Skip to content

Allow non matching schemas on EXPORT PARTITION#2111

Open
k-morozov wants to merge 6 commits into
antalya-26.3from
feature/antalya/26.3/allow_non_matching_schema_export_partition
Open

Allow non matching schemas on EXPORT PARTITION#2111
k-morozov wants to merge 6 commits into
antalya-26.3from
feature/antalya/26.3/allow_non_matching_schema_export_partition

Conversation

@k-morozov

@k-morozov k-morozov commented Jul 23, 2026

Copy link
Copy Markdown

Closes: #1717

Problem. EXPORT PART/EXPORT PARTITION matched source and destination columns positionally, exactly like INSERT INTO dest SELECT * FROM src, and required the column count to match exactly. This is unnecessarily strict for a common
real scenario: a MergeTree table's schema grows over time (new trailing columns added), but older partitions still need to be exported to an external table (Iceberg/object storage) whose schema was fixed when it was created. Such exports were rejected outright with NUMBER_OF_COLUMNS_DOESNT_MATCH, with no way to just drop the newer columns and export the rest.

Solution. Added the export_merge_tree_part_schema_mismatch_mode setting (default strict, preserving the old behavior). The new ignore_extra_source_columns_by_position mode allows the source to have more columns than the destination: the extra trailing source columns (by declared/readable position, the same order INSERT SELECT uses) are dropped, both at synchronous validation time (verifyExportSchemaCastable, so a real mismatch still fails immediately at ALTER TABLE ... EXPORT time) and at actual data-export time (ExportPartTask::addExportConvertingActions, via a projection step before the existing positional CAST). The reverse direction — destination has more columns than source — is still always rejected in both modes.

Known follow-up. Ignored columns are still fully read and decompressed from disk before being discarded in the projection step, so no I/O is saved. Skipping the read up front isn't safe as a naive truncation: a kept
ALIAS/MATERIALIZED column can legally forward-reference an ignored one, so it would need a real dependency-graph walk instead. Left as a possible future optimization

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added the export_merge_tree_part_schema_mismatch_mode setting. With ignore_extra_source_columns_by_position (default: strict), EXPORT PART/EXPORT PARTITION allows a source table with extra trailing columns — they are simply
ignored instead of failing with NUMBER_OF_COLUMNS_DOESNT_MATCH.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov
k-morozov force-pushed the feature/antalya/26.3/allow_non_matching_schema_export_partition branch from 0d6a23a to 267e79f Compare July 24, 2026 07:07
@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown

Workflow [PR], commit [e74a863]

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov k-morozov changed the title [WIP] Allow non matching schemas on EXPORT PARTITION Allow non matching schemas on EXPORT PARTITION Jul 24, 2026
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov
k-morozov marked this pull request as ready for review July 24, 2026 14:34
@arthurpassos

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: e74a863681

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@k-morozov

Copy link
Copy Markdown
Author

All current CI failures on this PR are pre-existing / unrelated to this PR's changes:

  • Docker server image, Grype Scan: Docker Hub outage (500 Internal Server Error from auth.docker.io) prevented pulling base images — infrastructure issue.
  • RegressionTestsRelease / Iceberg (2): almost all failures are EXPORT PARTITION is not implemented for engine MergeTree for the plain merge tree mode — expected, since that support isn't merged into antalya-26.3 yet; a handful of unrelated TABLE_ALREADY_EXISTS failures are known and already tracked.
  • RegressionTestsRelease / OAuth: a ModuleNotFoundError in the test suite itself, unrelated to this PR.
  • BuzzHouse (amd_debug): an unrelated LOGICAL_ERROR ("Inconsistent AST formatting") hit by a fuzzed query using GROUP BY ... WITH CUBE, unrelated to Iceberg/EXPORT PARTITION.

No failure traces back to the changes in this PR.

Comment thread src/Core/SettingsEnums.h
enum class MergeTreePartExportSchemaMismatchMode : uint8_t
{
strict,
ignore_extra_source_columns_by_position,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think name is confusing.
I expect that I can say "ignore columns at positions 3, 5 and 7", but actually it ignores last columns.
May be something like ignore_trailing_extra_columns?
/I'm not good in naming anyway/

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought to use 2 modes: ignore_extra_source_columns_by_position and ignore_extra_source_columns_by_name (in the future). Thank you for the example — I agree that the current naming is really confusing.

In the current name, the _by_position suffix meant that all columns from the prefix are matched 1-to-1, and the rest are ignored. I like the name ignore_trailing_extra_columns as the base. I'll suggest adding information that this is about the source. But I also wanted to convey that the columns in the prefix are matched sequentially:

ignore_trailing_extra_source_columns_match_by_sequence

node.query(f"DROP TABLE IF EXISTS {iceberg}")


def test_export_part_with_materialized_column(cluster):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to better understanding. Are this and next tests related to changes in current PR somehow? ignore_extra_source_columns_by_position is not used here, and tests cover different features.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, most of tests in this file are additional. However, some tests (eg test_export_part_ignore_extra_setting_drops_trailing_alias_column) show that simple resizing columns_to_read isn't enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow non matching schemas on export partition

3 participants