Fix LOGICAL_ERROR when loading an existing Compact part under a non-adaptive table policy#111626
Conversation
…daptive policy
canUsePolymorphicParts() is equivalent to canUseAdaptiveGranularity(). When a table's
current settings make it false (non-adaptive granularity), the part LOAD path rejected an
already-existing Compact part in two places, both deriving the part's validity from the
table's current create-time policy rather than the part's physical on-disk reality:
1. MergeTreeDataPartBuilder::build() threw "... does not support polymorphic parts".
2. Even without (1), the IMergeTreeDataPart constructor initializes index_granularity_info
from the current policy, building MarkType(adaptive=false, Compact), which throws
"Non-Wide data part type with non-adaptive granularity".
But a Compact part is always adaptive by invariant (see the MarkType constructor), and an
existing on-disk part's type is a fact, not a choice. The current policy only governs the
format chosen for NEW parts, which MergeTreeData::choosePartFormat() already enforces
(it returns {Wide, Full} when polymorphic parts are disabled). In debug and sanitizer
builds the exception aborts the server.
Reproduced by, for example, a read-only table that shares an on-disk path with another
table and reloads its parts via SYSTEM RESTART DISK or on server startup. Found by the
Stress test (amd_msan) check, recurring across several unrelated PRs and check types.
Fix:
- MergeTreeIndexGranularityInfo(storage, settings, type): a Compact part type now always
yields an adaptive mark type, regardless of the table's current policy; Wide keeps its
policy-derived adaptivity. This covers the load path and mutations that inherit a
Compact source part's format.
- MergeTreeDataPartBuilder::build(): removed the redundant polymorphic-parts guard.
New-part format selection is enforced by choosePartFormat; the type reaching build()
otherwise belongs to an existing part where it is a physical fact.
New-part creation behavior is unchanged. Added regression test
04371_load_compact_part_non_adaptive_reader.sh.
Since a corrupt part that looks Compact is now caught while its columns are read (and
detached as broken) instead of aborting at construction, the debug/sanitizer skip in the
existing test_merge_tree_load_parts_filesystem_error is no longer needed and was removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Internal second-model review — adjudication log (click to expand)Pre-publication review by an independent model (engine: codex; 2 findings across 1 full pass + 1 delta re-review; bounded).
Severity: ❌ blocker / Session id: cron:clickhouse-author-slot-1:20260723-131900 |
Pre-PR validation gate (click to expand)
Session id: cron:clickhouse-author-slot-1:20260723-131900 |
|
cc @hanfei1991 @CheSema — could you review this? It fixes a |
|
Workflow [PR], commit [0083c8c] Summary: ❌
|
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fixed a
LOGICAL_ERROR(Cannot create part with type Compact and storage type Full because table does not support polymorphic parts) raised when aMergeTreetable loads a pre-existingCompactpart while its current settings disable polymorphic parts (non-adaptive granularity). This could happen, for example, when a read-only table shares an on-disk path with another table and reloads its parts viaSYSTEM RESTART DISKor on server startup. In debug and sanitizer builds the exception aborts the server.Description
Found by the
Stress test (amd_msan)check on an unrelated PR (STID 3286-0cd6), with the same error recurring across several unrelated PRs and check types (Stress msan/tsan/asan, Integration msan/tsan) over the last two months. No tracking issue existed.Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=109360&sha=3791a7836dd7b2f089113ba6666a14b6f9c26492&name_0=PR&name_1=Stress%20test%20%28amd_msan%29
Root cause.
MergeTreeData::canUsePolymorphicParts()is equivalent tocanUseAdaptiveGranularity(). When a table's current settings make itfalse(non-adaptive granularity), the part LOAD path rejected an already-existingCompactpart in two places, both deriving the part's validity from the table's current create-time policy rather than the part's physical on-disk reality:MergeTreeDataPartBuilder::build()threw the... does not support polymorphic partsLOGICAL_ERRORdirectly.IMergeTreeDataPartconstructor initializesindex_granularity_infofrom the current policy, buildingMarkType(adaptive=false, Compact), which throwsNon-Wide data part type with non-adaptive granularity.But a
Compactpart is always adaptive by invariant (see theMarkTypeconstructor), and an existing on-disk part's type is a fact, not a choice. The current policy only governs the format chosen for NEW parts, whichMergeTreeData::choosePartFormat()already enforces (it returns{Wide, Full}when polymorphic parts are disabled).Fix.
MergeTreeIndexGranularityInfo(storage, settings, type): aCompactpart type now always yields an adaptive mark type, regardless of the table's current policy.Widekeeps its policy-derived adaptivity (aWidepart may legitimately be non-adaptive). This covers both the load path and mutations that inherit aCompactsource part's format.MergeTreeDataPartBuilder::build(): removed the redundant polymorphic-parts guard. New-part format selection is enforced bychoosePartFormat; the type reachingbuild()otherwise belongs to an existing part (read from disk, or inherited by a mutation), where it is a physical fact.New-part creation behavior is unchanged. The change only makes ClickHouse load pre-existing
Compactparts it currently rejects.Added regression test
04371_load_compact_part_non_adaptive_reader.sh(a non-adaptive reader shares aplain_rewritablepath with an adaptive writer and loads the writer'sCompactpart viaSYSTEM RESTART DISK). It aborts the server without the fix and passes with it.