Skip to content

Fix LOGICAL_ERROR when loading an existing Compact part under a non-adaptive table policy#111626

Open
groeneai wants to merge 1 commit into
ClickHouse:masterfrom
groeneai:groeneai-fix-load-compact-part-non-adaptive
Open

Fix LOGICAL_ERROR when loading an existing Compact part under a non-adaptive table policy#111626
groeneai wants to merge 1 commit into
ClickHouse:masterfrom
groeneai:groeneai-fix-load-compact-part-non-adaptive

Conversation

@groeneai

Copy link
Copy Markdown
Contributor

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fixed a LOGICAL_ERROR (Cannot create part with type Compact and storage type Full because table does not support polymorphic parts) raised when a MergeTree table loads a pre-existing Compact part while its current settings disable polymorphic parts (non-adaptive granularity). This could happen, for example, when a read-only table shares an on-disk path with another table and reloads its parts via SYSTEM RESTART DISK or on server startup. In debug and sanitizer builds the exception aborts the server.

Description

Found by the Stress test (amd_msan) check on an unrelated PR (STID 3286-0cd6), with the same error recurring across several unrelated PRs and check types (Stress msan/tsan/asan, Integration msan/tsan) over the last two months. No tracking issue existed.

Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=109360&sha=3791a7836dd7b2f089113ba6666a14b6f9c26492&name_0=PR&name_1=Stress%20test%20%28amd_msan%29

Root cause. MergeTreeData::canUsePolymorphicParts() is equivalent to canUseAdaptiveGranularity(). When a table's current settings make it false (non-adaptive granularity), the part LOAD path rejected an already-existing Compact part in two places, both deriving the part's validity from the table's current create-time policy rather than the part's physical on-disk reality:

  1. MergeTreeDataPartBuilder::build() threw the ... does not support polymorphic parts LOGICAL_ERROR directly.
  2. Even without (1), the IMergeTreeDataPart constructor initializes index_granularity_info from the current policy, building MarkType(adaptive=false, Compact), which throws Non-Wide data part type with non-adaptive granularity.

But a Compact part is always adaptive by invariant (see the MarkType constructor), and an existing on-disk part's type is a fact, not a choice. The current policy only governs the format chosen for NEW parts, which MergeTreeData::choosePartFormat() already enforces (it returns {Wide, Full} when polymorphic parts are disabled).

Fix.

  • MergeTreeIndexGranularityInfo(storage, settings, type): a Compact part type now always yields an adaptive mark type, regardless of the table's current policy. Wide keeps its policy-derived adaptivity (a Wide part may legitimately be non-adaptive). This covers both the load path and mutations that inherit a Compact source part's format.
  • MergeTreeDataPartBuilder::build(): removed the redundant polymorphic-parts guard. New-part format selection is enforced by choosePartFormat; the type reaching build() otherwise belongs to an existing part (read from disk, or inherited by a mutation), where it is a physical fact.

New-part creation behavior is unchanged. The change only makes ClickHouse load pre-existing Compact parts it currently rejects.

Added regression test 04371_load_compact_part_non_adaptive_reader.sh (a non-adaptive reader shares a plain_rewritable path with an adaptive writer and loads the writer's Compact part via SYSTEM RESTART DISK). It aborts the server without the fix and passes with it.

…daptive policy

canUsePolymorphicParts() is equivalent to canUseAdaptiveGranularity(). When a table's
current settings make it false (non-adaptive granularity), the part LOAD path rejected an
already-existing Compact part in two places, both deriving the part's validity from the
table's current create-time policy rather than the part's physical on-disk reality:

  1. MergeTreeDataPartBuilder::build() threw "... does not support polymorphic parts".
  2. Even without (1), the IMergeTreeDataPart constructor initializes index_granularity_info
     from the current policy, building MarkType(adaptive=false, Compact), which throws
     "Non-Wide data part type with non-adaptive granularity".

But a Compact part is always adaptive by invariant (see the MarkType constructor), and an
existing on-disk part's type is a fact, not a choice. The current policy only governs the
format chosen for NEW parts, which MergeTreeData::choosePartFormat() already enforces
(it returns {Wide, Full} when polymorphic parts are disabled). In debug and sanitizer
builds the exception aborts the server.

Reproduced by, for example, a read-only table that shares an on-disk path with another
table and reloads its parts via SYSTEM RESTART DISK or on server startup. Found by the
Stress test (amd_msan) check, recurring across several unrelated PRs and check types.

Fix:
  - MergeTreeIndexGranularityInfo(storage, settings, type): a Compact part type now always
    yields an adaptive mark type, regardless of the table's current policy; Wide keeps its
    policy-derived adaptivity. This covers the load path and mutations that inherit a
    Compact source part's format.
  - MergeTreeDataPartBuilder::build(): removed the redundant polymorphic-parts guard.
    New-part format selection is enforced by choosePartFormat; the type reaching build()
    otherwise belongs to an existing part where it is a physical fact.

New-part creation behavior is unchanged. Added regression test
04371_load_compact_part_non_adaptive_reader.sh.

Since a corrupt part that looks Compact is now caught while its columns are read (and
detached as broken) instead of aborting at construction, the debug/sanitizer skip in the
existing test_merge_tree_load_parts_filesystem_error is no longer needed and was removed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@groeneai

Copy link
Copy Markdown
Contributor Author
Internal second-model review — adjudication log (click to expand)

Pre-publication review by an independent model (engine: codex; 2 findings across 1 full pass + 1 delta re-review; bounded).

# Sev Finding Verdict Evidence / action
1 💡 Comments/test still say non-adaptive tables can't load Compact parts; the debug/sanitizer skip in test_merge_tree_load_parts_filesystem_error is obsolete AGREE — fixed Verified locally: corrupt "Compact-looking" part is now detached as broken (not a build-time abort), so the skip is unneeded and the test runs on all builds. Removed the skip + updated its rationale; clarified the MergeTreeDataPartCompact.h comment (new Compact parts aren't created for non-adaptive tables, but existing ones can be loaded).
2 ⚠️ A row-changing mutation of a Compact part under index_granularity_bytes = 0 could produce one-row granules DISAGREE Scenario is unreachable and this PR does not make it reachable: a fresh igb=0 table only creates Wide parts (choosePartFormat); ATTACH PARTITION of a Compact part is rejected ("inconsistent granularity with table"); the only loader of a Compact part under a non-adaptive policy is a shared plain_rewritable path whose holder must be readonly=true (mutations rejected with TABLE_IS_PERMANENTLY_READ_ONLY); a corruption-induced Compact-looking part is detached as broken, never mutated. Also, createMergeTreeIndexGranularity returns an empty adaptive object for Compact parts, so the cited 0 / size_of_row path is not taken. No change made (avoiding a speculative defensive edit).

Severity: ❌ blocker / ⚠️ major / 💡 nit. DISAGREE verdicts carry recorded evidence and are terminal per finding.

Session id: cron:clickhouse-author-slot-1:20260723-131900

@groeneai

Copy link
Copy Markdown
Contributor Author
Pre-PR validation gate (click to expand)
# Question Answer
a Deterministic repro? Yes. Two MergeTree tables share one plain_rewritable path; writer (adaptive) creates a Compact part; reader (index_granularity_bytes=0, non-adaptive) shares the path; SYSTEM RESTART DISK <reader-disk> reloads and loads the Compact part, aborting the server at MergeTreeDataPartBuilder::build(). Codified as 04371_load_compact_part_non_adaptive_reader.sh. Exact CI stack (STID 3286-0cd6) reproduced.
b Root cause explained? Yes. canUsePolymorphicParts() == canUseAdaptiveGranularity(). When it is false, the LOAD path rejects an existing Compact part in two spots deriving validity from the current create-time policy: (1) the build() guard throws directly; (2) the IMergeTreeDataPart ctor builds MarkType(adaptive=false, Compact) which throws. But a Compact part is always adaptive by invariant, and an existing part's type is a physical fact; new-part format is enforced separately by choosePartFormat.
c Fix matches root cause? Yes. (A) MergeTreeIndexGranularityInfo(storage,settings,type) forces adaptive=true for a Compact type (invariant), covering load + mutation-inherit. (B) removed the redundant create-time guard in build(). Directly targets the mechanism in (b).
d Test intent preserved / new tests added? Yes. New regression test 04371_.... No existing test weakened; choosePartFormat create-time enforcement untouched (sanity: small insert -> Compact; min_bytes_for_wide_part=0 -> Wide; igb=0 -> Wide). The existing test_merge_tree_load_parts_filesystem_error still detaches the corrupt part as broken (re-verified); its obsolete debug/sanitizer skip was removed so it now covers this fix on all builds.
e Both directions demonstrated? Yes. Baseline f2c8136d: test aborts the server (Received signal Aborted (6) at build():66). Fixed 1cbae537: passes (Compact / 1 Hello), server alive; 50/50 stable.
f Fix is general across code paths? Yes. Change A fixes the single point where mark-type adaptivity is decided, covering ALL load sites (13 withPartFormatFromDisk callers: load/clone/move/fetch/attach) AND mutation-inherit paths; Change B removes the guard for all callers. Not a symptom guard - it fixes the invariant at its origin.
g Fix generalizes across inputs? Yes. Keyed on part TYPE (the only dimension the guard/MarkType cares about): Compact (incl. cmrk4 with_substreams) -> adaptive by invariant; Wide keeps policy-derived adaptivity (a Wide part may legitimately be non-adaptive - unchanged). Verified Compact loads, non-adaptive table still creates Wide + queryable + survives DETACH/ATTACH.
h Backward compatible? Yes. Strictly more permissive: loads pre-existing Compact parts master currently rejects. No setting default / on-disk / wire / metadata format change; no new validation on existing data; no SettingsChangesHistory.cpp entry needed. New-part creation unchanged.
i Invariants and contracts preserved? Yes. The MarkType invariant "non-adaptive => Wide only" is upheld: Change A only ever sets adaptive=true for Compact, never adaptive=false for a non-Wide type. initializeIndexGranularityInfo still refines granularity from the on-disk mark type, so the member-init value is a consistent starting point. build()'s other assertions are untouched.

Session id: cron:clickhouse-author-slot-1:20260723-131900

@groeneai

Copy link
Copy Markdown
Contributor Author

cc @hanfei1991 @CheSema — could you review this? It fixes a LOGICAL_ERROR (server abort in debug/sanitizer builds) when a MergeTree table loads an already-existing Compact part while its current settings disable polymorphic parts (non-adaptive granularity). The load/construction path derived the part's validity from the table's current create-time policy instead of the part's on-disk reality; a Compact part is always adaptive by invariant, so it now loads regardless, while new-part format selection stays enforced by choosePartFormat.

@PedroTadim PedroTadim added the can be tested Allows running workflows for external contributors label Jul 23, 2026
@clickhouse-gh

clickhouse-gh Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [0083c8c]

Summary:

job_name test_name status info comment
Fast test FAIL
04371_load_compact_part_non_adaptive_reader FAIL cidb
Finish Workflow FAIL
python3 ./ci/jobs/scripts/workflow_hooks/new_tests_check.py FAIL
Code Review DROPPED
Fast test (arm_darwin) DROPPED
Build (amd_debug) DROPPED
Build (amd_asan_ubsan) DROPPED
Build (amd_tsan) DROPPED
Build (amd_msan) DROPPED
Build (amd_binary) DROPPED
Build (arm_debug) DROPPED

@clickhouse-gh clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants