test(inkless:consolidation): DeleteRecords across tiers e2e test [KC-332]#712
Draft
viktorsomogyi wants to merge 2 commits into
Draft
test(inkless:consolidation): DeleteRecords across tiers e2e test [KC-332]#712viktorsomogyi wants to merge 2 commits into
viktorsomogyi wants to merge 2 commits into
Conversation
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
from
July 21, 2026 15:02
474ab37 to
98c8387
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an end-to-end ducktape regression suite validating DeleteRecords semantics for Inkless consolidating topics across classic+remote tiers, including broker-to-broker earliest-offset consistency and post-failover safety.
Changes:
- Add a new e2e test covering cross-tier
DeleteRecordson switched (hybrid) consolidating topics, including leader failover and surviving-prefix readback verification. - Add verifier helpers to support the test: per-broker bootstrap pinning, per-broker earliest queries + convergence waiting,
DeleteRecordswrapper, born-consolidated topic creation, and a “tiered object count stabilized” waiter.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/kafkatest/tests/inkless/consolidation_delete_records_across_tiers_test.py | New e2e test suite for cross-tier DeleteRecords correctness on switched and born-consolidated topics, including failover coverage. |
| tests/kafkatest/services/inkless/consolidation_verifier.py | Adds helper APIs used by the new test (delete-records runner, per-broker earliest checks, born-consolidated topic creation, tiered-count stabilization waiter). |
Comment on lines
+266
to
+272
| state = {"last": -1, "stable": 0, "value": 0} | ||
|
|
||
| def check(): | ||
| cur = self.tiered_object_count() | ||
| state["stable"] = state["stable"] + 1 if cur == state["last"] else 0 | ||
| state["last"] = cur | ||
| state["value"] = cur |
| values = set(per_broker.values()) | ||
| agreed = len(values) == 1 and all(v > 0 for v in values) | ||
| cur = next(iter(values)) if agreed else None | ||
| state["stable"] = state["stable"] + 1 if (agreed and cur == state["last"]) else 0 |
viktorsomogyi
force-pushed
the
svv/ts-unification-reclaim-floor
branch
2 times, most recently
from
July 22, 2026 13:48
654af13 to
2336424
Compare
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
from
July 22, 2026 13:52
98c8387 to
cdfdd1c
Compare
viktorsomogyi
force-pushed
the
svv/ts-unification-reclaim-floor
branch
2 times, most recently
from
July 23, 2026 07:59
ea588ff to
66ddbb7
Compare
viktorsomogyi
changed the base branch from
svv/ts-unification-reclaim-floor
to
main
July 23, 2026 08:32
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
2 times, most recently
from
July 23, 2026 10:01
3d4a2a3 to
d52365c
Compare
viktorsomogyi
changed the base branch from
main
to
svv/ts-unification-reclaim-floor
July 23, 2026 10:06
viktorsomogyi
force-pushed
the
svv/ts-unification-reclaim-floor
branch
4 times, most recently
from
July 24, 2026 12:17
f6c0f80 to
e3f351d
Compare
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
from
July 24, 2026 12:17
d52365c to
796731c
Compare
viktorsomogyi
force-pushed
the
svv/ts-unification-reclaim-floor
branch
from
July 24, 2026 13:07
e3f351d to
5079dcd
Compare
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
from
July 24, 2026 13:11
796731c to
b17bfd0
Compare
…r earliest [KC-332] Followers keep a stale local logStartOffset (frozen at the seal) after a switch, so an older consumer routed there could read logically-deleted records. Consult the control-plane cross-tier earliest and reject such fetches with OFFSET_OUT_OF_RANGE (fail-open when unavailable). Adds unit tests. Co-authored-by: Cursor <cursoragent@cursor.com>
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
from
July 24, 2026 13:46
b17bfd0 to
c617830
Compare
…332] Adds consolidation_delete_records_across_tiers_test.py, the end-to-end regression for Problem A and Problem B on a switched consolidating topic: after DeleteRecords the earliest offset advances off 0, is identical on every broker, settles exactly at the delete boundary, and the surviving classic prefix [delete_before, seal) reads back contiguous. Also covers leader-failover metadata-propagation timing and born-consolidated topics. Adds the verifier helpers these need: delete_records, per-broker earliest_on_each_broker / wait_for_consistent_earliest_across_brokers, create_consolidated_topic, _broker_bootstrap, offset_at bootstrap pinning, and wait_for_tiered_count_stable (a settled pre-reclaim tiered-count peak). Co-authored-by: Cursor <cursoragent@cursor.com>
viktorsomogyi
force-pushed
the
svv/ts-unification-delete-records-e2e
branch
from
July 24, 2026 14:30
c617830 to
9bc2388
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds consolidation_delete_records_across_tiers_test.py, the end-to-end regression for Problem A and Problem B on a switched consolidating topic: after DeleteRecords the earliest offset advances off 0, is identical on every broker, settles exactly at the delete boundary, and the surviving classic prefix [delete_before, seal) reads back contiguous. Also covers leader-failover metadata-propagation timing and born-consolidated topics.
Adds the verifier helpers these need: delete_records, per-broker earliest_on_each_broker / wait_for_consistent_earliest_across_brokers, create_consolidated_topic, _broker_bootstrap, offset_at bootstrap pinning, and wait_for_tiered_count_stable (a settled pre-reclaim tiered-count peak).