Repository navigation
test(inkless): pin ListOffsets answers on born-diskless topics before consolidation [KC-537] - #820
Merged
Merged
Conversation
… consolidation Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker. The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test. It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request. Assisted-by: Claude Opus 5.5
`FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch. `DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places. Assisted-by: Claude Opus 5.5
`DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it. Assisted-by: Claude Opus 5.5
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The tests comprehensively pin the documented behavior without production changes.
0 open findings
What changed in this PR
Adds integration coverage and documentation for ListOffsets behavior on unconsolidated born-diskless topics.
Changes:
- Tests all timestamp types across both brokers.
- Verifies consumer fetch and leader-epoch validation paths.
- Corrects consolidation and leader-epoch documentation.
| File | Description |
|---|---|
docs/inkless/FEATURES.md |
Documents diskless ListOffsets behavior. |
docs/inkless/DISKLESS_CONSOLIDATION.md |
Corrects local-offset and epoch descriptions. |
core/src/test/java/kafka/server/InklessDisklessListOffsetsTest.java |
Adds end-to-end contract coverage. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Assisted-by: Claude Sonnet 5.5
viktorsomogyi
approved these changes
Oct 12, 2026
jeqo
added a commit
that referenced
this pull request
Oct 19, 2026
… consolidation [KC-537] (#820) * test(inkless): pin ListOffsets answers on born-diskless topics before consolidation Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker. The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test. It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request. Assisted-by: Claude Opus 5.5 * docs(inkless): describe ListOffsets answers for diskless topics `FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch. `DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places. Assisted-by: Claude Opus 5.5 * docs(inkless): name the diskless leader epoch in the consolidation doc `DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it. Assisted-by: Claude Opus 5.5
jeqo
added a commit
that referenced
this pull request
Oct 19, 2026
… consolidation [KC-537] (#820) * test(inkless): pin ListOffsets answers on born-diskless topics before consolidation Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker. The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test. It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request. Assisted-by: Claude Opus 5.5 * docs(inkless): describe ListOffsets answers for diskless topics `FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch. `DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places. Assisted-by: Claude Opus 5.5 * docs(inkless): name the diskless leader epoch in the consolidation doc `DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it. Assisted-by: Claude Opus 5.5
jeqo
added a commit
that referenced
this pull request
Oct 19, 2026
… consolidation [KC-537] (#820) * test(inkless): pin ListOffsets answers on born-diskless topics before consolidation Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker. The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test. It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request. Assisted-by: Claude Opus 5.5 * docs(inkless): describe ListOffsets answers for diskless topics `FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch. `DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places. Assisted-by: Claude Opus 5.5 * docs(inkless): name the diskless leader epoch in the consolidation doc `DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an integration test that pins the ListOffsets answers a client gets for a born-diskless topic before consolidation, and document those answers in
FEATURES.md. Also correct twoEARLIEST_LOCALstatements inDISKLESS_CONSOLIDATION.md. No production code changes.Why now
#798 changes how the broker runs ListOffsets jobs: it moves the lock and bounds the thread pool. #799 rewrites
list_offsets_v1and adds an index for timestamp lookups. Both change the path that answers every ListOffsets request on a diskless topic. Before this PR, no test runs a client against that path on a born-diskless topic before consolidation.ListOffsetsJobTesthas one case, and the cluster tests that calllistOffsetsuse switched or consolidating topics. This test gives both PRs a fixed answer set to compare against. If a change alters what a client sees, this test fails.Why this coverage is enough for this API
On a born-diskless topic before consolidation, the batch coordinator answers every timestamp type. No answer depends on broker-local state. The client-visible contract is therefore the answer for each timestamp type, on each broker, plus the two consumer paths that depend on those answers. The test covers each of these:
EARLIEST,LATEST,EARLIEST_LOCAL,LATEST_TIERED,MAX_TIMESTAMP, andEARLIEST_PENDING_UPLOAD. For a concrete timestamp, it checks a hit inside the first batch, a hit that falls between batches, and a miss after the last batch.MAX_TIMESTAMPand for timestamp lookups. A change from batch-level to record-level answers, or the reverse, fails the test.0in every answer,-1forLATEST_TIERED, the log start forEARLIEST_LOCAL, andUNKNOWN_SERVER_ERRORforEARLIEST_PENDING_UPLOAD. These are documented as current behavior, not as a target.LATEST - 1on each broker returns the last record.auto.offset.reset=none. The test asserts that the brokers receive anOffsetsForLeaderEpochrequest and that the consumer reads the next records. If the epoch in the answers and the epoch in the fetch disagree, the consumer throws.Out of scope: concurrency and pool rejection (#798), query cost on large partitions (#799), and topics with switch or consolidation, which have their own cluster and system tests.
Commits
test(inkless): pin ListOffsets answers on born-diskless topics before consolidation: addsInklessDisklessListOffsetsTest.docs(inkless): describe ListOffsets answers for diskless topics: adds theLIST_OFFSETSdetails toFEATURES.mdand corrects theEARLIEST_LOCALstatements inDISKLESS_CONSOLIDATION.md. On a consolidating topic,DisklessFetchOffsetRouteranswersEARLIEST_LOCALandLATEST_TIEREDfrom the local log of the broker that receives the request.docs(inkless): name the diskless leader epoch in the consolidation doc: replaces the symbolE_dwith the field namedisklessLeaderEpochinDISKLESS_CONSOLIDATION.md.See the commit messages for the details of each step.