Skip to content

test(inkless): pin ListOffsets answers on born-diskless topics before consolidation [KC-537] - #820

Merged
jeqo merged 4 commits into
mainfrom
jeqo/list-offsets-diskless-baseline
Oct 12, 2026
Merged

jeqo merged 4 commits into
mainfrom
jeqo/list-offsets-diskless-baseline

Conversation

@jeqo

@jeqo jeqo commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Add an integration test that pins the ListOffsets answers a client gets for a born-diskless topic before consolidation, and document those answers in FEATURES.md. Also correct two EARLIEST_LOCAL statements in DISKLESS_CONSOLIDATION.md. No production code changes.

Why now

#798 changes how the broker runs ListOffsets jobs: it moves the lock and bounds the thread pool. #799 rewrites list_offsets_v1 and adds an index for timestamp lookups. Both change the path that answers every ListOffsets request on a diskless topic. Before this PR, no test runs a client against that path on a born-diskless topic before consolidation. ListOffsetsJobTest has one case, and the cluster tests that call listOffsets use switched or consolidating topics. This test gives both PRs a fixed answer set to compare against. If a change alters what a client sees, this test fails.

Why this coverage is enough for this API

On a born-diskless topic before consolidation, the batch coordinator answers every timestamp type. No answer depends on broker-local state. The client-visible contract is therefore the answer for each timestamp type, on each broker, plus the two consumer paths that depend on those answers. The test covers each of these:

  • Every timestamp type: EARLIEST, LATEST, EARLIEST_LOCAL, LATEST_TIERED, MAX_TIMESTAMP, and EARLIEST_PENDING_UPLOAD. For a concrete timestamp, it checks a hit inside the first batch, a hit that falls between batches, and a miss after the last batch.
  • Each broker: the cluster has two brokers in two racks, and the client AZ marker sends each run to a different broker. The test asserts which broker serves, so the two runs cannot test the same broker.
  • Batch granularity: the second batch holds its max timestamp on its second record. Record-level and batch-level answers then differ for MAX_TIMESTAMP and for timestamp lookups. A change from batch-level to record-level answers, or the reverse, fails the test.
  • The answers that differ from classic Kafka: leader epoch 0 in every answer, -1 for LATEST_TIERED, the log start for EARLIEST_LOCAL, and UNKNOWN_SERVER_ERROR for EARLIEST_PENDING_UPLOAD. These are documented as current behavior, not as a target.
  • Consumer fetch at LATEST - 1 on each broker returns the last record.
  • Leader epoch validation: a consumer commits a position with the epoch from a fetched record. A new consumer on the other broker starts from that position with auto.offset.reset=none. The test asserts that the brokers receive an OffsetsForLeaderEpoch request and that the consumer reads the next records. If the epoch in the answers and the epoch in the fetch disagree, the consumer throws.

Out of scope: concurrency and pool rejection (#798), query cost on large partitions (#799), and topics with switch or consolidation, which have their own cluster and system tests.

Commits

  • test(inkless): pin ListOffsets answers on born-diskless topics before consolidation: adds InklessDisklessListOffsetsTest.
  • docs(inkless): describe ListOffsets answers for diskless topics: adds the LIST_OFFSETS details to FEATURES.md and corrects the EARLIEST_LOCAL statements in DISKLESS_CONSOLIDATION.md. On a consolidating topic, DisklessFetchOffsetRouter answers EARLIEST_LOCAL and LATEST_TIERED from the local log of the broker that receives the request.
  • docs(inkless): name the diskless leader epoch in the consolidation doc: replaces the symbol E_d with the field name disklessLeaderEpoch in DISKLESS_CONSOLIDATION.md.

See the commit messages for the details of each step.

jeqo added 3 commits October 12, 2026 10:52
… consolidation

Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker.

The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test.

It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request.

Assisted-by: Claude Opus 5.5
`FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch.

`DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places.

Assisted-by: Claude Opus 5.5
`DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it.

Assisted-by: Claude Opus 5.5
@jeqo
jeqo requested a balanced review from Copilot October 12, 2026 08:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The tests comprehensively pin the documented behavior without production changes.

0 open findings

What changed in this PR

Adds integration coverage and documentation for ListOffsets behavior on unconsolidated born-diskless topics.

Changes:

  • Tests all timestamp types across both brokers.
  • Verifies consumer fetch and leader-epoch validation paths.
  • Corrects consolidation and leader-epoch documentation.
File Description
docs/​inkless/​FEATURES.md Documents diskless ListOffsets behavior.
docs/​inkless/​DISKLESS_CONSOLIDATION.md Corrects local-offset and epoch descriptions.
core/​src/​test/​java/​kafka/​server/​InklessDisklessListOffsetsTest.java Adds end-to-end contract coverage.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jeqo
jeqo requested a review from viktorsomogyi October 12, 2026 09:14
Comment thread docs/inkless/FEATURES.md Resolved
@jeqo
jeqo merged commit e466e58 into main Oct 12, 2026
7 checks passed
@jeqo
jeqo deleted the jeqo/list-offsets-diskless-baseline branch October 12, 2026 13:18
jeqo added a commit that referenced this pull request Oct 19, 2026
… consolidation [KC-537] (#820)

* test(inkless): pin ListOffsets answers on born-diskless topics before consolidation

Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker.

The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test.

It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request.

Assisted-by: Claude Opus 5.5

* docs(inkless): describe ListOffsets answers for diskless topics

`FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch.

`DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places.

Assisted-by: Claude Opus 5.5

* docs(inkless): name the diskless leader epoch in the consolidation doc

`DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it.

Assisted-by: Claude Opus 5.5
jeqo added a commit that referenced this pull request Oct 19, 2026
… consolidation [KC-537] (#820)

* test(inkless): pin ListOffsets answers on born-diskless topics before consolidation

Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker.

The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test.

It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request.

Assisted-by: Claude Opus 5.5

* docs(inkless): describe ListOffsets answers for diskless topics

`FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch.

`DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places.

Assisted-by: Claude Opus 5.5

* docs(inkless): name the diskless leader epoch in the consolidation doc

`DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it.

Assisted-by: Claude Opus 5.5
jeqo added a commit that referenced this pull request Oct 19, 2026
… consolidation [KC-537] (#820)

* test(inkless): pin ListOffsets answers on born-diskless topics before consolidation

Add `InklessDisklessListOffsetsTest`. It runs a two-broker cluster with managed replicas and a born-diskless topic before consolidation. The client AZ marker sends each request to each broker.

The test pins the answers that differ from classic Kafka: batch granularity for `MAX_TIMESTAMP` and timestamp lookups, leader epoch 0 in every answer, and `UNKNOWN_SERVER_ERROR` for `EARLIEST_PENDING_UPLOAD`. A change to one of these answers fails the test.

It also pins a fetch at `LATEST - 1` on each broker. A position committed at leader epoch 0 passes `OffsetsForLeaderEpoch` validation after a restart with `auto.offset.reset=none`. The broker request count shows that the consumer sent the validation request.

Assisted-by: Claude Opus 5.5

* docs(inkless): describe ListOffsets answers for diskless topics

`FEATURES.md` lists `LIST_OFFSETS` with no detail. Add batch granularity, the `EARLIEST_LOCAL`, `LATEST_TIERED`, and `EARLIEST_PENDING_UPLOAD` answers, and the leader epoch.

`DISKLESS_CONSOLIDATION.md` says `EARLIEST_LOCAL` returns the diskless WAL log start on a consolidating topic. `DisklessFetchOffsetRouter` sends `EARLIEST_LOCAL` and `LATEST_TIERED` to the local log of the broker that receives the request. Correct both places.

Assisted-by: Claude Opus 5.5

* docs(inkless): name the diskless leader epoch in the consolidation doc

`DISKLESS_CONSOLIDATION.md` uses the symbol `E_d` for the diskless leader epoch. The code and the KRaft field call it `disklessLeaderEpoch`. Use the name in prose and the field name where the doc introduces it.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants