Skip to content

fix(inkless:switch): serve pending ListOffsets from the classic log and defer lookups past a stale high watermark [KC-756] [KC-763] - #873

Merged
viktorsomogyi merged 1 commit into
mainfrom
claude/kc-756-763-list-offsets
Nov 16, 2026
Merged

viktorsomogyi merged 1 commit into
mainfrom
claude/kc-756-763-list-offsets

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

This PR fixes two cases where DisklessFetchOffsetRouter asks the control plane for offsets that the control plane can't answer correctly.

Pending switch on a consolidating topic

A partition whose switch is pending (classicToDisklessStartOffset = -2) took the classic-only path only if the topic didn't consolidate. The switch sets remote.storage.enable, and the topic reads as consolidating from that point. The pending partition then took the hybrid path:

  • EARLIEST_TIMESTAMP, LATEST_TIMESTAMP, and MAX_TIMESTAMP went to the control plane.
  • The control plane has no entry for the partition until the seal commits, and answers UNKNOWN_TOPIC_OR_PARTITION.
  • DelayedRemoteListOffsets maps that exception to the client response.

kafka-topic-switch.sh state calls listOffsets for the earliest and latest offsets of every partition, and failed on a pending partition.

A pending partition now takes the classic-only path on every topic. For example, with a classic log that holds offsets 0 to 999, LATEST returns 1000 and EARLIEST returns 0 while the switch is pending.

The leader and the in-sync followers answer client requests. InklessTopicMetadataTransformer advertises any alive ISR member as the leader while the classic prefix lives only on local disks, which includes the pending state. A leader-only lookup would answer NOT_LEADER_OR_FOLLOWER to a client that the transformer sent to a follower. A replica outside the ISR, and every replica fetcher request, still gets the leader-only lookup. This rule also applies to a pending partition of a topic that doesn't consolidate, which was leader-only before.

Stale high watermark on a hybrid partition

For a timestamp lookup, the router searches the local log first and asks the control plane on a miss. A client lookup hides a match at or past the last fetchable offset: classicFetchOffset answers NONE with offset -1. The router read that answer as a miss.

For example, a consolidating replica restarts with high watermark 1100 and log end offset 1300, and the WAL is pruned up to 1200. A consumer asks for a timestamp whose first match is offset 1150. The local search hides the match, the control plane searches from 1200 and returns 1200, and the consumer skips offsets 1150 to 1199 without an error.

The guard from #870 compares the searchable end of the local log with the start of the diskless log. It used the log end offset (1300), which is past the diskless start (1200). For a client request, classicSearchRange now ends at the high watermark, or at the last stable offset for read_committed. In the example, 1100 is before 1200, and the lookup gets OFFSET_NOT_AVAILABLE (LEADER_NOT_AVAILABLE before version 5) until the high watermark moves past the WAL start. A replica request keeps the log end offset.

The same bound covers a switched leader whose high watermark is below the seal: the existing seal check defers the lookup.

Not in this PR

After the controller commits the seal and until the leader initializes the control plane, ListOffsets for the latest offset still fails with UNKNOWN_TOPIC_OR_PARTITION. On a consolidating topic, the earliest offset fails the same way. This PR doesn't fall back to the classic log there. PostgresControlPlane runs ListOffsets on the read connection, and a read replica that lags behind the primary reports a missing entry for a partition that already holds diskless records. A fallback would then return a stale offset instead of an error.

No system test runs kafka-topic-switch.sh state on a pending partition.

Testing

  • DisklessFetchOffsetRouterTest covers a pending partition of a consolidating topic for the latest, earliest, max, and specific timestamps. It also covers a client request on an in-sync replica and a replica fetcher request on an in-sync replica.
  • ReplicaManagerInklessTest covers a pending partition of a consolidating topic on a real local log: the latest and earliest offsets on an in-sync follower, and NOT_LEADER_OR_FOLLOWER outside the ISR. The control plane gets no call.
  • ReplicaManagerInklessTest covers a timestamp lookup on a replica with high watermark 700, log end offset 900, and WAL start 800. A client lookup gets OFFSET_NOT_AVAILABLE for a hidden match and for a miss. A client lookup with high watermark 850 and a replica lookup get the control-plane offset.
  • With the high watermark bound turned off, the two cases that expect OFFSET_NOT_AVAILABLE fail.
  • DisklessFetchOffsetRouterTest and ReplicaManagerInklessTest pass.

🤖 Generated with Claude Code

Assisted-by: Claude Fable 5.2

…nd defer lookups past a stale high watermark [KC-756] [KC-763]

The router sent a pending partition of a consolidating topic to the control
plane, which has no entry for the partition until the seal commits. The
client got `UNKNOWN_TOPIC_OR_PARTITION`, and `kafka-topic-switch.sh state`
failed on the partitions an operator needs it for. A pending partition now
answers from the classic log on every topic. The leader and the in-sync
followers answer client requests, because the metadata transformer sends
clients to any alive ISR member while the switch is pending.

A client timestamp lookup hides a match at or past the last fetchable
offset. The router read that answer as a miss and returned the control-plane
offset, so a consumer skipped the records between a stale high watermark and
the start of the diskless log. The searchable range of a client lookup now
ends at the last fetchable offset, so the existing guard answers
`OFFSET_NOT_AVAILABLE` while that offset is before the diskless log.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the ListOffsets routing and timestamp-lookup changes.

No open issues 🎉

What changed in this PR

This PR fixes ListOffsets routing during classic-to-diskless switches and prevents timestamp lookups from skipping retained records when a replica’s high watermark is stale. It extends the lookup-gap protection introduced in #870.

Changes:

  • Route pending-switch lookups to the classic log, allowing in-sync followers to answer client requests.
  • Bound client timestamp searches by the high watermark or last stable offset, while replica requests retain the log end offset.
  • Add regression tests and document pending-switch behavior and lookup deferrals.
File Description
docs/​inkless/​DISKLESS_CONSOLIDATION.md Explain client search bounds and stale-watermark deferrals.
docs/​inkless/​CLASSIC_TO_DISKLESS_SWITCH.md Document pending-switch routing and post-seal lookup errors.
core/​src/​test/​scala/​unit/​kafka/​server/​ReplicaManagerInklessTest.scala Add real-log tests for stale watermarks and pending-switch follower access.
core/​src/​test/​scala/​unit/​kafka/​server/​DisklessFetchOffsetRouterTest.scala Cover pending-switch routing and client-versus-replica access.
core/​src/​main/​scala/​kafka/​server/​ReplicaManager.scala Apply isolation-aware search bounds and pass the broker ID to the router.
core/​src/​main/​scala/​kafka/​server/​DisklessFetchOffsetRouter.scala Route pending switches through classic storage with ISR-aware follower access.

Review effort: Balanced

@viktorsomogyi
viktorsomogyi merged commit 41f177a into main Nov 16, 2026
8 checks passed
@viktorsomogyi
viktorsomogyi deleted the claude/kc-756-763-list-offsets branch November 16, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants