Skip to content

fix(inkless:consolidation): defer timestamp lookups during a rebuild [KC-719] - #848

Closed
jeqo wants to merge 1 commit into
mainfrom
jeqo/kc-673-listoffsets-unrebuilt
Closed

jeqo wants to merge 1 commit into
mainfrom
jeqo/kc-673-listoffsets-unrebuilt

Conversation

@jeqo

@jeqo jeqo commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

A ListOffsets request with a timestamp on a consolidating topic asks the classic log first and falls back to the control plane on "no match". A replica that has not rebuilt its tier state cannot search the remote tier: its local log is empty, or a rebuild cleared the leader epoch cache. It answers "no match" for a timestamp inside the remote-only prefix. The control plane knows only the WAL, so it returns the oldest WAL batch, and the consumer starts after the prefix with no error.

The router now checks the replica before and after the classic lookup. For a replica that has not rebuilt its tier state, a diskless answer becomes OFFSET_NOT_AVAILABLE (LEADER_NOT_AVAILABLE before ListOffsets v5, like the existing pending-error path) when RLMM has a readable remote segment whose maximum timestamp is at or after the target, or when RLMM can't tell yet. Classic picks that segment first and then searches inside it, so the check holds for timestamps that don't grow with the offset. It also covers a "no match" from the control plane. A new topic with nothing tiered, a timestamp after all tiered data, and every request on a replica that has rebuilt keep their answer. EARLIEST, LATEST, and MAX_TIMESTAMP already go to the control plane and do not change.

The second check covers a rebuild that starts while an async classic search runs. A rebuild that starts and ends within one search still gets through.

The check can defer a correct match while the WAL overlaps tiered data, and a timestamp that only a segment below the log start can match. The client retries until the replica rebuilds, which is the same wait as for fetches.

Metrics

New meter under kafka.server:type=ReplicaManager: ConsolidationDeferredListOffsetsRate, the rate of timestamp ListOffsets requests answered with the retriable error. A rate greater than zero for longer than a rebuild takes points at a replica that does not recover, or at RLMM that can't tell yet.

Testing

DisklessFetchOffsetRouterTest covers the deferral when a remote segment reaches the timestamp, RLMM that can't tell yet, a control-plane "no match" and an empty result in both cases, a rebuild that starts during an async classic search, a rebuilt replica, an error from the control plane, the pre-v5 error code, and the meter callback. The deferral tests fail without the check. RemoteLogManagerTest covers the new hasReadableRemoteSegmentWithMaxTimestampAtLeast for an unregistered partition, RLMM not ready, no segments, finished and transitional segments, and a failed listing. ReplicaManagerInklessTest covers hasRebuiltTierState for no log, an empty log, and a log with records.

KC-719

@jeqo
jeqo requested a balanced review from Copilot November 2, 2026 11:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues cause unnecessary retries for successful no-match responses and allow synchronous control-plane queries to block request handlers before timeout protection begins.

2 open issues
What changed in this PR

This PR guards timestamp-based ListOffsets lookups on unrebuilt consolidating replicas to prevent consumers from silently skipping remote-only data.

Changes:

  • Defer potentially incomplete diskless matches with a version-compatible retriable error.
  • Pass replica readiness and cross-tier earliest-offset information from ReplicaManager to the router.
  • Add tests for deferral, preserved responses, rebuilt replicas, and older request versions.
File Description
core/​src/​test/​scala/​unit/​kafka/​server/​DisklessFetchOffsetRouterTest.scala Adds timestamp-deferral and compatibility test cases.
core/​src/​main/​scala/​kafka/​server/​ReplicaManager.scala Supplies replica readiness and earliest-offset callbacks.
core/​src/​main/​scala/​kafka/​server/​DisklessFetchOffsetRouter.scala Guards diskless fallback results when replicas cannot search the remote tier.

Review effort: Balanced

@jeqo
jeqo force-pushed the jeqo/kc-673-listoffsets-unrebuilt branch 2 times, most recently from f0874b6 to 148c8b6 Compare November 4, 2026 13:11
@jeqo jeqo changed the title fix(inkless:consolidation): defer timestamp lookups when unrebuilt [KC-673] fix(inkless:consolidation): defer timestamp lookups during a rebuild [KC-673] Nov 4, 2026
@jeqo
jeqo requested a balanced review from Copilot November 4, 2026 13:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical lookup flaws can skip matching remote offsets during a concurrent cache rebuild or with out-of-order timestamps, returning an incorrect offset or no match.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment on lines +166 to +168
val tierRebuildPending = isConsolidatingPartition && !hasRebuiltTierState(topicPartition)
def guarded(lookup: () => Lookup): () => Lookup =
if (tierRebuildPending) () => deferPrefixMatch(lookup(), topicPartition, hasReadableRemoteLogCoverage, onListOffsetsDeferred, version) else lookup
@jeqo jeqo changed the title fix(inkless:consolidation): defer timestamp lookups during a rebuild [KC-673] fix(inkless:consolidation): defer timestamp lookups during a rebuild [KC-719] Nov 4, 2026
…[KC-719]

A ListOffsets request with a timestamp on a consolidating topic asks the
classic log first and falls back to the control plane on "no match". A
replica that hasn't rebuilt its tier state can't search the remote
tier: its local log is empty, or a rebuild cleared the leader epoch
cache. It answers "no match" for a timestamp inside the remote-only
prefix. The control plane knows only the WAL, so it returns the oldest
WAL batch. The consumer starts after the prefix with no error.

The router now checks the replica before and after the classic lookup.
For a replica that hasn't rebuilt its tier state, a diskless answer
becomes OFFSET_NOT_AVAILABLE (LEADER_NOT_AVAILABLE before ListOffsets
v5, which the existing pending-error path uses too) when RLMM has a
readable remote segment whose maximum timestamp is at or after the
target, or when RLMM can't tell yet. Classic picks that segment first
and then searches inside it, so the check holds for timestamps that
don't grow with the offset. It also covers a "no match" from the
control plane. A new topic with nothing tiered, a timestamp after all
tiered data, and every request on a replica that has rebuilt keep their
answer. EARLIEST, LATEST, and MAX_TIMESTAMP already go to the control
plane and don't change.

The second check covers a rebuild that starts while an async classic
search runs: the rebuild clears the leader epoch cache and the search
then finds nothing. A rebuild that starts and ends within one search
still gets through.

The check can defer a correct match while the WAL overlaps tiered data.
It also defers a timestamp that only a segment below the log start can
match. The client retries until the replica rebuilds, which is the same
wait as for fetches. A new meter, ConsolidationDeferredListOffsetsRate,
counts the deferred requests.

Related to KC-673, which fixes the same skip for consumer fetches.

Assisted-by: Claude Opus 5.5
@jeqo
jeqo force-pushed the jeqo/kc-673-listoffsets-unrebuilt branch from 148c8b6 to 30e7344 Compare November 4, 2026 13:53
@jeqo
jeqo requested a balanced review from Copilot November 4, 2026 13:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The epoch-readiness predicate can pass after a failed or incomplete rebuild, causing timestamp searches to skip remote data and silently return a later WAL offset.

3 open issues
1 resolved since last review

Review effort: Balanced

* yet too, because `UnifiedLog.fetchOffsetByTimestamp` skips the remote search for an empty log.
*/
private[server] def hasRebuiltTierState(topicPartition: TopicPartition): Boolean =
logManager.getLog(topicPartition).exists(log => !log.isEmpty && log.leaderEpochCache().earliestEntry().isPresent)
* yet too, because `UnifiedLog.fetchOffsetByTimestamp` skips the remote search for an empty log.
*/
private[server] def hasRebuiltTierState(topicPartition: TopicPartition): Boolean =
logManager.getLog(topicPartition).exists(log => !log.isEmpty && log.leaderEpochCache().earliestEntry().isPresent)
@jeqo

jeqo commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of #870

@jeqo jeqo closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants