Repository navigation
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Active issues cause unnecessary retries for successful no-match responses and allow synchronous control-plane queries to block request handlers before timeout protection begins.
2 open issues
What changed in this PR
This PR guards timestamp-based ListOffsets lookups on unrebuilt consolidating replicas to prevent consumers from silently skipping remote-only data.
Changes:
- Defer potentially incomplete diskless matches with a version-compatible retriable error.
- Pass replica readiness and cross-tier earliest-offset information from
ReplicaManagerto the router. - Add tests for deferral, preserved responses, rebuilt replicas, and older request versions.
| File | Description |
|---|---|
core/src/test/scala/unit/kafka/server/DisklessFetchOffsetRouterTest.scala |
Adds timestamp-deferral and compatibility test cases. |
core/src/main/scala/kafka/server/ReplicaManager.scala |
Supplies replica readiness and earliest-offset callbacks. |
core/src/main/scala/kafka/server/DisklessFetchOffsetRouter.scala |
Guards diskless fallback results when replicas cannot search the remote tier. |
Review effort: Balanced
f0874b6 to
148c8b6
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Critical lookup flaws can skip matching remote offsets during a concurrent cache rebuild or with out-of-order timestamps, returning an incorrect offset or no match.
2 open issues
2 resolved since last review
Review effort: Balanced
| val tierRebuildPending = isConsolidatingPartition && !hasRebuiltTierState(topicPartition) | ||
| def guarded(lookup: () => Lookup): () => Lookup = | ||
| if (tierRebuildPending) () => deferPrefixMatch(lookup(), topicPartition, hasReadableRemoteLogCoverage, onListOffsetsDeferred, version) else lookup |
…[KC-719] A ListOffsets request with a timestamp on a consolidating topic asks the classic log first and falls back to the control plane on "no match". A replica that hasn't rebuilt its tier state can't search the remote tier: its local log is empty, or a rebuild cleared the leader epoch cache. It answers "no match" for a timestamp inside the remote-only prefix. The control plane knows only the WAL, so it returns the oldest WAL batch. The consumer starts after the prefix with no error. The router now checks the replica before and after the classic lookup. For a replica that hasn't rebuilt its tier state, a diskless answer becomes OFFSET_NOT_AVAILABLE (LEADER_NOT_AVAILABLE before ListOffsets v5, which the existing pending-error path uses too) when RLMM has a readable remote segment whose maximum timestamp is at or after the target, or when RLMM can't tell yet. Classic picks that segment first and then searches inside it, so the check holds for timestamps that don't grow with the offset. It also covers a "no match" from the control plane. A new topic with nothing tiered, a timestamp after all tiered data, and every request on a replica that has rebuilt keep their answer. EARLIEST, LATEST, and MAX_TIMESTAMP already go to the control plane and don't change. The second check covers a rebuild that starts while an async classic search runs: the rebuild clears the leader epoch cache and the search then finds nothing. A rebuild that starts and ends within one search still gets through. The check can defer a correct match while the WAL overlaps tiered data. It also defers a timestamp that only a segment below the log start can match. The client retries until the replica rebuilds, which is the same wait as for fetches. A new meter, ConsolidationDeferredListOffsetsRate, counts the deferred requests. Related to KC-673, which fixes the same skip for consumer fetches. Assisted-by: Claude Opus 5.5
148c8b6 to
30e7344
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The epoch-readiness predicate can pass after a failed or incomplete rebuild, causing timestamp searches to skip remote data and silently return a later WAL offset.
3 open issues
1 resolved since last review
Review effort: Balanced
| * yet too, because `UnifiedLog.fetchOffsetByTimestamp` skips the remote search for an empty log. | ||
| */ | ||
| private[server] def hasRebuiltTierState(topicPartition: TopicPartition): Boolean = | ||
| logManager.getLog(topicPartition).exists(log => !log.isEmpty && log.leaderEpochCache().earliestEntry().isPresent) |
| * yet too, because `UnifiedLog.fetchOffsetByTimestamp` skips the remote search for an empty log. | ||
| */ | ||
| private[server] def hasRebuiltTierState(topicPartition: TopicPartition): Boolean = | ||
| logManager.getLog(topicPartition).exists(log => !log.isEmpty && log.leaderEpochCache().earliestEntry().isPresent) |
|
Closing in favor of #870 |
A
ListOffsetsrequest with a timestamp on a consolidating topic asks the classic log first and falls back to the control plane on "no match". A replica that has not rebuilt its tier state cannot search the remote tier: its local log is empty, or a rebuild cleared the leader epoch cache. It answers "no match" for a timestamp inside the remote-only prefix. The control plane knows only the WAL, so it returns the oldest WAL batch, and the consumer starts after the prefix with no error.The router now checks the replica before and after the classic lookup. For a replica that has not rebuilt its tier state, a diskless answer becomes
OFFSET_NOT_AVAILABLE(LEADER_NOT_AVAILABLEbeforeListOffsetsv5, like the existing pending-error path) when RLMM has a readable remote segment whose maximum timestamp is at or after the target, or when RLMM can't tell yet. Classic picks that segment first and then searches inside it, so the check holds for timestamps that don't grow with the offset. It also covers a "no match" from the control plane. A new topic with nothing tiered, a timestamp after all tiered data, and every request on a replica that has rebuilt keep their answer.EARLIEST,LATEST, andMAX_TIMESTAMPalready go to the control plane and do not change.The second check covers a rebuild that starts while an async classic search runs. A rebuild that starts and ends within one search still gets through.
The check can defer a correct match while the WAL overlaps tiered data, and a timestamp that only a segment below the log start can match. The client retries until the replica rebuilds, which is the same wait as for fetches.
Metrics
New meter under
kafka.server:type=ReplicaManager:ConsolidationDeferredListOffsetsRate, the rate of timestampListOffsetsrequests answered with the retriable error. A rate greater than zero for longer than a rebuild takes points at a replica that does not recover, or at RLMM that can't tell yet.Testing
DisklessFetchOffsetRouterTestcovers the deferral when a remote segment reaches the timestamp, RLMM that can't tell yet, a control-plane "no match" and an empty result in both cases, a rebuild that starts during an async classic search, a rebuilt replica, an error from the control plane, the pre-v5 error code, and the meter callback. The deferral tests fail without the check.RemoteLogManagerTestcovers the newhasReadableRemoteSegmentWithMaxTimestampAtLeastfor an unregistered partition, RLMM not ready, no segments, finished and transitional segments, and a failed listing.ReplicaManagerInklessTestcovershasRebuiltTierStatefor no log, an empty log, and a log with records.KC-719