feat(inkless): POD-2456 Enable offset fetch in consolidating partitions - #594
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Enables ListOffsets routing support for consolidating diskless topics by extending DisklessFetchOffsetRouter to recognize consolidation-enabled topics and applying routing decisions similar to existing hybrid (classic+diskless) behavior.
Changes:
- Add a consolidation-enabled flag to
DisklessFetchOffsetRouterand wire it fromReplicaManagerconfig. - Update routing logic to treat consolidating diskless topics as hybrid and allow follower access on the classic leg.
- Add unit tests covering consolidating-topic routing scenarios, including migration-pending and classic-only timestamps.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| core/src/main/scala/kafka/server/DisklessFetchOffsetRouter.scala | Adds consolidation-aware routing decisions for ListOffsets and follower allowance in classic leg. |
| core/src/main/scala/kafka/server/ReplicaManager.scala | Wires disklessRemoteStorageConsolidationEnabled into the router constructor. |
| core/src/test/scala/unit/kafka/server/DisklessFetchOffsetRouterTest.scala | Adds tests for consolidating-topic ListOffsets routing behavior and updates router construction helper. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
viktorsomogyi
force-pushed
the
svv/ts-unification-local-lso
branch
3 times, most recently
from
May 13, 2026 14:24
7e2313e to
ca597a3
Compare
viktorsomogyi
force-pushed
the
svv/ts-unification-list-offsets
branch
2 times, most recently
from
May 14, 2026 08:51
a078cee to
7934455
Compare
viktorsomogyi
marked this pull request as ready for review
May 14, 2026 12:37
jeqo
previously approved these changes
May 18, 2026
jeqo
left a comment
Contributor
There was a problem hiding this comment.
LGTM, just a suggestion on a potential test case to add.
Use local log's log start offset instead of the one that is stored in the diskless coordinator. This is needed for consolidating partitions as they expect that the log start at the beginning of the already consolidated log that is in UnifiedLog. Fetching with FetchHandler returns the log_start_offset from the coordinator which is actually the diskless start offset. If we pass this on, then ReplicaManager will believe that log_start_offset is the first offset and therefore enforces retention. Instead of this behavior we should use the local log start offset to properly respect the offset boundaries and retention.
Enable list offsets API for consolidating partitions. When they fetch offsets, the decision should be similar to what is already done for diskless partitions that have classic log tails.
…rTest.scala Co-authored-by: Jorge Esteban Quilcate Otoya <jorge.quilcate@aiven.io>
viktorsomogyi
force-pushed
the
svv/ts-unification-list-offsets
branch
from
May 19, 2026 12:11
0e3b86b to
7d817a1
Compare
jeqo
approved these changes
May 19, 2026
giuseppelillo
pushed a commit
that referenced
this pull request
May 28, 2026
…ns (#594) * fix(inkless): POD-1965 Use local log start in DisklessLeaderEndPoint Use local log's log start offset instead of the one that is stored in the diskless coordinator. This is needed for consolidating partitions as they expect that the log start at the beginning of the already consolidated log that is in UnifiedLog. Fetching with FetchHandler returns the log_start_offset from the coordinator which is actually the diskless start offset. If we pass this on, then ReplicaManager will believe that log_start_offset is the first offset and therefore enforces retention. Instead of this behavior we should use the local log start offset to properly respect the offset boundaries and retention. * feat(inkless): POD-2456 Enable offset fetch in consolidating partitions Enable list offsets API for consolidating partitions. When they fetch offsets, the decision should be similar to what is already done for diskless partitions that have classic log tails. * Update core/src/test/scala/unit/kafka/server/DisklessFetchOffsetRouterTest.scala Co-authored-by: Jorge Esteban Quilcate Otoya <jorge.quilcate@aiven.io> --------- Co-authored-by: Jorge Esteban Quilcate Otoya <jorge.quilcate@aiven.io>
giuseppelillo
pushed a commit
that referenced
this pull request
May 29, 2026
…ns (#594) * fix(inkless): POD-1965 Use local log start in DisklessLeaderEndPoint Use local log's log start offset instead of the one that is stored in the diskless coordinator. This is needed for consolidating partitions as they expect that the log start at the beginning of the already consolidated log that is in UnifiedLog. Fetching with FetchHandler returns the log_start_offset from the coordinator which is actually the diskless start offset. If we pass this on, then ReplicaManager will believe that log_start_offset is the first offset and therefore enforces retention. Instead of this behavior we should use the local log start offset to properly respect the offset boundaries and retention. * feat(inkless): POD-2456 Enable offset fetch in consolidating partitions Enable list offsets API for consolidating partitions. When they fetch offsets, the decision should be similar to what is already done for diskless partitions that have classic log tails. * Update core/src/test/scala/unit/kafka/server/DisklessFetchOffsetRouterTest.scala Co-authored-by: Jorge Esteban Quilcate Otoya <jorge.quilcate@aiven.io> --------- Co-authored-by: Jorge Esteban Quilcate Otoya <jorge.quilcate@aiven.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable list offsets API for consolidating partitions. When they fetch offsets, the decision should be similar to what is already done for diskless partitions that have classic log tails.