feat(diskless): add managed replicas routing to metadata transformer#504
Draft
jeqo wants to merge 3 commits intojeqo/pod-2001-controller-topic-metricsfrom
Draft
feat(diskless): add managed replicas routing to metadata transformer#504jeqo wants to merge 3 commits intojeqo/pod-2001-controller-topic-metricsfrom
jeqo wants to merge 3 commits intojeqo/pod-2001-controller-topic-metricsfrom
Conversation
6ddf9d8 to
742bb4f
Compare
deb173b to
2d516d0
Compare
742bb4f to
4de2bb4
Compare
… List Change return type from Iterable<Node> to List<Node> for simpler downstream usage. The underlying KRaftMetadataCache already returns a List, so this removes unnecessary abstraction.
Add support for managed replicas (RF > 1) in diskless topic metadata transformation: - Add MetadataView.isRemoteStorageEnabled() to check topic tiering config - For managed replicas, routing priority depends on remote storage: - Tiered (remote storage enabled): same-AZ replica > cross-AZ replica > unavailable - Diskless-only: same-AZ replica > same-AZ any broker > cross-AZ replica > cross-AZ any broker - For unmanaged replicas (RF=1), preserve legacy hash-based selection Update existing tests to use RF=1 to preserve original unmanaged behavior.
Add metrics to track routing behavior in diskless topic metadata transformation: - fallback-total: Count of fallbacks to non-replica brokers (diskless-only) - offline-replicas-routed-around: Routing decisions when some replicas are offline - cross-az-routing-total: Requests routed to a different AZ than the client These metrics help operators monitor: - How often routing falls back from assigned replicas to any available broker - Impact of replica availability on routing decisions - Cross-AZ traffic patterns for capacity planning
2d516d0 to
8763a1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds AZ-aware routing support for diskless topics with managed replicas (RF > 1). The transformer now routes metadata requests based on replica assignments and remote storage configuration.
Changes
MetadataView.getAliveBrokerNodes()to returnList<Node>for consistent APIisRemoteStorageEnabledcheck to determine routing constraints:fallback-total: Count of fallbacks to non-replica brokersoffline-replicas-routed-around: Routing decisions when some replicas are offlinecross-az-routing-total: Requests routed to different AZ than clientRouting Priority
For unmanaged replicas (RF = 1), the legacy hash-based selection from all brokers is preserved.
Test plan