feat(metadata:diskless): add controller metrics for diskless topics#503
Draft
jeqo wants to merge 5 commits intojeqo/pod-2001-diskless-managed-replicafrom
Draft
feat(metadata:diskless): add controller metrics for diskless topics#503jeqo wants to merge 5 commits intojeqo/pod-2001-diskless-managed-replicafrom
jeqo wants to merge 5 commits intojeqo/pod-2001-diskless-managed-replicafrom
Conversation
6ddf9d8 to
742bb4f
Compare
Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)
3e7a42b to
6fba0c9
Compare
When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)
6fba0c9 to
1f3c7e1
Compare
Add three new controller metrics to track diskless topics: - DisklessTopicCount: total number of diskless topics - DisklessUnmanagedReplicasTopicCount: diskless topics with RF=1 - DisklessManagedReplicasTopicCount: diskless topics with RF>1 These metrics update in real-time via delta processing, not just on snapshot load. For deleted topics, the diskless status is correctly looked up from prevImage since the config is already removed from newImage at that point. Also adds docker-compose overlay for testing managed replicas with 3 brokers across 3 availability zones.
…trics publisher This function is not used anymore.
742bb4f to
4de2bb4
Compare
82dbca5 to
5f41dde
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds controller metrics to track diskless topics, building on #492 (managed replicas).
Changes
New Metrics
DisklessTopicCountDisklessUnmanagedReplicasTopicCountDisklessManagedReplicasTopicCountLogging
Added INFO-level logging for operational visibility:
On snapshot load:
On new topic creation:
Implementation Details
prevImagesince config is already removed fromnewImageisDisklessTopicfunction parameter fromControllerMetadataMetricsPublisherconstructorTesting
Future Work (separate PRs)
Related