Skip to content

feat(metadata:diskless): add controller metrics for diskless topics#503

Draft
jeqo wants to merge 5 commits intojeqo/pod-2001-diskless-managed-replicafrom
jeqo/pod-2001-controller-topic-metrics
Draft

feat(metadata:diskless): add controller metrics for diskless topics#503
jeqo wants to merge 5 commits intojeqo/pod-2001-diskless-managed-replicafrom
jeqo/pod-2001-controller-topic-metrics

Conversation

@jeqo
Copy link
Contributor

@jeqo jeqo commented Feb 9, 2026

Adds controller metrics to track diskless topics, building on #492 (managed replicas).

Changes

New Metrics

Metric Description
DisklessTopicCount Total number of diskless topics
DisklessUnmanagedReplicasTopicCount Diskless topics with RF=1 (legacy)
DisklessManagedReplicasTopicCount Diskless topics with RF>1 (managed)

Logging

Added INFO-level logging for operational visibility:

On snapshot load:

Diskless topics summary: total=5, managed=3, unmanaged=2
Diskless topics with unmanaged replicas (RF=1): [topic-a, topic-b]

On new topic creation:

Created diskless topic 'my-topic' with managed replicas (RF=3)
Created diskless topic 'legacy-topic' with unmanaged replicas (RF=1)

Implementation Details

  • Metrics update in real-time via delta processing (not just on snapshot load)
  • For deleted topics, diskless status is correctly looked up from prevImage since config is already removed from newImage
  • Removed unused isDisklessTopic function parameter from ControllerMetadataMetricsPublisher constructor

Testing

  • Unit tests for all new metrics
  • Tests for snapshot and delta processing paths

Future Work (separate PRs)

  • Placement quality metrics (non_rack_aware, rf_mismatch, single_rack partitions)
  • Transformer metrics (fallback_total, cross_az_routing)

Related

@jeqo jeqo force-pushed the jeqo/pod-2001-controller-topic-metrics branch from 6ddf9d8 to 742bb4f Compare February 9, 2026 14:48
Add diskless.managed.rf.enable config (default: false) to control whether
diskless topics use managed replicas with RF=rack_count or legacy RF=1.

This config only affects topic creation. When enabled, new diskless topics
will be created with one replica per rack using standard KRaft placement.

Part of Phase 1: Diskless Managed Replicas
(See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)
@jeqo jeqo force-pushed the jeqo/pod-2001-diskless-managed-replica branch 2 times, most recently from 3e7a42b to 6fba0c9 Compare February 9, 2026 18:02
When diskless.managed.rf.enable=true, new diskless topics are created with
RF=rack_count using standard KRaft replica placement instead of legacy RF=1.

Changes:
- Compute RF from rack cardinality via rackCardinality()
- Use standard replicaPlacer.place() for rack-aware assignment
- Allow manual replica assignments when managed replicas enabled
- Add checkstyle suppression for extended createTopic method

Phase 1 limitations:
- Add Partitions inherits RF from existing partitions (Phase 3)
- Transformer not updated, uses legacy routing (Phase 2)
- Integration tests deferred to Phase 2
(See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)
@jeqo jeqo force-pushed the jeqo/pod-2001-diskless-managed-replica branch from 6fba0c9 to 1f3c7e1 Compare February 9, 2026 19:06
jeqo added 3 commits February 9, 2026 21:36
Add three new controller metrics to track diskless topics:
- DisklessTopicCount: total number of diskless topics
- DisklessUnmanagedReplicasTopicCount: diskless topics with RF=1
- DisklessManagedReplicasTopicCount: diskless topics with RF>1

These metrics update in real-time via delta processing, not just on
snapshot load. For deleted topics, the diskless status is correctly
looked up from prevImage since the config is already removed from
newImage at that point.

Also adds docker-compose overlay for testing managed replicas with
3 brokers across 3 availability zones.
…trics publisher

This function is not used anymore.
@jeqo jeqo force-pushed the jeqo/pod-2001-controller-topic-metrics branch from 742bb4f to 4de2bb4 Compare February 9, 2026 19:36
@jeqo jeqo force-pushed the jeqo/pod-2001-diskless-managed-replica branch 3 times, most recently from 82dbca5 to 5f41dde Compare February 10, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant