feat(inkless): add dedicated rate limit for consolidation fetch [KC-145]#636
Merged
Conversation
jeqo
force-pushed
the
jeqo/ts-consolidation-rate-limit
branch
from
June 5, 2026 20:14
0548aab to
9831673
Compare
jeqo
force-pushed
the
jeqo/ts-consolidation-rate-limit
branch
from
June 5, 2026 23:39
9831673 to
fdc9e9a
Compare
Introduce a separate ReplicationQuotaManager for consolidation fetchers, decoupled from the follower replication quota. This allows operators to cap object-storage read bandwidth (bytes/sec) without affecting regular replication. Key changes: - Add DISKLESS_CONSOLIDATION_FETCH QuotaType (JMX: DisklessConsolidationFetch) - Add diskless.consolidation.fetch.rate.limit.bytes.per.second config - Wire dedicated quota manager via ConsolidationReconciler (marks topics throttled on partition add, ensuring bytes are recorded to the sensor) - Zip consolidationQuotaManager into fetcher manager creation (no silent fallback to shared follower quota) Config lives in ServerConfigs (not InklessConfig) because: - It is a broker-level config accessed through KafkaConfig, same layer as num.replica.fetchers and leader.replication.throttled.rate - It uses the diskless.consolidation.* namespace alongside the existing diskless.remote.storage.consolidation.enable config - InklessConfig is for storage module internals (fetch pools, cache, upload) consumed by SharedState/Reader/Writer — not for fetcher-layer wiring Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jeqo
force-pushed
the
jeqo/ts-consolidation-rate-limit
branch
from
June 9, 2026 10:38
fdc9e9a to
d0d005c
Compare
jeqo
marked this pull request as ready for review
June 9, 2026 10:39
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a dedicated replication quota for Inkless consolidation fetchers so operators can rate-limit object-storage read bandwidth (bytes/sec) independently of follower replication throttling.
Changes:
- Add a new
QuotaType.DISKLESS_CONSOLIDATION_FETCH(JMX:DisklessConsolidationFetch) and a broker configdiskless.consolidation.fetch.rate.limit.bytes.per.second. - Wire a dedicated
ReplicationQuotaManagerfor consolidation fetchers viaReplicaManager/ConsolidationFetcherManagerand pass it throughConsolidationReconciler. - Add unit tests validating quota behavior/independence and update reconciler tests for the new constructor parameter.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| server-common/src/main/java/org/apache/kafka/server/quota/QuotaType.java | Adds DISKLESS_CONSOLIDATION_FETCH quota type identifier for metrics/JMX grouping. |
| server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java | Introduces the new broker config key/default/doc and registers it in the config definition. |
| core/src/main/scala/kafka/server/KafkaConfig.scala | Exposes the new rate-limit config via KafkaConfig. |
| core/src/main/scala/kafka/server/ReplicaManager.scala | Creates a dedicated ReplicationQuotaManager for consolidation and wires it into fetcher manager + reconciler. |
| core/src/main/scala/io/aiven/inkless/consolidation/ConsolidationReconciler.scala | Marks consolidation topics as throttled so bytes are recorded against the dedicated quota. |
| core/src/test/scala/io/aiven/inkless/consolidation/ConsolidationReconcilerTest.scala | Updates test construction for the new reconciler dependency. |
| core/src/test/scala/io/aiven/inkless/consolidation/ConsolidationQuotaManagerTest.scala | Adds coverage for quota exceed/recovery, independence from follower quota, metrics exposure, and dynamic quota updates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
viktorsomogyi
approved these changes
Jun 10, 2026
jeqo
added a commit
that referenced
this pull request
Jun 10, 2026
…45] (#636) Introduce a separate ReplicationQuotaManager for consolidation fetchers, decoupled from the follower replication quota. This allows operators to cap object-storage read bandwidth (bytes/sec) without affecting regular replication. Key changes: - Add DISKLESS_CONSOLIDATION_FETCH QuotaType (JMX: DisklessConsolidationFetch) - Add diskless.consolidation.fetch.rate.limit.bytes.per.second config - Wire dedicated quota manager via ConsolidationReconciler (marks topics throttled on partition add, ensuring bytes are recorded to the sensor) - Zip consolidationQuotaManager into fetcher manager creation (no silent fallback to shared follower quota) Config lives in ServerConfigs (not InklessConfig) because: - It is a broker-level config accessed through KafkaConfig, same layer as num.replica.fetchers and leader.replication.throttled.rate - It uses the diskless.consolidation.* namespace alongside the existing diskless.remote.storage.consolidation.enable config - InklessConfig is for storage module internals (fetch pools, cache, upload) consumed by SharedState/Reader/Writer, not for fetcher-layer wiring --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
jeqo
added a commit
that referenced
this pull request
Jun 10, 2026
…45] (#636) Introduce a separate ReplicationQuotaManager for consolidation fetchers, decoupled from the follower replication quota. This allows operators to cap object-storage read bandwidth (bytes/sec) without affecting regular replication. Key changes: - Add DISKLESS_CONSOLIDATION_FETCH QuotaType (JMX: DisklessConsolidationFetch) - Add diskless.consolidation.fetch.rate.limit.bytes.per.second config - Wire dedicated quota manager via ConsolidationReconciler (marks topics throttled on partition add, ensuring bytes are recorded to the sensor) - Zip consolidationQuotaManager into fetcher manager creation (no silent fallback to shared follower quota) Config lives in ServerConfigs (not InklessConfig) because: - It is a broker-level config accessed through KafkaConfig, same layer as num.replica.fetchers and leader.replication.throttled.rate - It uses the diskless.consolidation.* namespace alongside the existing diskless.remote.storage.consolidation.enable config - InklessConfig is for storage module internals (fetch pools, cache, upload) consumed by SharedState/Reader/Writer, not for fetcher-layer wiring --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a separate ReplicationQuotaManager for consolidation fetchers, decoupled from the follower replication quota. This allows operators to cap object-storage read bandwidth (bytes/sec) without affecting regular replication.
Key changes:
Config lives in ServerConfigs (not InklessConfig) because: