feat(inkless:consolidation): separate ConsolidationFetchBytesInPerSec from replication metric [KC-276] - #723
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a metrics accounting issue where consolidation fetcher throughput was being included in the inter-broker ReplicationBytesInPerSec meter (because both paths share ReplicaFetcherThread.processPartitionData). It introduces a dedicated consolidation meter so replication dashboards on consolidation-heavy brokers reflect true inter-broker replication traffic.
Changes:
- Add a
shouldRecordReplicationBytesInhook inReplicaFetcherThread(defaulttrue) and disable it inConsolidationFetcherThreadto prevent consolidation traffic from updatingReplicationBytesInPerSec. - Introduce a new broker-level meter
ConsolidationFetchBytesInPerSeconReplicaManager, marked fromConsolidationFetcherThreadusingLogAppendInfo.validBytes. - Add tests ensuring consolidation bytes are recorded and replication bytes are not recorded by consolidation fetchers.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| core/src/test/scala/io/aiven/inkless/consolidation/ConsolidationFetcherThreadTest.scala | Adds coverage for the new consolidation bytes meter and ensures replication bytes aren’t updated by consolidation fetchers. |
| core/src/main/scala/kafka/server/ReplicaManager.scala | Defines and registers the new ConsolidationFetchBytesInPerSec meter and exposes a recording method. |
| core/src/main/scala/kafka/server/ReplicaFetcherThread.scala | Adds the shouldRecordReplicationBytesIn hook and gates replication-bytes metering behind it. |
| core/src/main/scala/io/aiven/inkless/consolidation/ConsolidationFetcherThread.scala | Disables replication-bytes metering and records consolidation bytes based on append validBytes. |
jeqo
force-pushed
the
jeqo/kc-276-consolidation-fetch-metrics
branch
from
July 28, 2026 13:58
9452619 to
62144ad
Compare
… from replication metric ReplicationBytesInPerSec previously summed inter-broker replication AND consolidation fetcher appends, since both go through the AbstractFetcherThread processPartitionData path. That conflation made the metric meaningless on consolidation-heavy brokers and required operators to subtract the consolidation throughput to recover the inter-broker number. Add a shouldRecordReplicationBytesIn hook in ReplicaFetcherThread (default true) and override to false in ConsolidationFetcherThread. Add a new ConsolidationFetchBytesInPerSec meter on ReplicaManager, marked from ConsolidationFetcherThread.processPartitionData when validBytes > 0. Effect on existing dashboards: ReplicationBytesInPerSec on consolidation brokers will drop by the consolidation throughput (now reported separately). This is the corrected baseline. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jeqo
force-pushed
the
jeqo/kc-276-consolidation-fetch-metrics
branch
from
July 28, 2026 15:49
62144ad to
be1314e
Compare
jeqo
marked this pull request as ready for review
July 29, 2026 07:54
giuseppelillo
approved these changes
Jul 29, 2026
giuseppelillo
pushed a commit
that referenced
this pull request
Jul 29, 2026
… from replication metric (#723) ReplicationBytesInPerSec previously summed inter-broker replication AND consolidation fetcher appends, since both go through the AbstractFetcherThread processPartitionData path. That conflation made the metric meaningless on consolidation-heavy brokers and required operators to subtract the consolidation throughput to recover the inter-broker number. Add a shouldRecordReplicationBytesIn hook in ReplicaFetcherThread (default true) and override to false in ConsolidationFetcherThread. Add a new ConsolidationFetchBytesInPerSec meter on ReplicaManager, marked from ConsolidationFetcherThread.processPartitionData when validBytes > 0. Effect on existing dashboards: ReplicationBytesInPerSec on consolidation brokers will drop by the consolidation throughput (now reported separately). This is the corrected baseline. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
giuseppelillo
pushed a commit
that referenced
this pull request
Jul 30, 2026
… from replication metric (#723) ReplicationBytesInPerSec previously summed inter-broker replication AND consolidation fetcher appends, since both go through the AbstractFetcherThread processPartitionData path. That conflation made the metric meaningless on consolidation-heavy brokers and required operators to subtract the consolidation throughput to recover the inter-broker number. Add a shouldRecordReplicationBytesIn hook in ReplicaFetcherThread (default true) and override to false in ConsolidationFetcherThread. Add a new ConsolidationFetchBytesInPerSec meter on ReplicaManager, marked from ConsolidationFetcherThread.processPartitionData when validBytes > 0. Effect on existing dashboards: ReplicationBytesInPerSec on consolidation brokers will drop by the consolidation throughput (now reported separately). This is the corrected baseline. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ReplicationBytesInPerSec previously summed inter-broker replication AND consolidation fetcher appends, since both go through the AbstractFetcherThread processPartitionData path. That conflation made the metric meaningless on consolidation-heavy brokers and required operators to subtract the consolidation throughput to recover the inter-broker number.
Add a shouldRecordReplicationBytesIn hook in ReplicaFetcherThread (default true) and override to false in ConsolidationFetcherThread. Add a new ConsolidationFetchBytesInPerSec meter on ReplicaManager, marked from ConsolidationFetcherThread.processPartitionData when validBytes > 0.
Effect on existing dashboards: ReplicationBytesInPerSec on consolidation brokers will drop by the consolidation throughput (now reported separately). This is the corrected baseline.
KC-276