Repository navigation
fix(inkless:metadata): route consolidating partitions with one replica to the replica - #844
Conversation
…a to the replica [KC-651] `InklessTopicMetadataTransformer` picked the routing path from the replica count alone, so a partition with one replica always hashed over all alive brokers. Only the replica stores the local log of a consolidating topic, so consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata refresh returned the same broker again. A consolidating topic gets one replica when an untiered diskless topic created without managed replicas migrates to consolidation. The transformer now routes a partition to its replicas when it has more than one replica or when remote storage is on: same-AZ replica, then cross-AZ replica, then unavailable. Untiered topics with one replica keep hash-based routing. The migration procedure no longer requires raising the replication factor before the alter of `remote.storage.enable`. ## Test plan - Add `SingleReplicaDisklessTiered` transformer tests for Metadata and DescribeTopicPartitions. - Add a consolidating topic with replication factor 1 to `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without this change. - Run the transformer tests, checkstyle, spotbugs, and `make vale`. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
🟢 Approval recommended
The single-replica routing update has no active review issues.
No open issues 🎉
What changed in this PR
This PR fixes client routing for single-replica consolidating topics in Inkless. Metadata responses now direct clients to the assigned replica, avoiding repeated NOT_LEADER_OR_FOLLOWER errors from brokers that don't host the local log.
Changes:
- Route remote-storage-enabled partitions to replicas at every replication factor, while preserving hash-based routing for untiered single-replica topics.
- Add unit and migration-test coverage for single-replica routing, cross-zone reads, and replica unavailability.
- Update migration guidance to make increasing the replication factor optional and explain the availability tradeoff.
| File | Description |
|---|---|
storage/inkless/src/test/java/io/aiven/inkless/metadata/InklessTopicMetadataTransformerTest.java |
Tests single-replica routing and offline behavior for both metadata APIs. |
storage/inkless/src/main/java/io/aiven/inkless/metadata/InklessTopicMetadataTransformer.java |
Uses replica routing whenever remote storage is enabled. |
docs/inkless/DISKLESS_CONSOLIDATION.md |
Updates migration steps and single-replica tradeoffs. |
docs/inkless/CLIENT-BROKER-AZ-ALIGNMENT.md |
Clarifies that legacy single-replica hashing applies only to untiered topics. |
core/src/test/java/kafka/server/InklessUntieredDisklessMigrationTest.java |
Adds single-replica consolidation to routing and read checks across client zones. |
Review effort: Balanced
|
What is the motivation of this change? Is it just to simplify the routing patterns? |
I've updated the PR description with the description of the issue. It's really a fix rather than a refactor, if a diskless partition only has 1 replica then the consumers in different AZs could not fetch data. |
…a to the replica [KC-651] (#844) `InklessTopicMetadataTransformer` picked the routing path from the replica count alone, so a partition with one replica always hashed over all alive brokers. Only the replica stores the local log of a consolidating topic, so consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata refresh returned the same broker again. A consolidating topic gets one replica when an untiered diskless topic created without managed replicas migrates to consolidation. The transformer now routes a partition to its replicas when it has more than one replica or when remote storage is on: same-AZ replica, then cross-AZ replica, then unavailable. Untiered topics with one replica keep hash-based routing. The migration procedure no longer requires raising the replication factor before the alter of `remote.storage.enable`. ## Test plan - Add `SingleReplicaDisklessTiered` transformer tests for Metadata and DescribeTopicPartitions. - Add a consolidating topic with replication factor 1 to `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without this change. - Run the transformer tests, checkstyle, spotbugs, and `make vale`. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…a to the replica [KC-651] (#844) `InklessTopicMetadataTransformer` picked the routing path from the replica count alone, so a partition with one replica always hashed over all alive brokers. Only the replica stores the local log of a consolidating topic, so consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata refresh returned the same broker again. A consolidating topic gets one replica when an untiered diskless topic created without managed replicas migrates to consolidation. The transformer now routes a partition to its replicas when it has more than one replica or when remote storage is on: same-AZ replica, then cross-AZ replica, then unavailable. Untiered topics with one replica keep hash-based routing. The migration procedure no longer requires raising the replication factor before the alter of `remote.storage.enable`. ## Test plan - Add `SingleReplicaDisklessTiered` transformer tests for Metadata and DescribeTopicPartitions. - Add a consolidating topic with replication factor 1 to `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without this change. - Run the transformer tests, checkstyle, spotbugs, and `make vale`. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…a to the replica [KC-651] (#844) `InklessTopicMetadataTransformer` picked the routing path from the replica count alone, so a partition with one replica always hashed over all alive brokers. Only the replica stores the local log of a consolidating topic, so consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata refresh returned the same broker again. A consolidating topic gets one replica when an untiered diskless topic created without managed replicas migrates to consolidation. The transformer now routes a partition to its replicas when it has more than one replica or when remote storage is on: same-AZ replica, then cross-AZ replica, then unavailable. Untiered topics with one replica keep hash-based routing. The migration procedure no longer requires raising the replication factor before the alter of `remote.storage.enable`. ## Test plan - Add `SingleReplicaDisklessTiered` transformer tests for Metadata and DescribeTopicPartitions. - Add a consolidating topic with replication factor 1 to `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without this change. - Run the transformer tests, checkstyle, spotbugs, and `make vale`. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Consumers of a consolidating diskless topic with one replica get
NOT_LEADER_OR_FOLLOWERand stay on the broker that returned it.A diskless topic can have replication factor 1. Managed replicas accepts it, and a topic created before managed replicas stays at one replica. With consolidation on, that topic consolidates from creation, or an existing topic consolidates when
remote.storage.enableis set later. Either way it keeps that one replica, and only the replica stores the local log that serves reads.InklessTopicMetadataTransformerchose its path from the replica count, so a partition with one replica kept hashing over every alive broker. A client sent to any other broker gotNOT_LEADER_OR_FOLLOWER. The hash is stable, so the metadata refresh returned the same broker and the client retried the same failure.The migration guide required a replication-factor increase before the alter of an untiered topic. A topic created consolidating at replication factor 1 had no such step, and consumers failed the same way. The transformer now routes a partition to its replicas whenever remote storage is on, at every replication factor: same-AZ replica, then cross-AZ replica, then unavailable. A partition with more than one replica already took that path. Untiered topics with one replica keep hash-based routing, because every broker can serve them. Raising the replication factor stays optional, for same-AZ reads and for availability while the replica's broker is down.
Test plan
SingleReplicaDisklessTieredtransformer tests for Metadata and DescribeTopicPartitions.testClientRoutingOfUntieredAndConsolidatedTopics, which fails without this change.make vale.Assisted-by: Claude Opus 5.5