Skip to content

fix(inkless:metadata): route consolidating partitions with one replica to the replica - #844

Merged
viktorsomogyi merged 1 commit into
mainfrom
glillo/route-rf1-consolidating-to-replica
Nov 2, 2026
Merged

viktorsomogyi merged 1 commit into
mainfrom
glillo/route-rf1-consolidating-to-replica

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Consumers of a consolidating diskless topic with one replica get NOT_LEADER_OR_FOLLOWER and stay on the broker that returned it.

A diskless topic can have replication factor 1. Managed replicas accepts it, and a topic created before managed replicas stays at one replica. With consolidation on, that topic consolidates from creation, or an existing topic consolidates when remote.storage.enable is set later. Either way it keeps that one replica, and only the replica stores the local log that serves reads. InklessTopicMetadataTransformer chose its path from the replica count, so a partition with one replica kept hashing over every alive broker. A client sent to any other broker got NOT_LEADER_OR_FOLLOWER. The hash is stable, so the metadata refresh returned the same broker and the client retried the same failure.

The migration guide required a replication-factor increase before the alter of an untiered topic. A topic created consolidating at replication factor 1 had no such step, and consumers failed the same way. The transformer now routes a partition to its replicas whenever remote storage is on, at every replication factor: same-AZ replica, then cross-AZ replica, then unavailable. A partition with more than one replica already took that path. Untiered topics with one replica keep hash-based routing, because every broker can serve them. Raising the replication factor stays optional, for same-AZ reads and for availability while the replica's broker is down.

Test plan

  • Add SingleReplicaDisklessTiered transformer tests for Metadata and DescribeTopicPartitions.
  • Add a consolidating topic with replication factor 1 to testClientRoutingOfUntieredAndConsolidatedTopics, which fails without this change.
  • Run the transformer tests, checkstyle, spotbugs, and make vale.

Assisted-by: Claude Opus 5.5

…a to the replica [KC-651]

`InklessTopicMetadataTransformer` picked the routing path from the replica
count alone, so a partition with one replica always hashed over all alive
brokers. Only the replica stores the local log of a consolidating topic, so
consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata
refresh returned the same broker again. A consolidating topic gets one
replica when an untiered diskless topic created without managed replicas
migrates to consolidation.

The transformer now routes a partition to its replicas when it has more than
one replica or when remote storage is on: same-AZ replica, then cross-AZ
replica, then unavailable. Untiered topics with one replica keep hash-based
routing. The migration procedure no longer requires raising the replication
factor before the alter of `remote.storage.enable`.

## Test plan

- Add `SingleReplicaDisklessTiered` transformer tests for Metadata and
  DescribeTopicPartitions.
- Add a consolidating topic with replication factor 1 to
  `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without
  this change.
- Run the transformer tests, checkstyle, spotbugs, and `make vale`.

Assisted-by: Claude Opus 5.5
Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The single-replica routing update has no active review issues.

No open issues 🎉

What changed in this PR

This PR fixes client routing for single-replica consolidating topics in Inkless. Metadata responses now direct clients to the assigned replica, avoiding repeated NOT_LEADER_OR_FOLLOWER errors from brokers that don't host the local log.

Changes:

  • Route remote-storage-enabled partitions to replicas at every replication factor, while preserving hash-based routing for untiered single-replica topics.
  • Add unit and migration-test coverage for single-replica routing, cross-zone reads, and replica unavailability.
  • Update migration guidance to make increasing the replication factor optional and explain the availability tradeoff.
File Description
storage/​inkless/​src/​test/​java/​io/​aiven/​inkless/​metadata/​InklessTopicMetadataTransformerTest.java Tests single-replica routing and offline behavior for both metadata APIs.
storage/​inkless/​src/​main/​java/​io/​aiven/​inkless/​metadata/​InklessTopicMetadataTransformer.java Uses replica routing whenever remote storage is enabled.
docs/​inkless/​DISKLESS_CONSOLIDATION.md Updates migration steps and single-replica tradeoffs.
docs/​inkless/​CLIENT-BROKER-AZ-ALIGNMENT.md Clarifies that legacy single-replica hashing applies only to untiered topics.
core/​src/​test/​java/​kafka/​server/​InklessUntieredDisklessMigrationTest.java Adds single-replica consolidation to routing and read checks across client zones.

Review effort: Balanced

@giuseppelillo
giuseppelillo marked this pull request as ready for review October 29, 2026 12:25
@viktorsomogyi

Copy link
Copy Markdown
Contributor

What is the motivation of this change? Is it just to simplify the routing patterns?

@giuseppelillo

Copy link
Copy Markdown
Contributor Author

What is the motivation of this change? Is it just to simplify the routing patterns?

I've updated the PR description with the description of the issue. It's really a fix rather than a refactor, if a diskless partition only has 1 replica then the consumers in different AZs could not fetch data.

@viktorsomogyi
viktorsomogyi merged commit eea3c26 into main Nov 2, 2026
14 checks passed
@viktorsomogyi
viktorsomogyi deleted the glillo/route-rf1-consolidating-to-replica branch November 2, 2026 15:28
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…a to the replica [KC-651] (#844)

`InklessTopicMetadataTransformer` picked the routing path from the replica
count alone, so a partition with one replica always hashed over all alive
brokers. Only the replica stores the local log of a consolidating topic, so
consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata
refresh returned the same broker again. A consolidating topic gets one
replica when an untiered diskless topic created without managed replicas
migrates to consolidation.

The transformer now routes a partition to its replicas when it has more than
one replica or when remote storage is on: same-AZ replica, then cross-AZ
replica, then unavailable. Untiered topics with one replica keep hash-based
routing. The migration procedure no longer requires raising the replication
factor before the alter of `remote.storage.enable`.

## Test plan

- Add `SingleReplicaDisklessTiered` transformer tests for Metadata and
  DescribeTopicPartitions.
- Add a consolidating topic with replication factor 1 to
  `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without
  this change.
- Run the transformer tests, checkstyle, spotbugs, and `make vale`.

Assisted-by: Claude Opus 5.5

Co-authored-by: Cursor <cursoragent@cursor.com>
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…a to the replica [KC-651] (#844)

`InklessTopicMetadataTransformer` picked the routing path from the replica
count alone, so a partition with one replica always hashed over all alive
brokers. Only the replica stores the local log of a consolidating topic, so
consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata
refresh returned the same broker again. A consolidating topic gets one
replica when an untiered diskless topic created without managed replicas
migrates to consolidation.

The transformer now routes a partition to its replicas when it has more than
one replica or when remote storage is on: same-AZ replica, then cross-AZ
replica, then unavailable. Untiered topics with one replica keep hash-based
routing. The migration procedure no longer requires raising the replication
factor before the alter of `remote.storage.enable`.

## Test plan

- Add `SingleReplicaDisklessTiered` transformer tests for Metadata and
  DescribeTopicPartitions.
- Add a consolidating topic with replication factor 1 to
  `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without
  this change.
- Run the transformer tests, checkstyle, spotbugs, and `make vale`.

Assisted-by: Claude Opus 5.5

Co-authored-by: Cursor <cursoragent@cursor.com>
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…a to the replica [KC-651] (#844)

`InklessTopicMetadataTransformer` picked the routing path from the replica
count alone, so a partition with one replica always hashed over all alive
brokers. Only the replica stores the local log of a consolidating topic, so
consumers routed elsewhere got `NOT_LEADER_OR_FOLLOWER`, and the metadata
refresh returned the same broker again. A consolidating topic gets one
replica when an untiered diskless topic created without managed replicas
migrates to consolidation.

The transformer now routes a partition to its replicas when it has more than
one replica or when remote storage is on: same-AZ replica, then cross-AZ
replica, then unavailable. Untiered topics with one replica keep hash-based
routing. The migration procedure no longer requires raising the replication
factor before the alter of `remote.storage.enable`.

## Test plan

- Add `SingleReplicaDisklessTiered` transformer tests for Metadata and
  DescribeTopicPartitions.
- Add a consolidating topic with replication factor 1 to
  `testClientRoutingOfUntieredAndConsolidatedTopics`, which fails without
  this change.
- Run the transformer tests, checkstyle, spotbugs, and `make vale`.

Assisted-by: Claude Opus 5.5

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants