Skip to content

feat(inkless:switch): elect a chosen replica in phase 1 of the classic-to-diskless switch [KC-717] - #882

Open
giuseppelillo wants to merge 4 commits into
mainfrom
claude/kc-717-issue-065359
Open

giuseppelillo wants to merge 4 commits into
mainfrom
claude/kc-717-issue-065359

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

In phase 1 of the classic-to-diskless switch, only the replicas' local disks store the classic prefix, and the controller blocks unclean leader election. If the only ISR member loses its disk, the partition goes offline and no command brings it back:

  • If the log directory of the replica is offline, the controller elects no leader, because a phase 1 leader needs an online log directory.
  • If the broker returns with a log that ends below a committed seal, it leads the partition and marks it offline.
  • If the broker returns without a log for the partition after the seal commits, the controller keeps it as the leader, but ReplicaManager.applyLocalLeadersDelta doesn't create the partition. No broker serves it.

rearm re-sets the current leader and lower-seal keeps it, so neither command moves leadership to a replica that still has the classic prefix. This PR adds a command that does.

Add the elect subcommand

kafka-topic-switch.sh elect --topic <topic> --partition <partition> --replica <broker-id> elects the chosen replica as the leader of a partition in phase 1, even if the replica is outside the ISR. The command accepts --bootstrap-server or --bootstrap-controller.

The command sends AlterDisklessSwitch with a new LeaderId field and a seal offset of -1, which keeps the seal. The field joins version 0 of the request. The switch isn't used in production yet, so the request doesn't need a new version. AlterDisklessSwitchOptions.leaderId(int) sets the field, so the Admin interface keeps its methods.

On the controller, ReplicationControlManager.electPhaseOneLeader runs PartitionChangeBuilder with an unclean election and a leader predicate that accepts only the chosen replica:

  • If the replica is outside the ISR, the controller makes it the leader, resets the ISR to it, and sets the leader recovery state to RECOVERING.
  • If the replica is in the ISR, the controller makes it the leader and keeps the ISR.
  • The election runs whether or not the partition has a leader, with a pending switch or a committed seal.

The controller rejects the request with INVALID_REQUEST in these cases:

  • The partition isn't in phase 1.
  • The broker isn't a replica of the partition, is fenced, or has an offline log directory for its replica.
  • The broker already leads the partition.
  • The request has a seal offset other than -1, or a leader ID below -1.

The phase 1 guards on unclean leader election don't change. ElectLeaders with type UNCLEAN, the config check, and the automatic election paths still reject or skip the election.

The broker code doesn't change. A leader whose log ends at a committed seal serves the partition. A leader whose log ends below it marks the partition offline, and lower-seal then applies as it does after a last-known-leader election. A leader without a log for the partition doesn't create it, and the operator elects another replica.

CLASSIC_TO_DISKLESS_SWITCH.md has a new section, "Elect a chosen replica in phase 1", with the command, the rejections, the outcomes on the broker, and the limitations.

Limitations

  • The controller doesn't know replica log end offsets, so it doesn't check the log of the chosen replica.
  • With a pending switch, the elected leader seals at its own log end offset. A replica with a shorter log costs the partition the difference at the election.
  • The partition keeps only the records of the elected replica. This PR doesn't rebuild the part of the classic prefix that only remote storage has, which KC-717 also asks for.
  • A topic with one replica per partition has no other replica to elect.
  • No test in this PR takes a partition with a lowered seal into phase 2. KC-837 covers that path.

Test plan

  • ./gradlew :clients:checkstyleMain :metadata:checkstyleMain :metadata:checkstyleTest :tools:checkstyleMain :tools:checkstyleTest :core:checkstyleTest :core:spotlessJavaCheck :metadata:spotlessJavaCheck passes, and make vale reports no alerts.
  • These test classes pass: ReplicationControlManagerInklessTest, TopicSwitchCommandTest, RequestResponseTest, MessageTest, and InklessTopicTypeSwitcherClusterTest.
  • Controller: ReplicationControlManagerInklessTest elects a replica outside the ISR with a committed seal, with and without a live leader, and with a pending switch. It checks the leader, the ISR, the RECOVERING state, the leader epoch bump, and that the seal stays. It also elects an ISR member and checks that the ISR stays.
  • Controller rejections: the same class covers phase 2, a broker that isn't a replica, the current leader, a fenced replica, a replica with an offline log directory, a seal offset sent with the leader ID, and a negative leader ID. The fenced-replica test checks that a rejected election keeps the current leader.
  • Tool: TopicSwitchCommandTest checks that elect sends the replica with a seal offset of -1, rejects a negative replica before it calls the controller, accepts --bootstrap-controller, and requires --partition and --replica.
  • End to end: InklessTopicTypeSwitcherClusterTest.testElectedReplicaOutsideTheIsrRecoversAPhaseOnePartition switches a topic with two replicas and a committed seal. It shuts down the follower, then the leader, and restarts the follower. The partition has no leader. After elect, the follower leads with an ISR of one, a consumer reads the whole classic prefix, the partition takes diskless records, and the old leader rejoins the ISR after its restart. The test then creates the bucket of the tiered storage plugin, and the partition enters phase 2.
  • The cluster test needs the partition to stay in phase 1 until the election. It carries the tag blocks-remote-uploads, for which the setup configures the tiered storage plugin with a bucket that doesn't exist. Segment uploads fail until the test creates the bucket, which it does after the recovery. The test asserts that classicPrefixInRemoteStorage is unset before the election. In a local run with a temporary 15-second wait before that assertion, the test fails with the default bucket and passes with the missing one.
  • The barrier adds no class file on purpose. Gradle assigns the test classes of :core:test to its two forks in turn, by class file. An earlier version of this test added a nested class, which moved both 256-second cluster test classes to the same fork. :core:test then took 17 minutes instead of 11 in CI, and the test step reached its 30-minute limit.
  • checkstyle/suppressions.xml exempts InklessTopicTypeSwitcherClusterTest from ClassFanOutComplexity. The new test takes the class over the limit of 55.
  • No cluster test elects a replica while the switch is pending, a replica whose log ends below the seal, or a replica without a log.

🤖 Generated with Claude Code

…c-to-diskless switch [KC-717]

In phase 1 of the classic-to-diskless switch, only the replicas' local
disks store the classic prefix, and the controller blocks unclean leader
election. If the only ISR member loses its disk, the partition goes
offline, and neither `rearm` nor `lower-seal` moves leadership to a
replica that still has the prefix.

`kafka-topic-switch.sh elect --replica <broker-id>` elects the chosen
replica, even if it's outside the ISR. The command sends
`AlterDisklessSwitch` with a new `LeaderId` field and a seal offset of
`-1`. The controller runs an unclean election that accepts only that
replica: it resets the ISR to a replica outside the ISR and sets the
leader recovery state to `RECOVERING`. The election runs with or without
a leader, with a pending switch or a committed seal. The phase 1 guards
on unclean leader election don't change.

`ReplicationControlManagerInklessTest` and `TopicSwitchCommandTest`
cover the election and its rejections.
`InklessTopicTypeSwitcherClusterTest` recovers a partition end to end
after its last ISR member goes away.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test can still race with the zero-delay initial copy task, which may upload the sealed prefix and trigger phase 2 before the phase-1 assertion or election.

1 open issue
What changed in this PR

This PR adds operator-selected leader election to recover partitions stuck in phase 1 of the classic-to-diskless switch, without relaxing automatic unclean-election guards.

Changes:

  • Add the elect command and pass the chosen replica through the admin client and request protocol.
  • Validate and perform phase-1 elections while preserving the committed seal.
  • Document recovery behavior and limitations, and add controller, command, and cluster tests.
File Description
tools/​src/​test/​java/​org/​apache/​kafka/​tools/​TopicSwitchCommandTest.java Test election arguments and request options.
tools/​src/​main/​java/​org/​apache/​kafka/​tools/​TopicSwitchCommand.java Add the elect subcommand.
metadata/​src/​test/​java/​org/​apache/​kafka/​controller/​ReplicationControlManagerInklessTest.java Test election outcomes and rejection cases.
metadata/​src/​main/​java/​org/​apache/​kafka/​controller/​ReplicationControlManager.java Validate and elect a chosen phase-1 replica.
docs/​inkless/​CLASSIC_TO_DISKLESS_SWITCH.md Explain recovery commands, outcomes, and limitations.
core/​src/​test/​java/​kafka/​server/​InklessTopicTypeSwitcherClusterTest.java Exercise recovery through an out-of-ISR election.
clients/​src/​main/​resources/​common/​message/​AlterDisklessSwitchRequest.json Add the LeaderId request field.
clients/​src/​main/​java/​org/​apache/​kafka/​clients/​admin/​KafkaAdminClient.java Forward the chosen leader in requests.
clients/​src/​main/​java/​org/​apache/​kafka/​clients/​admin/​AlterDisklessSwitchOptions.java Expose the leader-selection option.
clients/​src/​main/​java/​org/​apache/​kafka/​clients/​admin/​Admin.java Document election semantics in the admin API.
checkstyle/​suppressions.xml Exempt the expanded cluster test from the fan-out limit.

Review effort: Balanced

… election test [KC-717]

The election test kept its partition in phase 1 with a one-hour remote
copy interval. The broker schedules the first copy run with no delay,
and the test didn't wait for that run, so a late first run could upload
the classic prefix and start phase 2 before the election.

The test now configures `GatedRemoteStorageManager`, a wrapper around
the tiered storage plugin that rejects segment uploads while the test
blocks them. The copy interval is back to the value the other tests
use. After the old leader rejoins the ISR, the test unblocks the uploads
and waits for phase 2.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The public API documentation incorrectly states that every election resets the ISR and should clarify that electing an existing ISR member preserves it.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Clarify ISR behavior for elections of existing ISR members

clients/​src/​main/​java/​org/​apache/​kafka/​clients/​admin/​Admin.java:387

This public API documentation says that every election resets the ISR, but electPhaseOneLeader preserves it when the chosen replica is already an ISR member, as testAlterDisklessSwitchElectsAnIsrMemberWithoutResettingTheIsr verifies. Describe both cases so callers don't expect an in-ISR election to remove the other replicas.

1 resolved since last review

Review effort: Balanced

…KC-717]

The Javadoc of `Admin.alterDisklessSwitch` said that every election
resets the ISR to the chosen replica. The controller resets the ISR only
for a replica outside the ISR, and keeps it for an ISR member. The
Javadoc now describes both cases.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed because the recovery command bypasses election safeguards and can discard committed data, while pending-switch and divergent or below-seal log recovery lack end-to-end coverage.

No open issues 🎉

Review effort: Balanced

…ead of a wrapper class [KC-717]

Gradle assigns the test classes of `:core:test` to its two forks in
turn, by class file. The nested `GatedRemoteStorageManager` class added
a class file, and every class after it moved to the other fork. Both
256-second cluster test classes then ran on the same fork. `:core:test`
took 17 minutes instead of 11, and the CI test step reached its
30-minute limit.

The election test now needs no new class. For this test, the setup
configures the tiered storage plugin with a bucket that doesn't exist,
so segment uploads fail and the partition stays in phase 1. The test
creates the bucket after the recovery and waits for phase 2.

Assisted-by: Claude Fable 5.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants