You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(inkless:switch): elect a chosen replica in phase 1 of the classic-to-diskless switch [KC-717] - #882
In phase 1 of the classic-to-diskless switch, only the replicas' local disks store the classic prefix, and the controller blocks unclean leader election. If the only ISR member loses its disk, the partition goes offline and no command brings it back:
If the log directory of the replica is offline, the controller elects no leader, because a phase 1 leader needs an online log directory.
If the broker returns with a log that ends below a committed seal, it leads the partition and marks it offline.
If the broker returns without a log for the partition after the seal commits, the controller keeps it as the leader, but ReplicaManager.applyLocalLeadersDelta doesn't create the partition. No broker serves it.
rearm re-sets the current leader and lower-seal keeps it, so neither command moves leadership to a replica that still has the classic prefix. This PR adds a command that does.
Add the elect subcommand
kafka-topic-switch.sh elect --topic <topic> --partition <partition> --replica <broker-id> elects the chosen replica as the leader of a partition in phase 1, even if the replica is outside the ISR. The command accepts --bootstrap-server or --bootstrap-controller.
The command sends AlterDisklessSwitch with a new LeaderId field and a seal offset of -1, which keeps the seal. The field joins version 0 of the request. The switch isn't used in production yet, so the request doesn't need a new version. AlterDisklessSwitchOptions.leaderId(int) sets the field, so the Admin interface keeps its methods.
On the controller, ReplicationControlManager.electPhaseOneLeader runs PartitionChangeBuilder with an unclean election and a leader predicate that accepts only the chosen replica:
If the replica is outside the ISR, the controller makes it the leader, resets the ISR to it, and sets the leader recovery state to RECOVERING.
If the replica is in the ISR, the controller makes it the leader and keeps the ISR.
The election runs whether or not the partition has a leader, with a pending switch or a committed seal.
The controller rejects the request with INVALID_REQUEST in these cases:
The partition isn't in phase 1.
The broker isn't a replica of the partition, is fenced, or has an offline log directory for its replica.
The broker already leads the partition.
The request has a seal offset other than -1, or a leader ID below -1.
The phase 1 guards on unclean leader election don't change. ElectLeaders with type UNCLEAN, the config check, and the automatic election paths still reject or skip the election.
The broker code doesn't change. A leader whose log ends at a committed seal serves the partition. A leader whose log ends below it marks the partition offline, and lower-seal then applies as it does after a last-known-leader election. A leader without a log for the partition doesn't create it, and the operator elects another replica.
CLASSIC_TO_DISKLESS_SWITCH.md has a new section, "Elect a chosen replica in phase 1", with the command, the rejections, the outcomes on the broker, and the limitations.
Limitations
The controller doesn't know replica log end offsets, so it doesn't check the log of the chosen replica.
With a pending switch, the elected leader seals at its own log end offset. A replica with a shorter log costs the partition the difference at the election.
The partition keeps only the records of the elected replica. This PR doesn't rebuild the part of the classic prefix that only remote storage has, which KC-717 also asks for.
A topic with one replica per partition has no other replica to elect.
No test in this PR takes a partition with a lowered seal into phase 2. KC-837 covers that path.
Test plan
./gradlew :clients:checkstyleMain :metadata:checkstyleMain :metadata:checkstyleTest :tools:checkstyleMain :tools:checkstyleTest :core:checkstyleTest :core:spotlessJavaCheck :metadata:spotlessJavaCheck passes, and make vale reports no alerts.
These test classes pass: ReplicationControlManagerInklessTest, TopicSwitchCommandTest, RequestResponseTest, MessageTest, and InklessTopicTypeSwitcherClusterTest.
Controller: ReplicationControlManagerInklessTest elects a replica outside the ISR with a committed seal, with and without a live leader, and with a pending switch. It checks the leader, the ISR, the RECOVERING state, the leader epoch bump, and that the seal stays. It also elects an ISR member and checks that the ISR stays.
Controller rejections: the same class covers phase 2, a broker that isn't a replica, the current leader, a fenced replica, a replica with an offline log directory, a seal offset sent with the leader ID, and a negative leader ID. The fenced-replica test checks that a rejected election keeps the current leader.
Tool: TopicSwitchCommandTest checks that elect sends the replica with a seal offset of -1, rejects a negative replica before it calls the controller, accepts --bootstrap-controller, and requires --partition and --replica.
End to end: InklessTopicTypeSwitcherClusterTest.testElectedReplicaOutsideTheIsrRecoversAPhaseOnePartition switches a topic with two replicas and a committed seal. It shuts down the follower, then the leader, and restarts the follower. The partition has no leader. After elect, the follower leads with an ISR of one, a consumer reads the whole classic prefix, the partition takes diskless records, and the old leader rejoins the ISR after its restart. The test then creates the bucket of the tiered storage plugin, and the partition enters phase 2.
The cluster test needs the partition to stay in phase 1 until the election. It carries the tag blocks-remote-uploads, for which the setup configures the tiered storage plugin with a bucket that doesn't exist. Segment uploads fail until the test creates the bucket, which it does after the recovery. The test asserts that classicPrefixInRemoteStorage is unset before the election. In a local run with a temporary 15-second wait before that assertion, the test fails with the default bucket and passes with the missing one.
The barrier adds no class file on purpose. Gradle assigns the test classes of :core:test to its two forks in turn, by class file. An earlier version of this test added a nested class, which moved both 256-second cluster test classes to the same fork. :core:test then took 17 minutes instead of 11 in CI, and the test step reached its 30-minute limit.
checkstyle/suppressions.xml exempts InklessTopicTypeSwitcherClusterTest from ClassFanOutComplexity. The new test takes the class over the limit of 55.
No cluster test elects a replica while the switch is pending, a replica whose log ends below the seal, or a replica without a log.
…c-to-diskless switch [KC-717]
In phase 1 of the classic-to-diskless switch, only the replicas' local
disks store the classic prefix, and the controller blocks unclean leader
election. If the only ISR member loses its disk, the partition goes
offline, and neither `rearm` nor `lower-seal` moves leadership to a
replica that still has the prefix.
`kafka-topic-switch.sh elect --replica <broker-id>` elects the chosen
replica, even if it's outside the ISR. The command sends
`AlterDisklessSwitch` with a new `LeaderId` field and a seal offset of
`-1`. The controller runs an unclean election that accepts only that
replica: it resets the ISR to a replica outside the ISR and sets the
leader recovery state to `RECOVERING`. The election runs with or without
a leader, with a pending switch or a committed seal. The phase 1 guards
on unclean leader election don't change.
`ReplicationControlManagerInklessTest` and `TopicSwitchCommandTest`
cover the election and its rejections.
`InklessTopicTypeSwitcherClusterTest` recovers a partition end to end
after its last ISR member goes away.
Assisted-by: Claude Fable 5.2
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The test can still race with the zero-delay initial copy task, which may upload the sealed prefix and trigger phase 2 before the phase-1 assertion or election.
This PR adds operator-selected leader election to recover partitions stuck in phase 1 of the classic-to-diskless switch, without relaxing automatic unclean-election guards.
Changes:
Add the elect command and pass the chosen replica through the admin client and request protocol.
Validate and perform phase-1 elections while preserving the committed seal.
Document recovery behavior and limitations, and add controller, command, and cluster tests.
… election test [KC-717]
The election test kept its partition in phase 1 with a one-hour remote
copy interval. The broker schedules the first copy run with no delay,
and the test didn't wait for that run, so a late first run could upload
the classic prefix and start phase 2 before the election.
The test now configures `GatedRemoteStorageManager`, a wrapper around
the tiered storage plugin that rejects segment uploads while the test
blocks them. The copy interval is back to the value the other tests
use. After the old leader rejoins the ISR, the test unblocks the uploads
and waits for phase 2.
Assisted-by: Claude Fable 5.2
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The public API documentation incorrectly states that every election resets the ISR and should clarify that electing an existing ISR member preserves it.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Clarify ISR behavior for elections of existing ISR members
This public API documentation says that every election resets the ISR, but electPhaseOneLeader preserves it when the chosen replica is already an ISR member, as testAlterDisklessSwitchElectsAnIsrMemberWithoutResettingTheIsr verifies. Describe both cases so callers don't expect an in-ISR election to remove the other replicas.
…KC-717]
The Javadoc of `Admin.alterDisklessSwitch` said that every election
resets the ISR to the chosen replica. The controller resets the ISR only
for a replica outside the ISR, and keeps it for an ISR member. The
Javadoc now describes both cases.
Assisted-by: Claude Fable 5.2
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human review is needed because the recovery command bypasses election safeguards and can discard committed data, while pending-switch and divergent or below-seal log recovery lack end-to-end coverage.
…ead of a wrapper class [KC-717]
Gradle assigns the test classes of `:core:test` to its two forks in
turn, by class file. The nested `GatedRemoteStorageManager` class added
a class file, and every class after it moved to the other fork. Both
256-second cluster test classes then ran on the same fork. `:core:test`
took 17 minutes instead of 11, and the CI test step reached its
30-minute limit.
The election test now needs no new class. For this test, the setup
configures the tiered storage plugin with a bucket that doesn't exist,
so segment uploads fail and the partition stays in phase 1. The test
creates the bucket after the recovery and waits for phase 2.
Assisted-by: Claude Fable 5.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In phase 1 of the classic-to-diskless switch, only the replicas' local disks store the classic prefix, and the controller blocks unclean leader election. If the only ISR member loses its disk, the partition goes offline and no command brings it back:
ReplicaManager.applyLocalLeadersDeltadoesn't create the partition. No broker serves it.rearmre-sets the current leader andlower-sealkeeps it, so neither command moves leadership to a replica that still has the classic prefix. This PR adds a command that does.Add the
electsubcommandkafka-topic-switch.sh elect --topic <topic> --partition <partition> --replica <broker-id>elects the chosen replica as the leader of a partition in phase 1, even if the replica is outside the ISR. The command accepts--bootstrap-serveror--bootstrap-controller.The command sends
AlterDisklessSwitchwith a newLeaderIdfield and a seal offset of-1, which keeps the seal. The field joins version 0 of the request. The switch isn't used in production yet, so the request doesn't need a new version.AlterDisklessSwitchOptions.leaderId(int)sets the field, so theAdmininterface keeps its methods.On the controller,
ReplicationControlManager.electPhaseOneLeaderrunsPartitionChangeBuilderwith an unclean election and a leader predicate that accepts only the chosen replica:RECOVERING.The controller rejects the request with
INVALID_REQUESTin these cases:-1, or a leader ID below-1.The phase 1 guards on unclean leader election don't change.
ElectLeaderswith typeUNCLEAN, the config check, and the automatic election paths still reject or skip the election.The broker code doesn't change. A leader whose log ends at a committed seal serves the partition. A leader whose log ends below it marks the partition offline, and
lower-sealthen applies as it does after a last-known-leader election. A leader without a log for the partition doesn't create it, and the operator elects another replica.CLASSIC_TO_DISKLESS_SWITCH.mdhas a new section, "Elect a chosen replica in phase 1", with the command, the rejections, the outcomes on the broker, and the limitations.Limitations
Test plan
./gradlew :clients:checkstyleMain :metadata:checkstyleMain :metadata:checkstyleTest :tools:checkstyleMain :tools:checkstyleTest :core:checkstyleTest :core:spotlessJavaCheck :metadata:spotlessJavaCheckpasses, andmake valereports no alerts.ReplicationControlManagerInklessTest,TopicSwitchCommandTest,RequestResponseTest,MessageTest, andInklessTopicTypeSwitcherClusterTest.ReplicationControlManagerInklessTestelects a replica outside the ISR with a committed seal, with and without a live leader, and with a pending switch. It checks the leader, the ISR, theRECOVERINGstate, the leader epoch bump, and that the seal stays. It also elects an ISR member and checks that the ISR stays.TopicSwitchCommandTestchecks thatelectsends the replica with a seal offset of-1, rejects a negative replica before it calls the controller, accepts--bootstrap-controller, and requires--partitionand--replica.InklessTopicTypeSwitcherClusterTest.testElectedReplicaOutsideTheIsrRecoversAPhaseOnePartitionswitches a topic with two replicas and a committed seal. It shuts down the follower, then the leader, and restarts the follower. The partition has no leader. Afterelect, the follower leads with an ISR of one, a consumer reads the whole classic prefix, the partition takes diskless records, and the old leader rejoins the ISR after its restart. The test then creates the bucket of the tiered storage plugin, and the partition enters phase 2.blocks-remote-uploads, for which the setup configures the tiered storage plugin with a bucket that doesn't exist. Segment uploads fail until the test creates the bucket, which it does after the recovery. The test asserts thatclassicPrefixInRemoteStorageis unset before the election. In a local run with a temporary 15-second wait before that assertion, the test fails with the default bucket and passes with the missing one.:core:testto its two forks in turn, by class file. An earlier version of this test added a nested class, which moved both 256-second cluster test classes to the same fork.:core:testthen took 17 minutes instead of 11 in CI, and the test step reached its 30-minute limit.checkstyle/suppressions.xmlexemptsInklessTopicTypeSwitcherClusterTestfromClassFanOutComplexity. The new test takes the class over the limit of 55.🤖 Generated with Claude Code