Repository navigation
fix(inkless:switch): seal and register a pending partition in the first delta after a restart [KC-752] - #877
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
Tests mock InitDisklessLogManager, leaving the crash/restart ordering between transaction sealing and aborts, broker registration, and controller epoch fencing unverified and requiring human validation.
No open issues 🎉
What changed in this PR
This PR lets a restarted leader resume a pending classic-to-diskless switch from its first metadata image, without waiting for another leader epoch or the old broker session to expire.
Changes:
- Route pending leaders through sealing, transaction abort, and
InitDisklessLogregistration even when noPartitionobject exists. - Add restart regression tests for consolidating and non-consolidating topics, and update sealing and registration expectations.
- Document how a restarted leader resumes a pending switch.
| File | Description |
|---|---|
docs/inkless/CLASSIC_TO_DISKLESS_SWITCH.md |
Explain pending-switch recovery on the first metadata image after restart. |
core/src/test/scala/unit/kafka/server/ReplicaManagerInklessTest.scala |
Cover first-delta sealing, registration, transaction abort, and consolidation-fetcher behavior. |
core/src/main/scala/kafka/server/ReplicaManager.scala |
Handle pending switches before consolidation routing and create missing partitions before sealing. |
Review effort: Balanced
282b4ff to
b5db59d
Compare
…st delta after a restart [KC-752] A broker that crashes and restarts before its session expires can apply a first metadata image that still shows it as the leader of a partition in `CLASSIC_TO_DISKLESS_SWITCH_PENDING`. No `Partition` object exists yet, so `applyLocalLeadersDelta` skipped the switch branch. For a consolidating topic, the consolidating branch made the broker the leader without sealing or registering, and no broker sent `InitDisklessLog` until the controller fenced the old session and bumped the leader epoch. A pending leader now always takes the switch branch, for a consolidating topic too. If the partition doesn't exist, the broker creates it, makes it the leader, seals it, aborts open transactions, and registers it. The post-restart branch handles only committed seals. Assisted-by: Claude Opus 5.5 Assisted-by: Claude Sonnet 5.5
b5db59d to
7830034
Compare
| case None => | ||
| // First delta after a restart while the switch is pending. The local log only becomes | ||
| // available after makeLeader, so the seal follows it. Produce can't append to the | ||
| // classic log in between, because it answers REPLICA_NOT_AVAILABLE for a pending | ||
| // partition before it reaches the log. | ||
| getOrCreatePartition(tp, delta, info.topicId).map { case (partition, isNew) => | ||
| partition.makeLeader(info.partition, isNew, offsetCheckpoints, Some(info.topicId), partitionAssignedDirectoryId) | ||
| partition.seal() | ||
| partition |
There was a problem hiding this comment.
Here makeLeader creates an empty log with HW = LEO = 0 when the broker has no local log on disk, seals it, and calls registerPartition to propose seal 0. On a wiped disk, creating that empty log also causes maybeUpdateTopicAssignment to move the partition's directory assignment to the new disk once the broker registers. If the controller then elects this broker via lastKnownElr or ELR, or if only the partition directory is missing on an online log directory, the controller commits seal 0 and the followers truncate away their classic prefix. Before this PR, non-consolidating partitions required logManager.getLog(tp).isDefined, and consolidating partitions didn't call registerPartition.
We may check logManager.getLog(tp).isDefined before creating the partition. If the local log is missing, log a warning and skip the partition, matching applyLocalFollowersDelta for non-consolidating pending switches. Add a test for consolidating and non-consolidating topics that applies a pending first delta without a local log and verifies that ReplicaManager doesn't create a log or call registerPartition.
…752] A broker can lead a pending switch without a local log, such as after a disk replacement or when its partition directory is missing and an ELR election picks it. The switch branch then created the partition, and makeLeader created an empty log. registerPartition proposed that log's high watermark of 0 as the seal, and the controller committed it, so the followers truncated the classic prefix they hold. If the local log is missing, the switch branch now logs a warning and skips the partition, as applyLocalFollowersDelta does for a pending switch. The broker doesn't create a log or register the partition. Assisted-by: Claude Opus 5.5
A broker that crashes and restarts while a classic-to-diskless switch is pending can apply a first metadata image that still shows it as the leader in the same leader epoch. The controller rejects the new registration with
DuplicateBrokerRegistrationExceptionuntil the old session expires, and the broker publishes its first image once it catches up to the metadata high watermark, without waiting for the registration.Problem
ReplicaManager.applyLocalLeadersDeltatakes the switch branch only for a partition that already has aPartitionobject. In the first delta after a restart, noPartitionexists. #874 covers a topic that doesn't consolidate: the post-restart branch seals, aborts open transactions, and registers the partition. A consolidating topic still misses it.consolidationActiveForis true while the switch is pending, so the consolidating branch runsmakeLeaderand queues the partition for consolidation fetching. It doesn't seal, abort, or register.No broker sends
InitDisklessLogfor that partition, and produce getsREPLICA_NOT_AVAILABLE. The switch resumes only after the controller fences the old session and bumps the leader epoch, so it's delayed by up tobroker.session.timeout.ms.Fix
makeLeader, and then seals it. The log exists only aftermakeLeader. Produce can't append to the classic log in between, becauseappendRecordsanswersREPLICA_NOT_AVAILABLEfor a pending partition before it reaches the log.applyLocalFollowersDeltadoes for a pending switch. This case happens after a disk replacement, or when the partition directory is missing and an ELR election picks the broker. OtherwisemakeLeadercreates an empty log,registerPartitionproposes its high watermark of 0 as the seal, the controller commits it, and the followers truncate the classic prefix they hold.abortOngoingTransactionsAsSealedLeader()and registers the partition withInitDisklessLogManager.maybeReconcileSwitchedLeader,maybeRollSwitchedLeaderForUpload, andmaybeDropLocalClassicPrefixreturn early for a negative seal, so they don't act on the pending partition.CLASSIC_TO_DISKLESS_SWITCH.mddescribes the restarted leader in the leadership-change paragraph.Interaction with #872
#872 adds
testRearmAfterLoweringMinIsrLetsAStuckPendingSwitchReachTheSeal, which starts from an empty image with this broker as the pending leader of a consolidating topic. With this change, that first delta seals and registers the partition, so the test now checks a registration after its first delta and expectsregisterPartitioncounts of 2 and 3 after the later deltas.Test plan
testFirstDeltaAsPendingLeaderOfConsolidatingTopicSealsAndRegistersandtestFirstDeltaAsPendingLeaderOfNonConsolidatingTopicSealsAndRegistersapply a first delta fromTopicsImage.EMPTYwith-2and this broker as the leader, on a local log with an open transaction. They check that the partition is online, leads, is sealed, and is registered once, that the transaction is aborted with a control batch in the restart leader epoch, and, for the consolidating topic, that no consolidation fetcher starts.testConsolidatingSwitchSealsRegistersThenStartsConsolidationOnClassicPrefixFlagCommitnow expects registration after its first delta and a second registration after the leader epoch bump.testApplyDeltaFencesConsolidatingLeaderBelowSealInPhaseOnenow expects the pending delta to seal and roll the active segment, and the seal commit to fence the leader without another roll.testFirstDeltaAsPendingLeaderOfConsolidatingTopicWithoutLocalLogSkipsThePartitionandtestFirstDeltaAsPendingLeaderOfNonConsolidatingTopicWithoutLocalLogSkipsThePartitionapply the same first delta with no local log. They check that the broker creates no log and noPartition, never callsregisterPartition, and, for the consolidating topic, starts no consolidation fetcher. Without the missing-log check, both tests fail.ReplicaManager.scalareset tomain, the consolidating test and the two updated tests fail. The non-consolidating test andtestApplyDeltaRegistersPendingSwitchLeaderWithoutExistingPartitionfrom fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] #874 pass, because fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] #874 already handles that case.ReplicaManagerInklessTest(297 tests),DisklessSwitchInvariantsTest(15 tests),InitDisklessLogManagerTest(65 tests), andPartitionTest(105 tests) pass, along with:core:checkstyleMainand:core:checkstyleTest. Vale reports no alerts on the added lines.KC-752
🤖 Generated with Claude Code