Skip to content

fix(inkless:switch): seal and register a pending partition in the first delta after a restart [KC-752] - #877

Merged
viktorsomogyi merged 2 commits into
mainfrom
claude/kc-752-restart-pending-leader
Nov 19, 2026
Merged

viktorsomogyi merged 2 commits into
mainfrom
claude/kc-752-restart-pending-leader

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

A broker that crashes and restarts while a classic-to-diskless switch is pending can apply a first metadata image that still shows it as the leader in the same leader epoch. The controller rejects the new registration with DuplicateBrokerRegistrationException until the old session expires, and the broker publishes its first image once it catches up to the metadata high watermark, without waiting for the registration.

Problem

ReplicaManager.applyLocalLeadersDelta takes the switch branch only for a partition that already has a Partition object. In the first delta after a restart, no Partition exists. #874 covers a topic that doesn't consolidate: the post-restart branch seals, aborts open transactions, and registers the partition. A consolidating topic still misses it. consolidationActiveFor is true while the switch is pending, so the consolidating branch runs makeLeader and queues the partition for consolidation fetching. It doesn't seal, abort, or register.

No broker sends InitDisklessLog for that partition, and produce gets REPLICA_NOT_AVAILABLE. The switch resumes only after the controller fences the old session and bumps the leader epoch, so it's delayed by up to broker.session.timeout.ms.

Fix

  • A pending leader always takes the switch branch, for a consolidating topic too.
  • If the partition doesn't exist, the switch branch creates it, runs makeLeader, and then seals it. The log exists only after makeLeader. Produce can't append to the classic log in between, because appendRecords answers REPLICA_NOT_AVAILABLE for a pending partition before it reaches the log.
  • If the partition doesn't exist and the broker has no local log, the switch branch logs a warning and skips the partition, as applyLocalFollowersDelta does for a pending switch. This case happens after a disk replacement, or when the partition directory is missing and an ELR election picks the broker. Otherwise makeLeader creates an empty log, registerPartition proposes its high watermark of 0 as the seal, the controller commits it, and the followers truncate the classic prefix they hold.
  • In both cases where the leader has a log, it then aborts open transactions with abortOngoingTransactionsAsSealedLeader() and registers the partition with InitDisklessLogManager.
  • The post-restart branch now handles only committed seals, so the pending-switch handling that fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] #874 added there moves into the switch branch.

maybeReconcileSwitchedLeader, maybeRollSwitchedLeaderForUpload, and maybeDropLocalClassicPrefix return early for a negative seal, so they don't act on the pending partition. CLASSIC_TO_DISKLESS_SWITCH.md describes the restarted leader in the leadership-change paragraph.

Interaction with #872

#872 adds testRearmAfterLoweringMinIsrLetsAStuckPendingSwitchReachTheSeal, which starts from an empty image with this broker as the pending leader of a consolidating topic. With this change, that first delta seals and registers the partition, so the test now checks a registration after its first delta and expects registerPartition counts of 2 and 3 after the later deltas.

Test plan

  • testFirstDeltaAsPendingLeaderOfConsolidatingTopicSealsAndRegisters and testFirstDeltaAsPendingLeaderOfNonConsolidatingTopicSealsAndRegisters apply a first delta from TopicsImage.EMPTY with -2 and this broker as the leader, on a local log with an open transaction. They check that the partition is online, leads, is sealed, and is registered once, that the transaction is aborted with a control batch in the restart leader epoch, and, for the consolidating topic, that no consolidation fetcher starts.
  • testConsolidatingSwitchSealsRegistersThenStartsConsolidationOnClassicPrefixFlagCommit now expects registration after its first delta and a second registration after the leader epoch bump.
  • testApplyDeltaFencesConsolidatingLeaderBelowSealInPhaseOne now expects the pending delta to seal and roll the active segment, and the seal commit to fence the leader without another roll.
  • testFirstDeltaAsPendingLeaderOfConsolidatingTopicWithoutLocalLogSkipsThePartition and testFirstDeltaAsPendingLeaderOfNonConsolidatingTopicWithoutLocalLogSkipsThePartition apply the same first delta with no local log. They check that the broker creates no log and no Partition, never calls registerPartition, and, for the consolidating topic, starts no consolidation fetcher. Without the missing-log check, both tests fail.
  • With ReplicaManager.scala reset to main, the consolidating test and the two updated tests fail. The non-consolidating test and testApplyDeltaRegistersPendingSwitchLeaderWithoutExistingPartition from fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] #874 pass, because fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] #874 already handles that case.
  • ReplicaManagerInklessTest (297 tests), DisklessSwitchInvariantsTest (15 tests), InitDisklessLogManagerTest (65 tests), and PartitionTest (105 tests) pass, along with :core:checkstyleMain and :core:checkstyleTest. Vale reports no alerts on the added lines.
  • No cluster test covers this case. It needs a broker that crashes without a controlled shutdown, and a controlled shutdown moves leadership before the broker restarts.

KC-752

🤖 Generated with Claude Code

@giuseppelillo
giuseppelillo changed the base branch from main to claude/kc-753-transaction-abort-b865d9 November 17, 2026 11:21
@giuseppelillo
giuseppelillo requested a balanced review from Copilot November 17, 2026 11:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Tests mock InitDisklessLogManager, leaving the crash/restart ordering between transaction sealing and aborts, broker registration, and controller epoch fencing unverified and requiring human validation.

No open issues 🎉

What changed in this PR

This PR lets a restarted leader resume a pending classic-to-diskless switch from its first metadata image, without waiting for another leader epoch or the old broker session to expire.

Changes:

  • Route pending leaders through sealing, transaction abort, and InitDisklessLog registration even when no Partition object exists.
  • Add restart regression tests for consolidating and non-consolidating topics, and update sealing and registration expectations.
  • Document how a restarted leader resumes a pending switch.
File Description
docs/​inkless/​CLASSIC_TO_DISKLESS_SWITCH.md Explain pending-switch recovery on the first metadata image after restart.
core/​src/​test/​scala/​unit/​kafka/​server/​ReplicaManagerInklessTest.scala Cover first-delta sealing, registration, transaction abort, and consolidation-fetcher behavior.
core/​src/​main/​scala/​kafka/​server/​ReplicaManager.scala Handle pending switches before consolidation routing and create missing partitions before sealing.

Review effort: Balanced

Base automatically changed from claude/kc-753-transaction-abort-b865d9 to main November 17, 2026 14:51
@giuseppelillo
giuseppelillo force-pushed the claude/kc-752-restart-pending-leader branch 4 times, most recently from 282b4ff to b5db59d Compare November 18, 2026 09:42
…st delta after a restart [KC-752]

A broker that crashes and restarts before its session expires can apply a
first metadata image that still shows it as the leader of a partition in
`CLASSIC_TO_DISKLESS_SWITCH_PENDING`. No `Partition` object exists yet, so
`applyLocalLeadersDelta` skipped the switch branch. For a consolidating topic,
the consolidating branch made the broker the leader without sealing or
registering, and no broker sent `InitDisklessLog` until the controller fenced
the old session and bumped the leader epoch.

A pending leader now always takes the switch branch, for a consolidating topic
too. If the partition doesn't exist, the broker creates it, makes it the
leader, seals it, aborts open transactions, and registers it. The post-restart
branch handles only committed seals.

Assisted-by: Claude Opus 5.5
Assisted-by: Claude Sonnet 5.5
@giuseppelillo
giuseppelillo force-pushed the claude/kc-752-restart-pending-leader branch from b5db59d to 7830034 Compare November 18, 2026 09:45
Comment on lines +4835 to +4843
case None =>
// First delta after a restart while the switch is pending. The local log only becomes
// available after makeLeader, so the seal follows it. Produce can't append to the
// classic log in between, because it answers REPLICA_NOT_AVAILABLE for a pending
// partition before it reaches the log.
getOrCreatePartition(tp, delta, info.topicId).map { case (partition, isNew) =>
partition.makeLeader(info.partition, isNew, offsetCheckpoints, Some(info.topicId), partitionAssignedDirectoryId)
partition.seal()
partition

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here makeLeader creates an empty log with HW = LEO = 0 when the broker has no local log on disk, seals it, and calls registerPartition to propose seal 0. On a wiped disk, creating that empty log also causes maybeUpdateTopicAssignment to move the partition's directory assignment to the new disk once the broker registers. If the controller then elects this broker via lastKnownElr or ELR, or if only the partition directory is missing on an online log directory, the controller commits seal 0 and the followers truncate away their classic prefix. Before this PR, non-consolidating partitions required logManager.getLog(tp).isDefined, and consolidating partitions didn't call registerPartition.

We may check logManager.getLog(tp).isDefined before creating the partition. If the local log is missing, log a warning and skip the partition, matching applyLocalFollowersDelta for non-consolidating pending switches. Add a test for consolidating and non-consolidating topics that applies a pending first delta without a local log and verifies that ReplicaManager doesn't create a log or call registerPartition.

…752]

A broker can lead a pending switch without a local log, such as after a disk
replacement or when its partition directory is missing and an ELR election
picks it. The switch branch then created the partition, and makeLeader created
an empty log. registerPartition proposed that log's high watermark of 0 as
the seal, and the controller committed it, so the followers truncated the
classic prefix they hold.

If the local log is missing, the switch branch now logs a warning and skips
the partition, as applyLocalFollowersDelta does for a pending switch. The
broker doesn't create a log or register the partition.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants