Skip to content

fix(inkless:switch): skip ISR expansion while a phase 1 leader's log ends below the seal [KC-773] - #880

Open
giuseppelillo wants to merge 1 commit into
mainfrom
claude/kc-773-investigation-ea3697
Open

giuseppelillo wants to merge 1 commit into
mainfrom
claude/kc-773-investigation-ea3697

Conversation

@giuseppelillo

Copy link
Copy Markdown
Contributor

ClassicToDisklessSwitchLastKnownLeaderClusterTest (loseTail=true) fails intermittently in CI. The cause is a race in the broker, not in the test.

After a last-known-leader election in phase 1 of the classic-to-diskless switch, the restarted leader runs makeLeader and then fences itself when its log ends below the committed seal. A follower fetch between those two steps expands the ISR. The controller then records a full ISR and a recovered leader for a partition that the leader marks offline. When the operator lowers the seal and restarts the leader, the controller elects another ISR member, and the test times out waiting for the restarted broker to lead.

This contradicts CLASSIC_TO_DISKLESS_SWITCH.md, which says the controller keeps an ISR of one after this election.

Partition.isFollowerInSync now returns false while the leader's log end offset is below the committed seal. The check reads replicationStopOffset() directly instead of the sealed-gated value, because seal() runs after makeLeader on a restarted leader.

Evidence

In the CI artifact for the failure on e3913e8c10 (run 44107792176), the leader is broker 2. When it shuts down in reSeal, broker 0 takes over, which only an ISR member can do.

To reproduce it locally, I added a 1.5-second sleep between makeLeader and maybeReconcileSwitchedLeader in ReplicaManager.applyLocalLeadersDelta. The test then failed on the first run with the ticket's signature: leader=2, isr=[0,1,2], leaderRecoveryState=RECOVERED. The logs of the failing run show:

  • The election sets the ISR to [1] and the state to RECOVERING.
  • Broker 1 logs ISR updated to 1,2 and then ISR updated to 2,1,0 before it marks the partition offline.

Testing

  • PartitionTest.testIsrNotExpandedWhileLeaderLogEndsBelowSeal fails without the change (expected: <0> but was: <1>) and passes with it. All 106 tests in PartitionTest pass.
  • With the same 1.5-second sleep and the change, ClassicToDisklessSwitchLastKnownLeaderClusterTest passes both cases. The sleep isn't part of this PR.
  • Without the sleep, 10 consecutive local runs of the cluster test passed before the change, so the local runs don't show the race on their own.

Fixes KC-773.

🤖 Generated with Claude Code

…ends below the seal [KC-773]

After a last-known-leader election in phase 1 of the classic-to-diskless
switch, the restarted leader runs `makeLeader` and then fences itself when its
log ends below the committed seal. A follower fetch between those two steps
expands the ISR. The controller then records a full ISR and a recovered
leader for a partition that the leader marks offline. When the operator lowers
the seal and restarts the leader, the controller elects another ISR member,
and `ClassicToDisklessSwitchLastKnownLeaderClusterTest` times out waiting for
the restarted broker to lead.

`Partition.isFollowerInSync` now returns false while the leader's log end
offset is below the committed seal. The check reads the stop offset directly,
because `seal()` runs after `makeLeader` on a restarted leader.

Assisted-by: Claude Sonnet 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant