Repository navigation
fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] - #874
Conversation
…e switch [KC-753] A transaction can be open on a classic partition when the switch to diskless starts. `Partition.seal()` aborted it on every replica, and the transaction coordinator couldn't complete it: - A follower appended its own `ABORT` marker with `appendAsLeader`. - The leader appended its marker before `makeLeader`, in the previous leader epoch. - `WriteTxnMarkers` for a diskless topic returned `INVALID_TOPIC_EXCEPTION`. `TransactionMarkerRequestCompletionHandler` has no case for that error, and the transaction stayed in `PrepareCommit`. `seal()` no longer aborts. The leader calls `abortOngoingTransactionsAsSealedLeader()` after `makeLeader`, only while the switch is pending, and a follower replicates the leader's markers. `WriteTxnMarkers` for a diskless topic returns `UNSUPPORTED_FOR_MESSAGE_FORMAT`, which the coordinator handles by dropping the partition from the transaction. Assisted-by: Claude Fable 5.2
…er aborts transactions The sealed leader appends its `ABORT` markers after `makeLeader`. If the leader is the only ISR member, no follower fetch advances the high watermark past the markers, and `InitDisklessLogManager` waits for `HW == LEO` without end. The leader now advances the high watermark itself after it appends the markers. Assisted-by: Claude Fable 5.2
There was a problem hiding this comment.
🟡 Changes recommended
The switch guidance must require coordinator-side transaction completion with no pending partitions, because an immediate switch after commitTransaction() returns can still abort acknowledged records.
1 open issue
What changed in this PR
This PR fixes transaction handling during classic-to-diskless topic switches. Only the leader writes abort markers, using the new leader epoch, and the coordinator can complete transactions involving switched topics.
Changes:
- Separate log sealing from leader-only transaction abortion and advance the high watermark after appending markers.
- Return a coordinator-supported error for diskless transaction markers.
- Add regression coverage and document the transaction atomicity limitation.
| File | Description |
|---|---|
docs/inkless/CLASSIC_TO_DISKLESS_SWITCH.md |
Document marker handling and transaction behavior during switching. |
core/src/test/scala/unit/kafka/server/KafkaApisTest.scala |
Update the expected diskless marker error. |
core/src/test/scala/unit/kafka/cluster/PartitionTest.scala |
Test leader-only abortion, marker epochs, and high watermark advancement. |
core/src/test/java/kafka/server/InklessTopicTypeSwitcherClusterTest.java |
Test transaction completion, replicated marker epochs, and consumer visibility. |
core/src/main/scala/kafka/server/ReplicaManager.scala |
Abort transactions after leader activation while the switch is pending. |
core/src/main/scala/kafka/server/KafkaApis.scala |
Return UNSUPPORTED_FOR_MESSAGE_FORMAT for diskless transaction markers. |
core/src/main/scala/kafka/cluster/Partition.scala |
Separate sealing from transaction abortion and close marker segments. |
Review effort: Balanced
…itch The coordinator sends the `EndTxn` response before it sends the markers to the partitions, so a commit that returns to the producer doesn't mean the partition's transaction is decided. The guidance now tells operators to stop the transactional producers and wait until the coordinator reports `CompleteCommit` or `CompleteAbort` with no pending partitions. Assisted-by: Claude Fable 5.2
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainer sign-off is needed on the atomicity tradeoff: the coordinator can report CompleteCommit while a switched partition retains ABORT markers and hides acknowledged records.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
viktorsomogyi
left a comment
There was a problem hiding this comment.
just a few nits, nothing serious
There was a problem hiding this comment.
Similarly to the one above, should this say "after sealing"?
There was a problem hiding this comment.
Yes. The abort now runs after the seal, so the log line says "after sealing". Fixed in 06afe65.
| log.roll() | ||
| } catch { | ||
| case e: java.io.IOException => | ||
| throw new KafkaStorageException(s"Failed to roll active segment while sealing " + |
There was a problem hiding this comment.
Should this say "after sealing"?
There was a problem hiding this comment.
Yes. The roll runs after the sealed flag is set, and again after the leader abort, so the message says "after sealing". Fixed in 06afe65.
| partition.makeLeader(info.partition, isNew, offsetCheckpoints, Some(info.topicId), partitionAssignedDirectoryId) | ||
| partition.seal() | ||
| if (isSwitchPending) { | ||
| partition.abortOngoingTransactionsAsSealedLeader() |
There was a problem hiding this comment.
Don't we need initDisklessLogManager.foreach(_.registerPartition(partition, info.topicId())) similarly as above? This would makes ReplicaManager.applyDelta self-consistent whenever existingPartition is empty on a pending switch (such as single-delta unit tests or snapshot loads).
There was a problem hiding this comment.
Yes, it needs it. Nothing else registers the partition, so a leader that loads a pending switch with no Partition yet sealed it and nothing proposed the seal offset until the next leader epoch. The branch now calls registerPartition when the switch is pending. ReplicaManagerInklessTest.testApplyDeltaRegistersPendingSwitchLeaderWithoutExistingPartition covers it and fails without the registration. Fixed in 06afe65.
… partition yet A leader that loads a pending switch with no `Partition` yet, such as after a restart, takes the post-restart branch of `applyLocalLeadersDelta`. That branch sealed the partition but didn't register it with `InitDisklessLogManager`, so nothing proposed the seal offset until the next leader epoch. The branch now registers the partition, like the branch for an existing partition. The log messages for the leader abort and the segment roll say "after sealing", because both now run after the seal. Assisted-by: Claude Fable 5.2
Diskless topics don't support transactions. A transaction can still be open on a partition when the switch starts: it added the partition and wrote to it while the topic was classic.
Partition.seal()aborted such a transaction on every replica, and the transaction coordinator couldn't complete it.Problem
seal()appended theABORTmarkers withappendAsLeaderand didn't check that the broker leads the partition. The follower path callsmakeFollowerand thenseal(), so a follower with the open transaction in its log wrote its own marker.seal()beforemakeLeader, so the leader wrote its marker in the previous leader epoch.KafkaApisansweredWriteTxnMarkersfor a diskless topic withINVALID_TOPIC_EXCEPTION.TransactionMarkerRequestCompletionHandlerhas no case for that error and throwsIllegalStateException. The transaction stayed inPrepareCommitwith the partition still pending.Fix
seal()sets the sealed flag and rolls the active segment. It doesn't abort.Partition.abortOngoingTransactionsAsSealedLeader()appends the markers and rolls the active segment again. It does nothing on a follower. It doesn't depend on whetherseal()ran in the same call sequence, so a replica that sealed as a follower aborts when it becomes the leader.InitDisklessLogManagerwaits forHW == LEO.ReplicaManager.applyLocalLeadersDeltacalls the abort aftermakeLeader, and only while the switch is pending. After the controller commits the seal offset, the leader doesn't append to the classic prefix.Partitionyet, such as after a restart, registers the partition withInitDisklessLogManager. Before, that branch sealed the partition and nothing proposed the seal offset until the next leader epoch.ReplicaManager.applyLocalFollowersDeltaonly seals. The follower replicates the leader's markers.KafkaApisanswersWriteTxnMarkersfor a diskless topic withUNSUPPORTED_FOR_MESSAGE_FORMATand logs the dropped marker. The coordinator handles that error by removing the partition from the transaction, so the upstream handler is unchanged.Known limitation
This change doesn't make the leader and the coordinator agree on the outcome of a transaction that the producer commits after the switch. The leader writes the
ABORTmarkers without the coordinator. If the producer commits a transaction that was open at the switch, the commit succeeds and the transaction reachesCompleteCommit, but the switched partition keeps itsABORTmarker, and aread_committedconsumer gets no records from it. The cluster test asserts this behavior.CLASSIC_TO_DISKLESS_SWITCH.mdhas a new section, "Open transactions at the switch", that describes it.Testing
PartitionTesthas four new tests: a sealed follower doesn't append a marker, a sealed follower that becomes the leader aborts in the new leader epoch and leaves the marker in a closed segment, the abort advances the high watermark when the leader is the only ISR member, and the abort throws on a partition that isn't sealed. The existing seal tests call the abort explicitly.ReplicaManagerInklessTest.testApplyDeltaRegistersPendingSwitchLeaderWithoutExistingPartitionapplies a pending switch to a broker with a local log and noPartition, and checks that the leader seals and registers. Without the registration, it fails.KafkaApisTest.testHandleWriteTxnMarkersRequestWithInklessTopicexpectsUNSUPPORTED_FOR_MESSAGE_FORMAT.InklessTopicTypeSwitcherClusterTest.testTransactionOpenAtSwitchIsAbortedAndCompleteswrites five records in a transaction, switches the topic, and commits. It checks that the seal offset is the five records plus one marker, that every replica has the marker in the same leader epoch and past the pre-switch epoch, that the transaction reachesCompleteCommitwith no pending partitions, that thetransactional.idcommits a second transaction on a classic topic, and that aread_committedconsumer gets no records from the switched partition.KafkaApischange reverted, it fails with the transaction inPrepareCommit. Without the high watermark change, the new only-ISR-member test fails with the high watermark one offset below the log end offset.PartitionTest,KafkaApisTest,ReplicaManagerInklessTest,InitDisklessLogManagerTest,DisklessSwitchFlowTest,InklessTopicTypeSwitcherClusterTest,ClassicToDisklessSwitchLastKnownLeaderClusterTest, andInklessConsolidatedDisklessTopicsTestpass, along with:core:checkstyleTest,:core:spotlessCheck, andmake vale.KC-753
🤖 Generated with Claude Code
Assisted-by: Claude Fable 5.2