Skip to content

fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753] - #874

Merged
giuseppelillo merged 4 commits into
mainfrom
claude/kc-753-transaction-abort-b865d9
Nov 17, 2026
Merged

giuseppelillo merged 4 commits into
mainfrom
claude/kc-753-transaction-abort-b865d9

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Diskless topics don't support transactions. A transaction can still be open on a partition when the switch starts: it added the partition and wrote to it while the topic was classic. Partition.seal() aborted such a transaction on every replica, and the transaction coordinator couldn't complete it.

Problem

  • seal() appended the ABORT markers with appendAsLeader and didn't check that the broker leads the partition. The follower path calls makeFollower and then seal(), so a follower with the open transaction in its log wrote its own marker.
  • The leader path called seal() before makeLeader, so the leader wrote its marker in the previous leader epoch.
  • KafkaApis answered WriteTxnMarkers for a diskless topic with INVALID_TOPIC_EXCEPTION. TransactionMarkerRequestCompletionHandler has no case for that error and throws IllegalStateException. The transaction stayed in PrepareCommit with the partition still pending.

Fix

  • seal() sets the sealed flag and rolls the active segment. It doesn't abort.
  • Partition.abortOngoingTransactionsAsSealedLeader() appends the markers and rolls the active segment again. It does nothing on a follower. It doesn't depend on whether seal() ran in the same call sequence, so a replica that sealed as a follower aborts when it becomes the leader.
  • After it appends the markers, the leader advances the high watermark itself. With the leader as the only ISR member, no follower fetch advances it, and InitDisklessLogManager waits for HW == LEO.
  • ReplicaManager.applyLocalLeadersDelta calls the abort after makeLeader, and only while the switch is pending. After the controller commits the seal offset, the leader doesn't append to the classic prefix.
  • A leader that loads a pending switch with no Partition yet, such as after a restart, registers the partition with InitDisklessLogManager. Before, that branch sealed the partition and nothing proposed the seal offset until the next leader epoch.
  • ReplicaManager.applyLocalFollowersDelta only seals. The follower replicates the leader's markers.
  • KafkaApis answers WriteTxnMarkers for a diskless topic with UNSUPPORTED_FOR_MESSAGE_FORMAT and logs the dropped marker. The coordinator handles that error by removing the partition from the transaction, so the upstream handler is unchanged.

Known limitation

This change doesn't make the leader and the coordinator agree on the outcome of a transaction that the producer commits after the switch. The leader writes the ABORT markers without the coordinator. If the producer commits a transaction that was open at the switch, the commit succeeds and the transaction reaches CompleteCommit, but the switched partition keeps its ABORT marker, and a read_committed consumer gets no records from it. The cluster test asserts this behavior. CLASSIC_TO_DISKLESS_SWITCH.md has a new section, "Open transactions at the switch", that describes it.

Testing

  • PartitionTest has four new tests: a sealed follower doesn't append a marker, a sealed follower that becomes the leader aborts in the new leader epoch and leaves the marker in a closed segment, the abort advances the high watermark when the leader is the only ISR member, and the abort throws on a partition that isn't sealed. The existing seal tests call the abort explicitly.
  • ReplicaManagerInklessTest.testApplyDeltaRegistersPendingSwitchLeaderWithoutExistingPartition applies a pending switch to a broker with a local log and no Partition, and checks that the leader seals and registers. Without the registration, it fails.
  • KafkaApisTest.testHandleWriteTxnMarkersRequestWithInklessTopic expects UNSUPPORTED_FOR_MESSAGE_FORMAT.
  • InklessTopicTypeSwitcherClusterTest.testTransactionOpenAtSwitchIsAbortedAndCompletes writes five records in a transaction, switches the topic, and commits. It checks that the seal offset is the five records plus one marker, that every replica has the marker in the same leader epoch and past the pre-switch epoch, that the transaction reaches CompleteCommit with no pending partitions, that the transactional.id commits a second transaction on a classic topic, and that a read_committed consumer gets no records from the switched partition.
  • With the fix reverted, the cluster test fails because the marker is in the pre-switch leader epoch. With only the KafkaApis change reverted, it fails with the transaction in PrepareCommit. Without the high watermark change, the new only-ISR-member test fails with the high watermark one offset below the log end offset.
  • PartitionTest, KafkaApisTest, ReplicaManagerInklessTest, InitDisklessLogManagerTest, DisklessSwitchFlowTest, InklessTopicTypeSwitcherClusterTest, ClassicToDisklessSwitchLastKnownLeaderClusterTest, and InklessConsolidatedDisklessTopicsTest pass, along with :core:checkstyleTest, :core:spotlessCheck, and make vale.

KC-753

🤖 Generated with Claude Code

Assisted-by: Claude Fable 5.2

…e switch [KC-753]

A transaction can be open on a classic partition when the switch to
diskless starts. `Partition.seal()` aborted it on every replica, and the
transaction coordinator couldn't complete it:

- A follower appended its own `ABORT` marker with `appendAsLeader`.
- The leader appended its marker before `makeLeader`, in the previous
  leader epoch.
- `WriteTxnMarkers` for a diskless topic returned
  `INVALID_TOPIC_EXCEPTION`. `TransactionMarkerRequestCompletionHandler`
  has no case for that error, and the transaction stayed in
  `PrepareCommit`.

`seal()` no longer aborts. The leader calls
`abortOngoingTransactionsAsSealedLeader()` after `makeLeader`, only while
the switch is pending, and a follower replicates the leader's markers.
`WriteTxnMarkers` for a diskless topic returns
`UNSUPPORTED_FOR_MESSAGE_FORMAT`, which the coordinator handles by
dropping the partition from the transaction.

Assisted-by: Claude Fable 5.2
…er aborts transactions

The sealed leader appends its `ABORT` markers after `makeLeader`. If the
leader is the only ISR member, no follower fetch advances the high
watermark past the markers, and `InitDisklessLogManager` waits for
`HW == LEO` without end. The leader now advances the high watermark
itself after it appends the markers.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The switch guidance must require coordinator-side transaction completion with no pending partitions, because an immediate switch after commitTransaction() returns can still abort acknowledged records.

1 open issue
What changed in this PR

This PR fixes transaction handling during classic-to-diskless topic switches. Only the leader writes abort markers, using the new leader epoch, and the coordinator can complete transactions involving switched topics.

Changes:

  • Separate log sealing from leader-only transaction abortion and advance the high watermark after appending markers.
  • Return a coordinator-supported error for diskless transaction markers.
  • Add regression coverage and document the transaction atomicity limitation.
File Description
docs/​inkless/​CLASSIC_TO_DISKLESS_SWITCH.md Document marker handling and transaction behavior during switching.
core/​src/​test/​scala/​unit/​kafka/​server/​KafkaApisTest.scala Update the expected diskless marker error.
core/​src/​test/​scala/​unit/​kafka/​cluster/​PartitionTest.scala Test leader-only abortion, marker epochs, and high watermark advancement.
core/​src/​test/​java/​kafka/​server/​InklessTopicTypeSwitcherClusterTest.java Test transaction completion, replicated marker epochs, and consumer visibility.
core/​src/​main/​scala/​kafka/​server/​ReplicaManager.scala Abort transactions after leader activation while the switch is pending.
core/​src/​main/​scala/​kafka/​server/​KafkaApis.scala Return UNSUPPORTED_FOR_MESSAGE_FORMAT for diskless transaction markers.
core/​src/​main/​scala/​kafka/​cluster/​Partition.scala Separate sealing from transaction abortion and close marker segments.

Review effort: Balanced

…itch

The coordinator sends the `EndTxn` response before it sends the markers
to the partitions, so a commit that returns to the producer doesn't mean
the partition's transaction is decided. The guidance now tells operators
to stop the transactional producers and wait until the coordinator
reports `CompleteCommit` or `CompleteAbort` with no pending partitions.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer sign-off is needed on the atomicity tradeoff: the coordinator can report CompleteCommit while a switched partition retains ABORT markers and hides acknowledged records.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@viktorsomogyi viktorsomogyi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a few nits, nothing serious

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly to the one above, should this say "after sealing"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The abort now runs after the seal, so the log line says "after sealing". Fixed in 06afe65.

log.roll()
} catch {
case e: java.io.IOException =>
throw new KafkaStorageException(s"Failed to roll active segment while sealing " +

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this say "after sealing"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The roll runs after the sealed flag is set, and again after the leader abort, so the message says "after sealing". Fixed in 06afe65.

partition.makeLeader(info.partition, isNew, offsetCheckpoints, Some(info.topicId), partitionAssignedDirectoryId)
partition.seal()
if (isSwitchPending) {
partition.abortOngoingTransactionsAsSealedLeader()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need initDisklessLogManager.foreach(_.registerPartition(partition, info.topicId())) similarly as above? This would makes ReplicaManager.applyDelta self-consistent whenever existingPartition is empty on a pending switch (such as single-delta unit tests or snapshot loads).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it needs it. Nothing else registers the partition, so a leader that loads a pending switch with no Partition yet sealed it and nothing proposed the seal offset until the next leader epoch. The branch now calls registerPartition when the switch is pending. ReplicaManagerInklessTest.testApplyDeltaRegistersPendingSwitchLeaderWithoutExistingPartition covers it and fails without the registration. Fixed in 06afe65.

… partition yet

A leader that loads a pending switch with no `Partition` yet, such as
after a restart, takes the post-restart branch of
`applyLocalLeadersDelta`. That branch sealed the partition but didn't
register it with `InitDisklessLogManager`, so nothing proposed the seal
offset until the next leader epoch. The branch now registers the
partition, like the branch for an existing partition.

The log messages for the leader abort and the segment roll say "after
sealing", because both now run after the seal.

Assisted-by: Claude Fable 5.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants