Skip to content

chore(inkless:release): cherry-pick main changes into inkless-4.3 for 0.52 - #892

Draft
jeqo wants to merge 24 commits into
inkless-4.3from
inkless-4.3-0.52
Draft

jeqo wants to merge 24 commits into
inkless-4.3from
inkless-4.3-0.52

Conversation

@jeqo

@jeqo jeqo commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Cherry-picks the 23 commits that main gained since inkless-release-0.51 into inkless-4.3 for the 0.52 release. No upstream sync is included: 4.3 already holds the upstream version of 0.51.

The last commit, sync(compile), adapts the picked tests and the JMH benchmark to MetadataDelta.Builder, and lets checkstyle skip the method length check for RemoteLogManager: #857 makes cleanupExpiredRemoteLogSegments 174 lines long, over the limit of 170.

Commits

See the commit messages for the reasoning behind each change.

Release notes

Testing

The storage:inkless unit and integration tests, the metadata controller tests, and the core Inkless and Diskless suites pass. InklessControlPlaneOutageClusterTest fails intermittently in the full run and passes alone.

gqmelo and others added 24 commits November 19, 2026 20:30
)

The 0.51 entry records the switch phase gates, the bounded consolidation prune, and their config, metric, and schema changes. The generator draft missed the new default of `inkless.consolidation.cleanup.interval.ms` and the broker and controller metrics that `metrics.rst` doesn't cover. Its `PostgresControlPlane` delta is documentation catch-up for metrics that 0.50 already registered.

The session logs record the cherry-pick of #823 through #854 to `inkless-4.1`, `inkless-4.2`, and `inkless-4.3`, the conflict resolutions, and the release-branch build fixes.

## Testing

- `make vale` reports no alerts.

Assisted-by: Claude Opus 5.5
(cherry picked from commit 25ac64c)
…861)

* docs(inkless:switch): drop the spanning-segment limitation from the forced deletion section

The section listed a follower that consolidated on a release that didn't wait
for phase 2 as a source of local prefix reads, and described the segment that
starts below the seal and ends past it as a limitation of forced deletion: a
follower below the seal or a log directory move on that replica waited for
local retention.

That segment can't form on the current code. `forceDeleteLocalSegmentsBelow`
rolls the active segment before it deletes, and `applyDelta` runs it before it
starts the consolidation fetchers. `Partition.seal()` rolls the leader at the
seal, and a log directory move rolls the future log at the seal. Only a replica
upgraded mid-switch from a release before 0.52 could hold one, and no
production partition switched on those releases. The remote-storage-off caveat
stays, because the broker still keeps the local prefix in that case.

Assisted-by: Claude Fable 5.1

* fix(inkless:switch): route pending-upload offsets to the classic log

A follower with `follower.fetch.last.tiered.offset.enable=true` asks the leader
for `EARLIEST_PENDING_UPLOAD_TIMESTAMP` before it rebuilds from tiered storage.
Only the local `UnifiedLog` knows that offset: it derives from the highest
offset in remote storage and the log start offset. A follower request on a
switched partition takes the direct classic route. A diskless topic with remote
storage whose switch is pending has no committed seal, and the router already
applies the consolidating rules to it, so its follower requests take the hybrid
route while the classic fetcher still replicates the prefix. There
`DisklessFetchOffsetRouter` sent the request to the diskless lookup first and
fell back to the classic log only on an empty answer. The router now routes the
timestamp to the classic lookup, like `EARLIEST_LOCAL_TIMESTAMP` and
`LATEST_TIERED_TIMESTAMP`.

## Testing

- `DisklessFetchOffsetRouterTest` covers the hybrid route, a follower request on
  a tiered diskless topic whose switch is pending, and a consolidating topic
  without a committed seal.
- `make vale` reports no alerts.

Assisted-by: Claude Fable 5.1
(cherry picked from commit 6a18c2a)
…changes [KC-727] (#864)

## Summary

When a partition's leader changes, `Partition.makeLeader` assigns a cache entry for the new leader epoch. Consolidation then appends batches in the lower diskless leader epoch. `LeaderEpochFileCache.assign` drops every entry whose epoch is equal or higher on a non-monotonic append. The consolidated records lose their epoch, and tiered storage can no longer match their segments to the lineage.

```diff
 makeLeader
   if the leader epoch is new
-    assignEpochStartOffset(newEpoch, log end offset)
+    if the consolidation epoch is already cached and lower than the new epoch
+      skip the new epoch entry
+    else
+      assignEpochStartOffset(newEpoch, log end offset)
```

The skip covers a switched partition in phase 2 and a born-diskless partition once the cache contains an entry for the consolidation epoch. Phase 1 and a pending switch still assign the entry, because consolidation doesn't append there. A broker restart takes the same path: a fresh `Partition` starts at leader epoch -1, so `makeLeader` treats the unchanged leader epoch as new.

A promoted follower whose log is still at the seal has no diskless leader epoch entry. `makeLeader` assigns the new epoch at the seal, so a lookup for an epoch between the last classic epoch and the diskless leader epoch returns the seal. The next consolidated append replaces that entry with the diskless leader epoch at the seal.

Brokers that ran without the skip already carry damaged caches. On a born-diskless partition, the damaged cache is `(0, X)`, where `X` is the log end offset at the leader change or restart. Offsets below `X` have no epoch, so remote copies of those segments fail and remote reads return `OFFSET_NOT_AVAILABLE`. The consolidation fetcher repairs the cache before every consolidated append:

```diff
 ConsolidationFetcherThread.processPartitionData
+  if the partition has no seal
+    assignEpochStartOffset(0, log start offset)
   stamp the diskless leader epoch
   append
```

Every consolidated batch of a born-diskless partition has epoch 0, so the correct cache is `(0, log start offset)`. The assignment changes nothing on a correct cache. It replaces a damaged `(0, X)` entry, and it drops a stale higher entry that a checkpoint kept from before the upgrade. The fetcher runs on leaders and followers, so the next append also repairs a cache that a tier-state rebuild restored from a damaged remote checkpoint. The guard checks the seal, not the diskless leader epoch. Partitions switched before the diskless leader epoch existed have a seal but no diskless leader epoch, and assigning epoch 0 there would drop their classic entries.

The become-leader log mentions the skip when it applies, and a comment next to the skip records the upstream `endOffsetFor` assumption that the skip breaks. `Partition` reads the consolidation epoch through a hook, the same way it reads `replicationStopOffset`. The glossary gains entries for Diskless Leader Epoch and Seal.

## Evidence

- **Before:** with the skip removed, `PartitionTest.testMakeLeaderKeepsDisklessLeaderEpochAfterLeaderChange` fails. The cache entry for the diskless leader epoch moves from the seal to the log end: `(4, 2)` becomes `(4, 4)`. The born-diskless case fails the same way: `(0, 0)` becomes `(0, 2)`, and so does the restart case, `PartitionTest.testMakeLeaderKeepsBornDisklessEpochAfterRestart`. An unconditional skip also fails `PartitionTest.testMakeLeaderAtSealAssignsEpochWhenDisklessEpochIsMissing`: a follower promoted at the seal keeps only `(2, 0)`, and `endOffsetFor(3)` returns undefined instead of the seal.
  **After:** the cached diskless leader epoch stays `(4, 2)`, and the born-diskless cache stays `(0, 0)` across a leader change and a restart. The promotion at the seal writes `(5, 2)`, `endOffsetFor(3)` returns `2`, and the following consolidated append leaves `(2, 0), (4, 2)`. Those `PartitionTest` cases pass.
- **Before:** with the repair call removed, `ConsolidationFetcherThreadTest` leaves a damaged cache at `(0, 10)` and a stale cache at `(0, 5), (4, 10)`. An empty cache doesn't get an entry. A real `UnifiedLog` rebuilt from a damaged checkpoint stays at `(0, 10)` after a consolidated append.
  **After:** with log start offset 5, each of those caches becomes `(0, 5)`. A correct cache, a pending switch, and switched partitions keep their entries. Those `ConsolidationFetcherThreadTest` cases pass.

## Test plan

`PartitionTest` covers a switched partition and a born-diskless partition across a leader change, a born-diskless partition across a restart into the same leader epoch, and a promotion at the seal before the cache has an entry for the diskless leader epoch. `ReplicaManagerInklessTest` covers `consolidationLeaderEpoch` in each switch phase, an end-to-end `applyDelta` promotion, and an `OffsetsForLeaderEpoch` query of the new leader epoch that the local cache can't answer. `ConsolidationFetcherThreadTest` covers the repair of a damaged cache, a stale higher entry, and an empty cache. It also checks that correct caches, a pending switch, and switched partitions with and without a diskless leader epoch keep their entries, and it rebuilds a real `UnifiedLog` from a damaged checkpoint before a consolidated append.

## Merge Danger

**Door:** two-way

Reverting the change restores the old assignment on the next leader change. A cache that the repair already rewrote keeps its `(0, log start offset)` entry, which the old code also reads correctly. A promotion at the seal writes a temporary entry for the new epoch, and the next consolidated append replaces it.

**Blast Radius:** consolidation

When the consolidation epoch is already cached, the latest cached epoch sits below the current leader epoch, so `endOffsetFor` returns undefined for the current epoch. Every caller handles that result today. Turning the broker consolidation flag on or off can still give a born-diskless partition the old behavior, because the check follows that flag.

A damaged born-diskless partition recovers on each replica at its next consolidated append. These limits remain:

- **Segment uploaded while damaged:** its remote metadata keeps `{0: X}` and indexes the segment at `X`. After the repair, reads between the segment's base offset and `X` return `OFFSET_OUT_OF_RANGE` instead of `OFFSET_NOT_AVAILABLE`, and a consumer that resets to the earliest offset can reach the same gap again. Every other offset below `X` becomes readable. The gap lasts until retention deletes that segment.
- **Partitions without new appends:** the repair waits for the next consolidated append. Until then, remote copies and reads below `X` keep failing.
- **Switched partitions:** the repair skips them. Their damaged range went to remote storage labeled with the last classic epoch, so a local-only repair would make those segments unreadable. They need a separate fix.
- **Canceled switch:** when `AlterDisklessSwitch` cancels a pending switch, the partition goes back to having no seal. The repair then treats it as born-diskless. The existing epoch-0 append already drops its classic entries, so the repair doesn't make that state worse.

Assisted-by: GPT-5.6 Luna Max
Assisted-by: Claude Opus 5.5
(cherry picked from commit 155da28)
…mmitted classic prefix [KC-702] (#857)

* fix(inkless:consolidation): bound the diskless start by the seal before the first prune [KC-702]

The control plane keeps a switched partition's WAL start at the classic log
start until the leader's first prune moves it to the seal. That prune runs only
once the leader's remote copy reaches `seal - 1`, which a leader with an empty
local log never reaches. Its consolidation fetch at offset 0 then reads the
first diskless batch from the WAL, and the local log hides the classic prefix
that remote storage holds.

`DisklessLeaderEndPoint` now uses the seal as the lowest diskless start for the
tiered-storage redirect and for `fetchEarliestLocalOffset`. The tier-state
machine then rebuilds the prefix and starts the local log at the seal, as it
does after the first prune. If remote storage doesn't cover `seal - 1`, the
endpoint returns `NOT_LEADER_OR_FOLLOWER` and sets
`ConsolidationRemotePrefixUnknown` instead of consolidating below the seal.

Retention or `DeleteRecords` can move the cross-tier earliest offset to the
seal while the WAL start stays below it. `DeleteRecords` at the seal runs only
the local leg. Without another check, the fetcher waits for coverage of
`seal - 1` that never returns, or rebuilds a deleted prefix from a segment that
remote storage still lists. If the cross-tier earliest offset is at the diskless
start or past it, the endpoint now returns `OFFSET_OUT_OF_RANGE` before it asks
remote storage. The out-of-range reset starts the local log at that offset.

## Testing

`DisklessLeaderEndPointTest` covers the redirect, the retry without remote
coverage, the reset when the cross-tier earliest offset is at the seal, a fetch
at the seal, and the earliest local offset before the first prune.

Assisted-by: Claude Opus 5.5

* fix(inkless:switch): recreate phase 2 replicas that don't hold the committed classic prefix [KC-702]

In phase 2 of the classic-to-diskless switch, the controller adds every
live replica to the ISR without checking its local copy of the classic
prefix, and any replica can lead. A replica can hold records below the
seal that the partition doesn't hold. It led an older leader epoch and
wrote records that the next leader didn't have, or an unclean leader
election before the switch discarded records below its high watermark.
Forced deletion keeps the leader epoch cache of that replica, and the
remote log manager uses that cache to pick remote segments. On that
replica, a tiered read below the seal fails with `OFFSET_OUT_OF_RANGE`.

The leader that requests the phase 2 flag now sends the leader epoch of
its record at `seal - 1`. The controller stores it with the flag as tag
`104` (`lastClassicLeaderEpoch`). Each epoch has one leader, so a
replica whose leader epoch cache has that epoch at `seal - 1` holds the
committed prefix, at any high watermark. The controller rejects an
epoch below `-1` or higher than the current leader epoch, which no
replica can match. `DescribeTopicPartitions` returns the epoch with the
other switch fields, and `kafka-topic-switch.sh state` prints it.

A consolidating replica checks its local log when it first applies the
partition and on every new leader epoch, after the broker stops the
replica, consolidation, and log directory move fetchers of the
partition. Stopping the move fetcher waits for a running promotion of
the future log. The replica keeps its log only if its last record below
the seal is in the recorded epoch, its records past the seal are
consolidated diskless records, its highest offset in remote storage is
below its log end offset, and its future log ends in the same epoch as
the current log at that offset. Otherwise, `applyDelta` stops the
replica as it stops a deleted partition. It deletes the current and
future logs, and the remote log manager cancels the partition tasks but
keeps the remote log metadata subscription. The leadership change in
the same delta creates an empty log, and the existing recovery rebuilds
it from remote storage.

A storage error can stop the deletion before it reaches the future log,
which would then restart the move on the new log. If the broker can't
delete both logs or cancel the remote log tasks, it marks the replica
offline with its topic ID, as after a storage error in a transition, so
later deltas skip it. `stopPartitions` cancels the remote log tasks even
if a deletion fails, because a copy task that keeps running also keeps
the copy progress of the deleted log.

The broker never truncates a log that it keeps in place. Truncating in
place left state behind that later steps trusted: the future log of a
log directory move, a highest offset in remote storage that only grows,
and remote log tasks that compute offsets without the log lock. Each of
them needed its own guard. A recreated log starts without that state,
and the remote log tasks that still run update only the deleted log.

A kept log can still have truncated before the check, to the seal in
phase 1 or to the new leader after an unclean leader election. The
remote log tasks compute the highest offset in remote storage without
the log lock. A follower task that read the discarded tail, or a copy
task that uploaded it, could install its offset after the check accepted
the log. `UnifiedLog` now counts truncations. A truncation counts itself
and changes the log under the log lock. The follower and copy tasks read
the count under that lock before they read the leader epoch cache or the
segments. A task never pairs the new count with the state from before
the truncation. `updateHighestOffsetInRemoteStorage(offset,
truncationCount)` installs the offset under the log lock only if the
count didn't change. After a truncation, the copy task also finds its
copied offset again instead of resuming after the discarded tail.

Upstream `TierStateMachine` rebuilds from the remote segment that holds
the record before the local log start of the source. In phase 2, a
classic leader or the current log of a log directory move can still
hold local classic segments below the seal. A segment from a discarded
epoch can hold that record in the same epoch as the committed prefix
and diverge after it, and it can also hold records past the seal.
Restoring such a segment restores the wrong epochs and starts the local
log at its end, past diskless records that the WAL prune then deletes.
A phase 2 rebuild now looks up the segment that holds `seal - 1` in the
recorded epoch, whatever the local log start of the source is, and
starts the local log at the seal. Its records below the seal are the
committed prefix, even if it also holds discarded records past the
seal. The rebuild keeps the epochs that start below the seal and skips
the epoch lookup on the source. It restores the producer snapshot only
if the segment ends at `seal - 1`. Sealing aborts undecided
transactions, so the empty producer state is correct at the seal.
Partitions without a recorded epoch keep the upstream rebuild.

- `testSwitchedLeaderInPhaseTwoRecreatesReplicaWithoutCommittedHistory`
  covers a leader with a divergent epoch at `seal - 1`, and a leader
  whose future log ends past the current log or in another epoch. The
  broker deletes both logs, stops the remote log tasks before the
  leadership change, and starts consolidation at 0 on an empty log. A
  matching future log keeps the move running.
- `testSwitchedLeaderInPhaseTwoDropsRemoteProgressPastTheSeal` runs the
  WAL pruner on leaders with a divergent epoch, a classic tail past the
  seal, or remote progress at their log end offset. The broker recreates
  them, and the pruner doesn't prune. A leader that holds the committed
  prefix keeps its log, and the pruner prunes at `seal - 1`.
- `testSwitchedLeaderInPhaseTwoKeepsConsolidatedReplica` keeps a leader
  whose log holds the committed prefix and consolidated records past the
  seal, and resumes consolidation at its log end offset.
- `testSwitchedConsolidatingFollowerInPhaseTwoRecreatesReplicaWithoutCommittedHistory`
  covers a follower at the seal with a matching and a divergent epoch.
- `testOnlineSwitchedReplicaWithoutCommittedHistoryRecreatedOnNewLeaderEpoch`
  checks that an ISR change doesn't recreate an online replica, and that
  the next leader epoch does.
- `testSwitchedReplicaStaysOfflineWhenItsDeletionFails` removes the
  directory of the current or the future log before the check. The
  broker cancels the remote log tasks and marks the replica offline. The
  next leader epoch doesn't recreate it, and no fetcher starts. Without
  the change, the broker recreates the replica.
- `testUpdateHighestOffsetInRemoteStorageIgnoresOffsetComputedBeforeTruncation`
  checks that `truncateTo` and `truncateFullyAndStartAt` make the log
  drop an offset from before them, and that a `truncateTo` without effect
  doesn't.
- `testTruncationCountWaitsForRunningTruncation` pauses `truncateTo`
  after it counts itself and before it truncates the leader epoch cache.
  A read of the count waits for the truncation to finish. If the read
  doesn't take the log lock, it returns the new count while the cache
  still holds the removed epoch, and the test fails.
- `testFollowerTaskInstallsHighestRemoteOffsetWithTruncationCountFromBeforeTheLookup`
  and `testCopyTaskFindsCopiedOffsetAgainAfterTruncationDuringUpload`
  truncate the log during the lookup and during the upload. Both tasks
  install the offset with the count from before, and the copy task finds
  its copied offset again on the next run. Without the change, both
  tests fail.
- `testTierStateMachineLooksUpLastClassicSegmentByRecordedEpoch` checks
  that the rebuild ignores a segment from a discarded epoch that covers
  `seal - 1`.
- `testTierStateMachineStopsAtSealForSegmentCrossingIt` rebuilds from a
  segment that crosses the seal. The segment contains one epoch, or a
  discarded epoch past the seal. The rebuild starts the log at the seal, keeps
  only the epochs below it, and skips the producer snapshot. Without
  the change, the fetch starts at the segment end.
- `testTierStateMachineLooksUpRecordedEpochForSourceWithLocalClassicPrefix`
  rebuilds from a source whose local log starts at 50, below seal 100.
  The segment that holds 49 in epoch 0 comes from discarded epoch 1,
  which epoch 2 replaced at 80. The rebuild uses the segment that holds
  99 in epoch 2 instead, which ends at the seal or crosses it. Without
  the change, the rebuild restores epoch 1.
- `PartitionRegistrationTest`, `PartitionChangeBuilderTest`,
  `ReplicationControlManagerInklessTest`, `InitDisklessLogManagerTest`,
  `InklessMetadataViewTest`, `DescribeTopicPartitionsRequestHandlerTest`,
  and `TopicSwitchCommandTest` cover the new field from the request to
  the broker metadata view, including the range check in the controller
  and the `kafka-topic-switch.sh state` output.

Assisted-by: Claude Opus 5.5

* fix(inkless:consolidation): keep referenced remote segments while the epoch cache rebuilds [KC-702]

A phase 2 leader that the broker recreates caches only its own leader
epoch until `TierStateMachine` restores the older epochs from remote
storage. A leader that lost its local log is in the same state. The
remote log manager runs its expiration task as soon as the broker
becomes the leader. That task deletes every remote segment whose epochs
are all older than the earliest cached epoch, and it doesn't check
offsets, because upstream assumes that those segments end below the log
start. For such a leader, the assumption doesn't hold, and the task
deletes the classic prefix that the rebuild reads and consumers fetch.

For a consolidating partition, this cleanup now deletes only the
segments that end below the cross-tier log start, which is the
authoritative log start of the partition. The other segments wait until
retention moves that start past them, like a divergent segment inside
the cached epochs does upstream. Classic topics keep the upstream
behavior.

- `testEpochCacheCleanupKeepsConsolidatingSegmentsFromCrossTierStart`
  runs the expiration task for a leader that caches only epoch 5, with
  remote segments from epochs 0 and 1 on both sides of the cross-tier
  log start. For a consolidating partition, the task deletes only the
  segment below that start. Without the change, it deletes both. For a
  classic partition, it deletes both, as upstream does.
- `testRecreatedLeaderKeepsRemotePrefixUntilRebuild` starts a real empty
  log in epoch 7, as a recreated leader does. Remote storage holds
  segments from classic epochs 0, 2, and 3 below seal 100 and from the
  diskless epoch 5, and the cross-tier log start is 40. Both before and
  after the rebuild, expiration deletes only the segment below 40.
  Without the guard, the first expiration deletes every remote segment.

Assisted-by: Claude Opus 5.5

* fix(inkless:switch): keep a replica whose future log ends at or below the log start [KC-702]

Retention or DeleteRecords can move the log start of the current log
while a partly copied log directory move waits behind other moves. The
current log then has no leader epoch for the last record of the future
log, so the phase 2 replica check fails and the broker recreates a
healthy replica.

The check now passes a future log that ends at or below the log start
offset of the current log. The current log holds none of its records,
and the move fetcher drops them on its next fetch.

Also reword two comments from review: the controller comment describes
the recreate instead of a truncation, and the endpoint comment passes
Vale.

Assisted-by: Claude Fable 5.2
(cherry picked from commit 8ac5704)
…when it applies the seal (#866)

* fix(inkless:switch): advance a follower's high watermark to the seal when it applies the seal

A follower at the seal can miss the leader's last high watermark update.
The fetch that moves the follower to the seal is also the fetch that
advances the leader's high watermark and triggers the seal commit. The
leader parks that fetch in purgatory, and the seal's leader epoch can
reach the follower before the response does. The follower then drops
the response, because its fetch state moved to the new epoch. The
leader's fetch session already recorded the high watermark at the seal
as sent, and the sealed log never changes, so the leader leaves the
partition out of every later response. The follower keeps its stale
high watermark until the next leader change.

With a high watermark below the seal, `lastOffsetForLeaderEpoch` and
the `ListOffsets` router treat the local classic prefix as incomplete
and refuse the follower with `NOT_LEADER_OR_FOLLOWER`. Metadata routes
consumers in that zone to the follower, so they retry until they time
out. `test_switch_succeeds_with_degraded_network` fails this way.

A committed seal proves that every offset below it is committed: the
leader requests the seal only after its high watermark reaches its
frozen log end offset. The leader already restores a stale high
watermark from that proof in `maybeReconcileSwitchedLeader`. The
follower now does the same when it applies a committed seal with its
log end offset at or past the seal, before it decides whether to
schedule a catch-up fetch. A follower below the seal keeps its high
watermark, because it still misses part of the prefix.

Testing

Unit tests in `ReplicaManagerInklessTest` cover a follower at the seal
with a stale high watermark in phase 1 and phase 2, and a follower below
the seal that keeps its high watermark. The promotion test stages the
stale high watermark again after the follower delta, so it still
exercises the leader-side restore.

KC-722

Assisted-by: Claude Fable 5.1

* fix(inkless:switch): restore the follower high watermark only for phase 1 ISR members

A replica outside the ISR, or any replica in phase 2, can reach the seal
with a divergent tail from an older leader epoch. Its log end offset
alone doesn't prove that it holds the committed prefix, and the restore
let such a replica skip the catch-up fetch that truncates the tail and
rebuilds its epoch cache from remote storage.

The restore now runs only when the partition is in phase 1 and this
broker is in the ISR of the new registration. In phase 1 the leader
admits a replica to the ISR only after a fetch at the seal validated
its log, so together with the committed seal that proves the prefix.
Phase 2 replicas and replicas outside the ISR keep their high watermark
and reconcile through the fetch, as before this change.

Testing

The phase 2 test now asserts that the high watermark stays and the
catch-up fetch is scheduled. A new test covers a phase 1 follower
outside the ISR the same way.

KC-722

Assisted-by: Claude Fable 5.1

* test(inkless:switch): cover the seal arriving in its own delta at the same leader epoch

The follower high watermark restore runs on every follower delta, but no
test applied the seal as a separate delta without a leader epoch bump.

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit 12d1e3c)
* fix(inkless:switch): cap tiered reads at the seal [KC-739]

After an unclean leader election before the switch, a remote segment
can cross the classic-to-diskless seal and hold discarded records past
it. `RemoteLogManager.read` returns records up to the fetch size, so a
consumer that reads below the seal receives those discarded records and
then skips the diskless records at the same offsets, without an error.

`ReplicaManager` now passes the seal as an upper bound offset in
`RemoteStorageFetchInfo` when a switched partition reads its classic
prefix from remote storage. `RemoteLogManager.read` keeps only the
whole batches that end below the bound, and the aborted transaction
lookup stops at the bound. Every other partition passes no bound, and
its read is unchanged.

Testing

`RemoteLogManagerTest` covers a segment that crosses the bound in one
epoch, one with a discarded epoch past the bound, one that ends below
the bound, a read with no bound, and the aborted transaction lookup.
`ReplicaManagerInklessTest` and `ReplicaManagerTest` check that the
fetch info carries the seal only for a switched partition.

Assisted-by: Claude Fable 5.2
Assisted-by: Claude Sonnet 5.5

* fix(inkless:switch): cap tiered reads at the seal in phase 1 too

The bound applied only in phase 2, because it reused the check that
requires the classic prefix in remote storage. In phase 1 the seal is
already committed, and a fetch below the local log start offset also
reads from remote storage, so it could return discarded records past
the seal.

`ReplicaManager` now sets the bound whenever the seal is committed and
the fetch starts below it.

Testing

`ReplicaManagerInklessTest` covers a phase 1 tiered read below the
local log start offset.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): log when the seal cuts off the first tiered batch [KC-739]

If the first batch of a tiered read crosses the upper bound offset, the
response is empty and the consumer fetches the same offset again. Log the
cause so the stall is diagnosable.

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit 0e3aed8)
* fix(inkless:release): list every Kafka-base tag in the release notes

A resumed release builds only the highest patch tag for each branch. The notes listed only the built tags, so adding Kafka 4.3.2 to release 0.51 removed Kafka 4.3.1 from the notes while its binary stayed attached. The notes step now also lists every existing Kafka-base tag of the version.

Assisted-by: Claude Sonnet 5.5

* docs(inkless:release): release an upstream sync under the current version

When a release branch changes only by an upstream Kafka patch sync, the Inkless version stays the same. The release sync guide now covers the tag, the build that the tag push starts, and the resume run of the release workflow that regenerates the notes and uploads the binary to the existing release.

The releases doc said such a tag stays off the GitHub Releases page. It now describes the resume run instead. The versioning strategy, the sync action plan, the prompt, and the session template link to the same steps.

Assisted-by: Claude Sonnet 5.5

* fixup! docs(inkless:release): release an upstream sync under the current version

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit 5e0dd75)
…og start [KC-761] (#870)

* fix(inkless:switch): defer timestamp ListOffsets below the diskless log start [KC-761]

A `ListOffsets` lookup by timestamp on a hybrid partition searches the
local log first and asks the control plane only on a "no match". The
control plane answers from the first retained WAL batch. A replica whose
local log ends before that batch leaves a range that neither search
reads, so a consumer that seeks into it starts at the first diskless
offset and skips records without an error. Two kinds of replica hit it:

- A switched leader below the seal: the KC-702 divergent-tail check
  recreates it empty, or it lost its disk and stays online for
  consolidation to rebuild it.
- A consolidating replica that the WAL prune passed: a reassignment
  target, a replica that lost its data, or a follower that lags the
  prune. The pruner doesn't wait for replicas, and any replica of a
  consolidating topic answers `ListOffsets`.

The router now returns the retriable `OFFSET_NOT_AVAILABLE`
(`LEADER_NOT_AVAILABLE` before version 5) instead, as the Fetch path
already does for the same window. A switched replica below the seal
gets the error without a control-plane call. For a consolidating
partition, `FetchOffsetHandler.Job.addWalStart` reads the WAL start in
the same control-plane call as the timestamp lookup, and the router
fails the lookup if the local log ends before the WAL start or the
seal. A match in the local log is still returned.

The searchable end of the local log is its log end offset, or its log
start offset while the leader epoch cache can't resolve the log start.
A tier-state rebuild truncates the log past the WAL start before it
restores that cache, and until then the remote search finds nothing.
The router reads this end before the local search and again when the
control plane answers, and uses the smaller value.

The WAL start check applies only to consolidating partitions. A
switched partition that doesn't consolidate keeps its log at the seal,
so diskless retention past the seal would block it permanently.

`FetchOffsetHandler.Job` now matches control-plane responses to
requests by position, because a partition can appear twice. Every
control plane answers in request order.

Testing

`DisklessFetchOffsetRouterTest` covers a switched replica below the
seal (sync and async local lookups, version mapping, a local match), a
consolidating replica below and past the WAL start, a diskless "no
match" below the WAL start, a log that a rebuild extends or a
recreation cuts back during the lookup, the seal bound before the
first prune, forwarded errors, and the fresh-job fallback. With the
guard disabled, the 9 tests that expect the error fail.
`FetchOffsetHandlerTest` covers request order, positional matching,
the cross-tier earliest cache, failures, and a response-count mismatch.
`ReplicaManagerInklessTest` checks that the KC-702 recreated leader and
an empty replica of a consolidating topic get `OFFSET_NOT_AVAILABLE`,
and that a caught-up replica gets the diskless offset. Four existing
fallback tests now create a local log at the seal.
`InklessDisklessListOffsetsTest` passes against Postgres.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): cover the unresolved epoch of the log start in timestamp lookups [KC-761]

No test exercised the branch of the searchable end that drops to the
log start while the leader epoch cache can't resolve it. The router
tests stub the searchable end, and the broker cases had no remote
prefix, so returning the log end offset unconditionally still passed.

The new `ReplicaManagerInklessTest` case leaves a remote-enabled log in
the state that a tier-state rebuild leaves before it restores the
cache: log start 0, local log start and log end 900, past the WAL
start at 800, and no epoch for the log start. It checks that a
timestamp lookup gets `OFFSET_NOT_AVAILABLE`, and that after the cache
is restored the lookup gets the diskless offset. With the branch
disabled, only this test fails. The consolidating replica tests now
share their broker setup.

Assisted-by: Claude Opus 5.5

* fix(inkless:switch): check that the search range is current when a timestamp lookup decides [KC-761]

The searchable end of the local log came from unsynchronized reads. A
tier-state rebuild could truncate the log between them and leave a
range that claims the remote prefix even though the local search can't
read it. If the rebuild restores the leader epoch cache before the
control plane answers, a second read agrees with the first, and the
guard returns a diskless offset that skips the prefix.

The broker now reads the log's truncation count before the range, as
`updateHighestOffsetInRemoteStorage` callers do, and the range counts
only while the log is the same instance with the same count. A
truncation that the reads straddle, a rebuild or truncation during the
lookup, and a recreation all make the range unknown, and the lookup
gets `OFFSET_NOT_AVAILABLE`. This check replaces the second read.

Testing

`ReplicaManagerInklessTest` runs a rebuild inside the remote search: it
truncates the log and restores the epoch cache before the control plane
answers. The lookup gets `OFFSET_NOT_AVAILABLE`, and a lookup without a
rebuild gets the diskless offset. Without the check, the test fails.
`DisklessFetchOffsetRouterTest` covers a range that stops being current
on a switched and on a consolidating partition, before routing and
before the control plane answers, and that the router reads the range
once before the classic lookup.

Assisted-by: Claude Opus 5.5

* fix(inkless:switch): end the search range when the log start moves or epochs restore in part [KC-761]

Two more states let a remote timestamp search read nothing while the
search range still counted as current, so the lookup returned a
diskless offset that skips retained records:

- Retention or `DeleteRecords` moves the log start while the remote
  search is queued. The search keeps the log start it read, and
  `truncateFromStartAsyncFlush` drops the epoch of that offset. The
  truncation count doesn't change.
- A tier-state rebuild restores the epoch cache with
  `LeaderEpochFileCache.assign(List)`, which took the write lock once
  per entry. A reader could see the epoch of the log start before the
  later epochs, and `RemoteLogManager` rejects segments in epochs that
  aren't in the cache yet.

The search range now also records the log start and counts only while
it's unchanged. `assign(List)` holds the write lock across all entries,
so a reader sees none of them or all of them. `INKLESS_OWNERSHIP` lists
`LeaderEpochFileCache.java` as interleaved.

Testing

`ReplicaManagerInklessTest` moves the log start inside the remote
search, checks that the epoch of the search's start offset is gone, and
expects `OFFSET_NOT_AVAILABLE`. A lookup without the move gets the
diskless offset. `LeaderEpochFileCacheTest` assigns 2000 entries 20
times while a reader thread records the cache sizes it sees, and checks
that it only sees 0 or 2000. Each test fails without its fix.
`LeaderEpochFileCacheTest`, `RemoteLogManagerTest`, `UnifiedLogTest`,
and `TierStateMachineTest` pass.

Assisted-by: Claude Opus 5.5

* fix(inkless:switch): skip the WAL start check in phase 1 and tighten checks [KC-761]

Review follow-ups for the timestamp lookup guard:

- The WAL start check ran in phase 1 of the switch, where the broker
  doesn't consolidate and the local log stays at the seal. A
  `DeleteRecords` past the seal moves the WAL start over deleted
  offsets, and every local miss then failed until phase 2. The check
  now runs only while consolidation is active, as
  `consolidationActiveFor` decides. The seal check still applies.
- A missing or negative WAL start now fails the check. Before, a
  negative offset passed it.
- `FetchOffsetHandler.Job` checks that each control-plane response is
  for the partition of the request at the same position, and fails
  every lookup on a mismatch.
- `LeaderEpochFileCache.assign(List)` validates every entry before it
  changes the cache, so an invalid entry leaves none of the list
  applied.

Testing

`DisklessFetchOffsetRouterTest` covers phase 1 with a WAL start past
the seal, the seal check in phase 1, and a negative WAL start.
`FetchOffsetHandlerTest` covers responses in another order.
`LeaderEpochFileCacheTest` covers a list with an invalid entry. Each
test fails without its fix. `ReplicaManagerInklessTest` and
`TierStateMachineTest` pass.

Assisted-by: Claude Opus 5.5

* feat(inkless:switch): count deferred timestamp lookups in a meter [KC-761]

A replica that can't search the range before the diskless log answers
timestamp `ListOffsets` lookups with `OFFSET_NOT_AVAILABLE` until it
rebuilds. If it never rebuilds, only client errors show it. The Fetch
path already counts its deferrals in
`ConsolidationDeferredPrefixFetchRate`.

`ReplicaManager` now registers `ConsolidationDeferredListOffsetsRate`,
and the router marks it each time it defers a timestamp lookup: on a
switched replica below the seal, and on a consolidating replica that
ends before the WAL start. Errors that the router forwards from the
control plane don't count. `DISKLESS_CONSOLIDATION.md` lists the
metric.

Testing

`DisklessFetchOffsetRouterTest` checks that each deferred lookup counts
once, and that a diskless result, forwarded errors, and phase 1 lookups
don't count. `ReplicaManagerInklessTest` reads the meter for the KC-702
recreated leader and for an empty and a caught-up consolidating
replica. Without the marks, those tests fail.

Assisted-by: Claude Opus 5.5
(cherry picked from commit 12c5120)
…e 1 [KC-759] (#871)

* docs(inkless:switch): describe the last-known-leader election in phase 1 [KC-759]

The phase 1 guards block unclean leader election, but they don't block
the last-known-leader election that KIP-966 adds. With ELR enabled, a
partition whose ISR and ELR members all shut down uncleanly elects its
last known leader when that broker returns, resets the ISR to it, and
loses any records that broker lost. The docs said such a partition has
no leader until an ISR member returns.

The switch doc now describes when the controller runs this election and
what it does to a partition in phase 1. If the leader lost records below
the seal, the leader marks the partition offline, the controller keeps
it as leader, and phase 2 doesn't start. The doc gives the recovery: a
downward re-seal to the leader's log end offset, a leader restart, and a
control-plane repair. That recovery loses the records between the
leader's log end offset and the old seal.

Testing

`ClassicToDisklessSwitchLastKnownLeaderClusterTest` holds phase 1 open
with a gated remote storage manager, drives the partition into the
last-known-leader election through unclean shutdowns, and checks that
the leader marks the partition offline. It then runs the re-seal
recovery and checks that phase 2 starts and that a consumer reads the
retained classic records followed by the diskless records.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): assert the ELR recovery states instead of logging them

The last-known-leader test logged each stage while it was being written.
It now asserts those states: the leader recovery state after the
election, the seal command's offset range, the partition staying offline
after the re-seal until the leader restarts, and the log end offset on
every replica in phase 2. The offline check waits for the leader to
apply the lower seal instead of sleeping for a fixed time.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): check replica log end offsets before phase 2

Once the upload gate opens, phase 2 consolidates from the seal, and a
replica can append the diskless records and move its log end offset
past the seal before the test checks it. The test now checks the log
end offsets after the ISR recovers and before it opens the gate, and
asserts that phase 2 hasn't started at that point.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): cover both outcomes of the last-known-leader election

The switch doc describes two outcomes of the last-known-leader election
in phase 1, and the test ran only the one where the leader lost records.
The test is now parametrized on that loss. Without it, the leader keeps
the partition online and finishes recovery, the ISR returns to 3, phase
2 starts at the original seal, and a consumer reads every record.

The doc also says that the `eligible.leader.replicas.version` feature is
on by default at 4.1 and later, instead of reading as an opt-in.

Assisted-by: Claude Opus 5.5
(cherry picked from commit 10a852b)
…cover faults before phase 2 [KC-747] (#869)

* test(inkless:switch): run switch system tests with consolidation and cover faults before phase 2 [KC-747]

Only one of the 14 switch system tests started brokers with consolidation.
The rest never left phase 1, so forced deletion of the local prefix, the
divergent-tail check, the rebuild from remote storage, and consolidation
from the seal ran under faults only in unit tests. `_wait_for_switch_complete`
also returned at control-plane init, because `InFlightPartitions` counts only
steps 3 to 5.

The brokers now consolidate by default, and the switch alter adds
`remote.storage.enable`. The completion wait then requires
`classicPrefixInRemoteStorage` on every partition through
`kafka-topic-switch.sh state`, and records the seal of each partition.

Each data check now compares every consumed record with the acked ledger:
partition, offset, key, and value. It fails on a missing, duplicated,
out-of-order, changed, or never-acked record, and labels each one below or at
or above the seal. Producers fail the test on any send error.

`test_fault_between_seal_commit_and_phase_2` holds every partition in
`WaitingForUpload` by dropping broker traffic to object storage, asserts the
hold through JMX and the switch state, then injects one fault: a leader
restart, SIGKILL, or SIGSTOP, a follower SIGKILL, a rolling restart, a leader
network partition, a reassignment, or a topic deletion. Three more faults
start as the hold ends, so they overlap the flag request and the phase 2
transition: a failover of the active controller of three, a Postgres outage,
and a rolling restart. The hold alone is the object-storage outage case.

Testing

The new diff and parsing helpers were checked offline against synthetic
records. `pyflakes` and `make vale` are clean. The Inkless System Tests
workflow runs the suite on this branch.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): reject every repeated acked offset in the switch test ledger

Each producer run restarts the same key and value sequence. If offsets
regressed, a new batch could be acked at existing positions with identical
content, and the ledger accepted it and hid the lost records. Any second
ack for a partition and offset now fails the test.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): wait on phase 2 alone when the brokers consolidate

A leader that takes over a consolidating partition in phase 2, for
example after a restart, doesn't seal it, so the partition drops out of
`SealedPartitionsCount`. The phase 1 wait then times out after a rolling
restart even though the switch finished. The leader requests the phase 2
flag only after control-plane init, so a consolidating cluster now waits
on the topic config and the phase 2 flag only.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): fix rolling restarts with a spare broker and pin the flag request to the controller failover

The Category D cluster has four brokers and a replication factor of three.
`_rolling_restart()` waited for every broker to join the ISR of partition 0,
and the broker outside its replicas never does. The helper now waits only
for the replicas of partition 0.

The URP gauge leaves out partitions that consolidate. In phase 2, the
switched partition no longer counts, and `__remote_log_metadata` could raise
the gauge instead. `test_switched_topic_urp_clears_after_replica_recovery`
now runs without consolidation, so it stays in phase 1.

`controller_failover_at_phase_2` could pass without a flag request meeting
the failover, because the remaining controllers elect a leader within
seconds. The test now stops two of three controllers, waits until a flag
request is pending for 10 s, and then starts one of them.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): keep a broker down across the phase 2 transition in the rolling restart fault

`rolling_restart_at_phase_2` lifted the hold and then started a rolling
restart. Every broker could finish the upload and get the flag before the
first broker stopped, so the test could pass without a restart overlapping
phase 2. The test now stops the leader of partition 0 before it lifts the
hold and keeps it down until every partition is in phase 2. The leader then
rejoins in phase 2, and the other brokers restart one at a time.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): stop the Postgres outage fault from leaking sessions into later tests

The `postgres_outage_at_phase_2` fault failed, and so did the four Category D
tests that ran after it.

The fault waited for phase 2 with `kafka-topic-switch.sh state`. That command
lists offsets, which needs the control plane, so every call failed during the
outage. The broker logs show that the controller accepted every flag about
20 s into the outage. The test now waits on the `WaitingForUpload` and
`SendingFlagToController` gauges, which don't need Postgres, and confirms
phase 2 with `state` after it lifts the block.

The outage drops broker traffic to Postgres. Postgres never sees the brokers
close their connections, so it keeps those sessions for about two hours.
The four tests after the fault failed with `too many clients already`. After
a test blocks the control plane, the teardown now stops the brokers and ends
the Postgres sessions from their addresses.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): find the partition leader from the broker gauge, not the client metadata

For a diskless topic, `InklessTopicMetadataTransformer` rewrites the leader
in Metadata and DescribeTopicPartitions responses to a replica it picks by
hash. `_get_leader_node` read `kafka-topics.sh --describe`, so after the
switch it could return a follower. A leader fault could then hit a follower,
and a follower fault could hit the leader, and the test would still pass.

The helper now reads the `ReplicasCount` gauge of the partition on each live
broker with a one-time `JmxTool` call. Only the leader reports a nonzero
value, in every phase of the switch. The helper waits until exactly one
broker reports it.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): end the Postgres sessions even if a broker doesn't stop in teardown

`KafkaService.stop_node` raises when a broker doesn't exit within 60 s.
The teardown then skipped `terminate_broker_sessions()`, and the stale
sessions filled `max_connections` for the tests that ran next. The teardown
now kills the brokers and ends the sessions in a `finally` block.

Assisted-by: Claude Opus 5.5
(cherry picked from commit e490e36)
…856)

Outside a git checkout, `git rev-parse HEAD` printed nothing, so the
Makefile passed an empty `-PcommitId=`. `determineCommitId()` saw the
property and used the empty value as the commit id.

Pass `-PcommitId` only when `git rev-parse HEAD` succeeds. Otherwise
Gradle falls back to "unknown".

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit f36b136)
…f sleeping [KC-349] (#795)

* fix(inkless:delete): skip file cleaner ticks during backoff instead of sleeping [KC-349]

`KafkaScheduler` runs the file cleaner with `scheduleAtFixedRate`. When
`run()` sleeps, for the idle backoff or the error backoff, the missed
ticks queue up and then fire back to back. Those cycles delete more than
`file.cleaner.max.files.per.cycle` files per interval, which breaks the
pacing the operator configured.

An empty cycle now returns immediately. After an error, the cleaner
records when the backoff ends and skips ticks until then, the same way
`TopicPurger` handles errors.

`InklessDisklessTopicDeleteTest` counts saturated `TopicPurger` cycles
from the log. The purger logs after its transaction commits, so the test
polls the captured log instead of counting once. Each broker checks
`moreRemain` before it commits, so only one cycle can see the drain
finish, and the test expects at least `batchesAtDelete - 1` saturated
cycles. `TopicPurger` exposes the log text that the test matches. The
test sets the `TopicPurger` log level to INFO, so the core checkstyle
rules now allow the log4j `Level` class in `kafka.server`, and
`INKLESS_OWNERSHIP` lists that rules file as interleaved.

Testing

`FileCleanerMockedTest` checks that an empty cycle doesn't advance time,
and that after an error the cleaner skips ticks until the backoff ends
and then runs again. `InklessDisklessTopicDeleteTest` checks that the
purger hits the per-cycle cap on every deleting cycle but the last.

Assisted-by: Claude Opus 5.5

* test(inkless:delete): drop the TopicPurger log level override [KC-349]

`core/src/test/resources/log4j2.yaml` already sets `io.aiven.inkless` to
DEBUG, so `InklessDisklessTopicDeleteTest` captures the `TopicPurger`
INFO lines without raising the level itself. Without that call, the test
no longer needs the log4j `Level` class, and
`checkstyle/import-control-core.xml` goes back to its state on `main`.

The `nextEligibleMs` comments in `FileCleaner` and `TopicPurger` now say
that a sleep in `run()` holds a thread of the shared `KafkaScheduler`,
and that missed ticks fire back to back only when a run outlasts the
interval.

Testing

`InklessDisklessTopicDeleteTest` passes without the override and logs 11
saturated `TopicPurger` cycles. `FileCleanerMockedTest` and
`TopicPurgerMockedTest` pass.

Assisted-by: Claude Opus 5.5
(cherry picked from commit 5491ff6)
…nd defer lookups past a stale high watermark [KC-756] [KC-763] (#873)

The router sent a pending partition of a consolidating topic to the control
plane, which has no entry for the partition until the seal commits. The
client got `UNKNOWN_TOPIC_OR_PARTITION`, and `kafka-topic-switch.sh state`
failed on the partitions an operator needs it for. A pending partition now
answers from the classic log on every topic. The leader and the in-sync
followers answer client requests, because the metadata transformer sends
clients to any alive ISR member while the switch is pending.

A client timestamp lookup hides a match at or past the last fetchable
offset. The router read that answer as a miss and returned the control-plane
offset, so a consumer skipped the records between a stale high watermark and
the start of the diskless log. The searchable range of a client lookup now
ends at the last fetchable offset, so the existing guard answers
`OFFSET_NOT_AVAILABLE` while that offset is before the diskless log.

Assisted-by: Claude Fable 5.2
(cherry picked from commit 41f177a)
…ic config (#875)

* test(inkless:metadata): add JMH benchmark for topic type flag reads

`InklessMetadataViewTopicFlagBenchmark` measures `isDisklessTopic` and
`isConsolidatingDisklessTopic` on an image of 1000 topics, with 0, 10,
or 30 additional overrides per topic. Half of the topics are diskless
with remote storage, and they also have the two flag overrides. Run it
with `-prof gc` to see the allocation per call.

`jmh-benchmarks` now depends on `:storage:inkless`, because
`InklessMetadataView` implements `MetadataView` from that module.

Assisted-by: Claude Opus 5.5

* perf(inkless:metadata): read topic type flags without copying the topic config

`InklessMetadataView.isDisklessTopic` and `isConsolidatingDisklessTopic`
run for each partition on the produce, fetch, and leader-change paths.
They called `KRaftMetadataCache.topicConfig`, which copies every override
of the topic into a new `Properties`, and `isConsolidatingDisklessTopic`
made two copies. `isRemoteStorageEnabled` read the same way.

The flags now read one key from
`ConfigurationsImage.configMapForResource`, a read-only view of the same
image. `isConsolidatingDisklessTopic` reads both flags from one image.
The `MetadataView` Javadoc states the contract: only topic overrides
count, and broker defaults don't apply.

With 10 additional overrides per topic, `isDisklessTopic` goes from
174 ns and 640 B to 16 ns and 24 B per call. `isConsolidatingDisklessTopic` goes
from 278 ns and 1036 B to 18 ns and 24 B. The cost no longer grows with
the number of overrides.

A value other than `true` or `false`, ignoring case, still throws, as
on `main`. `ConfigDef` trims before it validates, so an alter can store
a value such as ` true`. No boolean reading of that value is safe. False
misreads a diskless topic as classic, and a consolidating leader then
appends to its local log instead of the diskless WAL. True misreads a
classic topic, for which the controller wrote no switch records. The
broker logs a warning when a topic config update has such a value.

`getDisklessTopicPartitions` and `getConsolidatingDisklessTopicPartitions`
listed topics with `getAllTopics`, then called `numPartitions(t).get()`
and `getTopicId(t)` for each topic. Each call read the current image
again. If a new image deleted a topic during the scan, `Optional.get()`
threw `NoSuchElementException`, and `getTopicId` returned
`Uuid.ZERO_UUID`. The scans now read the topics, their IDs, partitions,
and configs from one image.

Assisted-by: Claude Opus 5.5
(cherry picked from commit a347e90)
…e switch [KC-753] (#874)

* fix(inkless:switch): abort open transactions only on the leader at the switch [KC-753]

A transaction can be open on a classic partition when the switch to
diskless starts. `Partition.seal()` aborted it on every replica, and the
transaction coordinator couldn't complete it:

- A follower appended its own `ABORT` marker with `appendAsLeader`.
- The leader appended its marker before `makeLeader`, in the previous
  leader epoch.
- `WriteTxnMarkers` for a diskless topic returned
  `INVALID_TOPIC_EXCEPTION`. `TransactionMarkerRequestCompletionHandler`
  has no case for that error, and the transaction stayed in
  `PrepareCommit`.

`seal()` no longer aborts. The leader calls
`abortOngoingTransactionsAsSealedLeader()` after `makeLeader`, only while
the switch is pending, and a follower replicates the leader's markers.
`WriteTxnMarkers` for a diskless topic returns
`UNSUPPORTED_FOR_MESSAGE_FORMAT`, which the coordinator handles by
dropping the partition from the transaction.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): advance the high watermark after the sealed leader aborts transactions

The sealed leader appends its `ABORT` markers after `makeLeader`. If the
leader is the only ISR member, no follower fetch advances the high
watermark past the markers, and `InitDisklessLogManager` waits for
`HW == LEO` without end. The leader now advances the high watermark
itself after it appends the markers.

Assisted-by: Claude Fable 5.2

* docs(inkless:switch): require coordinator-side completion before a switch

The coordinator sends the `EndTxn` response before it sends the markers
to the partitions, so a commit that returns to the producer doesn't mean
the partition's transaction is decided. The guidance now tells operators
to stop the transactional producers and wait until the coordinator
reports `CompleteCommit` or `CompleteAbort` with no pending partitions.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): register a pending switch when the leader has no partition yet

A leader that loads a pending switch with no `Partition` yet, such as
after a restart, takes the post-restart branch of
`applyLocalLeadersDelta`. That branch sealed the partition but didn't
register it with `InitDisklessLogManager`, so nothing proposed the seal
offset until the next leader epoch. The branch now registers the
partition, like the branch for an existing partition.

The log messages for the leader abort and the segment roll say "after
sealing", because both now run after the seal.

Assisted-by: Claude Fable 5.2
(cherry picked from commit e3913e8)
…earm and lower-seal (#872)

* refactor(inkless:switch): remove the seal and repair commands

Remove the `seal` and `repair` commands of `kafka-topic-switch.sh`, the
`RepairDisklessLog` RPC with its admin, broker, and control plane code and
metrics, and the `ClearProducerStates` field of `AlterDisklessSwitch`.
Migration `V32` drops the `repair_diskless_log_v1` function and types, and
the jOOQ code is regenerated.

Assisted-by: Claude Opus 5.5

* feat(inkless:switch): add rearm and lower-seal on AlterDisklessSwitch

`AlterDisklessSwitch` now accepts two seal offsets and rejects every other
one:

- `-2` re-arms a pending switch. The controller bumps the leader epoch so
  the leader seals again.
- A seal of at least 0 lowers a committed seal in phase 1, and must be
  lower than the committed seal. It keeps the recovery after a
  last-known-leader election picks a leader whose log ends below the seal.

`kafka-topic-switch.sh rearm` sends `-2`, and `lower-seal --offset N`
sends `N`. `lower-seal` rejects a negative offset, because the controller
reads `-2` as a re-arm. The last-known-leader recovery uses `lower-seal`
and has no `repair` step.

Assisted-by: Claude Opus 5.5

* test(inkless:switch): cover rearm and lower-seal and drop the seal and repair tests

Remove the tests of the `seal` and `repair` commands, `RepairDisklessLog`,
and `ClearProducerStates`. Add tests that check:

- `AlterDisklessSwitch` accepts `-2` only on a pending switch, and a lower
  seal only on a committed seal in phase 1. It rejects every other seal
  offset, and the lower-seal record survives serialization.
- `rearm` and `lower-seal` send the expected seal offset, and `lower-seal`
  rejects a negative offset without sending a request.
- Lowering `min.insync.replicas` and running `rearm` lets a pending switch
  that is stuck below `min.insync.replicas` reach the seal.
- A committed switch rejects a re-arm, an abort, and a seal that isn't
  lower, in a running cluster.

`ClassicToDisklessSwitchLastKnownLeaderClusterTest` lowers the seal with
`alterDisklessSwitch` and no longer runs `repair`.

Assisted-by: Claude Opus 5.5
(cherry picked from commit f0d5300)
…ends below the seal [KC-773] (#880)

After a last-known-leader election in phase 1 of the classic-to-diskless
switch, the restarted leader runs `makeLeader` and then fences itself when its
log ends below the committed seal. A follower fetch between those two steps
expands the ISR. The controller then records a full ISR and a recovered
leader for a partition that the leader marks offline. When the operator lowers
the seal and restarts the leader, the controller elects another ISR member,
and `ClassicToDisklessSwitchLastKnownLeaderClusterTest` times out waiting for
the restarted broker to lead.

`Partition.isFollowerInSync` now returns false while the leader's log end
offset is below the committed seal. The check reads the stop offset directly,
because `seal()` runs after `makeLeader` on a restarted leader.

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit 82f45b0)
…ess log init of others [KC-748] [KC-750] [KC-757] (#876)

* fix(inkless:switch): remove deleted partitions from the init tracking [KC-757]

`ReplicaManager` removed a partition from `InitDisklessLogManager` only
when it unsealed a classic topic or became a follower. A partition that
the broker deleted in `AwaitingMetadata` stayed tracked. Its
control-plane init retried without a limit, with a backoff of up to 10
seconds, and `InFlightPartitions` stayed above `0`. The tracking is
keyed by topic name, and `registerPartition` kept the old entry when a
topic with the same name started its switch. The broker then didn't
send `InitDisklessLog` for the new topic.

`applyDelta` now removes every deleted partition from the tracking. This
covers a topic deletion and a reassignment that takes the replica away
from the broker. `registerPartition` also drops an entry that has
another topic ID.

Testing

`ReplicaManagerInklessTest` checks that `applyDelta` removes the
partition for a deleted topic and for a reassignment.
`InitDisklessLogManagerTest` checks that a partition removed in
`AwaitingMetadata` stops retrying and leaves `InFlightPartitions` at
`0`, and that a topic with the same name and another topic ID replaces
the old entry and sends its own `InitDisklessLog` request.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): keep one partition from blocking the control-plane init of its batch [KC-748]

`AwaitingMetadata.sendBatch` builds one control-plane request per
partition with the local log start offset and the seal. The
`InitDisklessLogRequest` constructor throws if the log start offset is
past the seal. The exception left `sendBatch`, and
`RetriableInitDisklessLogBatchQueue.task` re-queued every partition of
the batch without logging anything. The partition that threw stayed in
the queue, and every later batch on that broker failed the same way. A
partition in the same batch whose seal had just committed didn't get its
control-plane row.

The request now sends `min(logStartOffset, seal)` as the log start
offset. If the log starts past the seal, the classic prefix is empty and
starts at the seal. The control plane answers `already_initialized` for
a partition that it already has, and the init completes.

If the leader can't build the request of a partition for another reason,
it logs the error and retries that partition alone. The queue also logs
the exception when a whole batch fails to send.

Testing

`InitDisklessLogManagerTest` checks a batch that mixes a healthy
partition with a partition whose log start is past the seal: one
control-plane call has both requests, the log start offset of the second
one is the seal, and both partitions leave the tracking. It also checks
that a partition whose request can't be built retries alone while the
healthy partition completes.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): bound the producer state in an InitDisklessLog request [KC-750]

The broker sends every ready partition in one `InitDisklessLog` request,
with one producer state entry per retained batch of each active
producer. The controller writes one `PartitionChangeRecord` per
partition, with 34 bytes per entry, and appends the records of the
request as one metadata batch. The Raft client rejects a batch over
8 MiB with `RecordBatchTooLargeException`.
`QuorumController.appendRecords` wraps that exception in a
`RuntimeException`, which the controller handles as a fault: the active
controller resigns, and the broker gets `UNKNOWN_SERVER_ERROR`. The
broker treats that error as retriable and sends the same request again,
while the partitions stay sealed.

The records of one request now have at most
`InitDisklessLogFields.MAX_PRODUCER_STATE_ENTRIES` producer state
entries. That's 123,361 entries, which take half of the Raft batch
size. A snapshot writes the producer states of a partition in one
record, and the same limit applies to a single partition.

- The controller rejects a partition whose entries don't fit in the
  request with `RECORD_LIST_TOO_LARGE` and doesn't write a record for
  it. The broker retries that partition in a later request.
- If the ready partitions exceed the limit together, the broker sends
  them in several requests. A partition that waits for a later request
  doesn't count as a retry.
- If one partition exceeds the limit, the broker sends only the newest
  batch of each producer. The control plane checks the next sequence of
  a producer against the highest sequence it stores for that producer,
  and a retry of an older batch then fails with
  `OUT_OF_ORDER_SEQUENCE_NUMBER`. If the producers still don't fit, the
  broker keeps the ones whose newest batch has the highest offset and
  logs a warning with the number of producers it dropped. The control
  plane rejects the next batch of a dropped producer with
  `OUT_OF_ORDER_SEQUENCE_NUMBER` until the producer bumps its epoch.

If the broker can't build the request entry of a partition, it logs the
error and retries that partition alone.

Testing

`ReplicationControlManagerInklessTest` sends a request whose records
exceed the Raft batch size. The controller commits two partitions per
request, rejects the others with `RECORD_LIST_TOO_LARGE`, and each
result stays under half of the batch size. A second test checks a
single partition one entry over the limit and one at the limit.

`InitDisklessLogManagerTest` runs with a limit of a few entries and
checks that the broker splits three partitions over two requests
without counting a retry, keeps the newest batch of each producer for a
partition over the limit, keeps the producers with the highest offsets
when the newest batches still don't fit, retries a partition that the
controller rejects with `RECORD_LIST_TOO_LARGE`, and retries alone a
partition whose producer state can't be read.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): split metadata snapshot batches by size [KC-750]

A metadata snapshot writes the committed producer states of a switched
partition in the `PartitionRecord` of that partition, and it writes the
partitions of a topic one after the other. `RaftSnapshotWriter` appended
up to 1,024 records as one batch without checking their size. The Raft
snapshot writer rejects a batch over 8 MiB with
`RecordBatchTooLargeException`. `SnapshotEmitter` rethrows the
exception, and `SnapshotGenerator` reports it to its fault handler.

With the limit per `InitDisklessLog` request, the partitions of one
topic commit their producer states over several requests. Three
partitions with 123,361 producer state entries each take 12.6 MB in one
snapshot batch.

`RaftSnapshotWriter` now starts a new batch when the next record doesn't
fit in 4 MiB, which is half of the Raft batch size. A record over that
size goes in a batch alone. `INKLESS_OWNERSHIP` lists the file as
interleaved.

Testing

`RaftSnapshotWriterTest` writes three such partition records through
the Raft `RecordsSnapshotWriter` with its 8 MiB batch size. The writer
rejects them as one batch, and `RaftSnapshotWriter` writes and freezes
the snapshot. The test fails with `RecordBatchTooLargeException` when
the size check is off. A second test checks the batch boundaries with a
small size limit.

Assisted-by: Claude Fable 5.2

* docs(inkless:switch): describe what a producer dropped by the trim loses [KC-750]

If the newest batches of the producers of one partition exceed the
producer state limit, the broker drops the producers that wrote first.
The control plane doesn't store any state for a dropped producer. The
docs and the comment on the trim described only the rejection of its
next batch with `OUT_OF_ORDER_SEQUENCE_NUMBER`.

They now also say that a dropped producer loses duplicate detection and
epoch fencing: the control plane appends a retry of a batch that starts
at sequence 0 again, and it accepts a batch from an older epoch of that
producer.

Assisted-by: Claude Fable 5.2

* fix(inkless:switch): keep a late controller response from advancing a replaced state [KC-757]

When an `InitDisklessLog` request succeeded, `InitDisklessLogManager`
advanced whatever `SendingToController` state it tracked for the
partition. The queue completes a promise outside its lock, and the
response thread can run the callback of a request after the metadata
thread replaced its state, such as with the state of a topic that
reuses the name of a deleted topic. The replacement then moved to
`AwaitingMetadata` before the controller accepted its own request. If
the controller rejected that request, the rejection didn't remove the
partition, which stayed tracked in `AwaitingMetadata`.

The success callback now advances only the state that queued the
request, compared by reference. The callbacks of the control-plane init
and of the flag request already compare the state that way.

Testing

`InitDisklessLogManagerTest` replaces a state whose request is in flight
with the state of a topic that has the same name, and then runs the
success callback of the old request. The replacement stays in
`SendingToController` until the controller accepts its own request. The
test fails without the reference check. The callback is now a method
that the test can call, because the manager has no other seam between
the queue releasing its lock and completing the promise.

Assisted-by: Claude Fable 5.2

* test(inkless:switch): cover the deferred re-queue and fix the batch wording [KC-750]

The re-queue in `deferOverflow` had two branches that no test reached. Removing the loop that
puts back the states that arrived during the task, or the check for a canceled promise, left
`InitDisklessLogManagerTest` green. Two tests now cover them: a partition registered while the
queue defers another one is still sent, and a deferred partition that `removePartition` cancels
isn't sent.

The docs and comments said the controller appends the records of an `InitDisklessLog` request
as one metadata batch. `QuorumController.appendRecords` splits a non-atomic result into batches of
at most 10,000 records. The text now says so.

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit 2157154)
…c switch state [KC-840] (#881)

* fix(inkless:switch): print partitions whose ListOffsets fails in topic switch state [KC-840]

`kafka-topic-switch.sh state` read the offsets with `ListOffsetsResult.all()`,
which fails if any partition fails. Between the seal commit and the control
plane initialization, `ListOffsets` fails with `UNKNOWN_TOPIC_OR_PARTITION`
for such a partition, so the command printed nothing for the topic. That is the
case an operator runs `state` for.

The command now reads each partition's result separately and prints
`unavailable` for an offset that failed.

Assisted-by: Claude Sonnet 5.5

* style(inkless:switch): fix import order in TopicSwitchCommandTest [KC-840]

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit e3c990e)
* feat(inkless:storage): add per-role storage config overrides

The fetch, produce, and background storage clients all read the same
`storage.*` keys. FileCleaner `DeleteObjects` requests (up to 1000 keys) time
out at the shared 2000 ms S3 call timeout, and a longer shared timeout would
also loosen the produce and fetch paths.

`InklessConfig.storage(metrics, role)` now builds the settings of each client
from three layers, lowest first: the role defaults, the shared `storage.*` keys,
and the `storage.<role>.*` keys. The roles are `fetch`, `produce`, and
`background`. The lagging fetch storage uses the `fetch` role.

The `background` role defaults to `s3.api.call.timeout=30000`,
`s3.api.call.attempt.timeout=10000`, and `gcs.read.timeout=10000`. It serves
FileCleaner deletes and consolidation cold reads. The `fetch` and `produce`
roles keep the shared defaults. Azure Blob Storage has no timeout setting, so
it has no role default. An operator can change any of the values with
`storage.background.s3.api.call.timeout` and similar keys.

Assisted-by: Claude Sonnet 5.5

* amend! feat(inkless:storage): add per-role storage config overrides

The fetch, produce, and background storage clients all read the same
`storage.*` keys. FileCleaner `DeleteObjects` requests (up to 1000 keys) time
out at the shared 2000 ms S3 call timeout, and a longer shared timeout would
also loosen the produce and fetch paths.

`InklessConfig.storage(metrics, role)` now builds the settings of each client
from three layers, lowest first: the role defaults, the shared `storage.*` keys,
and the `storage.<role>.*` keys. The roles are `fetch`, `produce`, and
`background`. The lagging fetch storage uses the `fetch` role.

The `background` role defaults to `s3.api.call.timeout=30000`,
`s3.api.call.attempt.timeout=10000`, and `gcs.read.timeout=10000`. It serves
FileCleaner deletes and consolidation cold reads. The `fetch` and `produce`
roles keep the shared defaults. Azure Blob Storage has no timeout setting, so
it has no role default. An operator can change any of the values with
`storage.background.s3.api.call.timeout` and similar keys.

The two S3 defaults apply as a pair. If the shared or role layer sets either S3
timeout, the `background` role keeps the shared default for the other one, so
the attempt timeout never exceeds the call timeout. The GCS default is
independent.

Assisted-by: Claude Sonnet 5.5
(cherry picked from commit 39946c3)
…st delta after a restart [KC-752] (#877)

* fix(inkless:switch): seal and register a pending partition in the first delta after a restart [KC-752]

A broker that crashes and restarts before its session expires can apply a
first metadata image that still shows it as the leader of a partition in
`CLASSIC_TO_DISKLESS_SWITCH_PENDING`. No `Partition` object exists yet, so
`applyLocalLeadersDelta` skipped the switch branch. For a consolidating topic,
the consolidating branch made the broker the leader without sealing or
registering, and no broker sent `InitDisklessLog` until the controller fenced
the old session and bumped the leader epoch.

A pending leader now always takes the switch branch, for a consolidating topic
too. If the partition doesn't exist, the broker creates it, makes it the
leader, seals it, aborts open transactions, and registers it. The post-restart
branch handles only committed seals.

Assisted-by: Claude Opus 5.5
Assisted-by: Claude Sonnet 5.5

* fix(inkless:switch): skip a pending leader that has no local log [KC-752]

A broker can lead a pending switch without a local log, such as after a disk
replacement or when its partition directory is missing and an ELR election
picks it. The switch branch then created the partition, and makeLeader created
an empty log. registerPartition proposed that log's high watermark of 0 as
the seal, and the controller committed it, so the followers truncated the
classic prefix they hold.

If the local log is missing, the switch branch now logs a warning and skips
the partition, as applyLocalFollowersDelta does for a pending switch. The
broker doesn't create a log or register the partition.

Assisted-by: Claude Opus 5.5
(cherry picked from commit 53b3422)
…r [KC-766] (#878)

* feat(inkless:metrics): add PartitionRateTracker [KC-766]

No broker metric shows how unequal client traffic is across
partitions. The broker meters `BytesInPerSec` and `BytesOutPerSec` are
per broker and per topic, so skew inside a topic is invisible.

`PartitionRateTracker` turns per-partition byte counts into the
distribution of partition rates on one broker. Callers record bytes. A
periodic snapshot computes the rate of each partition with traffic and
publishes, per direction and topic type: the `P50`, `P95`, `P99`,
`P999`, and `Max` rates, cumulative `le` buckets, the sum of the rates,
and the active and hosted partition counts.

The hot path does one map lookup and one `LongAdder` add. The snapshot
resolves the topic type and the role through a placement function, once
per partition per window, and drops the partitions that the broker no
longer hosts, which leaves the tracker without a removal hook. Each map
entry stores the counters of both directions of a partition.

HdrHistogram keeps the percentiles and the buckets within 1% of the
recorded rates. The snapshot resets and reuses the histograms. Apart
from the placement function, a snapshot allocates a constant amount.
The gauges read an immutable snapshot instead of the partitions.

With 100,000 partitions that have traffic in both directions, the
tracker uses about 225 B per partition and a snapshot takes about 4 ms.

The new `inkless.partition.rate.window.ms` config sets the window: 1 min
by default, and at least 1 s. The next commit adds the hooks that record
bytes.

Assisted-by: Claude Opus 5.5

* feat(inkless:metrics): record client bytes for partition rates [KC-766]

Record client bytes per partition and publish the partition rate
metrics under `kafka.server:type=ReplicaManager`.

The produce response callback of `KafkaApis` records bytes in for the
partitions with `Errors.NONE`, so a failed append doesn't count. The
handler keeps only the record sizes for the callback, and the records
can go while the request waits in purgatory.

`recordBytesOutMetric` records bytes out for consumer fetches. It runs
on the response that the broker sends, after the fetch quota decision.
A throttled fetch doesn't send records, so it doesn't count.

Coordinator writes and replication don't use these hooks, because the
hooks run only for client requests. They skip Kafka internal topics.

`ReplicaManager` gives the placement of each partition when the
snapshot runs: diskless, classic leader, classic follower, or not
hosted. It schedules the snapshot with the window and removes the
gauges at shutdown. `HostedPartitions` counts the classic partitions
that this broker leads and all diskless partitions in the cluster,
through the new `InklessMetadataView.getDisklessPartitionCount`.

The unit is one partition on one broker. A diskless partition, and a
classic partition that consumers fetch from followers, can have traffic
on several brokers. Each broker counts its slice. `FEATURES.md`
describes this.

Assisted-by: Claude Opus 5.5

* test(inkless:metrics): add JMH benchmark for the partition rate hooks [KC-766]

`InklessPartitionRateTrackerBenchmark` measures the produce and fetch
hooks per request, in conditions close to a broker. Each request has its
own copy of each topic name, as request parsing gives. Its partitions
come at random from a working set spread over the stored entries. A
thread takes a snapshot every second, and other threads stream through
memory to evict the caches. The fork uses the G1 flags of
`kafka-run-class.sh`.

The baseline methods build the same keys without the hooks. The cost of
a hook per partition is the hook minus its baseline, divided by the
partitions per request.

Run it with 8 threads, as the default `num.io.threads`:

    ./jmh-benchmarks/jmh.sh -t 8 InklessPartitionRateTrackerBenchmark

Assisted-by: Claude Opus 5.5

* fixup! feat(inkless:metrics): add PartitionRateTracker [KC-766]

Assisted-by: Claude Opus 5.5

* fixup! feat(inkless:metrics): record client bytes for partition rates [KC-766]

Assisted-by: Claude Opus 5.5

* amend! feat(inkless:metrics): add PartitionRateTracker [KC-766]

feat(inkless:metrics): add PartitionRateTracker [KC-766]

No broker metric shows how unequal client traffic is across
partitions. The broker meters `BytesInPerSec` and `BytesOutPerSec` are
per broker and per topic, so skew inside a topic is invisible.

`PartitionRateTracker` turns per-partition byte counts into the
distribution of partition rates on one broker. Callers record bytes. A
periodic snapshot computes the rate of each partition with traffic and
publishes, per direction and topic type: the `P50`, `P95`, `P99`,
`P999`, and `Max` rates, cumulative `le` buckets, the sum of the rates,
and the active and hosted partition counts.

The hot path does one map lookup and one `LongAdder` add. The snapshot
resolves the topic type and the role through a placement function, once
per partition per window. It expires an entry after a window without
bytes in either direction, and drops the partitions that the broker no
longer hosts, which leaves the tracker without a removal hook. The map
then holds only the partitions with traffic in the last window, wherever
the metadata transformer routes clients. Each map entry stores the
counters of both directions of a partition.

HdrHistogram keeps the percentiles and the buckets within 1% of the
recorded rates. The snapshot resets and reuses the histograms. Apart
from the placement function, a snapshot allocates a constant amount.
The gauges read an immutable snapshot instead of the partitions.

With 100,000 partitions that have traffic in both directions, the
tracker uses about 225 B per partition and a snapshot takes about 4 ms.

The new `inkless.partition.rate.window.ms` config sets the window: 1 min
by default, and at least 1 s. The next commit adds the hooks that record
bytes.

Assisted-by: Claude Opus 5.5

* amend! test(inkless:metrics): add JMH benchmark for the partition rate hooks [KC-766]

test(inkless:metrics): add JMH benchmark for the partition rate hooks [KC-766]

`InklessPartitionRateTrackerBenchmark` measures the produce and fetch
hooks per request, in conditions close to a broker. Each request has its
own copy of each topic name, as request parsing gives. Its partitions
come at random from a working set spread over the stored entries. A
thread takes a snapshot every second. Before each snapshot, it records
bytes for every stored entry, so that no entry expires. Other threads
stream through memory to evict the caches. The fork uses the G1 flags of
`kafka-run-class.sh`.

The baseline methods build the same keys without the hooks. The cost of
a hook per partition is the hook minus its baseline, divided by the
partitions per request.

Run it with 8 threads, as the default `num.io.threads`:

    ./jmh-benchmarks/jmh.sh -t 8 InklessPartitionRateTrackerBenchmark

Assisted-by: Claude Opus 5.5

* fixup! feat(inkless:metrics): add PartitionRateTracker [KC-766]

Assisted-by: Claude Opus 5.5

* amend! feat(inkless:metrics): record client bytes for partition rates [KC-766]

feat(inkless:metrics): record client bytes for partition rates [KC-766]

Record client bytes per partition and publish the partition rate
metrics under `kafka.server:type=ReplicaManager`.

The produce response callback of `KafkaApis` records bytes in for the
partitions with `Errors.NONE`, so a failed append doesn't count. The
handler keeps only the record sizes for the callback, and the records
can go while the request waits in purgatory.

`recordBytesOutMetric` records bytes out for consumer fetches. It runs
on the response that the broker sends, after the fetch quota decision.
A throttled fetch doesn't send records, so it doesn't count.

Coordinator writes and replication don't use these hooks, because the
hooks run only for client requests. They skip Kafka internal topics.

`ReplicaManager` gives the placement of each partition when the
snapshot runs: diskless, classic leader, classic follower, or not
hosted. It schedules the snapshot with the window and removes the
gauges at shutdown.

`HostedPartitions` counts the classic partitions that this broker leads
and the diskless partitions whose replicas include this broker, through
the new `InklessMetadataView.getDisklessReplicaCount`. For diskless, it
is a correct denominator for idle partitions only when the metadata
transformer keeps clients on the replicas: with remote storage on, or in
phase 1 of the classic-to-diskless switch.

The unit is one partition on one broker. A diskless partition, and a
classic partition that consumers fetch from followers, can have traffic
on several brokers. Each broker counts its slice. `FEATURES.md`
describes this.

Assisted-by: Claude Opus 5.5

* amend! test(inkless:metrics): add JMH benchmark for the partition rate hooks [KC-766]

test(inkless:metrics): add JMH benchmark for the partition rate hooks [KC-766]

`InklessPartitionRateTrackerBenchmark` measures the produce and fetch
hooks per request, in conditions close to a broker. Each request has its
own copy of each topic name, as request parsing gives. Its partitions
come at random from a working set spread over the stored entries. A
thread takes a snapshot every second. Before each snapshot, it records
bytes for every stored entry, so that no entry expires. Other threads
stream through memory to evict the caches. The fork uses the G1 flags of
`kafka-run-class.sh`.

The baseline methods build the same keys without the hooks. The cost of
a hook per partition is the hook minus its baseline, divided by the
partitions per request.

The class runs 8 threads, as the default `num.io.threads`. To measure
one thread, pass `-t 1`:

    ./jmh-benchmarks/jmh.sh -t 1 InklessPartitionRateTrackerBenchmark

`INKLESS_OWNERSHIP` lists the Inkless JMH benchmarks with one glob.

Assisted-by: Claude Opus 5.5

* fixup! test(inkless:metrics): add JMH benchmark for the partition rate hooks [KC-766]

Assisted-by: Claude Opus 5.5
(cherry picked from commit e843b1b)
The metadata view test and the JMH benchmark of the cherry-picked change
create a `MetadataDelta` with a constructor that 4.3 doesn't have. They now use
`MetadataDelta.Builder`.

The cherry-picked change to `RemoteLogManager` makes
`cleanupExpiredRemoteLogSegments` 174 lines long, over the 170 line limit of
checkstyle. The method length check now skips `RemoteLogManager`, as it
already does for `RemoteLogManagerConfig` and `UnifiedLog`.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants