Skip to content

fix(inkless:retention): size retention enforcement to keep up with produce - #744

Merged
jeqo merged 1 commit into
mainfrom
jeqo/fix-retention-enforcer-defaults
Aug 19, 2026
Merged

jeqo merged 1 commit into
mainfrom
jeqo/fix-retention-enforcer-defaults

Conversation

@jeqo

@jeqo jeqo commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

The defaults couldn't keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per writing broker, so a single writer already exceeds 4 batches/s, and in diskless any broker can write to any partition. The batches table grew without bound on busy partitions while data outlived its retention.

Raise capacity by ~10x:

  • retention.enforcement.interval.ms: 5 min -> 1 min
  • retention.enforcement.max.batches.per.request: 1000 -> 2000

That's 33 batches/s per partition, covering a partition written by ~8 brokers at the default commit interval.

The interval was the better lever. Capacity is cap/interval, so a 10x cap would also work, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22), and the delete holds the log's row: fewer, heavier passes hold locks longer, which V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The tradeoff is transaction rate: every due partition costs one transaction regardless of expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions).

consolidation.cleanup.interval.ms no longer derives from the retention interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost profile differs (whole-file scans, not per-partition scans), so it keeps 5 min explicitly.

Not addressed: these defaults assume ~8 brokers writing to a partition; nothing scales with cluster size. RetentionEnforcementScheduler already throttles per-broker cadence by brokerCount, but that bounds enforcement transactions, not delete throughput, which stays capped at cap/interval regardless of cluster size while producer fan-in per partition can grow with it. Scaling the cap by brokerCount would close that gap but reintroduce the heavier locked passes this change avoided; relaxing the cadence throttle instead grows transaction rate the same way. Both just move the cost this change traded away, so it is a scheduling design question, not a follow-up tweak: today every broker checks every partition independently with no coordination. Once managed-replica leaders are the norm, as tiered-storage consolidation becomes standard, a partition gets one natural scheduling owner instead of N uncoordinated brokers, which changes the shape of this problem. Left for that.

@jeqo
jeqo force-pushed the jeqo/fix-retention-enforcer-defaults branch 2 times, most recently from c0a9979 to 924893c Compare August 12, 2026 16:04
@jeqo
jeqo force-pushed the jeqo/kill-background-on-restart branch 7 times, most recently from b5a369c to c9e1a69 Compare August 19, 2026 07:07
@jeqo
jeqo force-pushed the jeqo/fix-retention-enforcer-defaults branch from 924893c to 71633c5 Compare August 19, 2026 07:10
@jeqo
jeqo changed the base branch from jeqo/kill-background-on-restart to main August 19, 2026 07:14
@jeqo
jeqo requested a balanced review from Copilot August 19, 2026 07:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates retention defaults to increase deletion throughput for continuously written diskless partitions.

Changes:

  • Reduces enforcement interval from five minutes to one minute.
  • Raises deletion cap from 1,000 to 2,000 batches.
  • Decouples WAL cleanup cadence and updates configuration documentation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
InklessConfig.java Updates retention and cleanup defaults.
InklessConfigTest.java Updates the batch-cap default assertion.
configs.rst Documents the new defaults and sizing guidance.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/inkless/configs.rst Resolved
…oduce

The defaults couldn't keep up with a continuously written partition. Sustained
deletion capacity per partition is max.batches.per.request per
retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A
partition gets roughly one batch per produce.commit.interval.ms (250ms) per
writing broker, so a single writer already exceeds 4 batches/s, and in
diskless any broker can write to any partition. The batches table grew
without bound on busy partitions while data outlived its retention.

Raise capacity by ~10x:
- retention.enforcement.interval.ms: 5 min -> 1 min
- retention.enforcement.max.batches.per.request: 1000 -> 2000

That's 33 batches/s per partition, covering a partition written by ~8 brokers
at the default commit interval.

The interval was the better lever. Capacity is cap/interval, so a 10x cap
would also work, but the per-pass boundary scan is O(cap)
(enforce_retention_v2 after V22), and the delete holds the log's row: fewer,
heavier passes hold locks longer, which V21/V22 set out to reduce. More
frequent, bounded passes keep the lock hold short. The tradeoff is
transaction rate: every due partition costs one transaction regardless of
expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide
at 100k partitions).

consolidation.cleanup.interval.ms no longer derives from the retention
interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost
profile differs (whole-file scans, not per-partition scans), so it keeps 5
min explicitly.

Not addressed: these defaults assume ~8 brokers writing to a partition;
nothing scales with cluster size. RetentionEnforcementScheduler already
throttles per-broker cadence by brokerCount, but that bounds enforcement
transactions, not delete throughput, which stays capped at cap/interval
regardless of cluster size while producer fan-in per partition can grow with
it. Scaling the cap by brokerCount would close that gap but reintroduce the
heavier locked passes this change avoided; relaxing the cadence throttle
instead grows transaction rate the same way. Both just move the cost this
change traded away, so it is a scheduling design question, not a follow-up
tweak: today every broker checks every partition independently with no
coordination. Once managed-replica leaders are the norm, as tiered-storage
consolidation becomes standard, a partition gets one natural scheduling
owner instead of N uncoordinated brokers, which changes the shape of this
problem. Left for that.
@jeqo
jeqo force-pushed the jeqo/fix-retention-enforcer-defaults branch from 71633c5 to 0aa4595 Compare August 19, 2026 07:48
@jeqo
jeqo requested a balanced review from Copilot August 19, 2026 07:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@jeqo
jeqo marked this pull request as ready for review August 19, 2026 07:53
@jeqo
jeqo requested a review from tvainika August 19, 2026 07:53
@jeqo
jeqo merged commit 86c4c66 into main Aug 19, 2026
11 checks passed
@jeqo
jeqo deleted the jeqo/fix-retention-enforcer-defaults branch August 19, 2026 08:21
jeqo added a commit that referenced this pull request Aug 19, 2026
…oduce (#744)

The defaults couldn't keep up with a continuously written partition. Sustained
deletion capacity per partition is max.batches.per.request per
retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A
partition gets roughly one batch per produce.commit.interval.ms (250ms) per
writing broker, so a single writer already exceeds 4 batches/s, and in
diskless any broker can write to any partition. The batches table grew
without bound on busy partitions while data outlived its retention.

Raise capacity by ~10x:
- retention.enforcement.interval.ms: 5 min -> 1 min
- retention.enforcement.max.batches.per.request: 1000 -> 2000

That's 33 batches/s per partition, covering a partition written by ~8 brokers
at the default commit interval.

The interval was the better lever. Capacity is cap/interval, so a 10x cap
would also work, but the per-pass boundary scan is O(cap)
(enforce_retention_v2 after V22), and the delete holds the log's row: fewer,
heavier passes hold locks longer, which V21/V22 set out to reduce. More
frequent, bounded passes keep the lock hold short. The tradeoff is
transaction rate: every due partition costs one transaction regardless of
expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide
at 100k partitions).

consolidation.cleanup.interval.ms no longer derives from the retention
interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost
profile differs (whole-file scans, not per-partition scans), so it keeps 5
min explicitly.

Not addressed: these defaults assume ~8 brokers writing to a partition;
nothing scales with cluster size. RetentionEnforcementScheduler already
throttles per-broker cadence by brokerCount, but that bounds enforcement
transactions, not delete throughput, which stays capped at cap/interval
regardless of cluster size while producer fan-in per partition can grow with
it. Scaling the cap by brokerCount would close that gap but reintroduce the
heavier locked passes this change avoided; relaxing the cadence throttle
instead grows transaction rate the same way. Both just move the cost this
change traded away, so it is a scheduling design question, not a follow-up
tweak: today every broker checks every partition independently with no
coordination. Once managed-replica leaders are the norm, as tiered-storage
consolidation becomes standard, a partition gets one natural scheduling
owner instead of N uncoordinated brokers, which changes the shape of this
problem. Left for that.
jeqo added a commit that referenced this pull request Aug 19, 2026
…oduce (#744)

The defaults couldn't keep up with a continuously written partition. Sustained
deletion capacity per partition is max.batches.per.request per
retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A
partition gets roughly one batch per produce.commit.interval.ms (250ms) per
writing broker, so a single writer already exceeds 4 batches/s, and in
diskless any broker can write to any partition. The batches table grew
without bound on busy partitions while data outlived its retention.

Raise capacity by ~10x:
- retention.enforcement.interval.ms: 5 min -> 1 min
- retention.enforcement.max.batches.per.request: 1000 -> 2000

That's 33 batches/s per partition, covering a partition written by ~8 brokers
at the default commit interval.

The interval was the better lever. Capacity is cap/interval, so a 10x cap
would also work, but the per-pass boundary scan is O(cap)
(enforce_retention_v2 after V22), and the delete holds the log's row: fewer,
heavier passes hold locks longer, which V21/V22 set out to reduce. More
frequent, bounded passes keep the lock hold short. The tradeoff is
transaction rate: every due partition costs one transaction regardless of
expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide
at 100k partitions).

consolidation.cleanup.interval.ms no longer derives from the retention
interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost
profile differs (whole-file scans, not per-partition scans), so it keeps 5
min explicitly.

Not addressed: these defaults assume ~8 brokers writing to a partition;
nothing scales with cluster size. RetentionEnforcementScheduler already
throttles per-broker cadence by brokerCount, but that bounds enforcement
transactions, not delete throughput, which stays capped at cap/interval
regardless of cluster size while producer fan-in per partition can grow with
it. Scaling the cap by brokerCount would close that gap but reintroduce the
heavier locked passes this change avoided; relaxing the cadence throttle
instead grows transaction rate the same way. Both just move the cost this
change traded away, so it is a scheduling design question, not a follow-up
tweak: today every broker checks every partition independently with no
coordination. Once managed-replica leaders are the norm, as tiered-storage
consolidation becomes standard, a partition gets one natural scheduling
owner instead of N uncoordinated brokers, which changes the shape of this
problem. Left for that.
jeqo added a commit that referenced this pull request Aug 19, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the
inkless-changelog skill and curated:

- Metric deltas cross-checked against source commits (#716, #735, #743,
  #746, #715); all trace to real, shipped metrics, no documentation
  catch-up noise this increment.
- Postgres migration headers (V23-V26) checked for lock impact; V25 takes
  ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26
  offers a pre-upgrade CONCURRENTLY workaround.
- Added two config default changes the generator's diff-only detection
  missed (existing keys, not new ones): file.cleaner.interval.ms
  (5min -> 2min, #745) and retention.enforcement.interval.ms /
  max.batches.per.request (#744).
jeqo added a commit that referenced this pull request Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the
inkless-changelog skill and curated:

- Metric deltas cross-checked against source commits (#716, #735, #743,
  #746, #715); all trace to real, shipped metrics, no documentation
  catch-up noise this increment.
- Postgres migration headers (V23-V26) checked for lock impact; V25 takes
  ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26
  offers a pre-upgrade CONCURRENTLY workaround.
- Added two config default changes the generator's diff-only detection
  missed (existing keys, not new ones): file.cleaner.interval.ms
  (5min -> 2min, #745) and retention.enforcement.interval.ms /
  max.batches.per.request (#744).
jeqo added a commit that referenced this pull request Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the
inkless-changelog skill and curated:

- Metric deltas cross-checked against source commits (#716, #735, #743,
  #746, #715); all trace to real, shipped metrics, no documentation
  catch-up noise this increment.
- Postgres migration headers (V23-V26) checked for lock impact; V25 takes
  ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26
  offers a pre-upgrade CONCURRENTLY workaround.
- Added two config default changes the generator's diff-only detection
  missed (existing keys, not new ones): file.cleaner.interval.ms
  (5min -> 2min, #745) and retention.enforcement.interval.ms /
  max.batches.per.request (#744).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants