Skip to content

fix(inkless:delete): skip file cleaner ticks during backoff instead of sleeping [KC-349] - #795

Open
viktorsomogyi wants to merge 2 commits into
mainfrom
svv/filecleaner-fix
Open

viktorsomogyi wants to merge 2 commits into
mainfrom
svv/filecleaner-fix

Conversation

@viktorsomogyi

@viktorsomogyi viktorsomogyi commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

FileCleaner slept inside run() for the idle backoff (8-12 s) and the error backoff (up to 60 s). At every interval, that sleep holds a thread of the shared KafkaScheduler, which also runs other broker background tasks. When the cycle and the sleep together outlast file.cleaner.interval.ms, scheduleAtFixedRate also fires the missed ticks back to back past file.cleaner.max.files.per.cycle. With the 2-minute default, that takes a short interval, or a slow cycle such as a failing storage call that runs for over a minute before the error sleep. The cleaner now skips ticks during a backoff instead of sleeping, the same way TopicPurger does.

 FileCleaner.run()
+  if now < nextEligibleMs
+    return
   fetch up to max.files.per.cycle + 1 files to delete
   if there are none
-    sleep 8-12 s
+    return
   delete the files, then dereference them in the control plane
   on error
-    sleep the backoff
+    nextEligibleMs = now + backoff

The test and manifest changes:

  • InklessDisklessTopicDeleteTest polls the captured TopicPurger log, because the purger logs after its transaction commits. It expects at least batchesAtDelete - 1 saturated cycles: each broker checks moreRemain before it commits, so only one cycle can see the drain finish.
  • TopicPurger.CAP_REACHED_LOG_FRAGMENT holds the log text that the test matches.
  • INKLESS_OWNERSHIP lists checkstyle/import-control-core.xml as interleaved, because it carries Inkless import allowances.

Evidence

  • Before: FileCleanerMockedTest against the FileCleaner from main (10a852b2b3) fails 3 of 14 tests:

    empty()                                 an empty cycle advanced the clock 10.2 s
    tracksLastSuccessfulCycle()             an empty cycle advanced the clock 11.3 s
    errorBackoffSkipsTicksUntilEligible()   the tick during the backoff ran a full cycle
    

    After: on this branch:

    FileCleanerMockedTest            14 tests, 0 failures
    TopicPurgerMockedTest             7 tests, 0 failures
    InklessDisklessTopicDeleteTest    1 test,  0 failures (two brokers, 11 saturated cycles logged)
    

    :core:checkstyleMain, :core:checkstyleTest, and make vale pass.

Merge danger

Door: two-way

No configs, metrics, or control-plane schema change. A revert restores the sleeps.

Blast radius: contained

  • The cleaner no longer holds a KafkaScheduler thread while it waits out a backoff, so other background tasks on that pool don't wait behind it.
  • With the default file.cleaner.interval.ms of 2 minutes, the next tick after an error runs on schedule, because the error backoff of at most 60 s is shorter than the interval.
  • With an interval under about 12 s, an idle broker now queries the control plane for files to delete once per interval. Before, the 8-12 s idle sleep slowed that rate down.
  • The "No files to delete" log line moves from INFO to DEBUG.

@viktorsomogyi

Copy link
Copy Markdown
Contributor Author

@jeqo @giuseppelillo addressed the comments from the earlier PR (#792 )

giuseppelillo
giuseppelillo previously approved these changes Sep 24, 2026
@viktorsomogyi viktorsomogyi changed the title fix(inkless:delete): do not sleep inside the file cleaner tick [KC-349] fix(inkless:delete): skip file cleaner ticks during backoff instead of sleeping [KC-349] Nov 16, 2026
@viktorsomogyi

Copy link
Copy Markdown
Contributor Author

@giuseppelillo @jeqo actualized this PR as it got dropped. Can you please review it once more?

giuseppelillo
giuseppelillo previously approved these changes Nov 16, 2026

@jeqo jeqo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor comment and finding on the description:
"When run() sleeps, the missed ticks queue up" holds only when the cycle time plus the sleep is longer than file.cleaner.interval.ms. With the 2min default, the sleep alone never is (idle 8-12s, error at most 60s). So the burst would need a short interval, or a slow cycle such as a failing storage call that runs for over a minute before the 60s sleep. Can you word the Summary that way, and name the effect that holds at every interval: the cleaner no longer holds a scheduler thread while it waits?

assertEquals(0, beforeDelete.deletedLogCount(), "Live logs must not be stamped deleted");

try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(TopicPurger.class)) {
appender.setClassLogger(TopicPurger.class, Level.INFO);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this override needed? io.aiven.inkless is already DEBUG in core/src/test/resources/log4j2.yaml

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, addressed in 0e85cdd.

…f sleeping [KC-349]

`KafkaScheduler` runs the file cleaner with `scheduleAtFixedRate`. When
`run()` sleeps, for the idle backoff or the error backoff, the missed
ticks queue up and then fire back to back. Those cycles delete more than
`file.cleaner.max.files.per.cycle` files per interval, which breaks the
pacing the operator configured.

An empty cycle now returns immediately. After an error, the cleaner
records when the backoff ends and skips ticks until then, the same way
`TopicPurger` handles errors.

`InklessDisklessTopicDeleteTest` counts saturated `TopicPurger` cycles
from the log. The purger logs after its transaction commits, so the test
polls the captured log instead of counting once. Each broker checks
`moreRemain` before it commits, so only one cycle can see the drain
finish, and the test expects at least `batchesAtDelete - 1` saturated
cycles. `TopicPurger` exposes the log text that the test matches. The
test sets the `TopicPurger` log level to INFO, so the core checkstyle
rules now allow the log4j `Level` class in `kafka.server`, and
`INKLESS_OWNERSHIP` lists that rules file as interleaved.

Testing

`FileCleanerMockedTest` checks that an empty cycle doesn't advance time,
and that after an error the cleaner skips ticks until the backoff ends
and then runs again. `InklessDisklessTopicDeleteTest` checks that the
purger hits the per-cycle cap on every deleting cycle but the last.

Assisted-by: Claude Opus 5.5
`core/src/test/resources/log4j2.yaml` already sets `io.aiven.inkless` to
DEBUG, so `InklessDisklessTopicDeleteTest` captures the `TopicPurger`
INFO lines without raising the level itself. Without that call, the test
no longer needs the log4j `Level` class, and
`checkstyle/import-control-core.xml` goes back to its state on `main`.

The `nextEligibleMs` comments in `FileCleaner` and `TopicPurger` now say
that a sleep in `run()` holds a thread of the shared `KafkaScheduler`,
and that missed ticks fire back to back only when a run outlasts the
interval.

Testing

`InklessDisklessTopicDeleteTest` passes without the override and logs 11
saturated `TopicPurger` cycles. `FileCleanerMockedTest` and
`TopicPurgerMockedTest` pass.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants