Repository navigation
fix(inkless:delete): skip file cleaner ticks during backoff instead of sleeping [KC-349] - #795
viktorsomogyi wants to merge 2 commits into
Conversation
|
@jeqo @giuseppelillo addressed the comments from the earlier PR (#792 ) |
3d3cf6d to
3506b1e
Compare
3506b1e to
a413757
Compare
01d2f1c to
cb19036
Compare
|
@giuseppelillo @jeqo actualized this PR as it got dropped. Can you please review it once more? |
jeqo
left a comment
There was a problem hiding this comment.
Just a minor comment and finding on the description:
"When run() sleeps, the missed ticks queue up" holds only when the cycle time plus the sleep is longer than file.cleaner.interval.ms. With the 2min default, the sleep alone never is (idle 8-12s, error at most 60s). So the burst would need a short interval, or a slow cycle such as a failing storage call that runs for over a minute before the 60s sleep. Can you word the Summary that way, and name the effect that holds at every interval: the cleaner no longer holds a scheduler thread while it waits?
| assertEquals(0, beforeDelete.deletedLogCount(), "Live logs must not be stamped deleted"); | ||
|
|
||
| try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(TopicPurger.class)) { | ||
| appender.setClassLogger(TopicPurger.class, Level.INFO); |
There was a problem hiding this comment.
Is this override needed? io.aiven.inkless is already DEBUG in core/src/test/resources/log4j2.yaml
cb19036 to
406a59e
Compare
…f sleeping [KC-349] `KafkaScheduler` runs the file cleaner with `scheduleAtFixedRate`. When `run()` sleeps, for the idle backoff or the error backoff, the missed ticks queue up and then fire back to back. Those cycles delete more than `file.cleaner.max.files.per.cycle` files per interval, which breaks the pacing the operator configured. An empty cycle now returns immediately. After an error, the cleaner records when the backoff ends and skips ticks until then, the same way `TopicPurger` handles errors. `InklessDisklessTopicDeleteTest` counts saturated `TopicPurger` cycles from the log. The purger logs after its transaction commits, so the test polls the captured log instead of counting once. Each broker checks `moreRemain` before it commits, so only one cycle can see the drain finish, and the test expects at least `batchesAtDelete - 1` saturated cycles. `TopicPurger` exposes the log text that the test matches. The test sets the `TopicPurger` log level to INFO, so the core checkstyle rules now allow the log4j `Level` class in `kafka.server`, and `INKLESS_OWNERSHIP` lists that rules file as interleaved. Testing `FileCleanerMockedTest` checks that an empty cycle doesn't advance time, and that after an error the cleaner skips ticks until the backoff ends and then runs again. `InklessDisklessTopicDeleteTest` checks that the purger hits the per-cycle cap on every deleting cycle but the last. Assisted-by: Claude Opus 5.5
`core/src/test/resources/log4j2.yaml` already sets `io.aiven.inkless` to DEBUG, so `InklessDisklessTopicDeleteTest` captures the `TopicPurger` INFO lines without raising the level itself. Without that call, the test no longer needs the log4j `Level` class, and `checkstyle/import-control-core.xml` goes back to its state on `main`. The `nextEligibleMs` comments in `FileCleaner` and `TopicPurger` now say that a sleep in `run()` holds a thread of the shared `KafkaScheduler`, and that missed ticks fire back to back only when a run outlasts the interval. Testing `InklessDisklessTopicDeleteTest` passes without the override and logs 11 saturated `TopicPurger` cycles. `FileCleanerMockedTest` and `TopicPurgerMockedTest` pass. Assisted-by: Claude Opus 5.5
406a59e to
0e85cdd
Compare
Summary
FileCleanerslept insiderun()for the idle backoff (8-12 s) and the error backoff (up to 60 s). At every interval, that sleep holds a thread of the sharedKafkaScheduler, which also runs other broker background tasks. When the cycle and the sleep together outlastfile.cleaner.interval.ms,scheduleAtFixedRatealso fires the missed ticks back to back pastfile.cleaner.max.files.per.cycle. With the 2-minute default, that takes a short interval, or a slow cycle such as a failing storage call that runs for over a minute before the error sleep. The cleaner now skips ticks during a backoff instead of sleeping, the same wayTopicPurgerdoes.The test and manifest changes:
InklessDisklessTopicDeleteTestpolls the capturedTopicPurgerlog, because the purger logs after its transaction commits. It expects at leastbatchesAtDelete - 1saturated cycles: each broker checksmoreRemainbefore it commits, so only one cycle can see the drain finish.TopicPurger.CAP_REACHED_LOG_FRAGMENTholds the log text that the test matches.INKLESS_OWNERSHIPlistscheckstyle/import-control-core.xmlas interleaved, because it carries Inkless import allowances.Evidence
Before:
FileCleanerMockedTestagainst theFileCleanerfrommain(10a852b2b3) fails 3 of 14 tests:After: on this branch:
:core:checkstyleMain,:core:checkstyleTest, andmake valepass.Merge danger
Door: two-way
No configs, metrics, or control-plane schema change. A revert restores the sleeps.
Blast radius: contained
KafkaSchedulerthread while it waits out a backoff, so other background tasks on that pool don't wait behind it.file.cleaner.interval.msof 2 minutes, the next tick after an error runs on schedule, because the error backoff of at most 60 s is shorter than the interval.