When object storage throttled deletes, the WAL file cleaner would abort the whole cycle and re-attempt keys it had already deleted, looping without draining and amplifying the throttle. This makes deletion drain monotonically: only the keys the backend confirms deleted are dereferenced, and the rest are retried on the next cycle.

Problem

S3Storage.delete(Set) threw on the first per-key or whole-request error. Because FileCleaner deletes from storage before updating the control plane, that throw meant no files were dereferenced for the whole batch — including keys that S3 had already deleted in earlier slices. The next cleaner cycle re-fetched the same set and re-issued DeleteObjects for objects that were already gone, so under sustained throttling the set never drained and the redundant requests amplified the throttle.

Change

The ObjectDeleter.delete(Set) contract now returns the subset of keys confirmed deleted (idempotent: already-absent keys count as deleted) instead of void, so callers can advance on partial progress.

• S3 (S3Storage): single pass per 1000-key batch, collecting DeleteObjectsResponse.deleted() and logging the rest (throttle vs hard error, for diagnostics only). It no longer throws for partial or whole-request failures (including a 503 the SDK's adaptive retry exhausted); undeleted keys stay marked for deletion and are retried next cycle.
• Azure (AzureBlobStorage): deletes one blob at a time, so it now accumulates and returns the confirmed-deleted subset instead of throwing on the first failure and abandoning the rest — same monotonic-drain benefit as S3.
• GCS (GcsStorage): stays all-or-nothing (documented inline). Storage.delete(Iterable) is an atomic batch that raises a genuine failure as a thrown exception rather than a per-blob flag, so a confirmed-deleted subset cannot be extracted; a thrown batch drains nothing and the whole set is retried next cycle.
• In-memory / config-test backends: return the input set on success (contract propagation only).
• FileCleaner: dereferences only the confirmed-deleted subset in the control plane and records that count.

No control-plane change is needed — deleteFiles already accepts any subset of keys.

Backoff

Backoff is layered rather than duplicated inside the delete call: the S3 client's ADAPTIVE_V2 retry strategy handles request-rate throttling within a cycle, and FileCleaner now applies its existing exponential backoff whenever a cycle drains nothing (previously only on a thrown exception, which throttling no longer produces). So sustained throttling no longer retries at a fixed cadence. There is deliberately no bespoke in-call retry in the backend, which would fight the adaptive limiter and block the cleaner thread.

Testing

• FileCleanerMockedTest gains monotonic-drain, no-progress-backoff, and storage-failure cases; the existing cases stub the new return value.
• New S3StorageDeleteTest (WireMock) covers single-pass partial return without retry, and a whole-request 503 that must not propagate.
• Existing Azure and GCS integration tests (BaseStorageTest.testDeletes, run against Azurite / fake-gcs) exercise the updated backends.