Problem

A production cluster (cdip-diskless-prod-spoke-*) threw a burst of FetchError / Read future failed exceptions. Every trace bottoms out in the same PostgreSQL condition on the read replica that serves find_batches_v2, in two escalating forms:

1. Statement cancellation

   org.postgresql.util.PSQLException: ERROR: canceling statement due to conflict with recovery
     Detail: User was holding shared buffer pin for too long.
   

2. Connection termination (escalation of the same conflict)

   org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery
     Detail: User was holding shared buffer pin for too long.
   

   which then surfaces downstream as server conn crashed?, Cannot commit transaction, and a suppressed Connection is closed / Cannot get autoCommit.

Both are hot-standby recovery conflicts: a read query holds a shared buffer pin while WAL replay needs the same page. They are transient and the server rolls the transaction back — but JobUtils had no retry (only a // TODO add retry with backoff), so a single conflict propagated all the way up as ControlPlaneException("Error finding batches") → FindBatchesException and FetchHandler returned UNKNOWN_SERVER_ERROR for every partition in the fetch.

Note: the DB-side mitigations (hot_standby_feedback = on, raised max_standby_streaming_delay) are already enabled on this cluster. This PR is the application-side resilience for the conflicts that still occur under replay pressure.

Change

Implement the missing retry in JobUtils: retry the whole job with jittered exponential backoff (3 attempts, ~50 ms → ~100 ms with 0.2 jitter, capped at 1 s; via the injected Time so it is fully virtual in tests).

Exceptions handled — retry is keyed on rollback-guaranteed PostgreSQL SQLStates:

Key design points:

• Retry wraps the whole transaction, not the inner lambda. Because the control-plane DSLContexts are Hikari-pool-backed, a retry pulls a fresh connection — this is what recovers from the connection-terminated form where the previous connection is dead.
• Full cause chain + suppressed throwables are scanned. The retriable PSQLException is buried under ControlPlaneException → DataAccessException; in the connection-killed form it appears as a suppressed throwable (the failed rollback), exactly as in the traces.
• Generic connection-loss states (08*, 57P01) are intentionally NOT retried — a drop during commit leaves a write in-doubt, so blindly replaying could double-apply. The recovery-conflict family that caused this incident is fully covered by the rollback-guaranteed states, so the retry is safe for both read and write jobs and can live in the shared helper.
• Error contract unchanged: on exhaustion or a non-retriable failure, the original exception is rethrown and mapped to ControlPlaneException / wrapped RuntimeException as before.

Hot-path safety

This retry sits on the fetch hot path shared by all 16 control-plane jobs, so it is designed to stay safe under a conflict storm rather than amplify one:

• Metrics stay accurate. measureDurationMs fires the duration callback in a finally, so timing each attempt directly would record N samples per run and pollute the latency/rate percentiles with failed-attempt durations. Only the decisive attempt (success, or the final failure) is reported to the callback → exactly one sample per run().
• Jittered backoff (ExponentialBackoff, reused from clients) prevents many jobs failing against the same struggling standby from retrying in lockstep and re-colliding on the same recovery-conflict window.
• Interrupt-aware. Utils.sleep restores the interrupt flag but returns normally; the loop checks Thread.isInterrupted() after backoff and aborts (keeping the flag set) rather than spinning through remaining attempts and delaying broker shutdown.
• One terminal log on exhaustion, so the final user-visible failure is greppable instead of appearing only as intermediate "retrying" warnings.

Tests

JobUtilsTest (pure unit — no DB container, uses MockTime): happy path, recovery-conflict-then-succeed (reproduces the incident with the exact wrapping from the traces), deadlock retry, max-attempts exhaustion preserving ControlPlaneException, non-retriable SQLState (23505) and plain runtime exceptions not retried, suppressed-throwable scanning, self-referential cause chain (terminates, no StackOverflow), plus: jittered backoff timing asserted via MockTime deltas, a single duration-callback per run (guards the metrics contract), no sleep on non-retriable failures, and interrupt-driven abort.

Follow-up (not in this PR)

• When retries are exhausted the client still gets UNKNOWN_SERVER_ERROR. Surfacing exhausted transient failures as a retriable Kafka error would be a separate change in the FetchHandler error-mapping path.
• The connection-terminated form relies on pgjdbc surfacing SQLState 40001; there is a residual driver-level race where a socket EOF beating the FATAL message could downgrade to 08006. Confirming this end-to-end warrants a hot-standby integration test.

🤖 Generated with Claude Code