Skip to content

feat: retry transient provider failures with backoff#1

Open
tmchow wants to merge 1 commit into
ajsai47:mainfrom
tmchow:feat/retry-transient-provider-failures
Open

feat: retry transient provider failures with backoff#1
tmchow wants to merge 1 commit into
ajsai47:mainfrom
tmchow:feat/retry-transient-provider-failures

Conversation

@tmchow

@tmchow tmchow commented May 25, 2026

Copy link
Copy Markdown

Summary

A single 429 or 5xx from the provider currently aborts the whole Claude Code turn, because client.py raises on the first non-200 response. This adds a bounded retry with exponential backoff in ProviderClient so a throttled free-tier provider recovers instead of killing the session.

Why this matters

The README points people at free tiers like Groq and NVIDIA NIM, and those tiers rate-limit hard. CONTRIBUTING.md lists "rate limit headers" as a wanted contribution, which is the gap this closes. LiteLLM and claude-code-router both retry transient failures already, so this brings Backdoor to parity, and it does so without adding a dependency.

Changes

Both complete and stream now retry on 429 and 5xx responses and on transport errors, up to PROVIDER_MAX_RETRIES (default 3) with exponential backoff, honoring a Retry-After header when the provider sends one. Errors that will not fix themselves on a retry, like a 404 for a bad model name, still raise immediately rather than sleeping through three attempts. Streaming is the subtle case: it only retries before the first chunk is yielded, so a mid-stream failure surfaces to the caller and the SSE stream is never restarted with a duplicate message_start. Setting PROVIDER_MAX_RETRIES=0 restores the old fail-fast behavior.

Testing

Added tests/test_retry.py with five cases on the existing pytest-asyncio setup: a 429 followed by a 200 retries and succeeds, a persistent 503 gives up after the retry budget and surfaces the error, a 404 raises with no retry, a zero retry budget makes a single attempt, and Retry-After parsing covers the delta-seconds and fallback paths. All pass, with no new dependencies beyond asyncio and stdlib email.utils.

429s and 5xx from the provider currently abort the Claude Code turn on the
first failure. Free tiers (Groq, NIM) throttle often, so a single rate-limit
response ends the session.

ProviderClient.complete and .stream now retry 429/500/502/503/504 and
transport errors up to PROVIDER_MAX_RETRIES (default 3) with exponential
backoff, honoring the Retry-After header when present. Non-retryable errors
(400/401/404/422) still raise immediately. Streaming only retries before the
first chunk is yielded so a mid-stream failure never re-emits message_start.
Set PROVIDER_MAX_RETRIES=0 to disable. No new dependencies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant