fix(proxy): retry passthrough on transient upstream connection close - #1513
Conversation
PR governanceThis PR follows the template and is marked ready for human review. |
JerrettDavis
left a comment
There was a problem hiding this comment.
The retry path looks narrowly scoped and matches the failure mode from #1112: only RemoteProtocolError is retried, the second failure is surfaced as a clean 502 instead of bubbling an exception, and the handler tests cover both transient recovery and persistent upstream close. No code changes requested.
There was a problem hiding this comment.
Reviewed the latest CodeQL follow-up. The client-facing 502 message is now generic while the detailed upstream protocol exception remains in the server-side warning log, so this preserves operator diagnosability without leaking upstream exception text to callers. No additional code changes requested.
This branch still needs to be updated from main: GitHub marks it conflicting, and the remaining pip-audit failure is the stale transformers CVE already fixed on main.
GET /v1/models and other buffered passthrough routes returned an opaque
HTTP 502 when an OpenAI-compatible upstream closed a pooled keep-alive
connection mid-response, surfacing httpx.RemoteProtocolError ("peer closed
connection without sending complete message body (incomplete chunked read)").
A direct curl works because it opens a fresh connection per call; Headroom
reuses pooled connections, so a stale connection fails on reuse even though
the upstream is healthy.
Add headroom.proxy.helpers.request_with_transient_retry, which retries a
buffered request once on a fresh connection (only on RemoteProtocolError, so
ConnectError/timeout handling is unchanged), and route handle_passthrough
through it with a clear upstream_protocol_error 502 when the error persists.
Closes headroomlabs-ai#1112
The RemoteProtocolError handler added for the headroomlabs-ai#1112 transient-retry fix interpolated the raw httpx exception into the JSON error message returned to the external client, so upstream stack-trace/exception text could leak to callers (CodeQL py/stack-trace-exposure, alert headroomlabs-ai#136 at openai.py:6341). Keep the full exception in the server-side logger.warning (unchanged) and return a generic "upstream closed the connection without sending a complete response" message with the same 502 status. Only the flagged path is touched; behaviour and the existing 502 contract are otherwise unchanged.
1cdcf8a to
1e1b6b3
Compare
|
Rebased onto the latest
No logic changes: the approved retry fix and the CodeQL sanitization (generic client-facing 502, full detail stays in the server-side log) are unchanged. Force-pushed Local checks on the affected area are green: |
JerrettDavis
left a comment
There was a problem hiding this comment.
Reviewed the rebased head. The merge kept the retry helper alongside the newer overload-status helper, and the CodeQL follow-up still keeps detailed upstream exception text in the server log while returning a generic upstream_protocol_error message to clients. I do not see a remaining code blocker.
Only governance checks are attached to this latest head while the substantive workflows wait for permission, so this approval is based on review of the final diff and the reported focused local checks rather than new GitHub CI signal.
🤖 I have created a release *beep* *boop* --- <details><summary>0.31.0</summary> ## [0.31.0](v0.30.0...v0.31.0) (2026-07-09) ### Features * **cache:** provider-agnostic cache-mode delta + cc-agnostic prefix comparison ([#1868](#1868)) ([7c2f0ea](7c2f0ea)) * **ccr:** wire retrieve-tool interception into OpenAI Responses handler ([#1898](#1898)) ([62cd307](62cd307)) * **compression:** add audit-safe mode with protected pattern matching ([#1899](#1899)) ([bb112dd](bb112dd)) * **content-router:** accept any real compression (remove min-savings floor) ([#1771](#1771)) ([6c31db9](6c31db9)) * **content-router:** lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold ([#1818](#1818)) ([60af15f](60af15f)) * **proxy:** add provider-only HTTP proxy ([#1807](#1807)) ([ebe0a3b](ebe0a3b)) * **proxy:** add turn-hook extension point for buffered model turns ([#1891](#1891)) ([ec950f7](ec950f7)) ### Bug Fixes * **build:** enable Intel macOS pip installs via ort-load-dynamic ([#1538](#1538)) ([32ce99e](32ce99e)) * **cache:** avoid fallback session collisions ([#1827](#1827)) ([0f606b6](0f606b6)) * **ccr:** make expired retrieve misses terminal ([#1781](#1781)) ([9cbdba4](9cbdba4)) * **ccr:** preserve Anthropic re-stream shape ([#1854](#1854)) ([f663894](f663894)) * **ccr:** preserve thinking blocks in buffered stream re-synthesis ([#1897](#1897)) ([ede085c](ede085c)) * **cli/proxy:** preserve explicit HEADROOM_MIN_TOKENS=0 / MAX_ITEMS=0 ([#1886](#1886)) ([3a33af1](3a33af1)) * **code-compressor:** CJK-aware relevance-query symbol matching ([#1747](#1747)) ([b38315c](b38315c)) * **codex:** discover updated Codex state stores ([#1889](#1889)) ([9d42eba](9d42eba)) * **codex:** OpenCode Zen telemetry attribution ([#1648](#1648)) ([f18c6bd](f18c6bd)) * **content-detector:** detect and compress space-separated JSON objects ([#1742](#1742)) ([5194bdc](5194bdc)) * **content-router:** token-measure lossless folds at the acceptance gate ([#1772](#1772)) ([c5493ea](c5493ea)) * **copilot:** normalize subscription routing host ([#1836](#1836)) ([afd9cbd](afd9cbd)) * **copilot:** route mixed-model requests per model ([#1785](#1785)) ([5af5e22](5af5e22)) * **dashboard:** deduplicate repeated savings metrics ([#1804](#1804)) ([88f935a](88f935a)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([#1900](#1900)) ([87f6e93](87f6e93)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([#1901](#1901)) ([361adcd](361adcd)) * **dashboard:** price proxy savings without litellm ([#1728](#1728)) ([188e382](188e382)) * detect and clear stale ANTHROPIC_BASE_URL from crashed wrap sessions ([#1768](#1768)) ([#1837](#1837)) ([84509a4](84509a4)) * **docker:** persist headroom workspace in compose ([#1839](#1839)) ([5e29c06](5e29c06)) * **docker:** report source build version ([#1862](#1862)) ([3807488](3807488)) * **evals:** default unparseable judge scores below pass threshold ([#1892](#1892)) ([42ebbc6](42ebbc6)) * **install:** pass sc.exe create as raw command line so binPath= quoting survives ([#1654](#1654)) ([#1702](#1702)) ([d6e0710](d6e0710)) * **install:** persist --no-http2 override through install apply ([#1676](#1676)) ([6fb5f3b](6fb5f3b)) * **mcp:** isolate ClaudeRegistrar CLI config env ([#1888](#1888)) ([1c947b1](1c947b1)) * **mcp:** surface dead proxy state ([#1786](#1786)) ([931eed8](931eed8)) * **memory:** resolve Trae cwd metadata from user reminders ([#1737](#1737)) ([#1887](#1887)) ([3e85eb1](3e85eb1)) * **opencode:** use local MCP config ([#1383](#1383)) ([4bd3ddf](4bd3ddf)) * **proxy/openai:** thread savings-profile kwargs into chat completions ([#1606](#1606)) ([7ff842d](7ff842d)) * **proxy/openai:** translate max_tokens -> max_completion_tokens on chat path ([#1774](#1774)) ([285808b](285808b)) * **proxy:** bound Codex WS compression fallback latency ([#1802](#1802)) ([d24a3f8](d24a3f8)) * **proxy:** bound HF tokenizer load and offload token counting off event loop ([#1738](#1738)) ([46d5d68](46d5d68)) * **proxy:** cancel retry backoff on shutdown ([#1834](#1834)) ([da2d8dc](da2d8dc)) * **proxy:** compress Anthropic user text blocks when enabled ([#1875](#1875)) ([e36439a](e36439a)) * **proxy:** freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting ([#1850](#1850)) ([248ae0f](248ae0f)) * **proxy:** fsync savings dir after atomic rename ([#1764](#1764)) ([7de2c1e](7de2c1e)) * **proxy:** keep cache_control bounded + stable so the freeze overlay stops busting ([#1852](#1852)) ([4820134](4820134)) * **proxy:** persist lifetime cache-read savings across restarts ([#1665](#1665)) ([908997e](908997e)) * **proxy:** preserve streaming passthrough beta headers ([#1783](#1783)) ([0f553a8](0f553a8)) * **proxy:** release _active_streams session lock on setup-phase errors ([#1864](#1864)) ([2ccd831](2ccd831)) * **proxy:** retry HTTP/2 stream resets instead of 502ing ([#1645](#1645)) ([2ce19c2](2ce19c2)) * **proxy:** retry passthrough on transient upstream connection close ([#1513](#1513)) ([5d14080](5d14080)) * **proxy:** route Foundry Anthropic messages ([#1878](#1878)) ([739f654](739f654)) * **proxy:** serve /favicon.ico locally instead of tunneling upstream ([#1787](#1787)) ([#1847](#1847)) ([3076e32](3076e32)) * **proxy:** stop rtk stat failures from corrupting session baseline ([#1693](#1693)) ([681b9a8](681b9a8)) * **proxy:** strip 1m model suffix before upstream forwarding ([#1840](#1840)) ([e22d745](e22d745)) * **proxy:** subtract cache write premiums from net savings ([#1800](#1800)) ([53a465b](53a465b)) * **router:** honor MCP aliases in excluded tools ([#1822](#1822)) ([#1863](#1863)) ([140d6e4](140d6e4)) * **rtk:** link managed rtk onto PATH instead of mutating the hook ([#1698](#1698)) ([140cb05](140cb05)) * **streaming:** preserve server_tool_use sse blocks ([#1826](#1826)) ([4ac5493](4ac5493)) * **toin:** publish skip compression recommendations ([#1782](#1782)) ([be51008](be51008)) * **transforms:** normalize diff compressor context ([#1801](#1801)) ([838c523](838c523)) * **transforms:** pass through ragged tables instead of misaligning columns ([#1713](#1713)) ([c7665ca](c7665ca)) * use rtk native Cursor hook instead of injecting .cursorrules ([#756](#756)) ([#1846](#1846)) ([1573f1f](1573f1f)) * **wrap:** replace stale-proxy detection with Vite-style port fallback ([#1406](#1406)) ([b4205c6](b4205c6)) ### Performance Improvements * **proxy:** cap compression workers to CPU count ([#1803](#1803)) ([0a3851b](0a3851b)) * **savings:** batch tracker persistence off the request hot path ([#1817](#1817)) ([451b9f0](451b9f0)) ### Dependencies * bump the cargo-minor-patch group across 1 directory with 7 updates ([#1909](#1909)) ([45601d9](45601d9)) * bump the npm-minor-patch group across 4 directories with 18 updates ([#1907](#1907)) ([8872bbc](8872bbc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…eadroomlabs-ai#1513) ## Description `GET /v1/models` (and other buffered passthrough routes) returned an opaque HTTP **502** when an OpenAI-compatible upstream closed a pooled keep-alive connection mid-response, surfacing `httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)`. The same upstream answers a direct `curl` with 200 because curl opens a fresh connection per call, while Headroom reuses pooled keep-alive connections — so the first request issued on a stale connection fails even though the upstream is healthy. The fix makes the buffered passthrough path retry once on a fresh connection (exactly what curl does), and return a clear error only if the upstream is genuinely sending an incomplete response. Closes headroomlabs-ai#1112 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom.proxy.helpers.request_with_transient_retry(client, *, request_id=None, max_retries=1, **request_kwargs)`: issues a buffered httpx request and retries on a **fresh connection** when (and only when) `httpx.RemoteProtocolError` is raised. Every other exception (`ConnectError`, timeouts, status errors) propagates immediately, so existing handling is unchanged. Documented as buffered-only (a streamed response can't be safely replayed once bytes reach the client). - Route `OpenAIHandlerMixin.handle_passthrough` through the helper, and add an `except httpx.RemoteProtocolError` arm that returns a clear `502` with error type `upstream_protocol_error` when the protocol error persists across the retry (instead of letting the raw error surface as an opaque/unhandled 502). - Add `tests/test_proxy_passthrough_transient_retry.py` (helper unit tests + handler-level tests covering the exact issue path). - Add a `CHANGELOG.md` entry under `Unreleased → Fixed`. Scope note: streaming `/v1/responses` is intentionally **out of scope** for this change — a streamed response cannot be safely retried after the first byte has been delivered to the client. The helper is written reusable so a streaming-aware follow-up can build on it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_proxy_passthrough_transient_retry.py All checks passed! $ mypy headroom/proxy/helpers.py --ignore-missing-imports Success: no issues found in 1 source file $ pytest tests/test_proxy_passthrough_transient_retry.py -q tests/test_proxy_passthrough_transient_retry.py ....... [100%] 7 passed in 0.27s # no regressions in the surrounding passthrough/handler suites: $ pytest tests/test_proxy_passthrough_transient_retry.py tests/test_proxy_handler_helpers.py \ tests/test_proxy_byte_faithful_forwarding.py \ tests/test_proxy/test_compression_failure_action.py tests/test_proxy_copilot_auth_hooks.py -q 80 passed, 1 warning in 6.88s ``` ## Real Behavior Proof Reproduced against a **real local TCP server** (no mocks) that speaks HTTP/1.1 and, when armed, emits a chunked body then closes the socket **without** the terminating `0\r\n\r\n` — the exact condition that makes httpx raise the `incomplete chunked read` error from this issue. - Environment: macOS arm64, Python 3.12, httpx 0.28.1 (same httpx major as the report), real loopback sockets via `asyncio.start_server`. - Exact command / steps: start the local server; (1) issue a single buffered request — the pre-fix `handle_passthrough` behaviour; (2) issue the same request through `request_with_transient_retry` — the fix. Verbatim: `python repro_1112.py`. - Observed result: BEFORE the fix a single request raises `httpx.RemoteProtocolError` ("incomplete chunked read") which `handle_passthrough` surfaced as an opaque HTTP 502; AFTER the fix the same request returns **HTTP 200** (the retry opened a fresh connection, mirroring a direct `curl`). Full terminal output: ```text upstream listening on http://127.0.0.1:62374/v1/models BEFORE (single buffered request, pre-fix behaviour): raised httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read) -> handle_passthrough surfaced this as an opaque HTTP 502 AFTER (request_with_transient_retry, the fix): HTTP 200 body={"object":"list","data":[]} -> first attempt hit the incomplete chunked read, retry on a fresh connection returned 200 (mirrors a direct curl) ``` The log line `Upstream closed connection mid-response (...incomplete chunked read); retrying on a fresh connection (attempt 1/1)` fires on the recovered request, confirming the retry path is what produced the 200. - Not tested: real third-party upstreams (LiteLLM/vLLM/etc.) — the local server reproduces the precise httpx error deterministically; the streaming `/v1/responses` path is intentionally out of scope (a streamed response cannot be safely retried after the first byte reaches the client). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - No new dependencies (httpx is already a proxy dependency), so no supply-chain justification is required. - The retry is deliberately narrow: only `httpx.RemoteProtocolError` is retried, capped at one retry, so a genuinely-down upstream still fails fast via the existing `ConnectError`/timeout path. - "Documentation" checklist item refers to the `CHANGELOG.md` entry; no user-facing docs pages needed for this internal resilience fix.
🤖 I have created a release *beep* *boop* --- <details><summary>0.31.0</summary> ## [0.31.0](headroomlabs-ai/headroom@v0.30.0...v0.31.0) (2026-07-09) ### Features * **cache:** provider-agnostic cache-mode delta + cc-agnostic prefix comparison ([headroomlabs-ai#1868](headroomlabs-ai#1868)) ([7c2f0ea](headroomlabs-ai@7c2f0ea)) * **ccr:** wire retrieve-tool interception into OpenAI Responses handler ([headroomlabs-ai#1898](headroomlabs-ai#1898)) ([62cd307](headroomlabs-ai@62cd307)) * **compression:** add audit-safe mode with protected pattern matching ([headroomlabs-ai#1899](headroomlabs-ai#1899)) ([bb112dd](headroomlabs-ai@bb112dd)) * **content-router:** accept any real compression (remove min-savings floor) ([headroomlabs-ai#1771](headroomlabs-ai#1771)) ([6c31db9](headroomlabs-ai@6c31db9)) * **content-router:** lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold ([headroomlabs-ai#1818](headroomlabs-ai#1818)) ([60af15f](headroomlabs-ai@60af15f)) * **proxy:** add provider-only HTTP proxy ([headroomlabs-ai#1807](headroomlabs-ai#1807)) ([ebe0a3b](headroomlabs-ai@ebe0a3b)) * **proxy:** add turn-hook extension point for buffered model turns ([headroomlabs-ai#1891](headroomlabs-ai#1891)) ([ec950f7](headroomlabs-ai@ec950f7)) ### Bug Fixes * **build:** enable Intel macOS pip installs via ort-load-dynamic ([headroomlabs-ai#1538](headroomlabs-ai#1538)) ([32ce99e](headroomlabs-ai@32ce99e)) * **cache:** avoid fallback session collisions ([headroomlabs-ai#1827](headroomlabs-ai#1827)) ([0f606b6](headroomlabs-ai@0f606b6)) * **ccr:** make expired retrieve misses terminal ([headroomlabs-ai#1781](headroomlabs-ai#1781)) ([9cbdba4](headroomlabs-ai@9cbdba4)) * **ccr:** preserve Anthropic re-stream shape ([headroomlabs-ai#1854](headroomlabs-ai#1854)) ([f663894](headroomlabs-ai@f663894)) * **ccr:** preserve thinking blocks in buffered stream re-synthesis ([headroomlabs-ai#1897](headroomlabs-ai#1897)) ([ede085c](headroomlabs-ai@ede085c)) * **cli/proxy:** preserve explicit HEADROOM_MIN_TOKENS=0 / MAX_ITEMS=0 ([headroomlabs-ai#1886](headroomlabs-ai#1886)) ([3a33af1](headroomlabs-ai@3a33af1)) * **code-compressor:** CJK-aware relevance-query symbol matching ([headroomlabs-ai#1747](headroomlabs-ai#1747)) ([b38315c](headroomlabs-ai@b38315c)) * **codex:** discover updated Codex state stores ([headroomlabs-ai#1889](headroomlabs-ai#1889)) ([9d42eba](headroomlabs-ai@9d42eba)) * **codex:** OpenCode Zen telemetry attribution ([headroomlabs-ai#1648](headroomlabs-ai#1648)) ([f18c6bd](headroomlabs-ai@f18c6bd)) * **content-detector:** detect and compress space-separated JSON objects ([headroomlabs-ai#1742](headroomlabs-ai#1742)) ([5194bdc](headroomlabs-ai@5194bdc)) * **content-router:** token-measure lossless folds at the acceptance gate ([headroomlabs-ai#1772](headroomlabs-ai#1772)) ([c5493ea](headroomlabs-ai@c5493ea)) * **copilot:** normalize subscription routing host ([headroomlabs-ai#1836](headroomlabs-ai#1836)) ([afd9cbd](headroomlabs-ai@afd9cbd)) * **copilot:** route mixed-model requests per model ([headroomlabs-ai#1785](headroomlabs-ai#1785)) ([5af5e22](headroomlabs-ai@5af5e22)) * **dashboard:** deduplicate repeated savings metrics ([headroomlabs-ai#1804](headroomlabs-ai#1804)) ([88f935a](headroomlabs-ai@88f935a)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([headroomlabs-ai#1900](headroomlabs-ai#1900)) ([87f6e93](headroomlabs-ai@87f6e93)) * **dashboard:** distinguish unavailable RTK from zero stats in Docker ([headroomlabs-ai#1901](headroomlabs-ai#1901)) ([361adcd](headroomlabs-ai@361adcd)) * **dashboard:** price proxy savings without litellm ([headroomlabs-ai#1728](headroomlabs-ai#1728)) ([188e382](headroomlabs-ai@188e382)) * detect and clear stale ANTHROPIC_BASE_URL from crashed wrap sessions ([headroomlabs-ai#1768](headroomlabs-ai#1768)) ([headroomlabs-ai#1837](headroomlabs-ai#1837)) ([84509a4](headroomlabs-ai@84509a4)) * **docker:** persist headroom workspace in compose ([headroomlabs-ai#1839](headroomlabs-ai#1839)) ([5e29c06](headroomlabs-ai@5e29c06)) * **docker:** report source build version ([headroomlabs-ai#1862](headroomlabs-ai#1862)) ([3807488](headroomlabs-ai@3807488)) * **evals:** default unparseable judge scores below pass threshold ([headroomlabs-ai#1892](headroomlabs-ai#1892)) ([42ebbc6](headroomlabs-ai@42ebbc6)) * **install:** pass sc.exe create as raw command line so binPath= quoting survives ([headroomlabs-ai#1654](headroomlabs-ai#1654)) ([headroomlabs-ai#1702](headroomlabs-ai#1702)) ([d6e0710](headroomlabs-ai@d6e0710)) * **install:** persist --no-http2 override through install apply ([headroomlabs-ai#1676](headroomlabs-ai#1676)) ([6fb5f3b](headroomlabs-ai@6fb5f3b)) * **mcp:** isolate ClaudeRegistrar CLI config env ([headroomlabs-ai#1888](headroomlabs-ai#1888)) ([1c947b1](headroomlabs-ai@1c947b1)) * **mcp:** surface dead proxy state ([headroomlabs-ai#1786](headroomlabs-ai#1786)) ([931eed8](headroomlabs-ai@931eed8)) * **memory:** resolve Trae cwd metadata from user reminders ([headroomlabs-ai#1737](headroomlabs-ai#1737)) ([headroomlabs-ai#1887](headroomlabs-ai#1887)) ([3e85eb1](headroomlabs-ai@3e85eb1)) * **opencode:** use local MCP config ([headroomlabs-ai#1383](headroomlabs-ai#1383)) ([4bd3ddf](headroomlabs-ai@4bd3ddf)) * **proxy/openai:** thread savings-profile kwargs into chat completions ([headroomlabs-ai#1606](headroomlabs-ai#1606)) ([7ff842d](headroomlabs-ai@7ff842d)) * **proxy/openai:** translate max_tokens -> max_completion_tokens on chat path ([headroomlabs-ai#1774](headroomlabs-ai#1774)) ([285808b](headroomlabs-ai@285808b)) * **proxy:** bound Codex WS compression fallback latency ([headroomlabs-ai#1802](headroomlabs-ai#1802)) ([d24a3f8](headroomlabs-ai@d24a3f8)) * **proxy:** bound HF tokenizer load and offload token counting off event loop ([headroomlabs-ai#1738](headroomlabs-ai#1738)) ([46d5d68](headroomlabs-ai@46d5d68)) * **proxy:** cancel retry backoff on shutdown ([headroomlabs-ai#1834](headroomlabs-ai#1834)) ([da2d8dc](headroomlabs-ai@da2d8dc)) * **proxy:** compress Anthropic user text blocks when enabled ([headroomlabs-ai#1875](headroomlabs-ai#1875)) ([e36439a](headroomlabs-ai@e36439a)) * **proxy:** freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting ([headroomlabs-ai#1850](headroomlabs-ai#1850)) ([248ae0f](headroomlabs-ai@248ae0f)) * **proxy:** fsync savings dir after atomic rename ([headroomlabs-ai#1764](headroomlabs-ai#1764)) ([7de2c1e](headroomlabs-ai@7de2c1e)) * **proxy:** keep cache_control bounded + stable so the freeze overlay stops busting ([headroomlabs-ai#1852](headroomlabs-ai#1852)) ([4820134](headroomlabs-ai@4820134)) * **proxy:** persist lifetime cache-read savings across restarts ([headroomlabs-ai#1665](headroomlabs-ai#1665)) ([908997e](headroomlabs-ai@908997e)) * **proxy:** preserve streaming passthrough beta headers ([headroomlabs-ai#1783](headroomlabs-ai#1783)) ([0f553a8](headroomlabs-ai@0f553a8)) * **proxy:** release _active_streams session lock on setup-phase errors ([headroomlabs-ai#1864](headroomlabs-ai#1864)) ([2ccd831](headroomlabs-ai@2ccd831)) * **proxy:** retry HTTP/2 stream resets instead of 502ing ([headroomlabs-ai#1645](headroomlabs-ai#1645)) ([2ce19c2](headroomlabs-ai@2ce19c2)) * **proxy:** retry passthrough on transient upstream connection close ([headroomlabs-ai#1513](headroomlabs-ai#1513)) ([5d14080](headroomlabs-ai@5d14080)) * **proxy:** route Foundry Anthropic messages ([headroomlabs-ai#1878](headroomlabs-ai#1878)) ([739f654](headroomlabs-ai@739f654)) * **proxy:** serve /favicon.ico locally instead of tunneling upstream ([headroomlabs-ai#1787](headroomlabs-ai#1787)) ([headroomlabs-ai#1847](headroomlabs-ai#1847)) ([3076e32](headroomlabs-ai@3076e32)) * **proxy:** stop rtk stat failures from corrupting session baseline ([headroomlabs-ai#1693](headroomlabs-ai#1693)) ([681b9a8](headroomlabs-ai@681b9a8)) * **proxy:** strip 1m model suffix before upstream forwarding ([headroomlabs-ai#1840](headroomlabs-ai#1840)) ([e22d745](headroomlabs-ai@e22d745)) * **proxy:** subtract cache write premiums from net savings ([headroomlabs-ai#1800](headroomlabs-ai#1800)) ([53a465b](headroomlabs-ai@53a465b)) * **router:** honor MCP aliases in excluded tools ([headroomlabs-ai#1822](headroomlabs-ai#1822)) ([headroomlabs-ai#1863](headroomlabs-ai#1863)) ([140d6e4](headroomlabs-ai@140d6e4)) * **rtk:** link managed rtk onto PATH instead of mutating the hook ([headroomlabs-ai#1698](headroomlabs-ai#1698)) ([140cb05](headroomlabs-ai@140cb05)) * **streaming:** preserve server_tool_use sse blocks ([headroomlabs-ai#1826](headroomlabs-ai#1826)) ([4ac5493](headroomlabs-ai@4ac5493)) * **toin:** publish skip compression recommendations ([headroomlabs-ai#1782](headroomlabs-ai#1782)) ([be51008](headroomlabs-ai@be51008)) * **transforms:** normalize diff compressor context ([headroomlabs-ai#1801](headroomlabs-ai#1801)) ([838c523](headroomlabs-ai@838c523)) * **transforms:** pass through ragged tables instead of misaligning columns ([headroomlabs-ai#1713](headroomlabs-ai#1713)) ([c7665ca](headroomlabs-ai@c7665ca)) * use rtk native Cursor hook instead of injecting .cursorrules ([headroomlabs-ai#756](headroomlabs-ai#756)) ([headroomlabs-ai#1846](headroomlabs-ai#1846)) ([1573f1f](headroomlabs-ai@1573f1f)) * **wrap:** replace stale-proxy detection with Vite-style port fallback ([headroomlabs-ai#1406](headroomlabs-ai#1406)) ([b4205c6](headroomlabs-ai@b4205c6)) ### Performance Improvements * **proxy:** cap compression workers to CPU count ([headroomlabs-ai#1803](headroomlabs-ai#1803)) ([0a3851b](headroomlabs-ai@0a3851b)) * **savings:** batch tracker persistence off the request hot path ([headroomlabs-ai#1817](headroomlabs-ai#1817)) ([451b9f0](headroomlabs-ai@451b9f0)) ### Dependencies * bump the cargo-minor-patch group across 1 directory with 7 updates ([headroomlabs-ai#1909](headroomlabs-ai#1909)) ([45601d9](headroomlabs-ai@45601d9)) * bump the npm-minor-patch group across 4 directories with 18 updates ([headroomlabs-ai#1907](headroomlabs-ai#1907)) ([8872bbc](headroomlabs-ai@8872bbc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Description
GET /v1/models(and other buffered passthrough routes) returned an opaqueHTTP 502 when an OpenAI-compatible upstream closed a pooled keep-alive
connection mid-response, surfacing
httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read). The same upstream answers a directcurlwith 200 because curl opens a fresh connection per call, while Headroomreuses pooled keep-alive connections — so the first request issued on a stale
connection fails even though the upstream is healthy.
The fix makes the buffered passthrough path retry once on a fresh connection
(exactly what curl does), and return a clear error only if the upstream is
genuinely sending an incomplete response.
Closes #1112
Type of Change
Changes Made
headroom.proxy.helpers.request_with_transient_retry(client, *, request_id=None, max_retries=1, **request_kwargs): issues a buffered httpx request and retries on a fresh connection when (and only when)httpx.RemoteProtocolErroris raised. Every other exception (ConnectError, timeouts, status errors) propagates immediately, so existing handling is unchanged. Documented as buffered-only (a streamed response can't be safely replayed once bytes reach the client).OpenAIHandlerMixin.handle_passthroughthrough the helper, and add anexcept httpx.RemoteProtocolErrorarm that returns a clear502with error typeupstream_protocol_errorwhen the protocol error persists across the retry (instead of letting the raw error surface as an opaque/unhandled 502).tests/test_proxy_passthrough_transient_retry.py(helper unit tests + handler-level tests covering the exact issue path).CHANGELOG.mdentry underUnreleased → Fixed.Scope note: streaming
/v1/responsesis intentionally out of scope for thischange — a streamed response cannot be safely retried after the first byte has
been delivered to the client. The helper is written reusable so a
streaming-aware follow-up can build on it.
Testing
pytest)ruff check .)mypy headroom)Test Output
Real Behavior Proof
Reproduced against a real local TCP server (no mocks) that speaks HTTP/1.1
and, when armed, emits a chunked body then closes the socket without the
terminating
0\r\n\r\n— the exact condition that makes httpx raise theincomplete chunked readerror from this issue.asyncio.start_server.handle_passthroughbehaviour; (2) issue the same request throughrequest_with_transient_retry— the fix. Verbatim:python repro_1112.py.httpx.RemoteProtocolError("incomplete chunked read") whichhandle_passthroughsurfaced as an opaque HTTP 502; AFTER the fix the same request returns HTTP 200 (the retry opened a fresh connection, mirroring a directcurl). Full terminal output:The log line
Upstream closed connection mid-response (...incomplete chunked read); retrying on a fresh connection (attempt 1/1)fires on the recoveredrequest, confirming the retry path is what produced the 200.
/v1/responsespath is intentionally out of scope (a streamed response cannot be safely retried after the first byte reaches the client).Review Readiness
Checklist
Additional Notes
httpx.RemoteProtocolErroris retried, capped at one retry, so a genuinely-down upstream still fails fast via the existingConnectError/timeout path.CHANGELOG.mdentry; no user-facing docs pages needed for this internal resilience fix.