Skip to content

fix(cloakserve): evict poll()-alive but CDP-dead browsers instead of serving 502s - #492

Open
DivMode wants to merge 1 commit into
CloakHQ:mainfrom
DivMode:fix/cloakserve-evict-dead-cdp-corpse
Open

DivMode wants to merge 1 commit into
CloakHQ:mainfrom
DivMode:fix/cloakserve-evict-dead-cdp-corpse

Conversation

@DivMode

@DivMode DivMode commented Aug 8, 2026 •

Copy link
Copy Markdown

Problem

ChromePool.get_or_launch decides whether to reuse a pooled Chrome for a seed with a single liveness check — proc.process.poll() is None. But poll() only reports whether the OS process has been reaped, which lags CDP availability in two common ways:

  • Graceful-shutdown window. Chrome tears down its DevTools/CDP listening socket seconds before the process actually exits. A client that closes a browser and immediately requests the same fingerprint seed is handed the still-"alive" process whose CDP port already refuses connections.
  • Wedged browser. A hung/unresponsive Chrome can keep its process alive indefinitely while its CDP port no longer answers.

In both cases the dead process is returned, and handle_json_version / handle_json_list then fail to reach http://127.0.0.1:<port>/json/version and return 502 {"error": "CDP endpoint unreachable"}. Crucially, the 502 path does not evict the dead pool entry, so every retry for that seed hits the same corpse — the failure is not self-healing until the OS finally reaps the process (or the idle timeout fires).

This is easy to trigger under any workflow that reuses seeds across browser restarts, or when a browser crashes/OOMs and lingers mid-shutdown. Under load it presents as a burst of dead-port 502s against a seed whose process still poll()s as alive.

Fix

  1. get_or_launch reuses a poll()-alive process only if its CDP port actually accepts a TCP connection (_cdp_port_alive, a sub-millisecond local connect). poll() short-circuits the check, so an already-exited process skips the probe and takes the existing cleanup+relaunch path. A poll-alive-but-dead-port corpse now falls through to _cleanup_process + a fresh launch, under the existing per-seed lock.
  2. handle_json_version and handle_json_list wrap the fetch in a 2-attempt loop: on the first connection failure they evict the stale entry and retry once, so a browser that dies in the narrow window between the probe and the fetch becomes a single cold relaunch instead of a persistent 502. Eviction matches the pooled process by object identity, never by recomputing the seed key, so a concurrent relaunch on the same seed is never killed.

The reuse fast-path stays fast (one cheap local connect) and behavior is unchanged for a genuinely-live browser.

Tests

Added to tests/test_cloakserve.py:

  • TestCdpPortAlive — the probe returns True for a listening port, False for a closed one.
  • TestPoolKeyForIdentity — identity match; returns None when a concurrent relaunch has replaced the entry.
  • TestGetOrLaunchReuseGuard — poll-alive-but-dead-port is evicted and relaunched; poll-alive-and-port-alive is reused after exactly one probe (no relaunch).
  • TestHandlerCorpseRecovery — json/version and json/list recover without a 502 by evicting once and retrying; they return 502 only after both attempts fail; identity-based eviction spares a concurrently-relaunched process.

Two existing tests were adjusted for the new reuse probe. pytest tests/test_cloakserve.py → 113 passed.

…serving 502s

process.poll() reports a Chrome alive until the OS reaps it, but Chrome drops
its DevTools/CDP socket seconds before it exits (and a wedged Chrome keeps the
process alive with a dead port indefinitely). ChromePool.get_or_launch reused
such a corpse on poll() alone, and handle_json_version / handle_json_list then
hit connection-refused on the CDP port and returned HTTP 502 without evicting
the pool entry -- so retries kept hitting the same dead browser. Not
self-healing.

- get_or_launch reuses a pooled Chrome only when its CDP port is actually
  accepting TCP connections (_cdp_port_alive). poll() short-circuits the probe,
  so an already-exited process skips straight to relaunch, while a poll()-alive
  but dead-port process falls through to _cleanup_process + relaunch.
- handle_json_version and handle_json_list wrap get_or_launch + the proxied CDP
  GET in a 2-attempt loop: on the first fetch failure they evict the stale
  entry by object identity (_pool_key_for) and retry; a second failure returns
  the original 502. Identity-based eviction never removes a concurrent
  same-seed relaunch. get_or_launch stays outside the fetch try, so its own
  errors propagate unchanged.

Adds tests for the reuse-guard eviction/reuse decision, handler recovery vs.
502-after-both-fail, the identity-based eviction guard, and _cdp_port_alive.
@DivMode
DivMode force-pushed the fix/cloakserve-evict-dead-cdp-corpse branch from 865d6fd to 7c69b11 Compare August 8, 2026 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant