Repository navigation
Conversation
…serving 502s process.poll() reports a Chrome alive until the OS reaps it, but Chrome drops its DevTools/CDP socket seconds before it exits (and a wedged Chrome keeps the process alive with a dead port indefinitely). ChromePool.get_or_launch reused such a corpse on poll() alone, and handle_json_version / handle_json_list then hit connection-refused on the CDP port and returned HTTP 502 without evicting the pool entry -- so retries kept hitting the same dead browser. Not self-healing. - get_or_launch reuses a pooled Chrome only when its CDP port is actually accepting TCP connections (_cdp_port_alive). poll() short-circuits the probe, so an already-exited process skips straight to relaunch, while a poll()-alive but dead-port process falls through to _cleanup_process + relaunch. - handle_json_version and handle_json_list wrap get_or_launch + the proxied CDP GET in a 2-attempt loop: on the first fetch failure they evict the stale entry by object identity (_pool_key_for) and retry; a second failure returns the original 502. Identity-based eviction never removes a concurrent same-seed relaunch. get_or_launch stays outside the fetch try, so its own errors propagate unchanged. Adds tests for the reuse-guard eviction/reuse decision, handler recovery vs. 502-after-both-fail, the identity-based eviction guard, and _cdp_port_alive.
DivMode
force-pushed
the
fix/cloakserve-evict-dead-cdp-corpse
branch
from
August 8, 2026 03:59
865d6fd to
7c69b11
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ChromePool.get_or_launchdecides whether to reuse a pooled Chrome for a seed with a single liveness check —proc.process.poll() is None. Butpoll()only reports whether the OS process has been reaped, which lags CDP availability in two common ways:In both cases the dead process is returned, and
handle_json_version/handle_json_listthen fail to reachhttp://127.0.0.1:<port>/json/versionand return502 {"error": "CDP endpoint unreachable"}. Crucially, the 502 path does not evict the dead pool entry, so every retry for that seed hits the same corpse — the failure is not self-healing until the OS finally reaps the process (or the idle timeout fires).This is easy to trigger under any workflow that reuses seeds across browser restarts, or when a browser crashes/OOMs and lingers mid-shutdown. Under load it presents as a burst of dead-port 502s against a seed whose process still
poll()s as alive.Fix
get_or_launchreuses apoll()-alive process only if its CDP port actually accepts a TCP connection (_cdp_port_alive, a sub-millisecond local connect).poll()short-circuits the check, so an already-exited process skips the probe and takes the existing cleanup+relaunch path. A poll-alive-but-dead-port corpse now falls through to_cleanup_process+ a fresh launch, under the existing per-seed lock.handle_json_versionandhandle_json_listwrap the fetch in a 2-attempt loop: on the first connection failure they evict the stale entry and retry once, so a browser that dies in the narrow window between the probe and the fetch becomes a single cold relaunch instead of a persistent 502. Eviction matches the pooled process by object identity, never by recomputing the seed key, so a concurrent relaunch on the same seed is never killed.The reuse fast-path stays fast (one cheap local connect) and behavior is unchanged for a genuinely-live browser.
Tests
Added to
tests/test_cloakserve.py:TestCdpPortAlive— the probe returns True for a listening port, False for a closed one.TestPoolKeyForIdentity— identity match; returns None when a concurrent relaunch has replaced the entry.TestGetOrLaunchReuseGuard— poll-alive-but-dead-port is evicted and relaunched; poll-alive-and-port-alive is reused after exactly one probe (no relaunch).TestHandlerCorpseRecovery—json/versionandjson/listrecover without a 502 by evicting once and retrying; they return 502 only after both attempts fail; identity-based eviction spares a concurrently-relaunched process.Two existing tests were adjusted for the new reuse probe.
pytest tests/test_cloakserve.py→ 113 passed.