Skip to content

fix(pty): reap process trees orphaned by an abnormal exit (#139) - #140

Merged
amirlehmam merged 1 commit into
masterfrom
fix/orphaned-pty-trees
Aug 3, 2026
Merged

fix(pty): reap process trees orphaned by an abnormal exit (#139)#140
amirlehmam merged 1 commit into
masterfrom
fix/orphaned-pty-trees

Conversation

@amirlehmam

Copy link
Copy Markdown
Owner

Closes #139.

What actually goes wrong

killAll() is reachable from exactly one place — app.on('will-quit') — and that does not run when the process dies abnormally. Windows does not tear down a process tree when its root dies, so a crashed wmux leaves every pane's shell resident, along with the agent under it and each of that agent's MCP servers. Restoring the session then spawns a fresh set beside them, so a crash-loop multiplies processes instead of replacing them. Six crash cycles is how the reporter reached 251 node.exe / 3.3 GB.

The tree-kill itself was never the problem: PtyManager.kill() has done taskkill /PID <pid> /T /F since #65, precisely because Claude Code's backend orphans otherwise. Every caller just required wmux to be alive and cooperating to reach it.

The fix

A ledger of spawned PIDs under APPDATA. Each spawn appends, each kill removes, killAll() clears. The next launch claims the file and reaps whatever a dead owner left behind.

On not killing the wrong process. This list feeds taskkill /T /F, so a stale entry whose PID has since been recycled would kill an unrelated process and its children. A recorded PID being alive is therefore not evidence that it is ours. Before killing anything, each PID is re-queried through Win32_Process and must still match the image name and the creation time recorded at spawn. Every failure path — no PowerShell, a query error, unparseable output — reaps nothing: leaking a process is recoverable, killing the user's editor is not. selectOrphans() is pure so those rules are tested without spawning anything.

Two smaller notes:

  • The ledger is injected into PtyManager rather than constructed inside it. The pty tests spawn real shells, and must not overwrite the ledger of an instance the user has running on the same machine.
  • takeOver() returns nothing when the recorded owner is still running — those PTYs have a live parent and are not orphans.

Also: two hook lifetimes

Found while chasing the same symptom, in wmux-hook.js:

  • The 1s stdin fallback timer was never cleared, so every hook process stayed resident for a full second even when its work finished in ~90 ms. Claude Code fires one per tool call, per pane. Measured 1049 ms → 101 ms.
  • The reply was never read. On Windows named pipes end() tears the connection down, but a paused socket has no queued read, so the EOF never surfaces to JS — 'end' and 'close' never fire. resume() makes the close observable and deterministic.

A 5s connect deadline matching the CLI's sendV1/sendV2 is added as a backstop. It is explicitly not a fix for an observed hang — measured against absent, responsive and wedged servers, the hook exits on its own in all three. I had assumed a hang and the measurement disproved it; the comment says so rather than claiming otherwise.

Verification

tests/unit/pty-ledger.test.ts — 22 cases, weighted toward what must not be killed: recycled PID under a different image, recycled PID outside the creation-time window, malformed ledger entries, a live owner. Plus a live Win32_Process probe of our own PID, which is what proves the PowerShell command string quotes correctly.

Beyond the suite, exercised against real process trees (two cmd.exeping trees, one recorded truthfully, one recorded under a wrong image):

PASS  recorded tree root killed
PASS  recorded tree GRANDCHILD killed (the MCP-server case)
PASS  image-mismatched tree root SPARED
PASS  image-mismatched tree grandchild SPARED
PASS  ledger now owned by this process

And the hook, pre-fix vs post-fix:

server shape    | before (HEAD)      | after (fix)
----------------+--------------------+------------------
absent          | 43ms               | 40ms
responsive      | 1044ms             | 100ms
wedged          | 1051ms             | 99ms

Full suite: 62 files, 689 tests passing. npm run typecheck clean; npm run lint unchanged from master (31 problems before and after, none in the touched lines).

What this does not fix

The crash itself. Six .dmp files means a native crash, and I have no way to diagnose one without them — asked for them on the issue. This change makes a crash survivable rather than cumulative; it does not stop it happening.

killAll() is reachable from exactly one place — app.on('will-quit') — which
does not run when the process dies abnormally. Windows does not tear down a
process tree when its root dies, so a crashed wmux leaves every pane's shell
resident, along with the agent under it and each of that agent's MCP servers.
Restoring the session then spawns a fresh set beside them, so a crash-loop
multiplies processes instead of replacing them: the reporter reached 251
node.exe / 3.3 GB across six crash cycles.

PtyManager already tree-kills correctly (taskkill /T /F, added for #65); every
caller just required wmux to be alive to reach it. So record the PIDs instead:
each spawn appends to a ledger under APPDATA, each kill removes it, and the
next launch claims the file and reaps whatever the dead owner left behind.

The ledger feeds taskkill /T /F, so a stale entry whose PID has been recycled
would kill an unrelated process AND its children. Liveness alone is therefore
not evidence: before killing anything, each PID is re-queried through
Win32_Process and must still match the image name and creation time recorded
at spawn. Every failure path — no PowerShell, a query error, unparseable
output — reaps nothing, because leaking a process is recoverable and killing
the user's editor is not. selectOrphans() is pure so those rules are tested
without spawning anything.

The ledger is injected rather than constructed inside PtyManager: the pty
tests spawn real shells, and must not overwrite the ledger of an instance the
user has running on the same machine.

Also two lifetime fixes in wmux-hook.js, found while chasing the same symptom:

  - the 1s stdin fallback timer was never cleared, so every hook process stayed
    resident for a full second even when its work finished in ~90ms. Claude
    Code fires one per tool call per pane. Measured 1049ms -> 101ms.
  - the reply was never read. On Windows named pipes end() tears the connection
    down, but a paused socket has no queued read, so the EOF never surfaces and
    'close' never fires. resume() makes the close observable and deterministic.

A 5s connect deadline matching the CLI's is added as a backstop. It is
explicitly not a fix for an observed hang: measured against absent, responsive
and wedged servers the hook exits on its own in all three.

Verified against real process trees: a recorded tree is killed down to its
grandchild, and an image-mismatched entry is left entirely alone.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 946c82b2ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/main/ipc-handlers.ts
Comment on lines +40 to +41
const ptyLedger = new PtyLedger(path.join(getAppDataDir(), 'pty-ledger.json'));
const orphanCandidates = ptyLedger.takeOver();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Claim the ledger only after winning the instance lock

When a second wmux launch is handed off to an already-running instance, this module-level takeOver() still executes during the import at index.ts:2, before requestSingleInstanceLock() runs at index.ts:310. Although takeOver() detects the live owner and returns no candidates, it then unconditionally flushes an empty ledger under the losing process's PID; if the active instance subsequently crashes without another ledger mutation, its PTY roots are no longer recoverable. Defer takeover until gotInstanceLock is true, or preserve the live owner's file.

Useful? React with 👍 / 👎.

@amirlehmam
amirlehmam merged commit 8807caa into master Aug 3, 2026
amirlehmam added a commit that referenced this pull request Aug 3, 2026
The probe ran with a 15s execFile timeout, which the first invocation on a
cold machine can exceed — PowerShell 5.1 pulling in .NET and the CIM
assemblies is slow before anything is warm. Timing out resolves to [] and
reaps nothing, and the moment that matters most is the first scan after a
crash-loop, which is exactly when the machine is coldest. It also failed CI
on a fresh runner (#140's live probe test), with no stderr to explain why,
because a killed process has none.

60s, and the warning now distinguishes a timeout from a real failure and
prints stderr rather than the whole command line. The live-probe tests budget
past the probe's own timeout so a slow machine reports the real problem
instead of a vitest timeout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VwpQ2e9r28DcBNqeq3F7qv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0.41.0: crash-loop leaves orphaned MCP child processes — CPU/RAM spike (30%/3.3GB, 251 processes)

1 participant