Skip to content

test: stop timing-dependent failures on a loaded Windows machine - #308

Merged
amirlehmam merged 1 commit into
amirlehmam:masterfrom
clemrdh:test/deflake-timing
Oct 19, 2026
Merged

amirlehmam merged 1 commit into
amirlehmam:masterfrom
clemrdh:test/deflake-timing

Conversation

@clemrdh

@clemrdh clemrdh commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

On a busy Windows desktop (an active wmux with ~28 panes and agents running), npx vitest run failed on almost every full run. The failures came from two test families, and the code under test was correct in both. Run in isolation, every one of these tests passed.

1. pipe-client-latency: a wall-clock budget measured the machine

These tests timed each pipe client from the server's reply to the client's exit, against 45 ms. That budget sits just under libuv's 50 ms eof_timeout, the cost of the old end()-then-wait client. Clients doing the right thing measured 50, 66, 68, 72, 119, 144, 188, 505, 647 and 1114 ms across our runs: that is node teardown and scheduling, not the pipe. The budget had already been raised once, from 20 ms to 45 ms (e95c2a3), after a CI runner hit 20.3 ms.

The regression these tests guard is a call sequence, so they now assert that sequence from inside the client:

  • A preloaded tracer, tests/helpers/socket-trace.cjs (loaded with node --require), logs the reply arriving and every end()/destroy() on the client's own connection. It never touches stdio.
  • The client must never half-close while waiting, and must destroy the connection first thing once the reply arrives. Real traces: data > destroy for wmux.js ping, wmux.js identify and wmux-hook.js.
  • A loose ceiling (4 s, under the CLI's 5 s deadline) still catches a client that ignores the reply and waits for its deadline.
  • A fourth test runs the old write(); end(); wait-for-close client and checks the tracer sees end before data, so the other three cannot pass vacuously.

2. Orchestrator bash suites: limits tighter than the scripts are slow

orchestrator-final-reap, orchestrator-reap, orchestrator-groups and orchestration-status-vocab drive bash scripts that start a node process for every state.json access, through Git Bash's fork emulation (and the antivirus).

  • On an idle desktop, single tests measured 18–30 s; one took 29.8 s against its 30 s limit. Inside a full parallel run they take about twice that.
  • The limits were 15, 30 or 60 s, depending on the file.
  • The scripts run through spawnSync/execFileSync, which block the worker. vitest's timer cannot fire during a script, so these timeouts never caught a hang: they only turned a slow pass into a failure.

Changes:

  • BASH_TEST_TIMEOUT_MS (300 s), in tests/helpers/bash-path.ts, is now the one limit for all four files, with the reason written down.
  • BASH_CALL_TIMEOUT_MS (120 s) is the real hang guard. It is set on the spawnSync in wmux-stub.ts and on status-vocab's execFileSync. A script that stops is killed, and the test fails with "bash did not finish" instead of freezing the worker.

Not in this PR

The Codex relay "session picker" test had the same kind of race: a 100 ms delay standing in for "the picker's backend answers first". It is fixed in the Codex PR (#309), which already changes that file.

Validation

  • The six affected files pass (102 tests).
  • Full suite with --maxWorkers=2 on the same busy machine: 4,815 passed, 0 failed, 14 skipped. Before this change, the same machine failed 3–5 of these tests on every full run.

🤖 Generated with Claude Code

Two families of tests failed on almost every full run on a busy Windows
desktop, with nothing wrong in the code under test.

pipe-client-latency timed each pipe client's exit against a 45 ms budget
set just under libuv's 50 ms eof_timeout. A client doing the right thing
measured 50-1114 ms there: node teardown and scheduling, not the pipe.
The budget had already been raised once, from 20 ms, after a CI runner
hit 20.3 ms. The regression it guards is a call sequence, so the tests
now assert that sequence from inside the client. A preloaded tracer
(tests/helpers/socket-trace.cjs) logs the reply arriving and every
end()/destroy() on the client's own connection. The client must never
half-close while waiting, and must destroy the connection first thing
once the reply is in. A loose ceiling well under the 5 s deadline still
catches a client that ignores the reply. A fourth test proves the tracer
sees the old end()-then-wait client.

The orchestrator bash suites (final-reap, reap, groups, status-vocab)
start a node process per state.json access, through Git Bash's fork
emulation. One test measured 18-30 s on an idle desktop and about twice
that inside a full run, against limits of 15, 30 or 60 s depending on
the file. Those scripts run through spawnSync/execFileSync, which block
the worker, so the vitest timeout never caught a hang: it only turned a
slow pass into a failure.
- All four files now share one limit, BASH_TEST_TIMEOUT_MS (300 s).
- The real hang guard is BASH_CALL_TIMEOUT_MS (120 s) on the call
  itself: a stuck script is killed and the test fails with a reason,
  instead of freezing the worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T11:20:30.979918Z c1a34b5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c1a34b53f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

expect(reply, `no reply reached the client: ${JSON.stringify(trace)}`).toBeGreaterThanOrEqual(0);
// An end() here is the bug: libuv's 50 ms eof_timeout before the client can exit.
expect(trace.slice(0, reply)).not.toContain('end');
expect(trace.slice(reply + 1).find((step) => step !== 'data')).toBe('destroy');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert that destroy runs in the data callback

When a client defers destroy() for any interval below four seconds (for example, with a three-second timer), the tracer still records exactly data, destroy, so this helper and the new LONG_BEFORE_DEADLINE_MS checks all pass even though every invocation now incurs multi-second latency. The previous 45 ms assertion caught that regression; bracket the call to the original emit (or otherwise record callback completion) so this assertion proves that destroy() occurs synchronously while dispatching the reply rather than merely sometime later.

Useful? React with 👍 / 👎.

@amirlehmam
amirlehmam merged commit 810cbc2 into amirlehmam:master Oct 19, 2026
@amirlehmam

Copy link
Copy Markdown
Owner

Merged, thank you. Asserting the call sequence instead of a wall-clock budget is the right fix, and the control test against the old end() client keeps it from passing vacuously.

One follow-up from the bot review: a client that defers destroy() by any delay under 4 s still produces the trace data > destroy, so the latency regression the 45 ms budget used to catch could slip through. Recording whether destroy runs inside the data callback (by wrapping the original emit) would close that. Not blocking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants