Skip to content

fix(codex): keep session capture through a restore burst; never save an empty thread - #309

Merged
amirlehmam merged 1 commit into
amirlehmam:masterfrom
clemrdh:fix/codex-resume-reliability
Oct 19, 2026
Merged

amirlehmam merged 1 commit into
amirlehmam:masterfrom
clemrdh:fix/codex-resume-reliability

Conversation

@clemrdh

@clemrdh clemrdh commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Restoring a session starts every saved Codex at once, and about half of them ran without session capture. This was seen on 2.28.1 with 10 restored Codex tabs:

  • 9 launchers started within 11 s;
  • 5 got the relay and 4 started natively, interleaved, so the cause is load, not configuration;
  • the 10th tab saved a thread Codex never wrote, so codex resume failed there on every restart.

A natively started Codex still shows its conversation, but wmux no longer follows that tab: a /new or a quit there is never recorded.

1. The restore burst (wmux-codex.ts)

Each launch hung on two tight limits, and failing either one silently meant "restore off":

  • pane.codex_restore_config got one 2 s pipe request. Requests are now asked again while wmux is merely busy: no reply, a dropped connection, EBUSY, ECONNRESET, EPIPE or ETIMEDOUT. Each attempt is longer than the last, within a budget: config 45 s, session reports 60 s, release 10 s. An absent wmux (ENOENT), "outside wmux" or an error reply from an older wmux still answers at once, as before. The pane prints a line when it waits, and another if it finally starts without restore.
  • codex --help got 5 s, once per pane, so the burst of probes was itself what ran out of time. The probe now gets 30 s. Its "yes" is remembered per executable (path, size and mtime, so an update invalidates it), which means a restore probes once. A probe that timed out is retried, and silence is no longer reported as "this Codex version cannot record sessions".

2. Empty threads (codex-session-tracker.ts, agent-state.ts)

The relay reported a thread at thread/start, but Codex writes a conversation only once it has content. A tab opened on Codex and left empty therefore saved an id that codex resume answers with -32600 "no rollout found for thread id <id>". Nothing ever released it: the pane never reported, and silence keeps a restored id (by design, so that slow restores keep their handle).

  • Prevention. A started thread is reported at its first content: turn/start, review/start, thread/shellCommand or thread/inject_items, all of which require threadId in the v2 protocol schema. Resumed and forked threads are reported at once, as before.
  • Clean-up. When the conversation a launch was asked to resume gets exactly that answer, naming that id, the launcher sends pane.codex_session_missing. The next save drops that id, and only that id, from that tab. A later report in the tab wins over it.

I first tried pruning saved ids that had no rollout file, as Claude's prune does, and abandoned it. Codex 0.163 migrated history into SQLite (legacy_to_paginated_v1): here 648 of 700 threads have no rollout file and still resume. The missing-thread signal therefore comes from Codex's own answer.

Verified live (Codex 0.163, Windows 11)

  • The branch's relay ran against the real codex app-server. The empty thread from the incident triggered onMissing with its id; a migrated thread with no rollout file resumed and triggered onSession.
  • turn/start, review/start, thread/shellCommand and thread/inject_items were checked against codex app-server generate-json-schema.

Tests

  • Tracker:
    • a started thread is reported at its first content and never when left empty;
    • each content method counts;
    • "missing" is raised only for the exact resumed thread named in Codex's answer.
  • Persistence:
    • only the refused id goes, only in its own tab;
    • another id is kept;
    • a later report wins over the refusal;
    • a hostile id is refused.
  • Launcher:
    • a busy wmux is asked again with longer attempts and one notice;
    • retrying stops at the end of the budget;
    • an absent wmux, an older wmux and "outside wmux" each answer at once;
    • codex --help is cached per executable and invalidated by an update;
    • a timed-out probe is retried, and silence is not read as "no";
    • resumeTarget parses the arguments correctly.
  • Four existing tests encoded "report at thread/start" and now send the first turn.
  • The "session picker" relay test is no longer timing-dependent. It used a 100 ms delay to order two backends' replies, and lost whenever the picker's own backend took longer than that to start. The fake server now holds the reply until the test releases it.
  • npm run typecheck and npm run verify:resources pass (resources/cli regenerated), and ESLint is clean on the changed files.
  • Full suite with --maxWorkers=2: 4,838 passed, 1 failed, 14 skipped. The one failure is pipe-client-latency's 45 ms wall-clock budget (188 ms on a busy machine), which test: stop timing-dependent failures on a loaded Windows machine #308 replaces with a deterministic check.

🤖 Generated with Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-19T11:14:42.728731Z ad1a9c0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

…an empty thread

Restoring a session starts every saved Codex at once, and about half of
them ran without session capture: wmux lost track of those tabs, so a
`/new` or a quit there was never recorded. Seen with 9 restored Codex
tabs: 5 relayed, 4 native, interleaved within 11 s. Each launch hung on
two tight limits, and failing either one silently meant "restore off":
- `pane.codex_restore_config` got one 2 s pipe request;
- `codex --help` got 5 s, run once per pane, so the burst of probes was
  itself what ran out of time.

- Pipe requests are now asked again while wmux is merely busy: no reply,
  a dropped connection, EBUSY/ECONNRESET/EPIPE/ETIMEDOUT. Each attempt
  is longer, within a budget: config 45 s, session reports 60 s,
  release 10 s. An absent wmux (ENOENT), "outside wmux" or an error reply
  from an older wmux still answers at once. The pane says when it waits,
  and when it finally starts without restore.
- The `--help` probe gets 30 s. Its "yes" is remembered per executable
  (path, size, mtime: an update invalidates it), so a restore probes
  once. A probe that timed out is retried, and silence is no longer
  reported as "this Codex is too old".

A tab opened on Codex and left empty saved an id that no restart could
reopen. The relay reported a thread at `thread/start`, but Codex writes
a conversation only once it has content, so `codex resume` answered
"no rollout found" on every restore. Nothing released it either: the
pane never reported, and silence keeps a restored id.
- A started thread is now reported at its first content: `turn/start`,
  `review/start`, `thread/shellCommand` or `thread/inject_items`.
  Resumed and forked threads are reported at once, as before.
- When the conversation a launch was asked to resume gets that exact
  answer, the launcher sends `pane.codex_session_missing`, and the next
  save drops that id, and only that id, from that tab. A later report in
  the tab wins over it.

The saved id is not checked against rollout files: Codex 0.163 migrated
history into SQLite (`legacy_to_paginated_v1`), so most valid threads
have no rollout file left. The missing-thread signal comes from Codex
itself, verified against codex 0.163 app-server: a migrated thread
resumes; a never-written one answers -32600 "no rollout found for
thread id <id>".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@amirlehmam
amirlehmam force-pushed the fix/codex-resume-reliability branch from ad1a9c0 to 68b1b1c Compare October 19, 2026 16:33
@amirlehmam
amirlehmam merged commit a675183 into amirlehmam:master Oct 19, 2026
@amirlehmam

Copy link
Copy Markdown
Owner

Merged, thank you. Retrying the pipe through a restore burst, caching the --help probe, and reporting a thread only once it has content are all well-argued, and using Codex's own 'no rollout found' answer instead of a file prune was the right call after the SQLite migration. This conflicted with #307 in stampAgentSessionIds, so I rebased it and kept both changes (dropMissingCodexIds on the Codex step, agy last). The combined tree passes the full suite. Ships in 2.30.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants