[Bug]: codex turn watchdogs have inconsistent defer guards and no wall-clock ceiling; attempt-idle aborts long quiet native tool calls #99272

Bug type

Behavior bug (incorrect output/state without crash)

Summary

The Codex turn watchdogs are inconsistent about when they defer: the completion and terminal watches defer while a native tool item or in-flight server request is open, but the attempt-idle watch does not, so a long quiet native tool call can be hard-aborted mid-execution. In the other direction, every current-turn notification (including methods the projector does not understand) resets the idle timers, and all four watchdogs are idle-based, so no wall-clock bound exists at the harness level. Blocker tracking also has an out-of-order hazard that can permanently suppress the completion watch.

Evidence

All in extensions/codex/src/app-server/, current main:

• Completion watch defers on open work: attempt-turn-watches.ts:121-128 (schedule) and :302-311 (fire) both bail when getActiveAppServerTurnRequests() > 0 or getActiveCompletionBlockerItemCount() > 0.
• Attempt-idle watch has no such guard: fireAttemptIdleTimeout (attempt-turn-watches.ts:260-300) checks only completed/aborted/armed, then calls params.onAbort("turn_progress_idle_timeout") at :300. A commandExecution item that produces no output deltas for longer than the attempt idle timeout gets the whole turn aborted even though the harness knows the item is still open (isCompletionBlockingItem, attempt-notifications.ts:87-101, lists commandExecution, mcpToolCall, dynamicToolCall, etc. as items completion must not time out under).
• Any current-turn notification counts as attempt progress: attempt-notification-state.ts:119-127 calls touchActivity(..., { attemptProgress: true }) for every non-delta current-turn notification, including methods the projector's dispatch does not know (event-projector.ts:310-311 default: break). A turn that keeps emitting anything never trips any watchdog.
• All four timers are idle-based, including the long stop-gap fuse: attempt-timeouts.ts:10-21 (CODEX_TURN_TERMINAL_IDLE_TIMEOUT_MS = 30 * 60_000 is idle time, not wall clock). There is no absolute per-turn ceiling anywhere in the harness; the only backstop is the core lane timeout, which releases the session lane out from under the attempt rather than ending the turn cleanly.
• Out-of-order blocker hazard: updateActiveCompletionBlockerItemIds (attempt-notifications.ts:66-85) deletes on item/completed and adds on item/started with no memory of already-completed ids. If item/completed arrives before its item/started (correlation is string equality with no sequence numbers, notification-correlation.ts:29-51), the late item/started adds the id and nothing ever removes it: a phantom blocker suppresses the completion and terminal watches for the rest of the turn.

Expected behavior

One shared guard set across all idle watches (open server requests and open blocker items defer every idle watch, including attempt-idle), complemented by an absolute wall-clock ceiling per turn so a genuinely hung native item still terminates. Only notification methods the projector actually understands count as progress. Blocker tracking remembers completed ids so ordering inversions cannot create phantom blockers.

Impact

Two opposite user-visible failure modes today: (1) legitimate long-running quiet native tool calls (builds, installs with buffered output) can be aborted mid-execution with turn_progress_idle_timeout; (2) a wedged turn that keeps emitting unrecognized notifications never times out at the harness level. Timeout behavior in this area has been a recurring fix theme (progress-aware timeout, terminal-idle watchdog, and idle-timer cap fixes all landed within the last two months).

Proposed fix

• Extract the completion watch's guard predicate and apply it to fireAttemptIdleTimeout (defer and reschedule while a blocker item or turn request is open).
• Add one absolute wall-clock turn ceiling (derived from the run timeout with a generous multiplier) that fires regardless of activity, as the complement that makes deferring safe.
• Track completed item ids in updateActiveCompletionBlockerItemIds so a late item/started for an already-completed item is ignored.
• Count only dispatch-known methods as attempt progress, or at minimum log unknown methods (see [Bug]: codex event projector fails open on unknown item statuses and silently drops unknown or mismatched notifications #99269).

Provenance

Found in a maintainer code audit of the Codex app-server harness. Line anchors verified by direct read on current main.