slurm: fall back to sacct when scontrol cannot answer - #10

The scontrol lookup at the top of wait_final_status is not guarded, and
run_command raises RuntimeError on any non-zero exit. scontrol only knows
jobs the controller still holds, so once a finished job is older than the
site's MinJobAge the poll raises instead of reading the answer from the
sacct call sitting directly beneath it. That sacct loop was the whole of
wait_final_status before fcf92e0 put scontrol in front of it; it is now
unreachable in exactly the case it exists for.

On NCSA DeltaAI (Slurm 25.11.1, MinJobAge = 300 s) a job that has aged out
of the controller gives:

    $ scontrol show job -o 3159304
    slurm_load_jobs error: Invalid job id specified        (exit 1)
    $ sacct -j 3159304 --parsable2 --noheader \
        --format=JobID,State,ExitCode,Elapsed,AllocCPUS,NodeList
    3159304|COMPLETED|0:0|00:03:40|64|gh007

Any poll interval or stall longer than MinJobAge therefore turns a
COMPLETED job into a failed flow. By inference from the code rather than
from a run here, the same happens on the very first poll wherever scontrol
is restricted or not installed, since create_subprocess_exec then raises
FileNotFoundError.

Asking the controller first is right: it answers while the job is live and
does not depend on accounting being configured. So this keeps scontrol and
makes it an optimisation that degrades. A failed scontrol is logged at
DEBUG and leaves stdout empty, and the loop reads sacct as before. A failed
sacct still raises, because with neither source there is nothing to report.

What that costs is written into the docstring rather than fixed here. If
scontrol fails and sacct exits 0 with no row for the job -- accounting
down, or the record purged -- the poll has no answer from either source and
the wait then ends only at timeout_seconds, which run_slurm_job leaves at
None. Today that combination ends the wait at once, by failing it. Bounding
the wait properly is a separate change; this one states the behaviour in
the docstring and pins it with a test.

Two things in the same few lines go with it. The raw scontrol record was
logged at INFO behind a "DEBUG:" prefix -- da81614, merged as qiskit-community#36, had just
promoted it there from a print() -- which at the default ten-second
interval writes several hundred copies of the full job record into the run
log of a job that waits half an hour. It is now a real DEBUG record,
formatted lazily so the string is not built when DEBUG is off. The
neighbouring INFO line announcing the final state still uses an f-string;
leaving it alone keeps the diff to one idea.

And the terminal-state list was written out twice, once as
_is_terminal_state() and once inline in the loop. The two copies agree
today, so dropping the inline one changes no outcome on the scontrol path;
it removes a second list that has to be maintained in step with the first.
The helper is the copy worth keeping because the inline one compared for
equality, and sacct decorates a state: over jobs run here since 2026-09-14,
sacct --parsable2 reports "CANCELLED by 33444" on 15 rows, and its default
fixed-width output truncates the same value to "CANCELLED+". Neither
spelling can reach this code today -- the loop passes --parsable2, so
nothing is truncated, and scontrol -o does not decorate at all (in one
snapshot of this controller all 2,169 JobState= values were bare). So that
tolerance is insurance for the sacct branch rather than a captured case.
The helper also gained a guard for an empty state string rather than
raising IndexError on one.

The tests drive run_command through a fake that dispatches on the command
name and records the full argv: a raising scontrol still returns the
terminal status from sacct, a missing scontrol does too, a scontrol answer
short-circuits without calling sacct, each of the nine terminal states ends
the wait while PENDING, RUNNING, COMPLETING and SUSPENDED fall through to
sacct, a still-running job is polled twice, a job neither source knows
raises WaitTimeout, a failing sacct still raises, and the scontrol record
appears at DEBUG and nowhere above it. Nine of the thirty new cases fail
against this file's previous contents; the rest pin paths that already work
and that this change must not drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>