fix(ci): reclaim the session e2e's state dir so the soak stops filling the runner - #1021
Conversation
…g the runner
`scripts/session-e2e.sh` never removed its `$WORK` state dir. Its two sibling
harnesses do — `bulk-upload-e2e.sh` ("leave nothing behind: a canary that
strands a session per run is worse than no canary") and `stress-session-e2e.sh`
both `rm -rf "$WORK"` on teardown. This one was the outlier, and it is the one
`soak-session-e2e.sh` runs ten times back-to-back on a single runner.
The dir is not just metadata. On a VM lane it holds the provider's per-VM
writable data volume, `minimal/providers/local-minvmd0/data-vol.raw` — a sparse
image whose HOST allocation is everything the guest wrote into it: its package
cache, the session rootfs, the workspace. `$WORK` is minted fresh per run, so
nothing is shared between iterations and each one pays that allocation again,
ten times over, with nothing reclaiming any of it.
That matches how the nightly fails. `session-e2e-soak` has been red since the
2026-07-25 run (last green 07-24) and every failure has the same shape: the
soak step stuck at `in_progress`, the job reporting `failure`, zero retrievable
log lines, no uploaded artifacts, and the runner dead well inside the 90-minute
budget. That is the runner AGENT being killed, not a test assertion failing —
the signature of ENOSPC on the runner. The single-session KVM lane, which runs
the same e2e once, stays green.
Two changes:
- `session-e2e.sh` removes `$WORK` on teardown, and writes its `min bug`
diagnostic bundle to /tmp rather than into `$WORK` when no boot-log dir is
set, so the bundle survives that teardown (the fallback `bulk-upload-e2e.sh`
already uses, for the same reason). Every diagnostic `fail` collects is
gathered before teardown runs.
- `soak-session-e2e.sh` measures free space on the filesystem holding those
state dirs, prints it with a per-check delta on every iteration, and stops
with a real `::error::` if it falls under a floor (4096 MiB, overridable via
`SOAK_MIN_FREE_MIB`). A genuine ENOSPC leaves nothing to read; this fails the
step while a runner is still alive to say why, and the deltas make it
obvious if reclamation ever regresses again.
Verified: `scripts/lint-shell.sh` passes (24 scripts). Before/after against a
stubbed `min` that drives the harness through its `fail` -> `teardown` path:
HEAD leaves a `/tmp/mnl-e2e.*` state dir behind, the patched script leaves
none. The headroom check was exercised from an isolated copy — floor above
actual free space aborts before running any iteration (exit 1, zero iterations
run), floor of 0 runs all iterations plus the bulk proof and reports the
deltas, a non-numeric floor is rejected with exit 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
Comment |
The failure
session-e2e-soakhas been red onnightly-testssince the 2026-07-25 run (last green: 30073487997, 07-24). Every failure since has the identical shape:Soak the session e2e (x10))in_progresslog not foundin_progressin_progressin_progressA step stuck at
in_progresswhile the job reportsfailure, zero retrievable log lines, no uploaded artifacts, and the runner dead well inside its budget — that is the runner agent being killed, not a test assertion failing. It is the signature of ENOSPC on the runner. The single-sessionci-linux-kvmlane, which runs the same e2e once, stays green throughout.For scale: the last green soak ran all ten iterations plus the bulk proof in 3m25s. The failing runs die inside that same step.
The cause
scripts/session-e2e.shnever removed its$WORKstate dir.Its two sibling harnesses both do.
bulk-upload-e2e.shis explicit about why — "Leave nothing behind: a canary that strands a session (or a VM) per run is worse than no canary" — andstress-session-e2e.shdoes the same.session-e2e.shwas the outlier, and it is the onesoak-session-e2e.shruns ten times back-to-back on a single runner.That dir is not just metadata. On a VM lane it holds the provider's per-VM writable data volume,
minimal/providers/local-minvmd0/data-vol.raw— a sparse image (DEFAULT_VOLUME_BYTES= 256 GiB nominal) whose host allocation is everything the guest wrote into it: its package cache, the session rootfs, the workspace.$WORKis minted fresh per iteration, so nothing is shared between them; each iteration re-pays that allocation, ten times, with nothing reclaiming any of it.The fix
scripts/session-e2e.sh—rm -rf "$WORK"on teardown, matching the siblings. Themin bugdiagnostic bundle's fallback moves from$WORKto/tmpso it survives that teardown (the same fallback, for the same reason, thatbulk-upload-e2e.shalready uses). Every diagnosticfail()collects is gathered before teardown runs, so nothing is lost.scripts/soak-session-e2e.sh— measure free space on the filesystem holding those state dirs, print it with a per-check delta on every iteration, and stop with a real::error::if it falls under a floor (4096 MiB, override withSOAK_MIN_FREE_MIB).The floor is the part that matters beyond this one bug: a genuine ENOSPC leaves nothing to read, which is why five nights of failures produced no diagnosis. This fails the step while a runner is still alive to say why, and the per-iteration deltas make it immediately visible if reclamation ever regresses again.
No workflow files touched —
.github/workflows/is frozen, and both changes land in thescripts/extension point CI already schedules over.Verification
scripts/lint-shell.sh— 24 scripts pass shellcheck (the gatecrates/common/tests/shell_lint.rsruns in CI).mindrives the harness through itsfail()→teardown()path — the same path every soak iteration exits by:exit 1, zero iterations run), emitting the::error::exit 2The real proof is the next nightly (or a
workflow_dispatchofnightly-tests): the soak step should complete and report a flat per-iteration disk delta.Note: the diff is shell-only and touches no Rust, so the
just ciRust gates cover nothing here;just lint-shellis the gate for it and passes. I deliberately did not runjust cion this host —cargo clippy --all-targets -p minvmdwould relinktarget/debug/minvmdand drop the hypervisor codesign out from under the running dev stack.🤖 Generated with Claude Code
Note
Reclaim session e2e state directory on teardown to stop soak runner disk fill
$WORK), reclaiming provider-managed data volumes after each run.mnl bugnow default to/tmpinstead of the state directory whenMINVMD_BOOT_LOGis unset, avoiding writes under the path being deleted.check_freeguard before each iteration and before the bulk upload proof, aborting early with a GitHub Actions error if free space falls below a configurableSOAK_MIN_FREE_MIBfloor.free_mibhelper measures available disk space in MiB for a given path using portabledfandawk, with platform-specific filesystem selection.Macroscope summarized 0e060ac.