test(scripts): canary the bulk host-to-guest upload path - #887

What this covers

Failure class: bulk host→guest transfer over the vsock/SSH transport.

A guest kernel bump (6.12.43 → 6.12.94, gominimal/pkgs#311, 2026-07-01) crossed Linux 6.12.92, where a new queued-sk_buff-overhead check turned a previously silent vsock protocol violation into a fatal connection reset — gominimal/minimal#869. From that day every min activate carrying a real project failed:

error: Failed to upload project files: copying tar stream to channel: channel closed

with, guest side in boot.log:

session ended with error error=Protocol error: No buffer space available (os error 105)

Nothing went red for three weeks, until a human hit it by hand. Nothing we run moves enough data through the guest vsock to trip it — session-e2e.sh uploads a two-line minimal.toml.

scripts/bulk-upload-e2e.sh is that missing proof: min activate a project carrying a large, compressible fixture, five times over, failing if any attempt errors or hangs.

Why the fixture has to be compressible

The trigger is write rate, not volume, duration or file count — the client has to hand the channel a large number of already-compressed bytes as fast as it can produce them. From the measurements on the issue (three runs each):

So the obvious fixture — /dev/urandom — is exactly the wrong one: it hides the bug. And 0.11 MB on the wire is equally useless. The fixture therefore has to be large after compression and cheap to compress.

Rather than commit a 49 MiB binary, the script synthesizes one: each MiB is a fresh 256 KiB of xorshift32 output written four times, so zstd (level 3, 2 MiB window — what async_compression's ZstdEncoder defaults to) resolves three quarters of every MiB to matches while the literal quarter stays incompressible enough to be stored raw. Fixed seed, so it is byte-identical every run and a failure is reproducible.

Measured locally: 51,380,224 bytes in → 12,884,004 bytes out (3.99:1) — within 1% of the 12,784,144 bytes that reproduce on the issue — generated in ~0.35 s of perl.

That ratio is then asserted at run time (zstd -3 -T1, band 8–20 MB): if a future change makes the fixture incompressible, or trivially compressible, the script fails loudly instead of silently disarming itself. The check is skipped with a warning where zstd is absent (it is on the nightly's ubuntu-latest; the justfile already lists it as a macOS prereq).

Two related traps are closed the same way: the issue measured 4/4 passes under MINVMD_KRUN_LOG=debug, and client-side RUST_LOG=debug is enough on its own to make the bug vanish. The script therefore pins RUST_LOG=warn and unsets MINVMD_KRUN_LOG (printing a note when it drops an inherited value) — an inherited debug level would otherwise leave the canary unable to fail.

Iteration count: 5

The failure is probabilistic (~89%, not 100%), so one activate is not a detector.

At the measured p = 0.89, three would already be enough (99.87%). Five is chosen for margin against CI hardware that fails to reproduce as reliably: even if the trigger rate halved to p = 0.5, five iterations still detect 96.9% of the time. The cost is five min activates against one already-booted VM (the daemon is warmed outside the loop, so a VM boot is not billed to any iteration's deadline).

Any single failure across the set fails the script ($fail -eq 0 && $pass -eq $ITER — the same shape soak-session-e2e.sh uses, which also refuses to call a short-circuited loop a pass).

Timeout handling (the hang case)

gominimal/inbox#335: the same defect sometimes hangs the client indefinitely instead of erroring (observed at 9 and 13 minutes before being killed). A canary that waits forever is a canary that reports nothing, so:

• every min activate runs under a wall-clock deadline (default 420 s, BULK_ACTIVATE_TIMEOUT_SECS), SIGTERM then SIGKILL, returning 124;
• 124 is reported as a distinct, named failure (the #869 hang), not a generic error;
• a hang stops the loop — five deadlines back to back would burn the nightly's whole job budget, and one hang is already conclusive. Plain errors are cheap, so those keep going: the pass/fail tally is what identifies the ~89% signature.

timeout(1) is deliberately not used: it does not exist on a stock macOS, and running the VM stack under it wedges the boot (libkrun tcsetattr()s from timeout's background process group and SIGTTOU stops the group). The hand-rolled watchdog runs the client with </dev/null for the same reason — which also puts min activate in its non-interactive mode, where it neither prompts for the non-VCS upload confirmation (#770) nor waits on a keypress for policy gating.

How it hooks into the nightly canary

No workflow change is required, and none is made (.github/workflows/ is frozen and CODEOWNER-gated).

The nightly test tier does not discover scripts by convention: nightly-tests.yml's session-e2e-soak job invokes one entry point explicitly, ./scripts/soak-session-e2e.sh 10 "$RUNNER_TEMP/soak-logs". That script is reviewed code, so the hook goes there: after its session-e2e reps it reaps, runs the bulk-upload proof once (with MINVMD_BOOT_LOG pointed into the same log dir that is already uploaded as an artifact), and folds the result into its tally and exit status. It inherits the job's whole VM setup — images, gvproxy, PATH — for free.

just bulk-upload [n] runs the same proof locally, and just soak keeps nightly parity.

One thing a CODEOWNER may need to follow up on: the proof now shares the session-e2e-soak job's timeout-minutes: 60 with the ten session-e2e reps. It should add roughly 5–10 minutes (one VM boot plus five activates against a warm cache), but I could not measure it. If that job starts timing out, the fix is a workflow edit I cannot make — either bump timeout-minutes, or split it into its own job:

      - name: Bulk host→guest upload proof
        env: # same MINVMD_*/E2E_* block as the soak step
        run: ./scripts/bulk-upload-e2e.sh 5

Either way the failure is legible, and notify already files a tracking issue when session-e2e-soak fails.

Cleanup

A canary that leaks 89 sessions a week is worse than no canary:

• each iteration destroys its session immediately on success;
• a failed iteration sweeps with min destroy --all --force before the next pass;
• the EXIT/INT/TERM trap runs destroy --all --force, stop --force, minvmd stop on VM targets, and removes the fixture, the project dir and the private state dir;
• the script runs in its own XDG_STATE_HOME, so the sweep can never touch a developer's real sessions.

Verification

Verified. The script's own logic was exercised end to end against a stub min on PATH (the real VM was never started — see below):

Not verified — needs a reviewer or CI to run it. Another agent is holding the only VM on this machine for a timing-sensitive measurement, and a second VM would collide on the provider directory, so I could not run this against a real stack. Specifically unproven:

1. The end-to-end run itself — that a real min activate of the 49 MiB project succeeds on a good stack (and, on a pre-fix stack, fails). Reviewer check: just bulk-upload on a KVM or macOS host. On a stack still carrying #869 it should fail ~89% of iterations with the channel closed line.
2. scripts/soak-session-e2e.sh's new phase — syntax-checked only; running it invokes scripts/reap-vms.sh, which would have killed the VM I was told not to touch.
3. Runtime cost inside the nightly's 60-minute budget (see the CODEOWNER note above).
4. perl and zstd behaviour on the runner image — verified on macOS/arm64 only. The generator uses 32-bit arithmetic precisely so it does not depend on perl's integer width, and session-e2e.sh already depends on perl.

Refs: #869, gominimal/inbox#335

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added a bulk-upload end-to-end verification task with a configurable repetition count (default 5).
  • Updated soak workflows to include a bulk host→guest upload proof after session testing.
• Bug Fixes
  • Soak now requires both session checks and bulk-upload verification to pass.
  • Improved timeout handling and failure diagnostics for repeated checks.
• Documentation
  • Updated task documentation to clarify the nightly soak flow and the added bulk upload step.