Skip to content

fix(scripts): make bench-minvmd-boot safe beside other checkouts, and n>1 actually work - #1023

Merged
norrietaylor merged 2 commits into
mainfrom
fix/bench-boot-scoped-teardown
Jul 29, 2026
Merged

fix(scripts): make bench-minvmd-boot safe beside other checkouts, and n>1 actually work#1023
norrietaylor merged 2 commits into
mainfrom
fix/bench-boot-scoped-teardown

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 29, 2026

Copy link
Copy Markdown
Member

Three bugs in scripts/bench-minvmd-boot.sh, all found while using it to A/B a new guest rootfs. Two made the bench actively destructive or useless; the third made it unrunnable on macOS.

1. Teardown killed other checkouts' VMs

teardown() { pkill -9 -f "__krun-vmm" 2>/dev/null; … }

That pattern matches every checkout's VM. Running the bench beside a parallel working copy SIGKILLs that copy's live dev stack — sessions and all — between each of N runs, and once more on exit.

It matched a sibling checkout's 44-minute-old VM:

$ pgrep -f "__krun-vmm"
30474 /Users/norrie/code/minimal/target/debug/minvmd __krun-vmm   ← different checkout

scripts/reap-vms.sh already documents the rule this violates:

Matching is scoped to THIS checkout's binaries […] a bare-name pkill would only add collateral — an unrelated checkout's live VM, or podman's gvproxy.

So the fix applies that existing discipline rather than inventing one. Both the supervisor and the __krun-vmm grandchild re-exec via current_exe(), so both carry the binary's absolute path in their cmdline. $BIN is resolved to an absolute path first, because it is conventionally passed relative (./target/debug/minvmd) and that would match nothing.

pattern matches
__krun-vmm (before) the sibling's VM — pid 30474
/…/minimal2/target/debug/minvmd (after) 0 processes

2. Only the first run was a real sample

Every run after the first reported FAILED (no vm-up).

teardown SIGKILLs the VM, which leaves the bound ssh.sock behind in the state dir. libkrun registers that path with listen=true on the next boot, so it hangs in krun_start_enter until the 12 s timeout. With one state dir shared across runs, n=10 really meant n=1.

Each boot now gets a fresh state dir. That is also the right fidelity — this is a cold-boot benchmark, and a reused state dir carries the previous run's data volume, which is not the state a cold boot starts from.

It additionally isolates the bench from any other checkout: minvmd refuses to boot when the default state dir has a VM registered, and that dir is shared across checkouts even though the binaries are not. A sibling dev stack alone was enough to fail every run with minvmd is already running.

3. mktemp -d -t blew the socket path limit

-t honours TMPDIR, which on macOS is a ~50-byte /var/folders/... path. The daemon socket beneath it then exceeds the 103-byte sun_path limit and the boot dies before it starts:

socket path /var/folders/…/providers/local-minvmd0/ssh.sock is 118 bytes, exceeds the 103-byte limit

A short /tmp template avoids it, matching what scripts/session-e2e.sh already does.

Verification

shellcheck clean at 0.11.0 and at CI's 0.10.0.

Exercised end-to-end on the job it was written for — boot-to-READY across two guest rootfs images, 10 runs each, with a sibling checkout's stack up throughout:

rootfs min median max
186 MB (current pin) 146 ms 154 ms 161 ms
45 MB (gominimal/pkgs#534) 122 ms 129 ms 147 ms

All 20 samples valid, distributions non-overlapping, and the sibling checkout was still running afterwards. Before this PR that comparison was not obtainable: run 1 of each set was the only sample, and the bench would have killed the sibling stack 22 times.

🤖 Generated with Claude Code

Note

Fix bench-minvmd-boot.sh teardown to target only the current binary and isolate per-run state

  • Adds BIN_ABS (absolute path to the minvmd binary) so teardown uses pkill -9 -f "$BIN_ABS" instead of broad pattern matches, avoiding killing processes from other checkouts.
  • Adds a SCRATCH directory under /tmp (short path) as the parent for per-run minimal state dirs, avoiding socket path length issues.
  • Introduces a boot_once function that creates a fresh --minimal-state-dir per run and checks for exact vm-up output; warmup and timed loops both use it.
  • Extends the EXIT trap to rm -rf "$SCRATCH" on exit.
  • Behavioral Change: each boot now runs against a fresh isolated state directory rather than shared state, which changes timing characteristics slightly.

Macroscope summarized d9fc641.

`teardown()` ran `pkill -9 -f "__krun-vmm"`, which matches EVERY
checkout's VM, not this one's. Running the bench beside a parallel
working copy SIGKILLs that copy's live dev stack — sessions and all —
between every one of N runs, and again on exit. Hit while benchmarking a
new rootfs with a second checkout's stack up: the pattern matched its
44-minute-old VM.

scripts/reap-vms.sh already documents the rule this broke — "a bare-name
pkill would only add collateral: an unrelated checkout's live VM" — and
scopes by absolute repo path. This applies the same discipline. Both the
supervisor and the __krun-vmm grandchild re-exec via current_exe(), so
both carry the binary's absolute path; $BIN is resolved to an absolute
path first because it is conventionally passed relative
(`./target/debug/minvmd`), which would match nothing.

Verified against a live sibling checkout: the old pattern matches its
__krun-vmm, the new one matches zero processes.

Also boots into a private --minimal-state-dir. The state dir is shared
across checkouts even though the binaries are not, so a sibling stack is
enough to make every run fail with "minvmd is already running" — the
script could not run at all in the situation the teardown fix is about.
It also keeps the bench off that stack's socket, pidfile, and logs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 14 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7970fca8-b4c3-4304-911e-5439ba6c2feb

📥 Commits

Reviewing files that changed from the base of the PR and between df22d71 and d9fc641.

📒 Files selected for processing (1)
  • scripts/bench-minvmd-boot.sh

Comment @coderabbitai help to get the list of available commands.

Every run after the first silently reported FAILED (no vm-up).

`teardown` SIGKILLs the VM, which leaves the bound `ssh.sock` behind in
the state dir. libkrun registers that path with listen=true on the next
boot, so it hangs in `krun_start_enter` until the 12 s timeout — a
single shared state dir across runs makes run 1 the only real sample.

Fresh per run is also the right fidelity: this is a COLD-boot
benchmark, and a reused state dir carries the previous run's data
volume, which is not the state a cold boot starts from.

Also fixes the state dir's template. `mktemp -d -t` honours TMPDIR, and
on macOS that yields a ~50-byte /var/folders/... path; the daemon socket
beneath it (`providers/local-minvmd0/ssh.sock`) then blows the 103-byte
sun_path limit and the boot dies before it starts:

    socket path /var/folders/.../providers/local-minvmd0/ssh.sock
    is 118 bytes, exceeds the 103-byte limit

A short /tmp template avoids it, matching scripts/session-e2e.sh.

The three boot invocations collapse into one `boot_once` helper rather
than repeating the mkdir in each.

Verified by the A/B this was written for: 2 rootfs images x 10 runs, all
20 samples valid, sibling checkout's stack still up afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@norrietaylor norrietaylor changed the title fix(scripts): scope bench-minvmd-boot teardown to this checkout fix(scripts): make bench-minvmd-boot safe beside other checkouts, and n>1 actually work Jul 29, 2026
@norrietaylor
norrietaylor enabled auto-merge (squash) July 29, 2026 19:28
@norrietaylor
norrietaylor merged commit a6b8191 into main Jul 29, 2026
29 checks passed
@norrietaylor
norrietaylor deleted the fix/bench-boot-scoped-teardown branch July 29, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants