feat(minvmd): add boot command and VMM child subcommand - #339

sequenceDiagram
  participant BootParent as minvmd boot
  participant MarkerSocket as Host UNIX socket
  participant Child as __krun-vmm
  participant Guest as VM guest
  BootParent->>MarkerSocket: bind marker socket
  BootParent->>Child: spawn with MINVMD_MARKER_SOCK env
  Child->>Child: start_enter() (boot VM)
  Guest->>Child: send READY\n on vsock:9799
  Child->>MarkerSocket: vsock 9799 forwarded -> write READY\n
  MarkerSocket-->>BootParent: accept() reads READY
  BootParent->>BootParent: write vmm.pid to state dir
  alt foreground set
    BootParent->>Child: wait for child exit
  else timeout or error
    BootParent->>Child: kill child
    BootParent-->>BootParent: return error
  end

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

/revise

Goal: make the R2.4 boot proof a covered CI check so sdd-validate's needs-human Blocker clears the spec-faithful way (proof artifact executed by a required check, not waived). The proof can't run today because no MINVMD_KERNEL_PATH is provisioned. Fix that by fetching the kernel from the minimal build cache in CI.

Verified mechanism (use exactly this — do not try to build the kernel):

1. The kernel is the virtio-linux package's vmlinuz, already prebuilt in the public GCS build cache. No kernel compile, no Linux VM needed — just a cache pull.

   • The package lives in upstream gominimal/pkgs (packages/virtio-linux/build.ncl) and emits usr/share/virtio-linux/vmlinuz.
   • The remote cache bucket is minimal-staging-cache (the default, DEFAULT_REMOTE_CACHE_BUCKET) and is public-read — rcache pulls it over anonymous HTTPS (RemoteCache::new_over_https). No GCS credentials required.

2. Fetch the prebuilt linux minimal CLI (it's Linux-only; don't compile it). It's published to gs://minimal-shim:

   • curl -sf https://storage.googleapis.com/minimal-shim/config/cli-arm64-linux.json → {"version":"<sha>"}
   • curl -sf https://storage.googleapis.com/minimal-shim/archives/cli-arm64-linux-<sha>.tar.zst → tar --zstd -x → bin/minimal (static aarch64 ELF). This is the pattern in gominimal/minimal-vm-mac:image/fetch-minimal.sh.

3. Extract the kernel with minimal materialize (in the minimal crate — cmd_materialize, OutputKind::RawFile). In a scratch dir create a minimal.toml whose [upstream] points at gominimal/pkgs (copy [upstream] from this repo's .minimal/minimal.toml, same locked_commit) plus:

   [outputs.virtio-kernel]
   type = "raw-file"
   packages = ["virtio-linux"]
   path = "usr/share/virtio-linux/vmlinuz"

   then minimal materialize --output <dest>/vmlinuz --output-name virtio-kernel --arch aarch64. This pulls virtio-linux from the cache (no build) and writes the vmlinuz.

CI wiring (.github/workflows/ci-macos.yml): two-job design so the self-hosted Mac runner needs no Docker/gcloud:

• New linux job (ubuntu-latest): steps 2–3 above → actions/upload-artifact the vmlinuz.
• The existing macOS job: actions/download-artifact the vmlinuz, stage the rootfs via scripts/fetch-alpine.sh, then add a step:

  MINVMD_E2E=1 MINVMD_KERNEL_PATH=<vmlinuz> MINVMD_ROOTFS_PATH=<rootfs> \
    cargo test -p minvmd --test boot_e2e -- --include-ignored
  

  (Mirror the existing krun_smoke E2E step at ci-macos.yml:79; the runner already has libkrun per the "Verify libkrun" step.)

Notes:

• aarch64 only for now (self-hosted runner is Apple Silicon); the kernel artifact for aarch64 is Image.gz/vmlinuz from virtio-linux.
• You run in a firewalled sandbox (index.crates.io blocked) so you cannot execute this workflow yourself — the macOS CI run on the PR is the verification. Make the YAML correct by construction.
• Keep the diff to ci-macos.yml (+ a small fetch script if helpful under scripts/). Do not touch crates/minvmd/** source.

Once boot_e2e runs as a required macOS check, the implementation-boundary proof-artifact gate is covered and needs-human can clear.

Commit pushed: e101111

Generated by sdd-execute (sonnet tier) · ● 17.2M

Commit pushed: fe4b131

Generated by sdd-execute (sonnet tier) · ● 13.7M

Boot proof manually executed — it FAILS (R2.4 not demonstrably met)

I fetched a real kernel and ran boot_e2e against this branch. The kernel pull works; the boot does not.

Kernel fetch (works): minimal materialize --output vmlinuz --arch aarch64 virtio-kernel against the public minimal-staging-cache pulls a 14 MB Image.gz (gzip magic 1f 8b, matches KRUN_KERNEL_FORMAT_IMAGE_GZ) — no auth, no build.

Boot (fails):

Error: krun_start_enter ... failed: Invalid argument (os error 22)
Error: boot timed out waiting for READY marker after 5 s
boot_e2e: did not receive 'vm-up' within 10 s; check that the guest rootfs writes READY\n to vsock port 9799

Ran with: MINVMD_E2E=1 MINVMD_KERNEL_PATH=<materialized vmlinuz> MINVMD_ROOTFS_PATH=<alpine-3.21.7-aarch64> cargo test -p minvmd --test boot_e2e -- --include-ignored on macOS arm64, libkrun 1.18.1, binary codesigned with minvmd.entitlements.

Root causes

1. vsock READY port mismatch. minvmd registers/awaits the marker on VSOCK_MARKER_PORT = 9799 (crates/minvmd/src/cmd/mod.rs:11), but the staged rootfs's etc/minvmd/manifest documents the guest emitting READY\n on vsock_port_ready=7350. Host and guest disagree → no marker can ever arrive.
2. No guest READY-writer. The staged Alpine minirootfs has sbin/init -> /bin/busybox; there is no init that connects to the marker vsock and writes READY\n. The R3.4 v0.1 vsock stub responder is not present in the rootfs.
3. krun_start_enter returns EINVAL (os error 22) — the VM does not start. vm.rs:44 calls set_kernel(path, kernel_format(), None, None) — no kernel cmdline is passed; a Linux boot generally needs one. (EINVAL may also be partly environmental — libkrun version / rootfs-as-dir — so confirm on the self-hosted runner, but the port and init mismatches are code/contract facts.)

Implication

Wiring boot_e2e into CI would make it a red required check, not a cleared gate — the proof genuinely does not pass. sdd-validate's needs-human is justified by an actual failure, not just missing infra. The host/guest vsock port contract (9799 vs 7350) and the missing guest stub need fixing before R2.4 can pass.

Why the earlier /revise couldn't add CI coverage

The /revise asked the agent to add the kernel-fetch + boot_e2e step to ci-macos.yml. The agent cannot edit .github/ — its safe-output config runs with protect_top_level_dot_folders: true, which strips any dot-folder change. It made an unrelated in-scope source edit instead. Filed upstream: norrietaylor/spectacles#220.

The kernel-fetch CI lane is ready to add (the materialize command above is verified) once the boot bugs are fixed.

sdd-validate · Implementation boundary

Boundary: Implementation (all changed files are implementation code, no spec or architecture files).
Gate set: Implementation gates.

Finding 1 · Blocker · Proof artifacts not covered by any gate

Gate: Proof artifacts re-executed and passing.

Both proof artifacts require macOS with libkrun, a kernel image, and a rootfs with a guest-side vsock READY writer — an infrastructure limit that prevents re-execution in this (Linux) agent container.

• CLI proof: MINVMD_KERNEL_PATH=... MINVMD_ROOTFS_PATH=... minvmd boot --foreground — requires macOS + libkrun + provisioned kernel/rootfs.
• Test proof: MINVMD_E2E=1 cargo test -p minvmd --test boot_e2e -- --include-ignored — requires macOS + libkrun + provisioned kernel/rootfs.

Consumer CI coverage check:

• ci.yml (Linux): runs cargo test -- --include-ignored, but boot_e2e.rs is #![cfg(target_os = "macos")] — excluded from compilation entirely.
• ci-macos.yml (macOS): runs cargo test -p minvmd (no --include-ignored) and cargo clippy -p minvmd --all-targets. The #[ignore] attribute on boot_e2e_ready_marker_round_trip means the test is skipped. The workflow does not set MINVMD_E2E=1 for the boot_e2e test.

Neither consumer CI workflow executes the declared proof artifacts. The proofs are verified by no gate — neither agent re-execution nor consumer CI — so this is a Blocker per the implementation gate rules.

Evidence: crates/minvmd/tests/boot_e2e.rs:27 (#[ignore]), .github/workflows/ci-macos.yml:79 (cargo test -p minvmd without --include-ignored or MINVMD_E2E=1).

Finding 2 · Warning · Changed file outside task scope

Gate: Changed files within task scope.

crates/minvmd/Cargo.toml is modified (adding serial_test dev-dependency) but is not listed in the task's files in scope: block (issue #326). This is not a protected path, so it is a Warning rather than a Blocker.

Evidence: PR diff crates/minvmd/Cargo.toml, task #326 files in scope: block.

Finding 3 · Info · No credentials in diff

Gate: No real credentials in the diff.

The diff introduces no secrets, tokens, keys, or credentials. Clean.

Result: 1 Blocker, 1 Warning, 1 Info. Applying needs-human for the Blocker.

Generated by sdd-validate for PR #339

Generated by sdd-validate for issue #339 · ● 17.7M · ◷

Resequenced the boot proof (rationale: norrietaylor/spectacles#221). This task (#326) no longer owns the full READY round-trip — that proof moved to #328, where the R3.4 guest vsock stub that emits READY is built. #326's proof is now host-side boot-to-userspace only (self-contained). The marker-port mismatch (host 9799 vs guest rootfs 7350) is to be reconciled in #328. Kernel-fetch + non-gating boot_e2e CI lane: #341.

Commit pushed: 792665a

Generated by sdd-execute (sonnet tier) · ● 15.1M

Correction to the boot diagnosis

My earlier "needs an ext4 disk image" claim was wrong — thanks for the push-back. The spec's directory rootfs (R2.2 fetch-alpine.sh → extracted Alpine minirootfs, consumed via krun_set_root) is the right design: libkrun.h shows krun_set_root provides a virtiofs root (KRUN_FS_ROOT_TAG "/dev/root"). So krun_set_kernel (external kernel) + krun_set_root (virtiofs directory) is a valid pairing. No disk image required.

What's confirmed correct

• Kernel format: KRUN_KERNEL_FORMAT_IMAGE_GZ (=4) is right — the materialized vmlinuz is gzip → a valid aarch64 Linux Image (MZ EFI stub + ARMd magic at 0x38).
• Root mechanism: set_root(<dir>) virtiofs, per KRUN_FS_ROOT_TAG.
• Port: reconciled 9799 → 7350 (8fbc32d5); still needed.

Leading hypothesis for the krun_start_enter EINVAL

The runner links the plain libkrun (1.18.1) with libkrunfw 5.3.0 — the bundled-kernel build. External-kernel boots (krun_set_kernel) are the domain of libkrun-efi (what krunkit, also installed, uses). A bundled-kernel libkrun rejecting an externally-set kernel at krun_start_enter is consistent with the EINVAL. The FFI smoke (krun_smoke, set_exec process-model) works on this same build, which fits.

This is a hypothesis, not yet confirmed — verifying needs either libkrun debug logging on the runner or linking the EFI/external-kernel libkrun variant. If it holds, #326's boot needs to link the libkrun build that supports krun_set_kernel (or boot via the bundled kernel), which is a real integration decision, not a code one-liner.

Net: the kernel-fetch lane (#341) and the port fix stand; the EINVAL is a libkrun-variant question, not a rootfs-format or kernel-format bug.

Boot root cause — kernel-format incompatibility (hardware-tested)

Diagnosed the krun_start_enter EINVAL on real hardware: macOS arm64, libkrun 1.18.1 + libkrunfw 5.3.0, a real virtio-linux kernel (materialized from the build cache) + Alpine aarch64 rootfs, binary codesigned with the hypervisor entitlement.

Root cause: the kernel format. crates/minvmd/src/vm.rs sets KRUN_KERNEL_FORMAT_IMAGE_GZ (4); the installed libkrun rejects it at config validation (EINVAL — before any boot).

Format matrix:

Ruled out (tested, no effect on the EINVAL):

• Full kernel cmdline (reboot=k panic=-1 … console=hvc0 rootfstype=virtiofs rw … init=/init.krun). libkrun's set_kernel cmdline replaces its default prolog, so passing None (current) or a partial string is wrong — but supplying the full prolog did not clear the EINVAL.
• Adding set_exec (an entrypoint for init=/init.krun).
• Disk-image vs directory rootfs — set_root (virtiofs, KRUN_FS_ROOT_TAG) is correct; no disk image needed. (Retracts my earlier comment on this PR.)

Conclusion. The installed libkrun and the 38 MiB virtio-linux Image.gz do not agree: the correct arm64 format (IMAGE_GZ) is rejected by this build, and the formats it accepts (RAW/PE_GZ) cannot fit a full Image in their load window. This is an environment/packaging issue, not a minvmd one-liner.

Fix options:

• (A) Link/use a libkrun build+version that accepts IMAGE_GZ (the standard arm64 Image format), or
• (B) Package the virtio-linux kernel so it RAW-loads within libkrun's window.

Supersedes my earlier libkrun-variant hypothesis on this PR. The 9799 → 7350 marker-port reconcile is unaffected and stands. The kernel-fetch CI lane (#341) is independent and unaffected.

Superseded by #344, which carries this branch's host plumbing plus the kernel-format fix (aarch64 PE_GZ), krun_set_exec wiring, the corrected guest READY direction, the rootfs/kernel image scripts, and the boot-E2E CI lane. #344 closes #326.