feat(minvmd): produce a viable non-EFI VM image and reach READY (closes #326) - #344
Conversation
Implements R2.3 and R2.4 from the minvmd spec: - cmd/mod.rs: declares the cmd module, VSOCK_MARKER_PORT (9799) and MARKER_SOCK_ENV constants shared between parent and child. - cmd/boot.rs: the `minvmd boot [--foreground]` subcommand. On macOS: validates MINVMD_KERNEL_PATH and MINVMD_ROOTFS_PATH, creates a UNIX socket listener for the READY marker, fork-execs `minvmd __krun-vmm` with MINVMD_MARKER_SOCK set to the socket path, writes the child PID to vmm.pid, then waits up to 5 s for the guest to connect and write READY\n (R2.4). On success prints vm-up. With --foreground, blocks until the VMM child exits. On Linux: bails immediately (no-op stub). - cmd/vmm_child.rs: the hidden `minvmd __krun-vmm` subcommand. On macOS: creates a libkrun context, applies VmConfig (kernel, rootfs, 2 vcpus, 512 MiB), registers VSOCK_MARKER_PORT pointing to the host UNIX socket, then calls krun_start_enter (R2.3). On Linux: bails immediately (no-op stub). - main.rs: wires Boot and KrunVmm subcommands to the CLI. - lib.rs: exports pub mod cmd. - tests/boot_e2e.rs: READY-marker round-trip E2E test (gated on MINVMD_E2E=1 and #[ignore], macOS only) (R2.4). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On the error and timeout paths in boot.rs, terminate and wait the VMM child and remove both vmm.pid and the marker socket before returning, so a failed boot does not leave stale state behind. In boot_e2e.rs, set XDG_STATE_HOME to an isolated tempdir for the spawned child so the E2E test does not clobber the developer's real state directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add #[serial] to boot_e2e_ready_marker_round_trip so it cannot race other stateful E2E tests that mutate XDG_STATE_HOME. The test already isolates its state directory via XDG_STATE_HOME=<tempdir>, so no additional isolation is needed; serialization ensures only one such test runs at a time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use PID + 4 random bytes from /dev/urandom for the READY-marker socket path instead of PID alone, making the path unpredictable and closing the TOCTOU race a local attacker could exploit to inject a spoofed READY. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The host awaited the boot READY marker on vsock 9799, but no guest emits on that port: the guest rootfs manifest documents 7350 (etc/minvmd/manifest: vsock_port_ready=7350). With 9799 the marker never arrived. Align the host to 7350. Necessary but not sufficient: krun_start_enter still returns EINVAL; root cause under investigation (see PR comment) — not a kernel-format or rootfs-format issue. Refs: #221 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The aarch64 libkrun loader implements only RAW and PE_GZ; IMAGE_GZ (=4) is x86_64-only and returns KernelFormatUnsupported, so krun_set_kernel failed before the VM could boot. Select KRUN_KERNEL_FORMAT_PE_GZ (=2) for the aarch64 Image.gz (the loader scans for the gzip magic and decompresses) and add the constant to the FFI surface. Verified against /opt/homebrew/include/libkrun.h (libkrun 1.18.1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VMM child configured a kernel + rootfs but never set a workload, so libkrun's /init.krun (pid-1) fell back to /bin/sh and never emitted the READY marker. Set the guest workload via krun_set_exec (default /sbin/minvmd-stub-init, MINVMD_EXEC overrides) with an explicit minimal envp — passing None would inherit the full host env and can overflow the ~2 KiB aarch64 kernel cmdline. Add opt-in early-boot console capture (MINVMD_BOOT_LOG) for diagnosing a stuck boot, and raise the VM from 512 MiB to 2 vCPU / 1024 MiB for cheap headroom under Hypervisor.framework. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage the non-EFI VM image production on main: - fetch-alpine.sh: pinned, sha256-verified Alpine 3.21.7 minirootfs. - build-rootfs.sh: overlay socat + the /sbin/minvmd-stub-init workload and the guest manifest onto the rootfs directory (consumed by krun_set_root as virtio-fs; no disk image). - fetch-virtio-kernel.sh: pull the prebuilt virtio-linux vmlinuz (Image.gz) from the public minimal build cache (carried from #341). The stub's READY writer connects OUT to the host (CID 2, port 7350) rather than listening: the host registers the marker with the plain krun_add_vsock_port (== listen=false), so the direction is guest->host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage the guest rootfs via fetch-alpine.sh + build-rootfs.sh (was a vanilla extract with no READY writer), capture the guest console as an artifact for debugging, and scope the workflow to scripts/**. The boot E2E stays non-gating (continue-on-error) until first-green on the self-hosted runner, which also validates the virtio-linux kernel config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Align the spec with libkrun's verified behaviour: aarch64 loads Image.gz via PE_GZ (not IMAGE_GZ); libkrun's /init.krun is pid-1 and execs the krun_set_exec workload (no init system in the rootfs); the kernel cmdline stays unset; boot uses 2 vCPU / 1024 MiB; and the READY marker is guest-initiated (krun_add_vsock_port, listen=false), the opposite of the R3 ssh.sock bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 28 minutes and 47 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughImplements macOS boot orchestration and hidden VMM child, adds scripts to fetch/build guest rootfs and virtio kernel, introduces arch-specific kernel formats and tests, adds a macOS boot e2e test and CI jobs, and updates spec/architecture docs. Changesminvmd macOS boot pipeline
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
The self-hosted runner's `/usr/bin/env bash` is the macOS system bash 3.2, which lacks associative arrays; `declare -A` made `[aarch64]=` parse as an unbound arithmetic index under `set -u`, failing build-rootfs.sh with "aarch64: unbound variable". Replace the per-arch sha256 arrays with plain vars + a case lookup. Verified under /bin/bash 3.2.57. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/minvmd/src/cmd/boot.rs`:
- Around line 54-59: The current nonce construction silently ignores errors and
yields 0 when std::fs::File::open("/dev/urandom") or read_exact fails; change it
to handle errors explicitly by checking the Result from File::open and
read_exact, and return or propagate an error (or panic with a clear message)
instead of silently using a zero nonce. Locate the nonce block in cmd::boot.rs
(the variable named nonce and the calls to std::fs::File::open and read_exact),
unwrap or map_err with context and propagate the failure from the surrounding
function (or call expect with a descriptive message) so a failed read from
/dev/urandom does not produce a predictable nonce.
In `@scripts/build-rootfs.sh`:
- Around line 40-43: The associative array SOCAT_SHA256 uses unquoted keys like
[aarch64] and [x86_64] which, under set -u, are treated as variable expansions
and cause "unbound variable" failures; update the array declaration for
SOCAT_SHA256 to quote the keys (e.g. ["aarch64"], ["x86_64"]) so bash treats
them as literal strings and the pipeline no longer errors when set -u is
enabled.
In `@scripts/fetch-alpine.sh`:
- Around line 26-29: The associative array PINNED_SHA256 is using unquoted
subscripts which fail under set -u; update the declaration of PINNED_SHA256 to
use quoted keys (e.g., "aarch64" and "x86_64") and ensure any later accesses to
PINNED_SHA256[...] also quote the subscript to avoid bash treating them as
undefined variables; locate the PINNED_SHA256 declaration and all uses of
PINNED_SHA256 in the script and replace unquoted keys/subscripts with quoted
strings.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3e255fd7-9172-486b-b2dc-73900057abc0
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (16)
.github/workflows/ci-macos.ymlcrates/minvmd/Cargo.tomlcrates/minvmd/src/cmd/boot.rscrates/minvmd/src/cmd/mod.rscrates/minvmd/src/cmd/vmm_child.rscrates/minvmd/src/image.rscrates/minvmd/src/krun/mod.rscrates/minvmd/src/krun/raw.rscrates/minvmd/src/lib.rscrates/minvmd/src/main.rscrates/minvmd/tests/boot_e2e.rsdocs/specs/01-spec-minvmd-host-daemon/01-spec-minvmd-host-daemon.mddocs/specs/01-spec-minvmd-host-daemon/architecture.mdscripts/build-rootfs.shscripts/fetch-alpine.shscripts/fetch-virtio-kernel.sh
The READY-writer stub runs socat, which dynamically links libreadline.so.8 (needs libncursesw.so.6) — neither is in the Alpine minirootfs, so socat aborted with "Error loading shared library libreadline.so.8" and never wrote READY. Overlay the readline and libncursesw apks (sha256-pinned) alongside socat via a fetch_apk helper. (libssl/libcrypto/libc are already in the base.) NOTE: the library ships in `libncursesw`, not the payload-less `ncurses-libs` metapackage. Verified locally on Apple Silicon: a codesigned `minvmd boot` against the prebuilt virtio-linux kernel boots Alpine and prints `vm-up`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cargo test` relinks target/debug/minvmd under the test profile, discarding the hypervisor entitlement applied by codesign — so krun_start_enter failed with EINVAL even though a prior step signed the binary. Build with `--no-run`, codesign, then execute the prebuilt boot_e2e test binary directly (no further cargo invocation), which preserves the signature. Stage the real rootfs via build-rootfs.sh and upload the guest console log. Verified locally: build --no-run -> codesign -> run test binary == 1 passed, binary still signed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
scripts/build-rootfs.sh (1)
156-161: 💤 Low valueConsider logging a warning if READY marker emission fails after all retries.
If all 50 attempts fail, the script silently proceeds to the echo bridge. For a bring-up stub this is likely acceptable, but a warning log would aid debugging vsock connectivity issues without blocking the boot.
i=0 while [ "$i" -lt 50 ]; do printf 'READY\n' | socat -t2 - VSOCK-CONNECT:2:7350 && break i=$((i + 1)) sleep 0.1 done +[ "$i" -ge 50 ] && echo "minvmd-stub-init: READY marker failed after 50 attempts" >&2🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/build-rootfs.sh` around lines 156 - 161, The retry loop that sends the 'READY' marker via socat (the while loop using variable i and the socat - VSOCK-CONNECT:2:7350 command) currently proceeds silently if all 50 attempts fail; add a post-loop check that detects the failed case (i reached 50 / the loop never broke) and emits a warning message to stderr or syslog before continuing to the echo bridge to aid debugging of vsock connectivity issues.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@scripts/build-rootfs.sh`:
- Around line 156-161: The retry loop that sends the 'READY' marker via socat
(the while loop using variable i and the socat - VSOCK-CONNECT:2:7350 command)
currently proceeds silently if all 50 attempts fail; add a post-loop check that
detects the failed case (i reached 50 / the loop never broke) and emits a
warning message to stderr or syslog before continuing to the echo bridge to aid
debugging of vsock connectivity issues.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ee86643e-8828-47fb-9b3a-b8c474f1a097
📒 Files selected for processing (2)
.github/workflows/ci-macos.ymlscripts/build-rootfs.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/ci-macos.yml
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/ci-macos.yml (1)
69-72:⚠️ Potential issue | 🟠 Major | ⚡ Quick winSerialize the two macOS jobs or widen
boot-e2e's timeout.
boot-e2eandbuild-macosboth target the single self-hosted ARM64 runner, butboot-e2etimes out after 20 minutes whilebuild-macosis allowed 30. Because queued time counts toward the job timeout here,boot-e2ecan fail in the queue before its VM boot even starts, which will make this new required gate flaky.Suggested fix
boot-e2e: if: ${{ vars.RUN_MACOS_CI != 'false' }} - needs: virtio-kernel + needs: [virtio-kernel, build-macos] runs-on: [self-hosted, macOS, ARM64] - timeout-minutes: 20 + timeout-minutes: 30Also applies to: 136-138
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/ci-macos.yml around lines 69 - 72, The two macOS jobs (boot-e2e and build-macos) contend for the single self-hosted ARM64 runner and boot-e2e currently has timeout-minutes: 20 which can expire while queued; either serialize these jobs so they don't run concurrently (e.g., add needs/depends-on from boot-e2e to build-macos or vice versa) or increase boot-e2e's timeout-minutes to match build-macos (e.g., 30+); locate the job definitions named boot-e2e and build-macos and update their depends/needs or the timeout-minutes field accordingly to prevent queue time from causing flaky failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/ci-macos.yml:
- Around line 69-72: The two macOS jobs (boot-e2e and build-macos) contend for
the single self-hosted ARM64 runner and boot-e2e currently has timeout-minutes:
20 which can expire while queued; either serialize these jobs so they don't run
concurrently (e.g., add needs/depends-on from boot-e2e to build-macos or vice
versa) or increase boot-e2e's timeout-minutes to match build-macos (e.g., 30+);
locate the job definitions named boot-e2e and build-macos and update their
depends/needs or the timeout-minutes field accordingly to prevent queue time
from causing flaky failures.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 09fed536-047f-46da-ad06-39bd3aba30ba
📒 Files selected for processing (2)
.github/workflows/ci-macos.ymldocs/specs/01-spec-minvmd-host-daemon/01-spec-minvmd-host-daemon.md
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/specs/01-spec-minvmd-host-daemon/01-spec-minvmd-host-daemon.md
Name build-rootfs.sh (overlays /sbin/minvmd-stub-init + socat + the readline/libncursesw closure), not just fetch-alpine.sh, and state the result is a virtio-fs directory (no disk image). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A failed open/read of /dev/urandom silently fell back to a zero nonce, making the marker socket path predictable and defeating the TOCTOU hardening the comment promises. Propagate the error with context instead of discarding it via `let _ =` (which the repo's standards ban). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
d6519a3 to
82ce054
Compare
Supersedes #339. Produces a viable non-EFI libkrun VM image (external
virtio-linuxkernel + Alpine virtio-fs rootfs) and wires the boot path sominvmd bootreaches the guestREADYmarker. Carries #339's host plumbing plus the fixes below.Root cause
The #339 boot E2E could not reach
READYfor four reasons:image.rsusedKRUN_KERNEL_FORMAT_IMAGE_GZ=4; the aarch64 libkrun loader implements onlyRAW=0andPE_GZ=2, sokrun_set_kernelreturnedKernelFormatUnsupportedbefore boot. Verified against/opt/homebrew/include/libkrun.h(1.18.1).vm.rsnorvmm_child.rscalledkrun_set_exec, so libkrun's/init.krun(pid-1) fell back to/bin/shand never emittedREADY.krun_add_vsock_port(≡listen=false, guest→host) and listens on the host UDS, but the rootfs stubsocat VSOCK-LISTEN:7350also listened. Both sides waited. Confirmed againstsrc/libkrun/src/lib.rs+ libkrun#246.Fix
fix(minvmd): aarch64 →KRUN_KERNEL_FORMAT_PE_GZ; add the constant.feat(minvmd):krun_set_execthe workload (/sbin/minvmd-stub-init,MINVMD_EXECoverrides) with an explicit minimal envp (avoids the ~2 KiB aarch64 cmdline overflow); opt-in console capture (MINVMD_BOOT_LOG); 2 vCPU / 1024 MiB. KernelcmdlinestaysNULL(libkrun's default carriesconsole=hvc0 rootfstype=virtiofs rw).feat(minvmd):scripts/{fetch-alpine,build-rootfs,fetch-virtio-kernel}.sh; the stub READY writer now connects out (socat - VSOCK-CONNECT:2:7350).ci(minvmd):boot-e2ebuilds the real rootfs and uploads the guest console log.docs(minvmd): spec R2.1/R2.3/R2.4 + process model corrected.Acceptance
cargo test -p minvmd,cargo clippy -p minvmd --all-targets -- -D warnings,cargo fmt --check— green locally.krun_smoke) green against real libkrun 1.18.1.build-rootfs.shassembles a rootfs with/sbin/minvmd-stub-init(connect-out) +socat.boot-e2egreen on the self-hosted Apple Silicon runner → then dropcontinue-on-errorto gate. First green also validates thevirtio-linuxkernel config (virtio-MMIO /VIRTIO_FS/VIRTIO_VSOCKETS/ HVC all=y).Note on #341
This PR carries #341's
fetch-virtio-kernel.shand an enhancedci-macos.ymlboot-e2e lane, so it supersedes #341's CI work. Merge #341 first and rebase, or close #341 — your call.References
include/libkrun.h,src/libkrun/src/lib.rs,init/init.c; libkrun#246🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests
Chores