Skip to content

ci(minvmd): virtio-linux kernel-fetch lane + non-gating boot E2E - #341

Closed
norrietaylor wants to merge 2 commits into
mainfrom
ci/minvmd-boot-e2e-kernel-lane
Closed

ci(minvmd): virtio-linux kernel-fetch lane + non-gating boot E2E#341
norrietaylor wants to merge 2 commits into
mainfrom
ci/minvmd-boot-e2e-kernel-lane

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jun 4, 2026

Copy link
Copy Markdown
Member

What

Adds a CI lane that pulls the prebuilt virtio-linux kernel from the public minimal build cache and runs the minvmd boot E2E on the self-hosted macOS runner.

  • scripts/fetch-virtio-kernel.sh — fetches the promoted Linux minimal CLI from gs://minimal-shim, then minimal materializes virtio-linux's vmlinuz from the public minimal-staging-cache (no kernel build, no GCS auth). Verified locally: produces a 14 MB Image.gz.
  • .github/workflows/ci-macos.yml:
    • virtio-kernel job (ubuntu) → uploads the vmlinuz artifact.
    • boot-e2e job (self-hosted macOS) → downloads the kernel, stages a rootfs, codesigns the binary (R1.4 hypervisor entitlement), runs boot_e2e.

Non-gating for now

boot-e2e is continue-on-error: true. The R2.4 READY round-trip cannot pass yet:

  • guest vsock stub (R3.4) is Vsock stub and bridge end-to-end test #328, still downstream;
  • marker-port contract unsettled: host VSOCK_MARKER_PORT=9799 vs guest rootfs 7350;
  • R2.2 rootfs staging (scripts/fetch-alpine.sh) is not on main (staged inline here as a stopgap; vanilla Alpine has no READY-writer).

Drop continue-on-error to make it a required check once #328 and the port contract land. The kernel-fetch lane is the durable, reusable piece.

Refs: #339 · kernel dep gominimal/pkgs virtio-linux

Summary by CodeRabbit

  • Chores
    • Added automated kernel artifact retrieval for CI to streamline test assets provisioning.
    • Added Apple Silicon end-to-end boot testing in CI: downloads kernel artifact, stages a minimal rootfs, builds and signs a debug binary, and runs non-blocking boot tests (tests do not gate the workflow).

Pull the prebuilt virtio-linux kernel from the public minimal build cache
(gs://minimal-shim CLI -> minimal materialize virtio-linux from
minimal-staging-cache; no build, no auth) on a Linux job and hand it to a
self-hosted macOS boot-e2e job.

boot-e2e is continue-on-error for now: the R2.4 READY round-trip needs the
R3.4 guest vsock stub (#328) and a settled marker-port contract (host
VSOCK_MARKER_PORT=9799 vs guest rootfs 7350), and R2.2 rootfs staging
(scripts/fetch-alpine.sh) is not yet on main. Drop continue-on-error to make
it required once those land.

Refs: #339

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a Linux helper script to fetch a prebuilt virtio-linux kernel and updates the macOS CI workflow to produce that kernel artifact on Ubuntu and run a conditional self-hosted Apple Silicon boot-e2e job that downloads the kernel, stages a rootfs, builds/codesigns minvmd, and runs boot tests.

Changes

E2E Boot Testing Setup

Layer / File(s) Summary
Kernel fetch helper script
scripts/fetch-virtio-kernel.sh
Shell script validates args/runner arch, downloads the promoted minimal CLI from public storage, creates a temporary minimal.toml project, materializes virtio-linuxvmlinuz, writes it to the destination, and cleans up temporary files.
CI workflow jobs for kernel and boot tests
.github/workflows/ci-macos.yml
Adds scripts/fetch-virtio-kernel.sh to workflow path filters; new Ubuntu virtio-kernel job runs the script and uploads virtio-kernel-aarch64; new conditional self-hosted macOS ARM64 boot-e2e job downloads the artifact, verifies libkrun.dylib, stages an Alpine aarch64 minirootfs, builds and codesigns minvmd, and runs boot_e2e with continue-on-error: true.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly Related PRs

  • gominimal/minimal#283: Modifies .github/workflows/ci-macos.yml to add macOS self-hosted Apple Silicon CI jobs for minvmd with overlapping macOS CI wiring.

Suggested Reviewers

  • jtnkminimal

Poem

🐰 A kernel hops from Linux ground to Mac,
Packaged, signed, then run — no hack,
A mini-rootfs gently laid,
minvmd boots in bright parade,
Tests tumble onward — tails held high.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: adding a CI lane for virtio-linux kernel fetching and non-gating boot E2E testing on macOS.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci-macos.yml:
- Around line 89-109: The workflow step currently wraps setup, build, codesign,
and the test run under continue-on-error; split it so the ROOTFS
download/extract (the curl | tar into ROOTFS), kernel staging, the cargo build
-p minvmd --bin minvmd, and the codesign invocation (codesign --entitlements ...
target/debug/minvmd) are separate steps that run normally (fail the job on
error), and keep continue-on-error: true only on the final step that runs
MINVMD_E2E=1 ... cargo test -p minvmd --test boot_e2e -- --include-ignored
--nocapture; ensure the final step still sets MINVMD_KERNEL_PATH,
MINVMD_ROOTFS_PATH and MINVMD_E2E as before so the test behavior is unchanged.
- Around line 49-50: The workflow’s path filter is missing the new script, so
changes to scripts/fetch-virtio-kernel.sh won't trigger the job; update the
ci-macos.yml paths filter to include "scripts/fetch-virtio-kernel.sh" (or a
matching pattern like "scripts/**") so that modifications to the fetch script
run this workflow; locate the job that runs ./scripts/fetch-virtio-kernel.sh and
add the script path to its paths inclusion list.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4eeea8ae-d2cb-4a67-86ff-18aff46c59ca

📥 Commits

Reviewing files that changed from the base of the PR and between 5862969 and 7f1046b.

📒 Files selected for processing (2)
  • .github/workflows/ci-macos.yml
  • scripts/fetch-virtio-kernel.sh

Comment thread .github/workflows/ci-macos.yml
Comment thread .github/workflows/ci-macos.yml Outdated
…he non-gating test

Address CodeRabbit review:
- Add scripts/fetch-virtio-kernel.sh to the ci-macos path filter so a
  change to the fetch logic alone still runs this workflow.
- Split rootfs staging, build, and codesign into their own steps that fail
  the job; keep continue-on-error only on the boot_e2e test invocation, so
  setup/build regressions are no longer masked by the known-non-gating test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@norrietaylor

Copy link
Copy Markdown
Member Author

@CodeRabbit review

@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@norrietaylor

Copy link
Copy Markdown
Member Author

closing, #344 supersedes

norrietaylor added a commit that referenced this pull request Jun 4, 2026
#326) (#344)

* feat(minvmd): add boot command and VMM child subcommand

Implements R2.3 and R2.4 from the minvmd spec:

- cmd/mod.rs: declares the cmd module, VSOCK_MARKER_PORT (9799) and
  MARKER_SOCK_ENV constants shared between parent and child.

- cmd/boot.rs: the `minvmd boot [--foreground]` subcommand.
  On macOS: validates MINVMD_KERNEL_PATH and MINVMD_ROOTFS_PATH,
  creates a UNIX socket listener for the READY marker, fork-execs
  `minvmd __krun-vmm` with MINVMD_MARKER_SOCK set to the socket
  path, writes the child PID to vmm.pid, then waits up to 5 s for
  the guest to connect and write READY\n (R2.4). On success prints
  vm-up. With --foreground, blocks until the VMM child exits.
  On Linux: bails immediately (no-op stub).

- cmd/vmm_child.rs: the hidden `minvmd __krun-vmm` subcommand.
  On macOS: creates a libkrun context, applies VmConfig (kernel,
  rootfs, 2 vcpus, 512 MiB), registers VSOCK_MARKER_PORT pointing
  to the host UNIX socket, then calls krun_start_enter (R2.3).
  On Linux: bails immediately (no-op stub).

- main.rs: wires Boot and KrunVmm subcommands to the CLI.
- lib.rs: exports pub mod cmd.
- tests/boot_e2e.rs: READY-marker round-trip E2E test (gated on
  MINVMD_E2E=1 and #[ignore], macOS only) (R2.4).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(minvmd): reap child and clean up pid/socket on boot failure

On the error and timeout paths in boot.rs, terminate and wait the VMM
child and remove both vmm.pid and the marker socket before returning,
so a failed boot does not leave stale state behind.

In boot_e2e.rs, set XDG_STATE_HOME to an isolated tempdir for the
spawned child so the E2E test does not clobber the developer's real
state directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(minvmd): serialize boot E2E test and add serial_test dep

Add #[serial] to boot_e2e_ready_marker_round_trip so it cannot race
other stateful E2E tests that mutate XDG_STATE_HOME. The test already
isolates its state directory via XDG_STATE_HOME=<tempdir>, so no
additional isolation is needed; serialization ensures only one such
test runs at a time.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(minvmd): harden marker socket path with random nonce

Use PID + 4 random bytes from /dev/urandom for the READY-marker socket
path instead of PID alone, making the path unpredictable and closing
the TOCTOU race a local attacker could exploit to inject a spoofed
READY.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(minvmd): align READY marker vsock port to canonical 7350

The host awaited the boot READY marker on vsock 9799, but no guest emits
on that port: the guest rootfs manifest documents 7350
(etc/minvmd/manifest: vsock_port_ready=7350). With 9799 the marker never
arrived. Align the host to 7350.

Necessary but not sufficient: krun_start_enter still returns EINVAL; root
cause under investigation (see PR comment) — not a kernel-format or
rootfs-format issue.

Refs: #221

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): load aarch64 Image.gz with PE_GZ format

The aarch64 libkrun loader implements only RAW and PE_GZ; IMAGE_GZ (=4)
is x86_64-only and returns KernelFormatUnsupported, so krun_set_kernel
failed before the VM could boot. Select KRUN_KERNEL_FORMAT_PE_GZ (=2)
for the aarch64 Image.gz (the loader scans for the gzip magic and
decompresses) and add the constant to the FFI surface.

Verified against /opt/homebrew/include/libkrun.h (libkrun 1.18.1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(minvmd): set guest workload, capture console, raise boot resources

The VMM child configured a kernel + rootfs but never set a workload, so
libkrun's /init.krun (pid-1) fell back to /bin/sh and never emitted the
READY marker. Set the guest workload via krun_set_exec (default
/sbin/minvmd-stub-init, MINVMD_EXEC overrides) with an explicit minimal
envp — passing None would inherit the full host env and can overflow the
~2 KiB aarch64 kernel cmdline.

Add opt-in early-boot console capture (MINVMD_BOOT_LOG) for diagnosing a
stuck boot, and raise the VM from 512 MiB to 2 vCPU / 1024 MiB for cheap
headroom under Hypervisor.framework.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(minvmd): add VM image build scripts with guest READY writer

Stage the non-EFI VM image production on main:

- fetch-alpine.sh: pinned, sha256-verified Alpine 3.21.7 minirootfs.
- build-rootfs.sh: overlay socat + the /sbin/minvmd-stub-init workload
  and the guest manifest onto the rootfs directory (consumed by
  krun_set_root as virtio-fs; no disk image).
- fetch-virtio-kernel.sh: pull the prebuilt virtio-linux vmlinuz
  (Image.gz) from the public minimal build cache (carried from #341).

The stub's READY writer connects OUT to the host (CID 2, port 7350)
rather than listening: the host registers the marker with the plain
krun_add_vsock_port (== listen=false), so the direction is guest->host.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd): boot E2E on the real Alpine rootfs + virtio kernel

Stage the guest rootfs via fetch-alpine.sh + build-rootfs.sh (was a
vanilla extract with no READY writer), capture the guest console as an
artifact for debugging, and scope the workflow to scripts/**. The boot
E2E stays non-gating (continue-on-error) until first-green on the
self-hosted runner, which also validates the virtio-linux kernel config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(minvmd): correct kernel format, init model, and marker direction

Align the spec with libkrun's verified behaviour: aarch64 loads Image.gz
via PE_GZ (not IMAGE_GZ); libkrun's /init.krun is pid-1 and execs the
krun_set_exec workload (no init system in the rootfs); the kernel cmdline
stays unset; boot uses 2 vCPU / 1024 MiB; and the READY marker is
guest-initiated (krun_add_vsock_port, listen=false), the opposite of the
R3 ssh.sock bridge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): make image scripts run under macOS bash 3.2

The self-hosted runner's `/usr/bin/env bash` is the macOS system bash
3.2, which lacks associative arrays; `declare -A` made `[aarch64]=` parse
as an unbound arithmetic index under `set -u`, failing build-rootfs.sh
with "aarch64: unbound variable". Replace the per-arch sha256 arrays with
plain vars + a case lookup. Verified under /bin/bash 3.2.57.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): add socat's runtime closure to the guest rootfs

The READY-writer stub runs socat, which dynamically links libreadline.so.8
(needs libncursesw.so.6) — neither is in the Alpine minirootfs, so socat
aborted with "Error loading shared library libreadline.so.8" and never
wrote READY. Overlay the readline and libncursesw apks (sha256-pinned)
alongside socat via a fetch_apk helper. (libssl/libcrypto/libc are already
in the base.) NOTE: the library ships in `libncursesw`, not the
payload-less `ncurses-libs` metapackage.

Verified locally on Apple Silicon: a codesigned `minvmd boot` against the
prebuilt virtio-linux kernel boots Alpine and prints `vm-up`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd): run the boot E2E binary directly so codesign survives

`cargo test` relinks target/debug/minvmd under the test profile, discarding
the hypervisor entitlement applied by codesign — so krun_start_enter failed
with EINVAL even though a prior step signed the binary. Build with
`--no-run`, codesign, then execute the prebuilt boot_e2e test binary
directly (no further cargo invocation), which preserves the signature.
Stage the real rootfs via build-rootfs.sh and upload the guest console log.

Verified locally: build --no-run -> codesign -> run test binary == 1 passed,
binary still signed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd): make the boot E2E a required gate

The READY round-trip is green on the self-hosted Apple Silicon runner
(test result: ok. 1 passed). Drop continue-on-error so a boot regression
fails the workflow. This is the acceptance gate for #311 / #326.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(minvmd): expand R2.2 for the rootfs overlay and socat closure

Name build-rootfs.sh (overlays /sbin/minvmd-stub-init + socat + the
readline/libncursesw closure), not just fetch-alpine.sh, and state the
result is a virtio-fs directory (no disk image).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minvmd): propagate /dev/urandom read failure for the marker nonce

A failed open/read of /dev/urandom silently fell back to a zero nonce,
making the marker socket path predictable and defeating the TOCTOU
hardening the comment promises. Propagate the error with context instead
of discarding it via `let _ =` (which the repo's standards ban).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: gominimal-aw-bot[bot] <281738952+gominimal-aw-bot[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@norrietaylor
norrietaylor deleted the ci/minvmd-boot-e2e-kernel-lane branch June 26, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant