feat(minimald): vsock guest (pid-1) mode for minvmd - #361
Conversation
Define the minvmd guest rootfs as a minimal package instead of the shell-script Alpine overlay. build.sh snapshots the closure of its build_deps (socat + bash + coreutils + e2fsprogs), drops in the bring-up init and the /etc/minvmd/manifest contract, prunes build-only bulk, and packs an ext4 image with mke2fs. Cross-layer deps are referenced via `upstream "<name>"`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the upstream gominimal/pkgs lock 0da02c7 -> 133db0c (the commit that adds the virtio-linux package; a 52-commit fast-forward) and add the raw-file outputs minvmd materializes: virtio-kernel (the guest kernel) and minvmd-rootfs (the guest ext4 image). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch the guest root from a virtiofs directory (krun_set_root) to a read-only ext4 block device. A block root has no libkrun /init.krun, so the kernel runs the workload directly via an explicit cmdline `root=/dev/vda rootfstype=ext4 ro init=<exec-target>`; the exec target (MINVMD_EXEC, default /sbin/minvmd-stub-init) moves from krun_set_exec into init=. Add the krun_add_disk2 binding + safe wrapper and an exec_target field on VmConfig. MINVMD_ROOTFS_PATH now resolves a file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…source Rewrite fetch-virtio-kernel.sh to build the minimal CLI from this repo (cargo build -p minimal) and materialize virtio-kernel against the repo's own minimal.toml, instead of downloading the promoted CLI from gs://minimal-shim. Drop `minimal update` (it would rewrite the tracked locked_commit and risk an uncached, full kernel build) and the scratch project. Linux-only, with a guard pointing at crates/minvmd/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewire ci-macos.yml: the virtio-kernel job builds minimal from source (toolchain + protoc + rust-cache); a new minvmd-rootfs job materializes the ext4 image on the self-hosted aarch64 runner (the build container does not emulate, so aarch64 packages need a native builder); boot-e2e and autospawn-e2e consume both as artifacts. Delete fetch-alpine.sh and build-rootfs.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add crates/minvmd/README.md (materialize kernel + rootfs into .scratch/, build, codesign-last, run minimal2 ls; E2E invocation; boot mechanics) and gitignore .scratch/. Closes the references from the kernel script and the CI error messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
libkrun's in-VMM gzip decompress of the aarch64 Image.gz cost ~77 ms of a ~146 ms boot-to-READY (measured N=10). Ship the kernel uncompressed and load it with KRUN_KERNEL_FORMAT_RAW, which skips the decompress: median boot-to-READY drops 146 -> ~67 ms. Add a local virtio-kernel-raw package that gunzips the upstream virtio-linux Image.gz, repoint the virtio-kernel output at it, and make RAW the aarch64 default in image.rs. The artifact grows 14 -> 38 MB, which is trivial next to per-boot latency. (Measured separately: quiet cmdline, vCPU count, and RAM size were all boot-noise.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build script panicked when 'git rev-parse' produced no output (for
example cross-compiling inside a container with no repo mounted, or
building from a release tarball), because strip_suffix("\n").unwrap()
hit None. Fall back to a GIT_HASH of "unknown" instead so the build
succeeds outside a git checkout. This unblocks the aarch64 cross-build
used for the minvmd guest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make minimald capable of running as the guest pid-1 / kernel init= target inside the minvmd microVM, replacing the socat stub init. - Generalize Connection::from_socket into from_stream<S: AsyncRead + AsyncWrite + Unpin + Send + 'static>, keeping from_socket as a thin UDS wrapper. The russh stack was already transport-agnostic. - Add Server::run_on_vsock(config, port): bind a tokio-vsock listener on CID_ANY and run the same accept -> from_stream -> JoinSet loop as the UDS path. The host-mediated vsock peer (net=none) is as trusted as the UDS peer, so connections are treated as local (Auth::Local). - Add a new guest module (Linux-only) implementing the boot contract and pid-1 hygiene: emit the one-shot READY marker to host CID 2 port 7350 with retry/backoff; mount /proc and /sys if absent (the kernel only auto-mounts devtmpfs on /dev); and reap orphaned children via a SIGCHLD waitpid(-1, WNOHANG) handler, since hakoniwa double-forks its namespace children. - Wire up main.rs: '--guest' (or the MINIMALD_VSOCK_PORT env) selects guest mode, serving on '--vsock-port' (default 2222, the bridge port the host registers via krun_add_vsock_port2). The UDS path remains the default, unaffected, native-Linux behavior. tokio-vsock is a Linux-only target dependency, so the macOS host build is untouched. Verified by cross-compiling, clippy (-D warnings), and testing minimald for aarch64-unknown-linux-musl. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR enables vsock-based SSH transport for ChangesGuest Mode Support
Sequence Diagram(s)sequenceDiagram
participant CLI as minimald CLI
participant Main as main::run_guest
participant Guest as guest helpers
participant Server as Server::run_on_vsock
participant Conn as Connection::from_stream
CLI->>Main: --guest --vsock-port=2222
Main->>Guest: mount_pseudo_filesystems()
Main->>Guest: install_child_reaper()
Main->>Guest: emit_ready_marker()
Main->>Server: run_on_vsock(config, 2222)
Server->>Server: bind vsock listener
Server->>Server: accept vsock stream
Server->>Conn: spawn from_stream session
Conn->>Conn: run russh server over stream
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
…ckage
The virtio-kernel-raw package forced an arm64 build, which the x86_64 CI
runner cannot do ("cannot build spec with target arm64/linux on
amd64/linux") — the upstream virtio-linux kernel only worked there because
it is a cache pull, not a build. Drop the package and gunzip the
cache-pulled Image.gz in fetch-virtio-kernel.sh instead; gunzip is
arch-agnostic, so it runs on any runner while still yielding the raw Image
that KRUN_KERNEL_FORMAT_RAW loads (skipping libkrun's ~77 ms decompress).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`minimal check`'s import-line checker requires destructured minimal.ncl identifiers in canonical order (lowercase first, then uppercase). Reorder to `upstream, BuildSpec, Local, OutputData` so minimal-check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/bench-minvmd-boot.sh times `minvmd boot` to the guest READY marker across N runs and reports min/median/max. This is the harness that found the gzip kernel decompress (~77 ms of ~146 ms) and verified the uncompressed-kernel fix (~67 ms). macOS-only; needs a codesigned minvmd and MINVMD_KERNEL_PATH/MINVMD_ROOTFS_PATH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-hosted runner cache-hit a minvmd-rootfs build entry that lacked rootfs.img (an earlier build produced no output yet got cached), so materialize failed with "copying output file ... rootfs.img: No such file". Add a `command -v mke2fs` precondition and a post-build assertion that the image exists and is non-empty (failing loudly instead of caching an empty result). The build.sh content change also changes the package's input hash, forcing a fresh build past the poisoned cache entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The image was sized tree + 5% + 4 MiB, but ext4's journal (~4 MiB+) plus inode-table metadata that `du` does not count can overflow that on a slightly larger closure — the likely cause of `mke2fs` producing no rootfs.img on the CI runner. The root mounts read-only, so the journal is pure overhead: build with `-O ^has_journal` and widen headroom to tree + 10% + 8 MiB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tiofs) The gating comments still described the old virtiofs path (PE_GZ load, virtio-fs root, /init.krun, Alpine, VIRTIO_FS). Update to the committed model: raw kernel load, ext4 block root via krun_add_disk2, kernel `init=` exec, VIRTIO_BLK/EXT4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-hosted runner fails to collect rootfs.img into the build cache despite a successful build (the build.sh output assertion does not trip), and it is not reproducible locally on the same shim version. Echo the shim version and, on failure, list the minvmd-rootfs output dir + most-recent cache dirs so the next run reveals whether the image was produced (→ a collection/disk issue) or not (→ a build issue). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diagnostics proved the minvmd-rootfs build succeeds and writes the image
to the host build cache (104 MB present), but `minimal materialize`'s copy
step cannot read it back through the shim VM's /host mount on the
self-hosted runner ("copying output file ... No such file" for a file that
exists on the host) — every build, same CLI version that works locally.
Work around it: the build still populates the cache, so on copy failure
take the freshly-built image straight from ~/.cache/minimal/built. TODO:
remove once the shim /host large-file read coherence issue is fixed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire scripts/bench-minvmd-boot.sh into the macOS boot-e2e job so every run reports boot-to-READY min/median/max, and document it in the README. Non-gating (boot correctness is gated by the e2e tests); reuses the already-codesigned minvmd. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…kages Now that gominimal/pkgs ships microvm-rootfs (ext4 guest rootfs) and virtio-kernel-raw (uncompressed kernel Image), drop the local stopgaps and dogfood the upstream packages: - minimal.toml: point [outputs.virtio-kernel] at virtio-kernel-raw (raw Image, no gunzip) and [outputs.minvmd-rootfs] at microvm-rootfs; bump locked_commit. - Delete the local .minimal/packages/minvmd-rootfs/ package. - fetch-virtio-kernel.sh: drop the gunzip step (the kernel is already raw). - Retarget the default guest exec /sbin/minvmd-stub-init -> /sbin/microvm-init (the upstream stub) and the manifest path; update docs + the CI cache path. Validated locally: both artifacts materialize from the upstream packages and boot_e2e + bridge_e2e pass against them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… job Now that the rootfs comes from the upstream microvm-rootfs package, its prebuilt aarch64 artifact is in the public cache — so `materialize --arch aarch64` is a cache pull, not a native build, and works on a cheap x86_64 Linux runner (exactly like the kernel). Materialize both in one `artifacts` job; the self-hosted runner only boots. - Generalize scripts/fetch-virtio-kernel.sh -> fetch-artifact.sh <output> <dest>. - Merge the virtio-kernel + minvmd-rootfs jobs into one Linux `artifacts` job. - Drop the self-hosted minvmd-rootfs materialize job and its shim /host large-file read workaround. - Point boot-e2e/autospawn-e2e at `needs: [artifacts]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
~67 ms was a one-off from a less-loaded machine; the reproducible CI bench is ~113 ms median (runner-dependent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The eager waitpid(-1) SIGCHLD reaper races tokio's process reaping and steals exec children's exit status (ECHILD -> every command reports the wrong exit code). Drop the reaper call: tokio reaps its own children. Reaping hakoniwa double-fork orphans needs a tokio-compatible reaper and is deferred (spec: revisit if zombies bite). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface accepted-connection + session-future errors instead of silently dropping the spawned future, so vsock-transport handshake failures are diagnosable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full SSH session over the krun_add_vsock_port2-bridged vsock stream dies with "early eof" shortly after accept; the deployed guest relays via socat to a UDS and serves run_on_uds. Document so run_on_vsock isn't read as a complete bridge serve path on its own. Doc-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ackport-361-fixes
3ca14ae to
40abc9e
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
scripts/fetch-artifact.sh (1)
38-38: ⚡ Quick winUse
--lockedfor deterministic CI artifact fetches.Line 38 should enforce lockfile fidelity so dependency resolution can’t drift unexpectedly in automation.
Suggested change
-cargo build -p minimal +cargo build --locked -p minimal🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/fetch-artifact.sh` at line 38, The CI build command currently runs "cargo build -p minimal" without lockfile enforcement; change that invocation to "cargo build -p minimal --locked" so Cargo will fail if Cargo.lock is out of sync and prevent nondeterministic dependency resolution in automation — update the command string in scripts/fetch-artifact.sh where "cargo build -p minimal" appears.scripts/bench-minvmd-boot.sh (1)
51-51: 💤 Low valueConsider using
mapfilefor safer array assignment.The current command substitution can misbehave if samples somehow contain whitespace (unlikely with integer ms values, but defensive). This also silences shellcheck SC2207.
♻️ Suggested fix
-sorted=($(printf '%s\n' "${samples[@]}" | sort -n)) +mapfile -t sorted < <(printf '%s\n' "${samples[@]}" | sort -n)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/bench-minvmd-boot.sh` at line 51, Replace the command-substitution array assignment for sorted which uses sorted=($(printf '%s\n' "${samples[@]}" | sort -n)) with a safe mapfile-based assignment to avoid word-splitting and silence SC2207: use mapfile -t sorted to read the sorted lines produced by printf/sort into the sorted array (keep using "${samples[@]}" and sort -n), and remove the old command-substitution form; this targets the sorted variable and the samples array usage in the script.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/minvmd/README.md`:
- Around line 62-67: The README currently picks only the boot_e2e-* binary via
the testbin assignment, so the bridge_e2e binary (built but ignored) is never
executed; update the documented invocation to run both E2E binaries (boot_e2e
and bridge_e2e) instead of only matching boot_e2e-*. For example, change the
single testbin selection to either iterate over both patterns or select a more
general pattern that matches both e2e binaries (referencing the testbin variable
and the boot_e2e-* pattern and the bridge_e2e name) and show running each found
binary with the same MINVMD_* env vars so the “boot + bridge” flow actually
executes both tests.
In `@crates/minvmd/src/cmd/vmm_child.rs`:
- Line 54: The current assignment to exec using
std::env::var("MINVMD_EXEC").unwrap_or_else(...) treats an empty MINVMD_EXEC as
a valid value; change the logic so that after reading the environment variable
you also treat an empty string as unset and fall back to the default
"/sbin/microvm-init". Concretely, read MINVMD_EXEC into an Option or String,
check if it is empty (e.g., None or trimmed().is_empty()), and only assign exec
to the env value when non-empty; otherwise set exec to the default. Update the
code that constructs exec in vmm_child.rs (the variable exec and the MINVMD_EXEC
lookup) accordingly.
---
Nitpick comments:
In `@scripts/bench-minvmd-boot.sh`:
- Line 51: Replace the command-substitution array assignment for sorted which
uses sorted=($(printf '%s\n' "${samples[@]}" | sort -n)) with a safe
mapfile-based assignment to avoid word-splitting and silence SC2207: use mapfile
-t sorted to read the sorted lines produced by printf/sort into the sorted array
(keep using "${samples[@]}" and sort -n), and remove the old
command-substitution form; this targets the sorted variable and the samples
array usage in the script.
In `@scripts/fetch-artifact.sh`:
- Line 38: The CI build command currently runs "cargo build -p minimal" without
lockfile enforcement; change that invocation to "cargo build -p minimal
--locked" so Cargo will fail if Cargo.lock is out of sync and prevent
nondeterministic dependency resolution in automation — update the command string
in scripts/fetch-artifact.sh where "cargo build -p minimal" appears.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c4e0f0e7-ec62-4bbb-973f-3a6b04c15415
📒 Files selected for processing (20)
.github/workflows/ci-macos.yml.gitignore.minimal/minimal.tomlcrates/minimald/src/server.rscrates/minvmd/README.mdcrates/minvmd/src/cmd/mod.rscrates/minvmd/src/cmd/vmm_child.rscrates/minvmd/src/image.rscrates/minvmd/src/krun/ctx.rscrates/minvmd/src/krun/mod.rscrates/minvmd/src/krun/raw.rscrates/minvmd/src/sock.rscrates/minvmd/src/vm.rscrates/minvmd/tests/boot_e2e.rscrates/minvmd/tests/bridge_e2e.rsscripts/bench-minvmd-boot.shscripts/build-rootfs.shscripts/fetch-alpine.shscripts/fetch-artifact.shscripts/fetch-virtio-kernel.sh
💤 Files with no reviewable changes (3)
- scripts/fetch-alpine.sh
- scripts/build-rootfs.sh
- scripts/fetch-virtio-kernel.sh
✅ Files skipped from review due to trivial changes (6)
- .gitignore
- crates/minvmd/src/cmd/mod.rs
- crates/minvmd/tests/bridge_e2e.rs
- crates/minvmd/tests/boot_e2e.rs
- crates/minvmd/src/sock.rs
- crates/minvmd/src/krun/mod.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/minimald/src/server.rs
Stacks on #361 (minimald vsock guest mode). Ships minimald as the initramfs /init (a cpio of the cross-compiled static binary) and serves a full session against the GENERIC upstream microvm-rootfs — no minimald baked into the rootfs. - minimald: run as /init (detect via argv[0]); mount devtmpfs; mount /dev/vda + chroot into the rootfs so /bin/sh + socat resolve; serve over a socat vsock->UDS relay + run_on_uds with tmpfs (/run/minimal) state. The block-root guest path gains a writable data disk + format-on-first-boot. - minvmd: VmConfig.initramfs + krun_set_kernel initramfs arg (MINVMD_INITRAMFS); provision + attach the rw data disk. - scripts/build-initramfs.sh + CI: cross-compile minimald with a fast `initramfs` profile, pack the cpio, cache the aarch64 target, and run the initramfs session e2e against the generic rootfs. Validated: minimald_exec_over_bridge passes booted via initramfs — exec stdout correct, exit 0. Boot-to-READY ~76 ms median, on par with block-root. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Closing in favor of #373 + #374, which re-slice this work along capability lines instead of delivery-mechanism lines:
The block-root boot path ( |
Makes
minimaldcapable of running as the minvmd guest pid-1 over vsock, replacing the socat bring-up stub. Stage 2 of the minvmd plan; the integration (baking minimald into the rootfs, flipping the exec target, e2e boot) is a follow-up. Builds on the block-root boot in #359 but is independent code (minimald crate).Gating risk resolved
cross build -p minimald --release --target aarch64-unknown-linux-muslsucceeds — hakoniwa builds on static musl (the top risk), producing a 27.3 MB static aarch64 ELF.cross clippy --all-targets -- -D warningsclean; 16 tests pass under QEMU.What changed
tokio-vsock(workspace-pinned), gated Linux-only in minimald.Connection::from_socket→ genericfrom_stream<S: AsyncRead + AsyncWrite + Unpin + Send + 'static>;from_socketkept as a thin UDS wrapper.Server::run_on_vsock(config, port)— bindsVsockListeneronVMADDR_CID_ANY, same accept→from_stream→JoinSetloop, peers treated asAuth::Local(host-mediated bridge,net=none).guest.rs: READY marker (connectVMADDR_CID_HOST:7350, writeREADY\n, retry-backoff);/proc+/sysmount; SIGCHLDwaitpid(-1, WNOHANG)reaper for hakoniwa's double-forked orphans.main.rs--guest/MINIMALD_VSOCK_PORTmode (default port 2222); the UDS path is unchanged default behavior.build.rspanic when git context is absent (blocked containerized builds).Caveats
Follow-up (integration)
minimaldinto theminvmd-rootfspackage at/sbin/minimald.MINVMD_EXEC=/sbin/minimald+--guest; retire the socat stub.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
--guestexecution mode to run minimald as a guest VM init process over vsock--vsock-portCLI option for configurable vsock port (configurable viaMINIMALD_VSOCK_PORTenvironment variable)Chores
tokio-vsockdependency (Linux-only)