feat(minvmd): boot a microVM from kernel + rootfs built as minimal packages by norrietaylor · Pull Request #367 · gominimal/minimal

norrietaylor · 2026-06-08T18:28:14Z

Stage 1 — the guest runs a socat bring-up stub (/sbin/microvm-init), not minimald. minimald-as-pid-1 is Stage 2 (#362).

What changed

Boots a real Linux microVM from artifacts the build system produces itself — removes the gs://minimal-shim CLI download and the pinned-Alpine apk overlay.
Kernel + rootfs are now upstream minimal packages (virtio-kernel-raw + microvm-rootfs), cache-pulled by the pinned locked_commit.
Ext4 block root instead of virtiofs — loaded via krun_add_disk2 (/dev/vda) with a kernel init= cmdline (a block root has no libkrun /init.krun).
Faster boot — ships the kernel uncompressed (raw Image), skipping libkrun's in-VMM gzip decompress (~77 ms); boot-to-READY ~113 ms median on CI.

How to run on Mac

Prereqs: Apple Silicon · brew install slp/krun/libkrun · the minimal shim on PATH.

mkdir -p .scratch
minimal materialize --output .scratch/vmlinuz    --arch aarch64 virtio-kernel
minimal materialize --output .scratch/rootfs.img --arch aarch64 minvmd-rootfs

cargo build -p minvmd --bin minvmd -p minimal2 --bin minimal2
# codesign LAST — a later cargo run/test relinks + unsigns it (krun_start_enter -> EINVAL)
codesign --entitlements crates/minvmd/minvmd.entitlements --force -s - target/debug/minvmd

export PATH="$PWD/target/debug:$PATH"
export MINVMD_KERNEL_PATH="$PWD/.scratch/vmlinuz"
export MINVMD_ROOTFS_PATH="$PWD/.scratch/rootfs.img"
minimal2 ls        # cold: auto-spawns minvmd, boots the VM, prints []
minvmd status      # running
minvmd stop

# boot-to-READY latency (min/median/max over N runs, default 10)
scripts/bench-minvmd-boot.sh

--output must be under the repo (the shim only syncs the project dir back to the host).

Define the minvmd guest rootfs as a minimal package instead of the shell-script Alpine overlay. build.sh snapshots the closure of its build_deps (socat + bash + coreutils + e2fsprogs), drops in the bring-up init and the /etc/minvmd/manifest contract, prunes build-only bulk, and packs an ext4 image with mke2fs. Cross-layer deps are referenced via `upstream "<name>"`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Bump the upstream gominimal/pkgs lock 0da02c7 -> 133db0c (the commit that adds the virtio-linux package; a 52-commit fast-forward) and add the raw-file outputs minvmd materializes: virtio-kernel (the guest kernel) and minvmd-rootfs (the guest ext4 image). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Switch the guest root from a virtiofs directory (krun_set_root) to a read-only ext4 block device. A block root has no libkrun /init.krun, so the kernel runs the workload directly via an explicit cmdline `root=/dev/vda rootfstype=ext4 ro init=<exec-target>`; the exec target (MINVMD_EXEC, default /sbin/minvmd-stub-init) moves from krun_set_exec into init=. Add the krun_add_disk2 binding + safe wrapper and an exec_target field on VmConfig. MINVMD_ROOTFS_PATH now resolves a file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…source Rewrite fetch-virtio-kernel.sh to build the minimal CLI from this repo (cargo build -p minimal) and materialize virtio-kernel against the repo's own minimal.toml, instead of downloading the promoted CLI from gs://minimal-shim. Drop `minimal update` (it would rewrite the tracked locked_commit and risk an uncached, full kernel build) and the scratch project. Linux-only, with a guard pointing at crates/minvmd/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rewire ci-macos.yml: the virtio-kernel job builds minimal from source (toolchain + protoc + rust-cache); a new minvmd-rootfs job materializes the ext4 image on the self-hosted aarch64 runner (the build container does not emulate, so aarch64 packages need a native builder); boot-e2e and autospawn-e2e consume both as artifacts. Delete fetch-alpine.sh and build-rootfs.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add crates/minvmd/README.md (materialize kernel + rootfs into .scratch/, build, codesign-last, run minimal2 ls; E2E invocation; boot mechanics) and gitignore .scratch/. Closes the references from the kernel script and the CI error messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

libkrun's in-VMM gzip decompress of the aarch64 Image.gz cost ~77 ms of a ~146 ms boot-to-READY (measured N=10). Ship the kernel uncompressed and load it with KRUN_KERNEL_FORMAT_RAW, which skips the decompress: median boot-to-READY drops 146 -> ~67 ms. Add a local virtio-kernel-raw package that gunzips the upstream virtio-linux Image.gz, repoint the virtio-kernel output at it, and make RAW the aarch64 default in image.rs. The artifact grows 14 -> 38 MB, which is trivial next to per-boot latency. (Measured separately: quiet cmdline, vCPU count, and RAM size were all boot-noise.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ckage The virtio-kernel-raw package forced an arm64 build, which the x86_64 CI runner cannot do ("cannot build spec with target arm64/linux on amd64/linux") — the upstream virtio-linux kernel only worked there because it is a cache pull, not a build. Drop the package and gunzip the cache-pulled Image.gz in fetch-virtio-kernel.sh instead; gunzip is arch-agnostic, so it runs on any runner while still yielding the raw Image that KRUN_KERNEL_FORMAT_RAW loads (skipping libkrun's ~77 ms decompress). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`minimal check`'s import-line checker requires destructured minimal.ncl identifiers in canonical order (lowercase first, then uppercase). Reorder to `upstream, BuildSpec, Local, OutputData` so minimal-check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scripts/bench-minvmd-boot.sh times `minvmd boot` to the guest READY marker across N runs and reports min/median/max. This is the harness that found the gzip kernel decompress (~77 ms of ~146 ms) and verified the uncompressed-kernel fix (~67 ms). macOS-only; needs a codesigned minvmd and MINVMD_KERNEL_PATH/MINVMD_ROOTFS_PATH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The self-hosted runner cache-hit a minvmd-rootfs build entry that lacked rootfs.img (an earlier build produced no output yet got cached), so materialize failed with "copying output file ... rootfs.img: No such file". Add a `command -v mke2fs` precondition and a post-build assertion that the image exists and is non-empty (failing loudly instead of caching an empty result). The build.sh content change also changes the package's input hash, forcing a fresh build past the poisoned cache entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The image was sized tree + 5% + 4 MiB, but ext4's journal (~4 MiB+) plus inode-table metadata that `du` does not count can overflow that on a slightly larger closure — the likely cause of `mke2fs` producing no rootfs.img on the CI runner. The root mounts read-only, so the journal is pure overhead: build with `-O ^has_journal` and widen headroom to tree + 10% + 8 MiB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tiofs) The gating comments still described the old virtiofs path (PE_GZ load, virtio-fs root, /init.krun, Alpine, VIRTIO_FS). Update to the committed model: raw kernel load, ext4 block root via krun_add_disk2, kernel `init=` exec, VIRTIO_BLK/EXT4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The self-hosted runner fails to collect rootfs.img into the build cache despite a successful build (the build.sh output assertion does not trip), and it is not reproducible locally on the same shim version. Echo the shim version and, on failure, list the minvmd-rootfs output dir + most-recent cache dirs so the next run reveals whether the image was produced (→ a collection/disk issue) or not (→ a build issue). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Diagnostics proved the minvmd-rootfs build succeeds and writes the image to the host build cache (104 MB present), but `minimal materialize`'s copy step cannot read it back through the shim VM's /host mount on the self-hosted runner ("copying output file ... No such file" for a file that exists on the host) — every build, same CLI version that works locally. Work around it: the build still populates the cache, so on copy failure take the freshly-built image straight from ~/.cache/minimal/built. TODO: remove once the shim /host large-file read coherence issue is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Wire scripts/bench-minvmd-boot.sh into the macOS boot-e2e job so every run reports boot-to-READY min/median/max, and document it in the README. Non-gating (boot correctness is gated by the e2e tests); reuses the already-codesigned minvmd. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…kages Now that gominimal/pkgs ships microvm-rootfs (ext4 guest rootfs) and virtio-kernel-raw (uncompressed kernel Image), drop the local stopgaps and dogfood the upstream packages: - minimal.toml: point [outputs.virtio-kernel] at virtio-kernel-raw (raw Image, no gunzip) and [outputs.minvmd-rootfs] at microvm-rootfs; bump locked_commit. - Delete the local .minimal/packages/minvmd-rootfs/ package. - fetch-virtio-kernel.sh: drop the gunzip step (the kernel is already raw). - Retarget the default guest exec /sbin/minvmd-stub-init -> /sbin/microvm-init (the upstream stub) and the manifest path; update docs + the CI cache path. Validated locally: both artifacts materialize from the upstream packages and boot_e2e + bridge_e2e pass against them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… job Now that the rootfs comes from the upstream microvm-rootfs package, its prebuilt aarch64 artifact is in the public cache — so `materialize --arch aarch64` is a cache pull, not a native build, and works on a cheap x86_64 Linux runner (exactly like the kernel). Materialize both in one `artifacts` job; the self-hosted runner only boots. - Generalize scripts/fetch-virtio-kernel.sh -> fetch-artifact.sh <output> <dest>. - Merge the virtio-kernel + minvmd-rootfs jobs into one Linux `artifacts` job. - Drop the self-hosted minvmd-rootfs materialize job and its shim /host large-file read workaround. - Point boot-e2e/autospawn-e2e at `needs: [artifacts]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

~67 ms was a one-off from a less-loaded machine; the reproducible CI bench is ~113 ms median (runner-dependent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… rootfs) minimald ships as the initramfs /init (a ~27 MB cpio) and serves a full session against the GENERIC upstream microvm-rootfs — no minimald baked into the rootfs, no local rootfs package, no data disk, no mke2fs. - minvmd: VmConfig.initramfs + krun_set_kernel initramfs arg (MINVMD_INITRAMFS); attaches the rootfs as /dev/vda for the initramfs /init to mount. Also carries the vsock guest-mode plumbing (data disk, etc.) for the block-root path. - minimald: runs as /init (detect via argv[0]); mounts devtmpfs; mounts /dev/vda + chroots into the rootfs so /bin/sh + socat resolve; serves over the socat vsock->UDS relay + run_on_uds with tmpfs (/run/minimal) state. - Session e2e (minimald_exec_over_bridge) booted via initramfs against the generic upstream microvm-rootfs: exec stdout correct, exit 0. Stacks on the generic-rootfs Stage 1 (#367); supersedes the local-rootfs integration (#362). Persistent state (data disk + mke2fs) is a follow-up; the session runs on tmpfs here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

evanspearman · 2026-06-09T19:55:35Z


+    /// Add a disk image as a virtio-blk block device backing `block_id`.
+    /// `disk_format` is one of `raw::KRUN_DISK_FORMAT_*`.
+    pub fn add_disk(


Not requesting a change here, just want to point out a convention that's common for these sorts of interfaces in Rust that you might want to consider in the future.

Instead of something like:

let mut thing = Thing::new(); thing.set_whatever(whatever)?; thing.set_otherthing(otherthing)?; thing.set_infallible(infallible);

Something like the following usage is preferred:

let thing = Thing::new() .with_whatever(whatever)? .with_otherthing(otherthing)? .with_infallible();

Or in cases where there are invariants that involve multiple fields:

let thing = Thing::builder() .with_whatever(whatever) .with_otherthing(otherthing) .with_infallible(infallible) .build()?;

There are a few different ways to implement this

Consuming immutable form:

pub fn with_whatever(self, whatever: Whatever) -> Result<Self, Error> { Self { whatever, ..self } }

Mutable reference form:

pub fn with_whatever(&mut self, whatever: Whatever) -> Result<&mut Self, Error> { self.whatever = whatever; self }

And other forms like consuming mutable, etc. that can be inferred.

The immutable form may seem less efficient, but the compiler actually ends up doing the same thing under the hood for these, so generally I recommend the consuming immutable form unless there's another reason not to.

Again, not asking for a change here, just wanted to point out this convention.

evanspearman · 2026-06-09T20:04:04Z

+        disk_format: u32,
+        read_only: bool,
+    ) -> Result<(), VmError> {
+        let id_cstr = cstring_from_str(block_id, "block_id")?;


FYI: you can do:

let id_cstr = c"block_id";

Or in cases where you need an owned type rather than a static reference:

let id_cstr = CString::from(c"block_id");

evanspearman · 2026-06-09T20:11:09Z

+    pub fn add_disk(
+        &mut self,
+        block_id: &str,
+        disk_path: impl AsRef<Path>,


Minor, but for paths we should move towards using the path newtypes in our paths crate which enforce some invariants at compile time. They're also based on camino::Utf8Path so we don't need to worry about non-utf8 characters.

evanspearman · 2026-06-09T20:13:47Z

 // - x86_64  `virtio-linux` ships `bzImage`  → `KRUN_KERNEL_FORMAT_ELF`. libkrun
 //   loads bzImage as ELF; `IMAGE_BZ2` (=3) is bzip2 blobs, not the bzImage
 //   container format.
+pub const KRUN_KERNEL_FORMAT_RAW: u32 = 0;


Just wanted to point out that you can do:

#[repr(i32)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum MyType { Whatever = 0, Variants = 1, You = 2, Need = 3, }

Not sure if that's the right call for this case or not, but in cases where you want some additional type safety this is an available tool.

Replace the bare `KRUN_KERNEL_FORMAT_*` / `KRUN_DISK_FORMAT_*` u32 constants with `#[repr(u32)]` enums (`KernelFormat`, `DiskFormat`), so `set_kernel`/`add_disk` can only be handed a known format. The FFI boundary still passes `format as u32`. Per Evan's review on #367. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Ship minimald as the kernel's initramfs `/init`: `build-initramfs.sh` cross-compiles the static aarch64 binary and packs it as a newc cpio. The kernel boots the initramfs and runs `/init` (minimald) as pid-1; minimald mounts `/dev` (devtmpfs), mounts the generic upstream rootfs (`/dev/vda`) + pseudo-fs, chroots in, and emits the boot READY marker. This replaces the block-root boot path (kernel `init=/sbin/<exec>` on an ext4 root): minimald is no longer baked into the rootfs, so the upstream `microvm-rootfs` package stays generic. Session state is on a tmpfs (`/run/minimal`); no data disk. - minvmd: `VmConfig` boots an initramfs (required) + the rootfs as a `/dev/vda` block device; drops `exec_target` and the block-root cmdline. `MINVMD_INITRAMFS` is the new required boot input. - minimald: `/init` detection (argv[0] basename) + `run_initramfs` (mount + READY + serve on the guest UDS); `guest` module mount helpers. - Type the libkrun kernel/disk formats as `#[repr(u32)]` enums (`KernelFormat`/`DiskFormat`) per review on #367. - CI: cross-build the initramfs on the Linux artifacts job; `boot_e2e` boots it with `MINVMD_INITRAMFS` and asserts READY. The vsock session round-trip (relay + `run_on_uds` serving) lands in a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Ship minimald as the kernel's initramfs `/init`: `build-initramfs.sh` cross-compiles the static aarch64 binary and packs it as a newc cpio. The kernel boots the initramfs and runs `/init` (minimald) as pid-1; minimald mounts `/dev` (devtmpfs), mounts the generic upstream rootfs (`/dev/vda`) + pseudo-fs, chroots in, and emits the boot READY marker. This replaces the block-root boot path (kernel `init=/sbin/<exec>` on an ext4 root): minimald is no longer baked into the rootfs, so the upstream `microvm-rootfs` package stays generic. Session state is on a tmpfs (`/run/minimal`); no data disk. - minvmd: `VmConfig` boots an initramfs (required) + the rootfs as a `/dev/vda` block device; drops `exec_target` and the block-root cmdline. `MINVMD_INITRAMFS` is the new required boot input. - minimald: `/init` detection (argv[0] basename) + `run_initramfs` (mount + READY + serve on the guest UDS); `guest` module mount helpers. - Type the libkrun kernel/disk formats as `#[repr(u32)]` enums (`KernelFormat`/`DiskFormat`) per review on #367. - CI: cross-build the initramfs on the Linux artifacts job; `boot_e2e` boots it with `MINVMD_INITRAMFS` and asserts READY. The vsock session round-trip (relay + `run_on_uds` serving) lands in a follow-up. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

norrietaylor and others added 19 commits June 6, 2026 17:02

docs(minvmd): correct the boot-latency figure in the README

5cf9e39

~67 ms was a one-off from a less-loaded machine; the reproducible CI bench is ~113 ms median (runner-dependent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

norrietaylor mentioned this pull request Jun 8, 2026

feat(minvmd,minimald): Stage 2 — minimald as pid-1 via initramfs #368

Closed

norrietaylor mentioned this pull request Jun 9, 2026

feature: minvmd VM provider host daemon #311

Closed

5 tasks

norrietaylor enabled auto-merge (squash) June 9, 2026 06:33

evanspearman approved these changes Jun 9, 2026

View reviewed changes

norrietaylor merged commit 00c1dc8 into main Jun 9, 2026
21 checks passed

norrietaylor deleted the minvmd-rootfs-as-package branch June 9, 2026 20:15

norrietaylor mentioned this pull request Jun 9, 2026

feat(minvmd,minimald): boot minimald as pid-1 via initramfs #373

Merged

norrietaylor mentioned this pull request Jun 12, 2026

[Minimal One] Linux host support for minvmd — run sessions in a microVM on Linux #396

Closed

gominimal-aw-bot Bot mentioned this pull request Jun 12, 2026

docs(spec-minvmd-linux-kvm): Linux host support for minvmd (KVM backend + vsock bridge) #399

Merged

Navigation Menu

Uh oh!

feat(minvmd): boot a microVM from kernel + rootfs built as minimal packages - #367

feat(minvmd): boot a microVM from kernel + rootfs built as minimal packages#367
norrietaylor merged 19 commits into
mainfrom
minvmd-rootfs-as-package

norrietaylor commented Jun 8, 2026

Uh oh!

evanspearman Jun 9, 2026

Uh oh!

evanspearman Jun 9, 2026

Uh oh!

evanspearman Jun 9, 2026

Uh oh!

evanspearman Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

norrietaylor commented Jun 8, 2026

What changed

How to run on Mac

Uh oh!

evanspearman Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

evanspearman Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

evanspearman Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

evanspearman Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants