[WIP] feat(minvmd): boot from an ext4 guest image built as a minimal package by norrietaylor · Pull Request #359 · gominimal/minimal

norrietaylor · 2026-06-07T00:03:36Z

Makes the minvmd guest kernel and rootfs minimal packages, switches the boot path to a read-only ext4 block device, and ships the kernel uncompressed for a ~2× faster boot. Stage 1 of the plan; Stage 2 (minimald-as-pid-1) is #361.

What changed

Rootfs as a package — .minimal/packages/minvmd-rootfs/ builds the guest rootfs as an ext4 image from upstream socat/bash/coreutils (+e2fsprogs for mke2fs), replacing scripts/build-rootfs.sh + scripts/fetch-alpine.sh (deleted). No journal (-O ^has_journal) since the root is read-only.
Block-device root — minvmd loads the rootfs via krun_add_disk2 as /dev/vda (was virtiofs krun_set_root). A block root has no libkrun /init.krun, so the exec target moves to the kernel init= cmdline. Adds the krun_add_disk2 binding + wrapper and VmConfig.exec_target.
Kernel from source, uncompressed — scripts/fetch-virtio-kernel.sh builds the minimal CLI from this repo, materializes virtio-kernel against the repo's own minimal.toml (no promoted-CLI download, no minimal update), and gunzips the Image.gz so minvmd loads it raw (KRUN_KERNEL_FORMAT_RAW).
Lock bump — upstream gominimal/pkgs 0da02c7 → 133db0c (adds virtio-linux; 52-commit fast-forward).
CI — ci-macos.yml: kernel materialized on a Linux runner; rootfs on the self-hosted aarch64 runner (the build container does not emulate, so aarch64 packages need a native builder); both handed to boot jobs as artifacts.

Performance — ~2× faster boot

Profiled boot-to-READY (N=10, Apple Silicon) with scripts/bench-minvmd-boot.sh (added here):

change	median boot→READY
baseline (gzip `Image.gz`, PE_GZ)	146 ms
uncompressed `Image` (RAW)	67 ms

libkrun's in-VMM gzip decompress was ~77 ms — over half the boot. Loading the kernel raw skips it. Artifact grows 14 → 38 MB (trivial vs per-boot latency).

Measured and ruled out as noise: quiet/loglevel cmdline, vCPU count (1 vs 2), RAM (512 vs 1024) — boot is serial; printk and SMP bringup don't matter at this size.

Root-shape choice (ext4 vs squashfs vs virtiofs) was also measured: all ~150 ms before the kernel fix, so backend is boot-noise. ext4 chosen (squashfs kernel decompressor is zlib-only; ext4 is read-write for Stage 2 state).

Otel? Not for this — a single ~150 ms minvmd → __krun-vmm → guest chain; SDK/exporter overhead would distort the measurement. Lightweight Instant/tracing + kernel printk timestamps suffice. Reserve Otel for production minimald-session observability.

Validation (local, Apple Silicon)

minimal materialize --arch aarch64 minvmd-rootfs → 95 MB ext4 image, built from upstream packages.
Boot from /dev/vda ext4: devtmpfs mounted → Run /sbin/minvmd-stub-init → vm-up.
boot_e2e + bridge_e2e (R2.4 + R3.3/R3.4) pass with no env toggle.
minimal2 ls cold-autospawn + warm + minvmd stop (R4.5).
cargo clippy -p minvmd --all-targets -- -D warnings, cargo fmt, unit tests clean.
scripts/bench-minvmd-boot.sh: median ~67 ms boot-to-READY.

CR-loop fixes (this PR)

virtio-kernel CI was failing (cannot build spec with target arm64/linux on amd64/linux) — the kernel decompression was a package forcing a cross-arch build. Moved to gunzip in the script (arch-agnostic).
minimal-check — fixed build.ncl import order (canonical: lowercase first).
minvmd-rootfs CI — a poisoned cache entry (a prior build produced no rootfs.img yet got cached); added a mke2fs precondition + output assertion (fail loud, not silent) and the journal/headroom fix (the likely cause).

Caveats

Stage 2 deferred (feat(minimald): vsock guest (pid-1) mode for minvmd #361) — exec target remains the bring-up stub; minimald-as-pid-1 is scoped there.
Spec docs (docs/specs/01-spec-minvmd-host-daemon/) still describe the old Alpine//init.krun/virtiofs model — follow-up doc update.

🤖 Generated with Claude Code

Define the minvmd guest rootfs as a minimal package instead of the shell-script Alpine overlay. build.sh snapshots the closure of its build_deps (socat + bash + coreutils + e2fsprogs), drops in the bring-up init and the /etc/minvmd/manifest contract, prunes build-only bulk, and packs an ext4 image with mke2fs. Cross-layer deps are referenced via `upstream "<name>"`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Bump the upstream gominimal/pkgs lock 0da02c7 -> 133db0c (the commit that adds the virtio-linux package; a 52-commit fast-forward) and add the raw-file outputs minvmd materializes: virtio-kernel (the guest kernel) and minvmd-rootfs (the guest ext4 image). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Switch the guest root from a virtiofs directory (krun_set_root) to a read-only ext4 block device. A block root has no libkrun /init.krun, so the kernel runs the workload directly via an explicit cmdline `root=/dev/vda rootfstype=ext4 ro init=<exec-target>`; the exec target (MINVMD_EXEC, default /sbin/minvmd-stub-init) moves from krun_set_exec into init=. Add the krun_add_disk2 binding + safe wrapper and an exec_target field on VmConfig. MINVMD_ROOTFS_PATH now resolves a file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…source Rewrite fetch-virtio-kernel.sh to build the minimal CLI from this repo (cargo build -p minimal) and materialize virtio-kernel against the repo's own minimal.toml, instead of downloading the promoted CLI from gs://minimal-shim. Drop `minimal update` (it would rewrite the tracked locked_commit and risk an uncached, full kernel build) and the scratch project. Linux-only, with a guard pointing at crates/minvmd/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rewire ci-macos.yml: the virtio-kernel job builds minimal from source (toolchain + protoc + rust-cache); a new minvmd-rootfs job materializes the ext4 image on the self-hosted aarch64 runner (the build container does not emulate, so aarch64 packages need a native builder); boot-e2e and autospawn-e2e consume both as artifacts. Delete fetch-alpine.sh and build-rootfs.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add crates/minvmd/README.md (materialize kernel + rootfs into .scratch/, build, codesign-last, run minimal2 ls; E2E invocation; boot mechanics) and gitignore .scratch/. Closes the references from the kernel script and the CI error messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-06-07T00:03:43Z

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title '[WIP] feat(minvmd): boot from an ext4 guest image built as a minimal package' accurately describes the primary change: switching minvmd to boot from an ext4 guest root filesystem image that is built as a minimal package, which is the core architectural change detailed throughout the PR.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.github/workflows/ci-macos.yml (1)

10-25: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Include .minimal/** in workflow path filters.

Lines 10-25 currently omit .minimal/**, but this workflow now depends on .minimal/minimal.toml and .minimal/packages/minvmd-rootfs/** for kernel/rootfs artifact production. PRs that only change those files can bypass this macOS gate.

Suggested fix

   push:
     branches: ["main"]
     paths:
+      - ".minimal/**"
       - "crates/minvmd/**"
       - "crates/minimal2/**"
       - "Cargo.toml"
       - "Cargo.lock"
       - ".github/workflows/ci-macos.yml"
       - "scripts/**"
   pull_request:
     branches: ["main"]
     paths:
+      - ".minimal/**"
       - "crates/minvmd/**"
       - "crates/minimal2/**"
       - "Cargo.toml"
       - "Cargo.lock"
       - ".github/workflows/ci-macos.yml"
       - "scripts/**"

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci-macos.yml around lines 10 - 25, Update the path filters
in .github/workflows/ci-macos.yml to include the new .minimal/** files so PRs
touching .minimal/minimal.toml or .minimal/packages/minvmd-rootfs/** trigger the
workflow; specifically add ".minimal/**" to both the push and pull_request paths
arrays alongside the existing entries (e.g., the blocks currently listing
"crates/minvmd/**", "crates/minimal2/**", "Cargo.toml", "Cargo.lock",
".github/workflows/ci-macos.yml", "scripts/**").

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.minimal/packages/minvmd-rootfs/build.ncl:
- Line 1: The import line's identifier order violates the repository's canonical
import ordering rule; update the let import in
.minimal/packages/minvmd-rootfs/build.ncl so the imported identifiers
(BuildSpec, Local, OutputData, upstream, ..) are sorted into the canonical order
enforced by the checker (reorder the names in the destructuring list
accordingly) to make `minimal --no-fetch check` pass.

In @.minimal/packages/minvmd-rootfs/build.sh:
- Around line 71-76: The subshell that performs the prune step "( cd \"$STAGE\"
&& rm -rf ... && find . \( -name '*.a' -o -name '*.la' -o -name '*.o' \) -delete
2>/dev/null || true )" masks failures with the trailing "|| true"; remove that
"|| true" so errors (from cd, rm, or find) propagate as failures, or if you only
want to ignore find stderr keep "2>/dev/null" but drop the final "|| true" so
the subshell fails on real errors.

In `@crates/minvmd/src/cmd/vmm_child.rs`:
- Around line 50-55: Normalize and validate the MINVMD_EXEC value before using
it as the kernel init: read the env var into exec, call trim() and treat an
empty/blank result as unset (fall back to "/sbin/minvmd-stub-init"), and if the
trimmed value contains any whitespace characters reject it (return an error or
exit) instead of accepting it; update the code that constructs exec in
vmm_child.rs and ensure VmConfig::apply receives only the validated,
non-whitespace init string.

---

Outside diff comments:
In @.github/workflows/ci-macos.yml:
- Around line 10-25: Update the path filters in .github/workflows/ci-macos.yml
to include the new .minimal/** files so PRs touching .minimal/minimal.toml or
.minimal/packages/minvmd-rootfs/** trigger the workflow; specifically add
".minimal/**" to both the push and pull_request paths arrays alongside the
existing entries (e.g., the blocks currently listing "crates/minvmd/**",
"crates/minimal2/**", "Cargo.toml", "Cargo.lock",
".github/workflows/ci-macos.yml", "scripts/**").

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 33a7f3a4-1814-45b0-89a3-2090e624056b

📥 Commits

Reviewing files that changed from the base of the PR and between 35fe939 and 169daae.

📒 Files selected for processing (18)

.github/workflows/ci-macos.yml
.gitignore
.minimal/minimal.toml
.minimal/packages/minvmd-rootfs/build.ncl
.minimal/packages/minvmd-rootfs/build.sh
crates/minvmd/README.md
crates/minvmd/src/cmd/vmm_child.rs
crates/minvmd/src/image.rs
crates/minvmd/src/krun/ctx.rs
crates/minvmd/src/krun/mod.rs
crates/minvmd/src/krun/raw.rs
crates/minvmd/src/sock.rs
crates/minvmd/src/vm.rs
crates/minvmd/tests/boot_e2e.rs
crates/minvmd/tests/bridge_e2e.rs
scripts/build-rootfs.sh
scripts/fetch-alpine.sh
scripts/fetch-virtio-kernel.sh

💤 Files with no reviewable changes (2)

scripts/build-rootfs.sh
scripts/fetch-alpine.sh

libkrun's in-VMM gzip decompress of the aarch64 Image.gz cost ~77 ms of a ~146 ms boot-to-READY (measured N=10). Ship the kernel uncompressed and load it with KRUN_KERNEL_FORMAT_RAW, which skips the decompress: median boot-to-READY drops 146 -> ~67 ms. Add a local virtio-kernel-raw package that gunzips the upstream virtio-linux Image.gz, repoint the virtio-kernel output at it, and make RAW the aarch64 default in image.rs. The artifact grows 14 -> 38 MB, which is trivial next to per-boot latency. (Measured separately: quiet cmdline, vCPU count, and RAM size were all boot-noise.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ckage The virtio-kernel-raw package forced an arm64 build, which the x86_64 CI runner cannot do ("cannot build spec with target arm64/linux on amd64/linux") — the upstream virtio-linux kernel only worked there because it is a cache pull, not a build. Drop the package and gunzip the cache-pulled Image.gz in fetch-virtio-kernel.sh instead; gunzip is arch-agnostic, so it runs on any runner while still yielding the raw Image that KRUN_KERNEL_FORMAT_RAW loads (skipping libkrun's ~77 ms decompress). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`minimal check`'s import-line checker requires destructured minimal.ncl identifiers in canonical order (lowercase first, then uppercase). Reorder to `upstream, BuildSpec, Local, OutputData` so minimal-check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scripts/bench-minvmd-boot.sh times `minvmd boot` to the guest READY marker across N runs and reports min/median/max. This is the harness that found the gzip kernel decompress (~77 ms of ~146 ms) and verified the uncompressed-kernel fix (~67 ms). macOS-only; needs a codesigned minvmd and MINVMD_KERNEL_PATH/MINVMD_ROOTFS_PATH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The self-hosted runner cache-hit a minvmd-rootfs build entry that lacked rootfs.img (an earlier build produced no output yet got cached), so materialize failed with "copying output file ... rootfs.img: No such file". Add a `command -v mke2fs` precondition and a post-build assertion that the image exists and is non-empty (failing loudly instead of caching an empty result). The build.sh content change also changes the package's input hash, forcing a fresh build past the poisoned cache entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The image was sized tree + 5% + 4 MiB, but ext4's journal (~4 MiB+) plus inode-table metadata that `du` does not count can overflow that on a slightly larger closure — the likely cause of `mke2fs` producing no rootfs.img on the CI runner. The root mounts read-only, so the journal is pure overhead: build with `-O ^has_journal` and widen headroom to tree + 10% + 8 MiB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tiofs) The gating comments still described the old virtiofs path (PE_GZ load, virtio-fs root, /init.krun, Alpine, VIRTIO_FS). Update to the committed model: raw kernel load, ext4 block root via krun_add_disk2, kernel `init=` exec, VIRTIO_BLK/EXT4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The self-hosted runner fails to collect rootfs.img into the build cache despite a successful build (the build.sh output assertion does not trip), and it is not reproducible locally on the same shim version. Echo the shim version and, on failure, list the minvmd-rootfs output dir + most-recent cache dirs so the next run reveals whether the image was produced (→ a collection/disk issue) or not (→ a build issue). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Diagnostics proved the minvmd-rootfs build succeeds and writes the image to the host build cache (104 MB present), but `minimal materialize`'s copy step cannot read it back through the shim VM's /host mount on the self-hosted runner ("copying output file ... No such file" for a file that exists on the host) — every build, same CLI version that works locally. Work around it: the build still populates the cache, so on copy failure take the freshly-built image straight from ~/.cache/minimal/built. TODO: remove once the shim /host large-file read coherence issue is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Wire scripts/bench-minvmd-boot.sh into the macOS boot-e2e job so every run reports boot-to-READY min/median/max, and document it in the README. Non-gating (boot correctness is gated by the e2e tests); reuses the already-codesigned minvmd. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

norrietaylor · 2026-06-07T04:21:23Z

Blocked on gominimal/pkgs#229 — keep in draft until it lands

This PR currently carries the guest kernel + rootfs as local packages under .minimal/packages/ plus CI workarounds. The packages are being promoted upstream in gominimal/pkgs#229 (virtio-kernel-raw + microvm-rootfs). Take this PR out of draft only after #229 merges.

Integration steps (once #229 lands)

Bump the upstream lock — set .minimal/minimal.toml locked_commit to the gominimal/pkgs merge commit that includes virtio-kernel-raw and microvm-rootfs.
Use the upstream packages, drop the local copies
- [outputs.virtio-kernel] → packages = ["virtio-kernel-raw"], path = "usr/share/virtio-linux/Image".
- [outputs.minvmd-rootfs] → packages = ["microvm-rootfs"], path = "usr/share/microvm-rootfs/rootfs.img".
- delete .minimal/packages/minvmd-rootfs/.
- revert the gunzip-in-script step in scripts/fetch-virtio-kernel.sh (the virtio-kernel-raw package now produces the raw Image directly).
Point minvmd at the upstream package's generic names — MINVMD_EXEC default /sbin/minvmd-stub-init → /sbin/microvm-init; manifest path /etc/minvmd/manifest → /etc/microvm/manifest.
Simplify CI — with both packages upstream and in the remote cache, the virtio-kernel job cache-pulls virtio-kernel-raw and the rootfs can be cache-pulled too. Drop the self-hosted minvmd-rootfs materialize job and the build-cache copy workaround (added for a shim /host read issue on the runner — track that separately). Re-run boot-e2e / autospawn-e2e and the boot bench.
Take this PR out of draft.

…kages Now that gominimal/pkgs ships microvm-rootfs (ext4 guest rootfs) and virtio-kernel-raw (uncompressed kernel Image), drop the local stopgaps and dogfood the upstream packages: - minimal.toml: point [outputs.virtio-kernel] at virtio-kernel-raw (raw Image, no gunzip) and [outputs.minvmd-rootfs] at microvm-rootfs; bump locked_commit. - Delete the local .minimal/packages/minvmd-rootfs/ package. - fetch-virtio-kernel.sh: drop the gunzip step (the kernel is already raw). - Retarget the default guest exec /sbin/minvmd-stub-init -> /sbin/microvm-init (the upstream stub) and the manifest path; update docs + the CI cache path. Validated locally: both artifacts materialize from the upstream packages and boot_e2e + bridge_e2e pass against them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… job Now that the rootfs comes from the upstream microvm-rootfs package, its prebuilt aarch64 artifact is in the public cache — so `materialize --arch aarch64` is a cache pull, not a native build, and works on a cheap x86_64 Linux runner (exactly like the kernel). Materialize both in one `artifacts` job; the self-hosted runner only boots. - Generalize scripts/fetch-virtio-kernel.sh -> fetch-artifact.sh <output> <dest>. - Merge the virtio-kernel + minvmd-rootfs jobs into one Linux `artifacts` job. - Drop the self-hosted minvmd-rootfs materialize job and its shim /host large-file read workaround. - Point boot-e2e/autospawn-e2e at `needs: [artifacts]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

norrietaylor · 2026-06-08T18:28:11Z

Superseded by a fresh PR (same branch, clean description). Closing this WIP.

norrietaylor and others added 6 commits June 6, 2026 17:02

coderabbitai Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread .minimal/packages/minvmd-rootfs/build.ncl Outdated

Comment thread .minimal/packages/minvmd-rootfs/build.sh Outdated

Comment thread crates/minvmd/src/cmd/vmm_child.rs Outdated

norrietaylor mentioned this pull request Jun 7, 2026

feat(minimald): vsock guest (pid-1) mode for minvmd #361

Closed

norrietaylor and others added 9 commits June 6, 2026 19:58

norrietaylor mentioned this pull request Jun 7, 2026

add microvm-rootfs and virtio-kernel-raw packages gominimal/pkgs#229

Merged

norrietaylor mentioned this pull request Jun 7, 2026

[WIP] feat(minvmd,minimald): Stage 2 — minimald as guest pid-1 (e2e integration) #362

Closed

norrietaylor changed the title ~~feat(minvmd): boot from an ext4 guest image built as a minimal package~~ [WIP] feat(minvmd): boot from an ext4 guest image built as a minimal package Jun 7, 2026

norrietaylor force-pushed the minvmd-rootfs-as-package branch from 493315c to 33aaf2f Compare June 8, 2026 17:54

norrietaylor force-pushed the minvmd-rootfs-as-package branch from 8bb536d to 00f1c1e Compare June 8, 2026 18:00

norrietaylor closed this Jun 8, 2026

Navigation Menu

Uh oh!

[WIP] feat(minvmd): boot from an ext4 guest image built as a minimal package - #359

[WIP] feat(minvmd): boot from an ext4 guest image built as a minimal package#359
norrietaylor wants to merge 18 commits into
mainfrom
minvmd-rootfs-as-package

norrietaylor commented Jun 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

norrietaylor commented Jun 7, 2026

Uh oh!

norrietaylor commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

norrietaylor commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Performance — ~2× faster boot

Validation (local, Apple Silicon)

CR-loop fixes (this PR)

Caveats

Uh oh!

coderabbitai Bot commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

norrietaylor commented Jun 7, 2026

Blocked on gominimal/pkgs#229 — keep in draft until it lands

Integration steps (once #229 lands)

Uh oh!

norrietaylor commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

norrietaylor commented Jun 7, 2026 •

edited

Loading

coderabbitai Bot commented Jun 7, 2026 •

edited

Loading