Skip to content

feat(minvmd,minimald): Stage 2 — minimald as pid-1 via initramfs - #368

Closed
norrietaylor wants to merge 2 commits into
minvmd-stage2-minimald-pid1from
minvmd-stage2-initramfs
Closed

feat(minvmd,minimald): Stage 2 — minimald as pid-1 via initramfs#368
norrietaylor wants to merge 2 commits into
minvmd-stage2-minimald-pid1from
minvmd-stage2-initramfs

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jun 8, 2026

Copy link
Copy Markdown
Member

Stacks on #367 (Stage 1, generic upstream rootfs). Supersedes #362 (the local-rootfs integration).

Approach

minimald runs the guest as pid-1, shipped as the initramfs /init (a ~27 MB cpio of the cross-compiled static binary) — not baked into the rootfs. The guest root stays the generic upstream microvm-rootfs. minimald mounts /dev, mounts the rootfs (/dev/vda), chroots in so /bin/sh + socat resolve, then serves a session over the bridge (socat vsock→UDS relay + run_on_uds, tmpfs state at /run/minimal).

Why this over the local-rootfs approach (#362)

local rootfs (#362) initramfs (this PR)
guest root local minvmd-rootfs package generic upstream microvm-rootfs
minimald delivery debugfs-baked into the image cpio via krun_set_kernel
CI materialize self-hosted runner rootfs cache-pulls on ubuntu
divergence from #367 toml/CI/README fork none
boot-to-READY ~82 ms ~76 ms

Validated locally

minimald_exec_over_bridge booted via initramfs against the generic upstream microvm-rootfs (no minimald, no mke2fs in it): CreateSession + exec → stdout="MINIMALD_SESSION_OK" exit=0. Boot-to-READY ~76 ms median (n=10), on par with block-root.

CI

artifacts job (ubuntu) builds the kernel (cache pull) + the initramfs (cross-compile minimald → cpio) and uploads both; boot-e2e runs the initramfs session e2e on the self-hosted runner.

Follow-up (not blocking, flagged)

Session state is on tmpfs (ephemeral). A persistent host key + session store needs the data disk, which needs mke2fs to format (the generic rootfs excludes e2fsprogs). Options: ship a small mke2fs in the initramfs, a Rust ext4 formatter, or host-side pre-format. The session itself is proven.

Draft until CI is green on the runner and the persistence decision is made.

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

  • New Features

    • Extended macOS CI to build and publish guest initramfs artifact alongside kernel and rootfs images.
    • Added initramfs boot mode support for guest VMs with writable state disk provisioning.
    • Added persistent data disk support for guest virtual machines.
    • Extended macOS E2E test coverage with new session-based tests.
  • Bug Fixes

    • Session/handshake failures on UDS transport are now logged instead of silently ignored.
  • Improvements

    • Replaced kernel and disk format constants with strongly-typed enums for better type safety.
    • Enhanced guest boot flow with improved rootfs mounting and vsock-to-UDS relay mechanism.
  • Documentation

    • Updated documentation with Stage 2 initramfs workflow and build/boot instructions.

@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a980c322-3428-454a-b360-8a83068156b4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • ✅ Review completed - (🔄 Check again to review again)

Comment @coderabbitai help to get the list of available commands and usage tips.

Stacks on #361 (minimald vsock guest mode). Ships minimald as the initramfs
/init (a cpio of the cross-compiled static binary) and serves a full session
against the GENERIC upstream microvm-rootfs — no minimald baked into the rootfs.

- minimald: run as /init (detect via argv[0]); mount devtmpfs; mount /dev/vda +
  chroot into the rootfs so /bin/sh + socat resolve; serve over a socat
  vsock->UDS relay + run_on_uds with tmpfs (/run/minimal) state. The block-root
  guest path gains a writable data disk + format-on-first-boot.
- minvmd: VmConfig.initramfs + krun_set_kernel initramfs arg (MINVMD_INITRAMFS);
  provision + attach the rw data disk.
- scripts/build-initramfs.sh + CI: cross-compile minimald with a fast `initramfs`
  profile, pack the cpio, cache the aarch64 target, and run the initramfs session
  e2e against the generic rootfs.

Validated: minimald_exec_over_bridge passes booted via initramfs — exec stdout
correct, exit 0. Boot-to-READY ~76 ms median, on par with block-root.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@norrietaylor
norrietaylor force-pushed the minvmd-stage2-initramfs branch from c1db7cd to 294c8bc Compare June 9, 2026 05:20
@norrietaylor
norrietaylor changed the base branch from main to minvmd-stage2-minimald-pid1 June 9, 2026 05:20
Replace the bare `KRUN_KERNEL_FORMAT_*` / `KRUN_DISK_FORMAT_*` u32
constants with `#[repr(u32)]` enums (`KernelFormat`, `DiskFormat`), so
`set_kernel`/`add_disk` can only be handed a known format. The FFI
boundary still passes `format as u32`.

Per Evan's review on #367.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (4)
crates/minimald/src/guest.rs (1)

82-94: 💤 Low value

Consider logging the initial mount error for diagnostics.

The function formats the disk on any mount failure, which is pragmatic for the controlled VM environment. However, logging the specific error from the first mount attempt would aid debugging when the failure isn't due to a blank disk (e.g., device missing, wrong device type).

💡 Optional improvement
 pub fn mount_state_disk(device: &str, target: &str) -> std::io::Result<()> {
     std::fs::create_dir_all(target)?;
-    if raw_mount_ext4(device, target).is_ok() {
+    if let Err(e) = raw_mount_ext4(device, target) {
+        tracing::info!(device, error = %e, "state disk mount failed; formatting ext4");
+        format_ext4(device)?;
+        raw_mount_ext4(device, target)?;
+        tracing::info!(device, target, "formatted + mounted state disk");
+    } else {
         tracing::info!(device, target, "mounted state disk");
-        return Ok(());
     }
-    // Likely blank on first boot — format ext4, then retry.
-    tracing::info!(device, "state disk mount failed; formatting ext4");
-    format_ext4(device)?;
-    raw_mount_ext4(device, target)?;
-    tracing::info!(device, target, "formatted + mounted state disk");
     Ok(())
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/guest.rs` around lines 82 - 94, In mount_state_disk,
capture the Result/error returned from the first raw_mount_ext4(device, target)
attempt and include that error in the log before formatting; specifically,
replace the current is_ok() check by matching or binding the Err to a variable
(from the first raw_mount_ext4 call) and pass that error into tracing::info/warn
so the message includes the actual mount failure (referencing mount_state_disk
and raw_mount_ext4), then proceed to format_ext4(device) and retry as before.
crates/minimald/src/main.rs (1)

297-303: 💤 Low value

Consider checking for specific errors when removing stale socket.

let _ = std::fs::remove_file(&guest_uds) ignores all errors, which could mask issues like permission denied or the path being a directory. While unlikely at early boot, checking for NotFound specifically is slightly safer.

💡 Optional improvement
-    let _ = std::fs::remove_file(&guest_uds);
+    if let Err(e) = std::fs::remove_file(&guest_uds) {
+        if e.kind() != std::io::ErrorKind::NotFound {
+            tracing::warn!(error = %e, "failed to remove stale guest UDS");
+        }
+    }

This same pattern appears at line 349 in run_guest.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/main.rs` around lines 297 - 303, The removal of the stale
UDS currently swallows all errors via let _ = std::fs::remove_file(&guest_uds),
which can hide real issues; update the code around guest_uds (the remove_file
call used before UnixListener::bind and the analogous call in run_guest) to
explicitly handle errors by matching the result: treat
std::io::ErrorKind::NotFound as OK (ignore) but propagate or return other errors
(e.g., wrap them with MainError::IO) so permission/invalid-path errors are not
silently ignored before calling UnixListener::bind or guest::spawn_vsock_relay.
scripts/build-initramfs.sh (1)

27-27: 💤 Low value

Consider removing unused staging directories.

The script creates dev, proc, sys, and newroot directories but only uses the staging root to place the init binary. These empty directories are included in the cpio archive but aren't referenced by the initramfs code.

🧹 Suggested simplification
 STAGE="$(mktemp -d)"
 trap 'rm -rf "$STAGE"' EXIT
-mkdir -p "$STAGE/dev" "$STAGE/proc" "$STAGE/sys" "$STAGE/newroot"
 cp "$BIN" "$STAGE/init"
 chmod +x "$STAGE/init"

If these directories are planned for future use (e.g., pre-creating mount points), consider adding a comment explaining their purpose.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/build-initramfs.sh` at line 27, Remove or document the unused staging
directories created by the mkdir call: if dev, proc, sys, and newroot under the
STAGE variable are not used by the initramfs (they only add empty entries to the
cpio archive), delete them from the mkdir invocation so only the necessary
staging root and any required directories (e.g., where the init binary is
placed) are created; alternatively, if you intend to keep them as pre-created
mount points, add a short comment next to the mkdir "$STAGE/..." line explaining
their purpose (reference: the mkdir invocation that creates "$STAGE/dev"
"$STAGE/proc" "$STAGE/sys" "$STAGE/newroot" and the STAGE variable).
.github/workflows/ci-macos.yml (1)

78-81: 💤 Low value

Consider pinning the cross version for reproducible CI builds.

The taiki-e/install-action will install the latest cross by default. If cross releases a breaking change, CI builds could become non-reproducible or fail unexpectedly.

📌 Suggested change to pin cross version
       - name: Install cross
         uses: taiki-e/install-action@v2
         with:
-          tool: cross
+          tool: cross@0.2.5  # or latest stable version
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci-macos.yml around lines 78 - 81, The GitHub Actions step
named "Install cross" currently uses taiki-e/install-action@v2 with only "tool:
cross", which installs the latest cross and risks non-reproducible CI; update
that step to pin the cross tool to a specific release by adding a version input
(e.g., add a "version: 'X.Y.Z'" field) to the taiki-e/install-action invocation
so the step installs a fixed cross version instead of the latest.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/minvmd/src/bin/krun_smoke_child.rs`:
- Around line 69-73: The kernel format selection sets KernelFormat::ImageGz for
aarch64 which is unsupported; change the conditional in krun_smoke_child.rs
where you assign the variable format so that if cfg!(target_arch = "aarch64")
you use KernelFormat::Raw instead of KernelFormat::ImageGz, otherwise keep the
existing non-aarch64 branch (KernelFormat::Elf); update the assignment that
produces format (the KernelFormat enum usage) so aarch64 loads the uncompressed
RAW kernel.

---

Nitpick comments:
In @.github/workflows/ci-macos.yml:
- Around line 78-81: The GitHub Actions step named "Install cross" currently
uses taiki-e/install-action@v2 with only "tool: cross", which installs the
latest cross and risks non-reproducible CI; update that step to pin the cross
tool to a specific release by adding a version input (e.g., add a "version:
'X.Y.Z'" field) to the taiki-e/install-action invocation so the step installs a
fixed cross version instead of the latest.

In `@crates/minimald/src/guest.rs`:
- Around line 82-94: In mount_state_disk, capture the Result/error returned from
the first raw_mount_ext4(device, target) attempt and include that error in the
log before formatting; specifically, replace the current is_ok() check by
matching or binding the Err to a variable (from the first raw_mount_ext4 call)
and pass that error into tracing::info/warn so the message includes the actual
mount failure (referencing mount_state_disk and raw_mount_ext4), then proceed to
format_ext4(device) and retry as before.

In `@crates/minimald/src/main.rs`:
- Around line 297-303: The removal of the stale UDS currently swallows all
errors via let _ = std::fs::remove_file(&guest_uds), which can hide real issues;
update the code around guest_uds (the remove_file call used before
UnixListener::bind and the analogous call in run_guest) to explicitly handle
errors by matching the result: treat std::io::ErrorKind::NotFound as OK (ignore)
but propagate or return other errors (e.g., wrap them with MainError::IO) so
permission/invalid-path errors are not silently ignored before calling
UnixListener::bind or guest::spawn_vsock_relay.

In `@scripts/build-initramfs.sh`:
- Line 27: Remove or document the unused staging directories created by the
mkdir call: if dev, proc, sys, and newroot under the STAGE variable are not used
by the initramfs (they only add empty entries to the cpio archive), delete them
from the mkdir invocation so only the necessary staging root and any required
directories (e.g., where the init binary is placed) are created; alternatively,
if you intend to keep them as pre-created mount points, add a short comment next
to the mkdir "$STAGE/..." line explaining their purpose (reference: the mkdir
invocation that creates "$STAGE/dev" "$STAGE/proc" "$STAGE/sys" "$STAGE/newroot"
and the STAGE variable).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7f948338-bc9f-4dc8-b91d-85738654c7de

📥 Commits

Reviewing files that changed from the base of the PR and between 40abc9e and 2c22555.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (17)
  • .github/workflows/ci-macos.yml
  • Cargo.toml
  • crates/minimald/src/guest.rs
  • crates/minimald/src/main.rs
  • crates/minimald/src/server.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/src/bin/krun_smoke_child.rs
  • crates/minvmd/src/cmd/vmm_child.rs
  • crates/minvmd/src/image.rs
  • crates/minvmd/src/krun/ctx.rs
  • crates/minvmd/src/krun/mod.rs
  • crates/minvmd/src/krun/raw.rs
  • crates/minvmd/src/state.rs
  • crates/minvmd/src/vm.rs
  • crates/minvmd/tests/minimald_session_e2e.rs
  • scripts/build-initramfs.sh

Comment on lines 69 to 73
let format = if cfg!(target_arch = "aarch64") {
KRUN_KERNEL_FORMAT_IMAGE_GZ
KernelFormat::ImageGz
} else {
KRUN_KERNEL_FORMAT_ELF
KernelFormat::Elf
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify aarch64 kernel format usage across the codebase

echo "=== Checking image.rs kernel_format() implementation ==="
rg -n -A 3 -B 3 'target_arch = "aarch64"' crates/minvmd/src/image.rs

echo ""
echo "=== Checking raw.rs documentation on ImageGz support ==="
rg -n -C 2 'ImageGz.*x86_64' crates/minvmd/src/krun/raw.rs

echo ""
echo "=== Checking for other aarch64 kernel format usage ==="
rg -n 'KernelFormat::(Raw|ImageGz|Elf|PeGz)' crates/minvmd/src/ --type rust

Repository: gominimal/minimal

Length of output: 1897


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect krun_smoke_child.rs around lines 55-90 ==="
nl -ba crates/minvmd/src/bin/krun_smoke_child.rs | sed -n '55,95p'
echo
echo "=== Search for usage of variable `format` in this file ==="
rg -n "let format|\\bformat\\b" crates/minvmd/src/bin/krun_smoke_child.rs

Repository: gominimal/minimal

Length of output: 177


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect crates/minvmd/src/bin/krun_smoke_child.rs around lines 55-95 (cat -n) ==="
cat -n crates/minvmd/src/bin/krun_smoke_child.rs | sed -n '55,95p'

echo
echo "=== Search for usages of variable `format` in this file ==="
rg -n --no-heading "\\bformat\\b" crates/minvmd/src/bin/krun_smoke_child.rs

Repository: gominimal/minimal

Length of output: 2009


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect crates/minvmd/src/bin/krun_smoke_child.rs (top + kernel selection) ==="
cat -n crates/minvmd/src/bin/krun_smoke_child.rs | sed -n '1,160p'

echo
echo "=== Inspect crates/minvmd/src/image.rs (kernel_format + how kernel file is prepared) ==="
cat -n crates/minvmd/src/image.rs | sed -n '1,220p'

echo
echo "=== Locate KernelFormat enum in crates/minvmd/src/krun ==="
rg -n "enum KernelFormat|KernelFormat::" crates/minvmd/src/krun -S

Repository: gominimal/minimal

Length of output: 9809


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect crates/minvmd/src/krun/raw.rs around KernelFormat + ImageGz docs ==="
rg -n "ImageGz" crates/minvmd/src/krun/raw.rs
echo
cat -n crates/minvmd/src/krun/raw.rs | sed -n '1,120p'

Repository: gominimal/minimal

Length of output: 6565


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Find where MINVMD_KERNEL_PATH is set/used (tests + minvmd) ==="
rg -n "MINVMD_KERNEL_PATH" -S .

echo
echo "=== Inspect tests/krun_smoke.rs ==="
ls -la tests || true
cat -n tests/krun_smoke.rs 2>/dev/null || true

Repository: gominimal/minimal

Length of output: 4146


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect crates/minvmd/tests/krun_smoke.rs ==="
cat -n crates/minvmd/tests/krun_smoke.rs | sed -n '1,220p'
echo
echo "=== Search in crates/minvmd/tests for Image.gz / vmlinuz / uncompressed kernel ==="
rg -n "Image\.gz|vmlinuz|uncompressed|gunzip|ImageGz|KernelFormat::Raw|KernelFormat::ImageGz" crates/minvmd/tests crates/minvmd/src -S

Repository: gominimal/minimal

Length of output: 6102


Fix aarch64 kernel format selection in krun_smoke_child.rs

File: crates/minvmd/src/bin/krun_smoke_child.rs (lines 69-73)

The smoke child selects KernelFormat::ImageGz for aarch64, but minvmd/src/image.rs states the aarch64 kernel is shipped uncompressed and should be loaded with KernelFormat::Raw. Also, minvmd/src/krun/raw.rs documents that ImageGz (=4) is x86_64-only and returns KernelFormatUnsupported on aarch64 (aarch64 loader implements only RAW and PE_GZ). This mismatch can make the smoke test fail on aarch64.

let format = if cfg!(target_arch = "aarch64") {
    KernelFormat::ImageGz
} else {
    KernelFormat::Elf
};
🔧 Proposed fix
     let format = if cfg!(target_arch = "aarch64") {
-        KernelFormat::ImageGz
+        KernelFormat::Raw
     } else {
         KernelFormat::Elf
     };
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minvmd/src/bin/krun_smoke_child.rs` around lines 69 - 73, The kernel
format selection sets KernelFormat::ImageGz for aarch64 which is unsupported;
change the conditional in krun_smoke_child.rs where you assign the variable
format so that if cfg!(target_arch = "aarch64") you use KernelFormat::Raw
instead of KernelFormat::ImageGz, otherwise keep the existing non-aarch64 branch
(KernelFormat::Elf); update the assignment that produces format (the
KernelFormat enum usage) so aarch64 loads the uncompressed RAW kernel.

@norrietaylor

Copy link
Copy Markdown
Member Author

Closing in favor of #373 + #374, which re-slice this work along capability lines instead of delivery-mechanism lines:

The block-root boot path (init=/sbin/minimald on an ext4 root) is dropped — it was a stepping stone; the initramfs delivery keeps the upstream microvm-rootfs package generic. Rationale on #373.

@norrietaylor
norrietaylor deleted the minvmd-stage2-initramfs branch June 19, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant