[WIP] feat(minvmd,minimald): Stage 2 — minimald as guest pid-1 (e2e integration) - #362

Bump the upstream gominimal/pkgs lock 0da02c7 -> 133db0c (the commit that
adds the virtio-linux package; a 52-commit fast-forward) and add the
raw-file outputs minvmd materializes: virtio-kernel (the guest kernel)
and minvmd-rootfs (the guest ext4 image).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Switch the guest root from a virtiofs directory (krun_set_root) to a
read-only ext4 block device. A block root has no libkrun /init.krun, so
the kernel runs the workload directly via an explicit cmdline
`root=/dev/vda rootfstype=ext4 ro init=<exec-target>`; the exec target
(MINVMD_EXEC, default /sbin/minvmd-stub-init) moves from krun_set_exec
into init=. Add the krun_add_disk2 binding + safe wrapper and an
exec_target field on VmConfig. MINVMD_ROOTFS_PATH now resolves a file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…source

Rewrite fetch-virtio-kernel.sh to build the minimal CLI from this repo
(cargo build -p minimal) and materialize virtio-kernel against the repo's
own minimal.toml, instead of downloading the promoted CLI from
gs://minimal-shim. Drop `minimal update` (it would rewrite the tracked
locked_commit and risk an uncached, full kernel build) and the scratch
project. Linux-only, with a guard pointing at crates/minvmd/README.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rewire ci-macos.yml: the virtio-kernel job builds minimal from source
(toolchain + protoc + rust-cache); a new minvmd-rootfs job materializes
the ext4 image on the self-hosted aarch64 runner (the build container
does not emulate, so aarch64 packages need a native builder); boot-e2e
and autospawn-e2e consume both as artifacts. Delete fetch-alpine.sh and
build-rootfs.sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add crates/minvmd/README.md (materialize kernel + rootfs into .scratch/,
build, codesign-last, run minimal2 ls; E2E invocation; boot mechanics)
and gitignore .scratch/. Closes the references from the kernel script and
the CI error messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

libkrun's in-VMM gzip decompress of the aarch64 Image.gz cost ~77 ms of a
~146 ms boot-to-READY (measured N=10). Ship the kernel uncompressed and
load it with KRUN_KERNEL_FORMAT_RAW, which skips the decompress: median
boot-to-READY drops 146 -> ~67 ms.

Add a local virtio-kernel-raw package that gunzips the upstream
virtio-linux Image.gz, repoint the virtio-kernel output at it, and make
RAW the aarch64 default in image.rs. The artifact grows 14 -> 38 MB,
which is trivial next to per-boot latency. (Measured separately: quiet
cmdline, vCPU count, and RAM size were all boot-noise.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The build script panicked when 'git rev-parse' produced no output (for
example cross-compiling inside a container with no repo mounted, or
building from a release tarball), because strip_suffix("\n").unwrap()
hit None. Fall back to a GIT_HASH of "unknown" instead so the build
succeeds outside a git checkout. This unblocks the aarch64 cross-build
used for the minvmd guest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Make minimald capable of running as the guest pid-1 / kernel init=
target inside the minvmd microVM, replacing the socat stub init.

- Generalize Connection::from_socket into from_stream<S: AsyncRead +
  AsyncWrite + Unpin + Send + 'static>, keeping from_socket as a thin
  UDS wrapper. The russh stack was already transport-agnostic.
- Add Server::run_on_vsock(config, port): bind a tokio-vsock listener
  on CID_ANY and run the same accept -> from_stream -> JoinSet loop as
  the UDS path. The host-mediated vsock peer (net=none) is as trusted
  as the UDS peer, so connections are treated as local (Auth::Local).
- Add a new guest module (Linux-only) implementing the boot contract
  and pid-1 hygiene: emit the one-shot READY marker to host CID 2 port
  7350 with retry/backoff; mount /proc and /sys if absent (the kernel
  only auto-mounts devtmpfs on /dev); and reap orphaned children via a
  SIGCHLD waitpid(-1, WNOHANG) handler, since hakoniwa double-forks its
  namespace children.
- Wire up main.rs: '--guest' (or the MINIMALD_VSOCK_PORT env) selects
  guest mode, serving on '--vsock-port' (default 2222, the bridge port
  the host registers via krun_add_vsock_port2). The UDS path remains
  the default, unaffected, native-Linux behavior.

tokio-vsock is a Linux-only target dependency, so the macOS host build
is untouched. Verified by cross-compiling, clippy (-D warnings), and
testing minimald for aarch64-unknown-linux-musl.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ckage

The virtio-kernel-raw package forced an arm64 build, which the x86_64 CI
runner cannot do ("cannot build spec with target arm64/linux on
amd64/linux") — the upstream virtio-linux kernel only worked there because
it is a cache pull, not a build. Drop the package and gunzip the
cache-pulled Image.gz in fetch-virtio-kernel.sh instead; gunzip is
arch-agnostic, so it runs on any runner while still yielding the raw Image
that KRUN_KERNEL_FORMAT_RAW loads (skipping libkrun's ~77 ms decompress).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`minimal check`'s import-line checker requires destructured minimal.ncl
identifiers in canonical order (lowercase first, then uppercase). Reorder
to `upstream, BuildSpec, Local, OutputData` so minimal-check passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scripts/bench-minvmd-boot.sh times `minvmd boot` to the guest READY marker
across N runs and reports min/median/max. This is the harness that found
the gzip kernel decompress (~77 ms of ~146 ms) and verified the
uncompressed-kernel fix (~67 ms). macOS-only; needs a codesigned minvmd
and MINVMD_KERNEL_PATH/MINVMD_ROOTFS_PATH.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The self-hosted runner cache-hit a minvmd-rootfs build entry that lacked
rootfs.img (an earlier build produced no output yet got cached), so
materialize failed with "copying output file ... rootfs.img: No such
file". Add a `command -v mke2fs` precondition and a post-build assertion
that the image exists and is non-empty (failing loudly instead of caching
an empty result). The build.sh content change also changes the package's
input hash, forcing a fresh build past the poisoned cache entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The image was sized tree + 5% + 4 MiB, but ext4's journal (~4 MiB+) plus
inode-table metadata that `du` does not count can overflow that on a
slightly larger closure — the likely cause of `mke2fs` producing no
rootfs.img on the CI runner. The root mounts read-only, so the journal is
pure overhead: build with `-O ^has_journal` and widen headroom to
tree + 10% + 8 MiB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tiofs)

The gating comments still described the old virtiofs path (PE_GZ load,
virtio-fs root, /init.krun, Alpine, VIRTIO_FS). Update to the committed
model: raw kernel load, ext4 block root via krun_add_disk2, kernel `init=`
exec, VIRTIO_BLK/EXT4.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The self-hosted runner fails to collect rootfs.img into the build cache
despite a successful build (the build.sh output assertion does not trip),
and it is not reproducible locally on the same shim version. Echo the shim
version and, on failure, list the minvmd-rootfs output dir + most-recent
cache dirs so the next run reveals whether the image was produced (→ a
collection/disk issue) or not (→ a build issue).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Diagnostics proved the minvmd-rootfs build succeeds and writes the image
to the host build cache (104 MB present), but `minimal materialize`'s copy
step cannot read it back through the shim VM's /host mount on the
self-hosted runner ("copying output file ... No such file" for a file that
exists on the host) — every build, same CLI version that works locally.
Work around it: the build still populates the cache, so on copy failure
take the freshly-built image straight from ~/.cache/minimal/built. TODO:
remove once the shim /host large-file read coherence issue is fixed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Wire scripts/bench-minvmd-boot.sh into the macOS boot-e2e job so every run
reports boot-to-READY min/median/max, and document it in the README.
Non-gating (boot correctness is gated by the e2e tests); reuses the
already-codesigned minvmd.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…o minvmd-stage2-integration

As pid-1 the guest has no HOME and the root image is read-only, so the old
dirs::state_dir() path both panics and lands on a read-only fs. In guest
mode, mount a second virtio-blk device (/dev/vdb) at /var/lib/minimal and
resolve state + cache + the SSH host key underneath it, before serving.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The host attaches a freshly-provisioned blank data disk, so the first
mount fails ("Can't find ext4 filesystem"); mke2fs it (the rootfs ships
e2fsprogs) and retry. Subsequent boots find a formatted disk and mount
directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The guest root is read-only, but minimald needs to write its session
store + SSH host key. Provision a persistent sparse raw image under the
state dir (the guest formats it, so no host mke2fs) and attach it rw as a
second virtio-blk device via krun_add_disk2; the guest mounts it at
/var/lib/minimal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

/var/lib/minimal must exist in the read-only root image so the guest can
mount the writable data disk there (it cannot mkdir on a read-only root).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

libkrun's krun_add_vsock_port2 bridge delivers a reliable byte stream to a
guest vsock listener (proven by minvmd's bridge_e2e), but CLI tools
(ssh/socat/nc) do not round-trip it — even the socat-echo stub returns
nothing via the CLI while bridge_e2e (UnixStream + read_exact) passes. So
the direct tokio-vsock listen vs the relay cannot be distinguished with
manual ssh; both need a Rust SSH-client harness to validate.

Guest mode now relays vsock<port> -> a UDS on the writable data disk via
socat (reusing the bridge_e2e-proven vsock listen) and serves it with the
native run_on_uds path. Adds accept + session-error logging to both serve
loops so handshake failures are visible.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A reliable Rust counterpart to ad-hoc ssh/socat/nc testing (which does not
round-trip the bridge): boot minimald as the guest, connect a UnixStream to
the bridge UDS, and assert an SSH-2.0 banner comes back from the in-guest
minimald (russh) server — proving host UDS -> libkrun bridge -> guest vsock
-> socat relay -> minimald SSH. Gated MINVMD_E2E=1; retries the exchange to
absorb the post-READY startup race (matching bridge_e2e).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The vsock->uds relay is validated by the SSH-session e2e; remove the
`-d -d` diagnostic flags so the relay is quiet on the guest console.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Drives a real russh client over a UnixStream to the bridge UDS: authenticate
(auth_none, local peer), CreateSession RPC over the SSH subsystem, then exec a
command in that session and assert stdout + exit status. Proves host UDS ->
libkrun bridge -> guest vsock -> socat relay -> minimald SSH -> session exec.
Gated MINVMD_E2E=1; retries the session to absorb the post-READY startup race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Records minvmd's new dev-dependency edges (russh, tokio, sessions, paths)
added for the session e2e test, so CI's --locked build resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Experiment (swap the relay for direct Server::run_on_vsock) showed the prior
rationale was wrong: a direct tokio-vsock listener *does* accept the bridged
connection. It fails one layer up — the SSH session over the bridged vsock
stream dies with "Protocol error: early eof" within ~13ms of accept, so the
client hangs. libkrun's bridged vsock does not sustain a full bidirectional
SSH session (bridge_e2e's single short request/response works). socat
terminates the vsock and gives russh a stable UNIX-domain stream; the session
e2e completes only with the relay. Comment-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… job

Now that the rootfs comes from the upstream microvm-rootfs package, its
prebuilt aarch64 artifact is in the public cache — so `materialize --arch
aarch64` is a cache pull, not a native build, and works on a cheap x86_64
Linux runner (exactly like the kernel). Materialize both in one `artifacts`
job; the self-hosted runner only boots.

- Generalize scripts/fetch-virtio-kernel.sh -> fetch-artifact.sh <output> <dest>.
- Merge the virtio-kernel + minvmd-rootfs jobs into one Linux `artifacts` job.
- Drop the self-hosted minvmd-rootfs materialize job and its shim /host
  large-file read workaround.
- Point boot-e2e/autospawn-e2e at `needs: [artifacts]`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…invmd-stage2-integration

# Conflicts:
#	.minimal/packages/minvmd-rootfs/build.sh