feat(minvmd): source libkrun from the upstream package - #466

R2.4: before krun_start_enter, verify /dev/kvm is accessible on Linux so
`boot`/`run` fail fast with an actionable message instead of an opaque
libkrun error. ENOENT maps to "KVM module not loaded / no hardware
virtualization"; EACCES maps to "add your user to the `kvm` group". The
check runs in run_supervisor (covering --detach from the foreground
caller) and in run_boot. No-op on macOS, where Hypervisor.framework
availability is verified by krun_create_ctx itself.

The error-mapping helper is platform-independent and unit-tested on every
libkrun build.

Also fix two misattributed (R2.4) comments on READY-marker code (the
READY round-trip is R2.1/R2.2; boot_e2e is R3.1) and update the
bench-minvmd-boot.sh header to document dual macOS/Linux use (codesigning
is a macOS-only prerequisite; Linux/KVM needs none).

Refs: #411
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the opt-in self-hosted minvmd-linux-kvm-e2e job (which needed a
preprovisioned [self-hosted, linux, kvm] runner with libkrun already
installed) with a path-scoped ci-linux-kvm.yml that runs on GitHub-hosted
runners across an x86_64 (ubuntu-latest) + aarch64 (ubuntu-24.04-arm)
matrix.

- Add scripts/build-libkrun.sh: builds libkrunfw v5.5.0 + libkrun v1.19.0
  (BLK=1, so krun_add_disk2 is exported) from source into a cacheable
  prefix; idempotent so a cache hit skips the kernel compile.
- Enable /dev/kvm for the runner user via a udev rule, failing fast if the
  runner has no KVM.
- Exercise the run/daemon path (run --detach -> status -> stop) alongside
  the existing boot/session/bridge e2e tests and the boot latency bench.
- Non-gating and path-scoped, mirroring ci-macos.yml.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The first CI (Linux KVM) run failed on both legs:

- x86_64: libkrunfw's Makefile forwards $(MAKEFLAGS) verbatim into the
  kernel's recursive make. Building it via `make -C` enables
  print-directory mode, which injects a bare `w` token into MAKEFLAGS that
  the kernel make then treats as a goal (`make w` -> "No rule to make
  target 'w'"). Build libkrunfw/libkrun with an in-dir `cd` + explicit
  `-jN` and no exported MAKEFLAGS, the upstream-standard invocation.
- aarch64: GitHub-hosted ubuntu-24.04-arm has no /dev/kvm, so the
  Linux/KVM e2e cannot run there. Drop the matrix and run x86_64-only;
  aarch64 + libkrun boot stays covered by the macOS lane (Apple Silicon).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The libkrun build now succeeds, but `fetch-artifact.sh` (cargo build -p
minimal) failed compiling remote-proto:

  protoc failed: google/protobuf/timestamp.proto: File not found.

The well-known protos live in libprotobuf-dev, which protobuf-compiler
only pulls via recommends. Since the dependency install uses
--no-install-recommends (to stay lean for the libkrun toolchain), name
libprotobuf-dev explicitly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

actions/cache's default post-job save is skipped when the job fails, so a
failure after the libkrun step forced a ~15 min kernel recompile on every
iteration. Split into cache/restore + an explicit cache/save placed right
after the build, so the expensive prefix is cached as soon as it exists —
before any later step (e2e/bench) can fail. Gated on a cache miss since
cache keys are immutable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The Linux/KVM boot e2e failed on x86_64 with `krun_start_enter ... Invalid
argument (EINVAL)`. The x86_64 `virtio-kernel` artifact is a bzImage: a
PE/`MZ` container that embeds the `vmlinux` ELF as a gzip member. minvmd
passed `KernelFormat::Elf`, so libkrun handed the PE container straight to
its ELF loader, which cannot parse it.

Use `KernelFormat::ImageGz` (already in the enum): libkrun scans for the
gzip magic (1f 8b 08), inflates the single member to the vmlinux ELF, and
ELF-loads it. Verified against the materialized artifact — the gzip member
at the first magic match decompresses to a 46 MB x86-64 ELF vmlinux.

aarch64 is unaffected (ships an uncompressed Image → Raw).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

With the ImageGz kernel fix the guest now boots on x86_64 KVM and Boot E2E
+ Session E2E pass, but Bridge E2E failed: it read back "SSH-2.0-russh_0.6"
instead of its echo payload.

bridge_e2e targets the Stage-1 guest "vsock stub" — a socat/cat echo on
port 2222 — which minimald-as-pid1 replaced with a direct SSH session
server. The test is obsolete against the current rootfs. The macOS lane
already runs only boot_e2e + minimald_session_e2e; match it. Bridge
session coverage comes from Session E2E + the daemon lifecycle step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Address CodeRabbit review feedback:

- build-libkrun.sh: gate the skip-rebuild on a version stamp, not mere
  presence of libkrun.so. A restored cache (or local prefix reuse) built
  from different pins is now rebuilt instead of silently accepted, so a
  pin bump can't yield a stale install.
- ci-linux-kvm.yml: assert the daemon state with `jq -e` (preinstalled on
  GitHub runners) instead of grepping the raw JSON, so the lifecycle check
  doesn't break on harmless formatting changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The previous cache was saved before build-libkrun.sh wrote its version
stamp, so the new stamp-gated idempotency check would treat the restored
prefix as stale and rebuild — yet the save step is gated on a cache miss,
so the rebuilt (stamped) prefix would never be re-saved. Every run would
then rebuild libkrun from scratch.

Bump the key (`-blk` → `-blk-stamped`) so the stamped prefix is saved
fresh once, after which restore hits and the build skips.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The daemon lifecycle step failed asserting state=="running" right after
`run --detach`: status showed `{"state":"starting","vmm_pid":null}`.

`run --detach` returns once the host UDS accepts connections, which libkrun
opens early in VM setup — ahead of the supervisor's Starting->Running
transition (set after the guest READY marker). Poll status for up to ~15s
for Running instead of asserting immediately, removing the race.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The daemon lifecycle step reached Running (the poll fixed the earlier race)
and stopped cleanly, but still exited 1: `minvmd status --json` exits 1
when stopped (documented), and `status --json | tee status-after.json`
tripped `set -o pipefail` under `set -e`. The bare `status` was already
guarded with `if`; the piped `--json` query was not.

Capture it with `> status-after.json || true`, then assert state==stopped
via jq and re-confirm the non-zero exit separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Consume the libkrun KVM backend from the upstream `libkrun` package
(gominimal/pkgs#271) instead of compiling it from source in CI. The
from-source path built libkrunfw, which compiles a guest kernel on every
run — far too heavy.

- Bump the upstream pkgs pin to the merge commit of #271, which also
  carries the new microvm-rootfs package.
- Add a `libkrun` oci-image output backed by the upstream package.
- Add scripts/fetch-libkrun.sh: a cache fetch (keyed by the pinned
  commit) that materializes the libkrun OCI image and extracts only
  libkrun.so* + libkrunfw.so* into a flat link/runtime prefix. The
  package's glibc/libgcc layers are left out so they cannot shadow the
  host glibc the minvmd binary is linked against.
- Rewrite the ci-linux-kvm.yml libkrun steps to fetch the prefix and
  drop the kernel-compile apt toolchain (flex/bison/clang/patchelf/…).
- Remove scripts/build-libkrun.sh.

Validated on aarch64: the upstream cache served a prebuilt libkrun (no
kernel compile), the extracted prefix satisfies both the build.rs link +
rpath and the runtime dlopen of libkrunfw.so.5, and minvmd builds as the
real (non-stub) impl.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Document the Linux/KVM flow for the next dev: fetch libkrun + guest
artifacts, build the initramfs (cross or native musl), build and boot
minvmd, and run the gated e2e tests. Covers the kvm-group durability
fix and the `boot` vs `run` state.toml lifecycle gotcha. Fix the stale
"macOS-only" opener and retitle the existing E2E section to "(macOS)".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scripts/build-libkrun.sh is no longer referenced by any workflow now
that CI materializes libkrun from the upstream package via
scripts/fetch-libkrun.sh. Drop the dead script and its dangling
reference in .minimal/minimal.toml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>