microvm-rootfs: ship e2fsprogs + util-linux and /var/lib/minimal mountpoint - #365

* feat(minvmd): per-VM writable /dev/vdb — attach, provision, sync tunables

Host side of the per-VM writable ext4 volume (spec #583 Unit 1). Adds the
krun_add_disk3 FFI + SyncMode tri-state (KRUN_SYNC_{NONE,RELAXED,FULL}, a u32
not a bool) and Context::add_disk_with_sync. volume.rs provisions a sparse raw
image (ftruncate, allocate-on-write) at the resolved path — MINVMD_DATA_VOLUME_PATH
override, else <state>/data-vol.raw honouring XDG_STATE_HOME — and is idempotent.
vmm_child attaches /dev/vdb on every boot with the MINVMD_DISK_SYNC /
MINVMD_DISK_DIRECT_IO tunables (default relaxed / false).

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(minimald): mount + relocate cache/state onto /dev/vdb

The guest formats /dev/vdb on first boot (ext4 superblock-gated mkfs, sized
1 MiB below the device to survive libkrun's backing-file trailer shave) and
mounts it read-write at /var/lib/minimal; cache + state then relocate onto it,
resolving the EXDEV hardlink constraint and persisting session state across
clean restarts. Mount presence uses try_exists so a stat error is not silently
treated as "no volume".

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* build: pin pkgs rootfs shipping /var/lib/minimal + e2fsprogs

microvm-rootfs now ships the /var/lib/minimal mountpoint and e2fsprogs +
util-linux (gominimal/pkgs#365/#366) so the guest can mkfs.ext4 and mount
/dev/vdb (and fstrim-reclaim it). Re-pin locked_commit to 12f324d8.

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(minvmd,minimald): assert libkrun >= 1.19 in the VM boot jobs

krun_add_disk3 (this PR's /dev/vdb attach) needs libkrun >= 1.19.0. The macOS
"Verify libkrun" steps (boot-e2e / autospawn-e2e / build-macos) and the
Linux/KVM job now assert the libkrun in use exports krun_add_disk3, failing with
an actionable "upgrade the runner" message instead of a cryptic dyld
symbol-not-found at VM boot.

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minimald): format the data volume without a background lazy-init storm

On first boot the guest mkfs.ext4's the (default 32 GiB) /dev/vdb. With ext4's
default lazy inode-table init, the ext4lazyinit kernel thread zeroes ~512 MiB of
inode tables in the background right after mount — sustained I/O that lands
exactly when the guest is bringing up its vsock bridge + SSH server on 2 vCPUs.
That starved the first host->guest connect: a gvproxy-forwarder control request
timed out (5s) and `minimal ls` failed with `ssh connect: Disconnected`
(autospawn-e2e). boot-e2e was unaffected because its check used a 2 GiB volume.

Format with `-E lazy_itable_init=0,lazy_journal_init=0` (do the zeroing now, at
mkfs, not in the background) and `-i 65536` (~524K inodes / ~128 MiB table
instead of ~2M / ~512 MiB) so the synchronous init is small and one-time; later
boots detect the superblock and skip mkfs entirely.

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minimald): reject undersized data volumes before mkfs

run_mkfs_ext4 derived the ext4 block count as
`device_size_bytes.saturating_sub(MKFS_MARGIN_BYTES) / EXT4_BLOCK_BYTES`, which
is 0 for any device <= the 1 MiB margin — handing mkfs.ext4 a nonsensical size
argument. Reject a device below MKFS_MIN_DEVICE_BYTES (16 MiB, comfortably above
ext4's journal minimum) with InvalidInput before invoking mkfs. Only guards a
misconfigured MINVMD_VOLUME_BYTES or malformed device; the real volume is
GiB-scale.

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minimald): don't block SSH accept on the gvproxy egress-proxy publish

Server::run awaited start_host_proxies before entering the accept loop, and the
DM1 host-expose there (expose_proxy_on_host -> gvproxy /services/forwarder/expose)
blocks up to a 5s control-request timeout when the host gvproxy is slow. In that
window the SSH listener is bound but not accepting, so the first host->guest
connect lands in the gap and fails with `ssh connect: Disconnected` — seen in
autospawn-e2e, where `minimal ls` connects right after READY. Attaching the
per-VM data volume made the host gvproxy slow enough to hit this on macOS/HVF,
exposing the latent bug (main passes only because gvproxy responds fast).

The expose is already best-effort (warns + continues on failure), so detach it:
bind the proxy listeners synchronously as before, but spawn the host-expose so it
never gates the SSH accept loop.

Refs: #583

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* revert(minimald): restore awaited gvproxy egress-proxy publish

This reverts commit 671bec4.

Detaching the host-side expose removed an accidental serialization
that main relies on. With the expose spawned, the first host->guest
SSH connect overlaps the guest->host expose control request on
libkrun's single vsock device and wedges it (#588), so cold
`minimal ls` fails with `ssh connect: Disconnected` in autospawn-e2e.
Awaiting the expose (as on main) serializes vsock usage so the accept
loop's first connection stays clean. The 5s cold-boot delay when the
host gvproxy is absent is pre-existing on main and out of scope here.

Refs: #588

* fix(minvmd): warn on unrecognized disk-tunable env values

resolve_disk_flags silently swallowed unrecognized MINVMD_DISK_SYNC and
MINVMD_DISK_DIRECT_IO values, falling back to the defaults with no signal.
Emit a tracing::warn! naming the env var and value when a set value is
unrecognized, keeping the existing relaxed/false defaults.

* fix(minvmd): treat concurrent data-volume creation as idempotent

ensure_sparse_raw probes metadata then create_new; a provisioner racing
between the two surfaced AlreadyExists as VolumeError::Create, breaking
the documented idempotent contract. Treat an AlreadyExists creation race
as success (the volume now exists) and keep the image as-is.

* ci: cover crates/minimald in the macOS lane

ci-macos boots minimald as pid-1 in the boot/e2e VM, yet its paths filter
omitted crates/minimald — so guest-daemon-only changes skipped every macOS
VM check, including autospawn-e2e. Add crates/minimald/** to the push +
pull_request filters so guest-daemon changes get the coverage that
exercises them.

* ci: provision a host gvproxy for the macOS e2e job

The Auto-spawn E2E boots the guest, which publishes its host-side proxy via
the host gvproxy over the vsock shuttle. With no gvproxy the publish blocks
until its 5s control-request timeout, holding Server::run's SSH accept loop
closed; the cold `minimal ls` connect-retry deadline then expires and the
list fails (`ssh connect: Disconnected`). This is a pre-existing ~40ms race
that /dev/vdb's ~60ms first-boot mkfs tips red. Fetch gvproxy and export
MINVMD_GVPROXY_BIN so the publish completes immediately, matching a real host.

* refactor(minimald): gate the data-volume mount on a ListenArgs field

Replace the second `is_minimal_microvm()` branch (per Tom's review) with a
hidden `mk_mount_state_volume: Option<String>` field on ListenArgs, set once
in the microVM config block. The mount is then driven by `if let Some(dev) =
listen_args.mk_mount_state_volume`. State/cache relocation stays gated on the
mount succeeding — pointing state at an unmounted /var/lib/minimal would land
it on the read-only rootfs (that becomes eager once Unit 2 makes mount
failure loud).

* docs(minimald): frame the mkfs options as consequences, not tuning

Per Tom's review: make explicit that the non-default mkfs.ext4 options are
not independent tuning but fall out of two deliberate choices — eager
inode/journal init (whose entailment is the reduced inode count) and
surviving libkrun's backing-file trailer shave (whose entailment is the
explicit block count + pinned block size). ext4's defaults are otherwise
left alone.

* refactor(minvmd): resolve data-volume path via paths::minimal_state_dir

Per Tom's review, source the state-dir base from the shared
`paths::minimal_state_dir()` resolver instead of the minvmd-local
`StateDir::default_path`. Keep the explicit `XDG_STATE_HOME` override first:
`dirs` (and thus `minimal_state_dir`) ignores it on macOS, and the e2e tests
isolate state via `XDG_STATE_HOME` — without it the volume would escape the
tests' temp dir onto the real state dir. Promote `paths` from a dev- to a
regular dependency.

* feat(minvmd): default the data volume to 256 GiB

Resolves Tom's review point: the image is sparse and offline resize is
hard, so size it big up front. Measured first-boot cost is flat in the
volume size — mkfs.ext4 writes only ~5-8 MiB of metadata regardless
(sparse backing + reduced inode ratio), so format+mount stays ~40 ms at
32/64/128/256 GiB on an M-series host. No boot-time penalty, so the default
goes to 256 GiB; MINVMD_VOLUME_BYTES still overrides.

Refs: #583

* refactor(minimald): drop the redundant mkfs -i override

Measurement (32/64/128/256 GiB) shows mke2fs already defaults to a ~65536
bytes/inode ratio at these volume sizes — `-i 65536` produces an identical
inode count (32 GiB -> 524288, 256 GiB -> 4194304, both 65536 B/inode). And
the eager inode-table init is cheap because the volume is sparse (zero-writes
land in holes; ~sub-ms at every size), not because of the inode count. So the
override earns nothing: drop it plus MKFS_BYTES_PER_INODE and correct the doc
rationale. Per Tom's review.

Refs: #583

* style(minvmd): rustfmt the path resolver + correct the flat-cost note

The wrapped closure exceeded rustfmt width (fmt CI). Also drop the stale
"reduced inode ratio" credit from the DEFAULT_VOLUME_BYTES note now that the
-i override is gone — the flat first-boot cost is purely the sparse backing.

* fix(minimald): cap the boot-path gvproxy expose so it can't stall SSH accept

start_host_proxies awaits expose_proxy_on_host on Server::run's boot path,
before the SSH accept loop serves. The `/services/forwarder/expose` control
request runs to the full 5s GVPROXY_CONTROL_TIMEOUT even with a host gvproxy
present (the forwarder control path isn't reachable over the shuttle in every
deployment), holding the accept loop closed. The cold `minimal ls`
connect-retry deadline (~5s) then expires as the loop would resume, so the
first list fails with `ssh connect: Disconnected` — the volume's ~60ms
first-boot mkfs tips a pre-existing race red on the e2e autospawn gate.

Cap the publish at HOST_EXPOSE_PUBLISH_TIMEOUT (1s): a best-effort publish
must never stall the SSH bridge. Best-effort semantics unchanged.

Refs: #588

* ci: drop the ineffective gvproxy provisioning from the e2e job

Provisioning a host gvproxy did not fix the autospawn stall — the boot log
shows the `/services/forwarder/expose` control request still times out at 5s
even with gvproxy present. The bounded expose (HOST_EXPOSE_PUBLISH_TIMEOUT)
fixes it in the daemon instead, so the fetch step + MINVMD_GVPROXY_BIN export
are dead weight; remove them.

* ci: reap the Session-E2E VM before the auto-spawn step

With boot-e2e and autospawn-e2e merged into one `e2e` job (upstream), the
session harness's leftover `__krun-vmm` (its Drop kills minvmd but not the
detached VMM grandchild) lingers into Auto-spawn. As a second concurrent VM
it wedges the host->guest bridge over libkrun's vsock (#588): with the expose
now bounded, the accept loop is up early yet the boot log shows no connection
ever accepted. Reap the VMM so Auto-spawn runs single-VM, as it did when it
was a standalone job.

The proper fix is the session harness reaping its own grandchild (process
group); tracked separately.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>