feat(release): ship minvmd to Linux as a static musl binary - #1104

* feat(minvmd): build against a static musl libkrun

minvmd is the only one of the four binaries that cannot ship to Linux
users. min, mip and minimald are self-contained musl builds; minvmd is a
native-glibc build that links libkrun.so and dlopens libkrunfw.so.5, so
shipping it means ~29 MB of libraries plus RUNPATH machinery, a bin/lib
sibling constraint and a glibc floor on users' disks. release.yml states
the constraint outright: "it can't join the static-musl cross build".

This removes the constraint. It does not yet wire it into the release —
that is the next step, and it is where x86_64 gets proven.

scripts/build-libkrun-linux.sh is the Linux twin of the macOS script:
same vendored pin, same patch series, same trim, different link model.
Three things it has to do that the dylib build does not:

- Add `staticlib` to libkrun's crate-type. Upstream declares
  ["cdylib", "lib"], so cargo emits no archive to link. Applied as a
  guarded sed rather than a carried patch: it is not an upstream fix but
  a property of how we consume the crate, and it survives a pin bump
  that would reject a line-anchored patch.
- Merge THEN localize. libkrun.a carries its whole Rust dependency
  closure including a copy of libstd, which collides with minvmd's own.
  `ld -r --whole-archive` into a single object first, and only then
  objcopy down to the krun_* globals: objcopy over an archive rewrites
  each member independently, so localizing first would break references
  that cross members.
- Drop libkrunfw entirely. It exists only to carry a bundled guest
  kernel that minvmd never uses (it supplies its own via ctx.set_kernel).
  That is what makes this possible at all — a static musl binary cannot
  dlopen — and it takes a GPL-2 kernel blob out of the shipped closure.

build.rs treats a libkrun.a in the prefix as the selector for a static
link, emitting minvmd_libkrun_static and recording no rpath (a static
link has nothing to resolve at load time). raw.rs's `#[link]` becomes
conditional on that cfg. Linux only: macOS builds and ships a dylib, and
silently switching that proven path because some unrelated libkrun
install dropped an archive in /opt/homebrew/lib would be a surprise.

Also fixes a latent bug in the macOS script. It ran cargo with
--manifest-path from the minimal repo root, but cargo resolves
.cargo/config.toml from the CWD, so it read ours and ignored libkrun's.
On darwin that only drops a test runner; libkrun's config also carries

    [target.'cfg(target_env = "musl")']
    rustflags = ["--cfg", "musl_v1_2_3"]

which is load-bearing for the musl build. Fixed in both scripts so they
cannot disagree about what they compiled.

Verified end-to-end on aarch64 (rust:alpine, binutils 2.45.1):

- libkrun.a builds; cargo confirms the crate-type edit landed by warning
  that it dropped the now-inert cdylib for a crt-static target
- 67 krun_* symbols kept global, 0 non-krun_ globals left after
  localization
- minvmd links it into an 8,335,392-byte ELF that `file` reports as
  "statically linked" and ldd rejects as "not a valid dynamic program"
- the real KVM backend is in there, not the stub: kvm_run,
  kvm_coalesced_mmio and KRUN_BLOCK_ROOT_FSTYPE are all present, and
  the binary runs

Not yet proven: x86_64, and booting a VM (needs KVM). Both belong in CI.

Refs #1065, #980

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(scripts): default the musl linker in build-libkrun-linux.sh

A musl-native host (Alpine) builds its own triple and needs no linker
override, which is how the script was verified. A glibc host with
musl-tools — every Ubuntu runner and any dev box — needs musl-gcc, and
without it the cargo build fails with a bare linker error that says
nothing about musl-tools.

Default cargo's documented `CARGO_TARGET_<TRIPLE>_LINKER` when the
target is not the host triple, and fail with an actionable message when
musl-gcc is missing. An explicit value always wins, so a caller with its
own toolchain is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(minvmd): keep the toolchain pin across the cd, and fail loudly on a
 missing libkrun

Two fixes, both about a build that should fail doing something worse.

**The toolchain pin escaped with the CWD.** Moving cargo into the fetched
libkrun tree fixed .cargo/config.toml resolution, but rustup resolves
rust-toolchain.toml from the CWD too. Leaving the repo silently fell back
to the DEFAULT toolchain — and in CI that is not the toolchain
`rustup target add` installed the musl target into, so the release job
died with:

    error[E0463]: can't find crate for `core`
    note: the `x86_64-unknown-linux-musl` target may not be installed

--manifest-path had the right toolchain and the wrong cargo config; the
cd had the right config and the wrong toolchain. Resolve the pin where
rust-toolchain.toml still applies and export RUSTUP_TOOLCHAIN across the
cd, so the build gets both.

Not merely convenience for the static build: libkrun.a is a Rust
staticlib linked into a Rust binary, and Rust has no stable ABI. libkrun
and minvmd must come out of the same rustc. Applied to the macOS script
too — its boundary is a C-ABI dylib so it is less exposed, but the two
scripts should not disagree about what they compiled.

**A missing libkrun produced a stub instead of an error.** When build.rs
finds no libkrun it builds a runtime-bailing stub. That is deliberate —
it keeps stock Linux CI green — but it means a build that SHOULD have
linked libkrun and didn't still succeeds, and ships a binary that
compiles, links, and then bails at VM boot. release.yml greps the built
binary for proof, but the build itself was happy either way.

MINVMD_REQUIRE_LIBKRUN closes that. `static` demands a libkrun.a and a
static link; any other non-empty value demands some libkrun. Unset (or
`0`) keeps stub-on-miss, so no existing build changes. The release lanes
opt in, turning a silent bad binary into a build error that names the
prefix it looked in and the script that fills it.

Verified all four paths: require=static with no libkrun and with only a
libkrun.so both panic with actionable messages; unset still builds the
stub; require=1 accepts a dynamic libkrun.

Refs #1065

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(scripts): drop the bashism that broke build-libkrun-linux.sh under sh

The musl-linker default used bash's ${!VAR} indirect expansion, so
running the script under any POSIX shell died before it built anything:

    scripts/build-libkrun-linux.sh: line 130: syntax error: bad substitution

CI never saw it — Ubuntu runners have bash and the shebang picks it up.
But Alpine, the natural place to build a musl artifact, ships no bash at
all, and `sh scripts/build-libkrun-linux.sh` is a reasonable way to
invoke it. Nothing else in the script needs bash; this was one line.

`eval` instead. `set -o pipefail` stays: it is load-bearing here, and
busybox ash supports it even though POSIX does not require it.

Found by running the script the way a user might, in a rust:alpine
container, rather than only the way CI does.

Re-verified end to end on aarch64 after the fix, under `sh`: the archive
builds (67 krun_* symbols kept, 0 leaked), MINVMD_REQUIRE_LIBKRUN=static
correctly refuses an empty prefix, and minvmd links to an 8,138,736-byte
binary that `file` reports as statically linked.

Refs #1065

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(release): ship minvmd to Linux as a static musl binary (#1104)

* feat(release): ship minvmd to Linux as a static musl binary

Completes the Linux half of #980. minvmd was the only one of the four
binaries not staged for Linux, and the only one that could not be: it
was a native-glibc build linking libkrun.so and dlopening libkrunfw.so.5,
so shipping it meant ~29 MB of libraries plus RUNPATH machinery, a
bin/lib sibling constraint and a glibc floor on users' disks.

Built on #1070, which made minvmd linkable against a static libkrun.

release.yml: both Linux jobs now build a static libkrun.a from the
vendored pin and link minvmd against it as musl, alongside the three
binaries that were already musl. arm64 gains a minvmd artifact for the
first time — the dynamic build needed a native glibc host per arch and
only amd64 had one, a constraint a static build does not have. The
libkrun build rides a new build-libkrun-static-linux composite, modelled
on setup-libkrun-macos: pin-keyed cache, build-on-miss, and a verify
step that re-asserts the API floor and the symbol surface on a
cache-restored archive.

Both jobs assert the built binary is `statically linked` and actually
contains the KVM backend. Neither is implied by a successful build: when
build.rs finds no libkrun it emits a stub that compiles and links
perfectly well, and would ship as a binary that bails at VM boot.

stage-release.sh: Linux gains bin/minvmd plus the guest payload
(data/{vmlinuz,rootfs.img,initramfs.cpio}) for both arches. All six
guest artifacts were already produced for both arches; only the rows
were missing. No lib/ component and no RUNPATH rewrite — that asymmetry
with darwin is the entire point.

nightly.yml: smoke-linux-kvm no longer materializes a libkrun prefix or
sets LD_LIBRARY_PATH, because the shipped binary needs neither. It
asserts that instead, so a regression to a dynamic build fails with a
clear message rather than a loader error mid-boot. The userns sysctl
stays — its justification cited the removed materialize step, but the
sandboxed build the e2e drives needs it independently.

setup-libkrun-linux is untouched and still used by ci-linux-kvm and
nightly-tests, which build against the dynamic upstream package.

Verified: workflow and composite YAML parse; shellcheck clean; a
dry-run stage against stub artifacts emits all eight new Linux rows with
correct dests, and the arm64 guest artifacts dedupe with darwin's as
intended.

Not verified locally: the builds themselves. x86_64 in particular has
never been exercised — the static-musl proof in #1070 is aarch64-only,
and this is the lane that settles it.

Refs #980, #1065

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(ci): boot the STATIC minvmd in the KVM lane, not a dynamic one

ci-linux-kvm is the only pre-merge lane that boots a real microVM, and
it was booting the wrong binary. It built minvmd against the dynamic
upstream libkrun package, shipped that .so in the testbed, and resolved
it at runtime via LD_LIBRARY_PATH. Meanwhile Linux now ships a
static-musl minvmd.

That left the shipped configuration first exercised by nightly's
smoke-linux-kvm — after merge. Nothing pre-merge compiled
build-libkrun-linux.sh at all, and nothing anywhere had ever BOOTED a
statically linked minvmd: release.yml only asserts `file` says static
and `strings` finds kvm_run, which proves the archive is well-formed and
the backend is linked, not that libkrun still works.

That gap matters here more than usual. libkrun.a is produced by merging
a whole Rust dependency closure with `ld -r` and then objcopy-localizing
everything outside the krun_* API — precisely the kind of transform that
links clean and misbehaves at runtime (two libstd copies, TLS, the panic
runtime). A symbol check cannot catch that; a boot can.

So this lane now builds libkrun statically from the vendored pin, builds
minvmd for x86_64-unknown-linux-musl, and boots that. The nextest
archive moves to the same target — the harnesses link libkrun too, and a
native-gnu archive would test a different binary than the one shipped.

Falls out of the change: no libkrun-prefix.tgz in the testbed, no
$HOME/.krun unpack, no LIBKRUN_PREFIX or LD_LIBRARY_PATH in the test
job. The binary carries its own libkrun.

Three assertions rather than assumptions, because a silent regression to
a dynamic link would quietly restore the gap this closes:
MINVMD_REQUIRE_LIBKRUN=static on both cargo invocations (a build.rs that
cannot find libkrun.a is now an error, not a stub), and a `file ... |
grep statically linked` check at each end of the artifact handoff.

Two path-filter fixes: vendor/libkrun/** was never listed, so a pin or
patch bump could not re-run the lane that now builds from it; and the
composite entry follows the lane onto build-libkrun-static-linux.

The dynamic path keeps its coverage in nightly-tests.yml, which still
builds against the upstream package — it remains the dev/`just up-kvm`
configuration, just no longer the one CI proves before merge.

Refs #980, #1065

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): accept file(1)'s "static-pie linked" when asserting a static minvmd

x86_64-unknown-linux-musl links a static PIE, which file(1) reports as
"static-pie linked", not "statically linked". The assertion grepped only for
the latter, so build-release-linux-amd64 failed on a binary that was correctly
static while the arm64 job passed: aarch64-unknown-linux-musl emits a plain
non-PIE static executable.

Neither form has a PT_INTERP or a DT_NEEDED, so both satisfy what the check is
actually for. Accept either spelling at all five assertion sites; the arm64 and
testbed ones pass today only by accident of that target's current default and
would break the same way if it ever switches to static-pie.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(minvmd): gate the static libkrun on musl and harden the build script

Review follow-ups on #1070.

build.rs gated `has_static` on `target_os == "linux"` alone, so a default
*-linux-gnu build pointed at a staged prefix selected kind = "static" and would
try to link a musl staticlib into a glibc binary. Worse, it satisfied
MINVMD_REQUIRE_LIBKRUN=static, whose entire job is to prove the link model we
ship. Require target_env == "musl" too, and report that mismatch distinctly
from a genuinely missing archive, since "no libkrun.a in <prefix>" sends the
reader hunting for a file that is sitting right there.

build-libkrun-linux.sh looked up the cargo linker override with `eval` over a
variable name derived from $TARGET, which is only suffix-validated, so a
crafted triple could inject shell syntax into the build. `printenv` reads the
variable without a second round of expansion, and is not a bashism either,
which is what the eval was working around in the first place.

The archive rewrite ran the HOST ld/nm/objcopy/ar unconditionally. That is
correct for CI, where each arch builds on its own runner, but silently wrong
for the cross-build the usage text advertised: the host linker cannot merge
foreign objects. Prefer target-prefixed binutils when the arches differ, fail
with an actionable message when none are installed, and correct the two
comments that overstated the cross-build support.

Verified on aarch64: the script still stages the archive end-to-end (67 krun_*
symbols) and passes shellcheck; a gnu build with a staged archive now fails
with the new message instead of attempting a static link; the musl build still
produces a statically linked minvmd.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ci): assert the two libstds stay separate, and drop three pipefail traps

Pins the property the #1070 review flagged as fragile ("coalescing the rust
internal symbols between libkrun and the minvmd binary"), and fixes the
pipefail hazards CodeRabbit found in the verification steps.

libkrun.a carries its own copy of libstd, localized so only the 67 krun_*
entry points stay global. The other half of that isolation was never checked:
if any Rust symbol is left UNDEFINED, the final link resolves it against
MINVMD's libstd instead, coalescing two independent Rust runtimes into one
binary. Assert there are none. A clean archive imports only C, musl libc plus
the _Unwind_* ABI, which is shared deliberately (one unwinder per process is
correct; two would be the bug). Measured on the current pin: 221 libc, 14
_Unwind_*, 0 Rust. Worth asserting rather than assuming, because the failure is
invisible at build time and only surfaces at VM boot, and because `ld -r` plus
`objcopy --keep-global-symbols` is binutils behaviour rather than a stability
contract, so a toolchain upgrade is the plausible regression path.

Three pipefail traps, all the same shape the script already documents for nm:

- release.yml piped `strings -a` over a ~10 MB binary into `grep -q` in both
  Linux jobs. `grep -q` exits at its first match, SIGPIPEs `strings`, and
  pipefail turns the dead producer into a step failure, so a PASSING check can
  fail a good release. Timing-dependent, which is worse than deterministic.
  Materialize the dump, then grep the file.
- keep.syms ended in `grep '^krun_'`, which exits 1 on no match and aborted the
  script before the "no krun_* symbols found" error could explain why.

Also corrects release-pipeline.md, which still quoted `statically linked` as
the sole accepted spelling after the check learned `static-pie linked`.

Verified on aarch64: full script run stages the archive (67 symbols) with the
new assertion passing, shellcheck clean, and the detector fires on synthetic
_ZN/_R/__rust_ input while staying silent on libc and _Unwind_*.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>