Skip to content

feat(release): ship the VM stack to Linux installs [blocked on pkgs#533] #988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
norrietaylor wants to merge 8 commits into from
Closed

feat(release): ship the VM stack to Linux installs [blocked on pkgs#533] #988

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8a29870
feat(release): ship the VM stack to Linux installs; rename gvproxy
norrietaylor Jul 27, 2026
892d238
test(scripts): harness for the Linux shipped-linkage rewriter
norrietaylor Jul 27, 2026
ad2ba06
fix(scripts): drop the SC2015 pattern from the install migration
norrietaylor Jul 27, 2026
ab25706
refactor(release): name the installed switch binary gvproxy-min
norrietaylor Jul 27, 2026
83be120
fix(release): address review findings on the Linux VM stack
norrietaylor Jul 27, 2026
40185cf
test(switch): make the system-path fallback tests host-independent
norrietaylor Jul 28, 2026
146ec31
docs(specs): record the Linux VM stack in the installer and minvmd specs
norrietaylor Jul 28, 2026
403c784
fix(release): do not ship libkrunfw
norrietaylor Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 68 additions & 43 deletions .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,10 +140,15 @@ jobs:
# the SHIPPED minvmd and pass the session e2e over the vsock bridge, the
# same proof ci-linux-kvm's `test-kvm` runs. Every moving part is a
# downloaded release artifact: the native-glibc `minvmd`, the guest kernel
# + ext4 rootfs + initramfs, and the pinned gvproxy switch. libkrun itself
# is NOT shipped on Linux (users resolve it from the system / the upstream
# package), so it is materialized here exactly as the release build linked
# it, reusing the shipped static `mip` to drive the cache fetch.
# + ext4 rootfs + initramfs, the pinned `gvproxy-min` switch, and the
# shipped libkrun — laid out as a `lib/` sibling of `bin/` exactly as
# scripts/install.sh drops it, so minvmd resolves it through its
# $ORIGIN/../lib RUNPATH (mirrors what smoke-macos proves for the dylib;
# like macOS, no libkrunfw is shipped — minvmd brings its own kernel, so
# libkrun's bundled-GPL-kernel library is never loaded). Deliberately
# NO LD_LIBRARY_PATH and no materialize step: this lane is what catches a
# shipped libkrun whose RUNPATH is wrong, and either would mask exactly
# that failure.
needs: [release]
runs-on: ubuntu-latest # x86_64; KVM-capable
timeout-minutes: 30
Expand All @@ -168,15 +173,10 @@ jobs:
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
ls -l /dev/kvm
- name: Enable unprivileged user namespaces
# `mip materialize` (the libkrun cache fetch below) runs the build
# pipeline, which needs userns; Ubuntu 24.04 restricts it via
# AppArmor by default.
run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
- name: Download shipped linux-amd64 binaries
# Pulls min, minvmd, mip and gvproxy (plus minimald, unused on the
# host — it ships inside the initramfs). `*-linux-amd64` is a clean
# glob here: only darwin artifacts carry the `temp-` prefix.
- name: Download shipped linux-amd64 artifacts
# Pulls min, minvmd, mip, gvproxy and libkrun (plus minimald, unused on the host — it ships inside the
# initramfs). `*-linux-amd64` is a clean glob here: only darwin
# artifacts carry the `temp-` prefix.
uses: actions/download-artifact@v8
with:
pattern: "*-linux-amd64"
Expand All @@ -197,32 +197,54 @@ jobs:
with:
name: minimald-initramfs-amd64
path: ${{ runner.temp }}/guest
- name: Put the shipped binaries on PATH
# Rename off the platform suffix and prepend to PATH so `min`
# autospawns `minvmd` by bare name. gvproxy is NOT resolved from
# PATH — minvmd only honours MINVMD_GVPROXY_BIN, the installer
# bin dir, or /usr/lib/minimal/bin (see resolve_gvproxy_path) —
# so the e2e step below points MINVMD_GVPROXY_BIN at it.
- name: Assemble the installer layout (bin/ + lib/)
# Rename off the platform suffix and prepend bin/ to PATH so `min`
# autospawns `minvmd` by bare name. The switch is NOT resolved from
# PATH — minvmd honours MINVMD_GVPROXY_BIN, the installer bin dir,
# or /usr/lib/minimal/bin (see resolve_gvproxy_path) — so the e2e
# step below points MINVMD_GVPROXY_BIN at it.
#
# The .so moves to a lib/ SIBLING of bin/ under its soname, which is
# where the shipped RUNPATH looks. Verified explicitly:
# `ldd` must resolve libkrun out of that lib/ and leave nothing
# "not found", so a bad RUNPATH fails here with a readable error
# instead of an opaque VM boot timeout later.
run: |
set -euo pipefail
BIN="$RUNNER_TEMP/bin"
LIB="$RUNNER_TEMP/lib"
mkdir -p "$LIB"
mv "$BIN/minimal-linux-amd64" "$BIN/min"
mv "$BIN/minvmd-linux-amd64" "$BIN/minvmd"
mv "$BIN/mip-linux-amd64" "$BIN/mip"
mv "$BIN/gvproxy-linux-amd64" "$BIN/gvproxy"
chmod +x "$BIN/min" "$BIN/minvmd" "$BIN/mip" "$BIN/gvproxy"
mv "$BIN/gvproxy-linux-amd64" "$BIN/gvproxy-min"
mv "$BIN/libkrun-linux-amd64.so" "$LIB/libkrun.so.1"
chmod +x "$BIN/min" "$BIN/minvmd" "$BIN/mip" "$BIN/gvproxy-min"
echo "$BIN" >> "$GITHUB_PATH"
- name: Materialize libkrun prefix (upstream package)
# minvmd links libkrun.so + dlopens libkrunfw.so.5 at runtime; the
# Linux release ships neither (its rpath resolves them from the
# system / LD_LIBRARY_PATH). Reuse the shipped static mip to drive
# the cache fetch (no Rust toolchain) — the exact libkrun the release
# build linked against. Publishes LIBKRUN_PREFIX and
# LD_LIBRARY_PATH=$HOME/.krun to later steps.
uses: ./.github/actions/setup-libkrun-linux
with:
arch: x86_64
mip: ${{ runner.temp }}/bin/mip
# Capture once, then match on the string. Piping into `grep -q`
# under `set -o pipefail` can report the pipeline as SIGPIPE
# (141) when grep exits early, which would make the guard below
# silently evaluate false and pass a genuinely broken binary.
ldd_out="$(ldd "$BIN/minvmd")"
printf '%s\n' "$ldd_out"
case "$ldd_out" in
*"not found"*)
echo "::error::shipped minvmd has unresolved libraries in the installer layout" >&2
exit 1 ;;
esac
# ld.so expands $ORIGIN but does NOT normalize the rest of the
# entry, so it reports <bin>/../lib/libkrun.so.1 verbatim.
# Comparing raw strings against "$LIB/libkrun.so.1" could never
# match; canonicalize both sides instead.
resolved="$(printf '%s\n' "$ldd_out" | awk '/libkrun\.so\.1 =>/ {print $3; exit}')"
if [ -z "$resolved" ] || [ ! -e "$resolved" ]; then
echo "::error::shipped minvmd did not resolve libkrun at all (RUNPATH wrong)" >&2
exit 1
fi
if [ "$(readlink -f "$resolved")" != "$(readlink -f "$LIB/libkrun.so.1")" ]; then
echo "::error::shipped minvmd resolved libkrun from $resolved, not $LIB (RUNPATH wrong)" >&2
exit 1
fi
- name: Point the microVM env at the shipped guest artifacts
run: |
{
Expand All @@ -240,7 +262,7 @@ jobs:
# session mint — dies with "Could not resolve host".
env:
MINVMD_BOOT_LOG: ${{ runner.temp }}/cli-e2e-boot.log
MINVMD_GVPROXY_BIN: ${{ runner.temp }}/bin/gvproxy
MINVMD_GVPROXY_BIN: ${{ runner.temp }}/bin/gvproxy-min
run: E2E_VM=1 E2E_MINIMAL_ARGS="--provider local-minvmd" E2E_PROJECT_DIR=/tmp ./scripts/session-e2e.sh
- name: Reap leftover VM processes
# The autospawned minvmd can leak a __krun-vmm child on a failed
Expand Down Expand Up @@ -320,26 +342,29 @@ jobs:
path: ${{ runner.temp }}/dl
- name: Assemble the installer layout
# Reproduce the on-disk tree scripts/install.sh writes (spec 07):
# bin/{min,minvmd,gvproxy}, lib/libkrun.1.dylib (bin's sibling, the
# @rpath target), data/{vmlinuz,rootfs.img,initramfs.cpio}. The dylib
# basename MUST be libkrun.1.dylib — minvmd's load command is
# @rpath/libkrun.1.dylib. Then bin/ joins PATH so `min` autospawns
# `minvmd` (which finds libkrun via @loader_path/../lib) by bare
# name. gvproxy is NOT resolved from PATH — the e2e step below
# points MINVMD_GVPROXY_BIN at the laid-out binary.
# bin/{min,minvmd,gvproxy-min}, lib/libkrun.1.dylib (bin's sibling,
# the @rpath target), data/{vmlinuz,rootfs.img,initramfs.cpio}. The
# dylib basename MUST be libkrun.1.dylib — minvmd's load command is
# @rpath/libkrun.1.dylib. The switch installs as `gvproxy-min`, not
# `gvproxy`: bin/ goes on PATH here exactly as ~/.local/bin is for a
# real install, and the upstream name would collide with podman's.
# Then bin/ joins PATH so `min` autospawns `minvmd` (which finds
# libkrun via @loader_path/../lib) by bare name. The switch is NOT
# resolved from PATH — the e2e step below points
# MINVMD_GVPROXY_BIN at the laid-out binary.
run: |
set -euo pipefail
DL="$RUNNER_TEMP/dl"
ROOT="$RUNNER_TEMP/install"
mkdir -p "$ROOT/bin" "$ROOT/lib" "$ROOT/data"
mv "$DL/minimal-macos-arm64" "$ROOT/bin/min"
mv "$DL/minvmd-macos-arm64" "$ROOT/bin/minvmd"
mv "$DL/gvproxy-darwin-arm64" "$ROOT/bin/gvproxy"
mv "$DL/gvproxy-darwin-arm64" "$ROOT/bin/gvproxy-min"
mv "$DL/libkrun-macos-arm64.dylib" "$ROOT/lib/libkrun.1.dylib"
mv "$DL/vmlinuz-arm64" "$ROOT/data/vmlinuz"
mv "$DL/rootfs-arm64.img" "$ROOT/data/rootfs.img"
mv "$DL/initramfs-arm64.cpio" "$ROOT/data/initramfs.cpio"
chmod +x "$ROOT/bin/min" "$ROOT/bin/minvmd" "$ROOT/bin/gvproxy"
chmod +x "$ROOT/bin/min" "$ROOT/bin/minvmd" "$ROOT/bin/gvproxy-min"
echo "$ROOT/bin" >> "$GITHUB_PATH"
- name: Reap stale VM processes before the smoke
# The mini is persistent and shared: a prior run's stray minvmd,
Expand All @@ -360,7 +385,7 @@ jobs:
MINVMD_ROOTFS_PATH: ${{ runner.temp }}/install/data/rootfs.img
MINVMD_INITRAMFS: ${{ runner.temp }}/install/data/initramfs.cpio
MINVMD_BOOT_LOG: ${{ runner.temp }}/cli-e2e-boot.log
MINVMD_GVPROXY_BIN: ${{ runner.temp }}/install/bin/gvproxy
MINVMD_GVPROXY_BIN: ${{ runner.temp }}/install/bin/gvproxy-min
run: E2E_VM=1 E2E_PROJECT_DIR=/tmp ./scripts/session-e2e.sh
- name: Reap VM processes
# Leave the shared mini clean for the next job (Guest teardown does
Expand Down
69 changes: 66 additions & 3 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,9 @@ jobs:
fetch-depth: 0
fetch-tags: true
- name: Install dependencies
run: sudo apt-get update && sudo apt-get install -y musl-tools protobuf-compiler
# patchelf: rewrites the shipped minvmd/libkrun RUNPATHs for the
# installer's bin/ + lib/ layout (scripts/rewrite-linux-linkage.sh).
run: sudo apt-get update && sudo apt-get install -y musl-tools protobuf-compiler patchelf
- name: Add Rust target
run: rustup target add x86_64-unknown-linux-musl
- name: Cache Rust build artifacts (dependency-pruned)
Expand Down Expand Up @@ -193,20 +195,42 @@ jobs:
mip: ${{ github.workspace }}/target/x86_64-unknown-linux-musl/release/mip-linux-amd64
- name: Build native minvmd binary
# Native host target (x86_64-unknown-linux-gnu), NOT musl: the binary
# links libkrun.so (and libkrunfw.so.5 is dlopen'd at runtime), so unlike
# links libkrun.so, so unlike
# the three above it is not self-contained — running it still needs those
# .so's reachable via rpath/LD_LIBRARY_PATH. LIBKRUN_PREFIX — published
# by the setup-libkrun-linux composite above — drives the build.rs link
# search and the `minvmd_libkrun` cfg (real, non-stub impl).
run: |
cargo build --release --locked -p minvmd --bin minvmd
mv target/release/minvmd target/release/minvmd-linux-amd64
- name: Rewrite + verify minvmd/libkrun linkage (RUNPATH)
# The Linux twin of the macOS @rpath rewrite: point minvmd at
# $ORIGIN/../lib (bin/minvmd -> lib/libkrun.so.1). Also hard-fails
# on a soname bump, which would silently break the lib/ dest in
# stage-release.sh.
run: ./scripts/rewrite-linux-linkage.sh target/release/minvmd-linux-amd64 "$LIBKRUN_PREFIX"
- name: Stage libkrun for upload
# Resolve the soname symlink chain to the real file and copy it
# under a platform-suffixed name, so merge-multiple's basename
# flatten does not collide with the arm64 build. The SONAME inside
# is unchanged; the installer stages it back under that soname
# (lib/libkrun.so.1). libkrunfw is deliberately not shipped: it
# carries a bundled GPL-2 kernel and minvmd supplies its own.
run: |
mkdir -p "$RUNNER_TEMP/libs"
cp "$(readlink -f "$LIBKRUN_PREFIX/libkrun.so.1")" "$RUNNER_TEMP/libs/libkrun-linux-amd64.so"
- name: Upload binary (minvmd)
uses: actions/upload-artifact@v7
with:
name: minvmd-linux-amd64
path: target/release/minvmd-linux-amd64
retention-days: 7
- name: Upload libkrun (linux-amd64)
uses: actions/upload-artifact@v7
with:
name: libkrun-linux-amd64
path: ${{ runner.temp }}/libs/libkrun-linux-amd64.so
retention-days: 7

build-release-linux-arm64:
runs-on: ubuntu-26.04-arm
Expand Down Expand Up @@ -234,7 +258,8 @@ jobs:
fetch-depth: 0
fetch-tags: true
- name: Install dependencies
run: sudo apt-get update && sudo apt-get install -y musl-tools protobuf-compiler
# patchelf: see the amd64 job — the shipped RUNPATH rewrite.
run: sudo apt-get update && sudo apt-get install -y musl-tools protobuf-compiler patchelf
- name: Add Rust target
run: rustup target add aarch64-unknown-linux-musl
- name: Cache Rust build artifacts (dependency-pruned)
Expand Down Expand Up @@ -283,6 +308,44 @@ jobs:
name: minimald-linux-arm64
path: target/aarch64-unknown-linux-musl/release/minimald-linux-arm64
retention-days: 7
- name: Materialize libkrun prefix (upstream package)
# Mirrors the amd64 job: minvmd links libkrun's KVM backend
# dynamically, so it can't join the static-musl build above — it
# needs a native glibc build against a real libkrun. This runner
# IS aarch64, so the native target and the guest arch agree.
# Reuse the static mip just built (renamed above) rather than
# letting the fetch compile a second, debug mip to drive the
# cache pull.
uses: ./.github/actions/setup-libkrun-linux
with:
arch: aarch64
mip: ${{ github.workspace }}/target/aarch64-unknown-linux-musl/release/mip-linux-arm64
- name: Build native minvmd binary
# Native host target (aarch64-unknown-linux-gnu), NOT musl: links
# libkrun.so.1, so unlike the three above it is not
# self-contained — the installer ships it under lib/ and the
# RUNPATH rewrite below points at it.
run: |
cargo build --release --locked -p minvmd --bin minvmd
mv target/release/minvmd target/release/minvmd-linux-arm64
- name: Rewrite + verify minvmd/libkrun linkage (RUNPATH)
run: ./scripts/rewrite-linux-linkage.sh target/release/minvmd-linux-arm64 "$LIBKRUN_PREFIX"
- name: Stage libkrun for upload
run: |
mkdir -p "$RUNNER_TEMP/libs"
cp "$(readlink -f "$LIBKRUN_PREFIX/libkrun.so.1")" "$RUNNER_TEMP/libs/libkrun-linux-arm64.so"
- name: Upload binary (minvmd)
uses: actions/upload-artifact@v7
with:
name: minvmd-linux-arm64
path: target/release/minvmd-linux-arm64
retention-days: 7
- name: Upload libkrun (linux-arm64)
uses: actions/upload-artifact@v7
with:
name: libkrun-linux-arm64
path: ${{ runner.temp }}/libs/libkrun-linux-arm64.so
retention-days: 7

build-release-macos-arm64:
# Self-hosted Apple Silicon runner. minvmd links libkrun (Hypervisor.framework
Expand Down
3 changes: 3 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

59 changes: 59 additions & 0 deletions crates/common/tests/script_harnesses.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
//! Convention-discovered gate that runs the `scripts/*_test.sh` harnesses.
//!
//! Sibling of [`shell_lint`](./shell_lint.rs), which only *lints* `scripts/`.
//! A harness that no lane runs is not a gate: `just test-linkage` was the only
//! executable proof of `rewrite-linux-linkage.sh`'s RUNPATH logic, and `just
//! ci` does not include it, `ci-shell-installer.yml` hardcodes install.sh, and
//! `shell_lint` only drives shellcheck — so a regression that dropped the
//! `$ORIGIN/../lib` entry would have been green on all five required
//! aggregators and surfaced only as `libkrun.so.1: cannot open shared object
//! file` on users' machines.
//!
//! Widening the frozen workflow YAML is not an option (CODEOWNER-gated), so
//! this hangs the harness off the workspace test suite the always-running Linux
//! lanes already execute — the reviewed-code extension point CI schedules over
//! (docs/ci-strategy.md §10).
//!
//! `install_test.sh` is deliberately NOT run here: `ci-shell-installer.yml`
//! already runs it under three shells, and duplicating it would double a
//! multi-minute job for no added coverage.

use std::path::PathBuf;
use std::process::Command;

fn repo_root() -> PathBuf {
// CARGO_MANIFEST_DIR is crates/common; the workspace root is two up.
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.parent()
.and_then(|p| p.parent())
.expect("workspace root two levels above crates/common")
.to_path_buf()
}

/// Runs one harness and asserts it exits zero, surfacing its output on failure.
///
/// The harnesses stub every external tool they need, so this holds on any host
/// — including macOS, where `rewrite-linux-linkage.sh` itself can never run.
fn run_harness(name: &str) {
let root = repo_root();
let harness = root.join("scripts").join(name);
assert!(harness.is_file(), "missing script harness: {name}");

let output = Command::new("bash")
.arg(&harness)
.current_dir(&root)
.output()
.unwrap_or_else(|e| panic!("failed to run scripts/{name}: {e}"));

assert!(
output.status.success(),
"scripts/{name} failed\n--- stdout ---\n{}\n--- stderr ---\n{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr),
);
}

#[test]
fn rewrite_linux_linkage_harness_passes() {
run_harness("rewrite-linux-linkage_test.sh");
}
Loading