Skip to content

ci(linux-kvm): split the monolith into build and test phases - #699

Merged
norrietaylor merged 2 commits into
ci/kvm-gate-readyfrom
ci/kvm-split
Jul 10, 2026
Merged

ci(linux-kvm): split the monolith into build and test phases#699
norrietaylor merged 2 commits into
ci/kvm-gate-readyfrom
ci/kvm-split

Conversation

@norrietaylor

Copy link
Copy Markdown
Member

What

PR6 of the CI refactor (#687), stacked on the gate-ready PR (base = ci/kvm-gate-ready; retarget to main after it merges).

The 60-minute build+test monolith meant a flake at minute 50 cost a full-hour rerun. Split into a clean build → test pipeline:

  • build-linux (no /dev/kvm needed): deps (retried apt), libkrun prefix, initramfs, minvmd, and the three e2e harnesses via cargo test --no-run --message-format=json — whose JSON writes a testbins.json manifest mapping target names to the exact hash-suffixed binaries produced. One kvm-testbed artifact ships the binaries, the manifest, the initramfs, and the libkrun prefix the binaries were linked against (unpacked to the same ~/.krun path on the test runner, satisfying both the baked rpath and LD_LIBRARY_PATH).
  • test-kvm (needs build-linux): no Rust toolchain, no cargo, no cargo cache — KVM udev rule, unpack testbed (exec bits restored from the manifest, MINVMD_BIN exported), materialize kernel/rootfs (prebuilt mip via the materialize action), fetch gvproxy, then boot / session / DM1-relay / daemon-lifecycle against the shipped binaries. A rerun of this job alone is ~15 minutes.
  • scripts/ci/run-testbin.sh resolves harnesses through the manifest — replaces the fragile ls -1t target/debug/deps/<name>-* | grep -v .d | head -1 pattern, and fails loudly on a missing entry.
  • Test-code enabler (the one non-CI change): boot_e2e and minimald_session_e2e spawned minvmd via compile-time env!("CARGO_BIN_EXE_minvmd") — an absolute path that only exists on the build runner. Both now prefer a MINVMD_BIN env override with fallback to the baked path; local cargo test is unchanged. (vsock_relay_e2e spawns no minvmd — verified — so it needs no override.)

Verification (local, before pushing)

  • The manifest jq pipeline run against a real --no-run JSON stream produces the expected {name: hash-suffixed-binary} map for all three harnesses.
  • run-testbin.sh resolves and execs a staged harness (self-skips cleanly without the env gates → exit 0) and errors correctly on a missing manifest entry.
  • The MINVMD_BIN change compiles (--no-run build) and cargo fmt --check is clean.
  • This PR's own run executes the full split pipeline end to end — the real proof that the shipped binaries boot VMs on a runner that never compiled them.

Drill before the ruleset flip (from #687)

Force a test-kvm-only rerun to confirm the ~15-minute figure, and break a harness deliberately on a scratch branch to confirm the aggregator goes red.

🤖 Generated with Claude Code

norrietaylor and others added 2 commits July 9, 2026 17:33
boot_e2e and minimald_session_e2e spawned minvmd through the
compile-time env!("CARGO_BIN_EXE_minvmd") absolute path, which only
exists on the machine that compiled the harness. The KVM lane's
build/test split ships prebuilt harnesses to a different runner, so
both now prefer a MINVMD_BIN env override and fall back to the baked
path — local `cargo test` behavior is unchanged.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 60-minute build+test monolith meant every flake rerun repeated
the full workspace build. Split it:

- build-linux (no KVM): deps, libkrun prefix, initramfs, minvmd, and
  the three e2e harnesses via `cargo test --no-run
  --message-format=json`, whose output writes a testbins.json
  manifest mapping target names to the exact hash-suffixed binaries
  produced. Everything ships as one `kvm-testbed` artifact,
  including the libkrun prefix the binaries were linked against.
- test-kvm (needs build-linux): no Rust toolchain, no cargo — KVM
  udev rule, unpack the testbed (restore exec bits, MINVMD_BIN,
  LIBKRUN_PREFIX/LD_LIBRARY_PATH), materialize kernel/rootfs
  (prebuilt mip via the materialize action), fetch gvproxy, then run
  boot / session / DM1-relay / daemon-lifecycle against the shipped
  binaries. A rerun of this job alone is ~15 minutes.
- scripts/ci/run-testbin.sh resolves harnesses through the manifest,
  replacing the fragile `ls -1t target/debug/deps/... | head` shape.

Verified locally: the manifest jq pipeline produces the expected
{name: binary} map from the --no-run JSON, run-testbin.sh resolves
and execs a staged harness (self-skipping cleanly without the env
gate) and fails loudly on a missing manifest entry, and the
MINVMD_BIN test change compiles.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e0141937-de4f-4432-b663-b96c41fb4aaa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

norrietaylor added a commit that referenced this pull request Jul 10, 2026
Generalize the native lane's daemon e2e into scripts/session-e2e.sh,
the SAME proof for all three deployment targets — the CLI already
abstracts where the daemon lives, so the script only needs knobs:

  Linux native (DM2)  no knobs (host daemon, repo as project)
  Linux KVM    (DM1)  E2E_VM=1 E2E_MINIMAL_ARGS=--minvmd
  macOS HVF           E2E_VM=1 (macOS is always VM-backed)

Flow everywhere: clean-state `minimal activate` (the cold activate IS
the autospawn) -> ls --raw -> `attach --command` exec asserting
stdout + exit (sessions are Linux environments on every target, so
`uname -s` == Linux even on macOS hosts) -> warm call -> destroy ->
stop. Timing reported, not asserted. VM targets get minvmd teardown
and guest-boot-log diagnostics. E2E_PROJECT_DIR parameterizes the
project: /tmp on VM targets — the guest sees no host project dir
until project sync lands, and /tmp exists in the guest image.
Portability handled: /tmp-rooted state dirs (macOS sun_path 104),
perl ms-clock fallback (BSD date has no %N).

The mac lane's autospawn step is retired: it only proved `minimal ls`
cold/warm against the spawned VM, which the unified e2e's cold
activate subsumes — and the CLI's macOS session path (activate /
attach / destroy over the vsock bridge) gets covered for the first
time. The native lane invokes the same script unchanged in behavior.
The KVM lane's adoption (E2E_MINIMAL_ARGS=--minvmd) follows once the
build/test split (#699) lands, to avoid cross-PR conflicts on that
file.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@norrietaylor
norrietaylor merged commit 62ae1aa into ci/kvm-gate-ready Jul 10, 2026
1 check passed
@norrietaylor
norrietaylor deleted the ci/kvm-split branch July 10, 2026 02:02
norrietaylor added a commit that referenced this pull request Jul 10, 2026
… apt retries (#698)

* ci(linux-kvm): make the lane gate-ready — changes filter, aggregator, apt retries

Converts the KVM lane to the always-triggered pattern (#687): the
trigger-level path filter moves into a `changes` job
(dorny/paths-filter, SHA-pinned, pull-requests: read), the e2e gates
on its output (workflow_dispatch runs everything), and a new
`if: always()` aggregator — ci-linux-kvm-success — reports success
when the e2e passed OR was path-skipped and red only on
failure/cancellation. The context joins the required checks after
the lane soaks green; until then the lane stays advisory exactly as
before.

Retries apt installs, the one step class with recorded transient
failures in lane history: the sole main-branch KVM failure (run
28897380911) and both of its manual re-runs died in "Install build
dependencies", and the macOS artifacts job failed twice in "Install
protoc". scripts/ci/retry.sh wraps those three call sites (KVM
deps, setup-rust protoc, mac-artifacts protoc); fetches stay
un-retried — zero recorded failures.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci(linux-kvm): split the monolith into build and test phases (#699)

* ci: drop the apt retry wrapper

Remove scripts/ci/retry.sh and unwrap its three call sites (KVM build
deps, setup-rust protoc, ci-macos artifacts protoc). One recorded apt
transient in lane history does not justify carrying retry machinery;
if apt flakes recur we can reintroduce it with fresh evidence.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant