Skip to content

ci(macos): move unit tier to GitHub-hosted, reserve the mini for hypervisor e2e - #696

Merged
norrietaylor merged 23 commits into
mainfrom
ci/mac-unit-tier
Jul 10, 2026
Merged

ci(macos): move unit tier to GitHub-hosted, reserve the mini for hypervisor e2e#696
norrietaylor merged 23 commits into
mainfrom
ci/mac-unit-tier

Conversation

@norrietaylor

Copy link
Copy Markdown
Member

What

PR3 of the CI refactor (#687), stacked on #694 (base = ci/libkrun-own-build; retarget after it merges).

The mini's build-macos job needed macOS but not the hypervisor — except the krun_smoke FFI smoke, which GitHub-hosted macOS VMs can't run. This splits along exactly that line:

  • New unit job on macos-latest: cargo clippy -p minvmd --all-targets -- -D warnings, cargo test -p minvmd, cargo test -p sessions, with rust-cache and the pinned own-build libkrun via the composite. Kept lean — private repo, mac minutes bill at 10×.
  • krun_smoke moves into e2e: the harness joins the existing --no-run build (before the single codesign — cargo must never run after signing), then runs as a direct binary. No kernel/rootfs env set → bring-up-only path, no entitlement needed. Building the harness also builds krun_smoke_child and bakes its CARGO_BIN_EXE path; build and run are same-machine, so it resolves.
  • New production-linkage static check in e2e: scripts/rewrite-macos-linkage.sh runs against a throwaway copy of the just-built minvmd, so a change that would break the shipped bin/../lib layout (lost @rpath load command, leaked non-system dylib) reds the PR instead of the next release.
  • build-macos deleted — the mini now runs exactly one job per mac-path PR (one checkout, one build, one codesign).

Descoped: cargo test -p minimal on macOS

Planned, but impossible today: the CLI's dev-dependencies pull minimald (test-support), whose ungated hakoniwa → libcgroups → procfs chain only builds on Linux (cargo build -p minimal is fine — no dev-deps; that's why the autospawn e2e works). Documented in-line in the workflow. Unblocking it means cfg(target_os = "linux")-gating minimald's sandbox deps — a code change worth its own PR if the coverage is wanted.

Verification (local, arm64 macOS, against the pinned own-build libkrun)

This PR's own run exercises the new unit job and the reshaped e2e on the mini — check the mini log for one build + one codesign, and krun_smoke + linkage check green before the VM steps.

🤖 Generated with Claude Code

norrietaylor and others added 14 commits July 9, 2026 08:28
…facts

Four new composite actions dissolve copy-pasted setup blocks:

- setup-rust: free-disk + protoc + rust-cache preamble, previously
  inlined in ci.yml clippy AND duplicated inside core-tests
- setup-libkrun-macos: the brew slp/krun tap install, previously
  copy-pasted across ci-macos.yml and release.yml (the release job's
  separate dylib-existence check is dropped like ci-macos's was in
  #678: a failed brew install already fails the step, and a missing
  libkrun still fails loudly at link/rewrite time)
- setup-libkrun-linux: fetch-libkrun.sh + the LIBKRUN_PREFIX /
  LD_LIBRARY_PATH exports, previously duplicated between
  ci-linux-kvm.yml and release.yml's amd64 build
- guest-artifacts: the kernel + rootfs cache pulls, previously
  repeated in ci-macos.yml, ci-linux-kvm.yml, and release.yml

The two fetch composites retry 3x to absorb transient cache/network
hiccups; commands, arguments, and env are otherwise unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ~60-line inline install_name_tool rewrite + otool verification in
build-release-macos-arm64 (grown across #664/#668/#670) moves to
scripts/rewrite-macos-linkage.sh, command-for-command. A script lets
CI exercise the production @rpath rewrite on a throwaway copy of the
debug binary later, instead of the rewrite only ever running at
release time. Signing stays the caller's job (Developer ID, last
mutation before upload, per #680).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Install Minimal step guarded on
steps.cache-minimal.outputs.cache-hit, but no step with id
cache-minimal exists (leftover from a removed cache step), so the
condition was always true. Remove it; the step runs unconditionally
as it already did in practice.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three latent bugs surfaced by review once the previously-inline code
became a parameterized, reusable script:

- otool -L's header line is the binary's own path; a binary living
  under a path containing "libkrun" (e.g. a throwaway CI copy) would
  false-match and silently skip the real rewrite. Skip the header
  (NR>1).
- install_name_tool -rpath errors when the bare @loader_path entry is
  already gone, so a second run on an already-rewritten binary
  hard-failed. Retarget only when the dev rpath is present.
- set -o pipefail aborted the `current=` capture before the annotated
  "no libkrun load command" diagnostic could fire when otool itself
  fails. Guard the capture with || true; the -z check handles both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Trigger the VM lanes when their composite actions change:
  ci-macos.yml and ci-linux-kvm.yml path filters now include the
  composites their setup runs through, so a composite-only edit can
  no longer merge unexercised and break the lanes post-merge.
- Restore the libkrun.dylib existence check in setup-libkrun-macos:
  brew exits 0 for an installed-but-unlinked keg, so "a failed
  install fails the step" was unsound on a persistent shared runner;
  without the check a release build dies later with a raw
  `ld: library 'krun' not found` instead of a provisioning pointer.
- Fail fast on deterministic errors: the mip CLI is validated/built
  once, outside the retry loops (guest-artifacts, setup-libkrun-linux),
  so a compile error or bad mip path fails in one attempt instead of
  being re-run into the job timeout. Only the network-bound
  materialize retries.
- One retry implementation: scripts/ci/retry.sh replaces the three
  hand-copied divergent loops, and the previously-unretried gvproxy
  downloads (ci-linux-kvm + release) now use it too.
- release.yml: drop the inline LIBKRUN_PREFIX re-hardcode (the
  composite publishes it); document why the job-wide LD_LIBRARY_PATH
  is safe (the prefix holds only libkrun/libkrunfw by construction);
  fix the linkage-step comment that implied a CI consumer exists.

Not fixed: ci-netns.yml still inlines its rust preamble — its
free-disk config differs (no remove_tool_cache) and the file is
slated for deletion when the networking proofs are mothballed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integrates #672 (per-VM /dev/vdb volume) with the composite extraction:

- The two copies of main's new libkrun >= 1.19.0 (krun_add_disk3)
  symbol check in ci-macos.yml fold into the setup-libkrun-macos
  composite's verify step, so the release mac build gets it too.
- The KVM lane's new krun_add_disk3 assert now reads LIBKRUN_PREFIX
  (published by the setup-libkrun-linux composite) — the KRUN_PREFIX
  env it referenced was retired with the composite extraction, so the
  auto-merged step would have probed an empty path.
- main's crates/minimald/** path additions and the Session-E2E VM
  reap step (#588 fix for the merged e2e job) are kept as-is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Actions history for both VM lanes (last ~200 runs each, plus the
first attempts of every manually re-run run) records not a single
failure in the retried steps: the kernel/rootfs cache pulls, the
libkrun prefix fetch, and the gvproxy download have never failed.
The retries were insurance without receipts; remove them and the
now-unused scripts/ci/retry.sh, returning the composites to plain
extractions of the original steps (the mip prebuild and input
pre-validation existed only to keep deterministic work out of the
retry loops, so they go too).

The transient failures the history DOES show are apt-get installs —
the only main-branch KVM lane failure in the window and both of its
manual reruns died in "Install build dependencies". Retry belongs
there if anywhere, left for a separate change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The __krun-vmm-only reap from #672 is insufficient: main run
29032763009 — on the very commit that added it — still failed the
cold `minimal ls` with `ssh connect: Disconnected` against a fully
healthy guest (READY emitted, minimald listening on vsock:2222, no
connection ever accepted): the #588 bridge wedge. The remaining
leftover is the host gvproxy switch, which minvmd owns and
Guest::drop never kills; the failing runs' proxy-publish WARN
corroborates a lingering gvproxy. Reap any stray minvmd first (so
nothing respawns), then the VMM and gvproxy.

The proper fix — the session harness reaping its own process group —
stays tracked under #588; this keeps the merged e2e job's steps
isolated in the meantime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI previously tested against the slp/krun Homebrew bottle (a
full-feature build, version drifting with the tap) while the release
shipped our trimmed source build (blk,net; no gpu, no init-blob) —
which nothing executed before it reached users. Consolidate both on
one build:

- vendor/libkrun/libkrun.lock pins the containers/libkrun version AND
  its resolved commit (replaces the LIBKRUN_REF env pin; the build
  fetches the commit, so a moved tag cannot change what we build).
- scripts/build-libkrun-macos.sh builds the trimmed dylib at the pin,
  asserts self-containment and the krun_add_disk3 (>= 1.19.0) API
  floor, sets the install name to @rpath/libkrun.1.dylib, ad-hoc
  signs, and stages libkrun.1.dylib + a libkrun.dylib linker symlink
  into a prefix.
- setup-libkrun-macos builds on miss into a commit-keyed prefix
  ($HOME/.cache/minimal-ci/libkrun/<commit>) — actions/cache on
  GitHub-hosted runners, the persistent directory itself on the mini —
  verifies the staged dylib, and publishes LIBKRUN_PREFIX. minvmd's
  build.rs prefers LIBKRUN_PREFIX over /opt/homebrew and bakes it as
  an rpath, so a leftover brew libkrun on the runner is ignored; brew
  leaves the CI path entirely (and with it the installed-but-unlinked
  keg failure mode).
- release build-libkrun-macos-arm64 ships from the same composite:
  the shipped dylib is by construction the one every macOS CI lane
  linked and booted. Developer ID signing flow unchanged (#680).

With the @rpath install name, minvmd records @rpath/libkrun.1.dylib
at link time — verified locally: the script builds a self-contained
dylib (Hypervisor.framework/libiconv/libSystem only), minvmd links
with the @rpath load command + @loader_path and prefix rpaths, and
the binary loads. rewrite-macos-linkage.sh's -change becomes a
natural no-op; only its @loader_path -> @loader_path/../lib retarget
still mutates release binaries.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves against #692, which introduced the materialize composite
(channel-released prebuilt mip; no source build) for the guest
kernel/rootfs pulls — superseding this branch's guest-artifacts
composite, which wrapped the same call sites around
scripts/fetch-artifact.sh. Resolution: adopt materialize at every
conflicted call site (ci-macos artifacts, ci-linux-kvm, release
fetch-release-guest-artifacts, including main's removal of the now
unneeded toolchain/protoc/cargo-cache steps in the release fetch
job), delete the guest-artifacts composite, and point the VM lanes'
composite path filters at .github/actions/materialize/** instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integrates the concurrently pushed clean merge of main@71b364b4
(rcache refactor); this branch's own merge of main@cd1c95eb already
contains that content.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Base moved to main's materialize composite (#692); union the mac
lane's composite path filters (materialize replaces guest-artifacts,
the vendor/libkrun/** pin entry stays).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rvisor e2e

The mini's build-macos job (clippy + minvmd/sessions unit tests + FFI
smoke) needed macOS but not the hypervisor — except the krun_smoke
FFI smoke, which GitHub-hosted macOS VMs cannot run. Split along that
line:

- New `unit` job on macos-latest: clippy -p minvmd, cargo test -p
  minvmd / -p sessions, rust-cache, own-build libkrun via the
  composite. `cargo test -p minimal` was planned but is impossible on
  macOS today (dev-deps pull minimald, whose ungated hakoniwa ->
  libcgroups -> procfs chain is Linux-only) — documented in-line; the
  CLI's mac coverage stays the autospawn e2e.
- krun_smoke moves into `e2e`: harness built in the existing no-run
  step before the single codesign, run as a direct binary (bring-up
  path only — no kernel/rootfs env, no entitlement).
- New production-linkage static check in `e2e`: run
  scripts/rewrite-macos-linkage.sh on a throwaway copy of the
  just-built minvmd, so a change that breaks the shipped bin/../lib
  layout reds the PR instead of the next release.
- build-macos deleted; the mini now runs exactly one job per mac-path
  PR.

Verified locally on arm64 macOS against the pinned own-build libkrun:
clippy clean, minvmd + sessions tests pass, krun_smoke bring-up
passes as a direct binary, the linkage check passes on a copy, and
the minimal-tests failure mode was reproduced (procfs build error)
before being descoped.

Refs: #687

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 23 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 83aba89b-2d90-455c-bc16-2dfd2680f864

📥 Commits

Reviewing files that changed from the base of the PR and between 7777fa8 and 652b77b.

📒 Files selected for processing (1)
  • .github/workflows/ci-macos.yml

Comment @coderabbitai help to get the list of available commands.

norrietaylor and others added 2 commits July 9, 2026 16:18
#685 landed on main as a squash (790a422), so this branch's real
history of the same content conflicts textually. Resolution keeps
this branch's side everywhere both touched — it is the squash content
plus this PR's libkrun-consolidation edits (own-build composite,
provision step names, vendor/libkrun path entries).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
norrietaylor and others added 3 commits July 9, 2026 16:27
- Key the on-disk libkrun prefix by BOTH build inputs (pinned commit
  + build-script hash), not commit alone: the persistent mini could
  otherwise keep serving a dylib built by an older
  build-libkrun-macos.sh after a script change. This mirrors the
  invalidation the actions/cache key already gave GitHub-hosted
  runners; stale sibling prefixes on the mini are inert.
- cargo build --locked: build exactly upstream's committed
  Cargo.lock (verified present at the pin) so a silent dependency
  re-resolve cannot undermine the reproducible-build guarantee.

Verified locally: the --locked build completes at the pin and stages
into the new hash-suffixed prefix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
norrietaylor and others added 3 commits July 9, 2026 17:42
The e2e's combined `cargo build -p minvmd --bin minvmd -p minimal
--bin minimal` unified minvmd's default `libkrun` feature into the
CLI's default-features=false opt-out, so `minimal` linked libkrun —
the exact footgun release.yml already documents and avoids with
separate invocations. The brew era masked it: brew's libkrun carries
an absolute /opt/homebrew install name, so the mislinked CLI still
loaded. The own-build dylib's @rpath install name exposed it — the
autospawn e2e died with dyld "Library not loaded:
@rpath/libkrun.1.dylib / no LC_RPATH's found" from target/debug/
minimal, which bakes no rpaths.

Split the build (mirroring release.yml) and add the release job's
"minimal links only system libraries" assert to the e2e, so a
unification regression fails at build time with a pointed message
instead of a dyld error mid-test. Net effect of this PR's own-build
switch: CI now catches a mislinked CLI that brew silently tolerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Union of this branch's krun_smoke no-run build with the base's
feature-unification fix (separate cargo invocation for minimal +
system-libraries verify step).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Base automatically changed from ci/libkrun-own-build to main July 10, 2026 01:20
#694 landed on main as a squash (7777fa8), so this branch's real
history of the same content conflicts textually. Both hunks resolve
to this branch's side (squash content + this PR's unit-tier edits:
krun_smoke in the no-run build, rust-cache on the unit job).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@norrietaylor
norrietaylor merged commit cfd38e8 into main Jul 10, 2026
12 checks passed
@norrietaylor
norrietaylor deleted the ci/mac-unit-tier branch July 10, 2026 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants