ci(nightly): fix nightly flake and smoke-test the shipped artifacts before pushing unstable - #759
Conversation
Prove the just-built RELEASE binaries actually run before publishing any of them. Three new jobs download the uploaded artifacts by name, put the SHIPPED binaries on PATH (never a fresh target/ build), and run the ONE shared session e2e (scripts/session-e2e.sh — the same proof the CI lanes run): - smoke-linux-amd64 (ubuntu): DM2 native minimald — the static-musl min/minimald boot a host daemon and pass the session e2e. - smoke-linux-kvm (KVM-capable ubuntu): DM3 — boot a real microVM from the shipped native-glibc minvmd + shipped guest kernel/rootfs/initramfs + pinned gvproxy; libkrun is materialized from the upstream package (not shipped on Linux) via the shipped static mip. - smoke-macos (self-hosted mini): DM1 — lay the SIGNED min/minvmd/libkrun.1.dylib/ gvproxy + arm64 guest payload into the installer layout (libkrun.1.dylib a lib/ sibling of bin/, resolved via @loader_path/../lib) and boot the microVM. A new skip-tolerant smoke-success aggregator (if: always(), fails only on failure/cancellation) is added to the release job's needs, so a smoke failure blocks every publish step while a skipped smoke-macos (RUN_MACOS_CI kill-switch) reads as pass. Smoke has no !inputs.dry_run guard: a dry run builds, signs, assembles, and SMOKES, then publishes nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 12 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
Comment |
release.yml's verify-ci gate (#757) requests checks: read at the job level. When release.yml runs as a workflow_call from nightly.yml, it cannot hold any scope the caller's permissions block doesn't grant, and nightly.yml granted only contents: write and id-token: write — so every scheduled run since #757 merged died at workflow compile time with startup_failure (Jul 15 and Jul 16), cutting no nightly release. Manual release.yml dispatches kept working because a directly-dispatched workflow uses its own permissions, which is why this went unnoticed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the three smoke jobs and their skip-tolerant aggregator from release.yml into nightly.yml. The workflow_call jobs are part of the caller's run, so each smoke job still downloads the just-built artifacts by name and runs the one shared session e2e against the shipped bytes; what changes is what the smoke gates. In release.yml it gated the publish step of every release; here it gates promote-nightly, so a failed smoke leaves the nightly channel pointing at yesterday's build instead of blocking the release machinery itself. On a no-op night (HEAD already staged) release and the smokes both skip and the channel flip proceeds against bytes that were smoked the night they were cut. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
minvmd never resolves gvproxy from PATH — resolve_gvproxy_path() honours only MINVMD_GVPROXY_BIN, the installer bin dir ($MINIMAL_BIN / ~/.local/bin), and /usr/lib/minimal/bin. Renaming the shipped binary onto PATH therefore left both VM smokes switchless: gvproxy resolution is best-effort, so minvmd boots without guest egress and the in-guest pkgs clone during session mint dies with 'Could not resolve host' — the exact 10/10 failure the Jul 16 session-e2e-soak hit for the same reason. Set MINVMD_GVPROXY_BIN on both session-e2e steps, as the KVM and macOS lanes do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#760 deepened session-e2e.sh with the in-sandbox min add proof, whose session mint clones the pkgs upstream inside the guest, and added fetch-gvproxy + MINVMD_GVPROXY_BIN to the KVM and macOS lanes — but not to nightly-tests' session-e2e-soak. With no gvproxy staged, minvmd boots the VM switchless (resolution is best-effort and never errors), the guest has no egress or DNS, and every one of the soak's 10 iterations dies with 'Could not resolve host: github.com' (first seen on the Jul 16 scheduled run, the first after #760 merged). Stage the pinned gvproxy and point MINVMD_GVPROXY_BIN at it, exactly as the KVM lane does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What
Adds three smoke jobs to
nightly.ymlthat download the just-built release artifacts and prove the shipped bytes actually run — a daemon boots, a microVM boots, a session round-trips — before thenightlychannel is advanced. Each downloads the uploaded artifacts by name, puts the shipped binaries onPATH(never a freshtarget/build), and runs the ONE shared session e2e (scripts/session-e2e.sh, the same proof the CI lanes run).This PR also carries two related nightly fixes as separate commits:
startup_failures: #757'sverify-cigate requestschecks: read, a called workflow cannot hold a scope its caller doesn't grant, andnightly.ymlgranted onlycontents: write+id-token: write— so every scheduled run since ci(release): verify-ci gate, dry_run, --locked, arm64 recollapse #757 merged (Jul 15, Jul 16) died at workflow compile time and no nightly was cut. Manualrelease.ymldispatches kept working (a directly-dispatched workflow uses its own permissions), which is why it went unnoticed.nightly.ymlnow grantschecks: read.session-e2e-soakfailing 10/10 (nightly-tests.yml): #760's deepened e2e clones thepkgsupstream inside the guest, and test: prove the in-sandbox min add path via interactive PTY attach on every lane #760 staged gvproxy on the KVM and macOS lanes but not on the soak — so minvmd boots the soak's VM switchless and every iteration dies with "Could not resolve host" (first seen Jul 16, the first scheduled run after test: prove the in-sandbox min add path via interactive PTY attach on every lane #760). The soak now runsfetch-gvproxy.shand setsMINVMD_GVPROXY_BIN, exactly as the KVM lane does.The three smoke jobs
smoke-linux-amd64ubuntu-latestmin/minimaldauto-spawn a host daemon and pass the session e2e. Mirrors ci-linux-native'snative-daemon-e2e.smoke-linux-kvmubuntu-latest(KVM-capable, x86_64)minvmd+ shipped guest kernel/rootfs/initramfs + pinnedgvproxy, session round-trips over the vsock bridge. libkrun is not shipped on Linux, so it is materialized from the upstream package (reusing the shipped staticmipto drive the cache fetch — no Rust toolchain), exactly as the release build linked it. Mirrors ci-linux-kvm'stest-kvm, incl. its reap step.smoke-macos[self-hosted, macOS, ARM64](the mini; holds the signing keychain)min/minvmd/libkrun.1.dylib/gvproxy+ arm64 guest payload, lay them out in the installer layout (bin/{min,minvmd,gvproxy},lib/libkrun.1.dylibas abin/sibling → resolved via minvmd's@loader_path/../librpath,data/{vmlinuz,rootfs.img,initramfs.cpio}— exactly whatscripts/install.shwrites), then boot the microVM.Needs-graph (how smoke gates the channel flip)
release.yml's build jobs uploaded, by the same names, with no cross-run plumbing.promote-nightlynow needssmoke-successand requires it to besuccess. A red smoke meansnightlykeeps serving yesterday's proven build. Note the trade-off vs the release.yml placement: the GitHub release + GCS staging inside the called workflow complete before the smoke verdict — only the channel flip is gated.smoke-successis skip-tolerant, mirroring the repo's*-successlane aggregators:if: always(), reds only oncontains(needs.*.result, 'failure'|'cancelled'). Askippedsmoke reads as pass; a real failure/cancellation reds it and blocks the flip.releaseskips, all three smokes skip with it (plainneeds: [release]),smoke-successpasses vacuously, andpromote-nightlyre-points the channel at bytes that were smoked the night they were cut.smoke-macosis gated on the samevars.RUN_MACOS_CI != 'false'kill-switch as the mac build/sign jobs. Taking the mini offline skips it asskipped(notfailure), and the nightly still advances on the two Linux proofs.