Skip to content

ci(nightly): fix nightly flake and smoke-test the shipped artifacts before pushing unstable - #759

Merged
twitchyliquid64 merged 8 commits into
mainfrom
ci/pr9b-release-smoke
Jul 17, 2026
Merged

ci(nightly): fix nightly flake and smoke-test the shipped artifacts before pushing unstable#759
twitchyliquid64 merged 8 commits into
mainfrom
ci/pr9b-release-smoke

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 14, 2026

Copy link
Copy Markdown
Member

What

Adds three smoke jobs to nightly.yml that download the just-built release artifacts and prove the shipped bytes actually run — a daemon boots, a microVM boots, a session round-trips — before the nightly channel is advanced. Each downloads the uploaded artifacts by name, puts the shipped binaries on PATH (never a fresh target/ build), and runs the ONE shared session e2e (scripts/session-e2e.sh, the same proof the CI lanes run).

This PR also carries two related nightly fixes as separate commits:

  1. nightly startup_failures: #757's verify-ci gate requests checks: read, a called workflow cannot hold a scope its caller doesn't grant, and nightly.yml granted only contents: write + id-token: write — so every scheduled run since ci(release): verify-ci gate, dry_run, --locked, arm64 recollapse #757 merged (Jul 15, Jul 16) died at workflow compile time and no nightly was cut. Manual release.yml dispatches kept working (a directly-dispatched workflow uses its own permissions), which is why it went unnoticed. nightly.yml now grants checks: read.
  2. session-e2e-soak failing 10/10 (nightly-tests.yml): #760's deepened e2e clones the pkgs upstream inside the guest, and test: prove the in-sandbox min add path via interactive PTY attach on every lane #760 staged gvproxy on the KVM and macOS lanes but not on the soak — so minvmd boots the soak's VM switchless and every iteration dies with "Could not resolve host" (first seen Jul 16, the first scheduled run after test: prove the in-sandbox min add path via interactive PTY attach on every lane #760). The soak now runs fetch-gvproxy.sh and sets MINVMD_GVPROXY_BIN, exactly as the KVM lane does.

The three smoke jobs

Job Runner Proves
smoke-linux-amd64 ubuntu-latest DM2 (native minimald, no VM): the shipped static-musl min/minimald auto-spawn a host daemon and pass the session e2e. Mirrors ci-linux-native's native-daemon-e2e.
smoke-linux-kvm ubuntu-latest (KVM-capable, x86_64) DM3: boot a real microVM from the shipped native-glibc minvmd + shipped guest kernel/rootfs/initramfs + pinned gvproxy, session round-trips over the vsock bridge. libkrun is not shipped on Linux, so it is materialized from the upstream package (reusing the shipped static mip to drive the cache fetch — no Rust toolchain), exactly as the release build linked it. Mirrors ci-linux-kvm's test-kvm, incl. its reap step.
smoke-macos [self-hosted, macOS, ARM64] (the mini; holds the signing keychain) DM1 (macOS is always VM-backed): download the signed min/minvmd/libkrun.1.dylib/gvproxy + arm64 guest payload, lay them out in the installer layout (bin/{min,minvmd,gvproxy}, lib/libkrun.1.dylib as a bin/ sibling → resolved via minvmd's @loader_path/../lib rpath, data/{vmlinuz,rootfs.img,initramfs.cpio} — exactly what scripts/install.sh writes), then boot the microVM.

Needs-graph (how smoke gates the channel flip)

check ─► release (workflow_call: release.yml — builds, signs, stages, publishes)
              │
              ├─► smoke-linux-amd64 ─┐
              ├─► smoke-linux-kvm ───┤
              └─► smoke-macos ───────┤
                                     ▼
                        smoke-success (if: always())
                                     │
check, release ──────────► promote-nightly ◄─┘   (flips the nightly channel)
  • The workflow_call jobs are part of the caller's run, so the smoke jobs download the artifacts release.yml's build jobs uploaded, by the same names, with no cross-run plumbing.
  • Smoke blocks the channel advance, not the build. promote-nightly now needs smoke-success and requires it to be success. A red smoke means nightly keeps serving yesterday's proven build. Note the trade-off vs the release.yml placement: the GitHub release + GCS staging inside the called workflow complete before the smoke verdict — only the channel flip is gated.
  • smoke-success is skip-tolerant, mirroring the repo's *-success lane aggregators: if: always(), reds only on contains(needs.*.result, 'failure'|'cancelled'). A skipped smoke reads as pass; a real failure/cancellation reds it and blocks the flip.
  • No-op nights stay no-ops. When HEAD is already staged, release skips, all three smokes skip with it (plain needs: [release]), smoke-success passes vacuously, and promote-nightly re-points the channel at bytes that were smoked the night they were cut.
  • mac smoke stays skip-tolerant. smoke-macos is gated on the same vars.RUN_MACOS_CI != 'false' kill-switch as the mac build/sign jobs. Taking the mini offline skips it as skipped (not failure), and the nightly still advances on the two Linux proofs.

Prove the just-built RELEASE binaries actually run before publishing any of
them. Three new jobs download the uploaded artifacts by name, put the SHIPPED
binaries on PATH (never a fresh target/ build), and run the ONE shared session
e2e (scripts/session-e2e.sh — the same proof the CI lanes run):

- smoke-linux-amd64 (ubuntu): DM2 native minimald — the static-musl
  min/minimald boot a host daemon and pass the session e2e.
- smoke-linux-kvm (KVM-capable ubuntu): DM3 — boot a real microVM from the
  shipped native-glibc minvmd + shipped guest kernel/rootfs/initramfs +
  pinned gvproxy; libkrun is materialized from the upstream package (not
  shipped on Linux) via the shipped static mip.
- smoke-macos (self-hosted mini): DM1 — lay the SIGNED min/minvmd/libkrun.1.dylib/
  gvproxy + arm64 guest payload into the installer layout (libkrun.1.dylib a
  lib/ sibling of bin/, resolved via @loader_path/../lib) and boot the microVM.

A new skip-tolerant smoke-success aggregator (if: always(), fails only on
failure/cancellation) is added to the release job's needs, so a smoke failure
blocks every publish step while a skipped smoke-macos (RUN_MACOS_CI kill-switch)
reads as pass. Smoke has no !inputs.dry_run guard: a dry run builds, signs,
assembles, and SMOKES, then publishes nothing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 12 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: dea90c8c-0036-405f-99e6-eda41f0b9e86

📥 Commits

Reviewing files that changed from the base of the PR and between 9e9f15e and 78c2d88.

📒 Files selected for processing (2)
  • .github/workflows/nightly-tests.yml
  • .github/workflows/nightly.yml

Comment @coderabbitai help to get the list of available commands.

norrietaylor and others added 4 commits July 15, 2026 17:27
release.yml's verify-ci gate (#757) requests checks: read at the job
level. When release.yml runs as a workflow_call from nightly.yml, it
cannot hold any scope the caller's permissions block doesn't grant, and
nightly.yml granted only contents: write and id-token: write — so every
scheduled run since #757 merged died at workflow compile time with
startup_failure (Jul 15 and Jul 16), cutting no nightly release. Manual
release.yml dispatches kept working because a directly-dispatched
workflow uses its own permissions, which is why this went unnoticed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the three smoke jobs and their skip-tolerant aggregator from
release.yml into nightly.yml. The workflow_call jobs are part of the
caller's run, so each smoke job still downloads the just-built artifacts
by name and runs the one shared session e2e against the shipped bytes;
what changes is what the smoke gates. In release.yml it gated the
publish step of every release; here it gates promote-nightly, so a
failed smoke leaves the nightly channel pointing at yesterday's build
instead of blocking the release machinery itself. On a no-op night
(HEAD already staged) release and the smokes both skip and the channel
flip proceeds against bytes that were smoked the night they were cut.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@norrietaylor norrietaylor changed the title ci(release): smoke-test the shipped artifacts before publish ci(nightly): smoke-test the shipped artifacts before advancing the channel Jul 17, 2026
norrietaylor and others added 2 commits July 16, 2026 23:56
minvmd never resolves gvproxy from PATH — resolve_gvproxy_path() honours
only MINVMD_GVPROXY_BIN, the installer bin dir ($MINIMAL_BIN /
~/.local/bin), and /usr/lib/minimal/bin. Renaming the shipped binary
onto PATH therefore left both VM smokes switchless: gvproxy resolution
is best-effort, so minvmd boots without guest egress and the in-guest
pkgs clone during session mint dies with 'Could not resolve host' —
the exact 10/10 failure the Jul 16 session-e2e-soak hit for the same
reason. Set MINVMD_GVPROXY_BIN on both session-e2e steps, as the KVM
and macOS lanes do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#760 deepened session-e2e.sh with the in-sandbox min add proof, whose
session mint clones the pkgs upstream inside the guest, and added
fetch-gvproxy + MINVMD_GVPROXY_BIN to the KVM and macOS lanes — but not
to nightly-tests' session-e2e-soak. With no gvproxy staged, minvmd
boots the VM switchless (resolution is best-effort and never errors),
the guest has no egress or DNS, and every one of the soak's 10
iterations dies with 'Could not resolve host: github.com' (first seen
on the Jul 16 scheduled run, the first after #760 merged). Stage the
pinned gvproxy and point MINVMD_GVPROXY_BIN at it, exactly as the KVM
lane does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@norrietaylor norrietaylor changed the title ci(nightly): smoke-test the shipped artifacts before advancing the channel ci(nightly): fix nightly flake and smoke-test the shipped artifacts before advancing the channel Jul 17, 2026
@norrietaylor
norrietaylor marked this pull request as ready for review July 17, 2026 07:00
@norrietaylor norrietaylor changed the title ci(nightly): fix nightly flake and smoke-test the shipped artifacts before advancing the channel ci(nightly): fix nightly flake and smoke-test the shipped artifacts before pushing unstable Jul 17, 2026
@twitchyliquid64
twitchyliquid64 merged commit 1c2cdea into main Jul 17, 2026
28 checks passed
@twitchyliquid64
twitchyliquid64 deleted the ci/pr9b-release-smoke branch July 17, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants