ci: add the nightly test tier by norrietaylor · Pull Request #720 · gominimal/minimal

norrietaylor · 2026-07-11T00:41:34Z

Part of the CI end-state migration (#687) — the mid-soak nightly tier. Touches no lane workflow (soak-safe).

What

New .github/workflows/nightly-tests.yml at 06:00 UTC — distinct from nightly.yml, which cuts the release channel at 10:00:

advisories — cargo-deny check advisories, blocking. The between-PRs counterpart of the PR-side check; catches RUSTSEC advisories published since the last PR touched the tree, and feeds notify.
session-e2e-soak — builds minvmd + the CLI, materializes the guest images, then loops the unified session e2e ×10 via scripts/soak-session-e2e.sh (reap between iterations) to shake out the boot / first-connect wedge class a single PR-lane run can't surface. Shares (and re-warms) the KVM lane's build cache.
beta-canary + update-canary — the workspace suite on beta rustc and on freshly-cargo updated deps, both continue-on-error (the 1.97 pin and the lockfile are the gate); both pass save-if: false so a throwaway build never consumes the shared cache pool.
hygiene — actionlint (pinned rhysd/actionlint image) + zizmor + cargo-machete, non-blocking.
notify — files or updates a single ci-nightly-labelled tracking issue when a blocking job fails or is cancelled, so a regression doesn't rot in the Actions tab.

Supporting:

scripts/reap-vms.sh — the leaked-__krun-vmm/gvproxy reap extracted to one place, used by the soak (the frozen lanes can adopt it later).
core-tests gains a save-if passthrough (the canaries need it).
promote.yml — the last missing timeout-minutes in the repo.

The CI-strategy doc + test-extension contract (and the junit step-summary proof) are deferred to a follow-up — they document the convention-based auto-discovery that isn't built yet.

Review

This branch already went through a max-effort /code-review; the mechanical findings are fixed here (prebuilt cross instead of cargo install, ITER validated + pass == ITER assertion, notify covers cancellation and dedups by label). The documentation-accuracy findings travel with the deferred docs branch.

Verification

Scripts shellcheck-clean; workflows parse; the ITER guard rejects 0/non-numeric.
The nightly tier is workflow_dispatch-able — the definitive proof is one manual dispatch (soak survives ×10; notify wiring exercised by a forced failure). The zizmor/actionlint installs get their first-dispatch confirmation there too.

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Enhanced nightly CI to include an additional installer verification stage, with notifications and failure tracking updated accordingly.
- Added automated issue creation for failures in key nightly tests.
- Added an e2e soak harness to repeatedly run session end-to-end tests and collect boot logs.
Bug Fixes
- Improved cleanup of leftover VM processes after failed or interrupted tests.
- Limited promotion jobs to 20 minutes.
- Restricted Rust cache writes to the main branch by default.

coderabbitai · 2026-07-11T00:41:53Z

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 46 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 337b0174-3d1e-4c1b-9b0c-25ac9026e964

📥 Commits

Reviewing files that changed from the base of the PR and between 7890db8 and 6642bde.

📒 Files selected for processing (3)

.github/workflows/nightly-tests.yml
scripts/reap-vms.sh
scripts/soak-session-e2e.sh

📝 Walkthrough

Walkthrough

Adds scheduled nightly CI with advisory, VM soak, installer, canary, hygiene, and failure-notification jobs. It introduces VM cleanup and soak scripts, gates Rust cache writes by branch, makes installer checks reusable, and limits promotion execution time.

Changes

Nightly CI

Layer / File(s)	Summary
Cache-write control `.github/actions/core-tests/action.yml`	Adds a `save-if` input defaulting to the `main` branch and forwards it to Rust setup.
VM soak helpers `scripts/reap-vms.sh`, `scripts/soak-session-e2e.sh`	Adds stranded-process cleanup and repeated session end-to-end execution with per-iteration logs and result tracking.
Nightly scheduling and blocking checks `.github/workflows/nightly-tests.yml`	Adds scheduled/manual triggers, advisory checks, and a KVM-backed ten-iteration session soak job.
Reusable installer validation `.github/workflows/ci-shell-installer.yml`, `.github/workflows/nightly-tests.yml`	Makes the installer workflow callable and invokes it from nightly CI.
Canaries, hygiene, and failure reporting `.github/workflows/nightly-tests.yml`	Adds beta, dependency-update, workflow hygiene, and conditional issue-notification jobs, including installer status.
Promotion execution limit `.github/workflows/promote.yml`	Sets a 20-minute timeout for the promote job.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant nightly-tests
  participant session-e2e-soak
  participant soak-session-e2e.sh
  participant reap-vms.sh
  participant session-e2e.sh
  participant microVM
  nightly-tests->>session-e2e-soak: start nightly soak job
  session-e2e-soak->>soak-session-e2e.sh: run ten iterations
  soak-session-e2e.sh->>reap-vms.sh: clean stranded VM processes
  soak-session-e2e.sh->>session-e2e.sh: run session test
  session-e2e.sh->>microVM: boot and exercise session
  microVM-->>session-e2e.sh: return result and boot log
  session-e2e.sh-->>soak-session-e2e.sh: report iteration status

Possibly related PRs

gominimal/minimal#620: Introduced the core-tests composite action extended here.
gominimal/minimal#703: Directly overlaps with save-if cache-write gating.
gominimal/minimal#712: Also modifies the installer workflow used by nightly CI.

Suggested labels: github_actions

Suggested reviewers: bryan-minimal, jtnkminimal, msample, twitchyliquid64

Poem

I’m a rabbit in the nightly queue,
Hopping through ten VM loops anew.
Caches save on main’s green ground,
Stray microVMs get cleared out cleanly.
CI reports when failures brew—
And promotion gets a timer too!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding a new nightly CI test tier.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (1)

.github/workflows/nightly-tests.yml (1)
186-192: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider uploading the resolved Cargo.lock when this canary fails.

cargo update mutates the lockfile in-place and it's discarded with the runner, so a failure can't be reproduced without knowing which versions were resolved. An artifact upload (on failure) makes triage far easier.
♻️ Suggested addition
      - name: Upload resolved Cargo.lock
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: update-canary-cargo-lock
          path: Cargo.lock
          if-no-files-found: warn
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly-tests.yml around lines 186 - 192, Add an
always-running “Upload resolved Cargo.lock” step after the updated-dependency
core tests in the nightly workflow, using actions/upload-artifact@v7 with
artifact name update-canary-cargo-lock, path Cargo.lock, and if-no-files-found
set to warn.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/reap-vms.sh`:
- Around line 9-12: Update the best-effort cleanup commands in the reap script
to invoke sudo non-interactively by adding -n to each sudo pkill command,
ensuring VM teardown fails fast when passwordless sudo is unavailable.

In `@scripts/soak-session-e2e.sh`:
- Around line 30-46: Install EXIT, INT, and TERM traps immediately after
defining reap() so cleanup runs when the script is cancelled during
session-e2e.sh; keep or remove the final explicit reap since cleanup is
idempotent, ensuring orphaned VMs are always reaped.

---

Nitpick comments:
In @.github/workflows/nightly-tests.yml:
- Around line 186-192: Add an always-running “Upload resolved Cargo.lock” step
after the updated-dependency core tests in the nightly workflow, using
actions/upload-artifact@v7 with artifact name update-canary-cargo-lock, path
Cargo.lock, and if-no-files-found set to warn.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 01d1fb55-959a-4109-96d4-29038ad9c224

📥 Commits

Reviewing files that changed from the base of the PR and between 0281323 and 3a67920.

📒 Files selected for processing (5)

.github/actions/core-tests/action.yml
.github/workflows/nightly-tests.yml
.github/workflows/promote.yml
scripts/reap-vms.sh
scripts/soak-session-e2e.sh

New nightly-tests.yml (06:00 UTC, distinct from nightly.yml's release cut): - advisories: cargo-deny check advisories, BLOCKING (between-PRs counterpart of the PR-side check); - session-e2e-soak: builds minvmd + the CLI, then loops the unified session e2e x10 with an inter-iteration reap (scripts/soak-session- e2e.sh) to shake out the boot/first-connect wedge class a single PR-lane run can't surface; installs prebuilt cross like the KVM lane so the guest musl build never falls back to cargo install; - installer: re-runs the curl|sh installer suite (shellcheck + the POSIX-sh matrix incl. macOS frozen /bin/sh) via workflow_call, so runner-image shell drift is caught even when install.sh is untouched (the PR lane is path-scoped to it); - beta-canary + update-canary: workspace suite on beta rustc and on freshly-updated deps, continue-on-error (the pin/lock are the gate); - hygiene: actionlint (pinned image) + zizmor + cargo-machete; - notify: files/updates a single ci-nightly-labelled issue on a blocking-job (advisories/soak/installer) failure or cancellation. Supporting: - scripts/reap-vms.sh: the leaked-VM reap extracted to one place, used by the soak (the lanes can adopt it later); - ci-shell-installer.yml gains a workflow_call trigger so the nightly reuses its suite instead of copying the matrix; - core-tests gains a save-if passthrough so the canaries never write the shared cache pool; - promote.yml: the missing timeout-minutes on its promote job. The CI-strategy doc + test-extension contract are deferred to a follow-up (they describe the auto-discovery convention not yet built). Refs: #687 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/nightly-tests.yml:
- Around line 263-264: The nightly-tests notification body always labels
unsuccessful runs as “failed,” including cancellations. Update the body
construction near the `body` variable to derive an appropriate status from the
blocking job results and report “cancelled” when applicable, while retaining
“failed” for actual failures.
- Around line 203-227: Ensure every hygiene step runs even when installation or
an earlier check fails: remove job-level continue-on-error from hygiene and add
continue-on-error: true to the install, actionlint, cargo-machete, and zizmor
steps. Update the steps in the hygiene job identified by “Install cargo-machete
+ zizmor”, “actionlint (workflow lint)”, “cargo-machete (unused deps)”, and
“zizmor (workflow security)”.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c79c3520-d333-4c53-97f7-70ad60fbf298

📥 Commits

Reviewing files that changed from the base of the PR and between 3a67920 and 7890db8.

📒 Files selected for processing (6)

.github/actions/core-tests/action.yml
.github/workflows/ci-shell-installer.yml
.github/workflows/nightly-tests.yml
.github/workflows/promote.yml
scripts/reap-vms.sh
scripts/soak-session-e2e.sh

✅ Files skipped from review due to trivial changes (2)

.github/workflows/ci-shell-installer.yml
scripts/reap-vms.sh

🚧 Files skipped from review as they are similar to previous changes (3)

.github/workflows/promote.yml
scripts/soak-session-e2e.sh
.github/actions/core-tests/action.yml

Addresses review feedback on the nightly tier's VM reaping: - reap-vms.sh: use `sudo -n` so the best-effort teardown fails fast when passwordless sudo is unavailable instead of blocking on a password prompt in CI. - soak-session-e2e.sh: install EXIT/INT/TERM traps right after `reap` is defined so a mid-run cancellation (a cancelled job sends TERM) still reaps orphaned VM processes. The EXIT trap preserves the script's exit status, so the final explicit reap is now redundant and removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6SWKgFLoRffVAUrkAWtz6

Addresses review feedback on the nightly-tests workflow: - hygiene job: add `if: always()` to actionlint, cargo-machete, and zizmor so a failed tool install (or an earlier failing check) no longer skips the remaining checks — each reports on its own. The job stays non-blocking via job-level continue-on-error. - notify job: the trigger fires on failure or cancellation, so the issue body now reads "failed or was cancelled" rather than always "failed"; the per-job result lines already carry the exact outcome. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6SWKgFLoRffVAUrkAWtz6

Documents the frozen-CI contract for contributors and agents (N8): - CONTRIBUTING.md — the test-extension table (unit / in-process integration / doctest / *_integration.rs harness / scripts/ e2e) with the discriminator between an integration harness and a full-CLI e2e script; adding a test never requires a workflow edit. - docs/ci-strategy.md — the full CI design, with a preamble mapping the strategy's terms to this repo (five lane workflows; justfile + scripts, not xtask; _integration binary-suffix filtersets, not a vm_e2e_ test-name prefix) and the section 10 contract table updated to the shipped convention. - CLAUDE.md — a pointer so agents inherit the "do not edit .github/workflows/" rule. - .config/nextest.toml + core-tests — JUnit output plus a $GITHUB_STEP_SUMMARY pass/fail table, so per-PR test evidence is visible without artifact downloads. Reconciled against what already merged: the nightly tier, soak script, and promote timeout landed with #720, and a repo-wide CODEOWNERS landed with #728 (already gating .github/ via the minimalists team), so the parked branch's copies of those are dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HbXL6jxHTY2YuLPaMxc1yU

Documents the frozen-CI contract for contributors and agents (N8): - CONTRIBUTING.md — the test-extension table (unit / in-process integration / doctest / *_integration.rs harness / scripts/ e2e) with the discriminator between an integration harness and a full-CLI e2e script; adding a test never requires a workflow edit. - docs/ci-strategy.md — the full CI design, with a preamble mapping the strategy's terms to this repo (five lane workflows; justfile + scripts, not xtask; _integration binary-suffix filtersets, not a vm_e2e_ test-name prefix) and the section 10 contract table updated to the shipped convention. - CLAUDE.md — a pointer so agents inherit the "do not edit .github/workflows/" rule. - .config/nextest.toml + core-tests — JUnit output plus a $GITHUB_STEP_SUMMARY pass/fail table, so per-PR test evidence is visible without artifact downloads. Reconciled against what already merged: the nightly tier, soak script, and promote timeout landed with #720, and a repo-wide CODEOWNERS landed with #728 (already gating .github/ via the minimalists team), so the parked branch's copies of those are dropped. Claude-Session: https://claude.ai/code/session_01HbXL6jxHTY2YuLPaMxc1yU Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread scripts/reap-vms.sh Outdated

Comment thread scripts/soak-session-e2e.sh Outdated

norrietaylor force-pushed the ci/nightly-and-docs branch from 3a67920 to 7890db8 Compare July 11, 2026 03:47

coderabbitai Bot reviewed Jul 11, 2026

View reviewed changes

Comment thread .github/workflows/nightly-tests.yml

Comment thread .github/workflows/nightly-tests.yml Outdated

norrietaylor and others added 2 commits July 11, 2026 12:42

norrietaylor mentioned this pull request Jul 11, 2026

feature: CI refactor — clear build/test phases, three gating targets #687

Open

19 tasks

twitchyliquid64 approved these changes Jul 13, 2026

View reviewed changes

norrietaylor merged commit ee69cef into main Jul 13, 2026
29 checks passed

norrietaylor deleted the ci/nightly-and-docs branch July 13, 2026 16:50

norrietaylor mentioned this pull request Jul 14, 2026

docs,ci: land the CI strategy and test-extension contract #742

Merged

Navigation Menu

Uh oh!

ci: add the nightly test tier - #720

ci: add the nightly test tier#720
norrietaylor merged 3 commits into
mainfrom
ci/nightly-and-docs

norrietaylor commented Jul 11, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

norrietaylor commented Jul 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Review

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

norrietaylor commented Jul 11, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading