test(common): gate the release job's CLI invocations pre-merge - #1044
Closed
norrietaylor wants to merge 1 commit into
Closed
test(common): gate the release job's CLI invocations pre-merge#1044norrietaylor wants to merge 1 commit into
norrietaylor wants to merge 1 commit into
Conversation
release.yml's "Generate completions" step runs the freshly built mip / min / minimald binaries, but only on a dispatched release — so a breaking CLI change clears every PR gate and only surfaces an hour into a release, taking the GCS upload, the GitHub Release and stage-installer down with it. That is how the `completions <shell>` → `completions print <shell>` split (#1009) shipped broken for twenty days (#1034). scripts/release-cli-smoke.sh reads those invocations back out of the workflow — both the artifact→binary map from the rename step and the commands themselves — and replays them against the debug binaries, so there is no second copy of the command list to drift. Each is checked for exit 0 and a non-empty file, that the shell it generates for matches the directory it lands in, and that the shim registers the command its destination filename implies. The last check is the one an exit-code-only gate would have missed when the #737 binary rename left `min`'s completions in files named `minimal`. Discovering zero invocations is a hard failure, so a moved or reshaped step cannot pass green having replayed nothing. `.github/workflows/` is frozen, so CI picks the harness up the way scripts/lint-shell.sh is: a convention-discovered workspace test (crates/common/tests/release_cli_smoke.rs) plus a `just test-release-cli` recipe. Verified by pointing the harness at the pre-#1034 workflow — the three `min completions <shell>` calls fail with `unrecognized subcommand`. minimald's three invocations self-skip on macOS (Linux-only crate) and run for real on the Linux lanes. Closes: #1035 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Caution Review failedAn error occurred during the review process. Please try again later. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1035.
Problem
release.yml's Generate completions step shells out to the freshly builtmip/min/minimaldbinaries — nine invocations atrelease.yml:724-732. They only ever execute on aworkflow_dispatchrelease, so a breaking CLI change clears every PR gate, merges, and sits until someone cuts a release. That is exactly how #1009 (completions <shell>→completions print <shell>) shipped: twenty days later it exited 2 on run 30425725370 and took the GCS archive upload, the GitHub Release, and the wholestage-installerjob down with it, an hour of build time in. #1034 fixed the call site; the structural gap stayed.Fix
scripts/release-cli-smoke.shreads those invocations back out ofrelease.ymland replays them against the debug binaries. Nothing here is a hand-maintained copy of the release job's command list — both the artifact→binary map (from themv min minimal-linux-amd64rename step) and the invocations themselves are parsed from the workflow, so a call site the release job adds or edits is covered on the next run without touching the harness. Discovering zero invocations is a hard failure, never a green skip.Each invocation is checked three ways, because exit 0 alone is too weak:
bash/<cmd>,zsh/_<cmd>,fish/<cmd>.fish.Check 3 is the one an exit-code-only gate would have missed: the #737 binary rename left the release job writing
min's completions to files namedminimal, which no shell autoloads, and that shipped silently until #1034. The check handles both shapes clap emits —mip/minimald's static (aot) generator andmin's dynamic (env) registration shim..github/workflows/is frozen and CODEOWNER-gated, so CI reaches the harness the same way it reachesscripts/lint-shell.sh(#899): a convention-discovered workspace test,crates/common/tests/release_cli_smoke.rs, run by the always-on Linux lanes' core-tests suite — the reviewed-code extension point in docs/ci-strategy.md §10. Plusjust test-release-clifor a direct local run.The test passes
--buildso a barecargo test -p commonbuilds what is missing rather than silently covering nothing. The build uses--workspace --bin <name>rather than-p <package> --bin <name>deliberately: selecting the whole workspace resolves features exactly as the workspace test build does, so after the core-tests suite it is a link or a no-op instead of a rebuild of every shared dependency under a narrower feature set (measured: 3 min for the narrow form locally).Verification
min/mipchange only surfaces on a release run #1035 pre-merge. Pointed at the pre-fix(release): callmin completions printin the release job #1034 workflow, the Rust test fails withFAIL … exited 2 — error: unrecognized subcommand 'bash'for all threemin completions <shell>calls.mincompletions written to a file namedminimal(the refactor(minimal): name the CLI binary targetmin#737 class) → fails check 3; a zsh shim written tobash/→ fails check 2; an artifact with no rename line; a missing binary without--build; a workflow with no invocations → the zero-discovery guard.main: 6 passed, 3 skipped.minimaldis Linux-only (platform matrix), so its three invocations self-skip on macOS by name and run for real on the Linux lanes. 0.24 s when the binaries already exist.shellcheckclean;just lint-shell25/25;cargo fmt --check;cargo clippy -p common --all-targets -- -D warnings;cargo test -p common. Fulljust ciis not runnable on this host — the macOS scope excludescommon, so the new test's real run is the Linux lanes on this PR.Audit
Per the issue's request to check the rest of the release path: the completions step is the only place the
releaseandstage-installerjobs invoke a shipped binary. Everything else they run isgcloud,gh,tar, or a reviewedscripts/helper that already has its own harness (install.sh→install_test.sh,verify-nightly-provenance.sh→verify-nightly-provenance_test.sh). Recorded in the script's SCOPE header, with a pointer to where discovery widens if a later step starts calling one of the binaries.Complements #687's PR9b (post-release artifact smoke); this is the pre-merge half.
🤖 Generated with Claude Code
Note
Gate release job CLI invocations with a pre-merge smoke test
--buildis passed, runs them to generate completions, and validates output is non-empty and registers the expected command.release_job_cli_invocations_still_workin crates/common/tests/release_cli_smoke.rs that invokes the script with--buildand fails the test suite if the script exits non-zero.test-release-clirecipe to justfile for running the smoke check locally.minimaldinvocations are skipped on non-Linux platforms via askip_on_this_osguard.cargo build --workspace --binfor each missing binary, so CI time will increase proportionally to how many binaries are not already built.Macroscope summarized 7898557.