Skip to content

test(common): gate the release job's CLI invocations pre-merge - #1044

Closed
norrietaylor wants to merge 1 commit into
mainfrom
chore/release-cli-smoke
Closed

test(common): gate the release job's CLI invocations pre-merge#1044
norrietaylor wants to merge 1 commit into
mainfrom
chore/release-cli-smoke

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 29, 2026

Copy link
Copy Markdown
Member

Closes #1035.

Problem

release.yml's Generate completions step shells out to the freshly built mip / min / minimald binaries — nine invocations at release.yml:724-732. They only ever execute on a workflow_dispatch release, so a breaking CLI change clears every PR gate, merges, and sits until someone cuts a release. That is exactly how #1009 (completions <shell>completions print <shell>) shipped: twenty days later it exited 2 on run 30425725370 and took the GCS archive upload, the GitHub Release, and the whole stage-installer job down with it, an hour of build time in. #1034 fixed the call site; the structural gap stayed.

Fix

scripts/release-cli-smoke.sh reads those invocations back out of release.yml and replays them against the debug binaries. Nothing here is a hand-maintained copy of the release job's command list — both the artifact→binary map (from the mv min minimal-linux-amd64 rename step) and the invocations themselves are parsed from the workflow, so a call site the release job adds or edits is covered on the next run without touching the harness. Discovering zero invocations is a hard failure, never a green skip.

Each invocation is checked three ways, because exit 0 alone is too weak:

  1. exits 0 and writes a non-empty file;
  2. the shell it generates for matches the directory it is written to;
  3. the shim registers the command its destination filename implies — bash/<cmd>, zsh/_<cmd>, fish/<cmd>.fish.

Check 3 is the one an exit-code-only gate would have missed: the #737 binary rename left the release job writing min's completions to files named minimal, which no shell autoloads, and that shipped silently until #1034. The check handles both shapes clap emits — mip/minimald's static (aot) generator and min's dynamic (env) registration shim.

.github/workflows/ is frozen and CODEOWNER-gated, so CI reaches the harness the same way it reaches scripts/lint-shell.sh (#899): a convention-discovered workspace test, crates/common/tests/release_cli_smoke.rs, run by the always-on Linux lanes' core-tests suite — the reviewed-code extension point in docs/ci-strategy.md §10. Plus just test-release-cli for a direct local run.

The test passes --build so a bare cargo test -p common builds what is missing rather than silently covering nothing. The build uses --workspace --bin <name> rather than -p <package> --bin <name> deliberately: selecting the whole workspace resolves features exactly as the workspace test build does, so after the core-tests suite it is a link or a no-op instead of a rebuild of every shared dependency under a narrower feature set (measured: 3 min for the narrow form locally).

Verification

Audit

Per the issue's request to check the rest of the release path: the completions step is the only place the release and stage-installer jobs invoke a shipped binary. Everything else they run is gcloud, gh, tar, or a reviewed scripts/ helper that already has its own harness (install.shinstall_test.sh, verify-nightly-provenance.shverify-nightly-provenance_test.sh). Recorded in the script's SCOPE header, with a pointer to where discovery widens if a later step starts calling one of the binaries.

Complements #687's PR9b (post-release artifact smoke); this is the pre-merge half.

🤖 Generated with Claude Code

Note

Gate release job CLI invocations with a pre-merge smoke test

  • Adds scripts/release-cli-smoke.sh, a Bash harness that reads the release workflow's CLI completion invocations, maps each artifact to a local binary, builds missing binaries if --build is passed, runs them to generate completions, and validates output is non-empty and registers the expected command.
  • Adds a Rust test release_job_cli_invocations_still_work in crates/common/tests/release_cli_smoke.rs that invokes the script with --build and fails the test suite if the script exits non-zero.
  • Adds a test-release-cli recipe to justfile for running the smoke check locally.
  • minimald invocations are skipped on non-Linux platforms via a skip_on_this_os guard.
  • Risk: the test runs cargo build --workspace --bin for each missing binary, so CI time will increase proportionally to how many binaries are not already built.

Macroscope summarized 7898557.

release.yml's "Generate completions" step runs the freshly built
mip / min / minimald binaries, but only on a dispatched release — so a
breaking CLI change clears every PR gate and only surfaces an hour into
a release, taking the GCS upload, the GitHub Release and stage-installer
down with it. That is how the `completions <shell>` → `completions print
<shell>` split (#1009) shipped broken for twenty days (#1034).

scripts/release-cli-smoke.sh reads those invocations back out of the
workflow — both the artifact→binary map from the rename step and the
commands themselves — and replays them against the debug binaries, so
there is no second copy of the command list to drift. Each is checked
for exit 0 and a non-empty file, that the shell it generates for matches
the directory it lands in, and that the shim registers the command its
destination filename implies. The last check is the one an exit-code-only
gate would have missed when the #737 binary rename left `min`'s
completions in files named `minimal`. Discovering zero invocations is a
hard failure, so a moved or reshaped step cannot pass green having
replayed nothing.

`.github/workflows/` is frozen, so CI picks the harness up the way
scripts/lint-shell.sh is: a convention-discovered workspace test
(crates/common/tests/release_cli_smoke.rs) plus a `just test-release-cli`
recipe.

Verified by pointing the harness at the pre-#1034 workflow — the three
`min completions <shell>` calls fail with `unrecognized subcommand`.
minimald's three invocations self-skip on macOS (Linux-only crate) and
run for real on the Linux lanes.

Closes: #1035

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Caution

Review failed

An error occurred during the review process. Please try again later.


Comment @coderabbitai help to get the list of available commands.

@norrietaylor
norrietaylor deleted the chore/release-cli-smoke branch July 29, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release-job CLI invocations have no pre-merge coverage — a breaking min/mip change only surfaces on a release run

1 participant