(via norrietaylor): fix(release): verify smoke tests actually ran in nightly provenance chec - #911
Merged
macroscopeapp[bot] merged 1 commit intoJul 22, 2026
Conversation
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
norrietaylor
added a commit
that referenced
this pull request
Jul 26, 2026
* docs: document the release and install pipeline Add docs/internal/release-pipeline.md, a maintainer-facing runbook for the release/install pipeline: release.yml, the stable/unstable/nightly channels, staging, promotion, pruning, and the operator steps. Marked internal (publicly visible in the repo, not published to the docs site). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document nightly-provenance gate and override_provenance input in release-pipeline.md (#900) Co-authored-by: macroscopeapp[bot] <170038800+macroscopeapp[bot]@users.noreply.github.com> * docs: remove the private docs-repo reference from the promote runbook The reference-docs promotion dispatch named the docs-site repo and its GitHub App. That repo is private, so this public repo should not announce it, and the promote pipeline is being reshaped besides. Describe the `reference-docs-promoted` dispatch abstractly (it triggers a docs.minimal.dev rebuild) without naming the target repo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): verify smoke tests actually ran in nightly provenance check (#911) Co-authored-by: macroscopeapp[bot] <170038800+macroscopeapp[bot]@users.noreply.github.com> * docs: remove em-dashes and fix the version-source binary in the release runbook * fix(scripts): close the no-op-nightly loophole in the provenance gate smoke-success is if: always() and skip-tolerant, so a no-op nightly run (SHA already staged, smokes skipped) still concluded success and could vouch for a version that was never smoked. The gate now also requires the Linux smoke jobs themselves to have run and passed; smoke-macos alone may be skipped by the RUN_MACOS_CI kill-switch. Test fixtures move to name/conclusion pairs and cover the real no-op shape and the kill-switch shape (31 tests). The runbook text now states the guarantee precisely, drops the unsourced retirement claim, and documents the channel-endpoint target override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: macroscopeapp[bot] <170038800+macroscopeapp[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds verification that smoke tests actually ran (and passed) before allowing nightly provenance checks to succeed.
Changes
Updated
scripts/verify-nightly-provenance.shto:smoke-successaggregator job hasconclusion == "success"(not "skipped")This ensures that versions promoted via nightly provenance actually completed the full smoke test suite, preventing releases that bypassed critical validation.
Testing
Added test coverage for:
smoke-successjobsmoke-successjobsmoke-successjobsmoke-successjobNote
Macroscope: Fix It For Me
Activity
Currently: Auto-merged 934546a
Previously
Note
Verify smoke tests passed before accepting nightly provenance in
verify-nightly-provenance.shsmoke-successjob exists with conclusionsuccess; non-success conclusions or a missing job cause a non-zero exit with a descriptive error message.smoke-successjob status.success,skipped,failure,cancelled, and missingsmoke-successoutcomes via canned stub responses.smoke-successjob.Macroscope summarized 4c8a03f.