fix(ci): stop the dogfood cargo cache from silently freezing at ENOSPC - #1076
Merged
Conversation
The dogfood lane's Cargo-state cache entry grew by ~100 MB on every save (cargo never GCs, so each run appends the new lockfile's artifacts on top of every older one). At ~5.9 GB the archive stopped fitting on the runner and every save from 2026-07-27 on died with `zstd: error 70 : Write error : cannot write block : No space left on device`. actions/cache/save downgrades that to `##[warning]Failed to save`, so the job stayed green while the entry froze. Every run since restored the stale entry through restore-keys, rebuilt the tree from scratch (55 s of cargo became 10-13 min) and the lane only turned red when a slow runner overran the 20 min timeout, taking ci-success with it via the cancelled dogfood. Three changes, all in that job: - Reclaim the tool cache (~8 GB) and the cloud CLIs, so the archive has headroom to grow again. Nothing in the job reads either. - Verify the save actually landed, on main, and fail if it did not. The silent warning is why this ran for two days unnoticed. - 20 -> 30 min timeout, so a cold build on a slow runner is not a red required check. The frozen 42f102 entry has been pruned out of band; the next main run reseeds it from a build that no longer carries three lockfiles of dead artifacts.
📝 WalkthroughWalkthroughThe dogfood CI job now performs broader disk cleanup, has explicit permissions and a longer timeout, and verifies on ChangesDogfood cache reliability
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/ci.yml (1)
99-128: 📐 Maintainability & Code Quality | 🔵 TrivialGet
@gominimal/minimalistsreview before merging this workflow change..github/workflows/ci.ymlis owned by that team, and the repo treats.github/workflows/as frozen/CODEOWNER-gated.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/ci.yml around lines 99 - 128, Before merging the workflow changes in the ci job configuration, obtain review and approval from the `@gominimal/minimalists` CODEOWNER team, preserving the repository’s ownership requirements for .github/workflows/.Sources: Coding guidelines, Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/ci-verify-cache-saved.sh`:
- Around line 37-40: Update the gh api failure handling in the cache
verification flow so deterministic authentication, permission, and
repository-configuration failures such as 403/404 return a nonzero status
instead of exiting successfully. Preserve warning-only behavior for transient
API outages, and keep the existing cache-key context in diagnostics.
- Around line 37-43: Update the gh API request in the total cache lookup to
include the ref parameter scoped to refs/heads/main, while preserving the
existing key filter and result handling. Ensure the total count only includes
cache entries from the main branch before the save-check logic evaluates it.
---
Nitpick comments:
In @.github/workflows/ci.yml:
- Around line 99-128: Before merging the workflow changes in the ci job
configuration, obtain review and approval from the `@gominimal/minimalists`
CODEOWNER team, preserving the repository’s ownership requirements for
.github/workflows/.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: dd70a233-e7e3-4cda-ac43-427fc81eae41
📒 Files selected for processing (2)
.github/workflows/ci.ymlscripts/ci-verify-cache-saved.sh
Comment on lines
+37
to
+40
| if ! total="$(gh api "repos/$repo/actions/caches?key=$key" --jq '.total_count')"; then | ||
| echo "::warning::could not query the Actions cache API; skipping the save check for $key" | ||
| exit 0 | ||
| fi |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file=scripts/ci-verify-cache-saved.sh
wc -l "$file"
sed -n '1,140p' "$file"Repository: gominimal/minimal
Length of output: 2482
Fail closed on cache API auth errors
This path returns 0 for any gh api failure, including 403/404s from missing actions: read or bad token/repo setup. Keep transient outages warning-only if needed, but treat deterministic permission/configuration failures as errors so the cache check doesn’t silently skip.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ci-verify-cache-saved.sh` around lines 37 - 40, Update the gh api
failure handling in the cache verification flow so deterministic authentication,
permission, and repository-configuration failures such as 403/404 return a
nonzero status instead of exiting successfully. Preserve warning-only behavior
for transient API outages, and keep the existing cache-key context in
diagnostics.
Comment on lines
+37
to
+43
| if ! total="$(gh api "repos/$repo/actions/caches?key=$key" --jq '.total_count')"; then | ||
| echo "::warning::could not query the Actions cache API; skipping the save check for $key" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if [ "$total" -gt 0 ]; then | ||
| echo "cache entry present: $key" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- file listing ---\n'
git ls-files scripts/ci-verify-cache-saved.sh
printf '\n--- relevant section with line numbers ---\n'
cat -n scripts/ci-verify-cache-saved.sh | sed -n '1,140p'
printf '\n--- search for cache lookup usage ---\n'
rg -n "actions/caches|ref=refs/heads/main|cache saved|cache entry present|gh api" scripts -S
printf '\n--- repository guidance references ---\n'
rg -n "actions cache|cache API|refs/heads/main|branch-scoped caches|save check" -S .Repository: gominimal/minimal
Length of output: 246
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- file listing ---'
git ls-files scripts/ci-verify-cache-saved.sh
echo
echo '--- relevant section with line numbers ---'
cat -n scripts/ci-verify-cache-saved.sh | sed -n '1,140p'
echo
echo '--- search for cache lookup usage ---'
rg -n "actions/caches|ref=refs/heads/main|cache saved|cache entry present|gh api" scripts -S
echo
echo '--- repository guidance references ---'
rg -n "actions cache|cache API|refs/heads/main|branch-scoped caches|save check" -S .Repository: gominimal/minimal
Length of output: 4333
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- invocations of scripts/ci-verify-cache-saved.sh ---'
rg -n "ci-verify-cache-saved\.sh|verify-cache-saved" -S .
echo
echo '--- ci strategy excerpt ---'
sed -n '190,220p' docs/ci-strategy.md
echo
echo '--- cache API docs hints in repo ---'
rg -n "actions/cache/save|restore-keys|save-if|refs/heads/main|cache" docs scripts -SRepository: gominimal/minimal
Length of output: 27000
Scope the cache lookup to refs/heads/main.
Without ref=refs/heads/main, this check can count a cache from another branch with the same key and miss a failed save on main.
Proposed fix
- if ! total="$(gh api "repos/$repo/actions/caches?key=$key" --jq '.total_count')"; then
+ if ! total="$(gh api "repos/$repo/actions/caches?key=$key&ref=refs/heads/main" --jq '.total_count')"; then📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if ! total="$(gh api "repos/$repo/actions/caches?key=$key" --jq '.total_count')"; then | |
| echo "::warning::could not query the Actions cache API; skipping the save check for $key" | |
| exit 0 | |
| fi | |
| if [ "$total" -gt 0 ]; then | |
| echo "cache entry present: $key" | |
| if ! total="$(gh api "repos/$repo/actions/caches?key=$key&ref=refs/heads/main" --jq '.total_count')"; then | |
| echo "::warning::could not query the Actions cache API; skipping the save check for $key" | |
| exit 0 | |
| fi | |
| if [ "$total" -gt 0 ]; then | |
| echo "cache entry present: $key" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ci-verify-cache-saved.sh` around lines 37 - 43, Update the gh API
request in the total cache lookup to include the ref parameter scoped to
refs/heads/main, while preserving the existing key filter and result handling.
Ensure the total count only includes cache entries from the main branch before
the save-check logic evaluates it.
evanspearman
approved these changes
Jul 29, 2026
twitchyliquid64
approved these changes
Jul 29, 2026
norrietaylor
enabled auto-merge (squash)
July 29, 2026 20:11
twitchyliquid64
disabled auto-merge
July 29, 2026 20:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
Run #30482500938 (PR #1064) showed a red
ci-success. That job is only the messenger:dogfoodwas cancelled, not failed, and the aggregator accepts onlysuccessorskipped.dogfoodwas cancelled because it hittimeout-minutes: 20still mid-Compiling mip. Nothing in #1064 broke anything, it only touches.minimal/minimal.toml.Root cause
The dogfood Cargo-state cache had been frozen since 2026-07-27, and the freeze was invisible.
The entry grows on every save. Cargo never GCs, so each run restores the previous archive, appends the new lockfile's artifacts on top of every older one, and writes back bigger. From the run that seeded the last good entry (30290858553):
At 5918 MB the archive stopped fitting. Every save since:
actions/cache/savedowngrades that to a warning, so the job stayed green. Ten consecutivemainruns sampled back to 2026-07-27T20:35Z all end this way (30480737509, 30478041284, 30475727135, 30462382952, 30425606668, 30421718357, 30303268598, 30307192493, 30315690735, 30323044555), andgh api …/actions/cachesconfirmed exactly onedogfood-minimal-cargo-*entry existed,42f102…, created 2026-07-27T17:56Z, three lockfile changes behindmain.So each run restored a stale entry via
restore-keys, got little reuse across three lockfile bumps, and rebuilt the tree: 55 seconds of cargo became 10-13 minutes. Add ~1 min of disk reclaim and ~4 min spent downloading and extracting 5.9 GB that barely helps, andmainruns were landing at 16-17 of the 20 available minutes. This PR's run drew a slower runner (a 4-minute stall betweenclap_completeandargs) and crossed the line.Changes
All in the
dogfoodjob.Reclaim more disk.
remove_tool_cache: true(~8 GB) plus the two cloud CLIs, so the archive has headroom to grow again. Nothing in the job reads either: the JS actions use the runner's bundled node,run-taskis a shell composite, and the build fetches over plain HTTPS.Note for reviewers:
remove_packagestakes a space-separated list, not a boolean. The action'smain.shvalidates for an embedded space andexit 0s ontrue— passing a boolean would silently skip the android/dotnet/haskell reclaims too.Make a failed save loud. New
scripts/ci-verify-cache-saved.shchecks, onmain, that the key actually exists after the save step, and fails with adf -hdump if not. The silent warning is the reason this ran unnoticed for two days and then surfaced as an unrelated-looking timeout. A missing entry is a hard error; an unusable API is only a warning, so a token or outage problem cannot fail an otherwise good build. This needsactions: read, added job-level (which replaces the workflow-level grant, hencecontents: readrestated).timeout-minutes: 20->30. Not the fix, the guard: a genuinely cold build on a slow runner should not be a red required check.Out-of-band
The frozen
42f102…entry (6,205,485,947 B) has been deleted. That alone restores the fast path: the nextmainrun reseeds from a build that no longer carries three lockfiles of dead artifacts, and PRs get an exact-key hit again.Note on the freeze
.github/workflows/is CODEOWNER-gated and frozen, and the contract routes new logic toscripts/. The verification logic lives there and is picked up byscripts/lint-shell.sh(25/25 scripts pass shellcheck). The three workflow edits are job config that cannot live anywhere else: an action input, a permission, and a timeout. Owner-directed.Verification
scripts/lint-shell.shpasses (the convention test covers the new script).ci.ymlparses; thedogfoodjob resolves totimeout: 30,permissions: {contents: read, actions: read}, 6 steps.::error::+df -h+ exit 1, missing arg -> exit 2, unreachable repo ->::warning::+ exit 0.🤖 Generated with Claude Code
Summary by CodeRabbit
Note
Fix dogfood cargo cache to fail visibly on ENOSPC instead of silently freezing
scripts/ci-verify-cache-saved.sh, which queries the GitHub Actions caches API for a given key and exits 1 if no entry exists, helping detect when a cache save silently failed due to disk exhaustion.mainbranch runs where the cache was not a hit, a new CI step invokes this script to confirm the cache was actually saved after the build.azure-cliandgoogle-cloud-cli.main, the dogfood job now fails explicitly if the cargo cache save did not produce an entry, rather than succeeding silently.Macroscope summarized 54a9621.