CI: don't cache half-built deps from cancelled/failed builds - #550
Open
fedorov wants to merge 1 commit into
Open
CI: don't cache half-built deps from cancelled/failed builds#550fedorov wants to merge 1 commit into
fedorov wants to merge 1 commit into
Conversation
The macOS and Windows workflows saved the DCMTK/ITK/zlib dependency cache with `if: always() && cache-hit != 'true'`. `always()` also fires when the job is cancelled, so a build killed mid-flight by the cancel-in-progress concurrency rule still ran the save step and archived a half-built tree (e.g. ITK with ITKConfig.cmake written but its libITKCommon-*.a not yet compiled). Once such a partial entry exists, every later run gets a cache hit -- which both restores the broken tree and skips this save step, so the bad entry is never overwritten -- and fails at link time with "libITKCommon-*.a ... missing and no known rule to make it". This poisoned the v1.5.6 release builds. Gate the save on the build step having actually succeeded (`!cancelled() && steps.build.outcome == 'success'`). This preserves the original intent of caching deps even when a *later* step (test, package) fails, but never snapshots a tree from a cancelled or failed build. `!cancelled()` keeps the step eligible after a failed test/package while `steps.build.outcome` does the real gating. Also harden the macOS "Validate restored cache" step to require a real ITK library, not just ITKConfig.cmake, so any partial cache that slips through is rejected and rebuilt cleanly instead of failing at link. Linux uses the combined actions/cache action, whose post-save does not run on cancellation, so it is unaffected and left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The v1.5.6 release builds failed on macOS (and were one step from the same fate on Windows) with:
Root cause was a poisoned dependency cache, not anything in the release. The macOS and Windows workflows save the DCMTK/ITK/zlib cache with:
always()also evaluates true when a job is cancelled. Thecancel-in-progressconcurrency rule cancels supersededmasterpushes mid-build, and the cancelled job still ran the save step — archiving a half-built tree. A half-compiled ITK hasITKConfig.cmake(written at configure time) but not itslibITKCommon-*.a(written later during compilation), so the archive is a valid cache of a broken tree.Verified from the step record of a cancelled run (
28198907054, job83532943609):Build dcmqi→cancelledat 20:38:15,Test/Package/Upload→skipped, butSave DCMTK, ITK and ZLIB cache→ success at 20:38:18 — the exactcreated_atof the resulting 57.7 MB partial cache (a complete one is ~785 MB).It was self-perpetuating: once a partial exists, later runs get a cache hit, which (a) restores the broken tree and (b) satisfies
cache-hit != 'true'so the save step is skipped — nothing ever overwrites the bad entry.Fix
Gate the save on the build step having actually succeeded:
test,package) fails —!cancelled()keeps the step eligible after such a failure.steps.build.outcome == 'success'is the real gate.Also hardens the macOS Validate restored cache step to require a real ITK library (
libITKCommon-*.a), not justITKConfig.cmake— exactly the gap that let the partial pass validation and then fail at link. Any partial that slips through is now rejected and rebuilt cleanly.Linux uses the combined
actions/cacheaction, whose post-save does not run on cancellation, so it never exhibited this and is left unchanged.Notes for reviewer / follow-up
windows-deps-*key should still be deleted manually (itsrestore-keysprefix fallback means it can be resurrected on a future Windows run); this PR stops new ones being created.🤖 Generated with Claude Code