Skip to content

CI: don't cache half-built deps from cancelled/failed builds - #550

Open
fedorov wants to merge 1 commit into
masterfrom
fix/ci-cache-poisoning-on-cancel
Open

CI: don't cache half-built deps from cancelled/failed builds#550
fedorov wants to merge 1 commit into
masterfrom
fix/ci-cache-poisoning-on-cancel

Conversation

@fedorov

@fedorov fedorov commented Jun 25, 2026

Copy link
Copy Markdown
Member

Problem

The v1.5.6 release builds failed on macOS (and were one step from the same fate on Windows) with:

ninja: error: '.../ITK-build/lib/libITKCommon-5.4.a', needed by 'GenerateCLP/bin/GenerateCLP',
missing and no known rule to make it

Root cause was a poisoned dependency cache, not anything in the release. The macOS and Windows workflows save the DCMTK/ITK/zlib cache with:

if: always() && steps.<restore>.outputs.cache-hit != 'true'

always() also evaluates true when a job is cancelled. The cancel-in-progress concurrency rule cancels superseded master pushes mid-build, and the cancelled job still ran the save step — archiving a half-built tree. A half-compiled ITK has ITKConfig.cmake (written at configure time) but not its libITKCommon-*.a (written later during compilation), so the archive is a valid cache of a broken tree.

Verified from the step record of a cancelled run (28198907054, job 83532943609): Build dcmqicancelled at 20:38:15, Test/Package/Uploadskipped, but Save DCMTK, ITK and ZLIB cache → success at 20:38:18 — the exact created_at of the resulting 57.7 MB partial cache (a complete one is ~785 MB).

It was self-perpetuating: once a partial exists, later runs get a cache hit, which (a) restores the broken tree and (b) satisfies cache-hit != 'true' so the save step is skipped — nothing ever overwrites the bad entry.

Fix

Gate the save on the build step having actually succeeded:

if: ${{ !cancelled() && steps.build.outcome == 'success' && steps.<restore>.outputs.cache-hit != 'true' }}
  • Preserves the original intent (the old comment): still caches deps when a later step (test, package) fails — !cancelled() keeps the step eligible after such a failure.
  • Never snapshots a tree from a cancelled or failed build — steps.build.outcome == 'success' is the real gate.

Also hardens the macOS Validate restored cache step to require a real ITK library (libITKCommon-*.a), not just ITKConfig.cmake — exactly the gap that let the partial pass validation and then fail at link. Any partial that slips through is now rejected and rebuilt cleanly.

Linux uses the combined actions/cache action, whose post-save does not run on cancellation, so it never exhibited this and is left unchanged.

Notes for reviewer / follow-up

  • The poisoned macOS caches were already deleted manually and v1.5.6 was re-run successfully — this PR prevents recurrence.
  • A lurking 69.7 MB partial cache under the windows-deps-* key should still be deleted manually (its restore-keys prefix fallback means it can be resurrected on a future Windows run); this PR stops new ones being created.

🤖 Generated with Claude Code

The macOS and Windows workflows saved the DCMTK/ITK/zlib dependency cache
with `if: always() && cache-hit != 'true'`. `always()` also fires when the
job is cancelled, so a build killed mid-flight by the cancel-in-progress
concurrency rule still ran the save step and archived a half-built tree
(e.g. ITK with ITKConfig.cmake written but its libITKCommon-*.a not yet
compiled). Once such a partial entry exists, every later run gets a cache
hit -- which both restores the broken tree and skips this save step, so the
bad entry is never overwritten -- and fails at link time with
"libITKCommon-*.a ... missing and no known rule to make it". This poisoned
the v1.5.6 release builds.

Gate the save on the build step having actually succeeded
(`!cancelled() && steps.build.outcome == 'success'`). This preserves the
original intent of caching deps even when a *later* step (test, package)
fails, but never snapshots a tree from a cancelled or failed build.
`!cancelled()` keeps the step eligible after a failed test/package while
`steps.build.outcome` does the real gating.

Also harden the macOS "Validate restored cache" step to require a real ITK
library, not just ITKConfig.cmake, so any partial cache that slips through
is rejected and rebuilt cleanly instead of failing at link.

Linux uses the combined actions/cache action, whose post-save does not run
on cancellation, so it is unaffected and left unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant