Improvements to GitHub Actions package caching#6803
Improvements to GitHub Actions package caching#6803trexfeathers wants to merge 20 commits intoSciTools:mainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6803 +/- ##
=======================================
Coverage 90.39% 90.39%
=======================================
Files 91 91
Lines 24761 24761
Branches 4639 4639
=======================================
Hits 22382 22382
Misses 1608 1608
Partials 771 771 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| cache_period: ${{ env.CACHE_PERIOD }} | ||
| env_name: ${{ env.ENV_NAME }} | ||
| install_packages: "cartopy nox pip" | ||
| - name: "install cartopy + nox + pip" |
There was a problem hiding this comment.
This could stand some improvement. The wheels job always finishes first and populates the cache. When the tests job - this one - finishes, it does not overwrite the cache. For the top level Nox environment, the Python version is not controlled, so the dependency stack will often be for a different Python version than what is in the cache, leading to lots of new downloads.
Solutions
- Make the Nox environment use
matrix.python-version, so that the Cartopy dependency stack comes from the cache. - Give the
testsworkflow 'control' of the cache, havingwheelsetc only use the cache but not populate it.
There was a problem hiding this comment.
Should benchmarking also be using the Cartopy cache?
EDIT: minimal benefit given the limited plotting that benchmarking involves. At the very least this is out of scope in this PR.
There was a problem hiding this comment.
(note this revealed another problem which I eventually figured out - the diff from all the debugging is not that much: https://github.com/SciTools/iris/pull/6803/files/20d3b372a8213b0b4c8c7bf3dc65b93891aa9281..957dc9f92a23c41b06d240319da7bec90f936827)
The problem
Our existing use of caching Conda content in GitHub Actions workflows clashes with Conda's own caching mechanism. Packages are stored in the Conda package-cache, and the packages are then included in individual environments via hard-links. GHA caching does not parse these hard-links, but we cache both the Conda package-cache AND the environment directories, causing two problems:
My proposed solution
tests,wheelsandbenchmarks.Pros and cons
Pros
Cons
This is largely mitigated: