Repository navigation
ci: stop saving the same rust cache many times per run - #608
Conversation
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
Each of the 40 bpf-tests jobs saved its own ~440 MB cache keyed by os and test binary, about 16 GB per run. That pushes the repository over its Actions cache quota and evicts entries within minutes, so later runs of the same PR rebuild from scratch. The shards build the same dependencies and rust-cache drops workspace crates, so key the cache by os only and let the c_tests shard save it. The install-rust action gains a save-cache input forwarded to rust-cache's save-if; other callers keep saving as before. Co-Authored-By: Claude <noreply@anthropic.com>
a3da312 to
4bb422e
Compare
Its jobs restore the entry without saving, so a cold run stores one union of their dependencies instead of one copy per job.
[Medium impact] The changes affect cache use and job ordering across a substantial part of CI. The PR appears safe to merge, but nine jobs will miss the intended shared cache until their extra key is removed.
|
Our Actions cache is capped at 10 GB, but a CI run that misses the cache saves over 16 GB. Each run evicts the previous run's caches, so the next one misses again. That's why
memtrack-benchmarkson #531 found no cache 15 minutes after saving it.Most of those 16 GB are copies of the same dependencies:
bpf-testsjobs each saved their own ~440 MB cache. Shards on the same OS build the same thing, so now one shard per OS saves it: 5 caches instead of 40.lint,tests,exec-harness-tests,basic-run-test,bpf-tests) also build mostly the same dependencies. A newprime-rust-cachejob builds them once into a shared cache, and those jobs only read from it.A run that misses the cache now saves about 6.4 GB instead of 16 GB.
The jobs reading the shared cache wait for
prime-rust-cache. On a cache hit it finishes in about a minute. The slowest job,memtrack-benchmarks, doesn't wait for it, so overall CI time shouldn't change.