Skip to content

ci(cache): rust-cache saves on pull requests — 13.4 GB of unreachable PR-ref caches over the 10 GB limit; add save-if and guard it #1216

Description

@zackees

Failure

running-process's Actions cache is over GitHub's 10 GB repository limit, and nearly all of it is unreachable: 32 entries, 13.36 GB, all on refs/pull/1215/merge. A cache saved on a PR's merge ref can only be restored by later runs of that same PR, yet it counts against the limit and evicts main's entries, so every other ref starts cold.

Every entry is Swatinem/rust-cache (v0-rust-*). The largest are 0.9–1.85 GB per job: v0-rust-windows-x86-* 1.85 GB, v0-rust-windows-arm-* 1.83 GB, v0-rust-output-shutd* 1.36 GB (6 entries), v0-rust-linux-x86-mu* 1.22 GB, and so on.

Mechanism

Swatinem/rust-cache@v2 saves on every event unless save-if is set. None of the ~12 uses in .github/workflows/ set it: _build.yml:93, ci-all-features.yml:105, ci-linux.yml:92, ci-output-shutdown.yml:56, ci-preflight-rust.yml:75,101,131, ci-preflight.yml:101, coverage.yml:69, independent-spawn.yml:52,117, linux-x86-build.yml:58, auto-release.yml:542, and the other per-platform build workflows. Many jobs also use a distinct key:, so one PR writes one ~1–2 GB entry per job.

Proposal

Fix

  • Add save-if: ${{ github.ref == 'refs/heads/main' }} (or github.event_name != 'pull_request') to every Swatinem/rust-cache step. PR runs keep restoring from main's entries.
  • Where jobs build the same graph on the same OS, share a key or shared-key so main keeps one entry per OS × graph instead of one per job.

Catch it so it does not recur

  • A lint-lane check (a small script under ci/) that fails when any Swatinem/rust-cache (or actions/cache save) step can save on pull_request without an explicit, commented exemption. RED on today's workflows.
  • Optionally, a scheduled check that reports PR-ref cache bytes and fails above a small allowance.

Acceptance criteria

  • After a PR run, no new refs/pull/* v0-rust-* entries exist; main holds one entry per OS × graph; the repository total stays well under 10 GB.
  • PR builds still restore warm from main (record hit rate and wall time before/after).
  • The check is RED before the change and GREEN after.

Related

Activity

  1. zackees commented on Sep 27, 2026

    @zackees
    OwnerAuthor

    Reopening: the prior main-only save gate prevented PR-ref cache growth but did not cap main-ref writes. Scheduled All-Features run 36319601595 (2026-09-27) saved three distinct v0-rust-*-all-features entries on refs/heads/main: macOS ARM ID 8176285621 (649,556,636 B), Windows x86 ID 8176434675 (850,009,549 B), macOS x86 ID 8176621221 (645,299,171 B), total 2,144,865,356 B. The Windows job failed one flaky TUI ANSI test, but cache-on-failure:true still saved its 850 MB archive. Afterward the usage endpoint and paginated cache list both settle at 11,027,243,685 B (list count differs from usage count; byte totals match), above the 10 GiB repository cap. Current ci-all-features.yml already has the #1216 main-only save-if gate; schedules have refs/heads/main, so this is allowed under the existing ownership rule. The gap is no live total-budget guard and no retention for this optional dedicated family. Follow-up will make that lane restore-only, retire only the exact three all-features main families through a reviewed workflow, and add RED/GREEN budget/ownership tests. No cache has been manually deleted.

  2. zackees commented on Sep 27, 2026

    @zackees
    OwnerAuthor

    Closure evidence (2026-09-27)

    The cache overage is resolved with reviewed producer/retention policy and main-run budget enforcement; no cache was manually deleted.

    • RED: the reopened audit found 11,027,243,685 B / 33 listed rows after the nightly all-features lane. Run 36332333469 later failed the conservative 9.5e9-byte guard at 12,811,688,983 B after the full producer cycle. These were real main-ref writes, not transient list lag.
    • The reviewed changes landed in PR #1226 and PR #1227. All-features and release target archives are restore-only; Windows ARM and coverage retain registry-only keys. Main-only, exact-family retirement runs before producers and after the writer barrier. Unrelated platform/profile caches remain protected.
    • On successful main budget run 36337888631, the reviewed early retirement removed only the two obsolete target-bearing rows:
      • ID 8181253810, 2,105,526,302 B, Windows ARM shared preflight target cache.
      • ID 8180667758, 944,115,359 B, coverage target cache.
        Total removed: 3,049,641,661 B. Their registry-only replacements remain: ID 8182796929 (86,534,903 B) and ID 8182505889 (99,540,621 B). The final guard passed with endpoint and list equal at 8,596,607,472 B (limit 9,500,000,000).
    • Restore validation: Coverage dispatch 36339999294 succeeded and restored the exact coverage registry key (99,540,621 B; full key match), with no target archive save. Windows dispatch 36340876690 succeeded on both x86 and ARM; ARM restored the exact 86,534,903 B registry-only key (full match), then reported "Cache up-to-date." The x86 leg also succeeded. No cache growth occurred in that dispatch.
    • Final endpoint/list total after both dispatches is 8,596,608,623 B (a 1,151 B setup-uv cache addition during Coverage), still below the 9.5e9 guard by 903,391,377 B and below GitHub's 10 GiB cap. No target-bearing all-features, release, Windows ARM, or coverage rows were recreated.

    Known validation exception: the optional all-features Windows run failed its pre-existing daemon_tui_repaint_test::raw_ansi_bytes_flow_through_pty_to_ring_buffer test (2,554/2,555 passed after retries); this was documented and did not affect the regular platform matrix or cache policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions