Skip to content

ci: drop the unhonoured pre-prune waiver (CACHE-034) - #533

Merged
zackees merged 2 commits into
mainfrom
fix/cache-034-drop-unhonoured-pre-prune
Oct 5, 2026
Merged

zackees merged 2 commits into
mainfrom
fix/cache-034-drop-unhonoured-pre-prune

Conversation

@zackees

@zackees zackees commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

What

ci.toml declared pre-prune = true on both writer flows (main and nightly).
No workflow in this repository has ever run ci-lint cache preprune -- verified by
grep -rn "preprune" .github/ ci/, which returns nothing.

ci_lint's CACHE-004 budget arithmetic honours the declaration on its own: with
every writer flow pre-pruning, the lockfile-change peak term is dropped from the
worst-case footprint. So this repository's modelled footprint was understated by
exactly that peak term -- 3.57GB -- for free.

This PR takes fix (B): it removes the declaration so the arithmetic counts the
peak honestly. It does not touch [cache].budget, any family size, or any live
cache entry.

Why not (A)

Adding ci-lint cache preprune --lockfile-changed before the saves would be the
better fix, but it is not reachable here without a structural change to the
workflow this PR should not make:

  • The saves live inside zackees/setup-soldr steps in the rust and linux jobs
    of .github/workflows/ci.yml (lines 145, 237). A pre-prune step would have to be
    ordered ahead of those, inside those jobs, each holding actions: write.
    This repository's ci.yml sets permissions: contents: read workflow-wide, and
    no build job holds actions: write today -- widening it on rust and linux is
    a security-relevant change that belongs in its own reviewed PR.
  • ci-pre.yml is the natural home (it already carries cache duties), but
    nothing in ci.yml needs: it. It is called as a job and deliberately
    un-gated, so its deletions would not be ordered ahead of the setup-soldr
    saves in the same run. A pre-prune that is not ordered before the save does not
    remove the peak.

CACHE-004 arithmetic, before and after

Verified with
uvx --from "git+https://github.com/zackees/ci.yml@92106df315bd28de24ef1808c893c91a5859122d" ci-lint precheck --repo . --local

before after
steady total 4,236,247,040 B 4,236,247,040 B
+ lockfile-change peak 3,565,158,400 B (waived) 3,565,158,400 B (counted)
+ [cache.pr].budget 0 B 0 B
worst case 4,236,247,040 B 7,801,405,440 B
budget 9,663,676,416 B 9,663,676,416 B
CACHE-004 pass pass

7.27GiB against a 9GB budget: the honest number still fits, with ~1.86GB of
headroom. Nothing was invented to achieve that -- the family max values and
per cardinalities are unchanged, and the pre-existing 490 violations / 13
needs_review are unchanged too (before and after).

CACHE-029 / the 99.2% cap

The prompt noted this repository's live cache sits at 9.92GiB of GitHub's 10GiB
cap, with a compile family holding 9.18GiB across 16 entries. Two notes:

  • This change does not affect that arithmetic. CACHE-029 is a separate
    finding, and it is not raised here: compile already declares evict = "lru"
    in ci.toml (line 110), so the static rule is satisfied. Before/after the
    modelled compile term is identical at 3,355,443,200 B.
  • The live-vs-declared gap is real and worth a follow-up: compile's declared
    max = "800MB" x 4 platforms models 3.36GB, while the live listing holds
    9.18GiB. That is entries outside the declared cardinality (and/or entries
    larger than max), which no ci.toml edit in this PR addresses. LRU eviction
    bounds future growth but does not reclaim the existing 9.18GiB. Tracked
    separately rather than folded in here.

Unresolved

  • Restoring pre-prune = true honestly needs a reviewed PR that grants
    actions: write to the rust and linux jobs and orders a pre-prune step
    ahead of their setup-soldr steps. That is the real fix; this PR only stops
    the waiver from being free.
  • No Local-Gate: trailer is attached. The precheck was run and is reported
    above; I did not fabricate a gate attestation. If this repository's policy
    requires one on PR branches, the maintainer should run the local gate.

Second commit: repoint the linter pin (fixes #534)

The first iteration of this PR sat red because ci.toml's linter pin named
zackees/ci.yml@fe965fc7... — a commit that does not exist — so ci-lint
could not be fetched and no Local-Gate: trailer could ever be produced
(#534). Commit 2 repoints linter at
0fcff997fefecf152ba075875029582d1ec05ad1, which resolves and carries the
CACHE-034 policy documentation (zackees/ci.yml#353).

Fixes #534

`pre-prune = true` was declared on both writer flows but no workflow here
ever runs `ci-lint cache preprune`. CACHE-004's arithmetic waives the
lockfile-change peak on that declaration alone, so the modelled footprint
was understated by 3.57GB for free.

Removing the declaration makes the number honest. Worst case goes
4.24GB -> 7.80GB, still inside the 9GB budget, so CACHE-004 keeps passing
and the violation count is unchanged at 490.
@zackees

zackees commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

CACHE-034 is implemented and merged — zackees/ci.yml#352 (docs #353). It fires on this repository's main today:

flow(s) 'main', 'nightly' declare pre-prune = true, which waives the lockfile-change peak from CACHE-004's worst case, but no workflow, composite action, or ci/ script invokes ci-lint cache preprune

So this PR is the remediation for an enforced rule, not a review suggestion — nothing mechanically prevents the declaration from coming back otherwise.

Verified independently

Your CACHE-004 numbers check out against main: compile models 800 MB × 4 = 3,355,443,200 B, and the honest worst case of 7,801,405,440 B still fits the 9 GB budget. bosn is the clean case of this batch — the problem here was purely the fiction, not the shape. Compare soldr (18.80 GB steady, already failing) and running-process (11.04 GB steady, already failing), where removing the waiver only makes a pre-existing failure larger.

Your reasoning for (B) over (A) is right: ci.yml is permissions: contents: read workflow-wide with no build job holding actions: write, and ci-pre.yml is not needs:-ordered ahead of the saves, so a pre-prune there would not be ordered before them.

The live-vs-declared gap you flagged

Agreed it deserves its own issue rather than folding in here: declared compile models 3.36 GB while the live listing holds 9.18 GiB across 16 entries. evict = "lru" bounds that going forward but does not reclaim what is already there, and bosn sits at 99.2% of GitHub's 10 GiB cap. Worth tracking on its own.

@zackees

zackees commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

One correction to something implied above: I framed option (A) — adding the pre-prune step — as "the smaller diff". For bosn specifically it is not currently expressible at all, and that is structural rather than a matter of effort.

Two independent blockers, both verified against your main:

  1. No actions: write. .github/workflows/ci.yml is permissions:\n contents: read workflow-wide; no job in the file holds it. ci-lint cache preprune deletes cache entries, so it needs that permission.
  2. No ordering. ci-pre.yml cannot host the step because nothing in ci.yml needs: it — zero references — so a prune there would not be ordered ahead of the saves in the same run.

So removing pre-prune = true is not a shortcut here; it is the only honest fix available without a reviewed permissions/order change first. Filed the shared prerequisite as zackees/ci.yml#354 rather than repeating it in each PR — it covers bosn, soldr and running-process, which all hit the same wall for the same two reasons.

This does not change the recommendation: this PR should still land. bosn's honest worst case (7.27 GiB) fits the 9 GB budget, so the waiver is a pure fiction with no underlying problem — unlike soldr and running-process, whose steady state is already over budget and which need a cardinality decision this change cannot make.

@zackees

zackees commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

Why this PR cannot go green: bosn's linter pin does not exist. Filed as zackees/bosn#534.

ci.toml line 14 pins zackees/ci.yml@fe965fc7e2c5203570072e1d0fd19d0935d98e39, but that commit is not in the repository — gh api repos/zackees/ci.yml/commits/<sha> returns 422, and git fetch returns upload-pack: not our ref. The pin is dead, not stale.

The consequence for this PR is mechanical:

  • Verify local gate fails GATE-003 (no Local-Gate: trailer), and Darwin wheel, Native wheel (×2), and Linux cascade from it — 6 failures, all within 3-8 seconds of each other.

  • The trailer cannot be produced, because ci-lint local-gate run resolves the declared pin before executing any lane, and that resolution fails first:

    $ uvx --from "git+https://github.com/zackees/ci.yml@fe965fc7..." ci-lint local-gate run
    failed to fetch commit fe965fc7e...
    local gate: not run after a heavy-lane failure
    

Running the gate against main instead gets further (guards and py-static pass) but the heavy tests lane then fails on a missing Bosn receipt, so there is still no honest attestation to push.

This is pre-existing and not caused by this PR — the pin is unchanged on main at 590e8ba and this branch does not touch it.

The underlying change here remains sound and independently verified: CACHE-004 passes at 7,801,405,440 B against the 9 GB budget before and after, and the compile term is identical either way (3,355,443,200 B), so removing the waiver is a pure honesty correction with no arithmetic consequence. It is held as a draft until the pin is repointed — at which point it should go green without further changes.

Repointing linter at a commit carrying CACHE-034 (merged as b7f512c) would fix the pin and make this repository's declared checks current in one move.

@zackees

zackees commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

The dead pin is fixed on this branch — 97228e6 repoints linter at 0fcff997 (carries CACHE-034). precheck now executes against a fetchable linter: 490 violations / 13 needs_review, and CACHE-034 is silent here because this branch already removed the waiver.

That was the blocker I described above, and it is now genuinely unblocked — local-gate run gets past pin resolution and runs lanes (guards and py-static pass).

What still blocks the attestation is unrelated to this change: the host. The bosn daemon refuses to start for /tmp/bosn-bosn3:

Protocol("registry missing but this machine already has Bosn-labelled objects")

with 79 orphaned bosn-* docker volumes listed and no registry database — so bosn declines to start rather than guess which volumes it owns. That is the standing host-hygiene gap raised in zackees/ci.yml#322; reclaiming 79 volumes is a destructive operation on someone's machine and is the owner's call, not something to do from inside a cache-policy PR.

I can stamp this branch as soon as the host is usable. The two changes on it are independent and both sound on their own:

  1. 97228e6 — repoint a pin that has never existed, which was making the gate unrunnable and GATE-003 fail on every PR.
  2. d67eed14 — remove the unhonoured pre-prune waiver (CACHE-004 honesty). Verified finding-neutral: CACHE-004 passes at 7,801,405,440 B against the 9 GB budget before and after, and the compile term is identical either way.

Holding as a draft for the host reason alone. If you would rather land (1) on its own — since it is what unblocks this repository's gate and its precheck, independent of the cache question — say so and I will split it out.

`fe965fc7e2c5203570072e1d0fd19d0935d98e39` is not in zackees/ci.yml -- the
GitHub API returns 422 and the object is absent from the store, so there is
no branch or tag to recover it from.

The pin was introduced in this repository's first ci.toml (#512) and was
never valid. SEC-004 did not catch it because the SHA is well-formed; a
syntactically valid reference to a nonexistent commit passes every existing
check. It surfaced only when something tried to RUN the gate -- local-gate
resolves the pin before executing any lane, so the resolution failed first
and no Local-Gate: trailer could be produced (GATE-003 on every PR).

Repointed at 0fcff99 (carries CACHE-034, merged). precheck now executes
against a fetchable linter: 490 violations / 13 needs_review.

See #534.

Local-Gate: v1 tree=705845b803adc59f10578868dfe72f0da94d2f75 secs=612 lanes=py-static:reused@67733cf35e83,guards:reused@6011cde8e2b0,rust:run,tests:run
Ci-Attestation: {"at":1791223142,"gate":"python/all/static","host":"linux-x86_64","key":"67733cf35e831723fd88a6ccf527858f2eca477748fe12f7fd3b8f4b5f495c21","lane":"py-static","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":null,"stamp":"79fc1b824301e53e76333fd5cf8d2be9","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"reused"}
Ci-Attestation: {"at":1791223142,"gate":"general/all/ci-policy","host":"linux-x86_64","key":"6011cde8e2b00205ef30534c9b853cbf8abb4424a0cea35162dcfd893a5aa6cc","lane":"guards","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":null,"stamp":"d5484777a28aea3b9707c21249f37f9e","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"reused"}
Ci-Attestation: {"at":1791223142,"gate":"rust/all/fmt","host":"linux-x86_64","key":"a01ef65e97480db9abcc7ce1e3ebd864a1a774cb14e7d67e14db49ae96242591","lane":"rust","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":298,"stamp":"311e622645ec6018a777a44000eaad49","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"run"}
Ci-Attestation: {"at":1791223142,"gate":"rust/all/boundary","host":"linux-x86_64","key":"a01ef65e97480db9abcc7ce1e3ebd864a1a774cb14e7d67e14db49ae96242591","lane":"rust","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":298,"stamp":"271d44451050f951110b4e85cc07d08d","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"run"}
Ci-Attestation: {"at":1791223142,"gate":"rust/x86_64-unknown-linux-gnu/clippy","host":"linux-x86_64","key":"a01ef65e97480db9abcc7ce1e3ebd864a1a774cb14e7d67e14db49ae96242591","lane":"rust","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":298,"stamp":"4cf65a24a4f240269489670738e35483","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"run"}
Ci-Attestation: {"at":1791223142,"gate":"rust/x86_64-unknown-linux-gnu/test","host":"linux-x86_64","key":"a01ef65e97480db9abcc7ce1e3ebd864a1a774cb14e7d67e14db49ae96242591","lane":"rust","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":298,"stamp":"d755f8a7fe4f219bb5470cd50e54c1d4","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"run"}
Ci-Attestation: {"at":1791223142,"gate":"python/x86_64-unknown-linux-gnu/test","host":"linux-x86_64","key":"d8be7b5119ad61ca86ae6d027f371f436fea741d235b656adec5e3da2490289f","lane":"tests","parents":["d67eed1413a5095445b6d90cd6e9de09aec6feea"],"secs":314,"stamp":"e8ff0e96fc281725de0adb9e0ca3458d","tree":"705845b803adc59f10578868dfe72f0da94d2f75","v":1,"via":"run"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci.toml's linter pin points at a commit that does not exist in zackees/ci.yml

1 participant