Repository navigation
Fix silent main build-cache save failures and cancel discarded queue runs - #735
Conversation
…runs Every main-branch "Save pool build" since 2026-10-16 failed because the runner ran out of disk while archiving .lake/build (52 MB free). The failure is only a warning, so runs stayed green while every PR, queue and main build kept restoring the 2026-10-16 cache and rebuilding weeks of changes (5,351 modules, ~3.5h for a one-project merge group). - Free ~25 GB of preinstalled toolchains before the build jobs. - Fail the run when the main build cache did not actually save. - Cancel merge-group runs whose queue ref GitHub already discarded, which happens after every out-of-queue push to main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proof profile (new / modified Lean files)No added or modified Lean files under Advisory only — never blocks merge. |
|
Advisory cross-file review against Lean Pool's supervisory rubric. This is not by itself a significance, source-verification, or merge verdict. [Medium impact] The PR appears safe to merge; both earlier findings are addressed and no new blocking issue was found. SummaryThe PR frees runner disk space, checks that main build caches were saved, and cancels discarded merge-queue runs.
Reviews (3) · Last reviewed commit: "Leave build-cache pruning to the schedul..." · Reviewed by Greptile |
The inline cancel loop stopped at the first missing ref: GitHub runs steps with bash -e, so the failing lookup exited the loop (Greptile P1). Move the logic into lean_pool.merge_queue_cleanup, which treats only a definite 404 as missing and keeps scanning, with tests for a missing ref followed by more runs. Also prune the superseded build cache right after a verified main save, so two 6+ GB build copies never sit over the 10 GB repository cache limit and push out the dependency or docs caches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#630 deliberately moved cache pruning off the build path into the hourly cache-maintenance workflow, and a contract test enforces that. Keep the save verification and drop the inline prune. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vilin97
left a comment
There was a problem hiding this comment.
{
"summary": "Post-merge review of published commit abb701f across all three changed files. The earlier shell-404 and inline-pruning findings are addressed. A remaining per-run timeout aborts the cleanup sweep; an offline mocked execution reproduced it. The focused Python-only repair is prepared in draft PR736, where all nine targeted tests and Ruff checks pass.",
"assessment": {
"scope": "CI cache health and discarded-run cleanup",
"risk": "medium",
"repair_status": "Draft PR736 isolates lookup and cancellation failures and reports only accepted cancellation requests.",
"integration": "Current-main qualification, all normal checks and the standing workflow authority conditions remain independent."
},
"verdict": "request_changes",
"findings": [
{
"priority": "P2",
"file": "python/lean_pool/merge_queue_cleanup.py",
"line": 39,
"rule": "isolate-best-effort-cleanup-failures",
"comment": "A ref lookup TimeoutExpired escapes branch_exists and aborts discarded_runs before later definite-404 runs are checked; even IDs selected before that timeout never reach the cancellation loop. A timeout in the cancellation loop at line84 likewise stops later requests. continue-on-error preserves the workflow but cannot resume the sweep. Catch lookup timeouts as unknown/keep and isolate cancellation errors per run, with timeout-followed-by-another-run regression coverage. Draft PR736 contains this repair."
}
],
"follow_up": "#736"
}
Problem
CI runs take hours even with a "cache hit", and the merge queue has duplicate runs.
LeanPoolBuildcache exists, saved on 2026-10-16. Since then every main push run'sSave pool buildstep failed with the runner out of disk (Free space left: 52 MB,tarexit 2). actions/cache treats that as a warning, so the runs stayed green. Every PR, merge-group and main run therefore restores the Oct 16 cache and rebuilds everything merged since. Example: merge group for Reduce Erdos97ConvexOctagon peak compiler memory #717 (one-project change) rebuilt 5,351 modules / 11.5 CPU-hours in 3h27m — mostly ConwaySoifer (190 min), Superdiffusion (174 min), HopfProblem (60 min), BrillNoetherGraphs (48 min).Changes
Free runner disk spacestep (removes dotnet / Android / GHC / CodeQL / Swift, ~25 GB) in the build, shard, finalize and rebase jobs.Verify pool build cachestep on main: fails the run if the cache key is not actually present after saving, so this cannot go silent again.Cancel discarded merge-group runsstep inplanfor merge_group events: cancels in-progress queue runs whosegh-readonly-queue/...ref returns 404 (continue-on-error, never blocks the run).plangetsactions: writefor this.Validation
actionlintclean;test_ci_cache_contract.py,test_rebase_fastpath.py,test_ci_artifacts.pypass (122 tests).🤖 Generated with Claude Code