feat(template): bound home-directory tool caches with an hourly cache GC - #954
Open
homelab-agent-bot[bot] wants to merge 1 commit into
Open
homelab-agent-bot[bot] wants to merge 1 commit into
homelab-agent-bot[bot] wants to merge 1 commit into
Conversation
The workspace home directory is a quota'd NFS share, and a full share fails every write in the pod. Go's build cache alone reached 201 GiB: Go keeps unused entries for five days, and builds from a fresh directory per run write entries no later build can hit. script-cache-gc.sh, run hourly by coder_script "cache_gc", evicts each cache by the signal its own tool writes, never by wiping it: - Go build and golangci-lint caches: least recently used first down to a cap (16 GiB, 1 GiB), never an entry used in the last three hours, with each entry's age checked again at deletion. This is the same unlocked mtime-based removal Go's own Trim() performs while builds run. - bun install cache: oldest first down to 4 GiB, counting only bytes no node_modules still hardlinks. Each entry is renamed out of bun's lookup path before deletion, so a concurrent install sees all of a package or none. - npm, node-gyp, Homebrew, pip and the bun transpiler cache: whole entries by age. Anything a live process uses is skipped. brew cleanup is avoided because it also removes kegs. - uv, mise and pre-commit: the tools' own prune commands, which take the tools' own locks. Removal costs a re-download or rebuild, never an installed tool. Runs are serialised by flock and run at idle priority. A missing tool is reported as skipped, and a failed step fails the run in the Coder UI. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
script-cache-gc.shand an hourlycoder_script(cache_gc) that keeps the tool caches under the workspace home directory bounded. The home directory is a quota'd NFS share, and when it fills every write in the pod fails, including the writes that hold agent sessions' state. That happened once, with Go's build cache at 201 GiB.How
Nothing wipes a cache. Each cache is evicted by the signal its own tool writes, through the tool's own prune command where one exists and is safe to run concurrently. A floor keeps anything a run in flight may still be reading.
Trim()deletes the same-a/-dentries by mtime with no lock while builds run. Go refreshes mtime on use at most hourly, so 3 hours covers that lag plus a long build. Each entry's age is checked again at the moment of deletionnode_modulesstill hardlinks. Prune abandoned extractions in.tmppackage.jsonexists, so each entry is first renamed into.tmpand deleted from there. A concurrent install sees the whole package or none of it_cacachefiles and_npxinstalls older than 14 days_npxinstall that any live process runs from is kept.incompletedownloads are left alone.brew cleanupis not used because it also removes old kegs and autoremoves formulaeuv cache prunemise cache prunewith a 7-day agepre-commit gcSizes are disk blocks, which is what the quota counts.
What removal costs
Every rule costs a re-download or a rebuild on the next use, never an installed tool.
node_modulestrees and venvs keep their files.pre-commit gc. Deleting a hook repo by hand leaves adb.dbrow pointing at nothing, and the next run fails.~/.cache/.bun/install/globalholds globally installed bun packages and is never touched.Operation
flock), and the script runs at the lowest CPU and IO priority.vscode_server_gc, where a guarded no-op was indistinguishable from success.CACHE_GC_GO_BUILD_CAP_MIB,CACHE_GC_GOLANGCI_LINT_CAP_MIBandCACHE_GC_BUN_CAP_MIB.--dry-runprints what would go.Why Go's cache grew
Go hashes the absolute package directory into every compile unless
-trimpathis set. A mutation-testing tool in another repository copies its module into a fresh temporary directory for every run, so every package recompiled under keys that are never hit again, at about 10 GB an hour. Go keeps unused entries for 5 days. This PR bounds the result whatever the cause. The source of the churn is a change in that tool and not part of this PR.Verification
node_modules, and the global install dir is untouched._npxdir in use by a live process survives.--dry-runagainst the live workspace, with these results:tofu validate, tflint and pre-commit pass.🤖 Generated with Claude Code