Skip to content

feat(template): bound home-directory tool caches with an hourly cache GC - #954

Open
homelab-agent-bot[bot] wants to merge 1 commit into
mainfrom
cache-gc
Open

homelab-agent-bot[bot] wants to merge 1 commit into
mainfrom
cache-gc

Conversation

@homelab-agent-bot

Copy link
Copy Markdown
Contributor

What

Adds script-cache-gc.sh and an hourly coder_script (cache_gc) that keeps the tool caches under the workspace home directory bounded. The home directory is a quota'd NFS share, and when it fills every write in the pod fails, including the writes that hold agent sessions' state. That happened once, with Go's build cache at 201 GiB.

How

Nothing wipes a cache. Each cache is evicted by the signal its own tool writes, through the tool's own prune command where one exists and is safe to run concurrently. A floor keeps anything a run in flight may still be reading.

Cache Rule Why it is safe while the tool runs
Go build cache Evict least recently used down to 16 GiB, never anything used in the last 3 hours Go's own Trim() deletes the same -a/-d entries by mtime with no lock while builds run. Go refreshes mtime on use at most hourly, so 3 hours covers that lag plus a long build. Each entry's age is checked again at the moment of deletion
golangci-lint cache Same, capped at 1 GiB golangci-lint uses a copy of Go's cache code and layout
bun install cache Evict oldest first down to 4 GiB, counting only bytes no node_modules still hardlinks. Prune abandoned extractions in .tmp bun counts an entry present if its package.json exists, so each entry is first renamed into .tmp and deleted from there. A concurrent install sees the whole package or none of it
bun transpiler cache Files older than 14 days Atomic writes, content hash verified on every read
npm _cacache files and _npx installs older than 14 days cacache is self-healing. An _npx install that any live process runs from is kept
node-gyp Header dirs for Node versions no longer installed, older than 7 days Skipped while any node-gyp process runs
Homebrew Downloads written more than 30 days ago .incomplete downloads are left alone. brew cleanup is not used because it also removes old kegs and autoremoves formulae
pip HTTP and wheel cache files older than 30 days pip writes to a temp file and renames
uv uv cache prune Takes uv's cache lock
mise mise cache prune with a 7-day age Installs never reference the cache
pre-commit pre-commit gc Takes pre-commit's store lock

Sizes are disk blocks, which is what the quota counts.

What removal costs

Every rule costs a re-download or a rebuild on the next use, never an installed tool.

  • bun and uv hardlink packages out of their caches, so node_modules trees and venvs keep their files.
  • mise installs and Homebrew kegs live outside their caches.
  • pre-commit's cache is the installed hook environments, so it is left to pre-commit gc. Deleting a hook repo by hand leaves a db.db row pointing at nothing, and the next run fails.
  • ~/.cache/.bun/install/global holds globally installed bun packages and is never touched.

Operation

  • An overlapping run skips (flock), and the script runs at the lowest CPU and IO priority.
  • A tool that is not installed is logged as skipped by name.
  • A step that fails makes the run fail in the Coder UI after the other steps have run. This follows the reasoning recorded on vscode_server_gc, where a guarded no-op was indistinguishable from success.
  • Caps can be overridden with CACHE_GC_GO_BUILD_CAP_MIB, CACHE_GC_GOLANGCI_LINT_CAP_MIB and CACHE_GC_BUN_CAP_MIB.
  • --dry-run prints what would go.

Why Go's cache grew

Go hashes the absolute package directory into every compile unless -trimpath is set. A mutation-testing tool in another repository copies its module into a fresh temporary directory for every run, so every package recompiled under keys that are never hit again, at about 10 GB an hour. Go keeps unused entries for 5 days. This PR bounds the result whatever the cause. The source of the churn is a change in that tool and not part of this PR.

Verification

  • A fixture home covering every cache, with all paths redirected. Checked there:
    • Recent Go entries and non-entry files survive. A cache without Go's marker files is refused.
    • A hardlinked bun package survives in node_modules, and the global install dir is untouched.
    • An _npx dir in use by a live process survives.
    • A Homebrew blob with an old mtime but a recent ctime survives.
    • An entry touched between the walk and the deletion survives.
  • --dry-run against the live workspace, with these results:
    • bun: 11.1 GiB of unshared data down to its 4 GiB cap, plus 1.1 GiB of abandoned extractions.
    • node-gyp: 544 MiB. pip: 485 MiB. golangci-lint: 944 MiB.
  • shellcheck, tofu validate, tflint and pre-commit pass.

🤖 Generated with Claude Code

The workspace home directory is a quota'd NFS share, and a full share fails
every write in the pod. Go's build cache alone reached 201 GiB: Go keeps
unused entries for five days, and builds from a fresh directory per run write
entries no later build can hit.

script-cache-gc.sh, run hourly by coder_script "cache_gc", evicts each cache
by the signal its own tool writes, never by wiping it:

- Go build and golangci-lint caches: least recently used first down to a cap
  (16 GiB, 1 GiB), never an entry used in the last three hours, with each
  entry's age checked again at deletion. This is the same unlocked
  mtime-based removal Go's own Trim() performs while builds run.
- bun install cache: oldest first down to 4 GiB, counting only bytes no
  node_modules still hardlinks. Each entry is renamed out of bun's lookup path
  before deletion, so a concurrent install sees all of a package or none.
- npm, node-gyp, Homebrew, pip and the bun transpiler cache: whole entries by
  age. Anything a live process uses is skipped. brew cleanup is avoided
  because it also removes kegs.
- uv, mise and pre-commit: the tools' own prune commands, which take the
  tools' own locks.

Removal costs a re-download or rebuild, never an installed tool. Runs are
serialised by flock and run at idle priority. A missing tool is reported as
skipped, and a failed step fails the run in the Coder UI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant