Skip to content

feat(komodo): prune the build cache on a schedule so Komodo's disk stays bounded - #126

Open
Teg79 wants to merge 4 commits into
mainfrom
feat/build-cache-prune
Open

Teg79 wants to merge 4 commits into
mainfrom
feat/build-cache-prune

Conversation

@Teg79

@Teg79 Teg79 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Komodo builds every image on its own LXC, and BuildKit's cache and the old images grow until the disk is full (#92). A small platform Stack now prunes them on a schedule.

  • platform/stacks/build-cache/compose.yaml (new): one docker:cli container on Komodo's LXC, with the Docker socket. Every every (24h by default, first run 10 minutes after start) it:
    1. drops build cache unused for keepFor (168h): docker builder prune --filter until=...;
    2. then trims the rest down to keepStorage (3GB): docker builder prune --keep-storage .... The two flags together prune only the old entries, so it takes two passes;
    3. drops images unused for keepFor: docker image prune --all --filter until=.... Images of running containers are never removed;
    4. logs one line with docker system df before and after (urbex build-cache: ...), which Loki collects.
  • internal/platform/config.go: komodo.buildCache.{keepStorage, keepFor, every} in urbex.platform.yaml, with defaults and validation (sizes like 3GB, durations of at least 1h).
  • internal/platform/catalog.go: the build-cache entry, kind platform, on the komodo LXC, GitOps-deployed, outbound only (ADR-0027).
  • internal/bootstrap/stacks.go: the Stack urbex-build-cache, on Komodo's server, its settings in config.env. A Stack without secrets now gets no secrets.sops.env and no compose wrapper: the sops wrapper isn't on Komodo's own LXC, and the first e2e run failed on it.
  • e2e/run.sh: the Stack runs, its config.env has the defaults, it has no secrets file; it stays after destroy staging.
  • CHANGELOG.md: under Unreleased.

Companion: urbforge/urbex#98.

Why

#92: Komodo's disk fills up with build cache, and the only fix today is pruning by hand.

Tests I ran

  • mise run test, mise run lint: pass.
  • mise run e2e: passed, 24 steps (bootstrap, a second bootstrap changes nothing, the Stack running, destroy staging keeps it).
  • The prune loop in docker-in-docker: with KEEP_STORAGE=1MB it freed 53 MB of build cache, logged before/after, and the container stayed running.
  • Not run on the lab: a bootstrap on the test node. Its effect on Komodo's disk shows after a day of builds.

Checklist before review

  • Linked issue, Conventional Commit title
  • Tests added (unit, e2e)
  • Service catalog entry, labels urbex.kind / urbex.service
  • CHANGELOG entry, docs in the companion PR
  • No secrets, tokens or private addresses
  • Run on a real Proxmox node

Fixes #92

🤖 Generated with Claude Code

https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA

Teg79 and others added 4 commits November 18, 2026 14:27
…ays bounded

A platform Stack on Komodo's LXC, urbex-build-cache, prunes every komodo.buildCache.every (24h) BuildKit's cache older than keepFor (168h) and then beyond keepStorage (3GB), and the images no container uses older than keepFor, and logs a line per run to Loki.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
…he compose wrapper

The wrapper (sops-tools) isn't on Komodo's own LXC: a platform Stack without secrets now has no secrets file and no wrapper, as the web frontends' Stacks on the builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182yxsfwRowwnxCfgrQwjnA
@Teg79
Teg79 marked this pull request as ready for review November 19, 2026 07:58
@Teg79

Teg79 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Run on the test node (2026-11-19, lab-cubotto/vault-test.sh, operator image from this branch): urbex bootstrap on the existing platform took 173 s, Ansible changed=0 everywhere, and added the Stack urbex-build-cache on Komodo's LXC. Its first pass, 10 minutes later, logged:

urbex build-cache: build cache: older than 168h: Total: 0B, beyond 3GB: Total: 0B; images: Total reclaimed space: 0B; before: Images 2.694GB ... Build Cache 939.2MB ; after: ... Build Cache 939.2MB

Nothing to free, as expected: 939 MB of cache, under 3GB and under a week old (the platform was rebuilt on 2026-11-17). The freeing itself was shown in docker-in-docker (see Tests). The Stack keeps running and will prune once the cache ages or grows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(komodo): prune the build cache on a schedule so Komodo's disk stays bounded

2 participants