ci: GC the Docker host per-run instead of on a nightly cron

[smunini]

Follow-up to #452. That PR stopped new volume orphans; this one reclaims the backlog and keeps it reclaimed.

Why not a scheduled janitor

An already-orphaned volume has no container left, so no amount of docker rm -fv will ever touch it. Something still has to sweep.

A nightly cron is the wrong shape for that job. The host fills during the working day, and a 05:30 UTC sweep cannot help a run that dies at 16:03 — by the time it fires the damage is twelve hours old and several runs have already gone red.

So the sweep is a per-run step instead. ci.yml fires on every push and PR, which means the host gets cleaned many times a day rather than once a night.

The change

A docker-host-gc composite action, used by:

Workflow	When	Why
ci.yml	after the reaps	every push/PR — this is the sweep that actually keeps the host under quota
ci-extended.yml	after the reaps	nightly; shouldn't leave the host dirtier than it found it
hts.yml	after the reaps	own nightly cron, can't rely on ci.yml's sweep
ui-tests-matrix.yml	before the matrix	pulls Playwright on top of ES/PG/Mongo — needs headroom up front, not afterwards

The three cleanup jobs had no checkout step, so each gets a cone-mode sparse checkout of .github/actions only — keeps them as cheap as they were.

Two deliberate departures from a naive prune

Volumes are matched on the 64-hex-char anonymous-volume name shape, not swept with docker volume prune -af.

This host also runs long-lived services. -a includes named volumes, so a prune racing a service redeploy — at the moment that service's named volume is briefly unreferenced — would delete its data, with no undo. The hex shape removes exactly the testcontainer leak and structurally cannot match a name a human chose. It is also stable across the Docker 23 change to what volume prune means without -a.

Volumes are freed first, and every step is non-fatal.

On a host already at its quota, image deletion itself fails — containerd needs a meta.db write to do it — while volume removal frees the same filesystem without one. The 2026-07 incident hit exactly this: image prunes erroring out while orphaned volumes held the bulk of usage. Free the bulk first and the rest starts working again.

Concurrency safety

The host is shared by every concurrent run, which is the constraint shaping each filter:

• Volumes — only unreferenced ones are touched, and a live container's volumes are by definition referenced. A volume still in use refuses removal; that's the intended outcome, not an error.
• Images — until=48h protects anything pulled recently, including an image a concurrent run has pulled but not yet started a container from. It also preserves the warm pull cache the nightly suites depend on, so this doesn't trade a disk problem for a slowness problem.
• Build cache / networks — age- and size-guarded for the same reason.

Every step is || true, so one failure can't abort the rest of the sweep or fail the cleanup job.

Verification

actionlint diffed before/after: no new findings.

Shell logic was exercised under GitHub's exact strict mode (bash --noprofile --norc -eo pipefail) including the daemon-unreachable path — every guard returns exit 0, so the GC can never fail a cleanup job.

Relationship to #451

This supersedes #451's docker-host-janitor.yml. #451 should not be closed outright — it also carries ci(ui-matrix): route the Elasticsearch self-traffic around the runner proxy, an unrelated real fix worth keeping. Trim it to that commit.

Note for the reviewer

The 64-hex-char assumption is worth one read-only confirmation on the host before merge:

docker volume ls --format '{{.Name}}' | grep -vE '^[0-9a-f]{64}$'

Anything that prints is a named volume the GC will never touch — it should list only real services, and nothing testcontainers created.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mauripunzueta]

Nice — the volumes-first ordering and the 64-hex-shape match are both exactly right, and they match what I measured on the host earlier today from a benchmark dispatch:

TYPE            TOTAL   ACTIVE   SIZE      RECLAIMABLE
Images           11       5       6.751GB    4.343GB (64%)
Containers       20      20     448.3MB         0B (0%)
Local Volumes   187      19      23.73GB   22.13GB (93%)

Volumes were 93% of the reclaimable space and containers were ~0. So "free the bulk first and the rest starts working again" is not just a theory about that incident — it's the shape of the host right now.

One gap: this wires the action into ci.yml, ci-extended.yml, hts.yml and ui-tests-matrix.yml, but not fhir-benchmark.yml — which is the single heaviest consumer on the host and the workflow that actually triggered the 2026-07-30 incident. Its Postgres leg imports the full Synthea corpus.

Two ways to close it, no strong preference:

1. Add the GC the Docker host step to fhir-benchmark.yml here, so one PR wires the action everywhere. Cleanest home for it, and no cross-PR dependency.
2. I add it in ci(benchmark): stop the FHIR benchmark from exhausting the shared Docker host #453 after this merges — but that makes ci(benchmark): stop the FHIR benchmark from exhausting the shared Docker host #453 depend on this branch, which I'd rather avoid.

I'd suggest (1). #453 stays independent: it fixes the benchmark's own leak (PG data was in the container's writable layer with no volume at all, so nothing could reclaim it) plus a start-of-job reaper for its orphans. Your GC is the complementary end-of-run sweep, and the two don't overlap — mine targets hfs-bench-* by name before allocating, yours sweeps the host generically after.

Worth noting for whoever runs it: neither PR reclaims the existing 22 GB retroactively, so the host still wants one manual docker system prune pass.