ci(ui-matrix): capture evidence when HFS cannot reach Elasticsearch

[angela-helios]

Closes #449

Slimmed after #452/#454 — Steve's PRs root-caused and fixed what my dispatches had been circling: every reap orphaned the testcontainers' anonymous volumes (docker rm -f without -v), and the choked host produced the nightly ES signature (readiness curl passes, then ES stalls on fsync and its mapped port drops SYNs — 30s connect timeouts from HFS's first request while the process still looks alive). My earlier janitor/cron approach here is superseded by the per-run docker-host-gc action and has been dropped from this branch.

Why this PR is still worth merging

• Evidence on failure — when "Wait for HFS" fails on an ES leg, dump the ES container's state, OOM flag, log tail, and a live reachability probe. The ci: nightly ui-tests-matrix — Elasticsearch leg failing (Postgres and MongoDB green) #449 incident was diagnosed blind, and composite sqlite-elasticsearch: CompartmentDefinitions never reach the ES index — /CompartmentDefinition serves empty and the UI caches the emptiness #462 is still open: the sqlite-elasticsearch leg loses its CompartmentDefinitions somewhere between the composite and the index, and the next failure needs exactly this evidence captured at the moment it happens instead of another round of artifact archaeology.
• NO_PROXY for the HFS process — no longer speculative. When first written this was defensive ("current runners show none set"). The terminology_integration failure on PR fix(bulk-submit): key entry results by their manifest output file #466's CI then demonstrated proxy-configured runners exist in the pool: the HTS client detoured through an egress proxy and hung 13 minutes (root-caused and fixed client-side in fix(terminology): the HTS client must fail fast, not hang behind a proxy #469). HFS's Elasticsearch client is the same reqwest with the same CIDR-no_proxy blind spot — on one of those runners, the matrix ES legs would fail with the same phantom timeouts. Naming the backend hosts exactly works for both curl and reqwest, and the step logs which proxy variables are present so any future incident can rule this in or out at a glance.

Validation

Dispatches on the pre-#452 branch demonstrated the failure and the recovery: with the host at quota, even image prunes refused (meta.db: disk quota exceeded); after a volume-first prune, run 30563256271 went green on sqlite-elasticsearch (the leg that failed every night) plus sqlite/postgres/mongodb, with s3-elasticsearch's single failure being the known #442 Recent bug (fix in #440).

Post-#452/#454 validation on this rebased branch (run 30582018605): the Docker-level failures are gone — both ES legs boot, seed, and run the suite; s3-elasticsearch effectively green (94/1, the 1 being #442); sqlite-elasticsearch surfaced the real product bug the choked host had been masking, filed as #462 — with the conformance probe and these diagnostics being what made that legible from workflow logs alone.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[smunini]

Heads up — #452 and #454 have both landed on main and overlap most of this PR. Flagging rather than touching the branch; the trim is yours to make.

Of the four commits here, one is still needed and three are superseded:

Commit	Status
ci(ui-matrix): route the Elasticsearch self-traffic around the runner proxy	Keep — unrelated real fix, nothing else covers it
ci: nightly janitor for the shared Docker host	Superseded by #454
ci(ui-matrix): prune the Docker host before the matrix runs	Superseded by #454
ci: prune volumes first, and keep pruning past failures	Superseded by #454

What replaced them. #454 moved the sweep from a nightly cron into a docker-host-gc composite action that runs per-run — ci.yml, ci-extended.yml and hts.yml after their reaps, and ui-tests-matrix's build gate before the matrix, which is the same placement your b025d8f8c chose and for the same reason. The reasoning for dropping the cron: the host fills during the working day, so a 05:30 UTC sweep can't help a run that dies at 16:03 — run 30559660973 was exactly that.

Your volumes-first ordering was right and is preserved — freeing volumes before images, with every step non-fatal, because a host at its quota fails image deletion (containerd needs a meta.db write) while volume removal frees the same filesystem without one. That lesson carried straight over.

One deliberate change from your version. The volume sweep is docker volume ls -qf dangling=true | grep -E '^[0-9a-f]{64}$' rather than docker volume prune -af. -a includes named volumes, so a sweep racing a service redeploy — while that service's named volume is briefly unreferenced — would delete its data with no undo. The host has no named volumes today (verified), so the two are equivalent right now; the narrow form is defence in depth for when that changes.

Root cause, for context. The underlying leak was docker rm -f without -v: the postgres/ES/mongo/minio/keycloak images all declare VOLUME, so every testcontainer created an anonymous volume and every reap orphaned it permanently. #452 fixed that at 84 call sites across 16 workflows. The host went from 168 dangling volumes / 42.21GB to 2 / 898MB.

So this PR should reduce to just 7099d5b4d. Worth retitling once it does, since the quota half of "heal the quota-choked Docker host" is now handled elsewhere.

[angela-helios]

Post-#452/#454 validation on this branch (run 30582018605): the Docker-level failures are gone — both ES legs boot, seed, and run the suite. s3-elasticsearch is effectively green (94 passed / 1 failed, the known #442 Recent bug whose fix rides in #440). sqlite-elasticsearch now shows a real product bug the choked host had been masking: CompartmentDefinitions never reach the ES index (SearchParameters do — 7,404 docs — while the CD index does not exist at all), so the conformance-fed UI surfaces fail as a cluster; filed as #462. The diagnostics in this PR are what made that legible from the workflow logs alone.