Containers CI: buildkit graceful_stop aborts a random matrix row during GHA cache export

[Xore]

Recurring, image-agnostic flake in .github/workflows/containers.yml on the self-hosted honeypot-ci runners. The image builds fine — every stage reports DONE — then the job dies during #13 exporting to GitHub Actions Cache:

ERROR: failed to build: failed to receive status: rpc error: code = Unavailable
desc = closing transport due to: connection error: desc = "error reading from server: EOF",
received prior goaway: code: NO_ERROR, debug data: "graceful_stop"

It is not the PR content

Three runs on 2026-08-29 failed this way, each on a different matrix row, on branches whose diffs did not touch the failing image at all:

run	row that failed	branch diff touched
33248870831	citrix-honeypot	sandbox/ghosts/** only
33247431081	dashboard-next	—
33246670261	vps-portbridge	—

Re-running the failed job is currently the only remedy, which costs a full serialized matrix pass each time.

What the box shows

• The buildkit container receives a graceful SIGTERM, not a kill: no oom-kill: / Killed process in dmesg anywhere near the failure timestamp, and the gRPC GOAWAY carries code: NO_ERROR.
• No prune/cleanup timer fires at the failure time — the only relevant units are honeypot-sandbox-cleanup.timer (last ran 00:22) and systemd-tmpfiles-clean.timer.
• Five CI runner services are registered on the box (github-ci-runner, -2 … -5), so several docker/setup-buildx-action builders exist concurrently. The docker journal shows moby/buildkit:buildx-stable-1 being pulled and a new buildx_buildkit_builder-* joining the bridge every ~30-60s during a matrix pass.
• Leaked builders outlive their jobs — e.g. a buildx_buildkit_builder-* container still Up 2 hours after its run had finished, i.e. some jobs are not running their setup-buildx post-cleanup.

That combination (concurrent builders + missed post-cleanup + graceful stop landing on an unrelated in-flight builder) is the leading hypothesis, but the specific actor sending the SIGTERM is not yet pinned down.

Suggested next steps

1. Pin the actor: enable docker daemon debug or watch docker events --filter event=kill --filter event=die across a full matrix pass and catch which PID/unit stops the builder.
2. Reap leaked builders between jobs so a stale buildx_buildkit_builder-* cannot be swept up by a later cleanup.
3. Independently of the root cause, make the build step tolerate it — a retry around docker/build-push-action in containers.yml, so a teardown during cache export costs a retry instead of a whole re-run. This is cross-cutting across all 18 matrix rows, so it wants its own PR rather than riding along on an unrelated one.

Filed while fixing an unrelated failure on PR #2637, which was collateral damage from this flake.

[Xore]

Root cause pinned: the five runners share one buildx current pointer

The missing actor from the original report is not an external pruner or a stray SIGTERM — it is each job's own legitimate setup-buildx-action post-cleanup, tearing down a builder that a different concurrent job had bound itself to.

The chain

1. All five runner services run as the same user with the same home:

   /etc/systemd/system/actions.runner.Xore-APIARY.supermicro-ci{,-2,-3,-4,-5}.service
     User=github-ci-runner
   $ getent passwd github-ci-runner | cut -d: -f6
     /opt/github-ci-runner
   

   buildx keeps its state in $DOCKER_CONFIG/buildx → /opt/github-ci-runner/.docker/buildx for all five.

2. docker/setup-buildx-action@v4 has a use input defaulting to true (use: Zl("use") in dist/index.cjs), i.e. docker buildx create --use, which writes the single shared .docker/buildx/current.

3. That action exports no per-job builder variable — grep -c BUILDX_BUILDER dist/index.cjs → 0, exportVariable → 0.

4. containers.yml passes no builder: to docker/build-push-action@v7, so the build resolves the builder from that one shared current at invocation time.

Caught in the act — two concurrent jobs, two builder instances, one current:

$ cat /opt/github-ci-runner/.docker/buildx/current
{"Key":"unix:///var/run/docker.sock","Name":"builder-9d8d25f3-...","Global":false}

$ ls /opt/github-ci-runner/.docker/buildx/instances/
builder-4e249123-730c-4aab-aea3-ef2c136f4af1     <- job A created this
builder-9d8d25f3-ef77-4b59-a4fb-bafbff72e730     <- job B created this, won `current`
builder-c3355694-843a-4155-ac33-f6a9245e1061     <- leaked, 7h old

Job A creates its builder, job B's --use overwrites current, then A's build step runs on B's builder. When B finishes, B's post-step docker buildx rm gracefully stops that buildkitd — and A, still streaming its GHA cache export, gets GOAWAY ... graceful_stop. NO_ERROR and the absence of any OOM are exactly right for a normal teardown, which is why the earlier hunt for a killer came up empty.

Why it looks "random" and hits unrelated matrix rows

Which row dies depends purely on which job loses the current race and how long its cache export runs — no relationship to the diff, which matches the three rows in the original table.

Corroborating timestamp

Two rows on different runners failed in the same millisecond:

dashboard-next (supermicro-ci-4)  16:22:13.8339  buildx failed with: ... graceful_stop
dicompot       (supermicro)       16:22:13.8334  buildx failed with: ... graceful_stop

with the docker journal recording a container stop at 16:22:13.810, ~20ms earlier. One teardown, two victims on two different runners — consistent with a shared builder, and not with a per-job actor. (Observation; the shared-current state above is the direct evidence.)

This regression was introduced by scaling the pool in #2572 — with one runner there is no concurrency, so the shared pointer never races.

Fix applied on the box (runner infrastructure only, no repo change)

Each runner now gets its own buildx state via its .env (the runner injects this into the job environment):

# /opt/github-ci-runner-N/.env
BUILDX_CONFIG=/opt/github-ci-runner-N/.docker/buildx

Previous .env files saved as .env.bak-20260829; all five services restarted. Verified afterwards that the pointers are now independent:

ci  : {"Name":"builder-8e51827b-...","Global":false}
ci-4: {"Name":"builder-f5043656-...","Global":false}

Also removed the leaked 7-hour buildx_buildkit_builder-c3355694-* container and its stale instance file, and deleted a bogus actions.runner.Xore-APIARY.supermicro-ci-.service unit (empty name, ExecStart=/bin/runsvc.sh, WorkingDirectory=) that had been sitting in failed state since 10:15.

Still worth doing in the repo (needs a PR, not done here)

1. Bind the builder explicitly — defence in depth, so this cannot regress if the runner layout changes:

   - uses: docker/setup-buildx-action@v4
     id: buildx
   - uses: docker/build-push-action@v7
     with:
       builder: ${{ steps.buildx.outputs.name }}

2. The retry-around-build-push-action idea from step 3 of the original report is still worthwhile as a general robustness measure.
3. Reap leaked buildx_buildkit_builder-* containers between jobs (step 2 above) — the leak is real and independently confirmed.

[Xore]

Fix confirmed under load

Follow-up to the root cause above. After giving each runner its own BUILDX_CONFIG:

• Containers run 33262958724 — completed/success, 19/19 rows green, zero graceful_stop — a full matrix pass under five-way concurrency with no lost row.
• Two PRs that were blocked purely by this flake plus restart collateral went green with no code change: fix(sandbox/windows): provision FakeNet runtime deps explicitly; add LASTEXITCODE guard #2640 and docs(benchmarks): #1805-c ghidra-slot matrix — cold-protocol re-measure, and promote gemma-4-26B-A4B #2641, both CLEAN at 89/89.
• No builder leaks accumulating since the change (buildkit container count now tracks the number of running jobs, and drops to 0 when idle).

Separately, the runner pool has been reduced to two active runners (supermicro-ci, supermicro-ci-2) pending a CPU upgrade on the box — the other three are stopped and disabled. The BUILDX_CONFIG entry is already present in all five .env files, so the remaining three come back isolated when they are re-enabled.

Leaving this open for the repo-side hardening in the "Still worth doing" list, in particular binding the builder explicitly with builder: ${{ steps.buildx.outputs.name }} so the workflow does not depend on runner-host layout.

[Xore]

Picked up for automated fix. Branch: orchestrator/issue-2639 (base main).

[Xore]

PR #2672 merged. CI: 108 passing, 0 failing, mergeState=CLEAN. Closes #2639 confirmed.