CI throughput: one runner, a 120 s docker client timeout, and a 67-minute release job on every branch #126

Objective. Restore CI throughput on the GitForge pipeline: the merge campaign is bounded by one runner's container-creation timeout and by the fact that every job rebuilds the world, not by FireCraft's own build times.

Why. Seven pull requests are in flight, all waiting on GitForge receipts, and the queue is serialized behind a single runner with capacity 4. A full branch run is 9 jobs of 10–40 minutes each, and the release job alone is ~67 minutes because it builds the DEB, the tarballs, the CLI and the web bundle on every branch push. Measured on 2026-11-11: 63 queued jobs across 7 runs, with clippy-test queued behind a resource_class_conflict for four different runs at once.

Two infra defects sit under that queue. Neither is a FireCraft code defect, and neither is visible in the FireCraft repo — they live in the runner's deployment — which is why they have gone unrecorded until now.

Measured evidence (from the authoritative store, /var/lib/gitforge/gitforge.db, jobs started 2026-11-10 onward)

One of the two timeout failures reddened main itself: run b61f198e (2026-11-11T17:04Z) failed at clippy-test with infrastructure_failure, cancelling its six downstream jobs. main currently has no green GitForge run of its own.

The chain's real cost, measured on the same store (2026-11-11/12)

One branch run, docs/deployment-receipt (PR #127, commit fcfe6db1):

The first job of that chain finished at 22:33Z and the seventh at 05:14Z — about seven hours of wall clock before the last job even started. Every one of those jobs compiled the workspace from scratch, because Cargo's target output went to a disposable directory inside the job container: no job ever started warm. The release job alone carries the merge receipt for a branch push.

Work

1. Make the two budgets agree. Set GITFORGE_DOCKER_TIMEOUT_SECS=600 on the runner service, so bollard's client timeout matches the acquisition window the runner already grants. The code comment at crates/gitforge-sandbox/src/docker.rs:270 states the rule — "keep bollard's default unless the operator raises it" — and the operator raised the acquisition window but not the client timeout.
2. Stop pulling images that cannot be pulled. Either pre-warm every referenced tag on every runner host before the commit lands, or make the runner fail fast with a clear "image not present on this host" message instead of a registry 404 that reads like a network fault.
3. Bring the second runner online. fedora-docker is registered with capacity 5 and has been offline since 2026-11-05; t5500-docker (capacity 2) has been offline since 2026-10-11. fedora is ready for it: the CI images :1, :2 and :chromium are already loaded there, the NAS is mounted, /nas/Temp/repos/GitForge is present, and both the git server (42782) and the API (42781) bind 0.0.0.0 and answer from it. Registering it roughly doubles fleet capacity.
4. Do not build the release artifact on every branch push. The release job costs ~67 minutes per run and produces artifacts nobody consumes for a branch. The :2 pipeline configuration already in flight on PR docs: reconcile FireCraft status, execution docs, and wiki sources #123 gates it with skip_if; landing that PR is the cheap fix and needs no runner changes.
5. Start every job from a warm Cargo target tree. The runner can bind-mount a host directory at /cargo-target and export CARGO_TARGET_DIR, so compiled artifacts survive the job — but it takes an exclusive advisory lock on every writable host directory a job mounts, for the job's whole lifetime (GitForge GF-11: two jobs writing one target tree clobber each other's binaries, measured on DIL on 2026-11-05). So the cache directory must be one per job, not one per image: a single shared directory would serialize every Cargo job in the fleet into one lane. PR feat(ci): a warm Cargo cache per job, so a run starts warm #129 (decision 0098) declares {cache_root}/cargo/{image}-<job> per job, and the offline validator now enforces that shape and runs in the GitHub backup CI's fmt job. The first run after the change is still cold; every later run of every branch reuses the artifacts.

Acceptance

• A GITFORGE_DOCKER_TIMEOUT_SECS value at least as large as GITFORGE_SANDBOX_ACQUIRE_SECS is set on every runner service, and the setting is recorded where the runner is deployed.
• A job whose workspace container takes longer than 120 s to create completes instead of failing with infrastructure_failure.
• fedora-docker reports a heartbeat and takes jobs; the runner inventory shows more than one online runner.
• main has a green GitForge run on its own head SHA.
• A branch push does not build the release artifacts.
• A warm run's per-job durations are read from the authoritative store and recorded here, so the throughput claim rests on a measurement rather than on the mechanism.

Boundaries. These are runner-deployment and GitForge-service changes, not FireCraft source changes. Changing the runner's service configuration, re-registering a runner, and touching the runner registration secret are operator actions that need explicit authorization — the registration token is a shared secret and was not read or moved while preparing this issue. Nothing in this issue has been applied.

Tracking: task #227; campaign RP-07.

Next action (owner: CI/release). Item 4 is a merge, not a change: PR #123 carries the skip_if pipeline. Item 5 is PR #129 and needs the same merge queue to drain before it can be rebased onto the :2 pipeline shape. Item 1 is a one-line service setting that removes a failure class the code already documents the remedy for, and it is the cheapest item on the list. Item 3 needs the runner registration secret, so it needs an operator with that secret; item 2 is a code change in GitForge.