feat: Add Docker default-address-pools policy, subnet-headroom health gate, and controller circuit breaker for managed fleet lifecycle

[Nickfost]

Docker subnet-address-pool exhaustion: missing fleet lifecycle, headroom, and rollback contract

Scope: ci-fleet design gap in managed-host lifecycle/capacity. This is not proof of a defect in any downstream consumer branch.

Observed GitHub evidence (no secrets)

• GitHub job metadata for the affected controller identifies failed runners as ci-fleet-rd-ci-fleet-01-* in runner group trusted-private-ci.
• Failed downstream consumer jobs report the Docker error: all predefined address pools have been fully subnetted.
• Reviewed private desired-state configuration declares controller rd-ci-fleet-01, pool trusted-ci (runner group trusted-private-ci), an exact ci-fleet engine_ref (d0398a831126594bd305a2cce39f1ce99841a84d), min_runners: 0, max_runners: 6, 4 CPU / 8192 MiB per runner. (Read-only verified; no credentials, addresses, or host identifiers were exposed.)
• The installed engine and current ci-fleet main install a five-minute health timer (ci-fleet-health.timer) and a daily cleanup timer (ci-fleet-cleanup.timer).
• ci-fleet main contains no Docker default-address-pools policy, no subnet-headroom health gate, and no controller circuit breaker (code-search for default-address-pools, subnet, headroom, circuit returns zero matches across the repository).
• ci-fleet scripts/cleanup.sh targets only its own managed label schema (io.randomdevelopment.ci-fleet.managed=true plus io.randomdevelopment.ci-fleet.expires-at), works off the expires-at label, and never performs a global docker system prune/docker network prune.

What is proven vs. what is not

Proven (from GitHub metadata and repository/configuration state above): ci-fleet's current managed-host contract has no address-pool policy, no subnet-headroom telemetry/gate, no low-water circuit breaker, and its cleanup contract only governs ci-fleet-managed labels — so project-originated Docker networks/volumes that do not carry the fleet expiry label are outside scope.

Cannot be known without inspecting the host: whether the immediate failure on the affected host is (a) leaked/orphaned networks accumulating past their use, (b) an undersized or overlapping default-address-pools definition, or (c) a combination of both. This issue deliberately does not assert which. The design gap is real regardless of which condition is live today.

Why this is a ci-fleet defect, not a consumer defect

The error surfaces on downstream consumer jobs, but the missing controls — address-pool policy validation, subnet-headroom telemetry, a circuit breaker, scoped orphan reconciliation, and a network-allocation-aware upgrade path — all belong to the fleet's managed-host lifecycle contract. Nothing in the observed evidence implicates downstream consumer branch logic; the jobs are ordinary consumers of the fleet's Docker networking. Fix the fleet contract; do not change application runtime logic.

Required fixes (design contract)

1. Common ownership/expiry label contract for project resources

Define a single fleet-wide label contract (owned by ci-fleet) that project containers, volumes, and networks must carry so cleanup/reconciliation can attribute and expire them:

• a fleet ownership label (e.g. io.randomdevelopment.ci-fleet.managed=true),
• a common expiry label (e.g. io.randomdevelopment.ci-fleet.expires-at=<epoch>),
• a project/owner label for provenance reporting.
  ci-fleet's cleanup and reconciliation must honor this contract for all three resource kinds, not only ci-fleet-managed runners.

2. Reviewed Docker address-pool policy, validated at install/update — never a moving branch

• Add a reviewed default-address-pools policy as a pinned, Git-authored artifact (private configuration or engine release).
• The installer and update path must validate this policy before applying and must never follow a moving branch (consistent with existing --ref full-SHA rule).
• Validation must fail closed if the policy is absent, malformed, or would reduce headroom below required capacity.

3. Subnet-headroom telemetry + runtime low-water circuit breaker

• Emit subnet-headroom telemetry (free vs. provisioned subnet capacity, per address pool) into the existing health/telemetry contract — no new off-host phone-home.
• Add a runtime low-water gate: when headroom drops below a reviewed threshold, the controller stops creating new runners while in-flight jobs drain. This is a fleet lifecycle control, not a job cancellation.

4. Frequent scoped reconciliation of expired/orphaned resources; forbid blind prune

• Reconcile expired and provably orphaned resources on a frequent schedule (scope-aligned with existing timers). "Provably orphaned" = resource with the fleet ownership label and no live owning job/controller reference.
• Forbid global or blind docker network prune (and docker system prune). Only label-scoped, dry-run-by-default, --apply-gated removal is permitted — consistent with the existing cleanup safety model.

5. Network-allocation-aware upgrade path with transactional rollback

• The upgrade/reconcile path must be able to recover when no new Docker network can be allocated: detect allocation failure, stop forward progress, and roll back to the last-known-good checkpoint transactionally (reusing the existing installer checkpoint/rollback machinery), without leaving partial networks or a wedged controller.

6. Capacity proof at configured max_runners

• Prove capacity at the configured max_runners (currently 6) covering: success, failure, cancellation, timeout, and repeated concurrency (sustained churn), asserting that runner creation halts at the low-water gate rather than exhausting subnets.

7. Coordinated downstream-consumer contract changes — app logic stays out

• Specify the coordinated change so affected downstream-consumer Compose/CI resources adopt the common fleet ownership/expiry label contract.
• The change is a consumer contract/labeling change only; application runtime logic must remain in the consumer repositories and out of ci-fleet.

8. TDD acceptance tests

Add/extend tests (red-green) covering:

• schema: address-pool policy schema and the common ownership/expiry label contract;
• renderer: renders the pinned policy and labels without secret leakage;
• installer: validates a reviewed address-pool policy and refuses a moving branch;
• health: emits subnet-headroom telemetry and trips the low-water gate;
• cleanup: removes only label-scoped, expired/orphaned resources; never global/blind prune;
• controller behavior: halts new-runner creation at low water while jobs drain;
• rollback: recovers when no Docker network can be allocated (no partial state);
• legacy resource reporting: reports pre-existing/non-conforming project resources without deleting them.

Authorization boundary (explicit)

Merging code that closes this issue does not authorize:

• deployment to any host,
• Docker cleanup or docker network prune/docker system prune on any host,
• Docker daemon restart or daemon.json change,
• host/SSH/console access,
• or any VM change.

Those remain explicit operator/infrastructure actions outside this repository's authority. This issue only defines the ci-fleet code contract.

Related (not duplicates)

• feat: Record per-runner and host utilization telemetry to drive max_runners and pool-budget capacity decisions #50 — capacity telemetry (runner/host utilization); does not own Docker subnet pools or orphan-network reconciliation.
• Add scoped cleanup scanner and host health checks #4 — scoped expiry cleanup; covers ci-fleet-managed labels only, no address-pool/headroom/circuit-breaker scope.
• Add fleet-wide host health monitoring and missed-heartbeat alerts #34 / Add fleet-wide host health monitoring #42 — fleet health monitoring; no subnet-headroom/circuit-breaker gate.
• fix(capacity): validate reviewed runner targets #47 — capacity preflight; validates runner targets, not Docker networking headroom.
• Add Git-authored controller desired state #33 / Define Git-authored desired state for fleet controllers and capacity #32 — Git-authored desired state; no address-pool policy field today.