From 5cbbdabd0d395e6fe149b72692b90b68d9bb3964 Mon Sep 17 00:00:00 2001
From: Nickfosts Hermes <me@nickfost.com>
Date: Fri, 17 Jul 2026 15:33:08 -0500
Subject: [PATCH 1/6] feat(ops): add post-pilot capacity promotion checks

---
 docs/ADDING-A-HOST.md              |   2 +
 docs/CAPACITY-PROMOTION.md         | 144 ++++++++++++++++++++
 scripts/capacity-preflight.sh      | 203 +++++++++++++++++++++++++++++
 scripts/test-capacity-preflight.sh | 180 +++++++++++++++++++++++++
 scripts/validate.sh                |   1 +
 5 files changed, 530 insertions(+)
 create mode 100644 docs/CAPACITY-PROMOTION.md
 create mode 100755 scripts/capacity-preflight.sh
 create mode 100755 scripts/test-capacity-preflight.sh

diff --git a/docs/ADDING-A-HOST.md b/docs/ADDING-A-HOST.md
index 6148dab9..3e42e1f5 100644
--- a/docs/ADDING-A-HOST.md
+++ b/docs/ADDING-A-HOST.md
@@ -151,6 +151,8 @@ The steady-state host should need no project-specific edits. Adding a project ch
 
 Raise `CI_FLEET_MAX_RUNNERS` only after measuring CPU, memory, disk, network, cache growth, collisions, cancellation, and cleanup. Keep explicit per-runner limits.
 
+Keep the one-runner pilot preflight unchanged. For any later increase, follow [Post-pilot capacity promotion](CAPACITY-PROMOTION.md), run `scripts/capacity-preflight.sh` before and after the controller-only recreation, and retain the new maximum only when the separately authorized workload and cleanup proof satisfy the predeclared resource policy.
+
 Use another host or location when it improves failure tolerance. Give every added host a new instance and scale-set name while retaining the shared routing label.
 
 ## 10. Drain, replace, or remove a host
diff --git a/docs/CAPACITY-PROMOTION.md b/docs/CAPACITY-PROMOTION.md
new file mode 100644
index 00000000..163e096d
--- /dev/null
+++ b/docs/CAPACITY-PROMOTION.md
@@ -0,0 +1,144 @@
+# Post-pilot capacity promotion
+
+Use this procedure only after the strict one-runner pilot has passed. It validates and changes one already isolated controller; it does not authorize a project workflow by itself.
+
+`scripts/preflight.sh` remains the initial pilot gate and accepts only `MIN=0`, `MAX=1`. `scripts/capacity-preflight.sh` is a separate, read-only post-pilot gate. It never edits host configuration.
+
+## Capacity policy
+
+Declare the policy before observing a larger workload:
+
+- requested MAX is explicit, positive, and bounded;
+- `MIN` remains zero;
+- the configured instance, scale set, routing label, runner group, Docker socket group, and per-runner limits must equal the running controller's effective values;
+- no managed runner, project job resource, unrelated running container, controller OOM, or current-boot kernel OOM evidence may exist;
+- Docker filesystem use must be below `CI_FLEET_DISK_WARN_PERCENT` (80 by default);
+- reserve the controller's Compose limit: 1 CPU and 512 MiB;
+- reserve 1 CPU and 1 GiB for Docker overhead;
+- reserve for the operating system the greater of 1 CPU or 15% of logical CPU capacity;
+- reserve for the operating system the greater of 2 GiB or 20% of physical memory;
+- admit `target MAX × CI_FLEET_RUNNER_CPUS` and `target MAX × CI_FLEET_RUNNER_MEMORY_MIB` only when those allocations plus all reserves fit;
+- require currently available memory to cover all target runners plus controller and Docker reservations.
+
+Runner limits are controller admission inputs. Project containers use the host Docker daemon as siblings of the runner container, so the separately authorized live proof must still observe whole-host CPU, memory, disk, collision, and cleanup behavior.
+
+During a live proof, retain the target only if every five-second sample keeps CPU busy below 85%, available memory at or above the greater of 2 GiB or 20% of total memory, and Docker filesystem use below 80%. Any OOM, unrelated workload, controller/Docker failure, third runner, observer gap, or cleanup residue requires restoration.
+
+## 1. Verify idle pilot state
+
+Gate all dispatches that can target this controller. Confirm no queued, assigned, or running fleet job and no instance-owned runner in any state. Keep the dispatch gate closed until post-change verification completes.
+
+From the reviewed checkout, load the root-only host configuration without tracing or printing it:
+
+```bash
+set -Eeuo pipefail
+set +x
+cd /opt/ci-fleet
+set -a
+. /etc/ci-fleet/ci-fleet.env
+set +a
+scripts/preflight.sh
+```
+
+Require exactly `PREFLIGHT_OK warnings=0`. This proves the original `MIN=0`, `MAX=1` pilot contract remains valid.
+
+## 2. Validate target capacity without changing it
+
+```bash
+scripts/capacity-preflight.sh --phase pre-change --target-max 2
+```
+
+Require `CAPACITY_PREFLIGHT_OK phase=pre-change target_max=2 configured_max=1 effective_max=1`. Record only the safe budget summaries.
+
+## 3. Create one protected backup
+
+Use one UTC timestamp and refuse to overwrite an existing path:
+
+```bash
+backup_dir=/etc/ci-fleet/backups
+stamp=$(date -u +%Y%m%dT%H%M%SZ)
+backup="$backup_dir/ci-fleet.env.before-max2.$stamp"
+install -d -o root -g root -m 0700 "$backup_dir"
+test ! -e "$backup"
+install -o root -g root -m 0600 /etc/ci-fleet/ci-fleet.env "$backup"
+cmp -s /etc/ci-fleet/ci-fleet.env "$backup"
+sha256sum "$backup" | cut -d' ' -f1
+```
+
+Record the path and checksum only. Never print or copy the backup contents, and do not delete the backup during the proof.
+
+## 4. Change one exact setting
+
+Before editing, require exactly one active assignment and the pilot value:
+
+```bash
+test "$(grep -c '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env)" -eq 1
+grep -qx 'CI_FLEET_MAX_RUNNERS=1' /etc/ci-fleet/ci-fleet.env
+```
+
+Edit only that line to `CI_FLEET_MAX_RUNNERS=2`. Then verify without printing a diff:
+
+```bash
+test "$(stat -c '%u:%g:%a' /etc/ci-fleet/ci-fleet.env)" = 0:0:600
+test "$(grep -c '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env)" -eq 1
+grep -qx 'CI_FLEET_MAX_RUNNERS=2' /etc/ci-fleet/ci-fleet.env
+cmp -s \
+  <(grep -v '^CI_FLEET_MAX_RUNNERS=' "$backup") \
+  <(grep -v '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env)
+docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet
+```
+
+Do not run the pilot preflight against MAX=2; it must continue to reject that value.
+
+## 5. Recreate only the controller
+
+Reconfirm zero runners and zero jobs immediately before stopping. Stop only the controller with enough grace for scale-set deletion:
+
+```bash
+docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller
+```
+
+Verify the exact old scale set is absent and there is no runner before starting the replacement. Do not delete an apparent duplicate until ownership and zero active jobs are proven.
+
+```bash
+docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \
+  up -d --no-deps --force-recreate --timeout 60 controller
+```
+
+Do not recreate `runner-image`, remove volumes, rebuild images, or touch unrelated resources.
+
+## 6. Verify effective state
+
+Wait boundedly for one sanitized `controller ready` record reporting `minRunners=0` and `maxRunners=2`. Require one running controller, restart count zero, one intended scale set, the unchanged experimental routing label and runner group, and no idle runner.
+
+Reload the host configuration and run:
+
+```bash
+scripts/capacity-preflight.sh --phase post-change --target-max 2
+scripts/healthcheck.sh
+```
+
+Require both to pass. The post-change preflight compares configured state with a filtered set of effective controller values and never prints arbitrary container environment entries.
+
+## 7. Run one separately authorized proof
+
+Start bounded runner, task-job, project-resource, and host-metric observers before dispatch. Dispatch exactly the approved workload once. Do not retry a failed proof and do not raise MAX above two.
+
+Observe runner creation/destruction, actual two-way overlap, no third runner, whole-host CPU/memory/disk thresholds, Docker/controller health, exact project identity, and automatic cleanup. Run the instance-scoped cleanup dry-run and healthcheck after all jobs terminate.
+
+## 8. Retain or restore (rollback)
+
+Retain MAX=2 only when the authorized workload succeeds, actual two-way job and runner overlap is proven, every predeclared resource threshold passes, no manual cleanup is required, all runner/project residue is zero, post-change capacity preflight passes, cleanup dry-run is empty, and healthcheck passes.
+
+On any failure, keep dispatch gated, wait for exact job termination, and restore the backup:
+
+```bash
+docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller
+install -o root -g root -m 0600 "$backup" /etc/ci-fleet/ci-fleet.env
+cmp -s "$backup" /etc/ci-fleet/ci-fleet.env
+docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet
+docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \
+  up -d --no-deps --force-recreate --timeout 60 controller
+```
+
+Verify a sanitized ready record with `MIN=0`, `MAX=1`, then run `scripts/preflight.sh`, `scripts/healthcheck.sh`, and the instance-scoped cleanup dry-run. Require zero runners, zero jobs, one intended scale set, no duplicate controller, and no residue before reopening dispatch.
diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh
new file mode 100755
index 00000000..ebcfa942
--- /dev/null
+++ b/scripts/capacity-preflight.sh
@@ -0,0 +1,203 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+phase=
+target_max=
+
+usage() {
+  printf 'usage: %s --phase pre-change|post-change --target-max N\n' "$0" >&2
+}
+
+die() {
+  printf 'FAIL %s\n' "$1" >&2
+  printf 'CAPACITY_PREFLIGHT_FAILED phase=%s\n' "${phase:-unknown}" >&2
+  exit 2
+}
+
+while (($#)); do
+  case "$1" in
+    --phase)
+      (($# >= 2)) || die '--phase requires a value'
+      phase=$2
+      shift 2
+      ;;
+    --target-max)
+      (($# >= 2)) || die '--target-max requires a value'
+      target_max=$2
+      shift 2
+      ;;
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    *)
+      usage
+      die 'unknown capacity-preflight argument'
+      ;;
+  esac
+done
+
+[[ "$phase" == pre-change || "$phase" == post-change ]] || die 'explicit --phase pre-change or post-change is required'
+[[ -n "$target_max" ]] || die 'explicit --target-max is required'
+[[ "$target_max" =~ ^[1-9][0-9]{0,3}$ ]] || die 'target MAX must be a positive integer'
+target_max=$((10#$target_max))
+
+required=(
+  CI_FLEET_GITHUB_URL
+  CI_FLEET_SCALE_SET_NAME
+  CI_FLEET_LABELS
+  CI_FLEET_RUNNER_GROUP
+  CI_FLEET_INSTANCE
+  CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE
+  CI_FLEET_DOCKER_GID
+  CI_FLEET_MIN_RUNNERS
+  CI_FLEET_MAX_RUNNERS
+  CI_FLEET_RUNNER_CPUS
+  CI_FLEET_RUNNER_MEMORY_MIB
+)
+for name in "${required[@]}"; do
+  [[ -n "${!name:-}" ]] || die "$name is required for capacity validation"
+done
+
+[[ "$CI_FLEET_GITHUB_URL" == https://github.com/* ]] || die 'CI_FLEET_GITHUB_URL must use https://github.com'
+[[ "$CI_FLEET_SCALE_SET_NAME" == *"$CI_FLEET_INSTANCE"* ]] || die 'scale-set name must retain the stable fleet instance ID'
+[[ ",$CI_FLEET_LABELS," != *,self-hosted,* ]] || die 'shared routing labels must not include self-hosted'
+[[ "$CI_FLEET_RUNNER_GROUP" != Default ]] || die 'post-pilot capacity requires the existing non-default runner group'
+[[ "$CI_FLEET_MIN_RUNNERS" == 0 ]] || die 'CI_FLEET_MIN_RUNNERS must be 0'
+[[ "$CI_FLEET_MAX_RUNNERS" =~ ^[1-9][0-9]{0,3}$ ]] || die 'configured MAX must be a positive integer'
+configured_max=$((10#$CI_FLEET_MAX_RUNNERS))
+[[ "$CI_FLEET_RUNNER_CPUS" =~ ^[1-9][0-9]{0,5}$ ]] || die 'per-runner CPU limit must be a positive integer'
+runner_cpus=$((10#$CI_FLEET_RUNNER_CPUS))
+[[ "$CI_FLEET_RUNNER_MEMORY_MIB" =~ ^[1-9][0-9]{0,9}$ ]] || die 'per-runner memory limit must be a positive integer MiB value'
+runner_memory_mib=$((10#$CI_FLEET_RUNNER_MEMORY_MIB))
+
+if [[ "$phase" == pre-change ]]; then
+  ((target_max > configured_max)) || die 'pre-change target MAX must be greater than configured MAX'
+else
+  ((configured_max == target_max)) || die 'post-change configured MAX does not match requested target'
+fi
+
+for command in docker stat df awk getconf free dmesg grep wc tr; do
+  command -v "$command" >/dev/null || die "$command is unavailable"
+done
+docker info >/dev/null 2>&1 || die 'Docker daemon is unreachable'
+docker compose version >/dev/null 2>&1 || die 'Docker Compose plugin is unavailable'
+
+socket_gid=$(stat -c '%g' /var/run/docker.sock 2>/dev/null || true)
+[[ -n "$socket_gid" && "$socket_gid" == "$CI_FLEET_DOCKER_GID" ]] || die 'Docker socket group does not match CI_FLEET_DOCKER_GID'
+
+secret=$CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE
+[[ -f "$secret" ]] || die 'GitHub App PEM file is missing'
+secret_mode=$(stat -c '%a' "$secret" 2>/dev/null || true)
+secret_owner=$(stat -c '%u' "$secret" 2>/dev/null || true)
+[[ "$secret_mode" == 600 && "$secret_owner" == 0 ]] || die 'GitHub App PEM must be root-owned mode 0600'
+
+warn_percent=${CI_FLEET_DISK_WARN_PERCENT:-80}
+[[ "$warn_percent" =~ ^[1-9][0-9]?$|^100$ ]] || die 'CI_FLEET_DISK_WARN_PERCENT must be an integer from 1 through 100'
+docker_root=$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)
+[[ "$docker_root" == /* ]] || die 'Docker root directory could not be determined'
+disk_used=$(df -P "$docker_root" 2>/dev/null | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
+[[ "$disk_used" =~ ^[0-9]{1,3}$ ]] || die 'Docker filesystem utilization could not be determined'
+((disk_used < warn_percent)) || die "Docker filesystem must remain below ${warn_percent}%"
+
+controller=${CI_FLEET_CONTROLLER_CONTAINER:-ci-fleet-controller-1}
+controller_state=$(docker inspect --format '{{.State.Status}}' "$controller" 2>/dev/null || true)
+[[ "$controller_state" == running ]] || die 'controller must be running and healthy'
+controller_oom=$(docker inspect --format '{{.State.OOMKilled}}' "$controller" 2>/dev/null || true)
+[[ "$controller_oom" == false ]] || die 'controller OOM evidence is present'
+
+controller_env=$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$controller" 2>/dev/null) || die 'effective controller configuration could not be inspected'
+effective_value() {
+  local wanted=$1 line key value= count=0
+  while IFS= read -r line; do
+    key=${line%%=*}
+    if [[ "$key" == "$wanted" ]]; then
+      value=${line#*=}
+      count=$((count + 1))
+    fi
+  done <<<"$controller_env"
+  ((count == 1)) || die "effective controller $wanted must occur exactly once"
+  printf '%s' "$value"
+}
+
+stable=(
+  CI_FLEET_GITHUB_URL
+  CI_FLEET_SCALE_SET_NAME
+  CI_FLEET_LABELS
+  CI_FLEET_RUNNER_GROUP
+  CI_FLEET_INSTANCE
+  CI_FLEET_MIN_RUNNERS
+  CI_FLEET_RUNNER_CPUS
+  CI_FLEET_RUNNER_MEMORY_MIB
+  CI_FLEET_DOCKER_GID
+)
+for name in "${stable[@]}"; do
+  effective=$(effective_value "$name")
+  [[ "$effective" == "${!name}" ]] || die "effective controller $name does not match configured state"
+done
+effective_min=$(effective_value CI_FLEET_MIN_RUNNERS)
+effective_max=$(effective_value CI_FLEET_MAX_RUNNERS)
+[[ "$effective_min" == 0 ]] || die 'effective controller MIN must be 0'
+[[ "$effective_max" =~ ^[1-9][0-9]{0,3}$ ]] || die 'effective controller MAX must be a positive integer'
+effective_max=$((10#$effective_max))
+if [[ "$phase" == pre-change ]]; then
+  ((effective_max == configured_max)) || die 'effective controller MAX does not match configured pre-change MAX'
+else
+  ((effective_max == target_max)) || die 'effective controller MAX does not match requested target'
+fi
+
+managed_count=$(docker ps -aq \
+  --filter label=io.randomdevelopment.ci-fleet.managed=true \
+  --filter label=io.randomdevelopment.ci-fleet.kind=runner \
+  --filter "label=io.randomdevelopment.ci-fleet.instance=$CI_FLEET_INSTANCE" | wc -l | tr -d ' ')
+[[ "$managed_count" == 0 ]] || die 'active managed runner or runner residue exists'
+job_count=$(docker ps -aq --filter label=ci-fleet.repository | wc -l | tr -d ' ')
+[[ "$job_count" == 0 ]] || die 'active fleet job container or job residue exists'
+
+while IFS= read -r running_name; do
+  [[ -z "$running_name" || "$running_name" == "$controller" ]] || die 'unrelated running Docker workload exists'
+done < <(docker ps --format '{{.Names}}')
+
+if ! oom_evidence=$(dmesg --level=err,crit,alert,emerg 2>/dev/null); then
+  die 'kernel OOM evidence could not be inspected'
+fi
+if grep -Eiq 'out of memory|oom-killer|killed process' <<<"$oom_evidence"; then
+  die 'OOM evidence exists for the current boot'
+fi
+
+logical_cpus=$(getconf _NPROCESSORS_ONLN 2>/dev/null || true)
+[[ "$logical_cpus" =~ ^[1-9][0-9]{0,5}$ ]] || die 'logical CPU count could not be determined'
+logical_cpus=$((10#$logical_cpus))
+read -r total_memory_mib available_memory_mib < <(free -m | awk '/^Mem:/ {print $2, $7}')
+[[ "$total_memory_mib" =~ ^[1-9][0-9]{0,9}$ ]] || die 'total memory could not be determined'
+[[ "$available_memory_mib" =~ ^[1-9][0-9]{0,9}$ ]] || die 'available memory could not be determined'
+total_memory_mib=$((10#$total_memory_mib))
+available_memory_mib=$((10#$available_memory_mib))
+
+# Fixed reservations match deploy/compose.yaml and the documented capacity policy.
+controller_cpu_millicpus=1000
+controller_memory_mib=512
+docker_cpu_reserve_millicpus=1000
+docker_memory_reserve_mib=1024
+total_cpu_millicpus=$((logical_cpus * 1000))
+os_cpu_reserve_millicpus=$(((total_cpu_millicpus * 15 + 99) / 100))
+((os_cpu_reserve_millicpus >= 1000)) || os_cpu_reserve_millicpus=1000
+os_memory_reserve_mib=$(((total_memory_mib * 20 + 99) / 100))
+((os_memory_reserve_mib >= 2048)) || os_memory_reserve_mib=2048
+required_cpu_millicpus=$((target_max * runner_cpus * 1000 + controller_cpu_millicpus + docker_cpu_reserve_millicpus + os_cpu_reserve_millicpus))
+required_memory_mib=$((target_max * runner_memory_mib + controller_memory_mib + docker_memory_reserve_mib + os_memory_reserve_mib))
+required_available_memory_mib=$((target_max * runner_memory_mib + controller_memory_mib + docker_memory_reserve_mib))
+
+((required_cpu_millicpus <= total_cpu_millicpus)) || die 'CPU capacity is insufficient for the requested target and reserves'
+((required_memory_mib <= total_memory_mib)) || die 'memory capacity is insufficient for the requested target and reserves'
+((required_available_memory_mib <= available_memory_mib)) || die 'available memory is insufficient for the requested target'
+
+printf 'OK configured_max=%d requested_target_max=%d\n' "$configured_max" "$target_max"
+printf 'OK effective_min=%d effective_max=%d\n' "$effective_min" "$effective_max"
+printf 'OK cpu_budget_millicpus=%d/%d os_reserve_millicpus=%d docker_reserve_millicpus=%d\n' \
+  "$required_cpu_millicpus" "$total_cpu_millicpus" "$os_cpu_reserve_millicpus" "$docker_cpu_reserve_millicpus"
+printf 'OK memory_budget_mib=%d/%d available_required_mib=%d available_mib=%d os_reserve_mib=%d docker_reserve_mib=%d\n' \
+  "$required_memory_mib" "$total_memory_mib" "$required_available_memory_mib" "$available_memory_mib" "$os_memory_reserve_mib" "$docker_memory_reserve_mib"
+printf 'OK docker_filesystem_used_percent=%d warning_percent=%d\n' "$disk_used" "$warn_percent"
+printf 'CAPACITY_PREFLIGHT_OK phase=%s target_max=%d configured_max=%d effective_max=%d\n' \
+  "$phase" "$target_max" "$configured_max" "$effective_max"
diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh
new file mode 100755
index 00000000..3320ac3c
--- /dev/null
+++ b/scripts/test-capacity-preflight.sh
@@ -0,0 +1,180 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+tmp=$(mktemp -d)
+trap 'rm -rf "$tmp"' EXIT
+mkdir -p "$tmp/bin"
+
+cat >"$tmp/bin/docker" <<'EOF'
+#!/usr/bin/env bash
+set -u
+case "${1:-}" in
+  info)
+    if [[ "$*" == *'.DockerRootDir'* ]]; then printf '/var/lib/docker\n'; fi
+    exit 0
+    ;;
+  compose)
+    [[ "${2:-}" == version ]] && exit 0
+    ;;
+  inspect)
+    args="$*"
+    if [[ "$args" == *'.State.Status'* ]]; then
+      printf '%s\n' "${FAKE_CONTROLLER_STATE:-running}"
+    elif [[ "$args" == *'.State.OOMKilled'* ]]; then
+      printf '%s\n' "${FAKE_CONTROLLER_OOM:-false}"
+    elif [[ "$args" == *'.Config.Env'* ]]; then
+      printf '%s\n' \
+        "CI_FLEET_GITHUB_URL=${CI_FLEET_GITHUB_URL}" \
+        "CI_FLEET_SCALE_SET_NAME=${CI_FLEET_SCALE_SET_NAME}" \
+        "CI_FLEET_LABELS=${CI_FLEET_LABELS}" \
+        "CI_FLEET_RUNNER_GROUP=${CI_FLEET_RUNNER_GROUP}" \
+        "CI_FLEET_INSTANCE=${CI_FLEET_INSTANCE}" \
+        "CI_FLEET_MIN_RUNNERS=${FAKE_EFFECTIVE_MIN:-${CI_FLEET_MIN_RUNNERS}}" \
+        "CI_FLEET_MAX_RUNNERS=${FAKE_EFFECTIVE_MAX:-${CI_FLEET_MAX_RUNNERS}}" \
+        "CI_FLEET_RUNNER_CPUS=${CI_FLEET_RUNNER_CPUS}" \
+        "CI_FLEET_RUNNER_MEMORY_MIB=${CI_FLEET_RUNNER_MEMORY_MIB}" \
+        "CI_FLEET_DOCKER_GID=${CI_FLEET_DOCKER_GID}" \
+        "UNRELATED_SECRET=${FAKE_SECRET_VALUE:-CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT}"
+    else
+      exit 1
+    fi
+    ;;
+  ps)
+    args="$*"
+    if [[ "$args" == *'--format'* ]]; then
+      printf '%s\n' "${CI_FLEET_CONTROLLER_CONTAINER:-ci-fleet-controller-1}"
+      [[ -z "${FAKE_UNRELATED_CONTAINER:-}" ]] || printf '%s\n' "$FAKE_UNRELATED_CONTAINER"
+    elif [[ "$args" == *'label=ci-fleet.repository'* ]]; then
+      for ((i=0; i<${FAKE_ACTIVE_JOBS:-0}; i++)); do printf 'job-%s\n' "$i"; done
+    elif [[ "$args" == *'io.randomdevelopment.ci-fleet.managed=true'* ]]; then
+      for ((i=0; i<${FAKE_MANAGED_RUNNERS:-0}; i++)); do printf 'runner-%s\n' "$i"; done
+    fi
+    ;;
+  *) exit 1 ;;
+esac
+EOF
+
+cat >"$tmp/bin/stat" <<'EOF'
+#!/usr/bin/env bash
+case "${2:-}" in
+  %g) printf '%s\n' "${CI_FLEET_DOCKER_GID:-999}" ;;
+  %a) printf '600\n' ;;
+  %u) printf '0\n' ;;
+  *) exit 1 ;;
+esac
+EOF
+
+cat >"$tmp/bin/df" <<'EOF'
+#!/usr/bin/env bash
+printf 'Filesystem 1024-blocks Used Available Capacity Mounted on\n'
+printf '/dev/test 100000 5000 95000 %s%% /var/lib/docker\n' "${FAKE_DISK_USED:-5}"
+EOF
+
+cat >"$tmp/bin/getconf" <<'EOF'
+#!/usr/bin/env bash
+printf '%s\n' "${FAKE_TOTAL_CPUS:-16}"
+EOF
+
+cat >"$tmp/bin/free" <<'EOF'
+#!/usr/bin/env bash
+total=${FAKE_TOTAL_MEMORY_MIB:-32768}
+available=${FAKE_AVAILABLE_MEMORY_MIB:-30000}
+printf '              total        used        free      shared  buff/cache   available\n'
+printf 'Mem:          %s        1000        1000           0       1000       %s\n' "$total" "$available"
+EOF
+
+cat >"$tmp/bin/dmesg" <<'EOF'
+#!/usr/bin/env bash
+[[ "${FAKE_OOM_EVIDENCE:-0}" == 0 ]] || printf 'Out of memory: Killed process 123\n'
+EOF
+chmod 700 "$tmp/bin"/*
+
+export PATH="$tmp/bin:$PATH"
+export CI_FLEET_GITHUB_URL=https://github.com/EXAMPLE-ORG
+export CI_FLEET_SCALE_SET_NAME=docker-ci-validation
+export CI_FLEET_LABELS=docker-ci-experimental
+export CI_FLEET_RUNNER_GROUP=trusted-private-ci-experimental
+export CI_FLEET_INSTANCE=validation
+export CI_FLEET_GITHUB_APP_CLIENT_ID=validation
+export CI_FLEET_GITHUB_APP_INSTALLATION_ID=1
+export CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE="$tmp/github-app.pem"
+export CI_FLEET_DOCKER_GID=999
+export CI_FLEET_MIN_RUNNERS=0
+export CI_FLEET_MAX_RUNNERS=1
+export CI_FLEET_RUNNER_CPUS=4
+export CI_FLEET_RUNNER_MEMORY_MIB=8192
+export CI_FLEET_CONTROLLER_CONTAINER=ci-fleet-controller-1
+export CI_FLEET_DISK_WARN_PERCENT=80
+printf 'fixture only\n' >"$CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE"
+chmod 600 "$CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE"
+
+fail() { printf 'FAIL %s\n' "$*" >&2; exit 1; }
+expect_success() {
+  local output
+  output=$("$@" 2>&1) || fail "expected success: $*; output=$output"
+  printf '%s\n' "$output"
+}
+expect_failure() {
+  local expected=$1 output
+  shift
+  if output=$("$@" 2>&1); then fail "expected failure: $*"; fi
+  grep -Fq -- "$expected" <<<"$output" || fail "missing failure [$expected]: $output"
+}
+reset_fixture() {
+  export CI_FLEET_MIN_RUNNERS=0 CI_FLEET_MAX_RUNNERS=1
+  export FAKE_EFFECTIVE_MIN=0 FAKE_EFFECTIVE_MAX=1
+  export FAKE_TOTAL_CPUS=16 FAKE_TOTAL_MEMORY_MIB=32768 FAKE_AVAILABLE_MEMORY_MIB=30000 FAKE_DISK_USED=5
+  export FAKE_MANAGED_RUNNERS=0 FAKE_ACTIVE_JOBS=0 FAKE_CONTROLLER_STATE=running FAKE_CONTROLLER_OOM=false
+  export FAKE_OOM_EVIDENCE=0 FAKE_UNRELATED_CONTAINER=
+}
+
+reset_fixture
+pilot=$(expect_success "$repo_root/scripts/preflight.sh")
+grep -Fq 'PREFLIGHT_OK warnings=0' <<<"$pilot" || fail 'pilot MAX=1 gate did not pass'
+CI_FLEET_MAX_RUNNERS=2 expect_failure 'CI_FLEET_MAX_RUNNERS must be 1 for the pilot' "$repo_root/scripts/preflight.sh"
+
+valid=$(expect_success "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2)
+grep -Fq 'CAPACITY_PREFLIGHT_OK phase=pre-change target_max=2 configured_max=1 effective_max=1' <<<"$valid" || fail 'valid MAX=2 pre-change summary missing'
+grep -Fq 'cpu_budget_millicpus=' <<<"$valid" || fail 'safe CPU budget missing'
+grep -Fq 'memory_budget_mib=' <<<"$valid" || fail 'safe memory budget missing'
+
+reset_fixture
+FAKE_TOTAL_CPUS=10 expect_failure 'CPU capacity is insufficient' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_TOTAL_MEMORY_MIB=20000 FAKE_AVAILABLE_MEMORY_MIB=18000 expect_failure 'memory capacity is insufficient' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_DISK_USED=80 expect_failure 'Docker filesystem must remain below 80%' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_MANAGED_RUNNERS=1 expect_failure 'active managed runner' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_ACTIVE_JOBS=1 expect_failure 'active fleet job container' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+CI_FLEET_MIN_RUNNERS=1 FAKE_EFFECTIVE_MIN=1 expect_failure 'CI_FLEET_MIN_RUNNERS must be 0' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_UNRELATED_CONTAINER=database expect_failure 'unrelated running Docker workload' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_OOM_EVIDENCE=1 expect_failure 'OOM evidence exists for the current boot' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+
+reset_fixture
+expect_failure 'explicit --target-max is required' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change
+for malformed in 0 -1 2x 1.5; do
+  expect_failure 'target MAX must be a positive integer' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max "$malformed"
+done
+
+reset_fixture
+CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2 expect_success "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 >/dev/null
+reset_fixture
+CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=1 expect_failure 'effective controller MAX does not match requested target' "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2
+reset_fixture
+CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2
+secret_output=$("$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 2>&1) || fail 'secret-output fixture unexpectedly failed'
+if grep -Fq 'CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT' <<<"$secret_output"; then fail 'capacity preflight printed an unrelated environment value'; fi
+
+for term in backup force-recreate --no-deps healthcheck retain restore rollback; do
+  grep -Fqi -- "$term" "$repo_root/docs/CAPACITY-PROMOTION.md" || fail "capacity procedure is missing $term"
+done
+grep -Fq 'scripts/capacity-preflight.sh' "$repo_root/docs/ADDING-A-HOST.md" || fail 'host guide does not link the capacity procedure'
+if grep -Riq --exclude='test-capacity-preflight.sh' 'docker system prune' "$repo_root/scripts"; then fail 'unrestricted prune exists in scripts'; fi
+
+printf 'Capacity preflight tests passed.\n'
diff --git a/scripts/validate.sh b/scripts/validate.sh
index 7710279b..9628d645 100755
--- a/scripts/validate.sh
+++ b/scripts/validate.sh
@@ -9,6 +9,7 @@ python3 -m py_compile .github/actions/plan/plan.py .github/actions/plan/test_pla
 python3 .github/actions/plan/test_plan.py
 python3 .github/actions/plan/plan.py --plan examples/project/scripts/ci/plan.json --group fast >/dev/null
 python3 .github/actions/plan/plan.py --plan examples/project/scripts/ci/plan.json --group full >/dev/null
+scripts/test-capacity-preflight.sh
 
 tmp=$(mktemp)
 trap 'rm -f "$tmp"' EXIT

From dd8043dac370cc73412835308ef7bf3f89775e16 Mon Sep 17 00:00:00 2001
From: Nickfosts Hermes <me@nickfost.com>
Date: Fri, 17 Jul 2026 15:41:52 -0500
Subject: [PATCH 2/6] fix(ops): close capacity promotion review gaps

---
 docs/CAPACITY-PROMOTION.md         | 20 +++++++++++++-------
 scripts/capacity-preflight.sh      | 10 +++++++---
 scripts/test-capacity-preflight.sh | 14 +++++++++++++-
 3 files changed, 33 insertions(+), 11 deletions(-)

diff --git a/docs/CAPACITY-PROMOTION.md b/docs/CAPACITY-PROMOTION.md
index 163e096d..293a23e8 100644
--- a/docs/CAPACITY-PROMOTION.md
+++ b/docs/CAPACITY-PROMOTION.md
@@ -85,23 +85,26 @@ grep -qx 'CI_FLEET_MAX_RUNNERS=2' /etc/ci-fleet/ci-fleet.env
 cmp -s \
   <(grep -v '^CI_FLEET_MAX_RUNNERS=' "$backup") \
   <(grep -v '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env)
-docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet
 ```
 
-Do not run the pilot preflight against MAX=2; it must continue to reject that value.
+Do not run the pilot preflight against MAX=2; it must continue to reject that value. Every Compose command below uses `env -i` so stale values exported when the pilot file was sourced cannot override the explicit `--env-file` during promotion or rollback.
 
 ## 5. Recreate only the controller
 
 Reconfirm zero runners and zero jobs immediately before stopping. Stop only the controller with enough grace for scale-set deletion:
 
 ```bash
-docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller
 ```
 
 Verify the exact old scale set is absent and there is no runner before starting the replacement. Do not delete an apparent duplicate until ownership and zero active jobs are proven.
 
 ```bash
-docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \
   up -d --no-deps --force-recreate --timeout 60 controller
 ```
 
@@ -133,11 +136,14 @@ Retain MAX=2 only when the authorized workload succeeds, actual two-way job and
 On any failure, keep dispatch gated, wait for exact job termination, and restore the backup:
 
 ```bash
-docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller
 install -o root -g root -m 0600 "$backup" /etc/ci-fleet/ci-fleet.env
 cmp -s "$backup" /etc/ci-fleet/ci-fleet.env
-docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet
-docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \
   up -d --no-deps --force-recreate --timeout 60 controller
 ```
 
diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh
index ebcfa942..73c4b9b6 100755
--- a/scripts/capacity-preflight.sh
+++ b/scripts/capacity-preflight.sh
@@ -108,7 +108,7 @@ controller_oom=$(docker inspect --format '{{.State.OOMKilled}}' "$controller" 2>
 
 controller_env=$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$controller" 2>/dev/null) || die 'effective controller configuration could not be inspected'
 effective_value() {
-  local wanted=$1 line key value= count=0
+  local wanted=$1 line key value='' count=0
   while IFS= read -r line; do
     key=${line%%=*}
     if [[ "$key" == "$wanted" ]]; then
@@ -151,8 +151,12 @@ managed_count=$(docker ps -aq \
   --filter label=io.randomdevelopment.ci-fleet.kind=runner \
   --filter "label=io.randomdevelopment.ci-fleet.instance=$CI_FLEET_INSTANCE" | wc -l | tr -d ' ')
 [[ "$managed_count" == 0 ]] || die 'active managed runner or runner residue exists'
-job_count=$(docker ps -aq --filter label=ci-fleet.repository | wc -l | tr -d ' ')
-[[ "$job_count" == 0 ]] || die 'active fleet job container or job residue exists'
+job_container_count=$(docker ps -aq --filter label=ci-fleet.repository | wc -l | tr -d ' ')
+[[ "$job_container_count" == 0 ]] || die 'active fleet job container or job residue exists'
+job_volume_count=$(docker volume ls -q --filter label=ci-fleet.repository | wc -l | tr -d ' ')
+[[ "$job_volume_count" == 0 ]] || die 'fleet job volume residue exists'
+job_network_count=$(docker network ls -q --filter label=ci-fleet.repository | wc -l | tr -d ' ')
+[[ "$job_network_count" == 0 ]] || die 'fleet job network residue exists'
 
 while IFS= read -r running_name; do
   [[ -z "$running_name" || "$running_name" == "$controller" ]] || die 'unrelated running Docker workload exists'
diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh
index 3320ac3c..577699af 100755
--- a/scripts/test-capacity-preflight.sh
+++ b/scripts/test-capacity-preflight.sh
@@ -51,6 +51,12 @@ case "${1:-}" in
       for ((i=0; i<${FAKE_MANAGED_RUNNERS:-0}; i++)); do printf 'runner-%s\n' "$i"; done
     fi
     ;;
+  volume|network)
+    kind=$1
+    [[ "${2:-}" == ls ]] || exit 1
+    if [[ "$kind" == volume ]]; then count=${FAKE_JOB_VOLUMES:-0}; else count=${FAKE_JOB_NETWORKS:-0}; fi
+    for ((i=0; i<count; i++)); do printf '%s-%s\n' "$kind" "$i"; done
+    ;;
   *) exit 1 ;;
 esac
 EOF
@@ -125,7 +131,8 @@ reset_fixture() {
   export CI_FLEET_MIN_RUNNERS=0 CI_FLEET_MAX_RUNNERS=1
   export FAKE_EFFECTIVE_MIN=0 FAKE_EFFECTIVE_MAX=1
   export FAKE_TOTAL_CPUS=16 FAKE_TOTAL_MEMORY_MIB=32768 FAKE_AVAILABLE_MEMORY_MIB=30000 FAKE_DISK_USED=5
-  export FAKE_MANAGED_RUNNERS=0 FAKE_ACTIVE_JOBS=0 FAKE_CONTROLLER_STATE=running FAKE_CONTROLLER_OOM=false
+  export FAKE_MANAGED_RUNNERS=0 FAKE_ACTIVE_JOBS=0 FAKE_JOB_VOLUMES=0 FAKE_JOB_NETWORKS=0
+  export FAKE_CONTROLLER_STATE=running FAKE_CONTROLLER_OOM=false
   export FAKE_OOM_EVIDENCE=0 FAKE_UNRELATED_CONTAINER=
 }
 
@@ -150,6 +157,10 @@ FAKE_MANAGED_RUNNERS=1 expect_failure 'active managed runner' "$repo_root/script
 reset_fixture
 FAKE_ACTIVE_JOBS=1 expect_failure 'active fleet job container' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
+FAKE_JOB_VOLUMES=1 expect_failure 'fleet job volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_JOB_NETWORKS=1 expect_failure 'fleet job network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
 CI_FLEET_MIN_RUNNERS=1 FAKE_EFFECTIVE_MIN=1 expect_failure 'CI_FLEET_MIN_RUNNERS must be 0' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
 FAKE_UNRELATED_CONTAINER=database expect_failure 'unrelated running Docker workload' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
@@ -174,6 +185,7 @@ if grep -Fq 'CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT' <<<"$secret_output"; then fa
 for term in backup force-recreate --no-deps healthcheck retain restore rollback; do
   grep -Fqi -- "$term" "$repo_root/docs/CAPACITY-PROMOTION.md" || fail "capacity procedure is missing $term"
 done
+grep -Fq 'env -i' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'capacity procedure does not isolate Compose interpolation from stale shell values'
 grep -Fq 'scripts/capacity-preflight.sh' "$repo_root/docs/ADDING-A-HOST.md" || fail 'host guide does not link the capacity procedure'
 if grep -Riq --exclude='test-capacity-preflight.sh' 'docker system prune' "$repo_root/scripts"; then fail 'unrestricted prune exists in scripts'; fi
 

From afab5e207d986c8fbbd4b77b91aebe032acda1a2 Mon Sep 17 00:00:00 2001
From: Nickfosts Hermes <me@nickfost.com>
Date: Fri, 17 Jul 2026 15:48:52 -0500
Subject: [PATCH 3/6] fix(ops): bound and redact capacity validation

---
 docs/CAPACITY-PROMOTION.md         | 2 +-
 scripts/capacity-preflight.sh      | 2 ++
 scripts/test-capacity-preflight.sh | 5 +++++
 3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/docs/CAPACITY-PROMOTION.md b/docs/CAPACITY-PROMOTION.md
index 293a23e8..8267483a 100644
--- a/docs/CAPACITY-PROMOTION.md
+++ b/docs/CAPACITY-PROMOTION.md
@@ -8,7 +8,7 @@ Use this procedure only after the strict one-runner pilot has passed. It validat
 
 Declare the policy before observing a larger workload:
 
-- requested MAX is explicit, positive, and bounded;
+- requested MAX is explicitly and exclusively two for this first post-pilot procedure; any other target is rejected;
 - `MIN` remains zero;
 - the configured instance, scale set, routing label, runner group, Docker socket group, and per-runner limits must equal the running controller's effective values;
 - no managed runner, project job resource, unrelated running container, controller OOM, or current-boot kernel OOM evidence may exist;
diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh
index 73c4b9b6..8402efa9 100755
--- a/scripts/capacity-preflight.sh
+++ b/scripts/capacity-preflight.sh
@@ -1,5 +1,6 @@
 #!/usr/bin/env bash
 set -Eeuo pipefail
+set +x
 
 phase=
 target_max=
@@ -41,6 +42,7 @@ done
 [[ -n "$target_max" ]] || die 'explicit --target-max is required'
 [[ "$target_max" =~ ^[1-9][0-9]{0,3}$ ]] || die 'target MAX must be a positive integer'
 target_max=$((10#$target_max))
+((target_max == 2)) || die 'target MAX must be exactly 2 for this post-pilot procedure'
 
 required=(
   CI_FLEET_GITHUB_URL
diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh
index 577699af..7906a2ce 100755
--- a/scripts/test-capacity-preflight.sh
+++ b/scripts/test-capacity-preflight.sh
@@ -172,6 +172,9 @@ expect_failure 'explicit --target-max is required' "$repo_root/scripts/capacity-
 for malformed in 0 -1 2x 1.5; do
   expect_failure 'target MAX must be a positive integer' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max "$malformed"
 done
+reset_fixture
+FAKE_TOTAL_CPUS=32 FAKE_TOTAL_MEMORY_MIB=65536 FAKE_AVAILABLE_MEMORY_MIB=60000 \
+  expect_failure 'target MAX must be exactly 2' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 3
 
 reset_fixture
 CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2 expect_success "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 >/dev/null
@@ -181,6 +184,8 @@ reset_fixture
 CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2
 secret_output=$("$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 2>&1) || fail 'secret-output fixture unexpectedly failed'
 if grep -Fq 'CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT' <<<"$secret_output"; then fail 'capacity preflight printed an unrelated environment value'; fi
+xtrace_output=$(bash -x "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 2>&1) || fail 'xtrace secret-output fixture unexpectedly failed'
+if grep -Fq 'CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT' <<<"$xtrace_output"; then fail 'capacity preflight exposed controller environment through inherited xtrace'; fi
 
 for term in backup force-recreate --no-deps healthcheck retain restore rollback; do
   grep -Fqi -- "$term" "$repo_root/docs/CAPACITY-PROMOTION.md" || fail "capacity procedure is missing $term"

From 109e0bed2a6da33acc5d1cf97e6a883a0097fbd2 Mon Sep 17 00:00:00 2001
From: Nickfosts Hermes <me@nickfost.com>
Date: Fri, 17 Jul 2026 15:59:08 -0500
Subject: [PATCH 4/6] fix(ops): reject generic Compose residue

---
 scripts/capacity-preflight.sh      | 21 +++++++++++++++++++++
 scripts/test-capacity-preflight.sh | 23 ++++++++++++++++++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh
index 8402efa9..3e0a0491 100755
--- a/scripts/capacity-preflight.sh
+++ b/scripts/capacity-preflight.sh
@@ -160,6 +160,27 @@ job_volume_count=$(docker volume ls -q --filter label=ci-fleet.repository | wc -
 job_network_count=$(docker network ls -q --filter label=ci-fleet.repository | wc -l | tr -d ' ')
 [[ "$job_network_count" == 0 ]] || die 'fleet job network residue exists'
 
+# Docker Compose always labels its resources even when a project omits optional
+# ci-fleet labels. Only the controller's committed Compose project may remain.
+compose_container_ids=$(docker ps -aq --filter label=com.docker.compose.project) || die 'Compose container residue could not be inspected'
+while IFS= read -r id; do
+  [[ -n "$id" ]] || continue
+  project=$(docker inspect --format '{{index .Config.Labels "com.docker.compose.project"}}' "$id" 2>/dev/null) || die 'Compose container identity could not be inspected'
+  [[ "$project" == ci-fleet ]] || die 'foreign Compose container residue exists'
+done <<<"$compose_container_ids"
+compose_volume_names=$(docker volume ls -q --filter label=com.docker.compose.project) || die 'Compose volume residue could not be inspected'
+while IFS= read -r name; do
+  [[ -n "$name" ]] || continue
+  project=$(docker volume inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose volume identity could not be inspected'
+  [[ "$project" == ci-fleet ]] || die 'foreign Compose volume residue exists'
+done <<<"$compose_volume_names"
+compose_network_names=$(docker network ls -q --filter label=com.docker.compose.project) || die 'Compose network residue could not be inspected'
+while IFS= read -r name; do
+  [[ -n "$name" ]] || continue
+  project=$(docker network inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose network identity could not be inspected'
+  [[ "$project" == ci-fleet ]] || die 'foreign Compose network residue exists'
+done <<<"$compose_network_names"
+
 while IFS= read -r running_name; do
   [[ -z "$running_name" || "$running_name" == "$controller" ]] || die 'unrelated running Docker workload exists'
 done < <(docker ps --format '{{.Names}}')
diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh
index 7906a2ce..a9a03e31 100755
--- a/scripts/test-capacity-preflight.sh
+++ b/scripts/test-capacity-preflight.sh
@@ -36,6 +36,8 @@ case "${1:-}" in
         "CI_FLEET_RUNNER_MEMORY_MIB=${CI_FLEET_RUNNER_MEMORY_MIB}" \
         "CI_FLEET_DOCKER_GID=${CI_FLEET_DOCKER_GID}" \
         "UNRELATED_SECRET=${FAKE_SECRET_VALUE:-CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT}"
+    elif [[ "$args" == *'com.docker.compose.project'* ]]; then
+      printf '%s\n' "${FAKE_COMPOSE_PROJECT:-ci-fleet}"
     else
       exit 1
     fi
@@ -47,14 +49,26 @@ case "${1:-}" in
       [[ -z "${FAKE_UNRELATED_CONTAINER:-}" ]] || printf '%s\n' "$FAKE_UNRELATED_CONTAINER"
     elif [[ "$args" == *'label=ci-fleet.repository'* ]]; then
       for ((i=0; i<${FAKE_ACTIVE_JOBS:-0}; i++)); do printf 'job-%s\n' "$i"; done
+    elif [[ "$args" == *'label=com.docker.compose.project'* ]]; then
+      for ((i=0; i<${FAKE_COMPOSE_CONTAINERS:-0}; i++)); do printf 'compose-container-%s\n' "$i"; done
     elif [[ "$args" == *'io.randomdevelopment.ci-fleet.managed=true'* ]]; then
       for ((i=0; i<${FAKE_MANAGED_RUNNERS:-0}; i++)); do printf 'runner-%s\n' "$i"; done
     fi
     ;;
   volume|network)
     kind=$1
+    if [[ "${2:-}" == inspect ]]; then
+      printf '%s\n' "${FAKE_COMPOSE_PROJECT:-ci-fleet}"
+      exit 0
+    fi
     [[ "${2:-}" == ls ]] || exit 1
-    if [[ "$kind" == volume ]]; then count=${FAKE_JOB_VOLUMES:-0}; else count=${FAKE_JOB_NETWORKS:-0}; fi
+    if [[ "$*" == *'label=com.docker.compose.project'* ]]; then
+      if [[ "$kind" == volume ]]; then count=${FAKE_COMPOSE_VOLUMES:-0}; else count=${FAKE_COMPOSE_NETWORKS:-0}; fi
+    elif [[ "$kind" == volume ]]; then
+      count=${FAKE_JOB_VOLUMES:-0}
+    else
+      count=${FAKE_JOB_NETWORKS:-0}
+    fi
     for ((i=0; i<count; i++)); do printf '%s-%s\n' "$kind" "$i"; done
     ;;
   *) exit 1 ;;
@@ -132,6 +146,7 @@ reset_fixture() {
   export FAKE_EFFECTIVE_MIN=0 FAKE_EFFECTIVE_MAX=1
   export FAKE_TOTAL_CPUS=16 FAKE_TOTAL_MEMORY_MIB=32768 FAKE_AVAILABLE_MEMORY_MIB=30000 FAKE_DISK_USED=5
   export FAKE_MANAGED_RUNNERS=0 FAKE_ACTIVE_JOBS=0 FAKE_JOB_VOLUMES=0 FAKE_JOB_NETWORKS=0
+  export FAKE_COMPOSE_CONTAINERS=0 FAKE_COMPOSE_VOLUMES=0 FAKE_COMPOSE_NETWORKS=0 FAKE_COMPOSE_PROJECT=ci-fleet
   export FAKE_CONTROLLER_STATE=running FAKE_CONTROLLER_OOM=false
   export FAKE_OOM_EVIDENCE=0 FAKE_UNRELATED_CONTAINER=
 }
@@ -161,6 +176,12 @@ FAKE_JOB_VOLUMES=1 expect_failure 'fleet job volume residue' "$repo_root/scripts
 reset_fixture
 FAKE_JOB_NETWORKS=1 expect_failure 'fleet job network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
+FAKE_COMPOSE_CONTAINERS=1 FAKE_COMPOSE_PROJECT=project-run expect_failure 'foreign Compose container residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_COMPOSE_VOLUMES=1 FAKE_COMPOSE_PROJECT=project-run expect_failure 'foreign Compose volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_COMPOSE_NETWORKS=1 FAKE_COMPOSE_PROJECT=project-run expect_failure 'foreign Compose network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
 CI_FLEET_MIN_RUNNERS=1 FAKE_EFFECTIVE_MIN=1 expect_failure 'CI_FLEET_MIN_RUNNERS must be 0' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
 FAKE_UNRELATED_CONTAINER=database expect_failure 'unrelated running Docker workload' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2

From e98bb1a203e86a18f4ba4b3bcd027d82c85ec5c2 Mon Sep 17 00:00:00 2001
From: Nickfosts Hermes <me@nickfost.com>
Date: Fri, 17 Jul 2026 16:10:08 -0500
Subject: [PATCH 5/6] fix(ops): narrow committed Compose exceptions

---
 scripts/capacity-preflight.sh      | 20 ++++++++++++++--
 scripts/test-capacity-preflight.sh | 38 +++++++++++++++++++++++++-----
 2 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh
index 3e0a0491..f1bf8ca7 100755
--- a/scripts/capacity-preflight.sh
+++ b/scripts/capacity-preflight.sh
@@ -163,23 +163,39 @@ job_network_count=$(docker network ls -q --filter label=ci-fleet.repository | wc
 # Docker Compose always labels its resources even when a project omits optional
 # ci-fleet labels. Only the controller's committed Compose project may remain.
 compose_container_ids=$(docker ps -aq --filter label=com.docker.compose.project) || die 'Compose container residue could not be inspected'
+compose_controller_count=0
 while IFS= read -r id; do
   [[ -n "$id" ]] || continue
-  project=$(docker inspect --format '{{index .Config.Labels "com.docker.compose.project"}}' "$id" 2>/dev/null) || die 'Compose container identity could not be inspected'
+  project=$(docker inspect --format '{{index .Config.Labels "com.docker.compose.project"}}' "$id" 2>/dev/null) || die 'Compose container project could not be inspected'
   [[ "$project" == ci-fleet ]] || die 'foreign Compose container residue exists'
+  service=$(docker inspect --format '{{index .Config.Labels "com.docker.compose.service"}}' "$id" 2>/dev/null) || die 'Compose container service could not be inspected'
+  resource_name=$(docker inspect --format '{{.Name}}' "$id" 2>/dev/null) || die 'Compose container name could not be inspected'
+  resource_name=${resource_name#/}
+  [[ "$service" == controller && "$resource_name" == "$controller" ]] || die 'unexpected Compose container residue exists'
+  compose_controller_count=$((compose_controller_count + 1))
 done <<<"$compose_container_ids"
+((compose_controller_count == 1)) || die 'committed controller Compose container is missing or duplicated'
+
 compose_volume_names=$(docker volume ls -q --filter label=com.docker.compose.project) || die 'Compose volume residue could not be inspected'
 while IFS= read -r name; do
   [[ -n "$name" ]] || continue
   project=$(docker volume inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose volume identity could not be inspected'
   [[ "$project" == ci-fleet ]] || die 'foreign Compose volume residue exists'
+  die 'unexpected Compose volume residue exists'
 done <<<"$compose_volume_names"
+
 compose_network_names=$(docker network ls -q --filter label=com.docker.compose.project) || die 'Compose network residue could not be inspected'
+compose_network_count=0
 while IFS= read -r name; do
   [[ -n "$name" ]] || continue
-  project=$(docker network inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose network identity could not be inspected'
+  project=$(docker network inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose network project could not be inspected'
   [[ "$project" == ci-fleet ]] || die 'foreign Compose network residue exists'
+  network_label=$(docker network inspect --format '{{index .Labels "com.docker.compose.network"}}' "$name" 2>/dev/null) || die 'Compose network label could not be inspected'
+  resource_name=$(docker network inspect --format '{{.Name}}' "$name" 2>/dev/null) || die 'Compose network name could not be inspected'
+  [[ "$network_label" == default && "$resource_name" == ci-fleet_default ]] || die 'unexpected Compose network residue exists'
+  compose_network_count=$((compose_network_count + 1))
 done <<<"$compose_network_names"
+((compose_network_count == 1)) || die 'committed controller Compose network is missing or duplicated'
 
 while IFS= read -r running_name; do
   [[ -z "$running_name" || "$running_name" == "$controller" ]] || die 'unrelated running Docker workload exists'
diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh
index a9a03e31..9c919b2a 100755
--- a/scripts/test-capacity-preflight.sh
+++ b/scripts/test-capacity-preflight.sh
@@ -37,7 +37,11 @@ case "${1:-}" in
         "CI_FLEET_DOCKER_GID=${CI_FLEET_DOCKER_GID}" \
         "UNRELATED_SECRET=${FAKE_SECRET_VALUE:-CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT}"
     elif [[ "$args" == *'com.docker.compose.project'* ]]; then
-      printf '%s\n' "${FAKE_COMPOSE_PROJECT:-ci-fleet}"
+      printf '%s\n' "${FAKE_COMPOSE_CONTAINER_PROJECT:-${FAKE_COMPOSE_PROJECT:-ci-fleet}}"
+    elif [[ "$args" == *'com.docker.compose.service'* ]]; then
+      printf '%s\n' "${FAKE_COMPOSE_SERVICE:-controller}"
+    elif [[ "$args" == *'{{.Name}}'* ]]; then
+      printf '/%s\n' "${FAKE_COMPOSE_CONTAINER_NAME:-ci-fleet-controller-1}"
     else
       exit 1
     fi
@@ -58,7 +62,20 @@ case "${1:-}" in
   volume|network)
     kind=$1
     if [[ "${2:-}" == inspect ]]; then
-      printf '%s\n' "${FAKE_COMPOSE_PROJECT:-ci-fleet}"
+      args="$*"
+      if [[ "$args" == *'com.docker.compose.project'* ]]; then
+        if [[ "$kind" == volume ]]; then
+          printf '%s\n' "${FAKE_COMPOSE_VOLUME_PROJECT:-${FAKE_COMPOSE_PROJECT:-ci-fleet}}"
+        else
+          printf '%s\n' "${FAKE_COMPOSE_NETWORK_PROJECT:-${FAKE_COMPOSE_PROJECT:-ci-fleet}}"
+        fi
+      elif [[ "$args" == *'com.docker.compose.network'* ]]; then
+        printf '%s\n' "${FAKE_COMPOSE_NETWORK_LABEL:-default}"
+      elif [[ "$args" == *'{{.Name}}'* ]]; then
+        printf '%s\n' "${FAKE_COMPOSE_NETWORK_NAME:-ci-fleet_default}"
+      else
+        exit 1
+      fi
       exit 0
     fi
     [[ "${2:-}" == ls ]] || exit 1
@@ -146,7 +163,10 @@ reset_fixture() {
   export FAKE_EFFECTIVE_MIN=0 FAKE_EFFECTIVE_MAX=1
   export FAKE_TOTAL_CPUS=16 FAKE_TOTAL_MEMORY_MIB=32768 FAKE_AVAILABLE_MEMORY_MIB=30000 FAKE_DISK_USED=5
   export FAKE_MANAGED_RUNNERS=0 FAKE_ACTIVE_JOBS=0 FAKE_JOB_VOLUMES=0 FAKE_JOB_NETWORKS=0
-  export FAKE_COMPOSE_CONTAINERS=0 FAKE_COMPOSE_VOLUMES=0 FAKE_COMPOSE_NETWORKS=0 FAKE_COMPOSE_PROJECT=ci-fleet
+  export FAKE_COMPOSE_CONTAINERS=1 FAKE_COMPOSE_VOLUMES=0 FAKE_COMPOSE_NETWORKS=1 FAKE_COMPOSE_PROJECT=ci-fleet
+  export FAKE_COMPOSE_CONTAINER_PROJECT=ci-fleet FAKE_COMPOSE_VOLUME_PROJECT=ci-fleet FAKE_COMPOSE_NETWORK_PROJECT=ci-fleet
+  export FAKE_COMPOSE_SERVICE=controller FAKE_COMPOSE_CONTAINER_NAME=ci-fleet-controller-1
+  export FAKE_COMPOSE_NETWORK_LABEL=default FAKE_COMPOSE_NETWORK_NAME=ci-fleet_default
   export FAKE_CONTROLLER_STATE=running FAKE_CONTROLLER_OOM=false
   export FAKE_OOM_EVIDENCE=0 FAKE_UNRELATED_CONTAINER=
 }
@@ -176,11 +196,17 @@ FAKE_JOB_VOLUMES=1 expect_failure 'fleet job volume residue' "$repo_root/scripts
 reset_fixture
 FAKE_JOB_NETWORKS=1 expect_failure 'fleet job network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
-FAKE_COMPOSE_CONTAINERS=1 FAKE_COMPOSE_PROJECT=project-run expect_failure 'foreign Compose container residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+FAKE_COMPOSE_NETWORKS=0 FAKE_COMPOSE_CONTAINER_PROJECT=project-run expect_failure 'foreign Compose container residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
-FAKE_COMPOSE_VOLUMES=1 FAKE_COMPOSE_PROJECT=project-run expect_failure 'foreign Compose volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+FAKE_COMPOSE_VOLUMES=1 FAKE_COMPOSE_NETWORKS=0 FAKE_COMPOSE_VOLUME_PROJECT=project-run expect_failure 'foreign Compose volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
-FAKE_COMPOSE_NETWORKS=1 FAKE_COMPOSE_PROJECT=project-run expect_failure 'foreign Compose network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+FAKE_COMPOSE_NETWORK_PROJECT=project-run expect_failure 'foreign Compose network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_COMPOSE_CONTAINER_NAME=rogue FAKE_COMPOSE_SERVICE=task expect_failure 'unexpected Compose container residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_COMPOSE_VOLUMES=1 expect_failure 'unexpected Compose volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
+reset_fixture
+FAKE_COMPOSE_NETWORK_NAME=ci-fleet_extra expect_failure 'unexpected Compose network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture
 CI_FLEET_MIN_RUNNERS=1 FAKE_EFFECTIVE_MIN=1 expect_failure 'CI_FLEET_MIN_RUNNERS must be 0' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2
 reset_fixture

From 115101a019760a94470902b8118a13504fdd864a Mon Sep 17 00:00:00 2001
From: Nickfosts Hermes <me@nickfost.com>
Date: Fri, 17 Jul 2026 16:17:30 -0500
Subject: [PATCH 6/6] fix(ops): reload restored pilot environment

---
 docs/CAPACITY-PROMOTION.md         | 11 ++++++++++-
 scripts/test-capacity-preflight.sh |  1 +
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/docs/CAPACITY-PROMOTION.md b/docs/CAPACITY-PROMOTION.md
index 8267483a..8d1bc507 100644
--- a/docs/CAPACITY-PROMOTION.md
+++ b/docs/CAPACITY-PROMOTION.md
@@ -147,4 +147,13 @@ env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/r
   up -d --no-deps --force-recreate --timeout 60 controller
 ```
 
-Verify a sanitized ready record with `MIN=0`, `MAX=1`, then run `scripts/preflight.sh`, `scripts/healthcheck.sh`, and the instance-scoped cleanup dry-run. Require zero runners, zero jobs, one intended scale set, no duplicate controller, and no residue before reopening dispatch.
+Verify a sanitized ready record with `MIN=0`, `MAX=1`, then run the pilot preflight and healthcheck from clean processes that source only the restored file:
+
+```bash
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  bash -c 'set -a; . /etc/ci-fleet/ci-fleet.env; set +a; exec scripts/preflight.sh'
+env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \
+  bash -c 'set -a; . /etc/ci-fleet/ci-fleet.env; set +a; exec scripts/healthcheck.sh'
+```
+
+Run the instance-scoped cleanup dry-run. Require zero runners, zero jobs, one intended scale set, no duplicate controller, and no residue before reopening dispatch.
diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh
index 9c919b2a..12de1e91 100755
--- a/scripts/test-capacity-preflight.sh
+++ b/scripts/test-capacity-preflight.sh
@@ -238,6 +238,7 @@ for term in backup force-recreate --no-deps healthcheck retain restore rollback;
   grep -Fqi -- "$term" "$repo_root/docs/CAPACITY-PROMOTION.md" || fail "capacity procedure is missing $term"
 done
 grep -Fq 'env -i' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'capacity procedure does not isolate Compose interpolation from stale shell values'
+grep -Fq 'exec scripts/preflight.sh' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'rollback does not run pilot preflight from a clean restored environment'
 grep -Fq 'scripts/capacity-preflight.sh' "$repo_root/docs/ADDING-A-HOST.md" || fail 'host guide does not link the capacity procedure'
 if grep -Riq --exclude='test-capacity-preflight.sh' 'docker system prune' "$repo_root/scripts"; then fail 'unrestricted prune exists in scripts'; fi