diff --git a/docs/ADDING-A-HOST.md b/docs/ADDING-A-HOST.md index 6148dab9..3e42e1f5 100644 --- a/docs/ADDING-A-HOST.md +++ b/docs/ADDING-A-HOST.md @@ -151,6 +151,8 @@ The steady-state host should need no project-specific edits. Adding a project ch Raise `CI_FLEET_MAX_RUNNERS` only after measuring CPU, memory, disk, network, cache growth, collisions, cancellation, and cleanup. Keep explicit per-runner limits. +Keep the one-runner pilot preflight unchanged. For any later increase, follow [Post-pilot capacity promotion](CAPACITY-PROMOTION.md), run `scripts/capacity-preflight.sh` before and after the controller-only recreation, and retain the new maximum only when the separately authorized workload and cleanup proof satisfy the predeclared resource policy. + Use another host or location when it improves failure tolerance. Give every added host a new instance and scale-set name while retaining the shared routing label. ## 10. Drain, replace, or remove a host diff --git a/docs/CAPACITY-PROMOTION.md b/docs/CAPACITY-PROMOTION.md new file mode 100644 index 00000000..8d1bc507 --- /dev/null +++ b/docs/CAPACITY-PROMOTION.md @@ -0,0 +1,159 @@ +# Post-pilot capacity promotion + +Use this procedure only after the strict one-runner pilot has passed. It validates and changes one already isolated controller; it does not authorize a project workflow by itself. + +`scripts/preflight.sh` remains the initial pilot gate and accepts only `MIN=0`, `MAX=1`. `scripts/capacity-preflight.sh` is a separate, read-only post-pilot gate. It never edits host configuration. + +## Capacity policy + +Declare the policy before observing a larger workload: + +- requested MAX is explicitly and exclusively two for this first post-pilot procedure; any other target is rejected; +- `MIN` remains zero; +- the configured instance, scale set, routing label, runner group, Docker socket group, and per-runner limits must equal the running controller's effective values; +- no managed runner, project job resource, unrelated running container, controller OOM, or current-boot kernel OOM evidence may exist; +- Docker filesystem use must be below `CI_FLEET_DISK_WARN_PERCENT` (80 by default); +- reserve the controller's Compose limit: 1 CPU and 512 MiB; +- reserve 1 CPU and 1 GiB for Docker overhead; +- reserve for the operating system the greater of 1 CPU or 15% of logical CPU capacity; +- reserve for the operating system the greater of 2 GiB or 20% of physical memory; +- admit `target MAX × CI_FLEET_RUNNER_CPUS` and `target MAX × CI_FLEET_RUNNER_MEMORY_MIB` only when those allocations plus all reserves fit; +- require currently available memory to cover all target runners plus controller and Docker reservations. + +Runner limits are controller admission inputs. Project containers use the host Docker daemon as siblings of the runner container, so the separately authorized live proof must still observe whole-host CPU, memory, disk, collision, and cleanup behavior. + +During a live proof, retain the target only if every five-second sample keeps CPU busy below 85%, available memory at or above the greater of 2 GiB or 20% of total memory, and Docker filesystem use below 80%. Any OOM, unrelated workload, controller/Docker failure, third runner, observer gap, or cleanup residue requires restoration. + +## 1. Verify idle pilot state + +Gate all dispatches that can target this controller. Confirm no queued, assigned, or running fleet job and no instance-owned runner in any state. Keep the dispatch gate closed until post-change verification completes. + +From the reviewed checkout, load the root-only host configuration without tracing or printing it: + +```bash +set -Eeuo pipefail +set +x +cd /opt/ci-fleet +set -a +. /etc/ci-fleet/ci-fleet.env +set +a +scripts/preflight.sh +``` + +Require exactly `PREFLIGHT_OK warnings=0`. This proves the original `MIN=0`, `MAX=1` pilot contract remains valid. + +## 2. Validate target capacity without changing it + +```bash +scripts/capacity-preflight.sh --phase pre-change --target-max 2 +``` + +Require `CAPACITY_PREFLIGHT_OK phase=pre-change target_max=2 configured_max=1 effective_max=1`. Record only the safe budget summaries. + +## 3. Create one protected backup + +Use one UTC timestamp and refuse to overwrite an existing path: + +```bash +backup_dir=/etc/ci-fleet/backups +stamp=$(date -u +%Y%m%dT%H%M%SZ) +backup="$backup_dir/ci-fleet.env.before-max2.$stamp" +install -d -o root -g root -m 0700 "$backup_dir" +test ! -e "$backup" +install -o root -g root -m 0600 /etc/ci-fleet/ci-fleet.env "$backup" +cmp -s /etc/ci-fleet/ci-fleet.env "$backup" +sha256sum "$backup" | cut -d' ' -f1 +``` + +Record the path and checksum only. Never print or copy the backup contents, and do not delete the backup during the proof. + +## 4. Change one exact setting + +Before editing, require exactly one active assignment and the pilot value: + +```bash +test "$(grep -c '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env)" -eq 1 +grep -qx 'CI_FLEET_MAX_RUNNERS=1' /etc/ci-fleet/ci-fleet.env +``` + +Edit only that line to `CI_FLEET_MAX_RUNNERS=2`. Then verify without printing a diff: + +```bash +test "$(stat -c '%u:%g:%a' /etc/ci-fleet/ci-fleet.env)" = 0:0:600 +test "$(grep -c '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env)" -eq 1 +grep -qx 'CI_FLEET_MAX_RUNNERS=2' /etc/ci-fleet/ci-fleet.env +cmp -s \ + <(grep -v '^CI_FLEET_MAX_RUNNERS=' "$backup") \ + <(grep -v '^CI_FLEET_MAX_RUNNERS=' /etc/ci-fleet/ci-fleet.env) +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet +``` + +Do not run the pilot preflight against MAX=2; it must continue to reject that value. Every Compose command below uses `env -i` so stale values exported when the pilot file was sourced cannot override the explicit `--env-file` during promotion or rollback. + +## 5. Recreate only the controller + +Reconfirm zero runners and zero jobs immediately before stopping. Stop only the controller with enough grace for scale-set deletion: + +```bash +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller +``` + +Verify the exact old scale set is absent and there is no runner before starting the replacement. Do not delete an apparent duplicate until ownership and zero active jobs are proven. + +```bash +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \ + up -d --no-deps --force-recreate --timeout 60 controller +``` + +Do not recreate `runner-image`, remove volumes, rebuild images, or touch unrelated resources. + +## 6. Verify effective state + +Wait boundedly for one sanitized `controller ready` record reporting `minRunners=0` and `maxRunners=2`. Require one running controller, restart count zero, one intended scale set, the unchanged experimental routing label and runner group, and no idle runner. + +Reload the host configuration and run: + +```bash +scripts/capacity-preflight.sh --phase post-change --target-max 2 +scripts/healthcheck.sh +``` + +Require both to pass. The post-change preflight compares configured state with a filtered set of effective controller values and never prints arbitrary container environment entries. + +## 7. Run one separately authorized proof + +Start bounded runner, task-job, project-resource, and host-metric observers before dispatch. Dispatch exactly the approved workload once. Do not retry a failed proof and do not raise MAX above two. + +Observe runner creation/destruction, actual two-way overlap, no third runner, whole-host CPU/memory/disk thresholds, Docker/controller health, exact project identity, and automatic cleanup. Run the instance-scoped cleanup dry-run and healthcheck after all jobs terminate. + +## 8. Retain or restore (rollback) + +Retain MAX=2 only when the authorized workload succeeds, actual two-way job and runner overlap is proven, every predeclared resource threshold passes, no manual cleanup is required, all runner/project residue is zero, post-change capacity preflight passes, cleanup dry-run is empty, and healthcheck passes. + +On any failure, keep dispatch gated, wait for exact job termination, and restore the backup: + +```bash +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml stop -t 60 controller +install -o root -g root -m 0600 "$backup" /etc/ci-fleet/ci-fleet.env +cmp -s "$backup" /etc/ci-fleet/ci-fleet.env +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml config --quiet +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + docker compose --env-file /etc/ci-fleet/ci-fleet.env -f deploy/compose.yaml \ + up -d --no-deps --force-recreate --timeout 60 controller +``` + +Verify a sanitized ready record with `MIN=0`, `MAX=1`, then run the pilot preflight and healthcheck from clean processes that source only the restored file: + +```bash +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + bash -c 'set -a; . /etc/ci-fleet/ci-fleet.env; set +a; exec scripts/preflight.sh' +env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root \ + bash -c 'set -a; . /etc/ci-fleet/ci-fleet.env; set +a; exec scripts/healthcheck.sh' +``` + +Run the instance-scoped cleanup dry-run. Require zero runners, zero jobs, one intended scale set, no duplicate controller, and no residue before reopening dispatch. diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh new file mode 100755 index 00000000..f1bf8ca7 --- /dev/null +++ b/scripts/capacity-preflight.sh @@ -0,0 +1,246 @@ +#!/usr/bin/env bash +set -Eeuo pipefail +set +x + +phase= +target_max= + +usage() { + printf 'usage: %s --phase pre-change|post-change --target-max N\n' "$0" >&2 +} + +die() { + printf 'FAIL %s\n' "$1" >&2 + printf 'CAPACITY_PREFLIGHT_FAILED phase=%s\n' "${phase:-unknown}" >&2 + exit 2 +} + +while (($#)); do + case "$1" in + --phase) + (($# >= 2)) || die '--phase requires a value' + phase=$2 + shift 2 + ;; + --target-max) + (($# >= 2)) || die '--target-max requires a value' + target_max=$2 + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + usage + die 'unknown capacity-preflight argument' + ;; + esac +done + +[[ "$phase" == pre-change || "$phase" == post-change ]] || die 'explicit --phase pre-change or post-change is required' +[[ -n "$target_max" ]] || die 'explicit --target-max is required' +[[ "$target_max" =~ ^[1-9][0-9]{0,3}$ ]] || die 'target MAX must be a positive integer' +target_max=$((10#$target_max)) +((target_max == 2)) || die 'target MAX must be exactly 2 for this post-pilot procedure' + +required=( + CI_FLEET_GITHUB_URL + CI_FLEET_SCALE_SET_NAME + CI_FLEET_LABELS + CI_FLEET_RUNNER_GROUP + CI_FLEET_INSTANCE + CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE + CI_FLEET_DOCKER_GID + CI_FLEET_MIN_RUNNERS + CI_FLEET_MAX_RUNNERS + CI_FLEET_RUNNER_CPUS + CI_FLEET_RUNNER_MEMORY_MIB +) +for name in "${required[@]}"; do + [[ -n "${!name:-}" ]] || die "$name is required for capacity validation" +done + +[[ "$CI_FLEET_GITHUB_URL" == https://github.com/* ]] || die 'CI_FLEET_GITHUB_URL must use https://github.com' +[[ "$CI_FLEET_SCALE_SET_NAME" == *"$CI_FLEET_INSTANCE"* ]] || die 'scale-set name must retain the stable fleet instance ID' +[[ ",$CI_FLEET_LABELS," != *,self-hosted,* ]] || die 'shared routing labels must not include self-hosted' +[[ "$CI_FLEET_RUNNER_GROUP" != Default ]] || die 'post-pilot capacity requires the existing non-default runner group' +[[ "$CI_FLEET_MIN_RUNNERS" == 0 ]] || die 'CI_FLEET_MIN_RUNNERS must be 0' +[[ "$CI_FLEET_MAX_RUNNERS" =~ ^[1-9][0-9]{0,3}$ ]] || die 'configured MAX must be a positive integer' +configured_max=$((10#$CI_FLEET_MAX_RUNNERS)) +[[ "$CI_FLEET_RUNNER_CPUS" =~ ^[1-9][0-9]{0,5}$ ]] || die 'per-runner CPU limit must be a positive integer' +runner_cpus=$((10#$CI_FLEET_RUNNER_CPUS)) +[[ "$CI_FLEET_RUNNER_MEMORY_MIB" =~ ^[1-9][0-9]{0,9}$ ]] || die 'per-runner memory limit must be a positive integer MiB value' +runner_memory_mib=$((10#$CI_FLEET_RUNNER_MEMORY_MIB)) + +if [[ "$phase" == pre-change ]]; then + ((target_max > configured_max)) || die 'pre-change target MAX must be greater than configured MAX' +else + ((configured_max == target_max)) || die 'post-change configured MAX does not match requested target' +fi + +for command in docker stat df awk getconf free dmesg grep wc tr; do + command -v "$command" >/dev/null || die "$command is unavailable" +done +docker info >/dev/null 2>&1 || die 'Docker daemon is unreachable' +docker compose version >/dev/null 2>&1 || die 'Docker Compose plugin is unavailable' + +socket_gid=$(stat -c '%g' /var/run/docker.sock 2>/dev/null || true) +[[ -n "$socket_gid" && "$socket_gid" == "$CI_FLEET_DOCKER_GID" ]] || die 'Docker socket group does not match CI_FLEET_DOCKER_GID' + +secret=$CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE +[[ -f "$secret" ]] || die 'GitHub App PEM file is missing' +secret_mode=$(stat -c '%a' "$secret" 2>/dev/null || true) +secret_owner=$(stat -c '%u' "$secret" 2>/dev/null || true) +[[ "$secret_mode" == 600 && "$secret_owner" == 0 ]] || die 'GitHub App PEM must be root-owned mode 0600' + +warn_percent=${CI_FLEET_DISK_WARN_PERCENT:-80} +[[ "$warn_percent" =~ ^[1-9][0-9]?$|^100$ ]] || die 'CI_FLEET_DISK_WARN_PERCENT must be an integer from 1 through 100' +docker_root=$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true) +[[ "$docker_root" == /* ]] || die 'Docker root directory could not be determined' +disk_used=$(df -P "$docker_root" 2>/dev/null | awk 'NR==2 {gsub(/%/, "", $5); print $5}') +[[ "$disk_used" =~ ^[0-9]{1,3}$ ]] || die 'Docker filesystem utilization could not be determined' +((disk_used < warn_percent)) || die "Docker filesystem must remain below ${warn_percent}%" + +controller=${CI_FLEET_CONTROLLER_CONTAINER:-ci-fleet-controller-1} +controller_state=$(docker inspect --format '{{.State.Status}}' "$controller" 2>/dev/null || true) +[[ "$controller_state" == running ]] || die 'controller must be running and healthy' +controller_oom=$(docker inspect --format '{{.State.OOMKilled}}' "$controller" 2>/dev/null || true) +[[ "$controller_oom" == false ]] || die 'controller OOM evidence is present' + +controller_env=$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$controller" 2>/dev/null) || die 'effective controller configuration could not be inspected' +effective_value() { + local wanted=$1 line key value='' count=0 + while IFS= read -r line; do + key=${line%%=*} + if [[ "$key" == "$wanted" ]]; then + value=${line#*=} + count=$((count + 1)) + fi + done <<<"$controller_env" + ((count == 1)) || die "effective controller $wanted must occur exactly once" + printf '%s' "$value" +} + +stable=( + CI_FLEET_GITHUB_URL + CI_FLEET_SCALE_SET_NAME + CI_FLEET_LABELS + CI_FLEET_RUNNER_GROUP + CI_FLEET_INSTANCE + CI_FLEET_MIN_RUNNERS + CI_FLEET_RUNNER_CPUS + CI_FLEET_RUNNER_MEMORY_MIB + CI_FLEET_DOCKER_GID +) +for name in "${stable[@]}"; do + effective=$(effective_value "$name") + [[ "$effective" == "${!name}" ]] || die "effective controller $name does not match configured state" +done +effective_min=$(effective_value CI_FLEET_MIN_RUNNERS) +effective_max=$(effective_value CI_FLEET_MAX_RUNNERS) +[[ "$effective_min" == 0 ]] || die 'effective controller MIN must be 0' +[[ "$effective_max" =~ ^[1-9][0-9]{0,3}$ ]] || die 'effective controller MAX must be a positive integer' +effective_max=$((10#$effective_max)) +if [[ "$phase" == pre-change ]]; then + ((effective_max == configured_max)) || die 'effective controller MAX does not match configured pre-change MAX' +else + ((effective_max == target_max)) || die 'effective controller MAX does not match requested target' +fi + +managed_count=$(docker ps -aq \ + --filter label=io.randomdevelopment.ci-fleet.managed=true \ + --filter label=io.randomdevelopment.ci-fleet.kind=runner \ + --filter "label=io.randomdevelopment.ci-fleet.instance=$CI_FLEET_INSTANCE" | wc -l | tr -d ' ') +[[ "$managed_count" == 0 ]] || die 'active managed runner or runner residue exists' +job_container_count=$(docker ps -aq --filter label=ci-fleet.repository | wc -l | tr -d ' ') +[[ "$job_container_count" == 0 ]] || die 'active fleet job container or job residue exists' +job_volume_count=$(docker volume ls -q --filter label=ci-fleet.repository | wc -l | tr -d ' ') +[[ "$job_volume_count" == 0 ]] || die 'fleet job volume residue exists' +job_network_count=$(docker network ls -q --filter label=ci-fleet.repository | wc -l | tr -d ' ') +[[ "$job_network_count" == 0 ]] || die 'fleet job network residue exists' + +# Docker Compose always labels its resources even when a project omits optional +# ci-fleet labels. Only the controller's committed Compose project may remain. +compose_container_ids=$(docker ps -aq --filter label=com.docker.compose.project) || die 'Compose container residue could not be inspected' +compose_controller_count=0 +while IFS= read -r id; do + [[ -n "$id" ]] || continue + project=$(docker inspect --format '{{index .Config.Labels "com.docker.compose.project"}}' "$id" 2>/dev/null) || die 'Compose container project could not be inspected' + [[ "$project" == ci-fleet ]] || die 'foreign Compose container residue exists' + service=$(docker inspect --format '{{index .Config.Labels "com.docker.compose.service"}}' "$id" 2>/dev/null) || die 'Compose container service could not be inspected' + resource_name=$(docker inspect --format '{{.Name}}' "$id" 2>/dev/null) || die 'Compose container name could not be inspected' + resource_name=${resource_name#/} + [[ "$service" == controller && "$resource_name" == "$controller" ]] || die 'unexpected Compose container residue exists' + compose_controller_count=$((compose_controller_count + 1)) +done <<<"$compose_container_ids" +((compose_controller_count == 1)) || die 'committed controller Compose container is missing or duplicated' + +compose_volume_names=$(docker volume ls -q --filter label=com.docker.compose.project) || die 'Compose volume residue could not be inspected' +while IFS= read -r name; do + [[ -n "$name" ]] || continue + project=$(docker volume inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose volume identity could not be inspected' + [[ "$project" == ci-fleet ]] || die 'foreign Compose volume residue exists' + die 'unexpected Compose volume residue exists' +done <<<"$compose_volume_names" + +compose_network_names=$(docker network ls -q --filter label=com.docker.compose.project) || die 'Compose network residue could not be inspected' +compose_network_count=0 +while IFS= read -r name; do + [[ -n "$name" ]] || continue + project=$(docker network inspect --format '{{index .Labels "com.docker.compose.project"}}' "$name" 2>/dev/null) || die 'Compose network project could not be inspected' + [[ "$project" == ci-fleet ]] || die 'foreign Compose network residue exists' + network_label=$(docker network inspect --format '{{index .Labels "com.docker.compose.network"}}' "$name" 2>/dev/null) || die 'Compose network label could not be inspected' + resource_name=$(docker network inspect --format '{{.Name}}' "$name" 2>/dev/null) || die 'Compose network name could not be inspected' + [[ "$network_label" == default && "$resource_name" == ci-fleet_default ]] || die 'unexpected Compose network residue exists' + compose_network_count=$((compose_network_count + 1)) +done <<<"$compose_network_names" +((compose_network_count == 1)) || die 'committed controller Compose network is missing or duplicated' + +while IFS= read -r running_name; do + [[ -z "$running_name" || "$running_name" == "$controller" ]] || die 'unrelated running Docker workload exists' +done < <(docker ps --format '{{.Names}}') + +if ! oom_evidence=$(dmesg --level=err,crit,alert,emerg 2>/dev/null); then + die 'kernel OOM evidence could not be inspected' +fi +if grep -Eiq 'out of memory|oom-killer|killed process' <<<"$oom_evidence"; then + die 'OOM evidence exists for the current boot' +fi + +logical_cpus=$(getconf _NPROCESSORS_ONLN 2>/dev/null || true) +[[ "$logical_cpus" =~ ^[1-9][0-9]{0,5}$ ]] || die 'logical CPU count could not be determined' +logical_cpus=$((10#$logical_cpus)) +read -r total_memory_mib available_memory_mib < <(free -m | awk '/^Mem:/ {print $2, $7}') +[[ "$total_memory_mib" =~ ^[1-9][0-9]{0,9}$ ]] || die 'total memory could not be determined' +[[ "$available_memory_mib" =~ ^[1-9][0-9]{0,9}$ ]] || die 'available memory could not be determined' +total_memory_mib=$((10#$total_memory_mib)) +available_memory_mib=$((10#$available_memory_mib)) + +# Fixed reservations match deploy/compose.yaml and the documented capacity policy. +controller_cpu_millicpus=1000 +controller_memory_mib=512 +docker_cpu_reserve_millicpus=1000 +docker_memory_reserve_mib=1024 +total_cpu_millicpus=$((logical_cpus * 1000)) +os_cpu_reserve_millicpus=$(((total_cpu_millicpus * 15 + 99) / 100)) +((os_cpu_reserve_millicpus >= 1000)) || os_cpu_reserve_millicpus=1000 +os_memory_reserve_mib=$(((total_memory_mib * 20 + 99) / 100)) +((os_memory_reserve_mib >= 2048)) || os_memory_reserve_mib=2048 +required_cpu_millicpus=$((target_max * runner_cpus * 1000 + controller_cpu_millicpus + docker_cpu_reserve_millicpus + os_cpu_reserve_millicpus)) +required_memory_mib=$((target_max * runner_memory_mib + controller_memory_mib + docker_memory_reserve_mib + os_memory_reserve_mib)) +required_available_memory_mib=$((target_max * runner_memory_mib + controller_memory_mib + docker_memory_reserve_mib)) + +((required_cpu_millicpus <= total_cpu_millicpus)) || die 'CPU capacity is insufficient for the requested target and reserves' +((required_memory_mib <= total_memory_mib)) || die 'memory capacity is insufficient for the requested target and reserves' +((required_available_memory_mib <= available_memory_mib)) || die 'available memory is insufficient for the requested target' + +printf 'OK configured_max=%d requested_target_max=%d\n' "$configured_max" "$target_max" +printf 'OK effective_min=%d effective_max=%d\n' "$effective_min" "$effective_max" +printf 'OK cpu_budget_millicpus=%d/%d os_reserve_millicpus=%d docker_reserve_millicpus=%d\n' \ + "$required_cpu_millicpus" "$total_cpu_millicpus" "$os_cpu_reserve_millicpus" "$docker_cpu_reserve_millicpus" +printf 'OK memory_budget_mib=%d/%d available_required_mib=%d available_mib=%d os_reserve_mib=%d docker_reserve_mib=%d\n' \ + "$required_memory_mib" "$total_memory_mib" "$required_available_memory_mib" "$available_memory_mib" "$os_memory_reserve_mib" "$docker_memory_reserve_mib" +printf 'OK docker_filesystem_used_percent=%d warning_percent=%d\n' "$disk_used" "$warn_percent" +printf 'CAPACITY_PREFLIGHT_OK phase=%s target_max=%d configured_max=%d effective_max=%d\n' \ + "$phase" "$target_max" "$configured_max" "$effective_max" diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh new file mode 100755 index 00000000..12de1e91 --- /dev/null +++ b/scripts/test-capacity-preflight.sh @@ -0,0 +1,245 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +mkdir -p "$tmp/bin" + +cat >"$tmp/bin/docker" <<'EOF' +#!/usr/bin/env bash +set -u +case "${1:-}" in + info) + if [[ "$*" == *'.DockerRootDir'* ]]; then printf '/var/lib/docker\n'; fi + exit 0 + ;; + compose) + [[ "${2:-}" == version ]] && exit 0 + ;; + inspect) + args="$*" + if [[ "$args" == *'.State.Status'* ]]; then + printf '%s\n' "${FAKE_CONTROLLER_STATE:-running}" + elif [[ "$args" == *'.State.OOMKilled'* ]]; then + printf '%s\n' "${FAKE_CONTROLLER_OOM:-false}" + elif [[ "$args" == *'.Config.Env'* ]]; then + printf '%s\n' \ + "CI_FLEET_GITHUB_URL=${CI_FLEET_GITHUB_URL}" \ + "CI_FLEET_SCALE_SET_NAME=${CI_FLEET_SCALE_SET_NAME}" \ + "CI_FLEET_LABELS=${CI_FLEET_LABELS}" \ + "CI_FLEET_RUNNER_GROUP=${CI_FLEET_RUNNER_GROUP}" \ + "CI_FLEET_INSTANCE=${CI_FLEET_INSTANCE}" \ + "CI_FLEET_MIN_RUNNERS=${FAKE_EFFECTIVE_MIN:-${CI_FLEET_MIN_RUNNERS}}" \ + "CI_FLEET_MAX_RUNNERS=${FAKE_EFFECTIVE_MAX:-${CI_FLEET_MAX_RUNNERS}}" \ + "CI_FLEET_RUNNER_CPUS=${CI_FLEET_RUNNER_CPUS}" \ + "CI_FLEET_RUNNER_MEMORY_MIB=${CI_FLEET_RUNNER_MEMORY_MIB}" \ + "CI_FLEET_DOCKER_GID=${CI_FLEET_DOCKER_GID}" \ + "UNRELATED_SECRET=${FAKE_SECRET_VALUE:-CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT}" + elif [[ "$args" == *'com.docker.compose.project'* ]]; then + printf '%s\n' "${FAKE_COMPOSE_CONTAINER_PROJECT:-${FAKE_COMPOSE_PROJECT:-ci-fleet}}" + elif [[ "$args" == *'com.docker.compose.service'* ]]; then + printf '%s\n' "${FAKE_COMPOSE_SERVICE:-controller}" + elif [[ "$args" == *'{{.Name}}'* ]]; then + printf '/%s\n' "${FAKE_COMPOSE_CONTAINER_NAME:-ci-fleet-controller-1}" + else + exit 1 + fi + ;; + ps) + args="$*" + if [[ "$args" == *'--format'* ]]; then + printf '%s\n' "${CI_FLEET_CONTROLLER_CONTAINER:-ci-fleet-controller-1}" + [[ -z "${FAKE_UNRELATED_CONTAINER:-}" ]] || printf '%s\n' "$FAKE_UNRELATED_CONTAINER" + elif [[ "$args" == *'label=ci-fleet.repository'* ]]; then + for ((i=0; i<${FAKE_ACTIVE_JOBS:-0}; i++)); do printf 'job-%s\n' "$i"; done + elif [[ "$args" == *'label=com.docker.compose.project'* ]]; then + for ((i=0; i<${FAKE_COMPOSE_CONTAINERS:-0}; i++)); do printf 'compose-container-%s\n' "$i"; done + elif [[ "$args" == *'io.randomdevelopment.ci-fleet.managed=true'* ]]; then + for ((i=0; i<${FAKE_MANAGED_RUNNERS:-0}; i++)); do printf 'runner-%s\n' "$i"; done + fi + ;; + volume|network) + kind=$1 + if [[ "${2:-}" == inspect ]]; then + args="$*" + if [[ "$args" == *'com.docker.compose.project'* ]]; then + if [[ "$kind" == volume ]]; then + printf '%s\n' "${FAKE_COMPOSE_VOLUME_PROJECT:-${FAKE_COMPOSE_PROJECT:-ci-fleet}}" + else + printf '%s\n' "${FAKE_COMPOSE_NETWORK_PROJECT:-${FAKE_COMPOSE_PROJECT:-ci-fleet}}" + fi + elif [[ "$args" == *'com.docker.compose.network'* ]]; then + printf '%s\n' "${FAKE_COMPOSE_NETWORK_LABEL:-default}" + elif [[ "$args" == *'{{.Name}}'* ]]; then + printf '%s\n' "${FAKE_COMPOSE_NETWORK_NAME:-ci-fleet_default}" + else + exit 1 + fi + exit 0 + fi + [[ "${2:-}" == ls ]] || exit 1 + if [[ "$*" == *'label=com.docker.compose.project'* ]]; then + if [[ "$kind" == volume ]]; then count=${FAKE_COMPOSE_VOLUMES:-0}; else count=${FAKE_COMPOSE_NETWORKS:-0}; fi + elif [[ "$kind" == volume ]]; then + count=${FAKE_JOB_VOLUMES:-0} + else + count=${FAKE_JOB_NETWORKS:-0} + fi + for ((i=0; i"$tmp/bin/stat" <<'EOF' +#!/usr/bin/env bash +case "${2:-}" in + %g) printf '%s\n' "${CI_FLEET_DOCKER_GID:-999}" ;; + %a) printf '600\n' ;; + %u) printf '0\n' ;; + *) exit 1 ;; +esac +EOF + +cat >"$tmp/bin/df" <<'EOF' +#!/usr/bin/env bash +printf 'Filesystem 1024-blocks Used Available Capacity Mounted on\n' +printf '/dev/test 100000 5000 95000 %s%% /var/lib/docker\n' "${FAKE_DISK_USED:-5}" +EOF + +cat >"$tmp/bin/getconf" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "${FAKE_TOTAL_CPUS:-16}" +EOF + +cat >"$tmp/bin/free" <<'EOF' +#!/usr/bin/env bash +total=${FAKE_TOTAL_MEMORY_MIB:-32768} +available=${FAKE_AVAILABLE_MEMORY_MIB:-30000} +printf ' total used free shared buff/cache available\n' +printf 'Mem: %s 1000 1000 0 1000 %s\n' "$total" "$available" +EOF + +cat >"$tmp/bin/dmesg" <<'EOF' +#!/usr/bin/env bash +[[ "${FAKE_OOM_EVIDENCE:-0}" == 0 ]] || printf 'Out of memory: Killed process 123\n' +EOF +chmod 700 "$tmp/bin"/* + +export PATH="$tmp/bin:$PATH" +export CI_FLEET_GITHUB_URL=https://github.com/EXAMPLE-ORG +export CI_FLEET_SCALE_SET_NAME=docker-ci-validation +export CI_FLEET_LABELS=docker-ci-experimental +export CI_FLEET_RUNNER_GROUP=trusted-private-ci-experimental +export CI_FLEET_INSTANCE=validation +export CI_FLEET_GITHUB_APP_CLIENT_ID=validation +export CI_FLEET_GITHUB_APP_INSTALLATION_ID=1 +export CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE="$tmp/github-app.pem" +export CI_FLEET_DOCKER_GID=999 +export CI_FLEET_MIN_RUNNERS=0 +export CI_FLEET_MAX_RUNNERS=1 +export CI_FLEET_RUNNER_CPUS=4 +export CI_FLEET_RUNNER_MEMORY_MIB=8192 +export CI_FLEET_CONTROLLER_CONTAINER=ci-fleet-controller-1 +export CI_FLEET_DISK_WARN_PERCENT=80 +printf 'fixture only\n' >"$CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE" +chmod 600 "$CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE" + +fail() { printf 'FAIL %s\n' "$*" >&2; exit 1; } +expect_success() { + local output + output=$("$@" 2>&1) || fail "expected success: $*; output=$output" + printf '%s\n' "$output" +} +expect_failure() { + local expected=$1 output + shift + if output=$("$@" 2>&1); then fail "expected failure: $*"; fi + grep -Fq -- "$expected" <<<"$output" || fail "missing failure [$expected]: $output" +} +reset_fixture() { + export CI_FLEET_MIN_RUNNERS=0 CI_FLEET_MAX_RUNNERS=1 + export FAKE_EFFECTIVE_MIN=0 FAKE_EFFECTIVE_MAX=1 + export FAKE_TOTAL_CPUS=16 FAKE_TOTAL_MEMORY_MIB=32768 FAKE_AVAILABLE_MEMORY_MIB=30000 FAKE_DISK_USED=5 + export FAKE_MANAGED_RUNNERS=0 FAKE_ACTIVE_JOBS=0 FAKE_JOB_VOLUMES=0 FAKE_JOB_NETWORKS=0 + export FAKE_COMPOSE_CONTAINERS=1 FAKE_COMPOSE_VOLUMES=0 FAKE_COMPOSE_NETWORKS=1 FAKE_COMPOSE_PROJECT=ci-fleet + export FAKE_COMPOSE_CONTAINER_PROJECT=ci-fleet FAKE_COMPOSE_VOLUME_PROJECT=ci-fleet FAKE_COMPOSE_NETWORK_PROJECT=ci-fleet + export FAKE_COMPOSE_SERVICE=controller FAKE_COMPOSE_CONTAINER_NAME=ci-fleet-controller-1 + export FAKE_COMPOSE_NETWORK_LABEL=default FAKE_COMPOSE_NETWORK_NAME=ci-fleet_default + export FAKE_CONTROLLER_STATE=running FAKE_CONTROLLER_OOM=false + export FAKE_OOM_EVIDENCE=0 FAKE_UNRELATED_CONTAINER= +} + +reset_fixture +pilot=$(expect_success "$repo_root/scripts/preflight.sh") +grep -Fq 'PREFLIGHT_OK warnings=0' <<<"$pilot" || fail 'pilot MAX=1 gate did not pass' +CI_FLEET_MAX_RUNNERS=2 expect_failure 'CI_FLEET_MAX_RUNNERS must be 1 for the pilot' "$repo_root/scripts/preflight.sh" + +valid=$(expect_success "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2) +grep -Fq 'CAPACITY_PREFLIGHT_OK phase=pre-change target_max=2 configured_max=1 effective_max=1' <<<"$valid" || fail 'valid MAX=2 pre-change summary missing' +grep -Fq 'cpu_budget_millicpus=' <<<"$valid" || fail 'safe CPU budget missing' +grep -Fq 'memory_budget_mib=' <<<"$valid" || fail 'safe memory budget missing' + +reset_fixture +FAKE_TOTAL_CPUS=10 expect_failure 'CPU capacity is insufficient' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_TOTAL_MEMORY_MIB=20000 FAKE_AVAILABLE_MEMORY_MIB=18000 expect_failure 'memory capacity is insufficient' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_DISK_USED=80 expect_failure 'Docker filesystem must remain below 80%' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_MANAGED_RUNNERS=1 expect_failure 'active managed runner' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_ACTIVE_JOBS=1 expect_failure 'active fleet job container' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_JOB_VOLUMES=1 expect_failure 'fleet job volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_JOB_NETWORKS=1 expect_failure 'fleet job network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_COMPOSE_NETWORKS=0 FAKE_COMPOSE_CONTAINER_PROJECT=project-run expect_failure 'foreign Compose container residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_COMPOSE_VOLUMES=1 FAKE_COMPOSE_NETWORKS=0 FAKE_COMPOSE_VOLUME_PROJECT=project-run expect_failure 'foreign Compose volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_COMPOSE_NETWORK_PROJECT=project-run expect_failure 'foreign Compose network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_COMPOSE_CONTAINER_NAME=rogue FAKE_COMPOSE_SERVICE=task expect_failure 'unexpected Compose container residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_COMPOSE_VOLUMES=1 expect_failure 'unexpected Compose volume residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_COMPOSE_NETWORK_NAME=ci-fleet_extra expect_failure 'unexpected Compose network residue' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +CI_FLEET_MIN_RUNNERS=1 FAKE_EFFECTIVE_MIN=1 expect_failure 'CI_FLEET_MIN_RUNNERS must be 0' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_UNRELATED_CONTAINER=database expect_failure 'unrelated running Docker workload' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 +reset_fixture +FAKE_OOM_EVIDENCE=1 expect_failure 'OOM evidence exists for the current boot' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 2 + +reset_fixture +expect_failure 'explicit --target-max is required' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change +for malformed in 0 -1 2x 1.5; do + expect_failure 'target MAX must be a positive integer' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max "$malformed" +done +reset_fixture +FAKE_TOTAL_CPUS=32 FAKE_TOTAL_MEMORY_MIB=65536 FAKE_AVAILABLE_MEMORY_MIB=60000 \ + expect_failure 'target MAX must be exactly 2' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 3 + +reset_fixture +CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2 expect_success "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 >/dev/null +reset_fixture +CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=1 expect_failure 'effective controller MAX does not match requested target' "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 +reset_fixture +CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2 +secret_output=$("$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 2>&1) || fail 'secret-output fixture unexpectedly failed' +if grep -Fq 'CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT' <<<"$secret_output"; then fail 'capacity preflight printed an unrelated environment value'; fi +xtrace_output=$(bash -x "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 2>&1) || fail 'xtrace secret-output fixture unexpectedly failed' +if grep -Fq 'CAPACITY_TEST_SECRET_SHOULD_NOT_PRINT' <<<"$xtrace_output"; then fail 'capacity preflight exposed controller environment through inherited xtrace'; fi + +for term in backup force-recreate --no-deps healthcheck retain restore rollback; do + grep -Fqi -- "$term" "$repo_root/docs/CAPACITY-PROMOTION.md" || fail "capacity procedure is missing $term" +done +grep -Fq 'env -i' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'capacity procedure does not isolate Compose interpolation from stale shell values' +grep -Fq 'exec scripts/preflight.sh' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'rollback does not run pilot preflight from a clean restored environment' +grep -Fq 'scripts/capacity-preflight.sh' "$repo_root/docs/ADDING-A-HOST.md" || fail 'host guide does not link the capacity procedure' +if grep -Riq --exclude='test-capacity-preflight.sh' 'docker system prune' "$repo_root/scripts"; then fail 'unrestricted prune exists in scripts'; fi + +printf 'Capacity preflight tests passed.\n' diff --git a/scripts/validate.sh b/scripts/validate.sh index 7710279b..9628d645 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -9,6 +9,7 @@ python3 -m py_compile .github/actions/plan/plan.py .github/actions/plan/test_pla python3 .github/actions/plan/test_plan.py python3 .github/actions/plan/plan.py --plan examples/project/scripts/ci/plan.json --group fast >/dev/null python3 .github/actions/plan/plan.py --plan examples/project/scripts/ci/plan.json --group full >/dev/null +scripts/test-capacity-preflight.sh tmp=$(mktemp) trap 'rm -f "$tmp"' EXIT