diff --git a/docs/CAPACITY-PROMOTION.md b/docs/CAPACITY-PROMOTION.md index a1615e87..f1aa0632 100644 --- a/docs/CAPACITY-PROMOTION.md +++ b/docs/CAPACITY-PROMOTION.md @@ -8,7 +8,7 @@ Schema-v3 private desired state owns capacity. Never edit `/etc/ci-fleet/ci-flee Declare the policy before observing a larger workload: -- requested MAX is exactly two for this first post-pilot procedure; +- requested MAX is the exact reviewed controller `max_runners`; - `MIN` remains zero; - both the controller `max_runners` and pool `capacity_budget` change in one reviewed private configuration PR; - the configured instance, scale set, routing label, runner group, Docker socket group, and per-runner limits remain unchanged; @@ -20,31 +20,32 @@ Declare the policy before observing a larger workload: Project containers use the host Docker daemon as siblings of the runner container. The separately authorized live proof must therefore observe whole-host CPU, memory, disk, collisions, and cleanup. -During that proof, retain MAX=2 only if every five-second sample keeps CPU busy below 85%, available memory at or above the greater of 2 GiB or 20% of total memory, and Docker filesystem use below 80%. Any OOM, unrelated workload, controller/Docker failure, third runner, observer gap, or cleanup residue requires restoration. +During that proof, retain the requested MAX only if every five-second sample keeps CPU busy below 85%, available memory at or above the greater of 2 GiB or 20% of total memory, and Docker filesystem use below 80%. Any OOM, unrelated workload, controller/Docker failure, runner above the requested maximum, observer gap, or cleanup residue requires restoration. -## 1. Gate dispatch and verify the one-runner state +## 1. Gate dispatch and verify the current state Block dispatches that can target this controller. Confirm no queued, assigned, or running fleet job and no instance-owned runner in any state. Keep dispatch gated through post-change verification. -Run the current installed preflight and health checks from clean processes. Require the existing one-runner contract, controller health, and an empty instance-scoped cleanup dry-run. +Run the current installed health check from a clean process. Require the configured and effective current MAX to match, controller health, and an empty instance-scoped cleanup dry-run. The target-capacity check in the next step replaces the pilot-only preflight for later increases. ## 2. Validate target capacity without changing it From the installed reviewed release, run: ```bash +TARGET_MAX=REVIEWED_CONTROLLER_MAX sudo env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root bash -c ' set -a . /etc/ci-fleet/ci-fleet.env set +a - /opt/ci-fleet/current/scripts/capacity-preflight.sh --phase pre-change --target-max 2 -' + /opt/ci-fleet/current/scripts/capacity-preflight.sh --phase pre-change --target-max "$1" +' _ "$TARGET_MAX" ``` Require: ```text -CAPACITY_PREFLIGHT_OK phase=pre-change target_max=2 configured_max=1 effective_max=1 +CAPACITY_PREFLIGHT_OK phase=pre-change target_max=REVIEWED_CONTROLLER_MAX configured_max=PREVIOUS_MAX effective_max=PREVIOUS_MAX ``` Record only the safe budget summary. @@ -53,7 +54,7 @@ Record only the safe budget summary. In the secret-free private configuration repository: -1. raise only the selected controller's `max_runners` from one to two; +1. raise only the selected controller's `max_runners` to the reviewed target; 2. raise its pool `capacity_budget` only as needed to admit that reviewed controller maximum; 3. keep `min_runners`, runner resources, identity, lifecycle, routing, trust, and engine pin unchanged; 4. run the complete strict validator, policy tests, and committed-secret scan; @@ -80,31 +81,32 @@ Do not recreate runner jobs, remove volumes, or touch unrelated Docker resources ## 5. Verify effective state -Require one running controller, restart count zero, one intended scale set, the unchanged routing label and runner group, no idle runner, and the exact reviewed MAX=2. +Require one running controller, restart count zero, one intended scale set, the unchanged routing label and runner group, no idle runner, and the exact reviewed target MAX. Run: ```bash +TARGET_MAX=REVIEWED_CONTROLLER_MAX sudo env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root bash -c ' set -a . /etc/ci-fleet/ci-fleet.env set +a - /opt/ci-fleet/current/scripts/capacity-preflight.sh --phase post-change --target-max 2 + /opt/ci-fleet/current/scripts/capacity-preflight.sh --phase post-change --target-max "$1" /opt/ci-fleet/current/scripts/healthcheck.sh -' +' _ "$TARGET_MAX" ``` Require both to pass, plus a clean desired-state `--check` and empty instance-scoped cleanup dry-run. ## 6. Run one separately authorized proof -Start bounded runner, task-job, project-resource, and host-metric observers before dispatch. Dispatch exactly the approved workload once. Do not retry a failed proof and do not raise MAX above two. +Start bounded runner, task-job, project-resource, and host-metric observers before dispatch. Dispatch exactly the approved workload once. Do not retry a failed proof or exceed the reviewed target MAX. -Observe runner creation/destruction, actual two-way overlap, no third runner, whole-host resource thresholds, Docker/controller health, exact project identity, and automatic cleanup. Repeat health, drift, and cleanup dry-run checks after all jobs terminate. +Observe runner creation/destruction, continuous target-way overlap, no runner above the requested maximum, whole-host resource thresholds, Docker/controller health, exact project identity, and automatic cleanup. A workload that never drives the full reviewed target does not qualify. Repeat health, drift, and cleanup dry-run checks after all jobs terminate. ## 7. Retain or restore -Retain MAX=2 only when every predeclared gate passes and no manual cleanup is required. +Retain the requested MAX only when every predeclared gate passes and no manual cleanup is required. On any failure, keep dispatch gated and apply the recorded previous private configuration commit through the same reviewed `--upgrade` path: @@ -116,4 +118,4 @@ sudo /opt/ci-fleet/manager/current/scripts/install-worker-controller.sh \ --controller CONTROLLER_ID ``` -Require the restored MAX=1, clean desired-state check, health pass, zero runners/jobs/residue, one intended scale set, and empty instance-scoped cleanup dry-run before reopening dispatch. Preserve installer checkpoint and proof evidence without recording credential values. +Require the restored PREVIOUS_MAX, clean desired-state check, health pass, zero runners/jobs/residue, one intended scale set, and empty instance-scoped cleanup dry-run before reopening dispatch. Preserve installer checkpoint and proof evidence without recording credential values. diff --git a/scripts/capacity-preflight.sh b/scripts/capacity-preflight.sh index f1bf8ca7..cfc49106 100755 --- a/scripts/capacity-preflight.sh +++ b/scripts/capacity-preflight.sh @@ -42,7 +42,6 @@ done [[ -n "$target_max" ]] || die 'explicit --target-max is required' [[ "$target_max" =~ ^[1-9][0-9]{0,3}$ ]] || die 'target MAX must be a positive integer' target_max=$((10#$target_max)) -((target_max == 2)) || die 'target MAX must be exactly 2 for this post-pilot procedure' required=( CI_FLEET_GITHUB_URL diff --git a/scripts/test-capacity-preflight.sh b/scripts/test-capacity-preflight.sh index 83ef6554..bd3699ee 100755 --- a/scripts/test-capacity-preflight.sh +++ b/scripts/test-capacity-preflight.sh @@ -221,7 +221,7 @@ for malformed in 0 -1 2x 1.5; do done reset_fixture FAKE_TOTAL_CPUS=32 FAKE_TOTAL_MEMORY_MIB=65536 FAKE_AVAILABLE_MEMORY_MIB=60000 \ - expect_failure 'target MAX must be exactly 2' "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 3 + expect_success "$repo_root/scripts/capacity-preflight.sh" --phase pre-change --target-max 6 >/dev/null reset_fixture CI_FLEET_MAX_RUNNERS=2 FAKE_EFFECTIVE_MAX=2 expect_success "$repo_root/scripts/capacity-preflight.sh" --phase post-change --target-max 2 >/dev/null @@ -242,6 +242,9 @@ grep -Fq 'PREVIOUS_PRIVATE_CONFIGURATION_COMMIT' "$repo_root/docs/CAPACITY-PROMO grep -Fq 'env -i' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'capacity procedure does not isolate preflight from the caller environment' if grep -Eq 'Edit only|force-recreate|ci-fleet\.env\.before-max2' "$repo_root/docs/CAPACITY-PROMOTION.md"; then fail 'capacity procedure still edits rendered host state'; fi grep -Fq 'scripts/capacity-preflight.sh' "$repo_root/docs/ADDING-A-HOST.md" || fail 'host guide does not link the capacity procedure' +grep -Fq 'Require the restored PREVIOUS_MAX' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'capacity rollback does not verify the previous reviewed maximum' +if grep -Fq 'one-runner state' "$repo_root/docs/CAPACITY-PROMOTION.md"; then fail 'generic capacity procedure still requires the pilot state'; fi +grep -Fq 'target-way overlap' "$repo_root/docs/CAPACITY-PROMOTION.md" || fail 'capacity proof does not require exercising the full reviewed target' if grep -Riq --exclude='test-capacity-preflight.sh' 'docker system prune' "$repo_root/scripts"; then fail 'unrestricted prune exists in scripts'; fi printf 'Capacity preflight tests passed.\n'