Add Git-authored controller desired state

.github/workflows/validate.yml

@@ -35,11 +35,33 @@ jobs:
 
       - name: Validate public configuration template
         run: |
+          cmp -s scripts/scan_committed_secrets.py templates/config-repository/scripts/scan_committed_secrets.py
           python3 templates/config-repository/scripts/test_policy.py
+          python3 templates/config-repository/scripts/scan_committed_secrets.py
           templates/config-repository/scripts/validate.sh
           templates/config-repository/scripts/validate.sh --config templates/config-repository/examples/multi-host/fleet.json
           temporary_directory="$(mktemp -d)"
+          if templates/config-repository/scripts/init.sh \
+            --organization test-company \
+            --project test-app \
+            --output "${temporary_directory}/missing-engine.json"; then
+            echo 'initializer accepted a missing engine revision' >&2
+            exit 1
+          fi
+          test ! -e "${temporary_directory}/missing-engine.json"
+          if templates/config-repository/scripts/init.sh \
+            --organization test-company \
+            --project test-app \
+            --registry BAD \
+            --engine-ref "$(git rev-parse HEAD)" \
+            --output "${temporary_directory}/invalid.json"; then
+            echo 'initializer accepted an invalid registry' >&2
+            exit 1
+          fi
+          test ! -e "${temporary_directory}/invalid.json"
           templates/config-repository/scripts/init.sh \
             --organization test-company \
             --project test-app \
+            --engine-ref "$(git rev-parse HEAD)" \
             --output "${temporary_directory}/fleet.json"
+          test "$(stat -c '%a' "${temporary_directory}/fleet.json")" = 644

.gitignore

@@ -18,6 +18,8 @@ _work/
 runner-data/
 logs/
 tmp/
+__pycache__/
+*.pyc
 
 # Editors and operating systems
 .vscode/

AGENTS.md

@@ -6,7 +6,7 @@ This repository defines a portable, public, self-hosted GitHub Actions CI fleet
 
 ## Current phase
 
-The project is in architecture definition and isolated proof-of-concept planning. Do not create or alter production deployment paths until the design decisions are recorded and the isolated proof of concept is approved.
+The isolated proof of concept is approved and the project is in controlled migration. The accepted scope and remaining gates are recorded in `docs/DESIGN-DECISIONS.md`. Managed ordinary-CI controller lifecycle is permitted only through reviewed schema-v3 desired state; production deployment paths remain separately gated.
 
 ## Sources of truth
 
@@ -41,6 +41,8 @@ Agents modifying this repository or adapting a project MUST apply these document
 - Do not install project language runtimes in the generic runner image.
 - Do not put repository names, project allowlists, or project-specific logic in runner images or host configuration.
 - Keep project workflows independent of host, site, instance, and scale-set names; route through shared capability labels.
+- Keep worker capacity in reviewed private infrastructure configuration. Application workflows submit all independent jobs and must not use `max-parallel` to model fleet size.
+- Treat schema-v3 controller desired state as authoritative for routing, capacity, lifecycle, and engine revision; host-local files contain only rendered state and credentials.
 - Do not allow projects to select privileged runner groups through untrusted inputs.
 
 ## Verification expectations

README.md

@@ -8,7 +8,7 @@
 
 Use one shared pool of disposable CI workers across multiple trusted private repositories, Docker hosts, virtual machines, bare-metal computers, home labs, remote sites, or VPS providers. Projects bring their own Dockerized build and test environment; fleet hosts stay generic.
 
-> **Status:** Experimental. The first controller host is deployed, and the first manually dispatched runner pilot is pending. The project is not production-ready.
+> **Status:** Experimental. Ephemeral runner pilots and multi-job workloads have been proven, and the schema-v3 Git-authored controller installer is available for reviewed adoption testing. The project is not production-ready.
 
 ## What problem does this solve?
 
@@ -93,7 +93,7 @@ GitHub runner-group policy decides which repositories may schedule work. A share
 | --- | --- | --- |
 | Public fleet repository | Runner image, controller, lifecycle, setup, health checks, scoped cleanup, standards, examples | Real credentials, private host inventory, production configuration |
 | Project repository | Test Dockerfile, services, fixtures, migrations, test plan, `scripts/ci/run.sh` | Fleet controller credentials or host-specific setup |
-| Private installation configuration | Organization settings, repository authorization, host capacity, network policy, monitoring, secrets | Project runtime dependencies or test logic |
+| Private installation configuration | Organization settings, repository authorization, logical controller state, capacity budgets, network policy, and required secret names | Secret values, host addresses, project runtime dependencies, or test logic |
 
 A public application can use the same fleet indirectly. Its public repository keeps pull-request validation unprivileged, while a separate private delivery repository checks out an approved immutable commit and performs protected CI, release, or deployment work. The public repository itself never receives privileged runner-group access or fleet credentials. See [Public projects, private delivery, and private configuration](docs/PUBLIC-PRIVATE-CONFIGURATION.md).
 
@@ -137,9 +137,9 @@ Read-only validation, repository-writing releases, staging, production deploymen
 | Public architecture, standards, examples, and migration rules | Available |
 | Docker runner and controller prototype | Available |
 | First isolated controller host | Deployed |
-| First manually dispatched private-repository pilot job | Pending |
-| One-command `sudo ./setup.sh` installer | Accepted design; not implemented |
-| MailThisForMe migration | Planned after pilot |
+| First manually dispatched private-repository pilot job | Proven |
+| Schema-v3 worker-controller installer | Available for experimental install and adoption |
+| MailThisForMe migration | Parallel task-matrix validation in progress |
 | TF2 Recommendation Engine migration | Planned after pilot |
 | Reusable tester and deployer components | Planned |
 | Production-ready release | Not yet |
@@ -169,6 +169,7 @@ Supported deployment shapes include virtual machines, dedicated physical machine
 | Browse every guide, concept, standard, and example | [Documentation index](docs/README.md) |
 | Evaluate the current prototype safely | [Live pilot runbook](docs/LIVE-PILOT.md) |
 | Add another Docker host or location | [How to add a host](docs/ADDING-A-HOST.md) |
+| Install, adopt, check, upgrade, or remove a controller | [Git-authored controller desired state](docs/DESIRED-STATE.md) |
 | Add a trusted private repository | [How to add a project](docs/ADDING-A-PROJECT.md) |
 | Convert existing GitHub Actions CI | [Migrating existing CI](docs/MIGRATING-EXISTING-CI.md) |
 | Build a compatible project contract | [Project CI standard](docs/PROJECT-STANDARD.md) |
@@ -236,17 +237,21 @@ Use this fleet only for explicitly trusted repositories and workflows. Keep depl
 
 Read [SECURITY.md](SECURITY.md) and the [secrets model](docs/SECRETS.md) before registration or deployment.
 
-## Installation direction
+## Install or adopt a controller
 
-The accepted operator experience is:
+After creating the GitHub App/runner group and placing the host-local identity file, run one command on the target machine with a reviewed private configuration commit:
 
 ```bash
-sudo ./setup.sh
+sudo ./scripts/install-worker-controller.sh \
+  --install \
+  --config-repo example-org/example-fleet-config \
+  --ref 1111111111111111111111111111111111111111 \
+  --controller example-ci-01
 ```
 
-That command will eventually bootstrap or validate GitHub configuration, install the controller and maintenance services, verify health, and support safe reruns, repair, upgrades, and removal directly from the target host. Phone interaction should be limited to unavoidable, non-secret GitHub approval links.
+Use `--adopt` for an existing manually installed controller. The same script provides `--check`, `--upgrade`, `--rollback`, and `--uninstall`. It validates schema-v3 desired state, renders only non-secret host configuration, installs the pinned engine revision, drains before replacement, records a rollback checkpoint, and installs health, scoped-cleanup, and drift timers.
 
-**That installer is not implemented yet.** For the experimental version, follow the [live pilot runbook](docs/LIVE-PILOT.md) and [deployment prototype](docs/DEPLOYMENT-PROTOTYPE.md).
+The installer remains experimental and intentionally does not create GitHub credentials or broaden runner-group access. Phone-friendly GitHub bootstrap is tracked separately. Read [Git-authored controller desired state](docs/DESIRED-STATE.md) before using it on a live host; retain the [live pilot runbook](docs/LIVE-PILOT.md) for first-job proof.
 
 ## Examples
 

controller/Dockerfile

@@ -8,6 +8,8 @@ ARG CI_FLEET_COMMIT=unknown
 RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${CI_FLEET_VERSION} -X main.commitSHA=${CI_FLEET_COMMIT}" -o /out/ci-fleet-controller .
 
 FROM debian:13.6-slim
+ARG CI_FLEET_COMMIT=unknown
+LABEL org.opencontainers.image.revision="${CI_FLEET_COMMIT}"
 RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/*
 COPY --from=build /out/ci-fleet-controller /usr/local/bin/ci-fleet-controller
 USER 65532:65532

deploy/compose.yaml

@@ -6,6 +6,7 @@ services:
     build:
       context: ../runner
       args:
+        CI_FLEET_COMMIT: ${CI_FLEET_COMMIT:-unknown}
         RUNNER_VERSION: ${CI_FLEET_RUNNER_VERSION:-2.335.1}
     command: ["/bin/true"]
     deploy:

docs/ADDING-A-HOST.md

@@ -1,6 +1,6 @@
 # How to add a fleet host
 
-Use this runbook to add a Proxmox VM, physical computer, remote-site machine, or VPS to an existing ci-fleet installation.
+Use this runbook to add a virtual machine, physical computer, remote-site machine, or VPS to an existing ci-fleet installation.
 
 ## Outcome
 
@@ -73,62 +73,41 @@ Before placing credentials on the host:
 
 A remote site needs outbound connectivity; it does not need project-specific inbound access. If one location is offline, compatible jobs remain queued for another available host.
 
-## 4. Install a pinned ci-fleet checkout
+## 4. Declare the host in private Git configuration
 
-Use a reviewed immutable commit:
+Create or update one schema-v3 controller entry in the organization's private configuration repository. Give it a unique logical ID and scale-set name, assign its shared runner pool, start with reviewed capacity, and pin a full ci-fleet engine commit.
 
-```bash
-sudo install -d -m 0755 /opt/ci-fleet
-# Populate /opt/ci-fleet from a reviewed ci-fleet commit.
-git -C /opt/ci-fleet rev-parse HEAD
-```
+Run that repository's strict validator before merge. Do not put the machine's address, VM ID, storage, backup identifier, or credential in Git.
 
-Do not follow a moving branch for unattended controller updates. Fleet image changes use reviewed rolling replacement and keep the previous version available for rollback.
+See [Git-authored controller desired state](DESIRED-STATE.md) for the complete contract.
 
-## 5. Create host-local configuration
+## 5. Prepare host-local identity
 
-Start from `deploy/ci-fleet.env.example`:
+From a reviewed ci-fleet checkout:
 
 ```bash
 sudo install -d -m 0700 /etc/ci-fleet/secrets
-sudo install -m 0600 deploy/ci-fleet.env.example /etc/ci-fleet/ci-fleet.env
+sudo install -m 0600 host/host.env.example /etc/ci-fleet/host.env
+sudo install -m 0600 /secure/source/github-app.pem /etc/ci-fleet/secrets/github-app.pem
 ```
 
-Set:
-
-- a unique `CI_FLEET_INSTANCE`;
-- a unique `CI_FLEET_SCALE_SET_NAME`;
-- the fleet's shared `CI_FLEET_LABELS`;
-- the shared `CI_FLEET_RUNNER_GROUP`;
-- host-specific maximum concurrency and runner CPU/memory limits;
-- `CI_FLEET_MIN_RUNNERS=0` so idle runners do not persist.
-
-Repository names do not belong in this file.
+Edit `/etc/ci-fleet/host.env` locally. Prefer a separately generated GitHub App private key per host so one location can be revoked without replacing every host key. Never expose the key to runner or project containers.
 
-Provision the GitHub App private key directly into `/etc/ci-fleet/secrets/github-app.pem` with mode `0600`. Prefer a separately generated App private key per host so one location can be revoked without replacing every host key. Never expose the key to runner or project containers.
-
-## 6. Validate before starting
+## 6. Install or adopt with one command
 
 ```bash
-cd /opt/ci-fleet
-set -a
-. /etc/ci-fleet/ci-fleet.env
-set +a
-scripts/preflight.sh
-docker compose -f deploy/compose.yaml build runner-image controller
-scripts/preflight.sh
+sudo ./scripts/install-worker-controller.sh \
+  --install \
+  --config-repo example-org/example-fleet-config \
+  --ref 1111111111111111111111111111111111111111 \
+  --controller example-ci-01
 ```
 
-Continue only when preflight reports `PREFLIGHT_OK warnings=0`.
-
-## 7. Start at zero and prove one job
+Use `--adopt` when converting an existing manual controller. The installer validates the complete configuration, renders host-local runtime state, installs the pinned engine, creates a controller checkpoint, builds images, installs maintenance timers, and verifies health.
 
-Start with maximum concurrency one even if the host will eventually run more:
+The configuration repository credential, when required, must be read-only and host-side. Credentials are never accepted in command arguments.
 
-```bash
-docker compose -f deploy/compose.yaml up -d --no-deps controller
-docker compose -f deploy/compose.yaml logs --tail=100 controller
-```
+## 7. Prove one job
 
 Confirm no runner exists while idle. Dispatch one manual, read-only job from an authorized private project and verify:
 
@@ -141,28 +120,29 @@ Confirm no runner exists while idle. Dispatch one manual, read-only job from an
 
 Follow [Live pilot](LIVE-PILOT.md) for the complete proof and rollback.
 
-## 8. Enable unattended operations
+## 8. Verify unattended operations
 
-Install the health and cleanup timers from [Host maintenance](HOST-MAINTENANCE.md). Monitor controller state, disk thresholds, Docker health, last successful job, cleanup failures, and pending reboot state.
+The installer enables health, scoped cleanup, and pinned desired-state drift timers. Run each service once and inspect its journal as described in [Host maintenance](HOST-MAINTENANCE.md). Monitor controller state, disk thresholds, Docker health, last successful job, cleanup failures, drift, and pending reboot state.
 
 The steady-state host should need no project-specific edits. Adding a project changes its repository, private fleet configuration, and GitHub runner-group policy—not this host.
 
-## 9. Increase capacity safely
+## 9. Increase capacity safely through Git
 
-Raise `CI_FLEET_MAX_RUNNERS` only after measuring CPU, memory, disk, network, cache growth, collisions, cancellation, and cleanup. Keep explicit per-runner limits.
+Raise the controller's schema-v3 `max_runners` and its pool `capacity_budget` through a reviewed private configuration change only after measuring CPU, memory, disk, network, cache growth, collisions, cancellation, and cleanup. Keep explicit per-runner limits.
 
 Keep the one-runner pilot preflight unchanged. For any later increase, follow [Post-pilot capacity promotion](CAPACITY-PROMOTION.md), run `scripts/capacity-preflight.sh` before and after the controller-only recreation, and retain the new maximum only when the separately authorized workload and cleanup proof satisfy the predeclared resource policy.
 
 Use another host or location when it improves failure tolerance. Give every added host a new instance and scale-set name while retaining the shared routing label.
 
 ## 10. Drain, replace, or remove a host
 
-1. Stop new capacity by setting maximum capacity to zero or stopping the controller.
-2. Confirm no managed runner is active.
-3. Run cleanup in dry-run mode, review candidates, then apply scoped cleanup.
-4. Verify the host's unique scale set is absent from GitHub.
-5. Revoke that host's GitHub App private key.
-6. Securely delete host-local credentials.
-7. Remove or restore the machine according to the site's recovery policy.
+1. Merge a private configuration change setting the controller to `drained`.
+2. Apply that exact commit with `install-worker-controller.sh --upgrade`.
+3. Confirm zero managed runners and run scoped cleanup.
+4. Run `install-worker-controller.sh --uninstall` when the host will not return.
+5. Verify the host's unique scale set is absent from GitHub.
+6. Revoke that host's GitHub App private key and explicitly remove preserved credentials.
+7. Set the controller to `disabled` or remove it from private configuration.
+8. Delete or repurpose the machine according to the site's recovery policy.
 
 Other hosts and project workflows continue using the shared routing label. Replacing one host must not require editing any project workflow.

docs/ARCHITECTURE.md

@@ -38,7 +38,8 @@ A host may run multiple runners when resource measurements and project isolation
 flowchart LR
     A["ci-fleet"] --> B["Runner lifecycle"]
     C["Project repo"] --> D["Test environment"]
-    E["Host config"] --> F["Secrets and capacity"]
+    E["Private Git config"] --> F["Routing and capacity desired state"]
+    G["Host-local config"] --> H["Credentials and rendered runtime state"]
 ```
 
 ### Fleet repository
@@ -71,19 +72,19 @@ Each project is responsible for:
 - run-scoped cleanup;
 - release and deployment behavior.
 
-### Deployment-local configuration
+### Private installation configuration
 
 Each installation is responsible for:
 
 - real organization and runner-group names;
-- host capacity;
+- logical controller identities and capacity budgets;
+- pinned controller engine revisions;
 - private network rules;
-- secret provisioning;
-- host inventory;
+- required secret names;
 - monitoring destinations;
 - maintenance windows.
 
-Deployment-local values must not be committed to this public repository.
+These secret-free values belong in a reviewed private schema-v3 configuration repository. Machine addresses, VM IDs, storage and backup identifiers, credentials, secret values, and rendered runtime files remain host-local or in an external secret manager. See [Git-authored controller desired state](DESIRED-STATE.md).
 
 ## Target runner lifecycle
 
@@ -127,7 +128,7 @@ The same fleet interface should support:
 
 - one Docker host with one or more runners;
 - several Docker hosts with one or more runners each;
-- Proxmox virtual machines;
+- virtual machines;
 - dedicated physical computers;
 - remote-site computers;
 - VPS infrastructure.
@@ -152,6 +153,7 @@ Host-wide pruning must not run as an uncoordinated per-job operation. Hard cance
 - Host security updates are applied automatically.
 - Reboots drain runner capacity before interrupting the host.
 - Fleet updates use rolling replacement and an explicit rollback version.
+- Controller routing, capacity, lifecycle, and engine updates use reviewed full commits from private desired-state configuration; hosts never follow an unreviewed moving branch.
 
 ## Project adoption