feat: prepare status receiver deployment

deploy/status-receiver/ci-fleet-status-receiver.service

@@ -0,0 +1,32 @@
+[Unit]
+Description=ci-fleet authenticated status receiver
+After=network.target
+
+[Service]
+Type=simple
+User=ci-fleet-status
+Group=ci-fleet-status
+ExecStart=/usr/bin/python3 /opt/ci-fleet-status/current/status_receiver.py --auth-config /etc/ci-fleet-status/auth.json --database /var/lib/ci-fleet-status/status.db --bind 127.0.0.1 --port 8080
+Restart=on-failure
+RestartSec=5s
+LogRateLimitIntervalSec=30s
+LogRateLimitBurst=20
+UMask=0077
+NoNewPrivileges=yes
+ProtectSystem=strict
+ProtectHome=yes
+PrivateTmp=yes
+PrivateDevices=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectControlGroups=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+MemoryDenyWriteExecute=yes
+CapabilityBoundingSet=
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+ReadOnlyPaths=/etc/ci-fleet-status
+ReadWritePaths=/var/lib/ci-fleet-status
+
+[Install]
+WantedBy=multi-user.target

deploy/status-receiver/nginx-location.conf.example

@@ -0,0 +1,10 @@
+# Terminate TLS in the existing reverse proxy. Keep the application on loopback.
+location / {
+    client_max_body_size 32k;
+    proxy_connect_timeout 5s;
+    proxy_read_timeout 20s;
+    proxy_send_timeout 20s;
+    proxy_pass http://127.0.0.1:8080;
+    proxy_set_header Host $host;
+    proxy_set_header X-Forwarded-Proto https;
+}

docs/README.md

@@ -62,6 +62,7 @@ These pages are normative for compatible projects and hosts:
 - [Host maintenance standard](HOST-MAINTENANCE.md)
 - [Fleet health monitoring](HEALTH-MONITORING.md)
 - [Authenticated controller status reporting](STATUS-REPORTING.md)
+- [Status receiver deployment](STATUS-RECEIVER-DEPLOYMENT.md)
 - [Git-authored controller desired state](DESIRED-STATE.md)
 - [Secrets model](SECRETS.md)
 - [Security policy](../SECURITY.md)

docs/STATUS-RECEIVER-DEPLOYMENT.md

@@ -0,0 +1,165 @@
+# Status receiver deployment
+
+This runbook prepares the existing authenticated status receiver for a dedicated
+Linux host. It does not create a host, provision credentials, change a controller,
+or deploy anything by itself.
+
+## Target and boundaries
+
+Use a small always-on VM or LXC separate from every runner controller: 2 vCPU,
+2–4 GiB RAM, and about 20 GiB disk. Confirm the next unused infrastructure ID
+from the live hypervisor inventory immediately before creation; repository state
+is not inventory evidence.
+
+The receiver host has no Docker socket, runner credentials, deployment
+credentials, or inbound connection to a controller. The Python application binds
+only to `127.0.0.1:8080`; an existing reverse proxy terminates HTTPS. Controllers
+submit outbound HTTPS. The read API is authenticated and has no mutation route.
+
+## Install from a reviewed commit
+
+On the prepared receiver host, use the supported `/usr/bin/python3` version 3.9
+or newer, check out the exact reviewed commit, and verify a clean tree. The
+installer creates the unprivileged `ci-fleet-status` account, release and state
+directories, a hardened systemd unit, and an atomic `current` link. It does not
+create credentials.
+
+```bash
+ref=$(git rev-parse HEAD)
+test -z "$(git status --porcelain)"
+sudo ./scripts/install-status-receiver.sh --install --ref "$ref"
+```
+
+A second identical invocation returns `NO_CHANGE`. Before activation, verify the
+unit definition:
+
+```bash
+sudo systemd-analyze verify \
+  /etc/systemd/system/ci-fleet-status-receiver.service
+```
+
+## One-time secret provisioning boundary
+
+Provision one independent 32–128 byte signing key per controller and one distinct
+32–128 byte visible-ASCII read token. Values never belong in Git, command
+arguments, chat, logs, issues, PRs, fixtures, or artifacts.
+
+An authorized human uses an approved secret manager or controlled provisioning
+workstation to place the same controller key at these host-local paths:
+
+- receiver: `/etc/ci-fleet-status/controller-keys/<controller-id>.key`;
+- controller: `/etc/ci-fleet/secrets/status-reporting.key`.
+
+The read token exists only at `/etc/ci-fleet-status/read-api.token`. On the
+receiver, every key, token, and `auth.json` is owned by `ci-fleet-status` with
+mode `0600`; both containing directories are mode `0700`. On the controller, the
+signing key is root-owned mode `0600`. Verify ownership, type, and mode without
+printing content. Stop and remove only the newly provisioned files if any check
+fails. Delete any provisioning-workstation copy after both destinations are
+verified. Do not enable SSH to provision or verify the controller.
+
+Create receiver-local `auth.json` with an editor that does not log content. It
+contains only controller-to-key-path mappings and the read-token path; use the
+fictional shape in [status reporting](STATUS-REPORTING.md#receiver). Never put a
+secret value in that JSON file.
+
+## HTTPS and activation
+
+Install the location block from
+`deploy/status-receiver/nginx-location.conf.example` in the existing HTTPS
+reverse proxy. Supply the real public certificate and private endpoint only in
+private infrastructure configuration. Do not expose port 8080.
+
+After receiver-local credential metadata and reverse-proxy configuration pass:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now ci-fleet-status-receiver.service
+sudo systemctl is-active --quiet ci-fleet-status-receiver.service
+sudo ./scripts/install-status-receiver.sh --check
+python3 - <<'PY'
+import json
+import urllib.request
+
+response = urllib.request.urlopen("http://127.0.0.1:8080/healthz", timeout=5)
+assert json.load(response) == {"status": "ok"}
+PY
+```
+
+Configure the controller's private monitoring policy with the HTTPS
+`/v1/status` URL and its host-local signing-key path. Preserve its existing
+identity, routing, capacity, resources, scale-to-zero behavior, and disabled SSH.
+A reporting failure must remain warning-only and must not interrupt runner
+management or reconciliation.
+
+## Verification
+
+1. Submit one scheduled report and confirm HTTP 202 without printing its body or
+   authorization headers.
+2. Read `/v1/controllers/<controller-id>` with the read token loaded from its
+   file by the client process, not placed in an argument or environment dump.
+3. Confirm an invalid signature, stale timestamp, replayed nonce, wrong
+   controller identity, and oversized payload are rejected.
+4. Restart the receiver and confirm `/healthz`, authenticated reads, and retained
+   bounded history recover.
+5. Stop the receiver for longer than one reporting interval. Confirm the
+   controller records only a reporting warning and continues reconciliation and
+   runner lifecycle; then restart the receiver and confirm reporting resumes.
+6. Stop the controller or take it offline. Confirm external monitoring of the
+   separate receiver still works and alerts on the latest report age.
+7. Confirm the receiver listens only on loopback and port 8080 is unreachable
+   externally. Confirm SSH remains disabled on the controller.
+
+The receiver suppresses request logs. Keep systemd journal retention bounded by
+the host's reviewed journald policy and monitor service restart count. SQLite
+retention is enforced independently by age and per-controller count. The status
+database is disposable. For database corruption, stop the service, quarantine the
+database and its journal files outside service-writable state, and start with a
+fresh database:
+
+```bash
+sudo systemctl stop ci-fleet-status-receiver.service
+quarantine="/var/lib/ci-fleet-status-installer/quarantine/$(date -u +%Y%m%dT%H%M%SZ)"
+sudo install -d -o root -g root -m 0700 "$quarantine"
+sudo find /var/lib/ci-fleet-status -maxdepth 1 -type f -name 'status.db*' \
+  -exec mv -t "$quarantine" -- {} +
+sudo systemctl start ci-fleet-status-receiver.service
+```
+
+A full host-loss recovery reinstalls the reviewed release and reprovisions
+credentials. Back up the database only if an operator separately decides that
+short status history is durable evidence.
+
+## Upgrade and rollback
+
+From a clean checkout at the newer reviewed commit:
+
+```bash
+ref=$(git rev-parse HEAD)
+sudo ./scripts/install-status-receiver.sh --upgrade --ref "$ref"
+sudo ./scripts/install-status-receiver.sh --check
+```
+
+The upgrade stages an immutable release, records the previous release, switches
+the symlink atomically, and restarts only an already-active service. If health,
+ingestion, read access, or retention verification fails:
+
+```bash
+sudo ./scripts/install-status-receiver.sh --rollback
+sudo ./scripts/install-status-receiver.sh --check
+```
+
+Rollback restores the selected release's application files and systemd unit. It
+preserves `auth.json`, keys, read token, database, reverse-proxy configuration,
+and journal policy. Reverse-proxy or schema changes require their own reviewed
+compatibility and rollback plan.
+
+## Test coverage
+
+`scripts/test-install-status-receiver.sh` exercises clean install, idempotent
+rerun, upgrade, check, and rollback in an isolated root. Receiver tests cover
+restart/key reload, incorrect secret permissions, authentication, controller
+isolation, replay/freshness, request and payload bounds, strict schema/redaction,
+retention, loopback binding, and read-only routes. Health and installer tests
+cover warning-only reporter outages, disabled SSH reporting, and preservation of
+runner lifecycle during delivery failure.

docs/STATUS-REPORTING.md

@@ -86,6 +86,10 @@ CI_FLEET_HEALTH_STATUS_KEY_FILE=/etc/ci-fleet/secrets/status-reporting.key
 ```
 
 The URL must be HTTPS with the exact `/v1/status` path and no embedded credentials, query, or fragment.
+Schema-v3 desired state may require reporting with only the fixed host-local
+configuration reference `/etc/ci-fleet/monitoring.env`; endpoint and key values
+remain outside Git. A required but missing or unsafe host-local configuration is
+reported as a redacted delivery warning and does not interrupt runner lifecycle.
 
 ## Threat model
 
@@ -121,3 +125,7 @@ Reports never contain:
 - Docker socket access or any host-control capability.
 
 The local full health result remains available for recovery, but only the status schema's allowlisted summary leaves the controller.
+
+For dedicated-host packaging, one-time secret boundaries, activation,
+verification, upgrade, and rollback, see
+[Status receiver deployment](STATUS-RECEIVER-DEPLOYMENT.md).

engine-capabilities.json

@@ -0,0 +1,7 @@
+{
+  "schema_version": 1,
+  "capabilities": {
+    "status_reporting_config": true,
+    "required_status_reporting": true
+  }
+}

scripts/desired_state.py

@@ -28,12 +28,40 @@
     "CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE",
 }
 HOST_OPTIONAL = {"CI_FLEET_RUNNER_TTL"}
+REQUIRED_STATUS_CAPABILITY = "required_status_reporting"
+STATUS_REPORTING_CONFIG_CAPABILITY = "status_reporting_config"
 
 
 class DesiredStateError(ValueError):
     """A safe operator-facing desired-state error."""
 
 
+def load_engine_capabilities(path: Path) -> set[str]:
+    def reject_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
+        value: dict[str, Any] = {}
+        for name, item in pairs:
+            if name in value:
+                raise ValueError("duplicate capability key")
+            value[name] = item
+        return value
+
+    try:
+        metadata = path.lstat()
+        if not stat.S_ISREG(metadata.st_mode):
+            raise DesiredStateError("engine capability declaration must be a regular file")
+        value = json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=reject_duplicates)
+    except FileNotFoundError as exc:
+        raise DesiredStateError("engine capability declaration is missing") from exc
+    except (json.JSONDecodeError, ValueError) as exc:
+        raise DesiredStateError("engine capability declaration is malformed") from exc
+    if not isinstance(value, dict) or set(value) != {"schema_version", "capabilities"} or value.get("schema_version") != 1:
+        raise DesiredStateError("engine capability declaration is malformed")
+    capabilities = value.get("capabilities")
+    if not isinstance(capabilities, dict) or any(type(supported) is not bool for supported in capabilities.values()):
+        raise DesiredStateError("engine capability declaration is malformed")
+    return {name for name, supported in capabilities.items() if supported}
+
+
 def load_template_validator():
     spec = importlib.util.spec_from_file_location("ci_fleet_template_validator", TEMPLATE_VALIDATOR)
     if spec is None or spec.loader is None:
@@ -127,6 +155,7 @@ def build_rendered_env(
     config_repository: str,
     config_ref: str,
     docker_gid: int,
+    engine_capabilities: set[str] | None = None,
 ) -> tuple[dict[str, str], dict[str, Any]]:
     controller, pool = select_controller(config, controller_id)
     engine_commit = controller["engine_ref"]
@@ -165,6 +194,14 @@ def build_rendered_env(
         "CI_FLEET_VERSION": short_commit,
         **validate_host_values(host_values),
     }
+    reporting_configured = "status_reporting" in controller
+    reporting_required = (controller.get("status_reporting") or {}).get("enabled") is True
+    if reporting_required and REQUIRED_STATUS_CAPABILITY not in (engine_capabilities or set()):
+        raise DesiredStateError("selected engine does not advertise required status reporting")
+    if reporting_configured and STATUS_REPORTING_CONFIG_CAPABILITY not in (engine_capabilities or set()):
+        raise DesiredStateError("selected engine does not support status reporting configuration")
+    if reporting_required:
+        rendered["CI_FLEET_STATUS_REPORTING_REQUIRED"] = "1"
     for name, value in rendered.items():
         if not SAFE_ENV_VALUE.fullmatch(value):
             raise DesiredStateError(f"rendered value for {name} contains unsafe characters")
@@ -183,6 +220,8 @@ def build_rendered_env(
         "config_ref": config_ref,
         "engine_ref": engine_commit,
         "engine_repository": config["organization"]["delivery_engine"],
+        "status_reporting_configured": reporting_configured,
+        "status_reporting_required": reporting_required,
     }
     return rendered, metadata
 
@@ -221,13 +260,15 @@ def command_extract_host(args: argparse.Namespace) -> None:
 def command_render(args: argparse.Namespace) -> None:
     config = load_and_validate_config(args.config)
     host_values = parse_env(args.host_config, allow_unknown=False)
+    capabilities = load_engine_capabilities(args.engine_capabilities) if args.engine_capabilities else set()
     values, metadata = build_rendered_env(
         config,
         args.controller,
         host_values,
         config_repository=args.config_repository,
         config_ref=args.config_ref,
         docker_gid=args.docker_gid,
+        engine_capabilities=capabilities,
     )
     write_private(args.output, render_env(values))
     write_private(args.metadata_output, json.dumps(metadata, indent=2, sort_keys=True) + "\n")
@@ -238,6 +279,22 @@ def command_render(args: argparse.Namespace) -> None:
     )
 
 
+def command_engine(args: argparse.Namespace) -> None:
+    config = load_and_validate_config(args.config)
+    controller, _ = select_controller(config, args.controller)
+    print(controller["engine_ref"])
+    print(config["organization"]["delivery_engine"])
+
+
+def command_validate_engine_capabilities(args: argparse.Namespace) -> None:
+    capabilities = load_engine_capabilities(args.manifest)
+    if args.require_status_reporting_config and STATUS_REPORTING_CONFIG_CAPABILITY not in capabilities:
+        raise DesiredStateError("selected engine does not support status reporting configuration")
+    if args.require_status_reporting and REQUIRED_STATUS_CAPABILITY not in capabilities:
+        raise DesiredStateError("selected engine does not advertise required status reporting")
+    print("ENGINE_CAPABILITIES_OK")
+
+
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description=__doc__)
     subparsers = parser.add_subparsers(dest="command", required=True)
@@ -258,9 +315,21 @@ def parse_args() -> argparse.Namespace:
     render.add_argument("--config-repository", required=True)
     render.add_argument("--config-ref", required=True)
     render.add_argument("--docker-gid", type=int, required=True)
+    render.add_argument("--engine-capabilities", type=Path)
     render.add_argument("--output", type=Path, required=True)
     render.add_argument("--metadata-output", type=Path, required=True)
     render.set_defaults(function=command_render)
+
+    engine = subparsers.add_parser("engine", help="select the immutable engine for one controller")
+    engine.add_argument("--config", type=Path, required=True)
+    engine.add_argument("--controller", required=True)
+    engine.set_defaults(function=command_engine)
+
+    capabilities = subparsers.add_parser("validate-engine-capabilities", help="validate an engine capability declaration")
+    capabilities.add_argument("--manifest", type=Path, required=True)
+    capabilities.add_argument("--require-status-reporting-config", action="store_true")
+    capabilities.add_argument("--require-status-reporting", action="store_true")
+    capabilities.set_defaults(function=command_validate_engine_capabilities)
     return parser.parse_args()