Remote configuration reconciliation via GitHub App token

docs/DESIRED-STATE.md

@@ -136,6 +136,42 @@ No mode uses global Docker prune or removes unrelated workloads.
 
 The host does not automatically follow or execute a moving branch. A new configuration becomes effective only when an operator or authorized external controller supplies its reviewed full commit SHA to `--upgrade`. Automatic dispatchers may watch a protected branch and invoke that exact command after merge, using read-only repository contents permission; their identity must remain host-side and unavailable to job runners.
 
+## Remote reconciliation
+
+`ci-fleet-reconcile.timer` runs every five minutes to fetch the desired configuration from the private desired-state repository and apply it if a newer reviewed commit is available. Authentication uses the existing GitHub App identity — no PAT, no SSH, no inbound management port.
+
+How it works:
+
+1. The timer invokes `scripts/remote-reconcile.sh`.
+2. The script generates a short-lived GitHub App installation token using the existing private key on the host (openssl + curl, no new dependencies).
+3. It fetches the default-branch HEAD of the desired-state repository over authenticated HTTPS.
+4. The fetched commit is resolved to an immutable SHA and compared with the installed SHA.
+5. If unchanged, a drift check confirms convergence (NO_CHANGE).
+6. If changed, the new configuration is validated (schema, secret scan, tree completeness) before any mutation.
+7. On success, the last-known-good state is updated and the controller runs with the new configuration.
+8. On failure, the controller rolls back to the last-known-good checkpoint via the existing installer mechanism.
+9. State is recorded at `/var/lib/ci-fleet/reconcile/state.json` with desired commit, applied commit, health, and failure description.
+
+Bounded retries (up to 3 attempts) handle transient fetch or API failures. All logging is sanitized — tokens, private keys, and raw credential values never appear in stdout, stderr, or state files.
+
+Prerequisites:
+
+- The controller's GitHub App must have `contents: read` permission and be authorized for the desired-state repository.
+- `host.env` must contain `CI_FLEET_GITHUB_APP_CLIENT_ID`, `CI_FLEET_GITHUB_APP_INSTALLATION_ID`, and `CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE`.
+
+Manual invocation:
+
+```bash
+# Check-only — validate without applying
+sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only
+
+# Full reconcile
+sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh
+
+# No-op — log what would be done
+sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --no-op
+```
+
 ## Drain and retirement
 
 1. Merge a private configuration change setting the controller to `drained`.

docs/HOST-MAINTENANCE.md

@@ -32,7 +32,8 @@ Review `/etc/apt/apt.conf.d/50unattended-upgrades` and confirm only the intended
 
 - `ci-fleet-health.timer` runs the complete [fleet health contract](HEALTH-MONITORING.md);
 - `ci-fleet-cleanup.timer` removes only expired inactive fleet-owned resources;
-- `ci-fleet-drift.timer` compares the installation with the exact pinned configuration commit without applying changes.
+- `ci-fleet-drift.timer` compares the installation with the exact pinned configuration commit without applying changes;
+- `ci-fleet-reconcile.timer` fetches the latest reviewed desired-state commit from the private repository over authenticated HTTPS and applies it automatically.
 
 Run each service manually once before relying on its timer:
 

host/systemd/ci-fleet-reconcile.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Reconcile ci-fleet controller with remote desired-state repository
+Documentation=https://github.com/RandomDevelopment/ci-fleet
+After=docker.service network-online.target ci-fleet-drift.service
+Wants=docker.service network-online.target ci-fleet-drift.service
+
+[Service]
+Type=oneshot
+User=root
+WorkingDirectory=/opt/ci-fleet/manager/current
+ExecStart=/opt/ci-fleet/manager/current/scripts/remote-reconcile.sh
+Restart=no
+# 0=noop, 3=drift/invalid — timer retries
+SuccessExitStatus=0 3

host/systemd/ci-fleet-reconcile.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Reconcile ci-fleet controller every five minutes
+Documentation=https://github.com/RandomDevelopment/ci-fleet
+
+[Timer]
+OnBootSec=10min
+OnUnitActiveSec=5min
+AccuracySec=30s
+Persistent=true
+
+[Install]
+WantedBy=timers.target

scripts/github-app-token.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Generate a short-lived GitHub App installation token.
+# Uses openssl (existing dependency) for JWT signing and curl for the API exchange.
+# Usage:
+#   github-app-token.sh --app-id ID --install-id ID --key-path PATH
+#   github-app-token.sh --env-file PATH  # reads APP_ID/INSTALL_ID/KEY_PATH from env file
+set -Eeuo pipefail
+
+app_id=
+install_id=
+key_path=
+env_file=
+
+while (($#)); do
+  case "$1" in
+    --app-id) app_id=$2; shift 2 ;;
+    --install-id) install_id=$2; shift 2 ;;
+    --key-path) key_path=$2; shift 2 ;;
+    --env-file) env_file=$2; shift 2 ;;
+    --help|-h) echo "usage: $(basename "$0") --app-id ID --install-id ID --key-path PATH"; exit 0 ;;
+    *) echo "ERROR: unknown argument: $1" >&2; exit 2 ;;
+  esac
+done
+
+if [[ -n "$env_file" ]]; then
+  [[ -f "$env_file" ]] || { echo "ERROR: env file not found: $env_file" >&2; exit 2; }
+  while IFS='=' read -r name value; do
+    case "$name" in
+      CI_FLEET_GITHUB_APP_CLIENT_ID) app_id=$value ;;
+      CI_FLEET_GITHUB_APP_INSTALLATION_ID) install_id=$value ;;
+      CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE) key_path=$value ;;
+    esac
+  done < <(grep -E '^(CI_FLEET_GITHUB_APP_CLIENT_ID|CI_FLEET_GITHUB_APP_INSTALLATION_ID|CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE)=' "$env_file" || true)
+fi
+
+[[ -n "$app_id" && -n "$install_id" && -n "$key_path" ]] || { echo "ERROR: --app-id, --install-id, and --key-path are required" >&2; exit 2; }
+[[ -f "$key_path" ]] || { echo "ERROR: private key file not found: $key_path" >&2; exit 2; }
+
+# Generate JWT
+now=$(date -u +%s)
+exp=$((now + 540))  # 9 minutes (GitHub max is 10, leave buffer)
+header='{"alg":"RS256","typ":"JWT"}'
+payload=$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$now" "$exp" "$app_id")
+
+b64url() { openssl base64 -e | tr '+/' '-_' | tr -d '=\n'; }
+b64header=$(printf '%s' "$header" | b64url)
+b64payload=$(printf '%s' "$payload" | b64url)
+signature=$(printf '%s.%s' "$b64header" "$b64payload" | openssl dgst -sha256 -sign "$key_path" | b64url)
+jwt="${b64header}.${b64payload}.${signature}"
+
+# Exchange JWT for installation token
+response=$(curl -sS -X POST \
+  -H "Authorization: Bearer ${jwt}" \
+  -H "Accept: application/vnd.github+json" \
+  "https://api.github.com/app/installations/${install_id}/access_tokens" 2>/dev/null) || {
+  echo "ERROR: token exchange request failed" >&2
+  exit 2
+}
+
+token=$(printf '%s' "$response" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('token',''))" 2>/dev/null) || token=
+if [[ -z "$token" ]]; then
+  msg=$(printf '%s' "$response" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('message',''))" 2>/dev/null)
+  echo "ERROR: token exchange rejected: ${msg:-unknown}" >&2
+  exit 2
+fi
+
+printf '%s' "$token"

scripts/install-worker-controller.sh

@@ -114,6 +114,9 @@ unit_names=(
   ci-fleet-drift.service ci-fleet-drift.timer
 )
 timer_names=(ci-fleet-health.timer ci-fleet-cleanup.timer ci-fleet-drift.timer)
+optional_unit_names=(
+  ci-fleet-reconcile.service ci-fleet-reconcile.timer
+)
 
 temporary=$(mktemp -d)
 cleanup_temporary() {
@@ -718,13 +721,17 @@ install_systemd_units() {
   install -m 0644 "$source/host/systemd/ci-fleet-cleanup.timer" "$systemd_dir/"
   install -m 0644 "$source/host/systemd/ci-fleet-drift.service" "$systemd_dir/"
   install -m 0644 "$source/host/systemd/ci-fleet-drift.timer" "$systemd_dir/"
+  local unit
+  for unit in "${optional_unit_names[@]}"; do
+    [[ -f "$source/host/systemd/$unit" ]] && install -m 0644 "$source/host/systemd/$unit" "$systemd_dir/"
+  done
   systemctl daemon-reload
 }
 
 remove_systemd_units() {
   systemctl disable --now "${timer_names[@]}" >/dev/null 2>&1 || true
   local unit
-  for unit in "${unit_names[@]}"; do rm -f "$systemd_dir/$unit"; done
+  for unit in "${unit_names[@]}" "${optional_unit_names[@]}"; do rm -f "$systemd_dir/$unit"; done
   systemctl daemon-reload
 }
 
@@ -803,6 +810,12 @@ PY
   chmod 0600 "$staged_state"
   mv -f "$staged_state" "$state_file"
   systemctl enable --now "${timer_names[@]}" >/dev/null
+  local opt_timer
+  for opt_timer in "${optional_unit_names[@]}"; do
+    case "$opt_timer" in *.timer)
+      systemctl enable --now "$opt_timer" >/dev/null 2>&1 || true
+    ;; esac
+  done
 }
 
 restore_systemd_snapshot() {
@@ -811,6 +824,9 @@ restore_systemd_snapshot() {
   for unit in "${unit_names[@]}"; do
     [[ ! -f "$checkpoint_dir/systemd/$unit" ]] || install -m 0644 "$checkpoint_dir/systemd/$unit" "$systemd_dir/$unit" || failed=1
   done
+  for unit in "${optional_unit_names[@]}"; do
+    [[ ! -f "$checkpoint_dir/systemd/$unit" ]] || install -m 0644 "$checkpoint_dir/systemd/$unit" "$systemd_dir/$unit" || failed=1
+  done
   systemctl daemon-reload || failed=1
   for timer in "${timer_names[@]}"; do
     if grep -Fxq "$timer" "$checkpoint_dir/enabled-timers"; then systemctl enable "$timer" >/dev/null || failed=1; else systemctl disable "$timer" >/dev/null 2>&1 || true; fi