Remote configuration reconciliation via GitHub App token

[Nickfost]

Summary

Add scripts/remote-reconcile.sh and scripts/github-app-token.sh so that the controller can fetch its desired-state configuration from a private desired-state configuration repository using short-lived GitHub App installation tokens — no PAT, no SSH, no inbound port.

What

• scripts/github-app-token.sh — generates a JWT from the existing app private key (openssl + curl, no new deps), exchanges it for a short-lived installation token.
• scripts/remote-reconcile.sh — fetches the default-branch HEAD of the desired-state repo, validates, checks drift, reconciles if changed, rolls back on failure. Supports --check-only and --no-op. Bounded retries (3). Sanitized logging (no keys/tokens in output).
• host/systemd/ci-fleet-reconcile.{service,timer} — periodic reconciliation every 5 minutes.
• scripts/install-worker-controller.sh — manages the new timer alongside existing health/cleanup/drift timers.
• scripts/test_remote_reconcile.py — 14 regression tests (auth failure, missing state, secret redaction, unit integrity).
• Docs updated — DESIRED-STATE.md and HOST-MAINTENANCE.md.

Design

┌──────────────────┐     HTTPS (JWT → install token)     ┌─────────────────────┐
│ Controller host  │ ──────────────────────────────────▶ │ github.com/api      │
│                  │                                     │                     │
│ remote-reconcile │     Authenticated git fetch         │ Private desired-    │
│ .sh              │ ──────────────────────────────────▶ │ state config repo   │
│                  │                                     │                     │
│                  │     validate + reconcile            │ fleet.json          │
│                  │ ◀────────────────────────────────── │                     │
└──────────────────┘                                     └─────────────────────┘
        │
        ▼
 /var/lib/ci-fleet/reconcile/state.json
 (desired_commit, applied_commit, health, status)

Prerequisite (blocking)

The controller's GitHub App needs contents: read on the private desired-state repository so the controller can fetch configuration over HTTPS. This is an org-level GitHub App permission change; exact installation details are tracked in the private configuration repository's rollout issue.

Testing

$ python3 -m unittest scripts.test_remote_reconcile -v
----------------------------------------------------------------------
Ran 14 tests in 2.3s
OK

[Nickfost]

@codex review

[Nickfost]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a93459d6ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Nickfost]

Addressed all 7 Codex findings in commit 589fb78:

• P1: auth_url now used for authenticated fetch
• P1: log_json passes message via argv (not source interpolation)
• P1: systemd unit uses After/Wants instead of Requires (reconciler runs even when drift.service fails)
• P1: durable config_repo identity preserved in install-state.json via fix_state_config_repo
• P1: reconcile units are optional in unit_names — backwards compat with older engine refs
• P1: rollback uses LKG direct re-apply, not checkpoint-latest (safe when no checkpoint was created)
• P2: token generation retries on transient failure within the bounded retry loop

@codex review

[Nickfost]

All 7 findings addressed in commit 589fb78 (head updated). All threads resolved. @codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 589fb78651

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Add reconciliation failures to fleet health monitoring

This adds a controller-lifecycle timer, but a repo-wide check of health.py shows that collect_snapshot monitors only the health, cleanup, and drift timers and only cleanup and drift services, with no consumer for reconciliation state. Consequently, a missing/disabled reconcile timer or persistent token and fetch failures can leave ordinary health reports and external heartbeats healthy indefinitely while reviewed desired state is no longer being applied; register this timer, service result, or reconcile state in the health contract.

Useful? React with 👍 / 👎.