docs: bound GitHub App lifecycle safety guidance

docs/GITHUB-APP-SETUP.md

@@ -44,11 +44,49 @@ Nothing else. No `write` on contents, no actions, no administration. If the
 controller ever needs more, that is a reviewed design change, not a settings
 tweak.
 
-## 3. Generate the private key
+## 3. Generate and transfer the private key
 
 On the app page: Private keys → Generate a private key. GitHub downloads one
-PEM. Store it only on the controller host, root-owned `0600`. It is never
-committed, printed, or copied elsewhere — see [SECRETS.md](SECRETS.md).
+PEM. The normal manual workflow in this guide stores it on the controller's
+local filesystem at `/etc/ci-fleet/secrets/github-app.pem`, owned by root with
+mode `0600`.
+
+GitHub's browser download is necessarily present briefly on a controlled
+management workstation. Choose a fresh temporary directory outside synchronized,
+indexed, and backed-up locations, restrict the downloaded file to the operator
+immediately, and transfer it at once. This is transient handling, not approved
+long-term credential storage; never claim that the key was absent from the
+workstation.
+
+Before any key bytes arrive, create the controller directory as root-owned mode
+`0700` and pre-create the destination as a root-owned regular file with mode
+`0600`. Initial setup uses only the active path:
+
+```bash
+sudo install -d -o root -g root -m 0700 /etc/ci-fleet/secrets
+sudo install -o root -g root -m 0600 /dev/null \
+  /etc/ci-fleet/secrets/github-app.pem
+```
+
+Verify the directory and destination ownership, type, and mode without reading
+the content. Then use an authenticated encrypted channel to stream into that
+already secured file; the transfer must not replace it with a default-mode node.
+Keep key bytes out of tracing, logs, stdout, process arguments, Git, issues, and
+PRs. Compare a SHA-256 digest at both ends without printing file content, then
+verify the destination again.
+
+Delete the workstation copy immediately after authenticated transfer and those
+transfer checks succeed, before token, reconciliation, health, or convergence
+checks. Stop if transfer verification or local deletion fails. The PEM must
+never be committed or printed; see [SECRETS.md](SECRETS.md).
+
+This manual workflow permits exactly two host-local files: the active path above
+and `/etc/ci-fleet/secrets/github-app.next.pem` while rotating. It does not cover
+other custom paths, symlinks, or an external secret manager's import, rotation,
+or deletion lifecycle. Those cases require provider-specific tested automation.
+Do not improvise them from these Markdown examples;
+[issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) tracks that
+automation.
 
 ## 4. Install the app
 
@@ -68,7 +106,7 @@ Record from the installation page URL and app page:
 ```bash
 CI_FLEET_GITHUB_APP_CLIENT_ID=<client id>
 CI_FLEET_GITHUB_APP_INSTALLATION_ID=<installation id>
-CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=<absolute path to the PEM>
+CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=/etc/ci-fleet/secrets/github-app.pem
 ```
 
 The controller exchanges a short-lived JWT signed with the PEM for an
@@ -77,18 +115,26 @@ stored.
 
 ## 6. Verify
 
-On the controller host:
+The token helper writes a token to stdout. Every verification invocation must
+redirect stdout to `/dev/null`; exit status alone is the result.
+
+Before installation, use only a reviewed checkout at the intended immutable
+commit. Confirm `git rev-parse HEAD` is that commit, ensure the checkout is
+clean, and run the checkout's helper:
 
 ```bash
-sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh \
-  --env-file /etc/ci-fleet/host.env
+sudo ./scripts/github-app-token.sh \
+  --env-file /etc/ci-fleet/host.env >/dev/null
 ```
 
-Prints nothing secret; exit 0 means JWT signing and token exchange work.
-Then a check-only reconcile validates the full fetch path without applying
-anything:
+**Stop** if the checkout or commit is not the reviewed source, the checkout is
+dirty, or the command fails. Do not substitute a downloaded standalone script.
+
+After installation, use the installed manager rather than a working tree:
 
 ```bash
+sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh \
+  --env-file /etc/ci-fleet/host.env >/dev/null
 sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only
 ```
 
@@ -98,15 +144,59 @@ sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only
 
 ## Troubleshooting
 
-| Symptom | Cause |
-| --- | --- |
-| `Repository not found` on fetch | repository not in the installation's selected list |
-| 403 on content API after granting permission | permission change saved on the app but not yet accepted on the installation — reopen the installation page and approve the pending permission request |
-| 401 on token exchange | wrong client ID, installation ID, or PEM path in `host.env` |
-
-## Rotation and removal
-
-- New controller: new app. Do not share one app across controllers.
-- Rotate: generate a new key, update the PEM path, delete the old key.
-- Retire: uninstall the app from the organization. The host keeps no usable
-  credential.
+- `Repository not found` on fetch: the repository is not in the installation's
+  selected list.
+- 403 on the content API after granting permission: the installation has not
+  accepted the app's pending permission change. Reopen the installation page
+  and approve it.
+- 401 on token exchange: `host.env` has the wrong client ID, installation ID,
+  or PEM path.
+
+## Rotation
+
+Use this ordered safety checklist for the normal host-local root-owned PEM
+workflow. It is a set of gates, not a copy-and-paste shell program.
+
+1. Generate a new GitHub key and transfer, verify, and install it as described
+   above at the one approved replacement path,
+   `/etc/ci-fleet/secrets/github-app.next.pem`. Pre-create it as root-owned
+   `0600` inside the root-owned `0700` directory before transfer. Keep the old
+   key and `/etc/ci-fleet/secrets/github-app.pem` active for rollback, and delete
+   the workstation copy before continuing.
+2. Update the protected controller identity configuration to select the new PEM.
+3. Activate the new key and require the installed manager's token verification,
+   reconciliation, health check, and installed-state convergence check all to
+   succeed with the new key.
+4. Confirm the controller remains healthy and converged after a fresh check.
+5. Only then revoke the old key in GitHub. Remove its exact old controller PEM
+   only after revocation is confirmed. Retain the now-active replacement PEM;
+   never use a wildcard in the shared secrets directory.
+
+**Stop before old-key revocation or deletion** if new-key activation,
+reconciliation, health, or convergence is incomplete or fails. Restore the old
+configuration while its key and old PEM remain valid. The approved replacement
+file is not cleanup residue after successful activation; it is the selected
+active key. Further rotation, canonical-path normalization, custom paths,
+symlinks, and secret managers require the tested automation tracked by
+[issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27); do not
+adapt this checklist into ad hoc shell.
+
+## Retirement
+
+1. In the reviewed private schema-v3 desired state, drain the controller and
+   converge that state; verify zero managed runners and zero effective capacity.
+2. Revoke every key for this controller in GitHub and uninstall its GitHub App.
+3. Confirm the controller can no longer authenticate.
+4. Remove the controller's PEM and its local GitHub App identity state, including
+   the protected client ID, installation ID, and PEM-path configuration.
+5. Through review, permanently disable or remove the controller declaration,
+   update lifecycle and capacity inventory as appropriate, and converge the
+   reviewed private desired state. Keep private identifiers out of this public
+   repository.
+6. Verify that no usable controller credential or identity state remains and
+   that the authoritative desired state no longer declares active capacity.
+
+**Stop and preserve evidence** if revocation cannot be confirmed or if the
+normal host-local files cannot be identified safely. Arbitrary paths, symlinks,
+and secret-manager lifecycle operations require the tested automation tracked
+by issue #27; operators must not improvise removal from Markdown examples.

scripts/test_quickstart.py

@@ -16,4 +16,28 @@
 assert quickstart.index("Cancel every queued job") < quickstart.index("3. Authorize the repository")
 assert "PROJECT_PREFIX=" not in raw_quickstart
 assert "managed controller managed controller" not in quickstart
+
+raw_app_setup = (Path(__file__).resolve().parents[1] / "docs" / "GITHUB-APP-SETUP.md").read_text()
+app_setup = " ".join(raw_app_setup.split())
+app_safety_contract = (
+    "sudo ./scripts/github-app-token.sh",
+    "sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh",
+    "Every verification invocation must redirect stdout to `/dev/null`",
+    "/etc/ci-fleet/secrets/github-app.next.pem",
+    "controlled management workstation",
+    "Before any key bytes arrive",
+    "Delete the workstation copy immediately after",
+    "before token, reconciliation, health, or convergence checks",
+    "new-key activation, reconciliation, health, or convergence",
+    "Revoke every key for this controller in GitHub",
+    "disable or remove the controller declaration",
+    "Remove the controller's PEM and its local GitHub App identity state",
+    "operators must not improvise removal from Markdown examples",
+    "issue #27",
+)
+for text in app_safety_contract:
+    assert text in app_setup, f"GitHub App safety contract missing: {text}"
+assert raw_app_setup.count("--env-file /etc/ci-fleet/host.env >/dev/null") == 2
+assert "LOCAL_PEMS=" not in raw_app_setup
+assert "declare -a" not in raw_app_setup
 print("quickstart_contract=PASS")