From b4d37aae43a192e11f774086bf3aa18a9d43e8a5 Mon Sep 17 00:00:00 2001 From: Nickfost <1572453+Nickfost@users.noreply.github.com> Date: Thu, 30 Jul 2026 19:13:28 -0500 Subject: [PATCH 1/2] docs: bound GitHub App lifecycle guidance --- docs/GITHUB-APP-SETUP.md | 96 +++++++++++++++++++++++++++++--------- scripts/test_quickstart.py | 19 ++++++++ 2 files changed, 93 insertions(+), 22 deletions(-) diff --git a/docs/GITHUB-APP-SETUP.md b/docs/GITHUB-APP-SETUP.md index 7ada9610..241d28b4 100644 --- a/docs/GITHUB-APP-SETUP.md +++ b/docs/GITHUB-APP-SETUP.md @@ -44,11 +44,26 @@ Nothing else. No `write` on contents, no actions, no administration. If the controller ever needs more, that is a reviewed design change, not a settings tweak. -## 3. Generate the private key +## 3. Generate and transfer the private key On the app page: Private keys → Generate a private key. GitHub downloads one -PEM. Store it only on the controller host, root-owned `0600`. It is never -committed, printed, or copied elsewhere — see [SECRETS.md](SECRETS.md). +PEM. The normal manual workflow in this guide stores it on the controller's +local filesystem at `/etc/ci-fleet/secrets/github-app.pem`, owned by root with +mode `0600`. + +Transfer the PEM to a fresh operator-owned temporary file on the controller +using an encrypted channel such as SSH. Compare a SHA-256 digest at both ends, +then install the verified file with `sudo install -o root -g root -m 0600`. +Remove the controller's temporary copy. Delete the workstation copy **only +after** the encrypted transfer, digest comparison, installation, ownership, +and mode checks have all succeeded. The PEM must never be committed or printed; +see [SECRETS.md](SECRETS.md). + +This manual example does not cover arbitrary custom paths, symlinks, or an +external secret manager's import, rotation, or deletion lifecycle. Those cases +require provider-specific tested automation. Do not improvise them from these +Markdown examples; [issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) +tracks that automation. ## 4. Install the app @@ -68,7 +83,7 @@ Record from the installation page URL and app page: ```bash CI_FLEET_GITHUB_APP_CLIENT_ID= CI_FLEET_GITHUB_APP_INSTALLATION_ID= -CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE= +CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=/etc/ci-fleet/secrets/github-app.pem ``` The controller exchanges a short-lived JWT signed with the PEM for an @@ -77,18 +92,26 @@ stored. ## 6. Verify -On the controller host: +The token helper writes a token to stdout. Every verification invocation must +redirect stdout to `/dev/null`; exit status alone is the result. + +Before installation, use only a reviewed checkout at the intended immutable +commit. Confirm `git rev-parse HEAD` is that commit, ensure the checkout is +clean, and run the checkout's helper: ```bash -sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh \ - --env-file /etc/ci-fleet/host.env +sudo ./scripts/github-app-token.sh \ + --env-file /etc/ci-fleet/host.env >/dev/null ``` -Prints nothing secret; exit 0 means JWT signing and token exchange work. -Then a check-only reconcile validates the full fetch path without applying -anything: +**Stop** if the checkout or commit is not the reviewed source, the checkout is +dirty, or the command fails. Do not substitute a downloaded standalone script. + +After installation, use the installed manager rather than a working tree: ```bash +sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh \ + --env-file /etc/ci-fleet/host.env >/dev/null sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only ``` @@ -98,15 +121,44 @@ sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only ## Troubleshooting -| Symptom | Cause | -| --- | --- | -| `Repository not found` on fetch | repository not in the installation's selected list | -| 403 on content API after granting permission | permission change saved on the app but not yet accepted on the installation — reopen the installation page and approve the pending permission request | -| 401 on token exchange | wrong client ID, installation ID, or PEM path in `host.env` | - -## Rotation and removal - -- New controller: new app. Do not share one app across controllers. -- Rotate: generate a new key, update the PEM path, delete the old key. -- Retire: uninstall the app from the organization. The host keeps no usable - credential. +- `Repository not found` on fetch: the repository is not in the installation's + selected list. +- 403 on the content API after granting permission: the installation has not + accepted the app's pending permission change. Reopen the installation page + and approve it. +- 401 on token exchange: `host.env` has the wrong client ID, installation ID, + or PEM path. + +## Rotation + +Use this ordered safety checklist for the normal host-local root-owned PEM +workflow. It is a set of gates, not a copy-and-paste shell program. + +1. Generate a new GitHub key and transfer, verify, and install it as described + above at a new root-owned `0600` host-local path. Keep the old key active. +2. Update the protected controller identity configuration to select the new PEM. +3. Activate the new key and require the installed manager's token verification, + reconciliation, health check, and installed-state convergence check all to + succeed with the new key. +4. Confirm the controller remains healthy and converged after a fresh check. +5. Only then revoke the old key in GitHub and remove the old controller PEM. + +**Stop before old-key revocation or deletion** if new-key activation, +reconciliation, health, or convergence is incomplete or fails. Restore the old +configuration while its key remains valid. Custom paths, symlinks, and secret +managers must use the provider-specific tested automation tracked by issue #27; +do not adapt this checklist into ad hoc shell. + +## Retirement + +1. Drain and stop the controller through its reviewed operational procedure. +2. Revoke every key for this controller in GitHub and uninstall its GitHub App. +3. Confirm the controller can no longer authenticate. +4. Remove the controller's PEM and its local GitHub App identity state, including + the protected client ID, installation ID, and PEM-path configuration. +5. Verify that no usable controller credential or identity state remains. + +**Stop and preserve evidence** if revocation cannot be confirmed or if the +normal host-local files cannot be identified safely. Arbitrary paths, symlinks, +and secret-manager lifecycle operations require the tested automation tracked +by issue #27; operators must not improvise removal from Markdown examples. diff --git a/scripts/test_quickstart.py b/scripts/test_quickstart.py index 55a82a24..ef3136d0 100644 --- a/scripts/test_quickstart.py +++ b/scripts/test_quickstart.py @@ -16,4 +16,23 @@ assert quickstart.index("Cancel every queued job") < quickstart.index("3. Authorize the repository") assert "PROJECT_PREFIX=" not in raw_quickstart assert "managed controller managed controller" not in quickstart + +raw_app_setup = (Path(__file__).resolve().parents[1] / "docs" / "GITHUB-APP-SETUP.md").read_text() +app_setup = " ".join(raw_app_setup.split()) +app_safety_contract = ( + "sudo ./scripts/github-app-token.sh", + "sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh", + "Every verification invocation must redirect stdout to `/dev/null`", + "Delete the workstation copy **only after**", + "new-key activation, reconciliation, health, or convergence", + "Revoke every key for this controller in GitHub", + "Remove the controller's PEM and its local GitHub App identity state", + "operators must not improvise removal from Markdown examples", + "issue #27", +) +for text in app_safety_contract: + assert text in app_setup, f"GitHub App safety contract missing: {text}" +assert raw_app_setup.count("--env-file /etc/ci-fleet/host.env >/dev/null") == 2 +assert "LOCAL_PEMS=" not in raw_app_setup +assert "declare -a" not in raw_app_setup print("quickstart_contract=PASS") From d8fdf866f97284f1635e3e1abfafd46b347d276d Mon Sep 17 00:00:00 2001 From: Nickfost <1572453+Nickfost@users.noreply.github.com> Date: Thu, 30 Jul 2026 23:56:06 -0500 Subject: [PATCH 2/2] docs: close bounded GitHub App lifecycle gaps --- docs/GITHUB-APP-SETUP.md | 78 ++++++++++++++++++++++++++++---------- scripts/test_quickstart.py | 7 +++- 2 files changed, 64 insertions(+), 21 deletions(-) diff --git a/docs/GITHUB-APP-SETUP.md b/docs/GITHUB-APP-SETUP.md index 241d28b4..d32990bb 100644 --- a/docs/GITHUB-APP-SETUP.md +++ b/docs/GITHUB-APP-SETUP.md @@ -51,19 +51,42 @@ PEM. The normal manual workflow in this guide stores it on the controller's local filesystem at `/etc/ci-fleet/secrets/github-app.pem`, owned by root with mode `0600`. -Transfer the PEM to a fresh operator-owned temporary file on the controller -using an encrypted channel such as SSH. Compare a SHA-256 digest at both ends, -then install the verified file with `sudo install -o root -g root -m 0600`. -Remove the controller's temporary copy. Delete the workstation copy **only -after** the encrypted transfer, digest comparison, installation, ownership, -and mode checks have all succeeded. The PEM must never be committed or printed; -see [SECRETS.md](SECRETS.md). - -This manual example does not cover arbitrary custom paths, symlinks, or an -external secret manager's import, rotation, or deletion lifecycle. Those cases -require provider-specific tested automation. Do not improvise them from these -Markdown examples; [issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) -tracks that automation. +GitHub's browser download is necessarily present briefly on a controlled +management workstation. Choose a fresh temporary directory outside synchronized, +indexed, and backed-up locations, restrict the downloaded file to the operator +immediately, and transfer it at once. This is transient handling, not approved +long-term credential storage; never claim that the key was absent from the +workstation. + +Before any key bytes arrive, create the controller directory as root-owned mode +`0700` and pre-create the destination as a root-owned regular file with mode +`0600`. Initial setup uses only the active path: + +```bash +sudo install -d -o root -g root -m 0700 /etc/ci-fleet/secrets +sudo install -o root -g root -m 0600 /dev/null \ + /etc/ci-fleet/secrets/github-app.pem +``` + +Verify the directory and destination ownership, type, and mode without reading +the content. Then use an authenticated encrypted channel to stream into that +already secured file; the transfer must not replace it with a default-mode node. +Keep key bytes out of tracing, logs, stdout, process arguments, Git, issues, and +PRs. Compare a SHA-256 digest at both ends without printing file content, then +verify the destination again. + +Delete the workstation copy immediately after authenticated transfer and those +transfer checks succeed, before token, reconciliation, health, or convergence +checks. Stop if transfer verification or local deletion fails. The PEM must +never be committed or printed; see [SECRETS.md](SECRETS.md). + +This manual workflow permits exactly two host-local files: the active path above +and `/etc/ci-fleet/secrets/github-app.next.pem` while rotating. It does not cover +other custom paths, symlinks, or an external secret manager's import, rotation, +or deletion lifecycle. Those cases require provider-specific tested automation. +Do not improvise them from these Markdown examples; +[issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) tracks that +automation. ## 4. Install the app @@ -135,28 +158,43 @@ Use this ordered safety checklist for the normal host-local root-owned PEM workflow. It is a set of gates, not a copy-and-paste shell program. 1. Generate a new GitHub key and transfer, verify, and install it as described - above at a new root-owned `0600` host-local path. Keep the old key active. + above at the one approved replacement path, + `/etc/ci-fleet/secrets/github-app.next.pem`. Pre-create it as root-owned + `0600` inside the root-owned `0700` directory before transfer. Keep the old + key and `/etc/ci-fleet/secrets/github-app.pem` active for rollback, and delete + the workstation copy before continuing. 2. Update the protected controller identity configuration to select the new PEM. 3. Activate the new key and require the installed manager's token verification, reconciliation, health check, and installed-state convergence check all to succeed with the new key. 4. Confirm the controller remains healthy and converged after a fresh check. -5. Only then revoke the old key in GitHub and remove the old controller PEM. +5. Only then revoke the old key in GitHub. Remove its exact old controller PEM + only after revocation is confirmed. Retain the now-active replacement PEM; + never use a wildcard in the shared secrets directory. **Stop before old-key revocation or deletion** if new-key activation, reconciliation, health, or convergence is incomplete or fails. Restore the old -configuration while its key remains valid. Custom paths, symlinks, and secret -managers must use the provider-specific tested automation tracked by issue #27; -do not adapt this checklist into ad hoc shell. +configuration while its key and old PEM remain valid. The approved replacement +file is not cleanup residue after successful activation; it is the selected +active key. Further rotation, canonical-path normalization, custom paths, +symlinks, and secret managers require the tested automation tracked by +[issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27); do not +adapt this checklist into ad hoc shell. ## Retirement -1. Drain and stop the controller through its reviewed operational procedure. +1. In the reviewed private schema-v3 desired state, drain the controller and + converge that state; verify zero managed runners and zero effective capacity. 2. Revoke every key for this controller in GitHub and uninstall its GitHub App. 3. Confirm the controller can no longer authenticate. 4. Remove the controller's PEM and its local GitHub App identity state, including the protected client ID, installation ID, and PEM-path configuration. -5. Verify that no usable controller credential or identity state remains. +5. Through review, permanently disable or remove the controller declaration, + update lifecycle and capacity inventory as appropriate, and converge the + reviewed private desired state. Keep private identifiers out of this public + repository. +6. Verify that no usable controller credential or identity state remains and + that the authoritative desired state no longer declares active capacity. **Stop and preserve evidence** if revocation cannot be confirmed or if the normal host-local files cannot be identified safely. Arbitrary paths, symlinks, diff --git a/scripts/test_quickstart.py b/scripts/test_quickstart.py index ef3136d0..a008a268 100644 --- a/scripts/test_quickstart.py +++ b/scripts/test_quickstart.py @@ -23,9 +23,14 @@ "sudo ./scripts/github-app-token.sh", "sudo /opt/ci-fleet/manager/current/scripts/github-app-token.sh", "Every verification invocation must redirect stdout to `/dev/null`", - "Delete the workstation copy **only after**", + "/etc/ci-fleet/secrets/github-app.next.pem", + "controlled management workstation", + "Before any key bytes arrive", + "Delete the workstation copy immediately after", + "before token, reconciliation, health, or convergence checks", "new-key activation, reconciliation, health, or convergence", "Revoke every key for this controller in GitHub", + "disable or remove the controller declaration", "Remove the controller's PEM and its local GitHub App identity state", "operators must not improvise removal from Markdown examples", "issue #27",