feat: add deployer host installer (#22) - #69
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 73f45a1a10
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed8c5e6351
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 777f8c534c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19a0c4e200
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37d29628fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 773c3ae74c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61b3d0eace
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ddea4d953
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: deb781caef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Permit the conventional /var/lock -> /run/lock symlink layout instead of rejecting the canonical parent path - Write a structured failed audit record through fd 8 for every failure after approval consumption, including post-adapter snapshot/publication failures, with the failing phase recorded - Reject unknown approval provider values instead of silently skipping GitHub capability validation - Validate the retained last-known-good state schema and cross-match it with the retained policy before rollback starts - Allow --repair to restore a missing or drifted owned active policy by skipping only the untrusted old-policy health probe while retaining candidate validation and post-activation health gates - Fix the repair transaction-failure fixture to fail the candidate health call under the new single-health-call sequence and assert the exact call count
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Nickfost
left a comment
Nickfost
left a comment
There was a problem hiding this comment.
[lane-a:pr69:publish-pending-dispositions] Publishing 16 previously drafted accepted/fixed evidence replies. Each corresponding Codex finding already has Nickfost +1; current exact-head CI is green and Codex found no major issues at 515b071.
|
Prepared isolated deployer proof completed on exact head Real Debian 13/systemd/Docker-host lifecycle covered install and unchanged rerun, deploy, exact request/approval/checkpoint binding, application-adapter validate/health/deploy/cleanup/rollback operations, immutable source/artifact state, failed cross-tier request preservation, upgrade, drain/refusal/resume, rollback to the prior application/core state, repeated uninstall, independent demo-b deployment, and complete cleanup. The proof exposed Debian’s canonical |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ad48c75c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2ad48c7 to
a07990f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a07990fcd0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 024614b803
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4fd48c36a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08e89046df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4d0bfb7bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4104f96ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| reject_deployer_role() { | ||
| local path | ||
| for path in "$root/etc/systemd/system/ci-fleet-deployer.service" "$root/etc/ci-fleet-deployer" "$root/var/lib/ci-fleet-deployer"; do |
There was a problem hiding this comment.
On a partially removed or drifted deployer host where only a secondary unit such as ci-fleet-deployer-cleanup.timer remains and the main service/state directories are absent, this check passes and the status receiver can be installed alongside deployer state. Check the complete ci-fleet-deployer* unit family, as the controller-side admission check does, so partial deployer state also fails closed and the host roles remain separable.
AGENTS.md reference: AGENTS.md:L33-L35
Useful? React with 👍 / 👎.
| perform_drain() { | ||
| if active_deployment; then block 'active deployment prevents drain'; fi | ||
| secure_directory "$state_root" 700 1 |
There was a problem hiding this comment.
When --drain is run on an uninstalled or ordinary-controller host that has a deployer config but no managed deployer state, this unconditional creation of state_root publishes a deployer role marker without first checking convergence or rejecting other roles. The host is then wedged in mixed-role state: controller convergence rejects /var/lib/ci-fleet-deployer, while deployer installation rejects the controller. Require an existing managed deployer installation and role isolation before creating the drain marker.
AGENTS.md reference: AGENTS.md:L33-L35
Useful? React with 👍 / 👎.
|
|
||
| reject_deployer_host() { | ||
| # A deployer host is a separate role; never run controller mutations on it. | ||
| if compgen -G "$(root_path /etc/systemd/system)/ci-fleet-deployer*" >/dev/null || [[ -e "$(root_path /var/lib/ci-fleet-deployer)" || -e "$(root_path /etc/ci-fleet-deployer)" ]]; then |
There was a problem hiding this comment.
If /var/lib/ci-fleet-deployer or /etc/ci-fleet-deployer is a dangling symlink—for example after an incomplete cleanup—-e is false, so this admission check allows controller convergence and creates a mixed-role host. Other role checks in this change explicitly test -L; do the same here so unsafe deployer markers fail closed before controller mutation.
AGENTS.md reference: AGENTS.md:L33-L35
Useful? React with 👍 / 👎.
| if [[ ${audit_pending:-0} == 1 && -n ${consumed_marker:-} && ! -e "$consumed_marker" ]]; then | ||
| if [[ -e "$consumed_root" || -L "$consumed_root" ]] && [[ ! -d "$consumed_root" || -L "$consumed_root" || $(stat -c '%u:%a' "$consumed_root" 2>/dev/null) != "$expected_uid:700" ]]; then | ||
| rm -rf -- "$consumed_root" 2>/dev/null || status=1 | ||
| fi | ||
| [[ -e "$consumed_root" ]] || install -d -m 0700 "$consumed_root" 2>/dev/null || status=1 | ||
| if [[ -d "$consumed_root" && ! -L "$consumed_root" ]]; then | ||
| install -m 0600 /dev/null "$consumed_marker" 2>/dev/null || status=1 |
There was a problem hiding this comment.
When an adapter recursively removes consumed-requests or the writable state root after multiple deployment attempts, this recovery recreates only the marker for the request currently being processed. Every older consumption marker is lost, and because replay validation consults only these markers plus the single last-request.conf, an older non-last request can later be accepted again when its protected policy/evidence is restored. Preserve and restore the complete consumed-request set outside the adapter-writable boundary rather than reconstructing only the current marker.
Useful? React with 👍 / 👎.
| if [[ -z ${CI_FLEET_DEPLOYER_REEXEC:-} ]]; then | ||
| exec 9<"$lock_dir" | ||
| flock -n 9 || die 'another deployer operation is running' | ||
| else | ||
| # The re-executed process inherits the locked descriptor 9 from its parent; | ||
| # reopening it would drop the lock during the handoff. | ||
| : |
There was a problem hiding this comment.
When the runtime starts with any nonempty CI_FLEET_DEPLOYER_REEXEC value but without the intended inherited descriptor 9, this branch skips flock entirely and deploy, cleanup, health, or drain can overlap another operation. Validate that fd 9 resolves to the expected lock directory and still carries the lock before accepting the internal re-exec flag; otherwise reject the invocation or acquire the lock normally.
AGENTS.md reference: AGENTS.md:L35-L35
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 48ddebe7be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| publish_deployed_snapshot "$active_policy" "$state_file" || return | ||
| rm -f "$previous_state" "$previous_policy" || return |
There was a problem hiding this comment.
When the root rollback adapter creates a safe commit marker but changes /opt/ci-fleet-deployer/current or any deployed unit before returning, finalize_committed_rollback validates only the active policy, state, adapter, and credential reference; it never verifies current_matches or units_match. These lines then publish the snapshot and remove the last-known-good pair before retiring the journal, permanently accepting a core that may not match the rolled-back application. Revalidate or restore the protected rolled-back pointer and units before finalization.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| fi | ||
| if [[ -n "$retired_snapshot" && -d "$deployed_root/$retired_snapshot" && ! -L "$deployed_root/$retired_snapshot" ]]; then rm -rf -- "${deployed_root:?}/$retired_snapshot"; fi | ||
| sync -f "$deployed_root" 2>/dev/null || sync "$deployed_root" 2>/dev/null || die 'retired deployed snapshot is not durable' | ||
| audit_pending=0 |
There was a problem hiding this comment.
When a successful deployment adapter truncates or removes active-policy.conf or install-state.json while leaving the request and audit boundaries intact, no success-path check detects that mutation. Clearing audit_pending here prevents deploy_exit lines 172-175 from restoring the protected policy/state copies, so the service reports success with the new application and deployed snapshot but subsequent health and deployment operations reject the missing or corrupted active state. Restore and verify those protected files before disabling recovery.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| [[ -z "$retired_snapshot" || "$retired_snapshot" =~ ^\.snapshot\.[A-Za-z0-9._-]+$ ]] || die 'current deployed snapshot pointer is unsafe' | ||
| [[ -z ${CI_FLEET_DEPLOYER_TEST_SIGNAL_SELF:-} || $testing != 1 ]] || kill -"$CI_FLEET_DEPLOYER_TEST_SIGNAL_SELF" $$ | ||
| if [[ -f "$request" && ! -L "$request" ]] && cmp -s "$request_snapshot" "$request"; then rm -f "$request"; fi | ||
| mv -Tf "$request_snapshot" "$last_request" |
There was a problem hiding this comment.
When the state and audit roots are on separate filesystems, a power loss after this rename and the success-audit fsync at line 612 but before the EXIT handler syncs state_root can discard last-request.conf even though the application change, deployed pointer, consumed marker, and success audit are already durable. On a first completed deployment this makes perform_rollback fail its mandatory completed-deployment gate, leaving the changed application without the supported rollback path. Fsync the marker and its parent directory before committing the success record or disarming recovery.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
1 participant
Summary
--check,--install,--upgrade,--repair,--rollback,--drain, and--uninstallmodesCloses #22
Acceptance matrix
scripts/install-deployer.shrequires exactly one mode and--configscripts/test-install-deployer.sh: missing/duplicate mode, check/install/upgrade/repair/rollback/drain/uninstallrequire_hostsupports Debian 12/13 and Ubuntu 22.04/24.04; Compose is conditionalKEY=valueparser; unknown, duplicate, malformed, missing, and ambiguous fields failscripts/test-deployer-units.shruns realsystemd-analyze verifyin a temporary rootsystemd-inhibitREPORTline carries action, result, environment, target, version, digest, health, changed, rollback, and next actionCHANGED,NO_CHANGE,BLOCKED,FAILED, rollback-available, and canary checksTrust boundaries
The deployer is a dedicated role. It never registers a runner and does not receive pull-request jobs. Ordinary workers/controllers receive no deployment credentials, deployer Docker socket, production authority, or deployment route. The installer rejects mixed-role hosts and unrelated Docker state instead of adopting it.
The systemd runtime runs as root because the Docker socket is root-equivalent. Units apply
NoNewPrivileges,ProtectSystem=strict, private temporary storage, explicit writable paths, and bounded operation timeouts, but do not pretend Docker access is unprivileged. The application adapter is a root-owned mode-0700 regular file pinned by SHA-256. Core defines its operation/timeout/redaction contract; application-specific staging, deployment, health, rollback, credential resolution, and exact-label cleanup remain application-owned.Secret values are never CLI fields. File references stay under the protected configuration tree; external references are resolved by the pinned adapter. Reports and audit logs contain only validated identities, commits, digests, and approval metadata. Network hosts are fed to DNS/HTTPS probes over standard input rather than command arguments.
GitHub-native Environment protection is optional.
github-environmentrequires separate exact-head capability evidence.manual-exact-headandexternal-exact-headprovide fail-closed alternatives for private repositories where the available GitHub plan does not supply the needed protection. A credential store by itself is never treated as approval.Operator examples
On a prepared isolated Linux deployer host, after creating the fictional root-owned configuration/evidence/credential files described in
docs/DEPLOYER-HOST.md:Rollback restores the last-known-good active state but deliberately leaves operator-owned desired configuration/evidence untouched; its report names the required reconciliation step before check.
Validation
scripts/test-install-deployer.shscripts/test-deployer-units.shshellcheck scripts/install-deployer.sh scripts/deployer-runtime.sh scripts/test-install-deployer.sh scripts/test-deployer-units.shscripts/validate.shpython3 scripts/scan_committed_secrets.pygit diff --checkNo host installation, Docker/systemd mutation, runner registration, infrastructure change, credential access, or production deployment was performed. Tests use an explicit temporary-root seam and mocked host commands. A prepared isolated deployment host still needs an operator proof of host prerequisites, application-adapter correctness, target policy, approval evidence, credentials, checkpoint evidence, deployment health, and rollback behavior before use.
Rollback and compatibility
The installer retains hashed core releases, active and last-known-good policy/state, audit logs, and host-local credentials. Failed candidate validation or activation restores the previous units, timer state, policy, state, and release pointer. Uninstall removes only the activation pointer and owned units/timers; retained data requires a separate explicit operator decision.
The public contract is application-neutral and Linux/Bash-only. Existing CI worker/controller behavior is unchanged.