Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions docs/ADDING-A-HOST.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The host must:
- use local or virtualized storage suitable for Docker's write load;
- have reliable DNS, time synchronization, and outbound HTTPS access to GitHub and required registries;
- permit local administration without requiring public inbound access;
- be disposable or recoverable from a documented build and backup procedure.
- declare its failure boundary: either **disposable** (rebuilt from reviewed Git-authored configuration and host-local identity re-provisioning, with no machine backup required) or **recoverable** (a documented build and backup procedure).

Runners sharing one Docker daemon share a security boundary. Add a host instead of increasing same-host concurrency when projects need stronger separation or when one site's failure must not stop the fleet.

Expand All @@ -61,15 +61,14 @@ Do not install Node, PHP, Python, Java, Composer, npm, database clients, or othe

Enable unattended security updates without automatic reboots. Configure Docker log rotation and host disk alerts. Follow [Host maintenance](HOST-MAINTENANCE.md).

## 3. Record backup and network readiness
## 3. Record recovery and network readiness

Before placing credentials on the host:

1. verify console or local recovery access;
2. verify the host has a unique name and address;
3. create a VM backup, snapshot, or equivalent recoverable baseline;
4. record the backup destination and successful timestamp outside this public repository;
5. verify the host can reach GitHub and required container/package registries.
3. for a **recoverable** host, create a VM backup, snapshot, or equivalent recoverable baseline and record the backup destination and successful timestamp outside this public repository; for a **disposable** controller, record that recovery is rebuilding the host from reviewed Git-authored desired state — no machine backup, snapshot, or tar archive is required or expected;
4. verify the host can reach GitHub and required container/package registries.

A remote site needs outbound connectivity; it does not need project-specific inbound access. If one location is offline, compatible jobs remain queued for another available host.

Expand Down
2 changes: 1 addition & 1 deletion docs/DESIRED-STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Legacy project-specific hosts remain until CI, promotion, and deployment no long

Before mutation, the installer records the prior rendered environment, installation metadata, runtime release, installer-manager release, and maintenance unit/timer state under `/var/lib/ci-fleet/checkpoints`. Each checkpoint is staged and atomically renamed with a completion marker; rollback ignores partial staging directories. Build and validation happen before the active release changes. A failed activation or health check drains the candidate, restores those artifacts, restarts the prior controller only when no managed runner is active, and verifies prior-release health before reporting rollback success. A host-local installer lock serializes every check and mutation. Runtime and installer-manager releases are staged on their respective target filesystems and renamed atomically so a failed copy cannot masquerade as an installed immutable release.

These controller checkpoints do not replace machine backups. Operators still create and verify VM snapshots, physical-host recovery media, or equivalent infrastructure backups according to their local policy.
Installer checkpoints and machine backups serve different failure classes. A checkpoint rolls back a single failed reconciliation. Recoverability of the machine itself is governed by each host's declared failure boundary (see [Adding a host](ADDING-A-HOST.md)): a disposable controller needs no machine backup at all — recovery is rebuilding from reviewed Git-authored desired state — while a non-disposable host follows its own documented local backup policy.

Monitoring thresholds, heartbeat endpoints, and backup hooks are host-local operational facts, not fleet desired state. Keep them in the protected file documented by [Fleet health monitoring](HEALTH-MONITORING.md); the installer preserves that file across upgrades and rollback.

Expand Down