Add fleet-wide host health monitoring

README.md

@@ -175,6 +175,7 @@ Supported deployment shapes include virtual machines, dedicated physical machine
 | Build a compatible project contract | [Project CI standard](docs/PROJECT-STANDARD.md) |
 | Verify project compliance | [Compliance checklist](docs/COMPLIANCE-CHECKLIST.md) |
 | Configure upgrades, cleanup, draining, and rebooting | [Host maintenance](docs/HOST-MAINTENANCE.md) |
+| Configure host health and external missed-heartbeat detection | [Fleet health monitoring](docs/HEALTH-MONITORING.md) |
 | Understand secret storage and injection | [Secrets model](docs/SECRETS.md) |
 | Use private fleet workers for a public project | [Public projects and private delivery](docs/PUBLIC-PRIVATE-CONFIGURATION.md) |
 | See planned work | [Roadmap](docs/ROADMAP.md) |

controller/Dockerfile

@@ -10,7 +10,8 @@ RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${CI_FLEET_
 
 FROM debian:13.6-slim
 ARG CI_FLEET_COMMIT=unknown
-LABEL org.opencontainers.image.revision="${CI_FLEET_COMMIT}"
+LABEL org.opencontainers.image.revision="${CI_FLEET_COMMIT}" \
+      io.randomdevelopment.ci-fleet.managed="true"
 RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/*
 COPY --from=build /out/ci-fleet-controller /usr/local/bin/ci-fleet-controller
 USER 65532:65532

docs/ADDING-A-HOST.md

@@ -122,7 +122,7 @@ Follow [Live pilot](LIVE-PILOT.md) for the complete proof and rollback.
 
 ## 8. Verify unattended operations
 
-The installer enables health, scoped cleanup, and pinned desired-state drift timers. Run each service once and inspect its journal as described in [Host maintenance](HOST-MAINTENANCE.md). Monitor controller state, disk thresholds, Docker health, last successful job, cleanup failures, drift, and pending reboot state.
+The installer enables health, scoped cleanup, and pinned desired-state drift timers. Run each service once and inspect its journal as described in [Host maintenance](HOST-MAINTENANCE.md). Configure redacted local checks and external missed-heartbeat detection as described in [Fleet health monitoring](HEALTH-MONITORING.md).
 
 The steady-state host should need no project-specific edits. Adding a project changes its repository, private fleet configuration, and GitHub runner-group policy—not this host.
 

docs/DESIRED-STATE.md

@@ -153,4 +153,6 @@ Before mutation, the installer records the prior rendered environment, installat
 
 These controller checkpoints do not replace machine backups. Operators still create and verify VM snapshots, physical-host recovery media, or equivalent infrastructure backups according to their local policy.
 
+Monitoring thresholds, heartbeat endpoints, and backup hooks are host-local operational facts, not fleet desired state. Keep them in the protected file documented by [Fleet health monitoring](HEALTH-MONITORING.md); the installer preserves that file across upgrades and rollback.
+
 This contract implements the engine portions of [issue #32](https://github.com/RandomDevelopment/ci-fleet/issues/32) and integrates the installer, documentation, phone-first bootstrap, and capacity work tracked by [#21](https://github.com/RandomDevelopment/ci-fleet/issues/21), [#24](https://github.com/RandomDevelopment/ci-fleet/issues/24), [#27](https://github.com/RandomDevelopment/ci-fleet/issues/27), and [#30](https://github.com/RandomDevelopment/ci-fleet/issues/30).

docs/HEALTH-MONITORING.md

@@ -0,0 +1,76 @@
+# Fleet health monitoring
+
+Every managed controller runs a redacted local health check every five minutes. An external monitor must also detect a host that cannot report because it is offline.
+
+## Local check
+
+```bash
+sudo /opt/ci-fleet/manager/current/scripts/healthcheck.sh
+sudo /opt/ci-fleet/manager/current/scripts/healthcheck.sh --json
+sudo cat /var/lib/ci-fleet/health/latest.json
+```
+
+Exit codes are `0` for healthy or intentional maintenance, `1` for warning, and `2` for unhealthy. The JSON schema is versioned and the latest result is replaced atomically. It contains controller identity, desired lifecycle state, status, timestamp, and redacted check results only.
+
+The check covers:
+
+- root and Docker filesystem space and inodes;
+- available memory, swap use, per-CPU load, and OOM evidence from the last 24 hours;
+- Docker availability, controller state/restarts, and configured versus effective capacity;
+- inactive, unhealthy, restarting, and stale fleet-labelled resources, including week-old build cache;
+- cleanup, drift, health, and update services/timers;
+- failed package state, pending reboot, and clock synchronization;
+- an optional host-local backup check;
+- optional outbound heartbeat delivery.
+
+It reports but never prunes, restarts, or repairs resources. Project source, logs, environment values, tokens, and private keys are never included.
+
+## Threshold overrides and hooks
+
+Defaults are intentionally conservative: disk and inode warning/critical at 80/90%, available memory warning/critical at 15/8%, sustained swap use under five-minute memory pressure warning/critical at 25/50%, per-CPU fifteen-minute load warning/critical at 1.0/1.5, and controller restart warning at 3.
+
+Optional overrides belong in `/etc/ci-fleet/monitoring.env`, owned by root with mode `0600`:
+
+```text
+CI_FLEET_HEALTH_DISK_WARN_PERCENT=80
+CI_FLEET_HEALTH_DISK_CRITICAL_PERCENT=90
+CI_FLEET_HEALTH_INODE_WARN_PERCENT=80
+CI_FLEET_HEALTH_INODE_CRITICAL_PERCENT=90
+CI_FLEET_HEALTH_MEMORY_WARN_AVAILABLE_PERCENT=15
+CI_FLEET_HEALTH_MEMORY_CRITICAL_AVAILABLE_PERCENT=8
+CI_FLEET_HEALTH_SWAP_WARN_PERCENT=25
+CI_FLEET_HEALTH_SWAP_CRITICAL_PERCENT=50
+CI_FLEET_HEALTH_LOAD_WARN_PER_CPU=1.0
+CI_FLEET_HEALTH_LOAD_CRITICAL_PER_CPU=1.5
+CI_FLEET_HEALTH_RESTART_WARN_COUNT=3
+CI_FLEET_HEALTH_BACKUP_CHECK=/usr/local/sbin/ci-fleet-backup-check
+CI_FLEET_HEALTH_HEARTBEAT_URL=https://monitor.example.invalid/heartbeat
+CI_FLEET_HEALTH_HEARTBEAT_TOKEN_FILE=/etc/ci-fleet/secrets/heartbeat-token
+```
+
+The backup hook must be an absolute, executable, root-owned file that is not group- or world-writable. Its output is discarded; only its exit status is reported. The heartbeat URL must use HTTPS. An optional token file must be root-owned and inaccessible to group/other users. The installer never creates, prints, commits, or removes this host-local file or its credentials, so rollback preserves them.
+
+## External missed-heartbeat detection
+
+A receiver accepts the redacted JSON POST and stores the most recent body as `<controller-id>.json`. Receiver implementation, endpoint, credential, address, and alert destination are provider-local. The external monitor evaluates those files against reviewed desired state:
+
+```bash
+python3 scripts/health.py heartbeats \
+  --config /srv/rd-delivery-config/fleet.json \
+  --input-dir /var/lib/ci-fleet-heartbeats \
+  --grace-seconds 900 \
+  --json
+```
+
+An active host with no fresh record is unhealthy. A drained host reports maintenance without a false alarm; a disabled host reports retired. A monitoring outage therefore cannot silently turn missing hosts healthy.
+
+## Operations
+
+- **Disk/inodes:** inspect fleet-labelled resources and run `scripts/cleanup.sh` in report mode first. Never use global Docker prune.
+- **Docker/controller:** drain if possible, inspect Docker and controller journals, then apply only reviewed desired state.
+- **Drift/timer failure:** run the named service manually and `install-worker-controller.sh --check`; repair by applying the reviewed pinned configuration, not by editing rendered files.
+- **Memory/OOM/load:** let active jobs drain, inspect kernel evidence, and adjust reviewed infrastructure capacity or runner resources.
+- **Updates/reboot:** drain before rebooting; verify all timers and the health result afterward.
+- **Missed heartbeat:** verify the receiver first, then use the provider console or out-of-band access. Inbound SSH is not required.
+- **Add/replace:** enroll the logical controller through reviewed private desired state, configure its host-local heartbeat credential, and verify a fresh external record before relying on it.
+- **Retire:** set lifecycle/state through reviewed desired state first. Delete no host, runner, or production resource without separate authorization.

docs/HOST-MAINTENANCE.md

@@ -30,7 +30,7 @@ Review `/etc/apt/apt.conf.d/50unattended-upgrades` and confirm only the intended
 
 `scripts/install-worker-controller.sh` installs and enables all three timer pairs:
 
-- `ci-fleet-health.timer` checks Docker, disk, and the controller's desired runtime state;
+- `ci-fleet-health.timer` runs the complete [fleet health contract](HEALTH-MONITORING.md);
 - `ci-fleet-cleanup.timer` removes only expired inactive fleet-owned resources;
 - `ci-fleet-drift.timer` compares the installation with the exact pinned configuration commit without applying changes.
 

docs/README.md

@@ -17,6 +17,7 @@ Use this index to find ci-fleet concepts, requirements, examples, and step-by-st
 | Make a project compliant | [Project CI standard](PROJECT-STANDARD.md) and [compliance checklist](COMPLIANCE-CHECKLIST.md) |
 | Split tests across parallel workers | [Project CI standard](PROJECT-STANDARD.md) and the [parallel workflow example](../examples/workflows/parallel-ci.yml.example) |
 | Configure automatic updates and cleanup | [Host maintenance](HOST-MAINTENANCE.md) |
+| Monitor hosts and detect missed heartbeats | [Fleet health monitoring](HEALTH-MONITORING.md) |
 | Handle GitHub App, workflow, or deployment secrets | [Secrets model](SECRETS.md) and [security policy](../SECURITY.md) |
 | Review accepted implementation scope | [Design decisions](DESIGN-DECISIONS.md) |
 | Run private CI or deployment for a public project | [Public projects, private delivery, and private configuration](PUBLIC-PRIVATE-CONFIGURATION.md) |
@@ -55,6 +56,7 @@ These pages are normative for compatible projects and hosts:
 - [Migration procedure](MIGRATING-EXISTING-CI.md)
 - [Compliance checklist](COMPLIANCE-CHECKLIST.md)
 - [Host maintenance standard](HOST-MAINTENANCE.md)
+- [Fleet health monitoring](HEALTH-MONITORING.md)
 - [Git-authored controller desired state](DESIRED-STATE.md)
 - [Secrets model](SECRETS.md)
 - [Security policy](../SECURITY.md)

host/systemd/ci-fleet-health.service

@@ -9,3 +9,4 @@ User=root
 WorkingDirectory=/opt/ci-fleet/manager/current
 EnvironmentFile=/etc/ci-fleet/ci-fleet.env
 ExecStart=/opt/ci-fleet/manager/current/scripts/healthcheck.sh
+SuccessExitStatus=1

runner/Dockerfile

@@ -6,7 +6,8 @@ ARG CI_FLEET_COMMIT=unknown
 ARG RUNNER_VERSION=2.335.1
 ARG RUNNER_SHA256_AMD64=4ef2f25285f0ae4477f1fe1e346db76d2f3ebf03824e2ddd1973a2819bf6c8cf
 ARG RUNNER_SHA256_ARM64=6d1e85bfd1a506a8b17c1f1b9b57dba458ffed90898799aaa9f599520b0d9207
-LABEL org.opencontainers.image.revision="${CI_FLEET_COMMIT}"
+LABEL org.opencontainers.image.revision="${CI_FLEET_COMMIT}" \
+      io.randomdevelopment.ci-fleet.managed="true"
 
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 RUN apt-get update \