Skip to content

Commit bb35601

Browse files
committed
docs(heartbeat): document the fleet-down notification and where it must run
1 parent 5a28c7f commit bb35601

1 file changed

Lines changed: 17 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,23 @@ echo "$(($(date +%s) + 600))" | gh gist create --filename arc-healthy-until --de
135135
```
136136
Confirm it refreshes: `gh gist view "$HEARTBEAT_GIST_ID"` must show a timestamp near `now + 600`, advancing roughly every 3 minutes.
137137

138+
#### Fleet-down notification
139+
140+
`runner-fallback-action` reroutes silently, so nothing otherwise tells a person that the fleet has no runners. `fleet-health.json` carries what a notifier needs to judge how long that has lasted: `at` (when the heartbeat last wrote), `unhealthy_since` (when it stopped vouching for the fleet, `null` while it does) and `not_ready_nodes` (each node whose Ready condition is not True, whether it is a control-plane server, and since when). The heartbeat carries each onset over from the document its previous tick published, read back from the gist, so a tick that cannot read it restarts the onsets at now, which delays an alert and never raises a false one.
141+
142+
`scripts/fleet-alert.sh` reads that document and announces an outage when any of these has held for at least `FLEET_ALERT_AFTER_SECONDS` (default 900, longer than the heartbeat's own 600 second window): the heartbeat has not written for that long, a control-plane server has not been Ready for that long, or the heartbeat has not vouched for the fleet for that long (a scale set missing or without a running listener, no ready controller, no Ready node, runner pods that cannot be scheduled; the reasons are in the notification). A worker node that is not Ready is not an outage by itself. An outage is announced once, and a recovery note follows when every condition has cleared. A heartbeat image older than the onset fields makes the script fail with a message saying so, rather than never alerting.
143+
144+
Delivery is independent per channel, and with none configured the script only prints what it found:
145+
146+
| Channel | Set | Notes |
147+
|---|---|---|
148+
| Issue | `FLEET_ALERT_ISSUE_REPO` (`owner/repo`) and `FLEET_ALERT_ISSUE_TOKEN` | Opens an issue and finds it again by a hidden marker in its body, so the open issue is the record that the outage was announced; closes it with a comment on recovery. The heartbeat's `HEARTBEAT_GH_TOKEN` is a gist-scoped token and cannot write issues, so this needs its own token with issue write access to that repository; a workflow's own `GITHUB_TOKEN` with `issues: write` suffices for the repository the workflow lives in. |
149+
| Webhook | `FLEET_ALERT_WEBHOOK_URL` and `FLEET_ALERT_STATE_FILE` | POSTs JSON: `event` (`firing` or `resolved`), `text` (a one-line summary), `conditions` (each with `kind`, `detail` and `since`) and `heartbeat_at`. It names no chat product, so a receiver maps `text` or `conditions` to its own format. The state file holds only whether the outage was announced; a delivery that fails is not recorded and the next pass retries. |
150+
151+
Both the URL and the tokens are passed to `curl` on its standard input, never its command line. Issue bodies name nodes and the heartbeat's reasons, so use a private repository.
152+
153+
**Where it runs matters.** A check that runs only inside the cluster cannot report that the cluster is down, and a check that runs on the fleet's own runners does not start when the fleet has no runners. So the check reads the gist, which the heartbeat writes, from outside: `examples/fleet-alert/fleet-alert.yml` is a scheduled workflow for a repository not served by the fleet, on a GitHub-hosted runner, and the script runs just as well from cron on any host outside the fleet. The tradeoff is that it depends on GitHub (Actions scheduling, which can be delayed and is limited to a run every five minutes, and the gist API) and on the heartbeat's write path: if GitHub is unreachable from the fleet the heartbeat goes stale and the alert fires, which is also what the fleet's users experience. It is not a cluster Deployment, deliberately. A repository with no GitHub-hosted runners available cannot use the workflow and should use cron. The workflow template keeps the webhook's state in `actions/cache`, which can be evicted; the worst case is one repeated announcement.
154+
138155
### Autoscaler
139156

140157
A scale set's static `maxRunners` has to be safe for the pool's real worst case, every pod at its limit at once, so it leaves real idle capacity unused whenever the jobs in flight need less than that. `autoscaler` (an in-cluster Deployment, same shape/placement as heartbeat above) raises and lowers `maxRunners` across every profile with `autoscale: true` (one or more, possibly from different orgs, named by `AUTOSCALER_TARGETS`), based on real, current memory usage and pressure, never on job identity or GitHub Actions queue depth: every consuming repo's CI keeps using its scale set's own `runs-on` label unchanged, and no job is ever classified as light or heavy. When more than one profile is pooled, they share one combined budget rather than each getting a fixed slice: the algorithm below runs against the pool's combined `currentRunners`/`maxRunners`, and a change is handed to whichever pooled profile's own headroom (its `maxRunners` minus its `currentRunners`) makes it the right target: raises go to the profile with the *least* headroom (closest to genuinely needing more), lowers take from the profile with the *most* (its capacity is going unused), one runner at a time, and a lower never takes a profile below its own `currentRunners`. RBAC is scoped to exactly what it reads/writes: cluster-wide `nodes.metrics.k8s.io` get/list; per pooled profile, `get`/`patch` on that `AutoscalingRunnerSet` by name, `get`/`list` on `ephemeralrunners` (to count the running runners and to name the job a runner is running), `get`/`list` on events and pods, and `pods.metrics.k8s.io` get/list in its own namespace; and `get`/`patch` on its own status ConfigMap by name, with no wildcards.

0 commit comments

Comments
 (0)