Skip to content

Commit e4d0054

Browse files
committed
docs(cluster): name the Compose profile in the regrade service's commands
1 parent 0d7a89d commit e4d0054

3 files changed

Lines changed: 3 additions & 3 deletions

File tree

‎roles/github_runner_cluster/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,7 @@ There are three levels, from the most capable down:
197197

198198
Giving up the vote is never part of it: a server that leaves etcd would have to be removed from the member list, which on a partition is unsafe, and k3s removes an etcd member when its Node object is deleted. The Node object of a stepped-down server stays, with its etcd role label.
199199

200-
A `regrade` service runs beside the server's k3s container (in its own Compose profile, which the role adds when the variable is true). It reads the memory pressure of the Docker host's kernel, the `some` line's `avg300` of `/proc/pressure/memory`, and `MemAvailable`; both are needed, because pressure alone also rises on reclaim while plenty is available, and a low `MemAvailable` alone is normal once the page cache has filled. A server counts as over the limit when the pressure is at least `github_runner_cluster_regrade_pressure_threshold` percent and `MemAvailable` is under `github_runner_cluster_regrade_available_floor_percent` of `MemTotal`. It steps down one level after being over for `github_runner_cluster_regrade_sustain_seconds` without a reading under the limit in between, and waits `github_runner_cluster_regrade_cooldown_seconds` after any step before the next. The cooldown has to exceed the pressure window, or the next reading still holds data from before the step; the role refuses a shorter one. The defaults are multiples of the window and the sample period, so they follow if either changes. The threshold and the floor are reasoned starting points, not measurements: read the service's log with `docker compose logs regrade` on a real host and tune them.
200+
A `regrade` service runs beside the server's k3s container (in its own Compose profile, which the role adds when the variable is true). It reads the memory pressure of the Docker host's kernel, the `some` line's `avg300` of `/proc/pressure/memory`, and `MemAvailable`; both are needed, because pressure alone also rises on reclaim while plenty is available, and a low `MemAvailable` alone is normal once the page cache has filled. A server counts as over the limit when the pressure is at least `github_runner_cluster_regrade_pressure_threshold` percent and `MemAvailable` is under `github_runner_cluster_regrade_available_floor_percent` of `MemTotal`. It steps down one level after being over for `github_runner_cluster_regrade_sustain_seconds` without a reading under the limit in between, and waits `github_runner_cluster_regrade_cooldown_seconds` after any step before the next. The cooldown has to exceed the pressure window, or the next reading still holds data from before the step; the role refuses a shorter one. The defaults are multiples of the window and the sample period, so they follow if either changes. The threshold and the floor are reasoned starting points, not measurements: read the service's log with `docker compose --profile regrade logs regrade` on a real host and tune them.
201201

202202
With `github_runner_cluster_regrade_step_up: automatic` (the default) the service also steps back up by itself, one level at a time. A server counts as calm when the pressure is under the threshold and `MemAvailable` is at or above the floor, both, which is stricter than not being over the limit: a reading between the two ends the streak towards stepping down and the streak towards stepping up alike, so the server does not step both ways on the same noise. It steps up after being calm for `github_runner_cluster_regrade_sustain_seconds`, once the cooldown since the last step, in either direction, has passed, through the same Lease and peer check as a step down; stepping up restarts the server's etcd at the first level that leaves etcd-only, so it never starts while a peer is down. A server that cannot hold the higher level cycles down and up no faster than once per cooldown and sustain period, which is the sign it is too small for the workloads it is given. `manual` leaves stepping up to `fleet-regrade up`.
203203

‎roles/github_runner_cluster/defaults/main.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ github_runner_cluster_regrade: false
3131
github_runner_cluster_regrade_step_up: automatic
3232
# The window of the longest pressure average the kernel reports (avg300 in /proc/pressure/memory), in seconds. A fact about the kernel, not a setting: every default below that is a time is a multiple of it or of the sample period, so that a reading always spans the level it describes.
3333
github_runner_cluster_regrade_pressure_window_seconds: 300
34-
# The "some" avg300 of /proc/pressure/memory, in percent, at or above which the server counts as under pressure: for that share of the last five minutes some task was stalled waiting for memory, a rate at which etcd's heartbeat and election timers, which run in tenths of a second, are being missed. A reasoned starting point, not a measurement: read the service's log (`docker compose logs regrade`) on a real host and tune it.
34+
# The "some" avg300 of /proc/pressure/memory, in percent, at or above which the server counts as under pressure: for that share of the last five minutes some task was stalled waiting for memory, a rate at which etcd's heartbeat and election timers, which run in tenths of a second, are being missed. A reasoned starting point, not a measurement: read the service's log (`docker compose --profile regrade logs regrade`) on a real host and tune it.
3535
github_runner_cluster_regrade_pressure_threshold: 10
3636
# MemAvailable, as a percentage of MemTotal, below which a pressure reading counts. It guards against stepping down on pressure caused by reclaim while plenty is available, and, as a share rather than a number of bytes, is right for any host. Chosen, not measured.
3737
github_runner_cluster_regrade_available_floor_percent: 15

‎roles/github_runner_cluster/tasks/regrade_service.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@
6262
msg: >-
6363
github_runner_cluster_regrade is false, but {{ inventory_hostname }} is running at the level
6464
{{ github_runner_cluster_regrade_level.stdout | trim }}, below full, and the regrade service is what steps it back up. Run
65-
`docker compose exec regrade fleet-regrade up` in {{ github_runner_cluster_dir }} until it reports full (status shows the
65+
`docker compose --profile regrade exec regrade fleet-regrade up` in {{ github_runner_cluster_dir }} until it reports full (status shows the
6666
level), then run again. Nothing was changed.
6767
when:
6868
- github_runner_cluster_regrade_level.rc == 0

0 commit comments

Comments
 (0)