Skip to content

Commit 90ca950

Browse files
committed
feat(autoscaler): lower maxRunners while a node's own I/O or cpu pressure is high
The autoscaler decided from memory only, so on a pool whose nodes differ it could keep a ceiling that a node's disk could not serve: jobs on that node stalled on I/O until tests missed their timeouts while memory and cpu requests said there was room. A psi-reporter DaemonSet, in the autoscaler's image, runs on each node runner pods may use and patches that node's key of one ConfigMap with the avg10 of its io, cpu and memory pressure. The figures come from the node's own /proc/pressure, not the kubelet's summary API, which reports every pressure field as 0 on a cgroup v1 node. The autoscaler lowers the pooled total by one a poll, never below the running count, while a node's I/O "full" stall is at or over a threshold (or its cpu "some" stall, when one is set), and does not raise the pool while a node has no report newer than three polls. Off by default. Closes #82
1 parent 206f108 commit 90ca950

11 files changed

Lines changed: 494 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,7 @@ Each poll (`AUTOSCALER_POLL_SECONDS`, default 45s):
150150
- Treats any unschedulable runner pod (`PodScheduled=False`, reason `Unschedulable`) like pressure: the memory budget sums every node's capacity, so it cannot see a node that is tainted for disk pressure or a request that fits no single node. It lowers the combined `maxRunners` to the running count and never raises while one exists.
151151
- Also collects the peak memory each job's runner measured for itself: the runner image's job-completed hook (`runner-hooks/job-completed.sh`, set through `ACTIONS_RUNNER_HOOK_JOB_COMPLETED`) reads the container cgroup's `memory.peak`, which a spike shorter than the poll cannot escape, and posts it as a `JobMemoryPeak` Event in the pod's namespace. The autoscaler reads those Events each poll and keeps the maximum per repository, workflow ref and job id under the status ConfigMap's `job-peak` key (`kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-peak}'`), in MiB. Keeping the maximum means a forged low figure lowers nothing and re-reading an Event changes nothing. The hook never fails the job it ends. To post an Event the runner pods' service account, which otherwise holds no permission, gets `create` on `events` in its own namespace and nothing else (`github_runner_arc_record_job_peaks`, default on; `false` removes the grant). A job's code runs in the same container and can read the same token, so the worst it can do with that permission is post events in its own runner namespace.
152152
- Records, per job, the highest memory a runner pod running it has been seen at, in the status ConfigMap's `job-memory` key (`kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-memory}'`), keyed by repository, workflow ref and job name with the peak in MiB, the number of observations and when it was last seen. It is sampled once per poll from the metrics server, so it is a lower bound that a spike shorter than the poll escapes; it shows how far below a pod's request jobs sit, which is what decides whether an accurate in-pod measurement (see the per-job sizing issue) is worth building. Entries unseen for 30 days are dropped, and a failure to keep the record never affects a capacity decision.
153+
- Optionally reads per-node pressure from the nodes' own `/proc/pressure` (`github_runner_arc_autoscaler_node_pressure_enabled`). A `psi-reporter` DaemonSet (`scripts/psi-reporter.sh`, in the autoscaler's image) runs on each node runner pods may use, tolerating every taint, and patches that node's own key of one ConfigMap with the `avg10` of its io, cpu and memory pressure. The reading is not taken from the kubelet's summary API, because on a cgroup v1 node that reports every pressure field as 0 whatever the node is doing. The autoscaler lowers the combined `maxRunners` by exactly one each poll, never below the running count, while any node's I/O "full" stall is at or over `AUTOSCALER_IO_PRESSURE_PCT` (default 10) or, when set above 0, its cpu "some" stall is at or over `AUTOSCALER_CPU_PRESSURE_PCT`, and it does not raise the pool while any node has no report newer than three polls, since whether that node can take more is then unknown. This is the signal for a pool whose nodes differ: memory can have room on a node whose disk is already the limit.
153154
- Fails safe on any memory measurement error (`kubectl top nodes`/`kubectl top pod` failing or unparseable): lowers the combined total toward `AUTOSCALER_FLOOR` (`github_runner_arc_autoscaler_floor`) immediately rather than skipping the cycle silently. A failure reading the scale sets' own state or their pods exits non-zero instead, since fail-safe needs that state to lower anything and the API server is then unreachable anyway. This combined floor need not equal any one profile's own values-file `maxRunners`, since each profile's Helm upgrade reverts to its own static floor independently of how the pool's combined floor is set.
154155

155156
Ships with `AUTOSCALER_DRY_RUN=true` by default: it computes and logs the target and writes its own status, but never patches. Generate real concurrent traffic to watch it react: `gh workflow run test-autoscaler.yml`, then watch `kubectl logs -n github-runner-platform deploy/autoscaler -f` and `kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.status\.json}'`. Only set `AUTOSCALER_DRY_RUN=false` after watching real dry-run output across genuine CI traffic. Status and its raise-confirm counter live in a shared Kubernetes ConfigMap (`autoscaler-status`, pre-created empty so neither pod's own ServiceAccount ever needs `create` RBAC on it, only `get`/`patch`) rather than a bind-mounted directory, since the two pods can land on different nodes now — `scripts/heartbeat.sh` reads that same ConfigMap back out and republishes it as a second file in the heartbeat gist, reusing its existing gist-write credential rather than giving the autoscaler its own.

‎autoscaler/Dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@ RUN apk add --no-cache bash curl jq kubectl
1313

1414
COPY autoscaler/loop.sh /app/loop.sh
1515
COPY scripts/autoscaler.sh /app/scripts/autoscaler.sh
16-
RUN chmod +x /app/loop.sh /app/scripts/autoscaler.sh
16+
COPY scripts/psi-reporter.sh /app/scripts/psi-reporter.sh
17+
RUN chmod +x /app/loop.sh /app/scripts/autoscaler.sh /app/scripts/psi-reporter.sh
1718

1819
WORKDIR /app
1920
ENTRYPOINT ["/app/loop.sh"]

‎roles/github_runner_arc/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ Helm 4 and server-side apply. The role reads `helm version --short` and, on Helm
3939
- `github_runner_arc_controller_replicas`: above 1, the chart turns on leader election and the role adds a preferred pod anti-affinity across nodes. `github_runner_arc_controller_extra_values` is merged over the role's controller values.
4040
- `github_runner_arc_metrics_enabled` (and the `_addr`/`_endpoint` settings): Prometheus metrics for the controller and every listener; the 0.14 chart only enables them together.
4141
- `github_runner_arc_listener_probes_enabled`: readiness and liveness probes on each listener's metrics endpoint, so a listener that starts and then fails its GitHub authentication is not counted ready during a rollout. Needs metrics on.
42+
- `github_runner_arc_autoscaler_node_pressure_enabled`, `github_runner_arc_autoscaler_io_pressure_pct`, `github_runner_arc_autoscaler_cpu_pressure_pct`, `github_runner_arc_psi_reporter_seconds`, `github_runner_arc_node_pressure_configmap`: per-node pressure for the autoscaler, from each node's own `/proc/pressure` through a `psi-reporter` DaemonSet on the nodes runner pods may use (see the Autoscaler section of the repository README). Needs a profile with `autoscale: true`; with the option off, or no autoscaled profile, the role removes the DaemonSet and its ConfigMap and RBAC.
4243
- `github_runner_arc_node_label_key`, `github_runner_arc_node_label_value`: when the key is set, the controller, listeners and runner pods are restricted to nodes carrying the label, and the autoscaler reads memory pressure from those nodes only. The role labels the nodes named in `github_runner_arc_labelled_nodes`.
4344
- `github_runner_arc_ghcr_username`, `github_runner_arc_ghcr_token`, `github_runner_arc_heartbeat_gh_token`: secrets, as plain variables. The pull credential is needed only for a static pull Secret.
4445
- `github_runner_arc_manage_secrets`: `false` stops the role writing the App Secrets and `heartbeat-gh-token`, and instead checks before any change that they already exist, and that each App Secret holds `github_app_id`, `github_app_installation_id` and `github_app_private_key`. The role then needs no `private_key` or `installation_id`, and no `app_id` either: it reads the App's id from the Secret where it needs it. An `app_id` that is set must match the Secret's `github_app_id`.

‎roles/github_runner_arc/defaults/main.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,12 @@ github_runner_arc_autoscaler_dry_run: true
144144
github_runner_arc_autoscaler_poll_seconds: 45
145145
github_runner_arc_autoscaler_raise_confirm_polls: 2
146146
github_runner_arc_autoscaler_mem_available_pressure_pct: 15
147+
# Per-node pressure for the autoscaler, from each node's own /proc/pressure (scripts/psi-reporter.sh, a DaemonSet on the nodes runner pods may use, publishing to one ConfigMap). The kubelet's summary API is not used: on a cgroup v1 node it reports every pressure field as 0. With it on, a node whose I/O "full" stall is at or over the first figure lowers the pooled maxRunners by one a poll, and one whose cpu "some" stall is at or over the second does the same when it is above 0. A node with no recent report keeps the pool from being raised. The cpu figure is off by default: a busy node's cpu "some" is normally high and there is no calibration yet for where it starts to cost jobs.
148+
github_runner_arc_autoscaler_node_pressure_enabled: false
149+
github_runner_arc_autoscaler_io_pressure_pct: 10
150+
github_runner_arc_autoscaler_cpu_pressure_pct: 0
151+
github_runner_arc_node_pressure_configmap: "autoscaler-node-pressure"
152+
github_runner_arc_psi_reporter_seconds: 15
147153

148154
# Node recovery (see tasks/install_node_recovery.yml and scripts/node-recovery.sh). When a node's kubelet stops reporting, the control plane marks the node's pods for deletion but only the kubelet can confirm it, so they stay Terminating, and an ARC listener on that node is never replaced. The watcher gives such a node the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint once its Ready condition has been Unknown for github_runner_arc_node_recovery_after_seconds, which makes the control plane evict its pods and force-delete the terminating ones, and removes the taint when the node reports Ready again. false removes the watcher and any taint it left.
149155
github_runner_arc_node_recovery_enabled: true

‎roles/github_runner_arc/meta/argument_specs.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,26 @@ argument_specs:
284284
type: int
285285
default: 15
286286
description: Available memory percentage below which the autoscaler treats a node as under pressure.
287+
github_runner_arc_autoscaler_node_pressure_enabled:
288+
type: bool
289+
default: false
290+
description: Installs a per-node reporter of /proc/pressure (a DaemonSet on the nodes runner pods may use) and has the autoscaler lower maxRunners by one a poll while a node's pressure is over the thresholds below, and hold off raising it while a node has no recent report. Needs an autoscaled profile.
291+
github_runner_arc_autoscaler_io_pressure_pct:
292+
type: int
293+
default: 10
294+
description: The "full" avg10 of a node's /proc/pressure/io, in percent, at or over which the autoscaler treats the node as under pressure. Used with github_runner_arc_autoscaler_node_pressure_enabled.
295+
github_runner_arc_autoscaler_cpu_pressure_pct:
296+
type: int
297+
default: 0
298+
description: The "some" avg10 of a node's /proc/pressure/cpu, in percent, at or over which the autoscaler treats the node as under pressure; 0 leaves the cpu signal off. Used with github_runner_arc_autoscaler_node_pressure_enabled.
299+
github_runner_arc_node_pressure_configmap:
300+
type: str
301+
default: autoscaler-node-pressure
302+
description: The ConfigMap in the platform namespace that each node's reporter patches its own key of.
303+
github_runner_arc_psi_reporter_seconds:
304+
type: int
305+
default: 15
306+
description: Seconds between each reporter's reports. The autoscaler treats a report older than three of its polls as no report.
287307
github_runner_arc_node_recovery_enabled:
288308
type: bool
289309
default: true

‎roles/github_runner_arc/tasks/install_platform.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,13 @@
498498
value: "{{ github_runner_arc_fleet.autoscaled | map(attribute='target') | join(' ') }}"
499499
- name: AUTOSCALER_STATUS_CONFIGMAP
500500
value: autoscaler-status
501+
# Empty leaves node pressure out; with it set the autoscaler reads each node's reporter entry from this ConfigMap (tasks/install_psi_reporter.yml).
502+
- name: AUTOSCALER_NODE_PRESSURE_CONFIGMAP
503+
value: "{{ github_runner_arc_node_pressure_configmap if github_runner_arc_autoscaler_node_pressure_enabled | bool else '' }}"
504+
- name: AUTOSCALER_IO_PRESSURE_PCT
505+
value: "{{ github_runner_arc_autoscaler_io_pressure_pct | string }}"
506+
- name: AUTOSCALER_CPU_PRESSURE_PCT
507+
value: "{{ github_runner_arc_autoscaler_cpu_pressure_pct | string }}"
501508
resources:
502509
requests:
503510
cpu: "50m"
@@ -507,3 +514,7 @@
507514
cpu: "200m"
508515
memory: "64Mi"
509516
ephemeral-storage: "{{ github_runner_arc_platform_ephemeral_storage_limit }}"
517+
518+
# The per-node pressure reporters the autoscaler reads, or their removal when turned off.
519+
- name: "Platform: install the node pressure reporter, or remove it when turned off"
520+
ansible.builtin.include_tasks: install_psi_reporter.yml

0 commit comments

Comments
 (0)