feat(autoscaler): lower maxRunners while a node's own I/O or cpu pressure is high

README.md

@@ -150,6 +150,7 @@ Each poll (`AUTOSCALER_POLL_SECONDS`, default 45s):
 - Treats any unschedulable runner pod (`PodScheduled=False`, reason `Unschedulable`) like pressure: the memory budget sums every node's capacity, so it cannot see a node that is tainted for disk pressure or a request that fits no single node. It lowers the combined `maxRunners` to the running count and never raises while one exists.
 - Also collects the peak memory each job's runner measured for itself: the runner image's job-completed hook (`runner-hooks/job-completed.sh`, set through `ACTIONS_RUNNER_HOOK_JOB_COMPLETED`) reads the container cgroup's `memory.peak`, which a spike shorter than the poll cannot escape, and posts it as a `JobMemoryPeak` Event in the pod's namespace. The autoscaler reads those Events each poll and keeps the maximum per repository, workflow ref and job id under the status ConfigMap's `job-peak` key (`kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-peak}'`), in MiB. Keeping the maximum means a forged low figure lowers nothing and re-reading an Event changes nothing. The hook never fails the job it ends. To post an Event the runner pods' service account, which otherwise holds no permission, gets `create` on `events` in its own namespace and nothing else (`github_runner_arc_record_job_peaks`, default on; `false` removes the grant). A job's code runs in the same container and can read the same token, so the worst it can do with that permission is post events in its own runner namespace.
 - Records, per job, the highest memory a runner pod running it has been seen at, in the status ConfigMap's `job-memory` key (`kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-memory}'`), keyed by repository, workflow ref and job name with the peak in MiB, the number of observations and when it was last seen. It is sampled once per poll from the metrics server, so it is a lower bound that a spike shorter than the poll escapes; it shows how far below a pod's request jobs sit, which is what decides whether an accurate in-pod measurement (see the per-job sizing issue) is worth building. Entries unseen for 30 days are dropped, and a failure to keep the record never affects a capacity decision.
+- Optionally reads per-node pressure from the nodes' own `/proc/pressure` (`github_runner_arc_autoscaler_node_pressure_enabled`). A `psi-reporter` DaemonSet (`scripts/psi-reporter.sh`, in the autoscaler's image) runs on each node runner pods may use, tolerating every taint, and patches that node's own key of one ConfigMap with the `avg10` of its io, cpu and memory pressure. The reading is not taken from the kubelet's summary API, because on a cgroup v1 node that reports every pressure field as 0 whatever the node is doing. The autoscaler lowers the combined `maxRunners` by exactly one each poll, never below the running count, while any node's I/O "full" stall is at or over `AUTOSCALER_IO_PRESSURE_PCT` (default 10) or, when set above 0, its cpu "some" stall is at or over `AUTOSCALER_CPU_PRESSURE_PCT`, and it does not raise the pool while any node has no report newer than three polls, since whether that node can take more is then unknown. This is the signal for a pool whose nodes differ: memory can have room on a node whose disk is already the limit.
 - Fails safe on any memory measurement error (`kubectl top nodes`/`kubectl top pod` failing or unparseable): lowers the combined total toward `AUTOSCALER_FLOOR` (`github_runner_arc_autoscaler_floor`) immediately rather than skipping the cycle silently. A failure reading the scale sets' own state or their pods exits non-zero instead, since fail-safe needs that state to lower anything and the API server is then unreachable anyway. This combined floor need not equal any one profile's own values-file `maxRunners`, since each profile's Helm upgrade reverts to its own static floor independently of how the pool's combined floor is set.
 
 Ships with `AUTOSCALER_DRY_RUN=true` by default: it computes and logs the target and writes its own status, but never patches. Generate real concurrent traffic to watch it react: `gh workflow run test-autoscaler.yml`, then watch `kubectl logs -n github-runner-platform deploy/autoscaler -f` and `kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.status\.json}'`. Only set `AUTOSCALER_DRY_RUN=false` after watching real dry-run output across genuine CI traffic. Status and its raise-confirm counter live in a shared Kubernetes ConfigMap (`autoscaler-status`, pre-created empty so neither pod's own ServiceAccount ever needs `create` RBAC on it, only `get`/`patch`) rather than a bind-mounted directory, since the two pods can land on different nodes now — `scripts/heartbeat.sh` reads that same ConfigMap back out and republishes it as a second file in the heartbeat gist, reusing its existing gist-write credential rather than giving the autoscaler its own.

autoscaler/Dockerfile

@@ -13,7 +13,8 @@ RUN apk add --no-cache bash curl jq kubectl
 
 COPY autoscaler/loop.sh /app/loop.sh
 COPY scripts/autoscaler.sh /app/scripts/autoscaler.sh
-RUN chmod +x /app/loop.sh /app/scripts/autoscaler.sh
+COPY scripts/psi-reporter.sh /app/scripts/psi-reporter.sh
+RUN chmod +x /app/loop.sh /app/scripts/autoscaler.sh /app/scripts/psi-reporter.sh
 
 WORKDIR /app
 ENTRYPOINT ["/app/loop.sh"]

roles/github_runner_arc/README.md

@@ -40,6 +40,7 @@ Helm 4 and server-side apply. The role reads `helm version --short` and, on Helm
 - `github_runner_arc_metrics_enabled` (and the `_addr`/`_endpoint` settings): Prometheus metrics for the controller and every listener; the 0.14 chart only enables them together.
 - `github_runner_arc_listener_probes_enabled`: readiness and liveness probes on each listener's metrics endpoint, so a listener that starts and then fails its GitHub authentication is not counted ready during a rollout. Needs metrics on.
 - `github_runner_arc_control_pods_priority_enabled`: runs the controller and every listener under the platform's PriorityClass (`github_runner_arc_platform_priority_class_name`, created by the role whether or not the platform is installed). A listener is one pod per scale set that ARC cannot replicate, and while it waits to be scheduled no new jobs are assigned; with the class a rescheduled one can take a full node's room from a runner pod, whose job ARC retries, instead of waiting behind it. Off by default, as it changes what a full node evicts.
+- `github_runner_arc_autoscaler_node_pressure_enabled`, `github_runner_arc_autoscaler_io_pressure_pct`, `github_runner_arc_autoscaler_cpu_pressure_pct`, `github_runner_arc_psi_reporter_seconds`, `github_runner_arc_node_pressure_configmap`: per-node pressure for the autoscaler, from each node's own `/proc/pressure` through a `psi-reporter` DaemonSet on the nodes runner pods may use (see the Autoscaler section of the repository README). Needs a profile with `autoscale: true`; with the option off, or no autoscaled profile, the role removes the DaemonSet and its ConfigMap and RBAC.
 - `github_runner_arc_node_label_key`, `github_runner_arc_node_label_value`: when the key is set, the controller, listeners and runner pods are restricted to nodes carrying the label, and the autoscaler reads memory pressure from those nodes only. The role labels the nodes named in `github_runner_arc_labelled_nodes`.
 - `github_runner_arc_ghcr_username`, `github_runner_arc_ghcr_token`, `github_runner_arc_heartbeat_gh_token`: secrets, as plain variables. The pull credential is needed only for a static pull Secret.
 - `github_runner_arc_manage_secrets`: `false` stops the role writing the App Secrets and `heartbeat-gh-token`, and instead checks before any change that they already exist, and that each App Secret holds `github_app_id`, `github_app_installation_id` and `github_app_private_key`. The role then needs no `private_key` or `installation_id`, and no `app_id` either: it reads the App's id from the Secret where it needs it. An `app_id` that is set must match the Secret's `github_app_id`.

roles/github_runner_arc/defaults/main.yml

@@ -147,6 +147,12 @@ github_runner_arc_autoscaler_dry_run: true
 github_runner_arc_autoscaler_poll_seconds: 45
 github_runner_arc_autoscaler_raise_confirm_polls: 2
 github_runner_arc_autoscaler_mem_available_pressure_pct: 15
+# Per-node pressure for the autoscaler, from each node's own /proc/pressure (scripts/psi-reporter.sh, a DaemonSet on the nodes runner pods may use, publishing to one ConfigMap). The kubelet's summary API is not used: on a cgroup v1 node it reports every pressure field as 0. With it on, a node whose I/O "full" stall is at or over the first figure lowers the pooled maxRunners by one a poll, and one whose cpu "some" stall is at or over the second does the same when it is above 0. A node with no recent report keeps the pool from being raised. The cpu figure is off by default: a busy node's cpu "some" is normally high and there is no calibration yet for where it starts to cost jobs.
+github_runner_arc_autoscaler_node_pressure_enabled: false
+github_runner_arc_autoscaler_io_pressure_pct: 10
+github_runner_arc_autoscaler_cpu_pressure_pct: 0
+github_runner_arc_node_pressure_configmap: "autoscaler-node-pressure"
+github_runner_arc_psi_reporter_seconds: 15
 
 # Node recovery (see tasks/install_node_recovery.yml and scripts/node-recovery.sh). When a node's kubelet stops reporting, the control plane marks the node's pods for deletion but only the kubelet can confirm it, so they stay Terminating, and an ARC listener on that node is never replaced. The watcher gives such a node the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint once its Ready condition has been Unknown for github_runner_arc_node_recovery_after_seconds, which makes the control plane evict its pods and force-delete the terminating ones, and removes the taint when the node reports Ready again. false removes the watcher and any taint it left.
 github_runner_arc_node_recovery_enabled: true

roles/github_runner_arc/meta/argument_specs.yml

@@ -288,6 +288,26 @@ argument_specs:
         type: int
         default: 15
         description: Available memory percentage below which the autoscaler treats a node as under pressure.
+      github_runner_arc_autoscaler_node_pressure_enabled:
+        type: bool
+        default: false
+        description: Installs a per-node reporter of /proc/pressure (a DaemonSet on the nodes runner pods may use) and has the autoscaler lower maxRunners by one a poll while a node's pressure is over the thresholds below, and hold off raising it while a node has no recent report. Needs an autoscaled profile.
+      github_runner_arc_autoscaler_io_pressure_pct:
+        type: int
+        default: 10
+        description: The "full" avg10 of a node's /proc/pressure/io, in percent, at or over which the autoscaler treats the node as under pressure. Used with github_runner_arc_autoscaler_node_pressure_enabled.
+      github_runner_arc_autoscaler_cpu_pressure_pct:
+        type: int
+        default: 0
+        description: The "some" avg10 of a node's /proc/pressure/cpu, in percent, at or over which the autoscaler treats the node as under pressure; 0 leaves the cpu signal off. Used with github_runner_arc_autoscaler_node_pressure_enabled.
+      github_runner_arc_node_pressure_configmap:
+        type: str
+        default: autoscaler-node-pressure
+        description: The ConfigMap in the platform namespace that each node's reporter patches its own key of.
+      github_runner_arc_psi_reporter_seconds:
+        type: int
+        default: 15
+        description: Seconds between each reporter's reports. The autoscaler treats a report older than three of its polls as no report.
       github_runner_arc_node_recovery_enabled:
         type: bool
         default: true

roles/github_runner_arc/tasks/install_platform.yml

@@ -498,6 +498,13 @@
                     value: "{{ github_runner_arc_fleet.autoscaled | map(attribute='target') | join(' ') }}"
                   - name: AUTOSCALER_STATUS_CONFIGMAP
                     value: autoscaler-status
+                  # Empty leaves node pressure out; with it set the autoscaler reads each node's reporter entry from this ConfigMap (tasks/install_psi_reporter.yml).
+                  - name: AUTOSCALER_NODE_PRESSURE_CONFIGMAP
+                    value: "{{ github_runner_arc_node_pressure_configmap if github_runner_arc_autoscaler_node_pressure_enabled | bool else '' }}"
+                  - name: AUTOSCALER_IO_PRESSURE_PCT
+                    value: "{{ github_runner_arc_autoscaler_io_pressure_pct | string }}"
+                  - name: AUTOSCALER_CPU_PRESSURE_PCT
+                    value: "{{ github_runner_arc_autoscaler_cpu_pressure_pct | string }}"
                 resources:
                   requests:
                     cpu: "50m"
@@ -507,3 +514,7 @@
                     cpu: "200m"
                     memory: "64Mi"
                     ephemeral-storage: "{{ github_runner_arc_platform_ephemeral_storage_limit }}"
+
+# The per-node pressure reporters the autoscaler reads, or their removal when turned off.
+- name: "Platform: install the node pressure reporter, or remove it when turned off"
+  ansible.builtin.include_tasks: install_psi_reporter.yml