Skip to content

feat(autoscaler): lower maxRunners while a node's own I/O or cpu pressure is high - #87

Merged
Mearman merged 1 commit into
mainfrom
feat/autoscaler-pressure
Oct 14, 2026
Merged

Mearman merged 1 commit into
mainfrom
feat/autoscaler-pressure

Conversation

@Mearman

@Mearman Mearman commented Oct 14, 2026

Copy link
Copy Markdown
Member

Adds an opt-in node-pressure signal to the autoscaler (github_runner_arc_autoscaler_node_pressure_enabled, off by default).

The source is each node's own /proc/pressure, read by a small psi-reporter DaemonSet in the autoscaler's image and published to one ConfigMap, one key per node. The kubelet's summary API looked like a free source and is not one: on a cgroup v1 node it reports every pressure field as 0 (checked on a node showing I/O "full" stall near 48% in /proc/pressure/io), so a reader built on it would find no pressure exactly where there is some. The detail is in the body of #82.

The autoscaler lowers the combined maxRunners by one a poll, never below the running count, while any eligible node's I/O "full" avg10 is at or over AUTOSCALER_IO_PRESSURE_PCT (default 10: the nodes that made jobs time out showed 20 to 50, healthy ones under one) or, when set above 0, its cpu "some" avg10 is at or over AUTOSCALER_CPU_PRESSURE_PCT. The cpu signal is off by default because a busy node's "some" is normally high and there is no calibration yet. A node with no report newer than three polls does not trigger lowering but keeps the pool from being raised, since whether it can take more is unknown. Pressure on a node outside the eligibility label is ignored, as memory already is.

The role installs the DaemonSet (tolerating every taint, on the nodes runner pods may use), its ServiceAccount, Roles scoped by resourceNames to the one ConfigMap and a get-only Role for the autoscaler, and pre-creates the ConfigMap empty like the status one. Turning the option off, or having no autoscaled profile, removes them.

Tests: the reporter against scripted pressure files (each line the kernel has, a line it does not report left out, a missing file, configurable names), and the autoscaler against reporter data (pressure lowers by one and names the node, never below the running count, healthy nodes change nothing, a stale or missing report holds the pool without lowering, a stale high reading is not pressure, cpu off unless asked, thresholds movable, nothing read without the ConfigMap, a node outside the selector ignored). I also ran the script in the real image on a cgroup v1 and a cgroup v2 node with a stub kubectl and it reported what /proc/pressure showed on each. The full unit suite and ansible-lint --profile production pass.

Closes #82

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

…sure is high

The autoscaler decided from memory only, so on a pool whose nodes differ it could keep
a ceiling that a node's disk could not serve: jobs on that node stalled on I/O until
tests missed their timeouts while memory and cpu requests said there was room.

A psi-reporter DaemonSet, in the autoscaler's image, runs on each node runner pods may
use and patches that node's key of one ConfigMap with the avg10 of its io, cpu and
memory pressure. The figures come from the node's own /proc/pressure, not the kubelet's
summary API, which reports every pressure field as 0 on a cgroup v1 node. The autoscaler
lowers the pooled total by one a poll, never below the running count, while a node's
I/O "full" stall is at or over a threshold (or its cpu "some" stall, when one is set), and
does not raise the pool while a node has no report newer than three polls. Off by default.

Closes #82
@Mearman
Mearman force-pushed the feat/autoscaler-pressure branch from 90ca950 to ae14fdc Compare October 14, 2026 16:16
@Mearman
Mearman merged commit 2f872ba into main Oct 14, 2026
16 checks passed
@Mearman
Mearman deleted the feat/autoscaler-pressure branch October 14, 2026 16:22
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 1.29.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Autoscaler: read I/O and CPU pressure per node, not only memory

1 participant