Repository navigation
feat(autoscaler): lower maxRunners while a node's own I/O or cpu pressure is high - #87
Merged
Merged
Conversation
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
…sure is high The autoscaler decided from memory only, so on a pool whose nodes differ it could keep a ceiling that a node's disk could not serve: jobs on that node stalled on I/O until tests missed their timeouts while memory and cpu requests said there was room. A psi-reporter DaemonSet, in the autoscaler's image, runs on each node runner pods may use and patches that node's key of one ConfigMap with the avg10 of its io, cpu and memory pressure. The figures come from the node's own /proc/pressure, not the kubelet's summary API, which reports every pressure field as 0 on a cgroup v1 node. The autoscaler lowers the pooled total by one a poll, never below the running count, while a node's I/O "full" stall is at or over a threshold (or its cpu "some" stall, when one is set), and does not raise the pool while a node has no report newer than three polls. Off by default. Closes #82
Mearman
force-pushed
the
feat/autoscaler-pressure
branch
from
October 14, 2026 16:16
90ca950 to
ae14fdc
Compare
|
🎉 This PR is included in version 1.29.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in node-pressure signal to the autoscaler (
github_runner_arc_autoscaler_node_pressure_enabled, off by default).The source is each node's own
/proc/pressure, read by a smallpsi-reporterDaemonSet in the autoscaler's image and published to one ConfigMap, one key per node. The kubelet's summary API looked like a free source and is not one: on a cgroup v1 node it reports every pressure field as 0 (checked on a node showing I/O "full" stall near 48% in/proc/pressure/io), so a reader built on it would find no pressure exactly where there is some. The detail is in the body of #82.The autoscaler lowers the combined
maxRunnersby one a poll, never below the running count, while any eligible node's I/O "full" avg10 is at or overAUTOSCALER_IO_PRESSURE_PCT(default 10: the nodes that made jobs time out showed 20 to 50, healthy ones under one) or, when set above 0, its cpu "some" avg10 is at or overAUTOSCALER_CPU_PRESSURE_PCT. The cpu signal is off by default because a busy node's "some" is normally high and there is no calibration yet. A node with no report newer than three polls does not trigger lowering but keeps the pool from being raised, since whether it can take more is unknown. Pressure on a node outside the eligibility label is ignored, as memory already is.The role installs the DaemonSet (tolerating every taint, on the nodes runner pods may use), its ServiceAccount, Roles scoped by resourceNames to the one ConfigMap and a get-only Role for the autoscaler, and pre-creates the ConfigMap empty like the status one. Turning the option off, or having no autoscaled profile, removes them.
Tests: the reporter against scripted pressure files (each line the kernel has, a line it does not report left out, a missing file, configurable names), and the autoscaler against reporter data (pressure lowers by one and names the node, never below the running count, healthy nodes change nothing, a stale or missing report holds the pool without lowering, a stale high reading is not pressure, cpu off unless asked, thresholds movable, nothing read without the ConfigMap, a node outside the selector ignored). I also ran the script in the real image on a cgroup v1 and a cgroup v2 node with a stub kubectl and it reported what
/proc/pressureshowed on each. The full unit suite andansible-lint --profile productionpass.Closes #82