diff --git a/README.md b/README.md index 76613d5..d1a2598 100644 --- a/README.md +++ b/README.md @@ -150,6 +150,7 @@ Each poll (`AUTOSCALER_POLL_SECONDS`, default 45s): - Treats any unschedulable runner pod (`PodScheduled=False`, reason `Unschedulable`) like pressure: the memory budget sums every node's capacity, so it cannot see a node that is tainted for disk pressure or a request that fits no single node. It lowers the combined `maxRunners` to the running count and never raises while one exists. - Also collects the peak memory each job's runner measured for itself: the runner image's job-completed hook (`runner-hooks/job-completed.sh`, set through `ACTIONS_RUNNER_HOOK_JOB_COMPLETED`) reads the container cgroup's `memory.peak`, which a spike shorter than the poll cannot escape, and posts it as a `JobMemoryPeak` Event in the pod's namespace. The autoscaler reads those Events each poll and keeps the maximum per repository, workflow ref and job id under the status ConfigMap's `job-peak` key (`kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-peak}'`), in MiB. Keeping the maximum means a forged low figure lowers nothing and re-reading an Event changes nothing. The hook never fails the job it ends. To post an Event the runner pods' service account, which otherwise holds no permission, gets `create` on `events` in its own namespace and nothing else (`github_runner_arc_record_job_peaks`, default on; `false` removes the grant). A job's code runs in the same container and can read the same token, so the worst it can do with that permission is post events in its own runner namespace. - Records, per job, the highest memory a runner pod running it has been seen at, in the status ConfigMap's `job-memory` key (`kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.job-memory}'`), keyed by repository, workflow ref and job name with the peak in MiB, the number of observations and when it was last seen. It is sampled once per poll from the metrics server, so it is a lower bound that a spike shorter than the poll escapes; it shows how far below a pod's request jobs sit, which is what decides whether an accurate in-pod measurement (see the per-job sizing issue) is worth building. Entries unseen for 30 days are dropped, and a failure to keep the record never affects a capacity decision. +- Optionally reads per-node pressure from the nodes' own `/proc/pressure` (`github_runner_arc_autoscaler_node_pressure_enabled`). A `psi-reporter` DaemonSet (`scripts/psi-reporter.sh`, in the autoscaler's image) runs on each node runner pods may use, tolerating every taint, and patches that node's own key of one ConfigMap with the `avg10` of its io, cpu and memory pressure. The reading is not taken from the kubelet's summary API, because on a cgroup v1 node that reports every pressure field as 0 whatever the node is doing. The autoscaler lowers the combined `maxRunners` by exactly one each poll, never below the running count, while any node's I/O "full" stall is at or over `AUTOSCALER_IO_PRESSURE_PCT` (default 10) or, when set above 0, its cpu "some" stall is at or over `AUTOSCALER_CPU_PRESSURE_PCT`, and it does not raise the pool while any node has no report newer than three polls, since whether that node can take more is then unknown. This is the signal for a pool whose nodes differ: memory can have room on a node whose disk is already the limit. - Fails safe on any memory measurement error (`kubectl top nodes`/`kubectl top pod` failing or unparseable): lowers the combined total toward `AUTOSCALER_FLOOR` (`github_runner_arc_autoscaler_floor`) immediately rather than skipping the cycle silently. A failure reading the scale sets' own state or their pods exits non-zero instead, since fail-safe needs that state to lower anything and the API server is then unreachable anyway. This combined floor need not equal any one profile's own values-file `maxRunners`, since each profile's Helm upgrade reverts to its own static floor independently of how the pool's combined floor is set. Ships with `AUTOSCALER_DRY_RUN=true` by default: it computes and logs the target and writes its own status, but never patches. Generate real concurrent traffic to watch it react: `gh workflow run test-autoscaler.yml`, then watch `kubectl logs -n github-runner-platform deploy/autoscaler -f` and `kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.status\.json}'`. Only set `AUTOSCALER_DRY_RUN=false` after watching real dry-run output across genuine CI traffic. Status and its raise-confirm counter live in a shared Kubernetes ConfigMap (`autoscaler-status`, pre-created empty so neither pod's own ServiceAccount ever needs `create` RBAC on it, only `get`/`patch`) rather than a bind-mounted directory, since the two pods can land on different nodes now — `scripts/heartbeat.sh` reads that same ConfigMap back out and republishes it as a second file in the heartbeat gist, reusing its existing gist-write credential rather than giving the autoscaler its own. diff --git a/autoscaler/Dockerfile b/autoscaler/Dockerfile index ad7155c..fc8ebff 100644 --- a/autoscaler/Dockerfile +++ b/autoscaler/Dockerfile @@ -13,7 +13,8 @@ RUN apk add --no-cache bash curl jq kubectl COPY autoscaler/loop.sh /app/loop.sh COPY scripts/autoscaler.sh /app/scripts/autoscaler.sh -RUN chmod +x /app/loop.sh /app/scripts/autoscaler.sh +COPY scripts/psi-reporter.sh /app/scripts/psi-reporter.sh +RUN chmod +x /app/loop.sh /app/scripts/autoscaler.sh /app/scripts/psi-reporter.sh WORKDIR /app ENTRYPOINT ["/app/loop.sh"] diff --git a/roles/github_runner_arc/README.md b/roles/github_runner_arc/README.md index 6472e66..19de6b4 100644 --- a/roles/github_runner_arc/README.md +++ b/roles/github_runner_arc/README.md @@ -40,6 +40,7 @@ Helm 4 and server-side apply. The role reads `helm version --short` and, on Helm - `github_runner_arc_metrics_enabled` (and the `_addr`/`_endpoint` settings): Prometheus metrics for the controller and every listener; the 0.14 chart only enables them together. - `github_runner_arc_listener_probes_enabled`: readiness and liveness probes on each listener's metrics endpoint, so a listener that starts and then fails its GitHub authentication is not counted ready during a rollout. Needs metrics on. - `github_runner_arc_control_pods_priority_enabled`: runs the controller and every listener under the platform's PriorityClass (`github_runner_arc_platform_priority_class_name`, created by the role whether or not the platform is installed). A listener is one pod per scale set that ARC cannot replicate, and while it waits to be scheduled no new jobs are assigned; with the class a rescheduled one can take a full node's room from a runner pod, whose job ARC retries, instead of waiting behind it. Off by default, as it changes what a full node evicts. +- `github_runner_arc_autoscaler_node_pressure_enabled`, `github_runner_arc_autoscaler_io_pressure_pct`, `github_runner_arc_autoscaler_cpu_pressure_pct`, `github_runner_arc_psi_reporter_seconds`, `github_runner_arc_node_pressure_configmap`: per-node pressure for the autoscaler, from each node's own `/proc/pressure` through a `psi-reporter` DaemonSet on the nodes runner pods may use (see the Autoscaler section of the repository README). Needs a profile with `autoscale: true`; with the option off, or no autoscaled profile, the role removes the DaemonSet and its ConfigMap and RBAC. - `github_runner_arc_node_label_key`, `github_runner_arc_node_label_value`: when the key is set, the controller, listeners and runner pods are restricted to nodes carrying the label, and the autoscaler reads memory pressure from those nodes only. The role labels the nodes named in `github_runner_arc_labelled_nodes`. - `github_runner_arc_ghcr_username`, `github_runner_arc_ghcr_token`, `github_runner_arc_heartbeat_gh_token`: secrets, as plain variables. The pull credential is needed only for a static pull Secret. - `github_runner_arc_manage_secrets`: `false` stops the role writing the App Secrets and `heartbeat-gh-token`, and instead checks before any change that they already exist, and that each App Secret holds `github_app_id`, `github_app_installation_id` and `github_app_private_key`. The role then needs no `private_key` or `installation_id`, and no `app_id` either: it reads the App's id from the Secret where it needs it. An `app_id` that is set must match the Secret's `github_app_id`. diff --git a/roles/github_runner_arc/defaults/main.yml b/roles/github_runner_arc/defaults/main.yml index 060e628..b381fbf 100644 --- a/roles/github_runner_arc/defaults/main.yml +++ b/roles/github_runner_arc/defaults/main.yml @@ -147,6 +147,12 @@ github_runner_arc_autoscaler_dry_run: true github_runner_arc_autoscaler_poll_seconds: 45 github_runner_arc_autoscaler_raise_confirm_polls: 2 github_runner_arc_autoscaler_mem_available_pressure_pct: 15 +# Per-node pressure for the autoscaler, from each node's own /proc/pressure (scripts/psi-reporter.sh, a DaemonSet on the nodes runner pods may use, publishing to one ConfigMap). The kubelet's summary API is not used: on a cgroup v1 node it reports every pressure field as 0. With it on, a node whose I/O "full" stall is at or over the first figure lowers the pooled maxRunners by one a poll, and one whose cpu "some" stall is at or over the second does the same when it is above 0. A node with no recent report keeps the pool from being raised. The cpu figure is off by default: a busy node's cpu "some" is normally high and there is no calibration yet for where it starts to cost jobs. +github_runner_arc_autoscaler_node_pressure_enabled: false +github_runner_arc_autoscaler_io_pressure_pct: 10 +github_runner_arc_autoscaler_cpu_pressure_pct: 0 +github_runner_arc_node_pressure_configmap: "autoscaler-node-pressure" +github_runner_arc_psi_reporter_seconds: 15 # Node recovery (see tasks/install_node_recovery.yml and scripts/node-recovery.sh). When a node's kubelet stops reporting, the control plane marks the node's pods for deletion but only the kubelet can confirm it, so they stay Terminating, and an ARC listener on that node is never replaced. The watcher gives such a node the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint once its Ready condition has been Unknown for github_runner_arc_node_recovery_after_seconds, which makes the control plane evict its pods and force-delete the terminating ones, and removes the taint when the node reports Ready again. false removes the watcher and any taint it left. github_runner_arc_node_recovery_enabled: true diff --git a/roles/github_runner_arc/meta/argument_specs.yml b/roles/github_runner_arc/meta/argument_specs.yml index ba09728..49f7354 100644 --- a/roles/github_runner_arc/meta/argument_specs.yml +++ b/roles/github_runner_arc/meta/argument_specs.yml @@ -288,6 +288,26 @@ argument_specs: type: int default: 15 description: Available memory percentage below which the autoscaler treats a node as under pressure. + github_runner_arc_autoscaler_node_pressure_enabled: + type: bool + default: false + description: Installs a per-node reporter of /proc/pressure (a DaemonSet on the nodes runner pods may use) and has the autoscaler lower maxRunners by one a poll while a node's pressure is over the thresholds below, and hold off raising it while a node has no recent report. Needs an autoscaled profile. + github_runner_arc_autoscaler_io_pressure_pct: + type: int + default: 10 + description: The "full" avg10 of a node's /proc/pressure/io, in percent, at or over which the autoscaler treats the node as under pressure. Used with github_runner_arc_autoscaler_node_pressure_enabled. + github_runner_arc_autoscaler_cpu_pressure_pct: + type: int + default: 0 + description: The "some" avg10 of a node's /proc/pressure/cpu, in percent, at or over which the autoscaler treats the node as under pressure; 0 leaves the cpu signal off. Used with github_runner_arc_autoscaler_node_pressure_enabled. + github_runner_arc_node_pressure_configmap: + type: str + default: autoscaler-node-pressure + description: The ConfigMap in the platform namespace that each node's reporter patches its own key of. + github_runner_arc_psi_reporter_seconds: + type: int + default: 15 + description: Seconds between each reporter's reports. The autoscaler treats a report older than three of its polls as no report. github_runner_arc_node_recovery_enabled: type: bool default: true diff --git a/roles/github_runner_arc/tasks/install_platform.yml b/roles/github_runner_arc/tasks/install_platform.yml index 0e62730..66425f9 100644 --- a/roles/github_runner_arc/tasks/install_platform.yml +++ b/roles/github_runner_arc/tasks/install_platform.yml @@ -498,6 +498,13 @@ value: "{{ github_runner_arc_fleet.autoscaled | map(attribute='target') | join(' ') }}" - name: AUTOSCALER_STATUS_CONFIGMAP value: autoscaler-status + # Empty leaves node pressure out; with it set the autoscaler reads each node's reporter entry from this ConfigMap (tasks/install_psi_reporter.yml). + - name: AUTOSCALER_NODE_PRESSURE_CONFIGMAP + value: "{{ github_runner_arc_node_pressure_configmap if github_runner_arc_autoscaler_node_pressure_enabled | bool else '' }}" + - name: AUTOSCALER_IO_PRESSURE_PCT + value: "{{ github_runner_arc_autoscaler_io_pressure_pct | string }}" + - name: AUTOSCALER_CPU_PRESSURE_PCT + value: "{{ github_runner_arc_autoscaler_cpu_pressure_pct | string }}" resources: requests: cpu: "50m" @@ -507,3 +514,7 @@ cpu: "200m" memory: "64Mi" ephemeral-storage: "{{ github_runner_arc_platform_ephemeral_storage_limit }}" + +# The per-node pressure reporters the autoscaler reads, or their removal when turned off. +- name: "Platform: install the node pressure reporter, or remove it when turned off" + ansible.builtin.include_tasks: install_psi_reporter.yml diff --git a/roles/github_runner_arc/tasks/install_psi_reporter.yml b/roles/github_runner_arc/tasks/install_psi_reporter.yml new file mode 100644 index 0000000..4bccaf6 --- /dev/null +++ b/roles/github_runner_arc/tasks/install_psi_reporter.yml @@ -0,0 +1,175 @@ +--- +# The per-node pressure reporter (scripts/psi-reporter.sh) and what the autoscaler needs to read it, installed when github_runner_arc_autoscaler_node_pressure_enabled is on and a profile is autoscaled, removed otherwise. One DaemonSet pod on each node runner pods may use, with the host's /proc/pressure mounted read-only; each patches its own node's key of one shared ConfigMap. Like the autoscaler's own state ConfigMap, that one is pre-created empty, so no ServiceAccount here is granted `create`, only get/patch scoped by resourceNames. + +- name: "Platform: decide whether the node pressure reporter is wanted" + ansible.builtin.set_fact: + github_runner_arc_psi_reporter_wanted: "{{ github_runner_arc_autoscaler_node_pressure_enabled | bool and github_runner_arc_fleet.autoscaled | length > 0 }}" + +- name: "Platform: remove the node pressure reporter and its state when it is not wanted" + when: not github_runner_arc_psi_reporter_wanted + kubernetes.core.k8s: + state: absent + api_version: "{{ item.api_version }}" + kind: "{{ item.kind }}" + name: "{{ item.name }}" + namespace: "{{ github_runner_arc_platform_namespace }}" + loop: + - api_version: apps/v1 + kind: DaemonSet + name: psi-reporter + - api_version: v1 + kind: ConfigMap + name: "{{ github_runner_arc_node_pressure_configmap }}" + - api_version: rbac.authorization.k8s.io/v1 + kind: RoleBinding + name: psi-reporter + - api_version: rbac.authorization.k8s.io/v1 + kind: Role + name: psi-reporter + - api_version: rbac.authorization.k8s.io/v1 + kind: RoleBinding + name: github-runner-autoscaler-node-pressure-reader + - api_version: rbac.authorization.k8s.io/v1 + kind: Role + name: github-runner-autoscaler-node-pressure-reader + - api_version: v1 + kind: ServiceAccount + name: psi-reporter + +- name: "Platform: install the node pressure reporter" + when: github_runner_arc_psi_reporter_wanted + block: + - name: "Platform: node pressure reporter service account" + kubernetes.core.k8s: + definition: + apiVersion: v1 + kind: ServiceAccount + metadata: + name: psi-reporter + namespace: "{{ github_runner_arc_platform_namespace }}" + + - name: "Platform: node pressure RBAC - the reporters patch the shared ConfigMap, the autoscaler reads it" + kubernetes.core.k8s: + definition: "{{ item }}" + loop: + - apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: psi-reporter + namespace: "{{ github_runner_arc_platform_namespace }}" + rules: + - apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["{{ github_runner_arc_node_pressure_configmap }}"] + verbs: ["get", "patch"] + - apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: psi-reporter + namespace: "{{ github_runner_arc_platform_namespace }}" + subjects: + - kind: ServiceAccount + name: psi-reporter + namespace: "{{ github_runner_arc_platform_namespace }}" + roleRef: + kind: Role + name: psi-reporter + apiGroup: rbac.authorization.k8s.io + - apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: github-runner-autoscaler-node-pressure-reader + namespace: "{{ github_runner_arc_platform_namespace }}" + rules: + - apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["{{ github_runner_arc_node_pressure_configmap }}"] + verbs: ["get"] + - apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: github-runner-autoscaler-node-pressure-reader + namespace: "{{ github_runner_arc_platform_namespace }}" + subjects: + - kind: ServiceAccount + name: autoscaler + namespace: "{{ github_runner_arc_platform_namespace }}" + roleRef: + kind: Role + name: github-runner-autoscaler-node-pressure-reader + apiGroup: rbac.authorization.k8s.io + + - name: "Platform: check whether the node pressure ConfigMap exists" + kubernetes.core.k8s_info: + kind: ConfigMap + namespace: "{{ github_runner_arc_platform_namespace }}" + name: "{{ github_runner_arc_node_pressure_configmap }}" + register: github_runner_arc_node_pressure_configmap_state + + - name: "Platform: pre-create the node pressure ConfigMap" + when: github_runner_arc_node_pressure_configmap_state.resources | length == 0 + kubernetes.core.k8s: + definition: + apiVersion: v1 + kind: ConfigMap + metadata: + name: "{{ github_runner_arc_node_pressure_configmap }}" + namespace: "{{ github_runner_arc_platform_namespace }}" + data: {} + + # The autoscaler's own image carries the script, so there is no image of its own to build. It runs on every node runner pods may use, taints included: a node that carries a taint to keep other workloads off is exactly one whose pressure matters, so it tolerates everything. + - name: "Platform: install/upgrade the node pressure reporter DaemonSet" + kubernetes.core.k8s: + definition: + apiVersion: apps/v1 + kind: DaemonSet + metadata: + name: psi-reporter + namespace: "{{ github_runner_arc_platform_namespace }}" + spec: + selector: + matchLabels: + app: psi-reporter + template: + metadata: + labels: + app: psi-reporter + spec: + serviceAccountName: psi-reporter + priorityClassName: "{{ github_runner_arc_platform_priority_class_name }}" + imagePullSecrets: "{{ github_runner_arc_platform_image_pull_secrets }}" + nodeSelector: "{{ {github_runner_arc_node_label_key: github_runner_arc_node_label_value | string} if github_runner_arc_node_label_key | length > 0 else {} }}" + tolerations: + - operator: Exists + containers: + - name: psi-reporter + image: "{{ github_runner_arc_autoscaler_image }}" + command: ["/app/scripts/psi-reporter.sh"] + env: + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: NAMESPACE + value: "{{ github_runner_arc_platform_namespace }}" + - name: NODE_PRESSURE_CONFIGMAP + value: "{{ github_runner_arc_node_pressure_configmap }}" + - name: REPORT_SECONDS + value: "{{ github_runner_arc_psi_reporter_seconds | string }}" + volumeMounts: + - name: pressure + mountPath: /host-pressure + readOnly: true + resources: + requests: + cpu: "10m" + memory: "16Mi" + ephemeral-storage: "{{ github_runner_arc_platform_ephemeral_storage_request }}" + limits: + cpu: "50m" + memory: "32Mi" + ephemeral-storage: "{{ github_runner_arc_platform_ephemeral_storage_limit }}" + volumes: + - name: pressure + hostPath: + path: /proc/pressure diff --git a/scripts/autoscaler.sh b/scripts/autoscaler.sh index d07294e..a3ee24e 100755 --- a/scripts/autoscaler.sh +++ b/scripts/autoscaler.sh @@ -10,6 +10,7 @@ # - AUTOSCALER_RAISE_CONFIRM_POLLS: consecutive polls of confirmed headroom before raising # - AUTOSCALER_MEM_AVAILABLE_PRESSURE_PCT: MemAvailable-of-MemTotal percentage treated as real pressure, evaluated as the worst case across the nodes runner pods may use (see the kubectl top nodes block below) - no swap-pressure equivalent, see that block's own comment for why # - AUTOSCALER_NODE_SELECTOR: a label selector (key=value) naming the nodes runner pods may be placed on, the role's node eligibility label; empty means every node. Only those nodes' memory is read, so a node that runs no runner pod, and whose metrics go missing when it drops out, never lowers the pool +# - AUTOSCALER_NODE_PRESSURE_CONFIGMAP: the ConfigMap the psi-reporter DaemonSet (scripts/psi-reporter.sh) keeps each node's own pressure stall figures in, in AUTOSCALER_STATE_NAMESPACE. Empty means no node-pressure signal, as before. With it set, a node over AUTOSCALER_IO_PRESSURE_PCT (the "full" avg10 of /proc/pressure/io, default 10) or over AUTOSCALER_CPU_PRESSURE_PCT (the "some" avg10 of /proc/pressure/cpu, default 0, which turns the cpu signal off) lowers the pooled total by one each poll, never below the running count, and a node with no report newer than three polls is not a reason to raise. # - AUTOSCALER_TARGETS: the AutoscalingRunnerSets to pool, as whitespace-separated namespace/release tokens (one per scale-set profile with autoscale: true; see plugins/filter/arc.py's own target field, which is exactly this namespace/release form) # - AUTOSCALER_STATE_NAMESPACE, AUTOSCALER_STATUS_CONFIGMAP: the ConfigMap this pod's own status/raise-confirm-count state lives in (shared with scripts/heartbeat.sh, which reads status.json back out to republish in the heartbeat gist) set -euo pipefail @@ -23,6 +24,14 @@ FLOOR="${AUTOSCALER_FLOOR:?AUTOSCALER_FLOOR must be set}" RAISE_CONFIRM_POLLS="${AUTOSCALER_RAISE_CONFIRM_POLLS:-2}" MEM_AVAILABLE_PRESSURE_PCT="${AUTOSCALER_MEM_AVAILABLE_PRESSURE_PCT:-15}" NODE_SELECTOR="${AUTOSCALER_NODE_SELECTOR:-}" +NODE_PRESSURE_CONFIGMAP="${AUTOSCALER_NODE_PRESSURE_CONFIGMAP:-}" +# Linux's own reading of "full" stall for I/O: the share of the last ten seconds in which every task on the node was waiting on I/O. A node doing its work shows well under a percent; the ones that made jobs miss their timeouts showed 20 to 50. Ten is where a disk is the limit, not a moment. +IO_PRESSURE_PCT="${AUTOSCALER_IO_PRESSURE_PCT:-10}" +# Off by default: a busy node's "some" for cpu is normally high, and there is no figure here yet for where it starts to cost jobs. +CPU_PRESSURE_PCT="${AUTOSCALER_CPU_PRESSURE_PCT:-0}" +# A report older than this many polls counts as no report, since a reporter that has stopped says nothing about the node. +NODE_PRESSURE_MAX_AGE_POLLS=3 +POLL_SECONDS="${AUTOSCALER_POLL_SECONDS:-45}" STATE_NAMESPACE="${AUTOSCALER_STATE_NAMESPACE:-github-runner-platform}" CONFIGMAP="${AUTOSCALER_STATUS_CONFIGMAP:-autoscaler-status}" # Where each job's observed peak memory is kept, in the same ConfigMap. A ConfigMap holds at most 1 MiB and an entry is under 512 bytes, so capping the entries at 2048 stays at half the limit; entries unseen for 30 days are dropped, since a job that has not run for that long no longer informs sizing. @@ -334,18 +343,53 @@ if [ "$mem_available_pct" -lt "$MEM_AVAILABLE_PRESSURE_PCT" ]; then pressure=true fi +# Per-node pressure from the reporters, when there are any. The eligible nodes are read from the cluster and each one's entry from the shared ConfigMap, so a node whose reporter is missing or stopped is seen as unmeasured instead of being left out. +node_pressure=false +node_pressure_reason="" +unmeasured_nodes=0 +if [ -n "$NODE_PRESSURE_CONFIGMAP" ]; then + NODES_ARGS=(get nodes -o name) + [ -n "$NODE_SELECTOR" ] && NODES_ARGS+=(-l "$NODE_SELECTOR") + ELIGIBLE_NODES="$(kubectl "${NODES_ARGS[@]}" 2>/dev/null)" || fail_safe "could not list the nodes runner pods may use" + PRESSURE_DATA="$(kubectl get configmap "$NODE_PRESSURE_CONFIGMAP" -n "$STATE_NAMESPACE" -o json 2>/dev/null | jq -c '.data // {}')" \ + || fail_safe "could not read the node pressure ConfigMap ${NODE_PRESSURE_CONFIGMAP}" + now="$(date +%s)" + max_age=$(( NODE_PRESSURE_MAX_AGE_POLLS * POLL_SECONDS )) + while read -r node_ref; do + [ -n "$node_ref" ] || continue + node="${node_ref#node/}" + entry="$(jq -r --arg n "$node" '.[$n] // empty' <<< "$PRESSURE_DATA")" + reported_at="$(jq -r '.at // empty' <<< "${entry:-null}" 2>/dev/null || true)" + case "$reported_at" in ''|*[!0-9]*) unmeasured_nodes=$(( unmeasured_nodes + 1 )); continue ;; esac + if [ $(( now - reported_at )) -gt "$max_age" ]; then + unmeasured_nodes=$(( unmeasured_nodes + 1 )) + continue + fi + io_full="$(jq -r '.io_full_avg10 // 0' <<< "$entry")" + cpu_some="$(jq -r '.cpu_some_avg10 // 0' <<< "$entry")" + # Percentages come as decimals ("19.41"); the comparison is awk's, not bash's integers. + if awk -v v="$io_full" -v t="$IO_PRESSURE_PCT" 'BEGIN { exit !(v >= t) }'; then + node_pressure=true + node_pressure_reason="${node} io stall ${io_full}% >= ${IO_PRESSURE_PCT}%" + elif [ "$CPU_PRESSURE_PCT" != "0" ] && awk -v v="$cpu_some" -v t="$CPU_PRESSURE_PCT" 'BEGIN { exit !(v >= t) }'; then + node_pressure=true + node_pressure_reason="${node} cpu stall ${cpu_some}% >= ${CPU_PRESSURE_PCT}%" + fi + done <<< "$ELIGIBLE_NODES" +fi + # ---- Algorithm --------------------------------------------------------- USABLE_BUDGET_MIB=$(( USABLE_BUDGET_GI * 1024 )) HEADROOM_MIB=$(( USABLE_BUDGET_MIB - USAGE_MIB )) -echo "R_total=${R_TOTAL} maxRunners_total=${MAX_TOTAL} Wh=${WH_MIB}MiB usage=${USAGE_MIB}MiB headroom=${HEADROOM_MIB}MiB mem_available=${mem_available_pct}% pressure=${pressure} unschedulable=${UNSCHEDULABLE_TOTAL} targets=${TARGETS[*]}" +echo "R_total=${R_TOTAL} maxRunners_total=${MAX_TOTAL} Wh=${WH_MIB}MiB usage=${USAGE_MIB}MiB headroom=${HEADROOM_MIB}MiB mem_available=${mem_available_pct}% pressure=${pressure} node_pressure=${node_pressure} unmeasured_nodes=${unmeasured_nodes} unschedulable=${UNSCHEDULABLE_TOTAL} targets=${TARGETS[*]}" # The ceiling is a hard cap on the pooled total, so a total above it (each profile's own static maxRunners can sum to more, and every Helm upgrade restores those) is lowered to it even when nothing else is wrong. over_ceiling=false [ "$MAX_TOTAL" -gt "$MAX_CEILING" ] && over_ceiling=true -if [ "$pressure" = "true" ] || [ "$HEADROOM_MIB" -lt "$WH_MIB" ] || [ "$UNSCHEDULABLE_TOTAL" -gt 0 ] || [ "$over_ceiling" = "true" ]; then +if [ "$pressure" = "true" ] || [ "$HEADROOM_MIB" -lt "$WH_MIB" ] || [ "$UNSCHEDULABLE_TOTAL" -gt 0 ] || [ "$over_ceiling" = "true" ] || [ "$node_pressure" = "true" ]; then # Lower immediately, no delay or averaging — lowering never disrupts in-flight jobs (ARC only gates new claims), so there is no cost to being trigger-happy in this direction. Can drop below FLOOR (even to 0, spread across targets by lower_to) if pressure is severe enough. configmap_set "raise-confirm-count" "0" configmap_set "$REBALANCE_CONFIRM_KEY" "0" @@ -354,12 +398,17 @@ if [ "$pressure" = "true" ] || [ "$HEADROOM_MIB" -lt "$WH_MIB" ] || [ "$UNSCHEDU if [ "$pressure" != "true" ] && [ "$HEADROOM_MIB" -ge "$WH_MIB" ] && [ "$UNSCHEDULABLE_TOTAL" -eq 0 ]; then target_total=$MAX_CEILING fi + # Only a node's own pressure: the pool has the memory, so it steps down by one a poll, not to the floor, and stops at the running count. + if [ "$node_pressure" = "true" ] && [ "$pressure" != "true" ] && [ "$HEADROOM_MIB" -ge "$WH_MIB" ] && [ "$UNSCHEDULABLE_TOTAL" -eq 0 ] && [ "$over_ceiling" != "true" ]; then + target_total=$(( MAX_TOTAL - 1 )) + fi [ "$pressure" = "true" ] && target_total=1 [ "$UNSCHEDULABLE_TOTAL" -gt 0 ] && target_total=0 # clamped to R_TOTAL below: stop asking for runners no node can host, never orphan a running job [ "$target_total" -lt "$R_TOTAL" ] && target_total=$R_TOTAL if [ "$target_total" -lt "$MAX_TOTAL" ]; then reason="lower: headroom=${HEADROOM_MIB}MiB < Wh=${WH_MIB}MiB" [ "$over_ceiling" = "true" ] && reason="lower: pooled maxRunners ${MAX_TOTAL} is above the ceiling ${MAX_CEILING}" + [ "$node_pressure" = "true" ] && reason="lower: node pressure (${node_pressure_reason})" [ "$pressure" = "true" ] && reason="lower: host pressure (mem_available=${mem_available_pct}%)" [ "$UNSCHEDULABLE_TOTAL" -gt 0 ] && reason="lower: ${UNSCHEDULABLE_TOTAL} runner pod(s) unschedulable, so the budgeted capacity is not all schedulable" lower_to "$target_total" @@ -367,7 +416,7 @@ if [ "$pressure" = "true" ] || [ "$HEADROOM_MIB" -lt "$WH_MIB" ] || [ "$UNSCHEDU else write_status "$MAX_TOTAL" "steady: already at or below the safe target" "$HEADROOM_MIB" fi -elif [ "$HEADROOM_MIB" -ge "$WH_MIB" ] && [ "$MAX_TOTAL" -lt "$MAX_CEILING" ]; then +elif [ "$HEADROOM_MIB" -ge "$WH_MIB" ] && [ "$MAX_TOTAL" -lt "$MAX_CEILING" ] && [ "$unmeasured_nodes" -eq 0 ]; then configmap_set "$REBALANCE_CONFIRM_KEY" "0" confirm_count="$(configmap_get 'raise-confirm-count')" [ -n "$confirm_count" ] || confirm_count=0 @@ -380,6 +429,11 @@ elif [ "$HEADROOM_MIB" -ge "$WH_MIB" ] && [ "$MAX_TOTAL" -lt "$MAX_CEILING" ]; t else write_status "$MAX_TOTAL" "watching: headroom confirmed for ${confirm_count}/${RAISE_CONFIRM_POLLS} polls" "$HEADROOM_MIB" fi +elif [ "$unmeasured_nodes" -gt 0 ] && [ "$HEADROOM_MIB" -ge "$WH_MIB" ] && [ "$MAX_TOTAL" -lt "$MAX_CEILING" ]; then + # Headroom would raise the pool, but a node has no fresh pressure report, so whether it can take more is not known. Holding is the safe reading, and it is not lowering, since nothing says there is pressure. + configmap_set "raise-confirm-count" "0" + configmap_set "$REBALANCE_CONFIRM_KEY" "0" + write_status "$MAX_TOTAL" "steady: ${unmeasured_nodes} node(s) have no fresh pressure report, so the pool is not raised" "$HEADROOM_MIB" else # Here the pool is healthy and holds as many runners as it may (below the ceiling with enough headroom is the raise branch above; pressure, an unschedulable pod and a total over the ceiling are the lower branch). Slots are fixed per target, so a target that has used all of its own while another sits idle would queue for capacity the pool has. Move one slot at a time, and only once the same imbalance has been seen for as many polls as a raise needs, so a burst that ends within a poll or two moves nothing. configmap_set "raise-confirm-count" "0" diff --git a/scripts/psi-reporter.sh b/scripts/psi-reporter.sh new file mode 100755 index 0000000..1940585 --- /dev/null +++ b/scripts/psi-reporter.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Publishes this node's own pressure stall information for the autoscaler. Runs in the `psi-reporter` DaemonSet, one pod per node a runner pod may use, with the host's /proc/pressure mounted read-only. +# +# The reading comes from the node's own /proc/pressure, not from the kubelet's summary API: on a cgroup v1 node that API reports every pressure field as 0 whatever the node is doing, so a reader built on it would find no pressure on exactly the nodes that have some. +# +# Env: +# - NODE_NAME: the node this pod runs on (the downward API's spec.nodeName). Required. +# - NAMESPACE: the namespace of the ConfigMap. Required. +# - NODE_PRESSURE_CONFIGMAP: the shared ConfigMap, pre-created empty, that every reporter patches its own key of (default: autoscaler-node-pressure). +# - PRESSURE_DIR: where the host's /proc/pressure is mounted (default: /host-pressure). +# - REPORT_SECONDS: seconds between reports (default: 15). The autoscaler treats a report older than a few polls as no report. +# - REPORT_ONCE: when "true", report once and exit, for tests. +# +# The value stored under the node's name is a JSON object: at (epoch seconds), and for each of io, cpu and memory the avg10 of its "some" and, where the kernel reports it, "full" lines, as __avg10. A line the kernel does not report (the root cpu "full" before kernel 5.13) is left out, not reported as 0. +set -euo pipefail + +: "${NODE_NAME:?NODE_NAME is required}" +: "${NAMESPACE:?NAMESPACE is required}" +NODE_PRESSURE_CONFIGMAP="${NODE_PRESSURE_CONFIGMAP:-autoscaler-node-pressure}" +PRESSURE_DIR="${PRESSURE_DIR:-/host-pressure}" +REPORT_SECONDS="${REPORT_SECONDS:-15}" +REPORT_ONCE="${REPORT_ONCE:-false}" + +# The avg10 figures of one resource's file as `__avg10: ` JSON members, one per line the file has. +members() { + local resource=$1 kind avg10 + [ -r "${PRESSURE_DIR}/${resource}" ] || return 0 + while read -r kind rest; do + avg10="$(sed -n 's/.*avg10=\([0-9.]*\).*/\1/p' <<< "$rest")" + [ -n "$avg10" ] && printf '"%s_%s_avg10": %s\n' "$resource" "$kind" "$avg10" + done < "${PRESSURE_DIR}/${resource}" +} + +report() { + local entry patch + entry="$( { printf '"at": %s\n' "$(date +%s)"; members io; members cpu; members memory; } | paste -sd, - )" + patch="$(mktemp)" + # The node's name is the key and the entry a JSON string, so one reporter's patch never touches another's key. The patch goes in a file: it is read back as a whole, and an argument has a size limit. + jq -n --arg node "$NODE_NAME" --arg entry "{${entry}}" '{data: {($node): $entry}}' > "$patch" + kubectl patch configmap "$NODE_PRESSURE_CONFIGMAP" -n "$NAMESPACE" --type merge --patch-file "$patch" + rm -f "$patch" +} + +while true; do + report + [ "$REPORT_ONCE" = "true" ] && exit 0 + sleep "$REPORT_SECONDS" +done diff --git a/tests/unit/test_autoscaler.py b/tests/unit/test_autoscaler.py index 8492479..4c2bda4 100644 --- a/tests/unit/test_autoscaler.py +++ b/tests/unit/test_autoscaler.py @@ -9,6 +9,7 @@ import subprocess import tempfile import textwrap +import time import unittest from pathlib import Path @@ -59,6 +60,16 @@ sys.exit(1) sys.stdout.write(json.dumps({"status": runner})) sys.exit(0) + if args[:2] == ["get", "nodes"]: + # `-o name`, optionally narrowed by the same key=value selector the autoscaler passes to `top nodes`. + selector = args[args.index("-l") + 1] if "-l" in args else None + for node in state["nodes"]: + if selector is None or selector in node["labels"]: + sys.stdout.write("node/%s\\n" % node["name"]) + sys.exit(0) + if args[:3] == ["get", "configmap", state["pressure_configmap"]] and "json" in args: + sys.stdout.write(json.dumps({"data": state["pressure"]})) + sys.exit(0) if args[:2] == ["top", "nodes"]: # -l takes one key=value selector, as the autoscaler passes it; a node is listed when it carries that label. Like the real kubectl, a node without metrics is printed as beside ones with them, and when no listed node has metrics at all (none matches, or every match is down) it prints nothing and fails. selector = args[args.index("-l") + 1] if "-l" in args else None @@ -134,6 +145,8 @@ def poll( job_memory_patch_fails: bool = False, events: dict[str, list[str]] | None = None, job_peak: str = "", + pressure: dict[str, str] | None = None, + extra_env: dict[str, str] | None = None, ) -> subprocess.CompletedProcess[str]: state = self.dir / "state.json" state.write_text( @@ -153,6 +166,8 @@ def poll( "job_peak": job_peak, "events": events or {}, "job_memory_patch_fails": job_memory_patch_fails, + "pressure_configmap": "node-pressure", + "pressure": pressure or {}, } ) ) @@ -174,6 +189,9 @@ def poll( } if node_selector is not None: env["AUTOSCALER_NODE_SELECTOR"] = node_selector + if pressure is not None: + env["AUTOSCALER_NODE_PRESSURE_CONFIGMAP"] = "node-pressure" + env.update(extra_env or {}) return subprocess.run(["bash", str(SCRIPT)], env=env, capture_output=True, text=True, check=False) def test_an_unschedulable_runner_pod_lowers_every_target_to_its_running_count(self) -> None: @@ -264,6 +282,70 @@ def test_an_unreachable_node_inside_the_selector_fails_safe(self) -> None: def recorded(self) -> dict[str, dict[str, object]]: return json.loads(self.job_memory_out.read_text()) + # Two nodes runner pods may use, each with a pressure report from its reporter, written as the reporter writes it. + NODES = ({"name": "a", "labels": [ELIGIBLE], "mem": "20%"}, {"name": "b", "labels": [ELIGIBLE], "mem": "30%"}) + + @staticmethod + def report(io_full: float, age_seconds: int = 0, cpu_some: float = 1.0) -> str: + return json.dumps({"at": int(time.time()) - age_seconds, "io_some_avg10": io_full + 1, "io_full_avg10": io_full, "cpu_some_avg10": cpu_some}) + + def pressure_poll(self, pressure: dict[str, str], **kwargs: object) -> subprocess.CompletedProcess[str]: + pods = {"ns-a": [SCHEDULED_POD], "ns-b": [SCHEDULED_POD]} + return self.poll(pods, ceiling=12, nodes=list(self.NODES), node_selector=self.ELIGIBLE, pressure=pressure, **kwargs) # type: ignore[arg-type] + + def test_io_pressure_on_one_node_lowers_the_pool_by_one_and_names_the_node(self) -> None: + result = self.pressure_poll({"a": self.report(19.4), "b": self.report(0.1)}) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("node_pressure=true", result.stdout) + self.assertIn("lower: node pressure (a io stall 19.4% >= 10%)", result.stdout) + # Six pooled runners, two of them running on each target: one fewer, and not the floor. + self.assertIn("would patch ns-a/release-ns-a's maxRunners to 2", result.stdout) + self.assertNotIn("FAIL-SAFE", result.stderr) + + def test_a_node_pressure_step_never_goes_below_the_running_count(self) -> None: + result = self.pressure_poll({"a": self.report(40.0), "b": self.report(0.0)}, max_runners={"ns-a": RUNNING, "ns-b": RUNNING}) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertNotIn("would patch", result.stdout) + + def test_healthy_nodes_with_fresh_reports_change_nothing(self) -> None: + result = self.pressure_poll({"a": self.report(0.2), "b": self.report(0.0)}, max_runners={"ns-a": 12, "ns-b": 0}) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("node_pressure=false unmeasured_nodes=0", result.stdout) + self.assertNotIn("node pressure", result.stdout) + + def test_a_node_with_no_report_or_a_stale_one_holds_the_pool_and_does_not_lower_it(self) -> None: + for pressure in ({"a": self.report(0.0)}, {"a": self.report(0.0), "b": self.report(0.0, age_seconds=1000)}): + result = self.pressure_poll(pressure) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("unmeasured_nodes=1", result.stdout) + self.assertIn("node_pressure=false", result.stdout) + self.assertNotIn("would patch", result.stdout) + + def test_a_stale_report_of_high_pressure_is_not_pressure(self) -> None: + result = self.pressure_poll({"a": self.report(0.0), "b": self.report(60.0, age_seconds=1000)}) + self.assertIn("node_pressure=false unmeasured_nodes=1", result.stdout) + + def test_the_cpu_signal_is_off_unless_a_threshold_is_given(self) -> None: + busy = {"a": self.report(0.0, cpu_some=90.0), "b": self.report(0.0)} + self.assertIn("node_pressure=false", self.pressure_poll(busy).stdout) + asked = self.pressure_poll(busy, extra_env={"AUTOSCALER_CPU_PRESSURE_PCT": "80"}) + self.assertIn("lower: node pressure (a cpu stall 90.0% >= 80%)", asked.stdout) + + def test_the_io_threshold_can_be_moved(self) -> None: + reports = {"a": self.report(19.4), "b": self.report(0.0)} + self.assertNotIn("node_pressure=true", self.pressure_poll(reports, extra_env={"AUTOSCALER_IO_PRESSURE_PCT": "25"}).stdout) + + def test_without_the_pressure_configmap_nothing_about_node_pressure_is_read(self) -> None: + result = self.poll({"ns-a": [SCHEDULED_POD], "ns-b": [SCHEDULED_POD]}, ceiling=12, nodes=list(self.NODES), node_selector=self.ELIGIBLE) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("node_pressure=false unmeasured_nodes=0", result.stdout) + + def test_a_node_outside_the_selector_with_pressure_is_not_pressure(self) -> None: + nodes = [*self.NODES, {"name": "x", "labels": [], "mem": "20%"}] + pods = {"ns-a": [SCHEDULED_POD], "ns-b": [SCHEDULED_POD]} + result = self.poll(pods, ceiling=12, nodes=nodes, node_selector=self.ELIGIBLE, pressure={"a": self.report(0.0), "b": self.report(0.0), "x": self.report(80.0)}, max_runners={"ns-a": 12, "ns-b": 0}) + self.assertIn("node_pressure=false unmeasured_nodes=0", result.stdout) + def test_a_jobs_highest_observed_memory_is_kept_across_polls(self) -> None: runners = {"runner-1": JOB_A} pods: dict[str, list[dict[str, object]]] = {} diff --git a/tests/unit/test_psi_reporter.py b/tests/unit/test_psi_reporter.py new file mode 100644 index 0000000..0e3aa90 --- /dev/null +++ b/tests/unit/test_psi_reporter.py @@ -0,0 +1,91 @@ +"""Tests for scripts/psi-reporter.sh: one report from a scripted /proc/pressure, with a stand-in for kubectl that keeps the patch it is given.""" + +from __future__ import annotations + +import json +import os +import subprocess +import tempfile +import textwrap +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "psi-reporter.sh" + +FAKE_KUBECTL = textwrap.dedent( + """\ + #!/usr/bin/env python3 + import json, os, sys + args = sys.argv[1:] + if args[:2] == ["patch", "configmap"] and "--patch-file" in args: + open(os.environ["FAKE_PATCH_OUT"], "w").write(open(args[args.index("--patch-file") + 1]).read()) + open(os.environ["FAKE_ARGS_OUT"], "w").write(" ".join(args)) + sys.exit(0) + sys.stderr.write("unexpected kubectl call: %s\\n" % args) + sys.exit(2) + """ +) + +IO = "some avg10=27.47 avg60=28.46 avg300=29.43 total=1624379039\nfull avg10=19.41 avg60=23.80 avg300=26.64 total=1491824298\n" +CPU_V1 = "some avg10=0.88 avg60=0.65 avg300=0.87 total=99315065\n" +MEMORY = "some avg10=0.08 avg60=0.04 avg300=0.03 total=26200355\nfull avg10=0.01 avg60=0.03 avg300=0.01 total=21701934\n" + + +class ReporterTest(unittest.TestCase): + def report(self, files: dict[str, str], **env: str) -> tuple[dict[str, object], str]: + with tempfile.TemporaryDirectory() as work: + root = Path(work) + pressure = root / "pressure" + pressure.mkdir() + for name, content in files.items(): + (pressure / name).write_text(content) + bin_dir = root / "bin" + bin_dir.mkdir() + kubectl = bin_dir / "kubectl" + kubectl.write_text(FAKE_KUBECTL) + kubectl.chmod(0o755) + patch_out, args_out = root / "patch.json", root / "args.txt" + full_env = {**os.environ, "PATH": f"{bin_dir}:{os.environ['PATH']}", "NODE_NAME": "node-a", "NAMESPACE": "platform", "PRESSURE_DIR": str(pressure), "REPORT_ONCE": "true", "FAKE_PATCH_OUT": str(patch_out), "FAKE_ARGS_OUT": str(args_out), **env} + subprocess.run(["bash", str(SCRIPT)], check=True, env=full_env, capture_output=True, text=True) + patch: dict[str, object] = json.loads(patch_out.read_text()) + return patch, args_out.read_text() + + def test_reports_each_line_the_kernel_has_under_the_nodes_own_key(self) -> None: + patch, args = self.report({"io": IO, "cpu": CPU_V1, "memory": MEMORY}) + data = patch["data"] + assert isinstance(data, dict) + self.assertEqual(list(data), ["node-a"]) + entry = json.loads(data["node-a"]) + self.assertEqual( + {key: value for key, value in entry.items() if key != "at"}, + {"io_some_avg10": 27.47, "io_full_avg10": 19.41, "cpu_some_avg10": 0.88, "memory_some_avg10": 0.08, "memory_full_avg10": 0.01}, + ) + self.assertGreater(entry["at"], 0) + self.assertIn("patch configmap autoscaler-node-pressure -n platform --type merge --patch-file", args) + + def test_a_line_the_kernel_does_not_report_is_left_out_not_reported_as_zero(self) -> None: + patch, _ = self.report({"io": IO, "cpu": CPU_V1, "memory": MEMORY}) + data = patch["data"] + assert isinstance(data, dict) + self.assertNotIn("cpu_full_avg10", json.loads(data["node-a"])) + + def test_a_missing_pressure_file_leaves_that_resource_out(self) -> None: + patch, _ = self.report({"io": IO}) + data = patch["data"] + assert isinstance(data, dict) + self.assertEqual(sorted(key for key in json.loads(data["node-a"]) if key != "at"), ["io_full_avg10", "io_some_avg10"]) + + def test_the_configmap_name_and_namespace_are_configurable(self) -> None: + _, args = self.report({"io": IO}, NODE_PRESSURE_CONFIGMAP="pressure-x", NAMESPACE="other") + self.assertIn("patch configmap pressure-x -n other", args) + + def test_it_refuses_to_start_without_a_node_name(self) -> None: + with tempfile.TemporaryDirectory() as work: + env = {key: value for key, value in os.environ.items() if key != "NODE_NAME"} + result = subprocess.run(["bash", str(SCRIPT)], env={**env, "NAMESPACE": "x", "PRESSURE_DIR": work, "REPORT_ONCE": "true"}, capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn("NODE_NAME is required", result.stderr) + + +if __name__ == "__main__": + unittest.main()