feat(arc): measure the runner ceiling from each node's free capacity

A profile's sizing can now set measured: true in place of uniform
node figures. At install time the role reads every node the runner
pods may use (the eligibility label and the profile's node_selector,
and no deliberately added NoSchedule or NoExecute taint) and every
pod still holding requests. For each node it subtracts those requests
and a configurable reserve_cpu and reserve_memory_gib from the node's
allocatable CPU and memory, and counts how many runner pods of the
profile's size fit. The sum, scaled by the safety margin, becomes the
release's maxRunners, or the autoscaler's ceiling on the autoscaled
profile.

Pods in any scale set's namespace and finished pods are left out, so
running runners never lower the ceiling they are sized by. A node in
a passing condition, such as not ready or cordoned, still counts,
since the ceiling lasts until the next run. A profile's own
max_runners still wins, and uniform sizing is unchanged.

Author: Mearman

plugins/filter/arc.py

@@ -6,7 +6,7 @@ It runs in two ways. Inside `playbooks/site.yml`, against a k3s server host that
 
 ## Validation
 
-`tasks/validate.yml` checks every input that needs neither secrets nor a cluster: each org's fields, profile names (derived or overridden) that would collide or make invalid Kubernetes names, two of an org's profiles sharing a release name, a `max_runners` that is not a whole number, the controller's release name and namespace, the same org configured on two hosts, more than one autoscaled profile, values files that do not exist, the autoscaler's pool settings, sizing inputs, and the controller, probe and node label settings. Both playbooks run it in a play of its own before anything changes; the role runs it itself when included some other way. It also bootstraps the heartbeat gist when `github_runner_arc_heartbeat_bootstrap_gist` is on and no host in the inventory has one: it creates the gist with the control node's `gh` session and stops, printing the id to record.
+`tasks/validate.yml` checks every input that needs neither secrets nor a cluster (a measured sizing's node figures are read later, when the role installs the profile): each org's fields, profile names (derived or overridden) that would collide or make invalid Kubernetes names, two of an org's profiles sharing a release name, a `max_runners` that is not a whole number, the controller's release name and namespace, the same org configured on two hosts, more than one autoscaled profile, values files that do not exist, the autoscaler's pool settings, sizing inputs, and the controller, probe and node label settings. Both playbooks run it in a play of its own before anything changes; the role runs it itself when included some other way. It also bootstraps the heartbeat gist when `github_runner_arc_heartbeat_bootstrap_gist` is on and no host in the inventory has one: it creates the gist with the control node's `gh` session and stops, printing the id to record.
 
 Before touching the cluster the role then checks the secrets it is about to write (each org's App key looks like a PEM key, the pull credential and heartbeat token are set) and proves the pull credential can read every image it installs from `github_runner_arc_image_pull_registry`, by getting a pull-scoped registry token and fetching each image's manifest. With an App-sourced pull Secret (see below) it mints each org's token and proves it the same way, before installing anything.
 
@@ -21,7 +21,7 @@ Before touching the cluster the role then checks the secrets it is about to writ
   - `runs_on_label`: the extra `scaleSetLabels` entry the runners carry. Default `<org>-runners`, which pools every profile of the org under one `runs-on` label; set a different one to keep a profile out of that pool.
   - `scale_set_labels`: the whole `scaleSetLabels` list, in place of `runs_on_label`. An empty list sets no `scaleSetLabels`, so jobs target the scale set by its release name.
   - `autoscale`: `true` on at most one profile in the whole inventory makes it the scale set the autoscaler manages.
-  - `sizing`: derive a runner ceiling from node capacity (see Sizing). On an ordinary profile without `max_runners` it sets `maxRunners`; on the autoscaled profile it sets the autoscaler's ceiling, leaving `maxRunners` as the floor.
+  - `sizing`: derive a runner ceiling from node capacity, given as uniform figures or measured from each eligible node (see Sizing). On an ordinary profile without `max_runners` it sets `maxRunners`; on the autoscaled profile it sets the autoscaler's ceiling, leaving `maxRunners` as the floor.
 - `github_runner_arc_values_dir`: the control-node directory relative `values_file` paths are read from.
 - `github_runner_arc_kubeconfig_path`: the kubeconfig on the host the role runs against. Defaults to `github_runner_cluster`'s kubeconfig; empty means `KUBECONFIG` or `~/.kube/config`.
 - `github_runner_arc_controller_chart_version`, `github_runner_arc_scaleset_chart_version`: chart pins; unset installs the latest chart.
@@ -57,6 +57,22 @@ sizing:
   safety_margin: 0.8                # optional, default 1
 ```
 
+That assumes the nodes are alike. Where their free capacity differs a lot, `measured: true` works it out from the cluster instead, each time the role runs. It reads every node the runner pods may be placed on: every node carrying the eligibility label and the profile's `node_selector`, with no `NoSchedule` or `NoExecute` taint other than those Kubernetes adds for a passing condition. The ceiling lasts until the next run, so a node that is not ready, unreachable or cordoned when it is measured still counts, as it will once it is back. For each one it takes the node's allocatable CPU and memory, subtracts what the pods already on it request and the per-node reserve, and divides what is left by one runner pod's requests; the tighter of the two gives the runners that fit there. The sum over the nodes, scaled by the safety margin and rounded down, is the ceiling. Pods in any scale set's namespace are left out, so running runners never lower the ceiling they are sized by, as are pods that have finished. The reserve covers what requests do not show, such as a workload that uses more than it requests. The kubeconfig needs cluster-wide read access to nodes and pods. The role prints each node's figures when it runs.
+
+```yaml
+sizing:
+  measured: true
+  pod_cpu_request: 1
+  pod_memory_request_gib: 2
+  sidecar_cpu_request: 0.25         # optional, default 0
+  sidecar_memory_request_gib: 0.5   # optional, default 0
+  reserve_cpu: 0.5                  # optional, per node, default 0
+  reserve_memory_gib: 1             # optional, per node, default 0
+  safety_margin: 0.8                # optional, default 1
+```
+
+A profile's own `max_runners` still takes precedence, and on the autoscaled profile the measured figure is the autoscaler's ceiling. Measured sizing uses the `exadev.github_runner.arc_measured_max_runners` filter.
+
 The filter behind it is `exadev.github_runner.arc_max_runners`.
 
 ## Image pull Secret from the GitHub App

roles/github_runner_arc/README.md

@@ -1,7 +1,7 @@
 ---
 # ARC (Actions Runner Controller) in an existing Kubernetes cluster: the controller, each org's runner scale sets, and the fleet-health platform (heartbeat and autoscaler). Each part is gated on this host's own github_runner_arc_orgs or github_runner_arc_heartbeat_gist_id, so any host that can reach the cluster's API can be the one Ansible installs from: a k3s server host brought up by github_runner_cluster, or localhost with a kubeconfig (playbooks/arc.yml). On a host github_runner_cluster made an agent, this role installs nothing.
 
-# Orgs whose runner scale sets this host installs. Each entry: name, app_id, image, private_key (the App's PEM private key, from any source Ansible reads) or private_key_op_reference (an op:// reference the 1Password adapter resolves into private_key), optionally installation_id (resolved from the App when empty), optionally app_secret_name (the App Secret's name in each of the org's namespaces, default <org>-github-app), and scale_set_profiles, a list of {suffix, namespace?, release_name?, values_file?, max_runners?, node_selector?, runs_on_label?, scale_set_labels?, autoscale?, sizing?}. Each profile becomes one namespace (namespace, default arc-runners-<org><suffix>), two Secrets in it, and one Helm release (release_name, default <org>-runners<suffix>); setting both overrides lets the role take over scale sets an existing install created under its own names. See the README for every profile key.
+# Orgs whose runner scale sets this host installs. Each entry: name, app_id, image, private_key (the App's PEM private key, from any source Ansible reads) or private_key_op_reference (an op:// reference the 1Password adapter resolves into private_key), optionally installation_id (resolved from the App when empty), optionally app_secret_name (the App Secret's name in each of the org's namespaces, default <org>-github-app), and scale_set_profiles, a list of {suffix, namespace?, release_name?, values_file?, max_runners?, node_selector?, runs_on_label?, scale_set_labels?, autoscale?, sizing?}. sizing takes uniform node figures, or measured: true to read each eligible node's free capacity from the cluster at install time. Each profile becomes one namespace (namespace, default arc-runners-<org><suffix>), two Secrets in it, and one Helm release (release_name, default <org>-runners<suffix>); setting both overrides lets the role take over scale sets an existing install created under its own names. See the README for every profile key.
 github_runner_arc_orgs: []
 
 # Directory on the control node that a relative scale_set_profiles[].values_file is read from. Values files are Jinja templates rendered on the control node, so the target host needs no checkout. A profile with no values_file uses the role's own templates/runner-scale-set-values.yaml.j2.

roles/github_runner_arc/defaults/main.yml

@@ -13,6 +13,7 @@ argument_specs:
           - Each entry's scale_set_profiles is a non-empty list of profiles, each with suffix and optionally namespace (default arc-runners-<org><suffix>), release_name (default <org>-runners<suffix>), values_file, max_runners, min_runners, container_mode, resources, node_selector, runs_on_label, scale_set_labels, autoscale and sizing.
           - max_runners, when set, is the release's maxRunners and takes precedence over its values file and its sizing.
           - scale_set_labels replaces runs_on_label with the whole list of scaleSetLabels; an empty list sets none, so jobs target the release name.
+          - sizing derives a runner ceiling from uniform node figures, or with measured true from each eligible node's allocatable CPU and memory less its current requests and reserve_cpu and reserve_memory_gib.
       github_runner_arc_values_dir:
         type: str
         default: ""

roles/github_runner_arc/meta/argument_specs.yml

@@ -328,6 +328,22 @@
         if github_runner_arc_controller_namespace != 'actions-runner-controller' else []
       }}
 
+- name: "Platform: measure the autoscaled profile's ceiling from the eligible nodes"
+  ansible.builtin.include_tasks: measure_sizing.yml
+  vars:
+    github_runner_arc_measure: "{{ github_runner_arc_fleet.autoscaled }}"
+  when: github_runner_arc_fleet.autoscaled is not none and github_runner_arc_fleet.autoscaled.sizing is not none and github_runner_arc_fleet.autoscaled.sizing.measured
+
+- name: "Platform: fail when the autoscaler's floor is above its measured ceiling"
+  ansible.builtin.fail:
+    msg: >-
+      github_runner_arc_autoscaler_floor ({{ github_runner_arc_autoscaler_floor }}) is above the ceiling measured for
+      {{ github_runner_arc_fleet.autoscaled.label }} ({{ github_runner_arc_measured_sizing[github_runner_arc_fleet.autoscaled.label].max_runners }}).
+  when:
+    - github_runner_arc_fleet.autoscaled is not none
+    - github_runner_arc_fleet.autoscaled.label in (github_runner_arc_measured_sizing | default({}))
+    - (github_runner_arc_autoscaler_floor | int) > (github_runner_arc_measured_sizing[github_runner_arc_fleet.autoscaled.label].max_runners | int)
+
 - name: "Platform: install/upgrade the autoscaler Deployment"
   when: github_runner_arc_fleet.autoscaled is not none
   kubernetes.core.k8s:
@@ -360,7 +376,13 @@
                   - name: AUTOSCALER_USABLE_BUDGET_GI
                     value: "{{ github_runner_arc_autoscaler_usable_budget_gi | string }}"
                   - name: AUTOSCALER_MAX_CEILING
-                    value: "{{ (github_runner_arc_fleet.autoscaled.sizing.max_runners if github_runner_arc_fleet.autoscaled.sizing else github_runner_arc_autoscaler_max_ceiling) | string }}"
+                    value: >-
+                      {{
+                        (github_runner_arc_measured_sizing[github_runner_arc_fleet.autoscaled.label].max_runners
+                         if github_runner_arc_fleet.autoscaled.label in (github_runner_arc_measured_sizing | default({}))
+                         else github_runner_arc_fleet.autoscaled.sizing.max_runners if github_runner_arc_fleet.autoscaled.sizing
+                         else github_runner_arc_autoscaler_max_ceiling) | string
+                      }}
                   - name: AUTOSCALER_FLOOR
                     value: "{{ github_runner_arc_autoscaler_floor | string }}"
                   - name: AUTOSCALER_RAISE_CONFIRM_POLLS

roles/github_runner_arc/tasks/install_platform.yml

@@ -59,6 +59,13 @@
 - name: Install or remove the pull Secret renewal for {{ profile.label }}
   ansible.builtin.include_tasks: install_pull_secret_renewal.yml
 
+# Only when the profile sets no max_runners of its own, which takes precedence; the autoscaled profile's measured sizing is its autoscaler's ceiling, measured in install_platform.yml.
+- name: Measure the eligible nodes' free capacity for {{ profile.label }}
+  ansible.builtin.include_tasks: measure_sizing.yml
+  vars:
+    github_runner_arc_measure: "{{ profile }}"
+  when: profile.sizing is not none and profile.sizing.measured and profile.max_runners is none and not profile.autoscale
+
 - name: Install/upgrade the scale-set release for {{ profile.label }}
   # runnerScaleSetName is left to default to the release name: it is the scale set's registration identity on GitHub, so two releases sharing it make the second one's listener crash-loop registering a duplicate. scaleSetLabels is the pooling mechanism instead: an extra label every profile's runners carry alongside their own unique one, so runs-on: <label> matches every profile sharing it. A profile sets its own runs_on_label to stay out of the shared pool, or scale_set_labels to choose the whole list (empty sets none, leaving jobs to target the release name).
   kubernetes.core.helm:

roles/github_runner_arc/tasks/install_scale_set_profile.yml

@@ -0,0 +1,45 @@
+---
+# Included with `github_runner_arc_measure` set to an expanded profile whose sizing is measured (see plugins/filter/arc.py's arc_measured_max_runners): from install_scale_set_profile.yml for an ordinary profile, where the result becomes the release's maxRunners, and from install_platform.yml for the autoscaled one, where it becomes the autoscaler's ceiling. Reads every Node and every running Pod, so the kubeconfig needs cluster-wide read access to both; read-only, so it also runs under --check. The result is recorded under the profile's label in github_runner_arc_measured_sizing.
+
+- name: Read the nodes for {{ github_runner_arc_measure.label }}
+  kubernetes.core.k8s_info:
+    kind: Node
+  register: github_runner_arc_measure_nodes
+
+- name: Read the pods that still hold requests for {{ github_runner_arc_measure.label }}
+  kubernetes.core.k8s_info:
+    kind: Pod
+    field_selectors:
+      - status.phase!=Succeeded
+      - status.phase!=Failed
+  register: github_runner_arc_measure_pods
+
+# The same nodeSelector the runner pods get (scale-set-overlay.yaml.j2), and the pods of every scale set in the fleet left out, so that runners already running never lower the ceiling they are sized by.
+- name: Work out how many runners fit on the eligible nodes for {{ github_runner_arc_measure.label }}
+  ansible.builtin.set_fact:
+    github_runner_arc_measured_sizing: >-
+      {{
+        github_runner_arc_measured_sizing | default({})
+        | combine({github_runner_arc_measure.label: github_runner_arc_measure.settings.sizing
+            | exadev.github_runner.arc_measured_max_runners(
+                github_runner_arc_measure_nodes.resources,
+                github_runner_arc_measure_pods.resources,
+                ({github_runner_arc_node_label_key: github_runner_arc_node_label_value | string} if github_runner_arc_node_label_key | length > 0 else {})
+                | combine(github_runner_arc_measure.node_selector),
+                github_runner_arc_fleet.profiles | map(attribute='namespace') | list)})
+      }}
+
+- name: Fail when the runner ceiling cannot be measured for {{ github_runner_arc_measure.label }}
+  ansible.builtin.fail:
+    msg: "{{ github_runner_arc_measure.label }}: {{ github_runner_arc_measured_sizing[github_runner_arc_measure.label].errors | join('; ') }}"
+  when: github_runner_arc_measured_sizing[github_runner_arc_measure.label].errors | length > 0
+
+- name: Report the measured runner ceiling for {{ github_runner_arc_measure.label }}
+  ansible.builtin.debug:
+    msg:
+      max_runners: "{{ github_runner_arc_measured.max_runners }}"
+      fit_before_the_safety_margin: "{{ github_runner_arc_measured.theoretical }}"
+      eligible_nodes: "{{ github_runner_arc_measured.nodes }}"
+      skipped_nodes: "{{ github_runner_arc_measured.skipped_nodes }}"
+  vars:
+    github_runner_arc_measured: "{{ github_runner_arc_measured_sizing[github_runner_arc_measure.label] }}"

roles/github_runner_arc/tasks/measure_sizing.yml

@@ -72,14 +72,17 @@
       github_runner_arc_autoscaler_max_ceiling, all whole numbers, with the floor no higher than the ceiling.
   vars:
     github_runner_arc_ceiling: "{{ github_runner_arc_fleet.autoscaled.sizing.max_runners if github_runner_arc_fleet.autoscaled.sizing else (github_runner_arc_autoscaler_max_ceiling | default('')) }}"
+    # A measured ceiling is only known once the cluster is read, so install_platform.yml checks the floor against it then.
+    github_runner_arc_ceiling_measured: "{{ github_runner_arc_fleet.autoscaled.sizing.measured | default(false) if github_runner_arc_fleet.autoscaled.sizing else false }}"
   when:
     - github_runner_arc_fleet.autoscaled is not none
     - github_runner_arc_heartbeat_gist_id | length > 0
     - >-
       github_runner_arc_autoscaler_usable_budget_gi | default('') | string is not match('^[0-9]+$')
       or github_runner_arc_autoscaler_floor | default('') | string is not match('^[0-9]+$')
-      or github_runner_arc_ceiling | string is not match('^[0-9]+$')
-      or (github_runner_arc_autoscaler_floor | int) > (github_runner_arc_ceiling | int)
+      or (not github_runner_arc_ceiling_measured
+          and (github_runner_arc_ceiling | string is not match('^[0-9]+$')
+               or (github_runner_arc_autoscaler_floor | int) > (github_runner_arc_ceiling | int)))
 
 # The platform installs from whichever host sets github_runner_arc_heartbeat_gist_id, so the question is whether ANY host in the inventory sets one. scripts/heartbeat.sh checks HEARTBEAT_GIST_ID at container start, but that only shows up as a crash-looping pod, and with no gist id anywhere the platform is not installed at all. This creates the gist the same way the README's manual instructions do, then fails on purpose: its id has to be recorded by hand in a host's host_vars (and in any consumer named in github_runner_arc_heartbeat_gist_consumer), and carrying on silently would leave that undone.
 - name: Bootstrap the heartbeat gist if no host in the fleet has one yet