feat(arc): let the node recovery watcher delete a labelled node that stayed Unknown

README.md

@@ -180,10 +180,12 @@ When a node stops (its host is switched off or asleep, Docker quits, or the k3s
 
 `node-recovery` (an in-cluster Deployment in the platform namespace, see `roles/github_runner_arc/templates/node-recovery.yaml.j2` and `scripts/node-recovery.sh`) applies Kubernetes' [non-graceful node shutdown](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#non-graceful-node-shutdown): every `NODE_RECOVERY_POLL_SECONDS` (default 15s) it gives each node whose `Ready` condition has been `Unknown` for `github_runner_arc_node_recovery_after_seconds` (default 120s) the `node.kubernetes.io/out-of-service=nodeshutdown:NoExecute` taint. The control plane then evicts every pod on that node at once, unless it tolerates that taint, and the pod garbage collector force-deletes the ones already terminating there, so ARC starts the listener again on a healthy node. When the node reports `Ready` again the watcher removes the taint, and the node's kubelet stops the containers of the pods that were deleted while it was away. With the defaults a stopped node stops holding up job pickup within roughly four minutes: the grace period (50s on current Kubernetes), the threshold, one poll, the garbage collector's 20-second sweep, and the listener's own restart. `github_runner_arc_node_recovery_enabled: false` removes the watcher and any taint it applied.
 
-The watcher only acts on a node whose kubelet has gone silent (`Unknown`), never on one reporting `NotReady` (`False`), whose kubelet is alive and finishes its pods' deletion itself. It never taints the node it runs on, removes only taints it applied (recognised by its `github-runner.exadev/out-of-service-applied-at` annotation), and sends each change as a JSON patch that first tests the node's `resourceVersion`, so a node that came back since it was read is left for the next poll. RBAC is `get`, `list` and `patch` on nodes, with no `delete`: the watcher never removes a Node object, and the taint does not touch k3s's etcd membership, which only deleting a server's Node object changes. Its own pod tolerates an unreachable or not-ready node for only 10 seconds, so if it was running on the node that stopped, its ReplicaSet starts a replacement elsewhere well before the threshold passes.
+The watcher only acts on a node whose kubelet has gone silent (`Unknown`), never on one reporting `NotReady` (`False`), whose kubelet is alive and finishes its pods' deletion itself. It never taints the node it runs on, removes only taints it applied (recognised by its `github-runner.exadev/out-of-service-applied-at` annotation), and sends each change as a JSON patch that first tests the node's `resourceVersion`, so a node that came back since it was read is left for the next poll. RBAC is `get`, `list` and `patch` on nodes, with no `delete` unless pruning is switched on (below): the taint does not touch k3s's etcd membership, which only deleting a server's Node object changes. Its own pod tolerates an unreachable or not-ready node for only 10 seconds, so if it was running on the node that stopped, its ReplicaSet starts a replacement elsewhere well before the threshold passes.
 
 The threshold is also where a stopped node's runner pods are given up, earlier than the five-minute default eviction would. Their containers stopped with the node, so their jobs are lost either way unless the node returns within that time; lower the threshold for faster recovery, or raise it to wait longer for a node that is only asleep.
 
+Pruning is opt-in (`github_runner_arc_node_recovery_prune_after_seconds`, empty by default). When it is set, the watcher also deletes the Node object of a node that has been `Unknown` for that long and carries the `github-runner.exadev.io/removable=true` label (`github_runner_arc_node_recovery_prune_label`), so a device that is never coming back stops being listed. It never deletes an unlabelled node, a node that is a control-plane or etcd member (even if labelled), its own node or a node without a `Ready` condition, and it deletes nothing unless at least three `Ready` nodes carrying the etcd role label would remain, because k3s removes a server's etcd member when its Node is deleted. It sends the delete with the node's `uid` and `resourceVersion` as preconditions so a node that has come back since it was listed is left alone. Turning it on gives the watcher a cluster-wide `delete` on nodes. Its pod is then confined to control-plane nodes, since a token that can delete nodes should not be readable from a worker. A pruned node comes back only when its k3s agent process restarts: a running kubelet does not register itself again once its Node object is gone, so pruning is safe only for nodes that restart their agent when their Node vanishes or that reboot, which the node has to arrange. See the role README's Node recovery section for the details.
+
 Its image, `github_runner_arc_node_recovery_image`, is pulled without a pull Secret, so the published package must be public.
 
 ## Conventions

node-recovery/Dockerfile

@@ -1,7 +1,7 @@
-# Image for the node recovery Deployment (see roles/github_runner_arc/templates/node-recovery.yaml.j2 and scripts/node-recovery.sh): bash, jq and kubectl, all from Alpine's apk, so it is small and builds for either architecture. Build context is the repository root so it can COPY the script. The Deployment pulls it without a pull Secret, so the published package must be public.
+# Image for the node recovery Deployment (see roles/github_runner_arc/templates/node-recovery.yaml.j2 and scripts/node-recovery.sh): bash, curl, jq and kubectl, all from Alpine's apk (curl sends the prune pass's delete with its preconditions, which kubectl cannot), so it is small and builds for either architecture. Build context is the repository root so it can COPY the script. The Deployment pulls it without a pull Secret, so the published package must be public.
 FROM alpine:3.20
 
-RUN apk add --no-cache bash jq kubectl
+RUN apk add --no-cache bash curl jq kubectl
 
 COPY node-recovery/loop.sh /app/loop.sh
 COPY scripts/node-recovery.sh /app/scripts/node-recovery.sh

roles/github_runner_arc/README.md

@@ -50,7 +50,7 @@ Helm 4 and server-side apply. The role reads `helm version --short` and, on Helm
 - `github_runner_arc_image_pull_registry`, `github_runner_arc_image_pull_secret_name`, `github_runner_arc_verify_image_pull`: the pull Secret's registry and name, and whether to prove the credential before writing it.
 - `github_runner_arc_heartbeat_gist_id`: install the fleet-health platform from this host, refreshing this gist. `github_runner_arc_heartbeat_bootstrap_gist`, `github_runner_arc_heartbeat_gist_description` and `github_runner_arc_heartbeat_gist_consumer` control the bootstrap described above.
 - `github_runner_arc_autoscaler_usable_budget_gi`, `github_runner_arc_autoscaler_max_ceiling`, `github_runner_arc_autoscaler_floor`: required when a profile sets `autoscale: true` and the platform is installed. The floor must equal the autoscaled profile's `maxRunners`, which every Helm upgrade reverts to. The other `github_runner_arc_autoscaler_*` and `github_runner_arc_heartbeat_*` settings have defaults; see `defaults/main.yml`.
-- `github_runner_arc_node_recovery_enabled` (default `true`), `github_runner_arc_node_recovery_after_seconds` (default 120), `github_runner_arc_node_recovery_poll_seconds` (default 15), `github_runner_arc_node_recovery_image`: the node recovery watcher (see [Node recovery](#node-recovery)).
+- `github_runner_arc_node_recovery_enabled` (default `true`), `github_runner_arc_node_recovery_after_seconds` (default 120), `github_runner_arc_node_recovery_poll_seconds` (default 15), `github_runner_arc_node_recovery_prune_after_seconds` (default empty, which leaves pruning off), `github_runner_arc_node_recovery_prune_label` (default `github-runner.exadev.io/removable`), `github_runner_arc_node_recovery_image`: the node recovery watcher (see [Node recovery](#node-recovery)).
 - `github_runner_arc_runner_reaper_enabled` (default `false`), `github_runner_arc_runner_reaper_max_job_seconds` (default 432000, five days), `github_runner_arc_runner_reaper_completed_grace_seconds` (default 300), `github_runner_arc_runner_reaper_schedule` (default every 5 minutes), `github_runner_arc_runner_reaper_dry_run` (default `false`), `github_runner_arc_runner_reaper_image` (default the renewal's image): the runner reaper (see [Runner reaper](#runner-reaper)).
 - `github_runner_arc_runner_active_deadline_seconds` (default unset): `activeDeadlineSeconds` on every runner pod (see [Runner reaper](#runner-reaper)).
 - `github_runner_arc_app_setup_*`, `github_runner_arc_app_manifest_code`, `github_runner_arc_app_private_key_path`: inputs to `playbooks/github_app_setup.yml` (see below), including `github_runner_arc_app_setup_write_secret`, `github_runner_arc_app_setup_secret_namespaces`, `github_runner_arc_app_setup_secret_name`, `github_runner_arc_app_setup_keep_private_key` and `github_runner_arc_app_setup_command`.
@@ -380,7 +380,13 @@ github_runner_arc_orgs:
 
 ## Node recovery
 
-Every host that installs anything also installs a small watcher, the `node-recovery` Deployment in `github_runner_arc_platform_namespace`, so that a node that stops does not leave its pods `Terminating` for ever. ARC waits for a listener's old pod to go before starting a new one, so a listener on a stopped node otherwise stalls its scale set until the pod is force-deleted by hand. Once a node's `Ready` condition has been `Unknown` for `github_runner_arc_node_recovery_after_seconds`, the watcher gives it the `node.kubernetes.io/out-of-service=nodeshutdown:NoExecute` taint of Kubernetes' non-graceful node shutdown, which makes the control plane evict the node's pods and force-delete the terminating ones; it removes the taint when the node reports `Ready` again. It never acts on a node reporting `NotReady`, never on its own node, and never removes a taint it did not apply. Its ClusterRole allows `get`, `list` and `patch` on nodes and nothing else. `github_runner_arc_node_recovery_enabled: false` removes the watcher, its RBAC and any taint it left behind. Its image is pulled without a pull Secret, so it must be public. The repository README's Node recovery section explains the default threshold.
+Every host that installs anything also installs a small watcher, the `node-recovery` Deployment in `github_runner_arc_platform_namespace`, so that a node that stops does not leave its pods `Terminating` for ever. ARC waits for a listener's old pod to go before starting a new one, so a listener on a stopped node otherwise stalls its scale set until the pod is force-deleted by hand. Once a node's `Ready` condition has been `Unknown` for `github_runner_arc_node_recovery_after_seconds`, the watcher gives it the `node.kubernetes.io/out-of-service=nodeshutdown:NoExecute` taint of Kubernetes' non-graceful node shutdown, which makes the control plane evict the node's pods and force-delete the terminating ones; it removes the taint when the node reports `Ready` again. It never acts on a node reporting `NotReady`, never on its own node, and never removes a taint it did not apply. Its ClusterRole allows `get`, `list` and `patch` on nodes and nothing else, unless pruning is switched on (below). `github_runner_arc_node_recovery_enabled: false` removes the watcher, its RBAC and any taint it left behind. Its image is pulled without a pull Secret, so it must be public. The repository README's Node recovery section explains the default threshold.
+
+Pruning is off by default. Setting `github_runner_arc_node_recovery_prune_after_seconds` to a whole number of seconds, greater than `github_runner_arc_node_recovery_after_seconds`, makes the watcher delete the Node object of a node whose `Ready` condition has been `Unknown` that long and which carries the label `github_runner_arc_node_recovery_prune_label` with the value `true`. The variable has no default value, since how long a node may be away before it is given up is the fleet's own decision. Use it for nodes that will not come back for a long time, such as agents on small devices that are unplugged, so that they stop being listed. The label is the node's own to set (for a k3s agent, `--node-label` on its command line, which it applies again every time it registers); a node without it is never deleted, and neither is a node with a `node-role.kubernetes.io/control-plane` or `node-role.kubernetes.io/etcd` label even if it also carries the removable one, because deleting a k3s server's Node removes its etcd member; the pass fails saying so until the label is removed. The watcher's own node and a node with no `Ready` condition are never deleted either. The pass also deletes nothing unless at least three `Ready` nodes carrying the `node-role.kubernetes.io/etcd` label would remain, three being the voters a cluster needs to tolerate losing one (a non-voting agent does not count, nor does a member that is not `Ready`); below that it keeps the node, fails the pass and says why, so a cluster with no etcd members, such as a single-server SQLite one, is never pruned. The floor is a constant in the script, not a setting. It exists because k3s removes a server's etcd member when that server's Node is deleted (`pkg/etcd/member_controller.go` in k3s: `Removing etcd member from cluster due to node delete`, done only for a Node carrying the etcd role label), so a prune must never be the thing that thins the voters.
+
+Turning pruning on adds `delete` on nodes to the watcher's ClusterRole. RBAC cannot select nodes by label, so that is a cluster-wide delete on nodes; the script is what narrows it. Because the script's checks are client-side, anyone who can read the pod's ServiceAccount token could call the API directly, so with pruning on the Deployment is confined to control-plane nodes (a required node affinity on `node-role.kubernetes.io/control-plane`, and a toleration for that taint) and not left to land on a lower-trust worker. That needs a control-plane node that can run the pod; with pruning off the pod stays unpinned. The delete is a request to the API carrying the node's `uid` and `resourceVersion` as preconditions (`kubectl delete` cannot send them, so the watcher image also contains `curl`), so a node that came back between being listed and being deleted is left alone.
+
+A pruned node returns only when its k3s agent process restarts. A running k3s agent does not register itself again when its Node object disappears: its kubelet registers at start-up and then logs `node "..." not found` for as long as it runs, however long the Node has been missing and whether or not the network came back. After a reboot or a power cut the agent starts, registers a new Node with its labels and becomes `Ready` without any step by hand. A node whose agent keeps running while it is unreachable (a network outage, a suspended machine) and is pruned in the meantime stays out of the cluster until its agent is restarted. Pruning is therefore safe only for nodes that restart their k3s agent when their own Node is gone, or that reboot; arranging that is the node's responsibility, not this role's. Do not label a node you cannot restart.
 
 ## Runner reaper
 

roles/github_runner_arc/defaults/main.yml

@@ -175,6 +175,10 @@ github_runner_arc_node_recovery_enabled: true
 github_runner_arc_node_recovery_after_seconds: 120
 # How often the watcher checks the nodes. A taint lands at most this long after the threshold passes.
 github_runner_arc_node_recovery_poll_seconds: 15
+# Opt-in pruning of nodes that went away for good. Empty turns it off, which also leaves the watcher without permission to delete a Node. A whole number of seconds, greater than github_runner_arc_node_recovery_after_seconds, makes the watcher delete the Node object of a node that has been Unknown that long and carries github_runner_arc_node_recovery_prune_label with the value true, so a node that does not return, such as a Raspberry Pi agent that was unplugged, stops being listed. A k3s agent that does return registers itself again. There is no default delay, since how long a node may be away before it is given up is the fleet's own decision. Servers, etcd members and any node without the label are never deleted, and nothing is deleted unless at least three Ready etcd members would remain. This is a cluster-wide delete on nodes, granted to the watcher's ServiceAccount only while this is set.
+github_runner_arc_node_recovery_prune_after_seconds: ""
+# The label a node must carry, with the value true, to be pruned. Set it on the nodes that may go (for a k3s agent, --node-label on its command line, which it applies again each time it registers), never on a server.
+github_runner_arc_node_recovery_prune_label: github-runner.exadev.io/removable
 # Image the watcher runs. Pulled without a pull Secret, so it must be public.
 github_runner_arc_node_recovery_image: "ghcr.io/exadev/github-runner-node-recovery:2.1.0"
 

roles/github_runner_arc/meta/argument_specs.yml

@@ -345,6 +345,16 @@ argument_specs:
         type: int
         default: 15
         description: How often the node recovery watcher checks the nodes.
+      github_runner_arc_node_recovery_prune_after_seconds:
+        type: str
+        default: ""
+        description:
+          - Empty turns pruning off. A whole number of seconds, greater than github_runner_arc_node_recovery_after_seconds, makes the watcher delete the Node object of a node that has been Unknown that long and carries github_runner_arc_node_recovery_prune_label with the value true.
+          - Setting it also gives the watcher's ClusterRole the delete verb on nodes, a cluster-wide permission. Servers and etcd members are never deleted, and nothing is deleted unless at least three Ready etcd members would remain.
+      github_runner_arc_node_recovery_prune_label:
+        type: str
+        default: github-runner.exadev.io/removable
+        description: The label key a node must carry, with the value true, to be pruned. A Kubernetes qualified name.
       github_runner_arc_node_recovery_image:
         type: str
         description: Image the node recovery watcher runs, pulled without a pull Secret.

roles/github_runner_arc/tasks/validate.yml

@@ -112,6 +112,21 @@
           and ([github_runner_arc_node_recovery_after_seconds, github_runner_arc_node_recovery_poll_seconds]
                | map('string') | reject('match', '^[1-9][0-9]*$') | list | length > 0)
         }}
+    - msg: github_runner_arc_node_recovery_prune_after_seconds must be empty or a whole number of seconds greater than github_runner_arc_node_recovery_after_seconds, so a node is tainted before it is deleted.
+      failed: >-
+        {{
+          github_runner_arc_node_recovery_enabled | bool
+          and github_runner_arc_node_recovery_prune_after_seconds | string | length > 0
+          and (github_runner_arc_node_recovery_prune_after_seconds | string is not match('^[1-9][0-9]*$')
+               or github_runner_arc_node_recovery_prune_after_seconds | int <= github_runner_arc_node_recovery_after_seconds | int)
+        }}
+    - msg: github_runner_arc_node_recovery_prune_label must be a Kubernetes label key, such as example.com/removable.
+      failed: >-
+        {{
+          github_runner_arc_node_recovery_enabled | bool
+          and github_runner_arc_node_recovery_prune_after_seconds | string | length > 0
+          and github_runner_arc_node_recovery_prune_label | string is not match('^([a-z0-9]([-a-z0-9.]{0,251}[a-z0-9])?/)?[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?$')
+        }}
     - msg: github_runner_arc_runner_reaper_max_job_seconds must be a whole number of seconds, at least 1, and github_runner_arc_runner_reaper_completed_grace_seconds a whole number of seconds.
       failed: >-
         {{