feat(cluster): step a server down and back up under memory pressure without losing its etcd vote

.github/workflows/ci.yml

@@ -111,6 +111,9 @@ jobs:
           ansible-galaxy collection build --output-path "${RUNNER_TEMP}/collection"
           tar -tzf "${RUNNER_TEMP}"/collection/*.tar.gz | grep -Fx roles/github_runner_cluster/files/docker-compose.yml
           tar -tzf "${RUNNER_TEMP}"/collection/*.tar.gz | grep -Fx roles/github_runner_cluster/files/k3s/node-entrypoint.sh
+          for file in Dockerfile fleet-regrade.sh fleet-regrade-core.sh fleet-regrade-docker.sh; do
+            tar -tzf "${RUNNER_TEMP}"/collection/*.tar.gz | grep -Fx "roles/github_runner_cluster/files/regrade/${file}"
+          done
 
   unit-tests:
     name: Unit tests (shard ${{ matrix.shard }})

.github/workflows/mesh-integration.yml

@@ -1,6 +1,6 @@
 name: Mesh integration
 
-# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. Only on changes that can affect how nodes join the mesh.
+# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. Only on changes that can affect how nodes join the mesh.
 #
 # Runs on GitHub-hosted ubuntu-latest, whose Docker daemon runs the privileged containers the test needs (k3s, and Tailscale inside it). The work directory sits under runner.temp, which the job always has write access to.
 
@@ -37,6 +37,7 @@ jobs:
           - {scenario: existing, compose: v5.5.1}
           - {scenario: hosted, compose: v5.5.1}
           - {scenario: in_cluster, compose: v5.5.1}
+          - {scenario: regrade, compose: v5.5.1}
           - {scenario: hosted, compose: v2.39.0}
           - {scenario: compose_faulty, compose: v2.38.2}
           - {scenario: in_cluster, compose: v5.5.1, pin: true}

.shellcheckrc

@@ -0,0 +1,2 @@
+# The scripts shared with other repositories are sourced or run as bash without a shebang line.
+shell=bash

README.md

@@ -137,9 +137,9 @@ Confirm it refreshes: `gh gist view "$HEARTBEAT_GIST_ID"` must show a timestamp
 
 #### Fleet-down notification
 
-`runner-fallback-action` reroutes silently, so nothing otherwise tells a person that the fleet has no runners. `fleet-health.json` carries what a notifier needs to judge how long that has lasted: `at` (when the heartbeat last wrote), `unhealthy_since` (when it stopped vouching for the fleet, `null` while it does) and `not_ready_nodes` (each node whose Ready condition is not True, whether it is a control-plane server, and since when). The heartbeat carries each onset over from the document its previous tick published, read back from the gist, so a tick that cannot read it restarts the onsets at now, which delays an alert and never raises a false one.
+`runner-fallback-action` reroutes silently, so nothing otherwise tells a person that the fleet has no runners. `fleet-health.json` carries what a notifier needs to judge how long that has lasted: `at` (when the heartbeat last wrote), `unhealthy_since` (when it stopped vouching for the fleet, `null` while it does) and `not_ready_nodes` (each node whose Ready condition is not True, whether it is a control-plane server, the `regrade_level` memory-pressure regrading has stepped it to if it has, and since when). The heartbeat carries each onset over from the document its previous tick published, read back from the gist, so a tick that cannot read it restarts the onsets at now, which delays an alert and never raises a false one.
 
-`scripts/fleet-alert.sh` reads that document and announces an outage when any of these has held for at least `FLEET_ALERT_AFTER_SECONDS` (default 900, longer than the heartbeat's own 600 second window): the heartbeat has not written for that long, a control-plane server has not been Ready for that long, or the heartbeat has not vouched for the fleet for that long (a scale set missing or without a running listener, no ready controller, no Ready node, runner pods that cannot be scheduled; the reasons are in the notification). A worker node that is not Ready is not an outage by itself. An outage is announced once, and a recovery note follows when every condition has cleared. A heartbeat image older than the onset fields makes the script fail with a message saying so, rather than never alerting.
+`scripts/fleet-alert.sh` reads that document and announces an outage when any of these has held for at least `FLEET_ALERT_AFTER_SECONDS` (default 900, longer than the heartbeat's own 600 second window): the heartbeat has not written for that long, a control-plane server has not been Ready for that long (except one regrading has stepped down, which is not Ready on purpose), or the heartbeat has not vouched for the fleet for that long (a scale set missing or without a running listener, no ready controller, no Ready node, runner pods that cannot be scheduled; the reasons are in the notification). A worker node that is not Ready is not an outage by itself. An outage is announced once, and a recovery note follows when every condition has cleared. A heartbeat image older than the onset fields makes the script fail with a message saying so, rather than never alerting.
 
 Delivery is independent per channel, and with none configured the script only prints what it found:
 
@@ -221,6 +221,7 @@ Commits follow Conventional Commits (`fix:`, `feat:`, `tune:`, and so on).
 - **A global `--etcd-arg` override is not scoped to the "real" server's own etcd instance - it also applies to k3s's separate, internal "temporary etcd", spun up on every *restart* (never on first boot) to reconcile with an already-initialized on-disk datastore before the real instance starts.** An earlier `--etcd-arg="listen-peer-urls=https://0.0.0.0:2380"` (kept as a "harmless" belt-and-braces bind override after fixing the peer-URL issue above) forced that temporary instance onto the same TLS `0.0.0.0:2380` bind it has no certificates for by design (it normally listens on a local, ephemeral, non-TLS port). Confirmed live: this only broke on a restart against an already-initialized datastore, never on a fresh `--cluster-init`, which is exactly why it wasn't caught until well after the first working deploy - fatal: `"cannot listen on TLS for [::]:2380: KeyFile and CertFile are not presented"`. `--node-ip` alone (above) is sufficient; etcd's own default `listen-peer-urls` already derives from the same `PrivateIP` correctly, so there's nothing left for an explicit override to add.
 - **Calling `tailscale up --authkey=...` unconditionally on every container restart re-registers a brand new tailnet device each time, instead of reconnecting the persisted one - even with the join key's own state volume intact.** A reusable Tailscale auth key is designed to authorize provisioning *new* devices, and a naive `tailscale up` call on every restart (needed here to learn this node's own Tailscale IP before `exec`ing k3s - see above) doesn't check whether the daemon is already authenticated first, so it keeps minting new devices under names like `node1`, `node1-1`, `node1-2`... k3s's own `--vpn-auth` integration avoids exactly this by checking the Tailscale backend state first and skipping its own `tailscale up` call entirely once already `"Running"` (confirmed against `pkg/vpn/vpn.go`'s `StartVPN()`) - the manual pre-empt call needs the identical guard: check `tailscale ip -4` succeeds *before* calling `tailscale up` at all, only calling it when not already authenticated.
 - **Tailscale assigns an IP and wires up its own MagicDNS resolver as two separate steps, not atomically - a node having a real IP is not proof DNS resolution against another tailnet member is working yet.** A joining server needs to resolve the bootstrap host's own MagicDNS hostname (`K3S_JOIN_SERVER_URL`) before it can bootstrap against it at all. Confirmed live: k3s failed outright with `"dial tcp: lookup k3s-server.example.ts.net: no such host"` immediately after `tailscale ip -4` had already returned a real address for the *local* node. Fix: wait for a real `nslookup` success against the join server's own hostname specifically, not just local Tailscale readiness, before invoking k3s (the `k3s` service's entrypoint only does this when `K3S_JOIN_SERVER_URL` is actually set - the bootstrap host has nothing else to resolve).
+- **A server that memory-pressure regrading has stepped to `etcd-only` is `NotReady` by design, and must stay registered.** With `github_runner_cluster_regrade` on, a server under sustained memory pressure steps down (`full`, `control-plane`, `etcd-only`) while it keeps its etcd member and vote; at `etcd-only` k3s runs with `--disable-apiserver --disable-controller-manager --disable-scheduler --disable-agent`, so there is no kubelet to keep its Node Ready, and the Node goes `NotReady` with `Unknown` conditions and stays so. That is not a failure to clear. Its Node object must not be deleted (k3s removes that server's etcd member with it), so anything that culls long-`NotReady` nodes must leave a Node carrying the `github-runner.exadev.com/regrade-level` label alone, and the role's wait for every node to be Ready already does. `kubectl` on such a server's own container is unreliable (its API server is off); the regrade service always reaches the cluster through another server. See the cluster role's README, Memory-pressure regrading. Confirmed against a three-server k3s cluster: the same etcd member ids before, during and after, the cluster taking writes with the stepped-down server as one of the two live voters, and the same Node and node password after stepping back up.
 - **A single remaining healthy etcd server can get stuck indefinitely if another member is registered but permanently unreachable (e.g. a partial join that never completed), even though that member is a non-voting learner and shouldn't block quorum.** Symptom: `"failed to publish local member to cluster through raft"` with `context deadline exceeded`, repeating forever, alongside `kubectl` returning `503 the server is currently unable to handle the request`. The fix is k3s's own documented recovery mechanism, run once against the affected node: stop the container, then `k3s server --cluster-reset` (with the SAME flags the real service uses, including `--node-ip` set to the node's own persisted Tailscale IP - getting this wrong re-registers the single remaining member under the *wrong* address, e.g. the invoking container's own ad-hoc bridge IP, and the real service then fails to start with `"this server is a not a member of the etcd cluster"`) - this resets *etcd cluster membership* back to a single member using the *existing local data*, it does not wipe anything. It's a one-shot operation: k3s writes a guard file after a successful reset and refuses to run `--cluster-reset` again until it's manually deleted (`rm /var/lib/rancher/k3s/server/db/reset-flag`) if a second reset is genuinely needed. After a reset, restart the service normally (without the flag) and clean up the stale member's own now-orphaned Kubernetes `Node` object and `node-password` Secret before it rejoins fresh.
 
 ## References

galaxy.yml

@@ -29,6 +29,7 @@ build_ignore:
   - .env*
   - .gitignore
   - .yamllint
+  - .shellcheckrc
   - AGENTS.md
   - CLAUDE.md
   - .ansible