Step a server down and back up under pressure without losing its etcd vote #112

A k3s server on a laptop or a small board can run short of memory or lose its power budget while it is still holding an etcd vote. Today the only choices are to leave it running full and risk the node, or to stop it and lose the vote.

I want a server to step down while keeping its etcd seat, and step back up by itself when the pressure passes. Three levels: full (control plane, etcd and workloads), control-plane (still a server and voter, cordoned and drained) and etcd-only (the API server, scheduler, controller manager and agent disabled). Giving up the vote is never automatic: removing a voting member on a partition is unsafe, and k3s removes the etcd member when its Node is deleted.

The decision logic is platform-neutral: memory pressure plus a MemAvailable floor, a cooldown longer than the pressure window, and one step at a time. Applying a level is not: a NixOS node switches a specialisation in place, a Docker node recreates the k3s container with a different flag set. So the collection would ship the core and a Docker actuator, and a Nix image can consume the same core by pinned release. The core calls only levels, current_level and apply_level <name>.

Transitions are serialised cluster-wide with a Lease in kube-system, taken only while the other voting Nodes are Ready, so two servers never step at once.

Open question, to be answered in a throwaway cluster before the Docker actuator is built: whether k3s accepts flipping an existing server between full and etcd-only by changing its flags and restarting.

Done when a server under sustained memory pressure steps down a level without losing quorum, steps back up after the pressure has gone, and a second server never transitions during the first one's step.

Answer to the open question: flipping works

Tested on a throwaway three-server k3s cluster (v1.34.1+k3s1, embedded etcd, three plain containers on one Docker network, each with named volumes for /var/lib/rancher/k3s and /etc/rancher/node). A server was flipped by stopping its container and creating it again with the same volumes and the extra flags --disable-apiserver --disable-controller-manager --disable-scheduler --disable-agent, and flipped back by creating it again without them. Done to a joined server and to the one that bootstrapped with --cluster-init, which keeps that flag on the etcd-only start.

• etcd membership is untouched. The member list had the same three IDs and peer URLs before, during and after, and endpoint health --cluster was healthy for all three while one server was etcd-only.
• It still votes. With one full server stopped, the remaining full server and the etcd-only one kept quorum and accepted writes. With the etcd-only one stopped as well, writes failed.
• Data is intact. ConfigMaps written before the flip were there after it, and writes made while a server was etcd-only (through another server) were there after step-up.
• The Node object stays. The etcd-only server has no kubelet, so its Node goes NotReady (Unknown conditions, unreachable taints) after the node controller's grace period and stays there for as long as the server is etcd-only, retaining its etcd role label and its EtcdIsVoter condition. It was not deleted over the minutes it was left, which matters because deleting it removes the etcd member. k3s's docs say an agentless server registers no Node; a server that was already registered keeps the old one.
• Step-up restores the same identity. The node registered again as the same Node (same creation time, same etcd member), became Ready within a few seconds of the container starting, and kept its cordon and taints until they were removed by hand, as for any restart.
• The node password is a separate concern. With the /etc/rancher/node volume the password file hash was identical before and after every flip. Creating the container again without that volume gave a new password file and a NodePasswordValidationFailed event ("hash does not match") against the Secret in kube-system, although the server still reported Ready on this version. So a recreate-based actuator needs the password volume from feat(cluster): keep a k3s node's password across recreation and recreate servers one at a time #108.
• A restarted etcd-only container keeps its flags. docker stop and docker start of an etcd-only container brought it back etcd-only, since the flags belong to the container's configuration. A level held only in the container's configuration therefore also has to be changed by creating the container again, or kept somewhere the entrypoint reads at every start.
• k3s server hides its flags. docker top and /proc/<pid>/cmdline of the running server show k3s server with none of them, so the level a node runs at cannot be read from its process arguments.
• kubectl on the etcd-only node itself is not reliable. Its kubeconfig points at its own supervisor port, which answered the API through the agent load balancer on one server and returned apiserver disabled on another, whose load balancer had listed its own address as the default for as long as it was observed. kubectl --server https://<another server>:6443 from inside the etcd-only container worked every time, for reads, writes and uncordon. The core's KUBE_SERVER setting is therefore needed for the Docker actuator too.
• Memory. An etcd-only server's container used a fraction of a full server's on an idle cluster (observation: about 100 to 130 MiB against roughly 400 MiB); with the agent left enabled (the three control-plane flags only) kubelet and containerd keep running and the Node stays Ready, at about 220 MiB.
• What it does to the rest. Other servers' peers_ready check sees the etcd-only server's Node as not Ready, so no second server can step down while one is etcd-only. That is the safe direction, and it also means the heartbeat reports a not-Ready server for as long as one is stepped down.

Delivered

The core, the Docker actuator and the role are in #114, with a mesh scenario that steps a server down and back up in CI; #125 adds the restart and recreate checks. The core also steps up by itself when STEP_UP=automatic (default manual, so an existing consumer is unchanged). The NixOS image still has its own copy of the core; it follows this collection's release as a follow-up, by taking the same core and passing STEP_UP=automatic from its module.