Repository navigation
feat(cluster): step a server down and back up under memory pressure without losing its etcd vote - #114
Merged
Conversation
The core steps a k3s server down through levels of reduced function under memory pressure, serialised across servers by a Lease and gated on the other etcd voters being Ready. It calls only the three functions of an actuator (levels, current_level, apply_level), so a Docker node and a NixOS node can drive the same decision logic. The files are the fleet image's core and tests unchanged; a .shellcheckrc names bash for them, since they carry no shebang.
With STEP_UP=automatic the watcher goes back up one level at a time once memory has been calm for the sustain period and the cooldown since the last step, in either direction, has passed. Calm is stricter than not being over the limit (pressure under the threshold and MemAvailable at or above the floor, both), so a reading between the two ends both streaks and a server does not step both ways on the same noise. A step up takes the same Lease and peer check as a step down, since it restarts etcd at the first level that leaves etcd-only. STEP_UP defaults to manual, which is what the core did before the setting existed, so a consumer that sets nothing is unchanged.
The Docker actuator records a level in a file on the k3s data volume and restarts the container when the level changes between etcd-only and another; full and control-plane run the same k3s and differ only in the cordon and taint the core puts on the Node. The container's entrypoint reads the file at every start and starts k3s with its API server, controller manager, scheduler and agent switched off at etcd-only, which leaves the server's etcd member, and its vote, in place. It records the level k3s started at on the container's tmpfs, since k3s hides its arguments from /proc, and that is what the actuator reports as the node's level. A restart returns only when etcd serves again and, for a level that runs the agent, the node is Ready on a heartbeat from the new container, because the core releases its Lease as soon as a step returns. The Node carries a label naming the level from the first step down until it is back at the first level. The regrade Compose service runs the core and the actuator beside the k3s container, in a profile of its own, with the Docker socket and its scripts mounted.
…essure The core decides on /proc/pressure/memory. A kernel built without CONFIG_PSI, or booted with psi=0, would make it fail on every sample, so watch and status now stop at once and say what is missing. Stepping by hand never reads the file and is not held up by it.
github_runner_cluster_regrade, off by default, runs the regrade service beside a server's k3s container and templates the settings it reads; its times are multiples of the kernel's longest pressure average and of the sample period, and its step up is automatic unless set to manual. The role refuses regrading on an agent, on a server with no other server in the group, and on the bootstrap server of an in-cluster Headscale mesh, whose Headscale server runs on the kubelet that etcd-only switches off. It also refuses to remove the service while a server is below the first level, for every host at once. The final wait for every node to be Ready, and the node-password self-healing that follows a failed one, leave out a Node carrying the level label: a server at etcd-only has no kubelet and is not Ready by design, and clearing its node-password Secret would be a change nobody asked for. The collection build check now asserts the service's files ship.
…rom the alert A server that memory-pressure regrading has stepped to etcd-only is not Ready by design, because it runs no kubelet. The heartbeat now publishes the level its Node's label names beside each not-Ready node, and fleet-alert.sh no longer announces a fleet outage for a server that carries one. A heartbeat image that predates the field leaves it out and alerts as before.
The regrade scenario turns regrading on for three real k3s servers behind Headscale and drives the watcher in the real service container with fixture pressure files. It checks that a held Lease refuses a step, that pressure steps a server down through control-plane to etcd-only with no other server below full meanwhile, that the server keeps its etcd member ids, its Node with the level label and its node password, and that the cluster takes writes with the third server stopped. It then checks that another server's step is refused while one is stepped down, that a role run changes nothing and turning regrading off is refused for every host, that calm steps the server back up by itself, that a server can be stepped by hand, and that turning regrading off with every server at full removes the services and restarts no node.
…m the k3s context
… reports as unreachable
Mearman
marked this pull request as ready for review
October 20, 2026 21:12
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
Member
Author
|
@codex security review |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.6.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds memory-pressure regrading for k3s servers to the cluster role: under sustained pressure a server steps down through
full,control-planeandetcd-only, keeping its etcd member and its vote the whole way, and steps back up by itself once memory has been calm. It is off by default (github_runner_cluster_regrade) and changes nothing on a running node until it is turned on. Fixes #112.The open question in the issue is answered in its body: k3s accepts flipping an existing server between
fullandetcd-onlyby changing its flags and restarting, bootstrap server included, with the same etcd member, the same Node and the same node password afterwards. A stepped-down server still counts for quorum (the cluster took writes with it as one of two live voters and refused them with it stopped as well).The decision logic is
files/regrade/fleet-regrade-core.sh, which calls onlylevels,current_levelandapply_level. The first commit is the fleet image's core and tests unchanged, so a diff against them is meaningful; a.shellcheckrcnames bash for them since they carry no shebang. The second addsSTEP_UP=automatic, because the issue asks for stepping up by itself and the fleet image's core only steps up by hand. It defaults tomanual, so a consumer that sets nothing, the NixOS image, behaves as before, and the role defaults it toautomatic. A server counts as calm when pressure is under the threshold and MemAvailable is at or above the floor, both, which is stricter than not being over, so a reading between the two ends both streaks. Stepping up shares the cooldown and the Lease with stepping down and restarts etcd at the first level that leaves etcd-only, so it never starts while a peer is down. A server that cannot hold a higher level cycles no faster than once per cooldown and sustain period; the NixOS module would need to passSTEP_UPto opt in.The Docker actuator does not recreate the container through Compose, which is what the issue sketches. It records the level in a file on the k3s data volume that the node entrypoint reads at every start, and restarts the container. Recreating would change the Compose configuration per level, so the next Ansible run would template the full configuration and put the server back to full under pressure, and every step would be a recreate that needs the node password volume from #108. A restart keeps the container, so neither applies: nothing the container is created from changes, the serialised server start of #108 is not involved, and a level survives the host restarting. k3s hides its flags from
/proc, so the entrypoint also records the level it started k3s at on the container's tmpfs, and that is whatcurrent_levelreads. A running container with neither record is an older entrypoint and counts asfull, so enabling regrading on a running fleet needs no restart until the first step to etcd-only.A restart returns only when etcd serves again (read from the container's log since it started) and, for a level that runs the agent, the node is Ready on a heartbeat from the new container, because the core releases its Lease as soon as
apply_levelreturns. Every call to the cluster goes to another server's API: on an etcd-only server the container's own kubeconfig answeredapiserver disabledon one server of the test cluster.A server at
etcd-onlyis NotReady by design, which touches three consumers. The Node carriesgithub-runner.exadev.com/regrade-levelwhile it is belowfull; the role's final wait for every node to be Ready and its node-password self-healing leave such a node out, the heartbeat publishesregrade_levelbeside each not-Ready node, andfleet-alert.shdoes not count a stepped-down server as an outage (a heartbeat image without the field alerts as before). Other servers' peer check also refuses to step while one is stepped down, which is the safe direction.The
regradeservice mounts the Docker socket, the same permission cost asdocker-prune, for the same reason: it acts on the host's other containers. It has a Compose profile of its own, so a host that does not enable regrading never builds or starts it, and its scripts are mounted read-only from the project directory rather than built into the image. The role refuses regrading on an agent, with no other server in the group (an etcd-only server needs another server's API), and on the bootstrap server of aheadscale_in_clustermesh (its Headscale server is a static pod on the kubelet etcd-only switches off); it also refuses to remove the service while a server is belowfull.Tests:
tests/regrade/regrade-recover.sh: the step-up decision at worked-out moments, manual mode never stepping, the cooldown and a fresh calm streak between steps, and the peer and Lease checks reaching a step up.tests/regrade/regrade-docker.sh: the actuator against a stand-indocker, including that etcd's line from before a restart does not count and that a node that never renews its lease fails the step.tests/unit/test_cluster_regrade.py: the role's checks, the settings file as the core sources it, the Compose service, the entrypoint's flags at each level, and the tasks that remove the service.regrademesh scenario: three real k3s servers behind Headscale, the watcher driven by fixture pressure files inside the real service container. It checks a held Lease refuses a step, a step down throughcontrol-planetoetcd-onlywith no other server belowfullmeanwhile, the same etcd member ids, label and node password, writes with the third server stopped, a refused step on another server, a role rerun with a server stepped down changing nothing, the step back up by itself, a step by hand, and the service being removed once every server is at full.Mutations of the code under test were run against each group above (for example dropping
--sincefrom the log read, the--disable-agentflag, the cooldown check and the heartbeat's label read) and each fails the tests.Not verified: real memory pressure on a small host (the threshold and floor defaults are reasoned starting points, and the scenario drives the watcher with fixture files), a Docker host whose kernel exposes no PSI (covered by a unit test of the service refusing to start, not by a real kernel), and macOS Docker Desktop's VM. The scenario was run on a Linux aarch64 Colima VM; CI runs it on x86_64.