Skip to content

Commit 85171e6

Browse files
committed
feat(cluster): register a node with taints set per host
A taint put on a node with kubectl is lost whenever the node object is deleted and registers again, which the cluster role does for an agent that fails to rejoin. github_runner_cluster_node_taints passes key=value:Effect entries to k3s as --node-taint for servers and agents, so a host that must stay free of ordinary workloads keeps that across re-registration.
1 parent 519b94e commit 85171e6

6 files changed

Lines changed: 23 additions & 4 deletions

File tree

‎roles/github_runner_cluster/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ Each site is its own cluster: its own inventory group, bootstrap server and mesh
9696
- `github_runner_cluster_tailnet_domain`: the mesh's MagicDNS domain. Each node's address is `<node name>.<domain>`, and join URLs and TLS SANs use it.
9797
- `github_runner_cluster_headscale_*`: the Headscale providers' settings, documented in `defaults/main.yml`.
9898
- `github_runner_cluster_node_name`: the k3s node name and mesh hostname, defaulting to the inventory name.
99+
- `github_runner_cluster_node_taints`: Kubernetes taints (`key=value:Effect`, no spaces) the host's node registers with, empty by default. `NoSchedule` keeps pods without a matching toleration off it and `NoExecute` also evicts the ones already there. k3s applies them on every registration, so they come back if the node object is deleted, which a taint set with `kubectl` does not.
99100
- Per-host overrides: `github_runner_cluster_node_role` (`server` or `agent`), `github_runner_cluster_bootstrap` (true picks the bootstrap host, false excludes a host), `github_runner_cluster_server_url` (join this server instead, for a cluster whose servers are outside the inventory), `github_runner_cluster_tls_sans`, `github_runner_cluster_node_address`. The run fails if the result has an even number of servers, no servers, or more than one bootstrap host.
100101
- `github_runner_cluster_docker_context` (default empty): pins the Docker context this host's node runs in (see [Docker contexts](#docker-contexts)).
101102
- `github_runner_cluster_compose_files`: extra Compose files for the host's k3s project, merged after `docker-compose.yml`, as paths relative to `github_runner_cluster_dir` or absolute. Several nodes on one Docker host each need their own `github_runner_cluster_dir` and `github_runner_cluster_compose_project`.

‎roles/github_runner_cluster/defaults/main.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ github_runner_cluster_node_name: ""
1818
# Derived, never set directly: the name every template and check actually uses for this host (k3s node name, Tailscale device hostname, TLS SAN self-check).
1919
github_runner_cluster_effective_node_name: "{{ github_runner_cluster_node_name if github_runner_cluster_node_name | length > 0 else inventory_hostname }}"
2020

21+
# Kubernetes taints this host's node registers with, as key=value:Effect strings (for example github-runner.example.com/excluded=true:NoExecute), none containing a space. A taint keeps pods without a matching toleration off the node; NoExecute also evicts the ones already there. They are applied by k3s each time the node registers, so they survive the node object being deleted and registering again, which a taint put on the node by hand does not. Per host, in its host_vars.
22+
github_runner_cluster_node_taints: []
23+
2124
github_runner_cluster_k3s_version: "latest"
2225

2326
# Secrets, as plain variables from any source Ansible reads (ansible-vault, a lookup, or a secrets adapter role such as github_runner_secrets_onepassword). github_runner_cluster_k3s_token is the shared node-join secret and has no default. github_runner_cluster_tailscale_join_key is, with the default tailscale mesh provider, what k3s's own --vpn-auth integration (see docker-compose.yml) uses on every host: it runs Tailscale INSIDE each node's own k3s container, joining the tailnet as its own device and using flannel's native "tailscale" backend for pod traffic. github_runner_cluster_tailscale_api_token, when set, lets tasks/mesh/tailscale_api.yml keep the tailnet's ACL grant and the join key itself provisioned; see that file's comments for what it manages and the one manual step (creating the API token) that has no API to automate.

‎roles/github_runner_cluster/files/docker-compose.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ services:
4949
- K3S_MESH_SELF_HOSTED_HEALTH_URL=${K3S_MESH_SELF_HOSTED_HEALTH_URL:-}
5050
# Empty (not :?required) on the bootstrap host - the one server with nothing to join. Every OTHER server host's own .env sets this to the bootstrap host's own URL.
5151
- K3S_JOIN_SERVER_URL=${K3S_JOIN_SERVER_URL:-}
52+
- K3S_NODE_TAINTS=${K3S_NODE_TAINTS:-}
5253
volumes:
5354
- k3s-data:/var/lib/rancher/k3s
5455
# Persists this container's own Tailscale device identity/keys across recreations - without this, every recreate would register a brand new device on the tailnet instead of reconnecting as the same one, since the join key is reusable by design.
@@ -87,6 +88,7 @@ services:
8788
# Not :?required, for the same reason as K3S_AGENT_SERVER_URL above: the self-hosted bootstrap server's first .env has no key yet, and Compose would reject it for this service too. k3s/node-entrypoint.sh fails fast without one instead.
8889
- K3S_VPN_AUTH_JOIN_KEY=${K3S_VPN_AUTH_JOIN_KEY:-}
8990
- K3S_VPN_AUTH_CONTROL_SERVER_URL=${K3S_VPN_AUTH_CONTROL_SERVER_URL:-}
91+
- K3S_NODE_TAINTS=${K3S_NODE_TAINTS:-}
9092
volumes:
9193
- k3s-agent-data:/var/lib/rancher/k3s
9294
# Same reasoning as the k3s service's own tailscale volume.

‎roles/github_runner_cluster/files/k3s/node-entrypoint.sh‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@ join_key="${K3S_VPN_AUTH_JOIN_KEY:-}"
1010
control_server_url="${K3S_VPN_AUTH_CONTROL_SERVER_URL:-}"
1111
self_hosted_health_url="${K3S_MESH_SELF_HOSTED_HEALTH_URL:-}"
1212

13+
# One --node-taint flag per entry of K3S_NODE_TAINTS (key=value:Effect entries separated by spaces, none containing a space), for the server and the agent alike. k3s applies them when the node registers, so a node object that is deleted and registers again gets them back; a taint put on a live node by hand is lost that way.
14+
node_taint_flags=""
15+
for node_taint in ${K3S_NODE_TAINTS:-}; do
16+
node_taint_flags="$node_taint_flags --node-taint $node_taint"
17+
done
18+
1319
# k3s's own vpn-auth integration only runs `tailscale up`; it does not start the daemon itself (without this, k3s fails outright with "tailscale up failed: ... it doesn't appear to be running"). Waits for the control socket before anything talks to it.
1420
mkdir -p /var/run/tailscale /var/lib/tailscale
1521
tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock &
@@ -124,8 +130,8 @@ supervise_self_hosted() {
124130
fi
125131
echo "Starting k3s server ($mode)"
126132
rm -f /output/kubeconfig.yaml
127-
# shellcheck disable=SC2086 # Word splitting of server_args is deliberate: every argument is a flag or a value without spaces.
128-
/bin/k3s server $server_args &
133+
# shellcheck disable=SC2086 # Word splitting of server_args and node_taint_flags is deliberate: every argument is a flag or a value without spaces.
134+
/bin/k3s server $server_args $node_taint_flags &
129135
k3s_pid=$!
130136
watcher_pid=""
131137
if [ "$mode" = direct ] && [ -n "$join_key" ]; then
@@ -148,10 +154,11 @@ case "$role" in
148154
rm -f /output/kubeconfig.yaml
149155
mesh_server_args
150156
# shellcheck disable=SC2086 # As above.
151-
exec /bin/k3s server $server_args
157+
exec /bin/k3s server $server_args $node_taint_flags
152158
;;
153159
agent)
154-
exec /bin/k3s agent --vpn-auth="$vpn_auth" --resolv-conf=/etc/k3s-resolv.conf
160+
# shellcheck disable=SC2086 # As above.
161+
exec /bin/k3s agent --vpn-auth="$vpn_auth" --resolv-conf=/etc/k3s-resolv.conf $node_taint_flags
155162
;;
156163
*)
157164
echo "usage: node-entrypoint.sh server|agent" >&2

‎roles/github_runner_cluster/templates/env-agent.j2‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,6 @@ K3S_VPN_AUTH_CONTROL_SERVER_URL={{ github_runner_cluster_mesh_control_server_url
99
{% endif %}
1010
# Unused by the k3s-agent service itself, but docker compose validates every service's env interpolation up front regardless of which --profile is active - the k3s service requires K3S_TLS_SAN_LIST, and an agent host's .env needs to set SOMETHING for that check to pass under `docker compose --profile agent`.
1111
K3S_TLS_SAN_LIST={{ github_runner_cluster_effective_tls_sans | join(' ') }}
12+
{% if github_runner_cluster_node_taints | length > 0 %}
13+
K3S_NODE_TAINTS={{ github_runner_cluster_node_taints | join(' ') }}
14+
{% endif %}

‎roles/github_runner_cluster/templates/env.j2‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,6 @@ K3S_JOIN_SERVER_URL={{ github_runner_cluster_effective_server_url }}
1111
K3S_DATASTORE=sqlite
1212
{% endif %}
1313
K3S_NODE_NAME={{ github_runner_cluster_effective_node_name }}
14+
{% if github_runner_cluster_node_taints | length > 0 %}
15+
K3S_NODE_TAINTS={{ github_runner_cluster_node_taints | join(' ') }}
16+
{% endif %}

0 commit comments

Comments
 (0)