You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 85171e6
Browse filesBrowse the repository at this point in the historyBrowse files
feat(cluster): register a node with taints set per host
A taint put on a node with kubectl is lost whenever the node object is deleted and registers again,
which the cluster role does for an agent that fails to rejoin. github_runner_cluster_node_taints
passes key=value:Effect entries to k3s as --node-taint for servers and agents, so a host that must
stay free of ordinary workloads keeps that across re-registration.
Copy file name to clipboardExpand all lines: roles/github_runner_cluster/README.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -96,6 +96,7 @@ Each site is its own cluster: its own inventory group, bootstrap server and mesh
96
96
-`github_runner_cluster_tailnet_domain`: the mesh's MagicDNS domain. Each node's address is `<node name>.<domain>`, and join URLs and TLS SANs use it.
97
97
-`github_runner_cluster_headscale_*`: the Headscale providers' settings, documented in `defaults/main.yml`.
98
98
-`github_runner_cluster_node_name`: the k3s node name and mesh hostname, defaulting to the inventory name.
99
+
-`github_runner_cluster_node_taints`: Kubernetes taints (`key=value:Effect`, no spaces) the host's node registers with, empty by default. `NoSchedule` keeps pods without a matching toleration off it and `NoExecute` also evicts the ones already there. k3s applies them on every registration, so they come back if the node object is deleted, which a taint set with `kubectl` does not.
99
100
- Per-host overrides: `github_runner_cluster_node_role` (`server` or `agent`), `github_runner_cluster_bootstrap` (true picks the bootstrap host, false excludes a host), `github_runner_cluster_server_url` (join this server instead, for a cluster whose servers are outside the inventory), `github_runner_cluster_tls_sans`, `github_runner_cluster_node_address`. The run fails if the result has an even number of servers, no servers, or more than one bootstrap host.
100
101
-`github_runner_cluster_docker_context` (default empty): pins the Docker context this host's node runs in (see [Docker contexts](#docker-contexts)).
101
102
-`github_runner_cluster_compose_files`: extra Compose files for the host's k3s project, merged after `docker-compose.yml`, as paths relative to `github_runner_cluster_dir` or absolute. Several nodes on one Docker host each need their own `github_runner_cluster_dir` and `github_runner_cluster_compose_project`.
# Derived, never set directly: the name every template and check actually uses for this host (k3s node name, Tailscale device hostname, TLS SAN self-check).
# Kubernetes taints this host's node registers with, as key=value:Effect strings (for example github-runner.example.com/excluded=true:NoExecute), none containing a space. A taint keeps pods without a matching toleration off the node; NoExecute also evicts the ones already there. They are applied by k3s each time the node registers, so they survive the node object being deleted and registering again, which a taint put on the node by hand does not. Per host, in its host_vars.
22
+
github_runner_cluster_node_taints: []
23
+
21
24
github_runner_cluster_k3s_version: "latest"
22
25
23
26
# Secrets, as plain variables from any source Ansible reads (ansible-vault, a lookup, or a secrets adapter role such as github_runner_secrets_onepassword). github_runner_cluster_k3s_token is the shared node-join secret and has no default. github_runner_cluster_tailscale_join_key is, with the default tailscale mesh provider, what k3s's own --vpn-auth integration (see docker-compose.yml) uses on every host: it runs Tailscale INSIDE each node's own k3s container, joining the tailnet as its own device and using flannel's native "tailscale" backend for pod traffic. github_runner_cluster_tailscale_api_token, when set, lets tasks/mesh/tailscale_api.yml keep the tailnet's ACL grant and the join key itself provisioned; see that file's comments for what it manages and the one manual step (creating the API token) that has no API to automate.
# Empty (not :?required) on the bootstrap host - the one server with nothing to join. Every OTHER server host's own .env sets this to the bootstrap host's own URL.
51
51
- K3S_JOIN_SERVER_URL=${K3S_JOIN_SERVER_URL:-}
52
+
- K3S_NODE_TAINTS=${K3S_NODE_TAINTS:-}
52
53
volumes:
53
54
- k3s-data:/var/lib/rancher/k3s
54
55
# Persists this container's own Tailscale device identity/keys across recreations - without this, every recreate would register a brand new device on the tailnet instead of reconnecting as the same one, since the join key is reusable by design.
@@ -87,6 +88,7 @@ services:
87
88
# Not :?required, for the same reason as K3S_AGENT_SERVER_URL above: the self-hosted bootstrap server's first .env has no key yet, and Compose would reject it for this service too. k3s/node-entrypoint.sh fails fast without one instead.
# One --node-taint flag per entry of K3S_NODE_TAINTS (key=value:Effect entries separated by spaces, none containing a space), for the server and the agent alike. k3s applies them when the node registers, so a node object that is deleted and registers again gets them back; a taint put on a live node by hand is lost that way.
# k3s's own vpn-auth integration only runs `tailscale up`; it does not start the daemon itself (without this, k3s fails outright with "tailscale up failed: ... it doesn't appear to be running"). Waits for the control socket before anything talks to it.
# Unused by the k3s-agent service itself, but docker compose validates every service's env interpolation up front regardless of which --profile is active - the k3s service requires K3S_TLS_SAN_LIST, and an agent host's .env needs to set SOMETHING for that check to pass under `docker compose --profile agent`.
0 commit comments