Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions roles/github_runner_cluster/files/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ services:
K3S_VERSION: "${K3S_VERSION:-latest}"
# k3s's own --vpn-auth integration (documented, if experimental) runs Tailscale INSIDE the node's container rather than depending on the host's own Tailscale client: each node's k3s process joins the mesh as its own device, flannel uses its native "tailscale" backend (not VXLAN) for pod traffic, and k3s takes the resulting mesh IP as its node IP. This replaced an earlier, considerably hackier approach (--node-external-ip + --flannel-external-ip layered on the HOST's own Tailscale, tunnelled through Docker's UDP port-publishing) that needed two nested tunnels for the same traffic and was correspondingly fragile.
#
# k3s/node-entrypoint.sh starts tailscaled, joins the mesh (Tailscale's own control plane, or the Headscale server K3S_VPN_AUTH_CONTROL_SERVER_URL names), then execs k3s; its comments explain each k3s flag (--node-ip, --cluster-init or --server, --tls-san, --resolv-conf) and the live failures behind them. Bind-mounted from this checkout below rather than built into the image: a host that already has the image built would otherwise keep running an old copy, since Compose only builds a missing image.
# k3s/node-entrypoint.sh starts tailscaled, joins the mesh (Tailscale's own control plane, or the Headscale server K3S_VPN_AUTH_CONTROL_SERVER_URL names), then execs k3s; its comments explain each k3s flag (--node-ip, --cluster-init or --server, --tls-san, --resolv-conf) and the live failures behind them. Bind-mounted from this project directory below rather than built into the image: a host that already has the image built would otherwise keep running an old copy, since Compose only builds a missing image.
entrypoint: ["/bin/sh", "/usr/local/bin/github-runner-node", "server"]
# Stable node name across container recreations. Without this k3s derives the node name from the container's hostname (its ID), so every recreate (e.g. on an image pull) would register a NEW node and leave the old one stale in etcd. Pinning it keeps `kubectl get nodes` to one entry. This is also the hostname Tailscale registers the container's own new device under, giving it a stable MagicDNS name too. Every server host sets its own K3S_NODE_NAME (Ansible defaults it to the host's own inventory name; the bootstrap host overrides it to keep its pre-existing "k3s-server" identity rather than forcing a disruptive rename - see host_vars/node-a.yml).
# Stable node name across container recreations. Without this k3s derives the node name from the container's hostname (its ID), so every recreate (e.g. on an image pull) would register a NEW node and leave the old one stale in etcd. Pinning it keeps `kubectl get nodes` to one entry. This is also the hostname Tailscale registers the container's own new device under, giving it a stable MagicDNS name too. Every server host sets its own K3S_NODE_NAME (Ansible defaults it to the host's own inventory name; a host that joined under a different name, such as a bootstrap host first brought up as "k3s-server", keeps that name through github_runner_cluster_node_name rather than being renamed disruptively).
hostname: ${K3S_NODE_NAME:-k3s-server}
container_name: github-runner-k3s
privileged: true
Expand Down
2 changes: 1 addition & 1 deletion roles/github_runner_cluster/files/k3s/node-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/sh
# Entry point for the k3s and k3s-agent services in docker-compose.yml: starts Tailscale inside the node's own container, then k3s with its --vpn-auth integration, so each node joins the mesh as its own device. Bind-mounted from the checkout rather than baked into the k3s image, so a host picks up a change to it with the Compose file that uses it, without rebuilding an image Compose would not otherwise rebuild.
# Entry point for the k3s and k3s-agent services in docker-compose.yml: starts Tailscale inside the node's own container, then k3s with its --vpn-auth integration, so each node joins the mesh as its own device. Bind-mounted from the Compose project directory the role manages (github_runner_cluster_dir) rather than baked into the k3s image, so a host picks up a change to it with the Compose file that uses it, without rebuilding an image Compose would not otherwise rebuild.
#
# Usage: node-entrypoint.sh server|agent. Reads, from the container's environment (docker-compose.yml, from .env): K3S_VPN_AUTH_JOIN_KEY, K3S_VPN_AUTH_CONTROL_SERVER_URL (empty for Tailscale's own control plane, a Headscale URL otherwise), K3S_JOIN_SERVER_URL, K3S_TLS_SAN_LIST and K3S_DATASTORE (server), and K3S_MESH_SELF_HOSTED_HEALTH_URL (the bootstrap server of a cluster that runs its own Headscale, see supervise_self_hosted below).
# No set -e: a failed `tailscale up` falls through to waiting for an address, as it always has, rather than exiting and restarting the container.
Expand Down