Skip to content

Commit dfdfe6d

Browse files
committed
docs(cluster): describe the node network modes and moving a node to a bridged Colima VM
1 parent 184934d commit dfdfe6d

2 files changed

Lines changed: 66 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ ARC's `gha-runner-scale-set` solves both by design: it runs one fresh pod per jo
100100

101101
The control plane itself runs genuine embedded-etcd HA across three `server`-role hosts, not a single node with worker capacity bolted on: one host bootstraps with `--cluster-init` (the one supported migration path from k3s's default single-node SQLite datastore to embedded etcd), and two further hosts join as full `k3s server` members - a real vote each in etcd's own raft quorum, not just schedulable capacity. 3 is the practical minimum: etcd tolerates the loss of any *minority* of its members without losing quorum, and with 2 servers that minority is zero - a second server can only ever make quorum strictly worse than staying at one, never better, so "2-node HA" isn't a real intermediate step, it's a trap. Getting there over Tailscale rather than a shared LAN needed real, non-obvious work - see the [Non-obvious behaviour](#non-obvious-behaviour) section's etcd-specific entries for what actually broke and why.
102102

103-
Two or more physical hosts on different home networks can't reach each other's LAN IPs directly, so cross-node cluster traffic (a second node joining, cross-node pod-to-pod traffic, the API server's proxy to a remote kubelet, and now etcd's own peer traffic between `server` hosts) needs a real overlay between them - see [Tailscale](#tailscale) below for how that overlay is built and the DNS pitfall it introduces.
103+
Hosts on different networks can't reach each other's LAN addresses, and even hosts that share a LAN can't from inside their Docker VMs by default: each node's k3s container sits behind its VM's NAT, so its own Tailscale client has no address to offer its peers and every pair of nodes is relayed. So cross-node cluster traffic (a node joining, cross-node pod-to-pod traffic, the API server's proxy to a remote kubelet, and etcd's own peer traffic between `server` hosts) runs over a real overlay with mesh addresses - see [Tailscale](#tailscale) below for how that overlay is built and the DNS pitfall it introduces. Where hosts do share a LAN, `github_runner_cluster_network_mode: host`, on a Docker VM that has an address of its own on that LAN, lets their Tailscale clients reach each other directly over it, with the relay as the fallback when a host leaves; nothing k3s uses changes (see `roles/github_runner_cluster/README.md`, Networking).
104104

105105
The custom runner toolchain (GCC 11/G++ for C++20, Bun, the `gh` CLI — see `Dockerfile`) carries over from the previous design, rebased onto ARC's own [`actions/actions-runner`](https://github.com/actions/runner/pkgs/container/actions-runner) base image instead of `catthehacker/ubuntu:act-latest`. ARC's runner pods use their own registration flow, so the previous design's vendored [myoung34/docker-github-actions-runner](https://github.com/myoung34/docker-github-actions-runner) scripts (registration, GitHub App JWT signing, ephemeral re-registration) are not needed.
106106

0 commit comments

Comments
 (0)