@@ -8,6 +8,44 @@ Which hosts are servers is decided automatically from the inventory group named
 
 The role reads `docker compose version` first and fails, before changing anything on the host, if it is 2.37.1 or later but older than 2.39.0. Those releases create a container from an image they have just built without recording that image's ID on it ([docker/compose#13047](https://github.com/docker/compose/pull/13047)), so the next run sees a different image and recreates the container, which restarts every node on the run after the one that built the k3s image. Releases before 2.37.1 and from 2.39.0 on keep the containers. GitHub's ubuntu-latest runner image has shipped 2.38.2, so a CI job that runs the role there needs a different Compose installed first, as `.github/workflows/mesh-integration.yml` does.
 
+## Versions
+
+Every node's image is built from two pinned tags: `github_runner_cluster_k3s_version` (a `rancher/k3s` release tag, k3s's version with `+` written as `-`, such as `v1.34.1-k3s1`; release candidates are not accepted) and `github_runner_cluster_tailscale_version` (a `tailscale/tailscale` tag such as `v1.102.4`). The defaults are the versions this collection's integration tests run. The role writes both into the host's `.env`, the Compose file refuses to build without them, and the Dockerfile has no default for either, so no node is ever built from `latest`. Set them once for the whole group, in the inventory or `group_vars`, not per host and not in play vars.
+
+Before it changes anything on a host the role fails:
+
+- if either is not a pinned tag;
+- if the hosts of `github_runner_cluster_group` do not all name the same k3s minor version, listing each host's version. Every host is read from the inventory, and one that sets nothing counts as the default, so the check gives the same answer on every host, including under `--limit`;
+- if the k3s version is older than, or more than one minor above, what this node's running container or any node of the cluster runs now. k3s upgrades one minor at a time and cannot be downgraded. The role reads `k3s --version` in this node's container, and the nodes' kubelet versions through every server of the group in the inventory (each on its own host, with its pinned Docker context, waiting at most `github_runner_cluster_cluster_read_timeout` for its API server), so one server stopped or being rebuilt does not hide what the others run. When servers run but none of them can be read, it refuses, naming each and why. When no server's container runs (a first bootstrap), or the inventory holds no server (a host `bootstrap.sh` joins to a cluster outside it), it compares only this node's own container and says so. `-e github_runner_cluster_allow_k3s_version_jump=true` skips this comparison for one run: for a deliberate downgrade of a node that was never upgraded, a cluster that is being replaced, or one that cannot be read.
+
+A new cluster can start on a newer k3s than the default by setting both variables; the integration tests have run only the defaults.
+
+### Upgrading from a release before the pins
+
+Releases before the pins built every node from `latest`, and this change refuses what they accepted:
+
+- `github_runner_cluster_k3s_version: latest`, or `K3S_VERSION=latest` in `.env.bootstrap`: set the `rancher/k3s` tag the cluster runs instead (`k3s --version` in a node's container, with `+` written as `-`), or leave it unset for the default;
+- a cluster that floated to a newer k3s minor than the default: set `github_runner_cluster_k3s_version` to the tag it runs, once for the whole group, before running the role;
+- a Compose file path (in `github_runner_cluster_dir` or `github_runner_cluster_compose_files`) with a colon in it: move it, since `.env` now lists the files in `COMPOSE_FILE`.
+
+Set `github_runner_cluster_tailscale_version` too if the nodes run another Tailscale than the default (`tailscale version` in a node's container). The first run after upgrading rewrites each node's `.env` and recreates nothing.
+
+Changing a pin changes no running node. Compose builds the image only when it is missing and leaves the build arguments out of a container's configuration hash, so the next run of the role rewrites `.env` and leaves the container alone. To move the cluster to a new version, run the role once with the new pins, so every host's `.env` names them, then rebuild and restart the nodes by hand, one at a time.
+
+The node's `.env` names the Compose files, as absolute paths, and the project the role runs the node with (`COMPOSE_FILE` and `COMPOSE_PROJECT_NAME`), so a `docker compose` command run by hand in `github_runner_cluster_dir`, or with `--project-directory` pointing at it, acts on the same project with the same files, including the extra file a `headscale_in_cluster` bootstrap server or a `github_runner_cluster_compose_files` entry adds. Each command still needs `--profile server` (or `--profile agent` on an agent), since every service sits in a profile, and, on a host with `github_runner_cluster_docker_context` set, that context: run it as `docker --context <context> compose ...`.
+
+1. Read k3s's release notes for every minor version between the old and the new one. A jump of more than one minor is several upgrades, each run to the end before the next, and the role refuses to build a node more than one minor ahead of the cluster.
+2. Servers first, one at a time. Servers run runner pods as well as the control plane, so before each one:
+   - check that every server's node is Ready (`kubectl get nodes`) and that each server's own API reports etcd healthy (`kubectl get --raw '/readyz?verbose'` through that server lists `[+]etcd ok`), so the cluster keeps its quorum while this one restarts: with three servers, at most one may be down;
+   - cordon it (`kubectl cordon <node>`) and wait until no runner pod is left on it (`kubectl get pods --all-namespaces --field-selector spec.nodeName=<node>`), so no job is cut off, then drain the rest (`kubectl drain <node> --ignore-daemonsets --delete-emptydir-data`).
+
+   Then, in `github_runner_cluster_dir`, run `docker compose --profile server build k3s` and `docker compose --profile server up -d k3s`. Wait until its node is Ready again and the etcd checks above pass on every server, then `kubectl uncordon <node>`, before the next server.
+3. Then agents, one at a time, the same way: cordon, wait for its runner pods to finish, drain, `docker compose --profile agent build k3s-agent` and `docker compose --profile agent up -d k3s-agent`, wait for Ready, uncordon.
+
+A Tailscale upgrade on its own follows the same order; it restarts each node's mesh client with the node.
+
+The role copies the new `docker-compose.yml` before it writes `.env`. A run that stops between the two (a failed check, a lost connection) leaves a Compose file that needs `TAILSCALE_VERSION` beside an `.env` without it, and every hand-run `docker compose` command in that directory fails with a missing-variable error until a run of the role succeeds.
+
 ## Docker host disk
 
 Each host also runs a `docker-prune` service that removes Docker build cache and dangling images older than `DOCKER_PRUNE_AFTER` (a week by default) once every `DOCKER_PRUNE_INTERVAL_SECONDS` (daily). The node's ephemeral storage lives on the host's Docker disk, shared with anything else that builds there, so an untrimmed cache eventually puts the node under disk pressure and taints it unschedulable. Tagged images, containers and volumes are never touched. Set either variable in the host's `.env` to change them.
@@ -95,6 +133,7 @@ Each site is its own cluster: its own inventory group, bootstrap server and mesh
 - `github_runner_cluster_mesh` (default `tailscale`), `github_runner_cluster_mesh_tag` (default `tag:github-runner-node`), `github_runner_cluster_pod_cidr` (default k3s's `10.42.0.0/16`).
 - `github_runner_cluster_tailnet_domain`: the mesh's MagicDNS domain. Each node's address is `<node name>.<domain>`, and join URLs and TLS SANs use it.
 - `github_runner_cluster_headscale_*`: the Headscale providers' settings, documented in `defaults/main.yml`.
+- `github_runner_cluster_k3s_version`, `github_runner_cluster_tailscale_version`: the pinned image tags every node is built from, `github_runner_cluster_allow_k3s_version_jump` (default false) to skip the comparison with what runs, and `github_runner_cluster_cluster_read_timeout` (default 30s) for each server's answer to it (see [Versions](#versions)). `github_runner_cluster_k3s_tested_version` is the k3s default, which the minor-version check compares hosts that set nothing against; set `github_runner_cluster_k3s_version` instead.
 - `github_runner_cluster_node_name`: the k3s node name and mesh hostname, defaulting to the inventory name.
 - `github_runner_cluster_node_taints`: Kubernetes taints (`key=value:Effect`, no spaces) the host's node registers with, empty by default. `NoSchedule` keeps pods without a matching toleration off it and `NoExecute` also evicts the ones already there. k3s applies them on every registration, so they come back if the node object is deleted, which a taint set with `kubectl` does not.
 - Per-host overrides: `github_runner_cluster_node_role` (`server` or `agent`), `github_runner_cluster_bootstrap` (true picks the bootstrap host, false excludes a host), `github_runner_cluster_server_url` (join this server instead, for a cluster whose servers are outside the inventory), `github_runner_cluster_tls_sans`, `github_runner_cluster_node_address`. The run fails if the result has an even number of servers, no servers, or more than one bootstrap host.