Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions .taskfiles/talos/Taskfile.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,12 @@ tasks:
- which talosctl sops yq envsubst

diff-node:
desc: Show what applying the rendered config would change on a node (dry run) [IP=required]
cmd: talosctl --nodes {{.IP}} apply-config --dry-run --file <(./render.sh "$({{.TALOS_DIR}}/nodes.sh {{.IP}} | cut -d' ' -f1)")
desc: Summarise what applying the rendered config would change on a node (dry run, no values shown) [IP=required] [FULL=false]
cmd: FULL={{.FULL}} ./diff.sh "$({{.TALOS_DIR}}/nodes.sh {{.IP}} | cut -d' ' -f1)"
dir: "{{.TALOS_DIR}}"
vars:
# FULL=true prints talosctl's raw diff, which contains CA keys and tokens.
FULL: '{{.FULL | default "false"}}'
requires:
vars: [IP]
preconditions:
Expand All @@ -33,12 +36,11 @@ tasks:
- which talosctl sops yq envsubst

diff-cluster:
desc: Dry-run the rendered config against every node
desc: Dry-run the rendered config against every node (summary only, no values shown)
dir: "{{.TALOS_DIR}}"
cmd: |
./nodes.sh | while read -r host ip _; do
echo "### $host ($ip)"
talosctl --nodes "$ip" apply-config --dry-run --file <(./render.sh "$host") | grep -E '^(@@|[+-] )' || echo "no changes"
./nodes.sh | while read -r host _; do
./diff.sh "$host" || true
done
preconditions:
- test -f {{.SOPS_AGE_KEY_FILE}}
Expand All @@ -57,7 +59,7 @@ tasks:
INSECURE: '{{.INSECURE | default "false"}}'
requires:
vars: [IP]
prompt: Apply the rendered Talos config to {{.IP}} (diff shown first)... continue?
prompt: Apply the rendered Talos config to {{.IP}} (summary of the change shown first)... continue?
preconditions:
- sh: '{{if eq .INSECURE "true"}}exit 0{{else}}talosctl --nodes {{.IP}} get machineconfig{{end}}'
msg: "Node {{.IP}} is not reachable or not configured. Use INSECURE=true for first-time setup."
Expand Down
11 changes: 5 additions & 6 deletions docs/src/operations/sso-cluster-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,11 @@ unreachable Authentik degrades only OIDC token validation, never apiserver start
`Headlamp OIDC` (confidential, `client_id: headlamp`).
- **Issuers**: `https://sso.chelonianlabs.com/application/o/kubectl/` and
`…/o/headlamp/` (trailing slash required).
- **apiserver config**: `kubernetes/bootstrap/talos/patches/controller/apiserver-oidc.yaml`
— the auth config is written via `machine.files` to `/var/lib/kubernetes/authentication/`
(must be a writable `/var` path — a `machine.files` entry under `/etc/kubernetes` makes
Talos overlay that dir read-only and breaks the kubelet's PKI writes) and mounted into
the apiserver pod. Adding another OIDC app = one more `jwt:` block. Applying this patch
requires a per-node reboot (`machine.files` changes are not immediate-mode).
- **apiserver config**: `kubernetes/bootstrap/talos/patches/controller/30-apiserver-oidc.yaml`
— a Talos `KubeAuthenticationConfig` document. Talos renders it into the apiserver's
`--authentication-config` file and restarts the apiserver when it changes (no reboot).
Anonymous access is turned off in the same document. Adding another OIDC app = one more
`jwt:` block.
- **RBAC**: `kubernetes/apps/kube-system/oidc-rbac/` binds
`oidc:kubernetes-admins` → `cluster-admin` and `oidc:kubernetes-viewers` → `view`;
both issuers map claims with the same `oidc:` prefix, so bindings apply to both.
Expand Down
59 changes: 47 additions & 12 deletions kubernetes/bootstrap/talos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,10 @@ the rendered output contains cluster secrets and is never written to disk by the
| `talsecret.sops.yaml` | `talosctl gen secrets` bundle, SOPS-encrypted |
| `patches/controller/*.yaml` | Control-plane-only patches |
| `patches/global/*.yaml` | Patches for every node |
| `nodes/<role>/<hostname>.yaml` | Per-node patch: install disk/image, labels, hostname and network documents |
| `nodes/<role>/<hostname>.yaml` | Per-node patch: install image/disk, labels, hostname and network documents |
| `render.sh <hostname>` | Prints the full machine config for a node |
| `nodes.sh [ip\|hostname]` | Lists nodes as `<hostname> <ip> <role> <install-image>` |
| `diff.sh <hostname>` | Dry run against the node; prints a summary of the change without values |
| `clusterconfig/talosconfig` | Admin client config (gitignored) |

The directory a node file sits in (`controlplane/` or `worker/`) is its role.
Expand All @@ -32,30 +33,64 @@ and lists are appended:
3. `patches/global/*.yaml`
4. `nodes/<role>/<hostname>.yaml`

Role patches come before global ones so list order matches what talhelper produced. Files apply
in name order, hence the numeric prefixes. `${TALOS_VERSION}` and `${KUBERNETES_VERSION}` are
substituted from `versions.yaml`, which is how the install images pick up the Talos version.
Files apply in name order, hence the numeric prefixes. `${TALOS_VERSION}` and
`${KUBERNETES_VERSION}` are substituted from `versions.yaml`, which is how the install images pick
up the Talos version.

`configContract` in `versions.yaml` is passed to `talosctl gen config --talos-version`. It stays at
`v1.13` until the patches are converted to the Talos 1.14 multi-document format; with a `v1.14`
contract the base config emits the new documents and the v1alpha1 patches conflict with them.
Patches are Talos 1.14 configuration documents (`kind: KubeAPIServerConfig`, `KubeletConfig`,
`ResolverConfig`, `UnattendedInstallConfig`, …). A patch document merges into the base document of
the same kind (and `name`, where the kind has one): maps merge, lists append, and
`$patch: delete` removes the base document. Only what has no document yet stays in the v1alpha1
`machine:` / `cluster:` form: etcd settings, `machine.certSANs`, and the containerd customization in
`patches/global/20-machine-files.yaml` (see the comment there).

Every secret (CAs, tokens, the service-account signing key, the etcd encryption key) comes from the
secrets bundle through the generated base. Patches never hold secret material.

## Config contract

`configContract` in `versions.yaml` is passed to `talosctl gen config --talos-version`. It decides
which documents the generated base contains **and their defaults**, so raising it is a migration,
not a version bump: diff the new base against the old one first.

The move from `v1.13` to `v1.14` (November 2026) was done without changing node behaviour. Three
things hold the old behaviour in place and are worth knowing about:

- `patches/global/70-pinned-defaults.yaml` removes the documents a v1.14 base adds that the cluster
never ran with: workload isolation, the secure `/var` mount and the weekly filesystem trim.
- `patches/controller/00-apiserver.yaml` sets `startupProbes: false`, and
`30-apiserver-oidc.yaml` sets `anonymous.enabled: false`. The v1.14 defaults are probes on and
anonymous access to the health paths. Change the two together or not at all: the probes are
anonymous requests, so probes on with anonymous off gets the apiserver killed.
- `render.sh` appends the `identity` provider to `KubeEtcdEncryptionConfig`, which is what Talos
generated before v1.14.

To rotate the service-account key again, add a `KubeServiceAccountConfig` patch with
`accepted.publicKeys`; the signing key itself stays in the bundle.

## Tasks

```sh
task talos:diff-node IP=10.100.0.50 # dry run: what would change on one node
task talos:diff-cluster # dry run against every node
task talos:apply-node IP=10.100.0.50 # shows the diff, asks, then applies
task talos:diff-node IP=10.100.0.50 # dry run: summary of what would change on one node
task talos:diff-cluster # the same for every node
task talos:apply-node IP=10.100.0.50 # shows the summary, asks, then applies
task talos:render-config IP=10.100.0.50 # print the config (contains secrets)
```

Always run a diff before applying. When the repo and the nodes agree, every node reports
`No changes.`
`no changes`.

The diff tasks print a summary only: whether a reboot is needed, how many lines change, and which
keys and documents are added or removed. Values are never shown, because a machine config diff
contains CA keys and tokens. That makes the tasks safe in CI logs and agent sessions.
`task talos:diff-node IP=… FULL=true` prints talosctl's raw diff for when you need to see the
values yourself.

## Adding a node

1. Create `nodes/<role>/<hostname>.yaml` (copy a sibling and change the MACs, addresses and disk).
`nodes.sh` finds a node by the first address on link `ethSel0`.
`nodes.sh` finds a node by the first address on link `ethSel0` and reads its install image
from the `UnattendedInstallConfig` document.
2. `task talos:add-node IP=<ip>` while the node is in maintenance mode.

## Schematics
Expand Down
47 changes: 47 additions & 0 deletions kubernetes/bootstrap/talos/diff.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Dry-run the rendered config against a node and summarise what would change, without
# printing any value: the diff of a machine config contains CA keys and tokens.
#
# ./diff.sh <hostname> summary: apply mode, line counts, changed keys and documents
# FULL=true ./diff.sh <host> the raw diff from talosctl (contains secrets; needs a terminal
# you trust, never a log or an agent session)
set -euo pipefail

dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
host="${1:?usage: diff.sh <hostname>}"
ip="$("$dir/nodes.sh" "$host" | cut -d' ' -f2)"

work="$(mktemp -d)"
chmod 700 "$work"
trap 'rm -rf "$work"' EXIT

# talosctl writes the dry-run report (summary and diff) to stderr.
if ! talosctl --nodes "$ip" apply-config --dry-run --file <("$dir/render.sh" "$host") > "$work/out" 2>&1; then
# A failure message can quote config lines: show only its first line.
echo "$host ($ip): dry run failed: $(grep -vE '^([+ -]|@@|$)' "$work/out" | head -1 | cut -c1-200)" >&2
exit 1
fi

if [ "${FULL:-false}" = "true" ]; then
cat "$work/out"
exit 0
fi

if grep -q 'No changes' "$work/out"; then
echo "$host ($ip): no changes"
exit 0
fi

mode="$(grep -E '^Applied configuration' "$work/out" | head -1 || true)"
added="$(grep -cE '^\+([^+]|$)' "$work/out" || true)"
removed="$(grep -cE '^-([^-]|$)' "$work/out" || true)"
echo "$host ($ip): ${mode:-changes} +$added -$removed lines"

# Changed lines reduced to "<sign><indent><key>:". Values, list items and block text are dropped.
keys() { grep -oE "^\\$1 *[A-Za-z0-9_.\$/-]+:" "$work/out" | sed -E "s/^\\$1//; s/:\$//" | sort -u | tr '\n' ' ' | sed -E 's/ +/ /g'; }
docs() { grep -E "^\\$1kind: [A-Za-z0-9]+\$" "$work/out" | sed -E "s/^\\$1kind: //" | sort | uniq -c | awk '{printf "%s%s ", $2, ($1 > 1 ? "×" $1 : "")}'; }
echo " documents added: $(docs +)"
echo " documents removed: $(docs -)"
echo " keys added: $(keys +)"
echo " keys removed: $(keys -)"
echo " (values are hidden; FULL=true shows the raw diff, secrets included)"
2 changes: 1 addition & 1 deletion kubernetes/bootstrap/talos/nodes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ for f in "$dir"/nodes/*/*.yaml; do
host="$(basename "$f" .yaml)"
role="$(basename "$(dirname "$f")")"
ip="$(yq -e 'select(.kind == "LinkConfig" and .name == "ethSel0") | .addresses[0].address' "$f" | cut -d/ -f1)"
image="$(yq -e 'select(document_index == 0) | .machine.install.image' "$f" | envsubst '${TALOS_VERSION}')"
image="$(yq -e 'select(.kind == "UnattendedInstallConfig") | .installer.image' "$f" | envsubst '${TALOS_VERSION}')"
if [ $# -eq 0 ] || [ "$1" = "$ip" ] || [ "$1" = "$host" ]; then
echo "$host $ip $role $image"
found=1
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# talos-control-1 (10.100.0.50). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/vda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/vda"
wipe: false
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# talos-control-2 (10.100.0.51). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/vda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/vda"
wipe: false
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# talos-control-3 (10.100.0.52). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/vda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/vda"
wipe: false
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
52 changes: 36 additions & 16 deletions kubernetes/bootstrap/talos/nodes/worker/talos-node-gpu-1.yaml
Original file line number Diff line number Diff line change
@@ -1,20 +1,40 @@
# talos-node-gpu-1 (10.100.0.53). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/sda
image: factory.talos.dev/installer/1f6caf935b35f840ea19acb12d2dcc6d8f9c7d47fc621b1fcebb4a04c5e3130d:${TALOS_VERSION}
nodeLabels:
hardware.cluster.io/gpu: "p100"
nvidia.com/gpu.present: "true"
nvidia.com/mps.capable: "true"
kernel:
modules:
- name: nvidia
- name: nvidia_uvm
- name: nvidia_drm
- name: nvidia_modeset
sysctls:
net.core.bpf_jit_harden: 1
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/1f6caf935b35f840ea19acb12d2dcc6d8f9c7d47fc621b1fcebb4a04c5e3130d:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/sda"
wipe: false
---
apiVersion: v1alpha1
kind: KubeNodeConfig
labels:
hardware.cluster.io/gpu: "p100"
nvidia.com/gpu.present: "true"
nvidia.com/mps.capable: "true"
---
apiVersion: v1alpha1
kind: SysctlConfig
params:
net.core.bpf_jit_harden: "1"
---
apiVersion: v1alpha1
kind: KernelModuleConfig
name: nvidia
---
apiVersion: v1alpha1
kind: KernelModuleConfig
name: nvidia_uvm
---
apiVersion: v1alpha1
kind: KernelModuleConfig
name: nvidia_drm
---
apiVersion: v1alpha1
kind: KernelModuleConfig
name: nvidia_modeset
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
26 changes: 18 additions & 8 deletions kubernetes/bootstrap/talos/nodes/worker/talos-node-large-1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,24 @@
# create user namespaces. Talos defaults this to 0 as hardening, which makes the
# sandbox fail with "fork/exec ...: no space left on device". Only enable it on
# nodes labelled workload.cluster.io/agent-sandbox, where the worker pool is pinned.
machine:
install:
disk: /dev/sda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
nodeLabels:
workload.cluster.io/agent-sandbox: "true"
sysctls:
user.max_user_namespaces: "11255"
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/sda"
wipe: false
---
apiVersion: v1alpha1
kind: KubeNodeConfig
labels:
workload.cluster.io/agent-sandbox: "true"
---
apiVersion: v1alpha1
kind: SysctlConfig
params:
user.max_user_namespaces: "11255"
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# talos-node-large-2 (10.100.0.55). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/sda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/sda"
wipe: false
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# talos-node-large-3 (10.100.0.56). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/sda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/sda"
wipe: false
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# talos-node-small-1 (10.100.0.60). Its role comes from the directory this file is in.
machine:
install:
disk: /dev/sda
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
apiVersion: v1alpha1
kind: UnattendedInstallConfig
installer:
image: factory.talos.dev/installer/5cc10e6e83c6e3b32167fe812094d4715c9fa3e1a77bcfe61fa4cf71b4b9fed3:${TALOS_VERSION}
provisioning:
diskSelector:
match: disk.dev_path == "/dev/sda"
wipe: false
---
apiVersion: v1alpha1
kind: HostnameConfig
Expand Down
Loading
Loading