Skip to content

feat(talos): rotate the Kubernetes service-account signing key - #4943

Merged
dapperdivers merged 3 commits into
mainfrom
feat/talos-sa-rotation
Nov 21, 2026
Merged

dapperdivers merged 3 commits into
mainfrom
feat/talos-sa-rotation

Conversation

@dapperdivers

@dapperdivers dapperdivers commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

The service-account signing key was exposed in a Claude Code session transcript on 2026-11-21. This rotates it with no downtime using Talos 1.14's KubeServiceAccountConfig.accepted.publicKeys, the only Talos config path that lets the apiserver trust more than one key at once. Talos refuses --service-account-key-file in extraArgs.

How

  • patches/controller/60-service-account.yaml moves the key from v1alpha1 .cluster.serviceAccount (deleted with $patch: delete, because Talos rejects both being set) into a KubeServiceAccountConfig document. render.sh fills issuer.privateKey from the SOPS bundle at render time, so only public keys live in git.
  • Stage A (commit 1): sign with the old key, accept the new one. Applied to control-1..3 one at a time.
  • Stage B (commit 2): sign with the new key, accept the old one. certs.k8sserviceaccount.key in talsecret.sops.yaml swapped via sops set --value-stdin; the rest of the bundle was checked byte-identical after decryption. Applied one control plane at a time.
  • The legacy observability/kube-prometheus-stack-prometheus-token Secret (used by the chart's control-plane ServiceMonitors) was re-signed with the new key.

Verified (both stages already applied live)

  • Each apiserver's service-account.pub holds both keys (fingerprints ed09c41d… old, 9effede6… new).
  • Tokens fresh from each apiserver carry the new key's kid, and old-key tokens are still accepted (Prometheus targets stayed up).
  • task talos:diff-cluster: all 11 nodes report No changes., so this PR matches what is running.
  • Workers are untouched.
  • Encrypted pre-rotation backups are in .private/backups/ and in Infisical /Infrastructure/Talos/Backups.
  • Stage C (commit 3): the old key was removed from accepted. Applied at 20:08 UTC, after all 171 pods with tokens carried the new key's kid (checked by reading the token header from each kubelet directory through the Talos API) and after the 1-hour lifetime of any controller-manager token issued before stage B had passed. Each apiserver now trusts only 9effede6…. No 401s, no target, pod or Flux regressions.

All three stages are live; task talos:diff-cluster reports No changes. on 11/11 nodes. The exposed key no longer authenticates.

@dunce-bot

dunce-bot Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Note

No rendered changes.

konflate 0.6.6 · rendered a25350b · full diff →

The old key was exposed in a session transcript. Stage A makes the
apiservers trust a new key while still signing with the old one:

- the key moves from v1alpha1 .cluster.serviceAccount into a Talos 1.14
  KubeServiceAccountConfig document (only that has accepted.publicKeys);
  render.sh fills issuer.privateKey from the SOPS bundle
- accepted.publicKeys lists the new key (sha256 DER 9effede6126fa48e)
- the new private key is staged in service-account-next.sops.yaml

Applied to talos-control-1..3 one at a time; each apiserver trusts both
keys and old-key tokens still authenticate.
Sign with the new key and keep trusting the old one:

- talsecret.sops.yaml certs.k8sserviceaccount.key is now the new key
  (set with `sops set --value-stdin`; nothing else in the bundle changed)
- accepted.publicKeys lists the old key (sha256 DER ed09c41d875c9a8d)
- service-account-next.sops.yaml removed (its key is in the bundle now)

Applied to talos-control-1..3 one at a time; each apiserver issues
new-key tokens (kid nv_t5hJvpI5c...) and still accepts old-key ones. The
legacy kube-prometheus-stack-prometheus-token Secret was re-signed with
the new key; Prometheus reloaded and its control-plane targets are up.

Stage C (drop the old key) follows once no pod holds an old-key token.
Drop the old (exposed) key from accepted.publicKeys. The apiservers now
trust only the new key (sha256 DER 9effede6126fa48e).

Done after every pod's projected token carried the new key's kid (171/171,
read from the kubelet dir via the Talos API, header only) and past the 1h
lifetime of any controller-manager token issued before stage B. Applied to
talos-control-1..3 one at a time: no 401s, no target or pod regressions.
@dapperdivers
dapperdivers force-pushed the feat/talos-sa-rotation branch from a25350b to 630163b Compare November 21, 2026 20:26
@dapperdivers
dapperdivers merged commit 96084e3 into main Nov 21, 2026
6 checks passed
@dapperdivers
dapperdivers deleted the feat/talos-sa-rotation branch November 21, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant