docs(upgrades): record 2026-10-04 wave and cert renewal

[surrealwolf]

Summary

Docs for the 2026-10-04 upgrade wave, plus the rancher_version default bump.

• UPGRADE_PLAN.md: items Add poc-apps cluster and standardize variable naming #1–Enable cert-manager gateway-shim controller for Gateway API support #6 marked done. New "Executed 2026-10-04" section with versions, PRs, backup locations, measured impact and learnings for each item: Rancher v2.15.2, democratic-csi removal, Envoy Gateway v1.9.2 / Gateway API v1.6.1, CNPG 1.30.1, Grafana 12.4.12, and rolling OS reboots of all 31 nodes.
• RKE2_CERT_ROTATION.md: marked done. New "Renewal on 2026-10-04" section with per-server before/after expiries. All leaf and agent certs now expire 2027-10-05 (poc-apps-1's server certs 2027-09-19); the next renewal window opens 2027-06-07.
• OPS_NOTES.md:
  • poc-apps wildcard cert marked RESOLVED (gitops-core#6, 2026-10-04); cert renewal recorded; versions bumped; CNPG now uses switchover.
  • New known issues: Gateway VIPs on :443 are caught by ingress-nginx's hostPort (root cause and fix options); TCPRoute/UDPRoute need migrating to v1; disk at 82–84% on two prd workers; game servers pinned to single nodes; Grafana chart frozen; system-upgrade-controller still v0.20.1.
  • Local Terraform state files are mode 0777.
• TRUENAS_CSI_MIGRATION.md: correction to docs(storage): list TrueNAS cleanup for democratic-csi #44. The key that was in democratic_csi_api_key is the same TrueNAS key (ID 4) as truenas_csi_api_key, and it is live in truenas-csi/truenas-api-credentials on nprd, prd and poc. Revoking it would break TrueNAS CSI everywhere. Only the older keys (IDs 1 and 2) are safe to revoke.
• CLUSTER_ACCESS_AND_SSO.md: break-glass and cert calendar expiries updated to 2027-10-05.
• CHANGELOG.md: entries for this wave and for fix(envoy-gateway): make module safe for upgrades, pin v1.9.2 #40–fix(envoy-gateway): verify controller pods by their real label #42.
• Terraform: rancher_version default and both tfvars examples changed v2.15.0 → v2.15.2, matching live and the local tfvars. deploy_rancher has no version trigger, so this changes no plan.

Test plan

• Docs only, plus the one variable default; no apply needed.
• Live values checked on 2026-10-04: Rancher image, webhook and Fleet versions; rke2 certificate check on all 12 servers and one agent per cluster; kernel and pending updates on all 31 nodes; the latest cert-sync job; TrueNAS CSI secret presence (values not printed).

---

Note

Low Risk
Documentation and a Terraform default pin only; no cluster apply or module logic changes in this diff.

Overview
Records the 2026-10-04 upgrade wave in ops docs and aligns the repo with live Rancher v2.15.2.

Docs: UPGRADE_PLAN.md marks P0 items #1–#6 done and adds an Executed 2026-10-04 section (Rancher, democratic-csi cleanup, Envoy Gateway / Gateway API, CNPG, Grafana, 31-node rolling reboots) with backups, measured impact, and runbook learnings. RKE2_CERT_ROTATION.md is marked complete with per-server renewal results (leaf certs 2027-10-05). OPS_NOTES.md, CLUSTER_ACCESS_AND_SSO.md, and CHANGELOG.md update versions, cert calendars, resolved poc-apps wildcard sync, and new known issues (Gateway VIP :443 vs ingress-nginx hostPort, Terraform state permissions, etc.). TRUENAS_CSI_MIGRATION.md corrects guidance: do not revoke the old democratic_csi_api_key — it is the live TrueNAS CSI key (ID 4).

Terraform: rancher_version default and both terraform.tfvars.example files change v2.15.0 → v2.15.2 to match clusters already upgraded; no functional deploy trigger beyond the pin for new installs.

^{Reviewed by Cursor Bugbot for commit 0f8b7f9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Approved. Cursor Bugbot completed successfully with no findings that need human review, and no approval policy requires additional review.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}