Skip to content

docs(upgrades): record 2026-10-04 wave and cert renewal #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
surrealwolf merged 1 commit into from
Oct 5, 2026
Merged

docs(upgrades): record 2026-10-04 wave and cert renewal #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- [docs/UPGRADE_PLAN.md](docs/UPGRADE_PLAN.md): prioritized next upgrade wave (reviewed 2026-10-04: live vs latest for Rancher, RKE2, Envoy Gateway, CNPG, OS, monitoring, apps, Terraform providers) and the cert-manager v1.21.2 rollout log; the completed 2.13→2.15 / 1.34→1.36 runbook is kept as history
- **cert-manager** pin `v1.19.2` → **`v1.21.2`** (latest 1.21 patch; supports Kubernetes 1.33–1.36). Live clusters were already on `v1.21.1` via manual Helm (2026-07-31); Terraform now owns the version again
- cert-manager module: Helm values `crds.enabled=true,crds.keep=true` (replaces deprecated `installCRDs`) and `config.gatewayAPI.enabled=true` for the gateway-shim (replaces `--controllers=*,gateway-shim`, which has not started the shim since cert-manager 1.15)
- **Rancher** pin `v2.15.0` → **`v2.15.2`** (live since 2026-10-04)
- **Envoy Gateway** `v1.6.1` → **`v1.9.2`** and **Gateway API** CRDs `v1.4.1` → **`v1.6.1`**. The module applies the CRDs server-side and waits for them before the controller, re-runs when the script changes, and verifies the controller by its real label ([#40](https://github.com/DataKnifeAI/rancher-deploy/pull/40), [#42](https://github.com/DataKnifeAI/rancher-deploy/pull/42))
- **CloudNativePG** default `1.28.0` → **`1.30.1`** ([#41](https://github.com/DataKnifeAI/rancher-deploy/pull/41))
- [docs/UPGRADE_PLAN.md](docs/UPGRADE_PLAN.md): execution log for items #1–#6 of the 2026-10-04 wave (Rancher, democratic-csi, Envoy Gateway, CNPG, Grafana, rolling OS reboots of all 31 nodes); [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md): renewal results; [docs/OPS_NOTES.md](docs/OPS_NOTES.md): new known issues

### Fixed
- CloudNativePG resources no longer have destroy provisioners: a version bump replaced the resource and deleted the operator manifest, CRDs included, and with them every Cluster/Backup ([#41](https://github.com/DataKnifeAI/rancher-deploy/pull/41))
- Envoy Gateway module no longer deletes Gateway API CRDs on a version mismatch ([#40](https://github.com/DataKnifeAI/rancher-deploy/pull/40))
- cert-manager module no longer deletes the cert-manager CRDs/namespace (and with them every Certificate/Issuer) when it cannot see the Helm release; it fails unless `cleanup_unmanaged_install = true`. Release detection uses `helm status` instead of grepping `helm list`
- Broken doc links (`TERRAFORM_VARIABLES.md`) and an unterminated code fence in `RANCHER_DOWNSTREAM_MANAGEMENT.md`
- Docs no longer claim `~/.kube/<cluster>.yaml` are RKE2 admin kubeconfigs (they were overwritten by Rancher-proxied ones); break-glass is `~/.kube/<cluster>-rke2.yaml`
Expand All @@ -34,7 +40,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Known issues
- Terraform Rancher API token expired; registration modules read a hard-coded `/home/lee/git/rancher-deploy/config/.rancher-api-token`
- RKE2 leaf certs on all control-plane nodes expire 2027-01-08 (poc-apps 2027-01-15) — rotate per [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md)
- RKE2 leaf certs were renewed on 2026-10-04 and now expire 2027-10-05 (poc-apps-1 2027-09-19); renew again from 2027-06-07 per [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md)
- Do **not** revoke the TrueNAS API key that was in `democratic_csi_api_key`: it is the key TrueNAS CSI uses ([docs/TRUENAS_CSI_MIGRATION.md](docs/TRUENAS_CSI_MIGRATION.md#truenas-manual))
- TrueNAS CSI restarts; two prd CNPG replicas need re-cloning — see [docs/OPS_NOTES.md](docs/OPS_NOTES.md)

## [1.2.0-beta.1] - 2026-08-31
Expand Down
14 changes: 7 additions & 7 deletions docs/CLUSTER_ACCESS_AND_SSO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

How people and tools log in to Rancher and the RKE2 clusters, and what to do when that breaks.

**Last verified:** 2026-10-04 — Rancher `v2.15.0`, RKE2 `v1.36.2+rke2r1` on all clusters, Rancher CLI `v2.15.2`, Authentik `2026.8.3`.
**Last verified:** 2026-10-04 — Rancher `v2.15.2`, RKE2 `v1.36.2+rke2r1` on all clusters, Rancher CLI `v2.15.2`, Authentik `2026.8.3`.

Kubeconfigs no longer embed long-lived Rancher tokens. `kubectl` calls the Rancher CLI (`rancher token`) as an exec credential plugin, the CLI logs you in through Authentik (SAML) and caches a token. Break-glass paths do not depend on Authentik.

Expand Down Expand Up @@ -250,10 +250,10 @@ Use the first layer that works:

| File | Cluster / user / context | Server | Client cert expires |
|------|--------------------------|--------|---------------------|
| `~/.kube/rancher-manager-rke2.yaml` | `rancher-manager-rke2` | `https://manager.dataknife.net:6443` | 2027-01-08 |
| `~/.kube/nprd-apps-rke2.yaml` | `nprd-apps-rke2` | `https://nprd-apps.dataknife.net:6443` | 2027-01-08 |
| `~/.kube/prd-apps-rke2.yaml` | `prd-apps-rke2` | `https://prd-apps.dataknife.net:6443` | 2027-01-08 |
| `~/.kube/poc-apps-rke2.yaml` | `poc-apps-rke2` | `https://poc-apps.dataknife.net:6443` | 2027-09-19 (pulled from poc-apps-1, already renewed) |
| `~/.kube/rancher-manager-rke2.yaml` | `rancher-manager-rke2` | `https://manager.dataknife.net:6443` | 2027-10-05 (re-fetched 2026-10-04 from rancher-manager-2) |
| `~/.kube/nprd-apps-rke2.yaml` | `nprd-apps-rke2` | `https://nprd-apps.dataknife.net:6443` | 2027-10-05 (re-fetched 2026-10-04 from nprd-apps-2) |
| `~/.kube/prd-apps-rke2.yaml` | `prd-apps-rke2` | `https://prd-apps.dataknife.net:6443` | 2027-10-05 (re-fetched 2026-10-04 from prd-apps-1) |
| `~/.kube/poc-apps-rke2.yaml` | `poc-apps-rke2` | `https://poc-apps.dataknife.net:6443` | 2027-10-05 (re-fetched 2026-10-04 from poc-apps-2) |

The `<cluster>.dataknife.net` names are round-robin DNS over the three control-plane nodes and are in every node's apiserver cert SANs (`tls-san`), so the files survive the loss of one server. There is no control-plane VIP: kube-vip runs with `cp_enable=false` (Service VIPs only). If DNS is down, point `server` at a control-plane IP (each node's own IP is in its SANs) — that is a single-node dependency.

Expand Down Expand Up @@ -329,8 +329,8 @@ The script logs in through the local provider, so it keeps working with SAML ena
| `rancher.dataknife.net` (`cattle-system/tls-rancher-ingress`, cert-manager, LE) | 2026-11-15 | Auto-renews (~2026-10-16) | Rancher UI/API TLS errors; CLI login fails |
| Authentik SAML signing cert (`authentik Self-signed Certificate`) | **2027-10-04** | Manual: create a new cert in Authentik, set it on the `Rancher` provider, re-download metadata, paste it into Rancher's SAML config (re-supply the SP key) and re-enable | **All SSO logins fail** |
| Rancher SP cert (`~/.config/rancher-saml/sp.crt`) | 2036-10-01 | Regenerate (above) and re-upload | SSO logins fail |
| RKE2 leaf certs (apiserver `:6443` serving, `client-admin`, `client-kube-apiserver`, `auth-proxy`, etcd, controller-manager, scheduler, supervisor) on control-plane nodes | **2027-01-08** (manager, nprd, prd); **2027-01-15** (poc-apps-2/-3; poc-apps-1 already renewed to 2027-09-19) | Not automatic unless rke2-server restarts within 120 days of expiry. Planned rolling renewal: [RKE2_CERT_ROTATION.md](RKE2_CERT_ROTATION.md) | Apiservers stop serving / components can't authenticate — cluster down |
| RKE2 admin client certs (`system:admin`) in `~/.kube/<cluster>-rke2.yaml` and the gitops-core `cert-sync-kubeconfig` Secret | 2027-01-08 (poc: 2027-09-19) | Re-fetch `rke2.yaml` after each RKE2 rotation ([fetch](#fallback-and-break-glass)) and re-run `gitops-core/scripts/create-cert-sync-kubeconfig-secret.sh` | Break-glass access fails; the `cert-sync` Job fails (it now exits non-zero) and downstream wildcard copies go stale |
| RKE2 leaf certs (apiserver `:6443` serving, `client-admin`, `client-kube-apiserver`, `auth-proxy`, etcd, controller-manager, scheduler, supervisor) on control-plane nodes; agent certs on all nodes | **2027-10-05** (renewed 2026-10-04 by the rolling reboots); poc-apps-1 server certs **2027-09-19** | Renewed automatically only when rke2 restarts within 120 days of expiry (from 2027-05-22 on poc-apps-1, 2027-06-07 elsewhere): plan a rolling reboot/RKE2 upgrade by 2027-09-01, or `rke2 certificate rotate` ([RKE2_CERT_ROTATION.md](RKE2_CERT_ROTATION.md)) | Apiservers stop serving / components can't authenticate — cluster down |
| RKE2 admin client certs (`system:admin`) in `~/.kube/<cluster>-rke2.yaml` and the gitops-core `cert-sync-kubeconfig` Secret | 2027-10-05 (all four, re-fetched 2026-10-04) | Re-fetch `rke2.yaml` after each RKE2 rotation ([fetch](#fallback-and-break-glass)) and re-run `gitops-core/scripts/create-cert-sync-kubeconfig-secret.sh` | Break-glass access fails; the `cert-sync` Job fails (it now exits non-zero) and downstream wildcard copies go stale |
| Rancher API / kubeconfig tokens | ≤ 90 days from issue | Re-login (humans) / recreate (automation, `config/.rancher-api-token`) | 401s; exec plugin asks for login |

Spot checks:
Expand Down
Loading
Loading