Skip to content

docs(upgrades): record 2026-10-05 app wave and removals #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
surrealwolf merged 1 commit into from
Oct 6, 2026
Merged

docs(upgrades): record 2026-10-05 app wave and removals #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Envoy Gateway** `v1.6.1` → **`v1.9.2`** and **Gateway API** CRDs `v1.4.1` → **`v1.6.1`**. The module applies the CRDs server-side and waits for them before the controller, re-runs when the script changes, and verifies the controller by its real label ([#40](https://github.com/DataKnifeAI/rancher-deploy/pull/40), [#42](https://github.com/DataKnifeAI/rancher-deploy/pull/42))
- **CloudNativePG** default `1.28.0` → **`1.30.1`** ([#41](https://github.com/DataKnifeAI/rancher-deploy/pull/41))
- [docs/UPGRADE_PLAN.md](docs/UPGRADE_PLAN.md): execution log for items #1–#6 of the 2026-10-04 wave (Rancher, democratic-csi, Envoy Gateway, CNPG, Grafana, rolling OS reboots of all 31 nodes); [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md): renewal results; [docs/OPS_NOTES.md](docs/OPS_NOTES.md): new known issues
- [docs/UPGRADE_PLAN.md](docs/UPGRADE_PLAN.md): execution log for the 2026-10-05 app wave (Postgres 18.6 / Harbor Postgres 16.15, pgbouncer 1.26.0, high-command-ui v0.29, proxmox-ve-mcp v0.43, unifi-network-mcp v0.12, Vector 0.58, removals); items #12, #18 and #20 marked done; [docs/OPS_NOTES.md](docs/OPS_NOTES.md): 2026-10-05 known issues

### Fixed
- CloudNativePG resources no longer have destroy provisioners: a version bump replaced the resource and deleted the operator manifest, CRDs included, and with them every Cluster/Backup ([#41](https://github.com/DataKnifeAI/rancher-deploy/pull/41))
Expand All @@ -36,13 +37,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Removed
- **democratic-csi**: `democratic_csi_*` variables, the `deploy_democratic_csi_{nprd,prd,poc}_apps` resources (never in state; count was 0), the `democratic_csi_config` output, `scripts/install-democratic-csi.sh`, `scripts/generate-helm-values-from-tfvars.sh` and `docs/DEMOCRATIC_CSI_TRUENAS_SETUP.md`. Nothing on any cluster used the driver; TrueNAS CSI is the only storage driver. Remove the `democratic_csi_*` entries from your local `terraform.tfvars`
- **GitHub ARC, MongoDB Community operator, OpenSearch operator**: the 9 `deploy_{arc,mongodb_community_operator,opensearch_operator}_{nprd,prd,poc}_apps` resources, the `github_arc_controller_version`, `mongodb_operator_version` and `opensearch_operator_version` variables, the ARC/GitHub App helper scripts and `install-mongodb-community-operator.sh`. Uninstalled live from all clusters and dropped from state on 2026-10-05; nothing used them. Remove those three entries from your local `terraform.tfvars` ([#46](https://github.com/DataKnifeAI/rancher-deploy/pull/46))
- **democratic-csi** `democratic_csi_config` output dropped from Terraform state (state edit only, no apply; resources unchanged). Remaining TrueNAS-side cleanup is listed in [docs/TRUENAS_CSI_MIGRATION.md](docs/TRUENAS_CSI_MIGRATION.md#truenas-manual)

### Known issues
- Terraform Rancher API token expired; registration modules read a hard-coded `/home/lee/git/rancher-deploy/config/.rancher-api-token`
- RKE2 leaf certs were renewed on 2026-10-04 and now expire 2027-10-05 (poc-apps-1 2027-09-19); renew again from 2027-06-07 per [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md)
- Do **not** revoke the TrueNAS API key that was in `democratic_csi_api_key`: it is the key TrueNAS CSI uses ([docs/TRUENAS_CSI_MIGRATION.md](docs/TRUENAS_CSI_MIGRATION.md#truenas-manual))
- TrueNAS CSI restarts; two prd CNPG replicas need re-cloning — see [docs/OPS_NOTES.md](docs/OPS_NOTES.md)
- Orphaned graylog `opensearch-bootstrap-password-webhook` (MutatingWebhookConfiguration, `failurePolicy: Fail`) and its RBAC on nprd-apps; delete them — see [docs/OPS_NOTES.md](docs/OPS_NOTES.md#known-issues-2026-10-05)

## [1.2.0-beta.1] - 2026-08-31

Expand Down
11 changes: 11 additions & 0 deletions docs/OPS_NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,15 @@ Example layout (mgmt `vmbr0`/`bond0`, storage jumbo `vmbr1`/`bond1`, aux `vmbr2`
- **Terraform's Rancher API token is expired** (`config/.rancher-api-token` = `rancher_api_token` in tfvars, HTTP 401), and the registration modules read a hard-coded `/home/lee/git/rancher-deploy/config/.rancher-api-token`. Refresh before any `register_downstream_cluster` apply — [RANCHER_API_TOKEN_CREATION.md](RANCHER_API_TOKEN_CREATION.md).
- **Break-glass kubeconfigs — resolved 2026-10-04.** Break-glass now lives in `~/.kube/<cluster>-rke2.yaml` (RKE2 admin, straight to `https://<cluster>.dataknife.net:6443`; `manager.dataknife.net` for rancher-manager). The old `~/.kube/<cluster>.yaml` files on the main workstation are Rancher-proxied token kubeconfigs (`rancher-manager.yaml`'s token is expired); don't use them for break-glass. Terraform's `get_kubeconfig` will overwrite them with RKE2 admin kubeconfigs on the next full apply. See [CLUSTER_ACCESS_AND_SSO.md § Fallback](CLUSTER_ACCESS_AND_SSO.md#fallback-and-break-glass).

## Known issues (2026-10-05)

- **Postgres images are pinned in git now.** Every CNPG Cluster sets `imageName`: coder, high-command and authentik on `ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie`, Harbor on `16.15-standard-trixie`. The high-command Pooler pins `pgbouncer:1.26.0` through its pod template. Change minors in the owning repo (gitops-dev, gitops-mcp, gitops-tools); CNPG rolls replicas first, then switches the primary over. The `system` flavor is deprecated: move the 18.x clusters to `standard` in a later window (Barman plugin backups don't need the in-image barman tools). See [UPGRADE_PLAN.md § Executed 2026-10-05](UPGRADE_PLAN.md#executed-2026-10-05-app-wave).
- **Coder restarts once about 5 minutes after a Postgres switchover.** Its pubsub watchdog times out ("pubsub Watchdog timed out") and the process exits; Kubernetes restarts it and it comes back in about 10 s. Expect it after every coder-postgres switchover (operator upgrades, drains, minors).
- **Orphaned graylog webhook on nprd-apps.** `MutatingWebhookConfiguration/opensearch-bootstrap-password-webhook` and `ClusterRole`/`ClusterRoleBinding` `opensearch-bootstrap-webhook-ca-bundle-injector` are left over from an old Fleet graylog release (created 2026-01-15). The webhook has `failurePolicy: Fail` and points at a service in `managed-graylog`, which no longer exists. It only selects namespaces labelled `name=managed-graylog`, so nothing is blocked today, but a namespace with that label would have every pod CREATE rejected. Delete all three objects.
- **Vector never parses UniFi CEF fields (nprd `vector-syslog`).** In the gitops-tools grafana overlay's VRL, `match!(message_text, r'CEF:.*')` returns a boolean, not the matched text, so `cef_message` becomes `"true"`/`"false"` and the `^CEF:` branch never runs. Syslog still reaches Loki, just without `device_vendor`, `signature_id`, `severity` and the other CEF labels. Fix with `parse_regex` (capture `CEF:.*`) and test with `vector vrl` before rolling. Present before the 0.58 upgrade.
- **high-command-ui runs blue/green.** Green (`v0.29`) is live behind `high-command-ui` Service; blue stays on `v0.23` as the rollback (switch the Service selector back). Bring blue to the current tag once green has soaked. Releases are applied with `kubectl apply` from the high-command-ui repo's `k8s/`, not by Fleet.
- **ARC, MongoDB and OpenSearch operators are gone** from all clusters (2026-10-05). Nothing used them. If you need one again, install it fresh at the current version; don't restore the old Terraform resources.

## Secrets hygiene

| Path | Purpose |
Expand All @@ -102,3 +111,5 @@ Example layout (mgmt `vmbr0`/`bond0`, storage jumbo `vmbr1`/`bond1`, aux `vmbr2`
| `state/terraform.tfstate*` | Local Terraform state (backend path `../state/`); holds secrets in plaintext (gitignored) |

On the main workstation `state/terraform.tfstate*` were mode `0777` on 2026-10-04 (and `terraform/terraform.tfstate.*.backup` `0644`). Fix with `chmod 600 state/terraform.tfstate* terraform/terraform.tfstate.*.backup`; keep `terraform.tfvars` at `0600`.

`terraform state rm` and `terraform state mv` write a full-state backup `terraform.tfstate.<epoch>.backup` into the current directory at mode `0644`. Move it into `~/terraform-state-backups/` (mode 600) or delete it right after.
Loading
Loading