From d64cfd47666fcd43c1f37d8388bb9b518163dd1e Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Mon, 5 Oct 2026 20:52:38 -0700 Subject: [PATCH] docs(upgrades): record 2026-10-05 app wave and removals UPGRADE_PLAN: execution log for the app wave (Postgres minors, pgbouncer, high-command-ui v0.29, MCP servers, Vector 0.58, ARC/MongoDB/OpenSearch and leftover removals); #12, #18, #20 marked done; grafana-mcp removed (gitops-mcp#18); TrueNAS key note kept consistent with #45. OPS_NOTES: 2026-10-05 known issues (Coder restart after switchover, orphaned graylog webhook, Vector CEF bug, image pins, state rm backups). Co-authored-by: Cursor --- CHANGELOG.md | 3 +++ docs/OPS_NOTES.md | 11 +++++++++++ docs/UPGRADE_PLAN.md | 46 ++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 56 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14766ce..41a9676 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Envoy Gateway** `v1.6.1` → **`v1.9.2`** and **Gateway API** CRDs `v1.4.1` → **`v1.6.1`**. The module applies the CRDs server-side and waits for them before the controller, re-runs when the script changes, and verifies the controller by its real label ([#40](https://github.com/DataKnifeAI/rancher-deploy/pull/40), [#42](https://github.com/DataKnifeAI/rancher-deploy/pull/42)) - **CloudNativePG** default `1.28.0` → **`1.30.1`** ([#41](https://github.com/DataKnifeAI/rancher-deploy/pull/41)) - [docs/UPGRADE_PLAN.md](docs/UPGRADE_PLAN.md): execution log for items #1–#6 of the 2026-10-04 wave (Rancher, democratic-csi, Envoy Gateway, CNPG, Grafana, rolling OS reboots of all 31 nodes); [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md): renewal results; [docs/OPS_NOTES.md](docs/OPS_NOTES.md): new known issues +- [docs/UPGRADE_PLAN.md](docs/UPGRADE_PLAN.md): execution log for the 2026-10-05 app wave (Postgres 18.6 / Harbor Postgres 16.15, pgbouncer 1.26.0, high-command-ui v0.29, proxmox-ve-mcp v0.43, unifi-network-mcp v0.12, Vector 0.58, removals); items #12, #18 and #20 marked done; [docs/OPS_NOTES.md](docs/OPS_NOTES.md): 2026-10-05 known issues ### Fixed - CloudNativePG resources no longer have destroy provisioners: a version bump replaced the resource and deleted the operator manifest, CRDs included, and with them every Cluster/Backup ([#41](https://github.com/DataKnifeAI/rancher-deploy/pull/41)) @@ -36,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Removed - **democratic-csi**: `democratic_csi_*` variables, the `deploy_democratic_csi_{nprd,prd,poc}_apps` resources (never in state; count was 0), the `democratic_csi_config` output, `scripts/install-democratic-csi.sh`, `scripts/generate-helm-values-from-tfvars.sh` and `docs/DEMOCRATIC_CSI_TRUENAS_SETUP.md`. Nothing on any cluster used the driver; TrueNAS CSI is the only storage driver. Remove the `democratic_csi_*` entries from your local `terraform.tfvars` +- **GitHub ARC, MongoDB Community operator, OpenSearch operator**: the 9 `deploy_{arc,mongodb_community_operator,opensearch_operator}_{nprd,prd,poc}_apps` resources, the `github_arc_controller_version`, `mongodb_operator_version` and `opensearch_operator_version` variables, the ARC/GitHub App helper scripts and `install-mongodb-community-operator.sh`. Uninstalled live from all clusters and dropped from state on 2026-10-05; nothing used them. Remove those three entries from your local `terraform.tfvars` ([#46](https://github.com/DataKnifeAI/rancher-deploy/pull/46)) - **democratic-csi** `democratic_csi_config` output dropped from Terraform state (state edit only, no apply; resources unchanged). Remaining TrueNAS-side cleanup is listed in [docs/TRUENAS_CSI_MIGRATION.md](docs/TRUENAS_CSI_MIGRATION.md#truenas-manual) ### Known issues @@ -43,6 +45,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - RKE2 leaf certs were renewed on 2026-10-04 and now expire 2027-10-05 (poc-apps-1 2027-09-19); renew again from 2027-06-07 per [docs/RKE2_CERT_ROTATION.md](docs/RKE2_CERT_ROTATION.md) - Do **not** revoke the TrueNAS API key that was in `democratic_csi_api_key`: it is the key TrueNAS CSI uses ([docs/TRUENAS_CSI_MIGRATION.md](docs/TRUENAS_CSI_MIGRATION.md#truenas-manual)) - TrueNAS CSI restarts; two prd CNPG replicas need re-cloning — see [docs/OPS_NOTES.md](docs/OPS_NOTES.md) +- Orphaned graylog `opensearch-bootstrap-password-webhook` (MutatingWebhookConfiguration, `failurePolicy: Fail`) and its RBAC on nprd-apps; delete them — see [docs/OPS_NOTES.md](docs/OPS_NOTES.md#known-issues-2026-10-05) ## [1.2.0-beta.1] - 2026-08-31 diff --git a/docs/OPS_NOTES.md b/docs/OPS_NOTES.md index c6e55d3..929140f 100644 --- a/docs/OPS_NOTES.md +++ b/docs/OPS_NOTES.md @@ -89,6 +89,15 @@ Example layout (mgmt `vmbr0`/`bond0`, storage jumbo `vmbr1`/`bond1`, aux `vmbr2` - **Terraform's Rancher API token is expired** (`config/.rancher-api-token` = `rancher_api_token` in tfvars, HTTP 401), and the registration modules read a hard-coded `/home/lee/git/rancher-deploy/config/.rancher-api-token`. Refresh before any `register_downstream_cluster` apply — [RANCHER_API_TOKEN_CREATION.md](RANCHER_API_TOKEN_CREATION.md). - **Break-glass kubeconfigs — resolved 2026-10-04.** Break-glass now lives in `~/.kube/-rke2.yaml` (RKE2 admin, straight to `https://.dataknife.net:6443`; `manager.dataknife.net` for rancher-manager). The old `~/.kube/.yaml` files on the main workstation are Rancher-proxied token kubeconfigs (`rancher-manager.yaml`'s token is expired); don't use them for break-glass. Terraform's `get_kubeconfig` will overwrite them with RKE2 admin kubeconfigs on the next full apply. See [CLUSTER_ACCESS_AND_SSO.md § Fallback](CLUSTER_ACCESS_AND_SSO.md#fallback-and-break-glass). +## Known issues (2026-10-05) + +- **Postgres images are pinned in git now.** Every CNPG Cluster sets `imageName`: coder, high-command and authentik on `ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie`, Harbor on `16.15-standard-trixie`. The high-command Pooler pins `pgbouncer:1.26.0` through its pod template. Change minors in the owning repo (gitops-dev, gitops-mcp, gitops-tools); CNPG rolls replicas first, then switches the primary over. The `system` flavor is deprecated: move the 18.x clusters to `standard` in a later window (Barman plugin backups don't need the in-image barman tools). See [UPGRADE_PLAN.md § Executed 2026-10-05](UPGRADE_PLAN.md#executed-2026-10-05-app-wave). +- **Coder restarts once about 5 minutes after a Postgres switchover.** Its pubsub watchdog times out ("pubsub Watchdog timed out") and the process exits; Kubernetes restarts it and it comes back in about 10 s. Expect it after every coder-postgres switchover (operator upgrades, drains, minors). +- **Orphaned graylog webhook on nprd-apps.** `MutatingWebhookConfiguration/opensearch-bootstrap-password-webhook` and `ClusterRole`/`ClusterRoleBinding` `opensearch-bootstrap-webhook-ca-bundle-injector` are left over from an old Fleet graylog release (created 2026-01-15). The webhook has `failurePolicy: Fail` and points at a service in `managed-graylog`, which no longer exists. It only selects namespaces labelled `name=managed-graylog`, so nothing is blocked today, but a namespace with that label would have every pod CREATE rejected. Delete all three objects. +- **Vector never parses UniFi CEF fields (nprd `vector-syslog`).** In the gitops-tools grafana overlay's VRL, `match!(message_text, r'CEF:.*')` returns a boolean, not the matched text, so `cef_message` becomes `"true"`/`"false"` and the `^CEF:` branch never runs. Syslog still reaches Loki, just without `device_vendor`, `signature_id`, `severity` and the other CEF labels. Fix with `parse_regex` (capture `CEF:.*`) and test with `vector vrl` before rolling. Present before the 0.58 upgrade. +- **high-command-ui runs blue/green.** Green (`v0.29`) is live behind `high-command-ui` Service; blue stays on `v0.23` as the rollback (switch the Service selector back). Bring blue to the current tag once green has soaked. Releases are applied with `kubectl apply` from the high-command-ui repo's `k8s/`, not by Fleet. +- **ARC, MongoDB and OpenSearch operators are gone** from all clusters (2026-10-05). Nothing used them. If you need one again, install it fresh at the current version; don't restore the old Terraform resources. + ## Secrets hygiene | Path | Purpose | @@ -102,3 +111,5 @@ Example layout (mgmt `vmbr0`/`bond0`, storage jumbo `vmbr1`/`bond1`, aux `vmbr2` | `state/terraform.tfstate*` | Local Terraform state (backend path `../state/`); holds secrets in plaintext (gitignored) | On the main workstation `state/terraform.tfstate*` were mode `0777` on 2026-10-04 (and `terraform/terraform.tfstate.*.backup` `0644`). Fix with `chmod 600 state/terraform.tfstate* terraform/terraform.tfstate.*.backup`; keep `terraform.tfvars` at `0600`. + +`terraform state rm` and `terraform state mv` write a full-state backup `terraform.tfstate..backup` into the current directory at mode `0644`. Move it into `~/terraform-state-backups/` (mode 600) or delete it right after. diff --git a/docs/UPGRADE_PLAN.md b/docs/UPGRADE_PLAN.md index f2e6ff1..d4cf4dc 100644 --- a/docs/UPGRADE_PLAN.md +++ b/docs/UPGRADE_PLAN.md @@ -2,7 +2,7 @@ Two parts: -1. **[Next wave (reviewed 2026-10-04)](#next-wave-reviewed-2026-10-04)**: what was live, what's behind, and a prioritized order. Items #1–#6 were applied on 2026-10-04 ([Executed 2026-10-04](#executed-2026-10-04-items-16)), along with the cert-manager work in [cert-manager v1.21.2 (done 2026-10-04)](#cert-manager-v1212-done-2026-10-04). Everything else is still plan only. +1. **[Next wave (reviewed 2026-10-04)](#next-wave-reviewed-2026-10-04)**: what was live, what's behind, and a prioritized order. Items #1–#6 were applied on 2026-10-04 ([Executed 2026-10-04](#executed-2026-10-04-items-16)), along with the cert-manager work in [cert-manager v1.21.2 (done 2026-10-04)](#cert-manager-v1212-done-2026-10-04). An app wave followed on 2026-10-05 ([Executed 2026-10-05](#executed-2026-10-05-app-wave)): Postgres minors, high-command-ui, two MCP servers, Vector, and removal of ARC, the MongoDB and OpenSearch operators and leftovers (#12, #18, #20). Everything else is still plan only. 2. **[History: 2026-07-31 → 2026-08-01 wave](#history-2026-07-31--2026-08-01-wave-rancher-213--215-rke2-134--136)**: the completed Rancher 2.13 → 2.15 / RKE2 1.34 → 1.36 runbook, kept as-is. Its gating, drain, and rollback procedures (§2, §5, §7) still apply to everything below. Ground rules carried over: back up Terraform state (`terraform state pull` → `~/terraform-state-backups/rancher-deploy/`, mode 600) before every apply; use `-refresh=false` and targeted plans, and apply only when the full plan contains exactly the intended resources; poc-apps first, then nprd-apps, then prd-apps, with the manager last unless noted; use the break-glass `~/.kube/-rke2.yaml` for drains. @@ -47,15 +47,15 @@ Priority: **P0** = security fix or EOL/unsupported on k8s 1.36, do next. **P1** | 9 | P1 | **Terraform providers / CLI** | rancher2 `3.4.0` → **`v15.x`** (`15.1.2`); proxmox `0.93.0` → `0.115.0`; kubernetes `2.38.0` → `3.3.0`; null `3.2.4` → `3.3.2`; local `2.6.1` → `2.9.1`; CLI `1.15.9` → `1.16.5` | rancher2 majors track the Rancher minor since v13 (v15 = Rancher 2.15); 3.x predates that. The proxmox provider jump is large: expect schema/plan diffs on all VMs. Fix the known refresh problems first (rancher-manager-1 refresh error, pve1/pve2 placement, Ubuntu image re-download), then upgrade one provider at a time and review `-refresh=false` plans. Commit `.terraform.lock.hcl`. Terraform's Rancher API token (`config/.rancher-api-token`) is expired and must be renewed before rancher2 resources can plan | `terraform/provider.tf`, lock file | M–L | | 10 | P1 | **kube-vip** | `v1.0.3` → `v1.2.4` | Two minors behind; every LoadBalancer VIP depends on it. poc first and watch for VIP flaps | `kube_vip_version` tfvars → `modules/kube-vip` | S–M | | 11 | P1 | **Monitoring stack** | kube-prometheus-stack `88.0.1` → `91.9.0`; Alloy image `v1.12.2` → `v1.20.1` (chart `1.13.0`); Loki `3.6.11` → `3.6.17` (or `3.7.8`, chart `7.3.0`) | kps is three majors behind: apply the prometheus-operator CRDs (`v0.94.x`) before the chart, one major at a time. The Alloy image is pinned well below its own chart's appVersion; drop the override. Loki patch first, 3.7 later | manual Helm + `gitops-tools` monitoring/grafana overlays | M | -| 12 | P1 | **ARC** (GitHub runners) | controller `0.13.1` → `0.15.0` | GitHub only supports the latest ARC; the controller and scale-set charts must move together, and CRDs need upgrading | `terraform/main.tf` (ARC resources) | S–M | +| 12 | P1 | **ARC** (GitHub runners) | controller `0.13.1` → **removed** (**done 2026-10-05**) | No runner scale sets existed on any cluster, so ARC was removed instead of upgraded: controller, CRDs (including the legacy `actions.summerwind.dev` ones on nprd/prd) and namespace on poc/nprd/prd, the Terraform resources and helper scripts ([#46](https://github.com/DataKnifeAI/rancher-deploy/pull/46)), and the unwired gitops-tools `github-runner` overlay ([gitops-tools#59](https://github.com/DataKnifeAI/gitops-tools/pull/59)). To bring it back, install the current `gha-runner-scale-set-controller` plus a scale set ([GITHUB_ARC_SETUP.md](GITHUB_ARC_SETUP.md)) | — | — | | 13 | P1 | **Harbor** (nprd) | `2.15.1` → `2.15.2` (chart `1.19.2`) | Patch release. The registry serves every cluster, so take a manual Barman backup of `harbor-postgresql` first | manual Helm + `gitops-tools` harbor overlay | S | | 14 | P1 | **Coder** (prd) | `2.35.3` → `2.35.8` now; `2.36.6` (stable) next | Patch within line first. The `gitops-dev` branch `chore/coder-2.32.0` is older than live; close it | manual Helm + `gitops-dev` overlay | S | | 15 | P2 | **cert-manager (manager)** | `v1.21.1` → `v1.21.2` | Apps clusters are done. Rancher 2.15 docs set no cert-manager cap (install with `crds.enabled=true`). Manager has no Gateway API CRDs, so **don't** set `config.gatewayAPI.enabled` there. Helm with `crds.enabled=true crds.keep=true`, before #1 in the same window. 1.21 stays supported until 1.23 (~early 2027) | manual Helm (`rancher_cluster/deploy-rancher.sh` only runs on install) | S | | 16 | P2 | **GitLab runner** (nprd) | chart `0.91.0` (19.2) → `0.93.0` (19.4) | Keep the runner within the GitLab server's supported skew | manual Helm + `gitops-tools` gitlab-runner overlay | S | | 17 | P2 | **TrueNAS CSI sidecars** | provisioner `5.0.1` → `6.3.0`, attacher `4.7.0` → `4.13.0`, resizer `1.12.0` → `2.3.0`, snapshotter `8.1.0` → `8.6.0`, registrar `2.12.0` → `2.18.0` | The driver (`v0.22`, fork of upstream `v1.3.0`, which is still upstream latest) is current; the sidecars are about two years old. Bump on poc and test PVC create/expand/snapshot | Terraform TrueNAS CSI resources (`truenas_csi_*`) | M | -| 18 | P2 | **Remove idle operators** | MongoDB community operator `0.13.0`; OpenSearch operator `2.8.0` | Neither manages anything: no `MongoDBCommunity` or `OpenSearchCluster` objects on any cluster. The MongoDB community operator repo is archived (successor `mongodb/mongodb-kubernetes` `1.13.0`). OpenSearch operator 3.0 is a breaking major, and ours runs a hand-patched kube-rbac-proxy image. Uninstall instead of upgrading | `terraform/main.tf` (`deploy_mongodb_community_operator_*`, OpenSearch resources) | S | +| 18 | P2 | **Remove idle operators** | MongoDB community operator `0.13.0`; OpenSearch operator `2.8.0` (**done 2026-10-05**, removed with ARC in [#46](https://github.com/DataKnifeAI/rancher-deploy/pull/46)) | Neither manages anything: no `MongoDBCommunity` or `OpenSearchCluster` objects on any cluster. The MongoDB community operator repo is archived (successor `mongodb/mongodb-kubernetes` `1.13.0`). OpenSearch operator 3.0 is a breaking major, and ours runs a hand-patched kube-rbac-proxy image. Uninstall instead of upgrading | `terraform/main.tf` (`deploy_mongodb_community_operator_*`, OpenSearch resources) | S | | 19 | P2 | **palworld-operator** (prd) | live digest `8036766f…` → tfvars digest `89efffac…` | Already in the Terraform drift. Confirm which digest is `v0.2.0-beta.1` and apply in a game-server maintenance window | `palworld_operator.tf` | S | -| 20 | P2 | **mcp-servers images** | `grafana-mcp` on `:latest` (upstream now `v2.0.0`, a major); others pinned (`high-command-mcp v0.12`, `proxmox-ve-mcp v0.41`, `unifi-network-mcp v0.10`, `unifi-protect-mcp v0.9`) | Pin grafana-mcp to an explicit version, and compare the others against each repo's CI tags | `gitops-mcp` | S | +| 20 | P2 | **mcp-servers images** | `grafana-mcp` on `:latest` (upstream now `v2.0.0`, a major); others pinned (`high-command-mcp v0.12`, `proxmox-ve-mcp v0.41`, `unifi-network-mcp v0.10`, `unifi-protect-mcp v0.9`) (**done 2026-10-05**: grafana-mcp removed in [gitops-mcp#18](https://github.com/DataKnifeAI/gitops-mcp/pull/18); `proxmox-ve-mcp v0.43`, `unifi-network-mcp v0.12`; high-command-mcp and unifi-protect-mcp were already on their latest tags) | Still open: unifi-network-mcp and unifi-protect-mcp build on Go 1.23 / Alpine 3.18, both EOL; rebuild on supported bases | `gitops-mcp` | S | | 21 | P2 | **Terraform state catch-up** | 5 to add | After the cert-manager applies, an untargeted `-refresh=false` plan shows only: `label_{poc,nprd,prd}_apps_truenas_topology` (labels `topology.truenas.io/pool=SAS` already on every node, so idempotent), `unattended_upgrades_nodes[0]` (already enabled/active on all nodes, so idempotent), and `deploy_palworld_operator_prd_apps[0]` (#19, a real change). Apply as separate targeted plans | `terraform/main.tf` | S | | — | — | Already current | Authentik `2026.8.3`, plugin-barman-cloud `v0.15.1`, TrueNAS CSI driver base `v1.3.0`, Rancher CLI `v2.15.2`, cert-manager apps `v1.21.2` | — | — | — | | — | — | Not checked | Proxmox VE version (no PVE API/SSH access used in this review): run `pveversion` on pve1/pve2 | — | — | — | @@ -104,6 +104,44 @@ In this order: Rancher, democratic-csi, Envoy Gateway, CloudNativePG, Grafana, n - Nodes may report NotReady for 10–20 s just after uncordon; they recover by themselves. - Pin floating tags before reboots: they re-pull on reschedule. Coder workspaces still run `archlinux:latest`. +### Executed 2026-10-05 (app wave) + +Pre-checks, backups and verification were done for each item; nothing was rolled back. Postgres clusters went one at a time (coder, high-command, authentik, Harbor last). Each started with an on-demand Barman backup `-pre-minor-upgrade-20261005` (all completed) and a `pg_dumpall` to `~/backups/cnpg/-20261005-.sql.gz` (mode 600; the Cluster spec is saved next to it). Each upgrade was an in-place minor: replicas first, then the primary by switchover. + +| Item | Before → after | What was done / verified | PRs | +|------|----------------|--------------------------|-----| +| coder-postgres (prd) | `18.1-system-trixie` → `18.6-system-trixie` | `imageName` now pinned in git. Switchover 5 → 4; replicas streaming; `SELECT version()` 18.6; WAL archiving OK | gitops-dev [#10](https://github.com/DataKnifeAI/gitops-dev/pull/10) | +| high-command-postgres + pooler (prd) | `18.1` → `18.6-system-trixie`; pgbouncer `1.25.2` → `1.26.0` | Switchover 3 → 4; Pooler 2/2 on `ghcr.io/cloudnative-pg/pgbouncer:1.26.0` (pinned through the Pooler pod template); API never restarted | gitops-mcp [#19](https://github.com/DataKnifeAI/gitops-mcp/pull/19) | +| authentik-postgres (prd) | `18.1` → `18.6-system-trixie` | Switchover 1 → 2; login page and `/-/health/ready/` 200; no pod restarts | gitops-tools [#56](https://github.com/DataKnifeAI/gitops-tools/pull/56) | +| harbor-postgresql (nprd) | `postgresql:16.2` (legacy Debian 11 image) → `16.15-standard-trixie` | Same major, so in place. The glibc jump (2.31 → 2.41) is safe here because every database uses the `C` collation and only `plpgsql` is installed. Barman plugin backups don't need the barman tools in the image, so `standard`. Switchover 4 → 2; 184 artifacts before and after; pulls through Harbor work | gitops-tools [#57](https://github.com/DataKnifeAI/gitops-tools/pull/57) | +| high-command-ui (prd) | `v0.23` → `v0.29` (blue/green) | Green deployment on `v0.29` checked through a port-forward, then the Service selector moved to green. Blue stays on `v0.23` as the rollback. cloudflared-tunnel pinned `v0.29` (cloudflared 2026.9.3, 4 tunnel connections). `hc.dataknife.ai` serves the new bundle; `/api/war/status`, `/api/campaigns`, `/api/planets` 200 | high-command-ui [#51](https://github.com/DataKnifeAI/high-command-ui/pull/51) | +| MCP servers (prd) | proxmox-ve-mcp `v0.41` → `v0.43`; unifi-network-mcp `v0.10` → `v0.12` | 3/3 Ready, no restarts. MCP `initialize` + `tools/list` over stdio inside a pod (the HTTP `/mcp` endpoint is a stub): 70 and 41 tools, same as before | gitops-mcp [#20](https://github.com/DataKnifeAI/gitops-mcp/pull/20) | +| Vector `vector-syslog` (nprd) | `0.51.0-alpine` → `0.58.0-alpine` | Config validated with the 0.58 image first. 0.55 moved the API from GraphQL to gRPC and rejects `api.playground`, so that key was removed (`/health` still serves the probes). VRL output identical on 0.51 and 0.58 for a sample event. Loki keeps receiving about 2,000 syslog events/min | gitops-tools [#58](https://github.com/DataKnifeAI/gitops-tools/pull/58) | +| ARC, MongoDB, OpenSearch operators (poc/nprd/prd) | ARC `0.13.1`, MongoDB `0.13.0`, OpenSearch `2.8.0` → removed | Zero custom resources for any of their CRDs on all four clusters, re-checked right before each cluster. `helm uninstall` of the three releases, then the leftover CRDs (ARC, legacy summerwind on nprd/prd, MongoDB) and the three namespaces. No CRDs, ClusterRoles, APIServices or webhooks left. Terraform: `terraform state rm` of the 9 `null_resource.deploy_*` addresses (168 → 159 in state); a targeted `-refresh=false` plan shows no changes | [#46](https://github.com/DataKnifeAI/rancher-deploy/pull/46), gitops-tools [#59](https://github.com/DataKnifeAI/gitops-tools/pull/59) | +| Orphaned HelmChart `managed-tools/harbor` (nprd) | Terminating since 2026-02-08 → gone | Its namespace no longer existed, so the helm-controller finalizer couldn't be patched. Recreated the namespace briefly, removed the finalizer, deleted the namespace. No uninstall job ran; the live `harbor/harbor` release stayed at revision 4 with the same pods | — | +| Leftover namespaces | → deleted | local `managed-tools` (stale `harbor-credentials` and an old wildcard TLS secret) and `mcp-servers` (empty); nprd `test-expansion` (empty); prd `managed-cicd` (only an unused `gitlab-runner-secret`; the runner lives on nprd) and `csi-test` (1Gi test PVC: truenas-csi deleted NFS share 198 and dataset `SAS/pvc-3ca9d9c4-0171-4f91-8db0-6076d68a67b5`). Not touched: dioptra and Satisfactory | — | + +**Impact.** Measured with an HTTP probe every ~2 s: +- coder: about 3 s of DB errors at the switchover. About 5 minutes later Coder exited on its own ("pubsub Watchdog timed out") and restarted: about 11 s of 502/503. +- high-command: `/api/health` timeouts for about 16 s at the pooler roll and switchover. +- authentik: 503s/timeouts for about 13 s at the switchover. +- Harbor and the MCP servers: nothing measurable. +- high-command-ui: a single 530 when cloudflared rolled. + +**Backups (mode 600).** `~/backups/cnpg/` (dumps + Cluster specs), Barman `*-pre-minor-upgrade-20261005`; `~/backups/high-command/pre-ui-v0.29-20261005-2002/`; `~/backups/vector/nprd-pre-0.58-20261005-2040/`; `~/backups/harbor-orphan-helmchart-20261005-2035/`; `~/backups/leftover-namespaces-20261005-2039/` (secrets and the csi-test PVC/PV); `~/backups/operators-removal-20261005//` (Helm release metadata, CRD definitions); Terraform state and tfvars under `~/terraform-state-backups/rancher-deploy/20261005-204227-pre-operator-removal/` and `20261005-204749-pre-state-rm-operators/`. + +**Learnings.** +- The archiver's `failed_count` goes up by one at every switchover: the new primary retries a `.history` file the old one had already archived. Harmless while `ContinuousArchiving` stays True. +- Expect Coder to restart once about 5 minutes after a Postgres switchover (pubsub watchdog). It recovers by itself. +- `terraform state rm` writes `terraform.tfstate..backup` into the working directory at mode 0644. Move or delete it right away; it holds the full state. +- RKE2 helm-controller HelmCharts whose namespace is gone can't be patched (NamespaceLifecycle). Recreating the namespace briefly is enough to clear the finalizer, and doesn't trigger an uninstall. + +**Still open from this wave.** +- **TrueNAS API keys (democratic-csi leftovers, see #6).** Don't revoke key ID 4: it's the key truenas-csi uses on nprd, prd and poc. Move truenas-csi to a new key first if you want it gone. Revoke the unused democratic-csi keys, IDs 1 and 2. +- Orphaned graylog webhook on nprd: `MutatingWebhookConfiguration/opensearch-bootstrap-password-webhook` plus `ClusterRole`/`ClusterRoleBinding` `opensearch-bootstrap-webhook-ca-bundle-injector`, left by an old Fleet graylog release. It has `failurePolicy: Fail` but only selects namespaces labelled `name=managed-graylog`, which no longer exist. Delete it ([OPS_NOTES.md](OPS_NOTES.md#known-issues-2026-10-05)). +- Vector's CEF parsing never runs (pre-existing VRL bug, see OPS_NOTES). +- high-command-ui: move blue to `v0.29` after a soak. `v0.31` exists (manifest-only change). The image still builds on `node:20-alpine` (Node 20 is EOL); rebase on Node 22/24. + ### cert-manager v1.21.2 (done 2026-10-04) [PR #36](https://github.com/DataKnifeAI/rancher-deploy/pull/36).