Skip to content

docs(ops): point break-glass at <cluster>-rke2.yaml; split CNPG bucket gap #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
surrealwolf merged 1 commit into from
Oct 5, 2026
Merged

docs(ops): point break-glass at <cluster>-rke2.yaml; split CNPG bucket gap #37

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/OPS_NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Short operational truths that do not belong in the README.
- Rancher `v2.15.0` at `https://rancher.dataknife.net`; auth providers **local** + **Keycloak (SAML)** pointed at Authentik (`https://auth.dataknife.net`, on prd-apps). Full guide: [CLUSTER_ACCESS_AND_SSO.md](CLUSTER_ACCESS_AND_SSO.md).
- kubectl: `./scripts/setup-rancher-kubeconfig.sh --install-cli --merge --auth-provider keyCloakProvider --break-glass`, then `kubectl --context <local|nprd-apps|prd-apps|poc-apps>`. Kubeconfigs embed no tokens; `rancher token` caches one (≤ 90 days) in `~/.rancher/cli2.json`.
- SAML site access is **`required`** by design: only principals in `allowedPrincipalIds` may log in via SSO (currently `keycloak_user://4` = akadmin). Allow people via an Authentik group, `keycloak_group://<exact group name>`; single users are `keycloak_user://<Authentik numeric pk>`, not the username. Local logins are unaffected. See [CLUSTER_ACCESS_AND_SSO.md § Site access](CLUSTER_ACCESS_AND_SSO.md#site-access-required).
- Break-glass: `<cluster>-local` contexts (Rancher local user), UI **Log in with Local User**, RKE2 admin kubeconfigs `~/.kube/<cluster>.yaml` (independent of Rancher/Authentik), Authentik recovery key.
- Break-glass: `<cluster>-local` contexts (Rancher local user), UI **Log in with Local User**, RKE2 admin kubeconfigs `~/.kube/<cluster>-rke2.yaml` (independent of Rancher/Authentik; client certs expire 2027-01-08, poc-apps 2027-09-19 — re-fetch after RKE2 cert rotation per [RKE2_CERT_ROTATION.md](RKE2_CERT_ROTATION.md)), Authentik recovery key.
- Global settings: `kubeconfig-generate-token=false`, `kubeconfig-default-token-ttl-minutes=129600`, `auth-token-max-ttl-minutes=129600` (default) — so **every** Rancher API token, including Terraform's `ttl: 0` one, expires in 90 days.
- Unattended jobs: use a dedicated scoped API token or RKE2 admin kubeconfigs, never a personal SSO token cache.

Expand Down Expand Up @@ -73,11 +73,12 @@ Example layout (mgmt `vmbr0`/`bond0`, storage jumbo `vmbr1`/`bond1`, aux `vmbr2`

- **TrueNAS CSI pods restart often on all app clusters.** The TrueNAS API (middlewared, `192.168.9.10`) freezes for ~20–60 s about every 7 minutes; the driver's liveness probe treats the lost API connection as fatal. A driver fix is in progress in the `truenas-csi` fork. Restarts are noisy but mounts recover; don't chase them per pod.
- **CNPG replicas stuck on timeline divergence (prd-apps) — fixed 2026-10-04.** `coder-postgres-1` and `high-command-postgres-1` were re-cloned with `kubectl cnpg destroy <cluster> <n> -n <ns>` (now `-6` and `-4`; both clusters 3/3). coder's dead slot had pinned about 115 GB of WAL by LSN (3.8G on disk thanks to ZFS compression). pg_wal is back to about 20M. Every CNPG cluster now sets `max_slot_wal_keep_size` (1–4GB, sized to the volume). Rebuild any future broken replica the same way. Use a `kubectl-cnpg` plugin that matches the operator version (1.28.x).
- **CNPG backups (since 2026-10-04).** All four clusters (prd-apps: coder, high-command, authentik; nprd-apps: harbor) archive WAL and take daily base backups through the Barman Cloud plugin (gitops-core `cnpg-barman-cloud`) to `s3://rke2-backups/cnpg/<k8s-cluster>/<db>/`. They use the rustfs S3 API on port **30292**; 30293 is the console. Retention is 14 days. Restore and checks are documented in each repo's README. Gaps: (1) backups share the Rancher-backup bucket and key, because that key can't create buckets; a dedicated `cnpg-backups` bucket and scoped key would be better. (2) Adding or changing the plugin rolls each cluster with an in-place primary restart (`primaryUpdateMethod: restart`). Authentik SSO was down about 4 min (its server pods were restarted by the liveness probe); Coder and high-command lost the DB for under a minute.
- **CNPG backups (since 2026-10-04).** All four clusters (prd-apps: coder, high-command, authentik; nprd-apps: harbor) archive WAL and take daily base backups through the Barman Cloud plugin (gitops-core `cnpg-barman-cloud`) to `s3://rke2-backups/cnpg/<k8s-cluster>/<db>/`. They use the rustfs S3 API on port **30292**; 30293 is the console. Retention is 14 days. Restore and checks are documented in each repo's README. Adding or changing the plugin rolls each cluster with an in-place primary restart (`primaryUpdateMethod: restart`). Authentik SSO was down about 4 min (its server pods were restarted by the liveness probe); Coder and high-command lost the DB for under a minute.
- **Open: CNPG backups share the `rke2-backups` bucket and key with Rancher backups.** The existing rustfs key can't create buckets. Create a dedicated `cnpg-backups` bucket and a key scoped to it in the rustfs console, then update the `cnpg-backup-rustfs` Secrets (coder, high-command, authentik on prd-apps; harbor on nprd-apps) and each ObjectStore's `destinationPath`.
- **high-command-postgres is now Fleet-managed** (gitops-mcp `k8s/high-command/`, adopted with `helm.takeOwnership`). Helm two-way-patches custom resources, so adoption only adds ownership metadata. Spec and Fleet labels were synced to git by hand once. Keep the manifest identical to live, including CNPG-defaulted fields such as `plugins[].enabled`, or Fleet's drift correction loops rollbacks.
- **poc-apps wildcard cert expired (2026-04-18).** `cert-manager/cert-sync` (gitops-core, manager) fails for poc-apps: its kubeconfig entry uses the Rancher proxy URL with a deleted token, and the job image fails x509 verification. nprd/prd copies are current. Fix in gitops-core (RKE2 admin kubeconfig entry for poc-apps). The other entries' client certs expire 2027-01-08. See [CLUSTER_ACCESS_AND_SSO.md § Certificate calendar](CLUSTER_ACCESS_AND_SSO.md#certificate-and-expiry-calendar).
- **Terraform's Rancher API token is expired** (`config/.rancher-api-token` = `rancher_api_token` in tfvars, HTTP 401), and the registration modules read a hard-coded `/home/lee/git/rancher-deploy/config/.rancher-api-token`. Refresh before any `register_downstream_cluster` apply — [RANCHER_API_TOKEN_CREATION.md](RANCHER_API_TOKEN_CREATION.md).
- **`~/.kube/<cluster>.yaml` on the main workstation are Rancher-proxied token kubeconfigs**, not the RKE2 admin kubeconfigs Terraform writes (`rancher-manager.yaml`'s token is already expired). Re-pull them before relying on them for break-glass or Terraform add-on steps — see [CLUSTER_ACCESS_AND_SSO.md § Fallback](CLUSTER_ACCESS_AND_SSO.md#fallback-and-break-glass).
- **Break-glass kubeconfigs — resolved 2026-10-04.** Break-glass now lives in `~/.kube/<cluster>-rke2.yaml` (RKE2 admin, straight to `https://<cluster>.dataknife.net:6443`; `manager.dataknife.net` for rancher-manager). The old `~/.kube/<cluster>.yaml` files on the main workstation are Rancher-proxied token kubeconfigs (`rancher-manager.yaml`'s token is expired); don't use them for break-glass. Terraform's `get_kubeconfig` will overwrite them with RKE2 admin kubeconfigs on the next full apply. See [CLUSTER_ACCESS_AND_SSO.md § Fallback](CLUSTER_ACCESS_AND_SSO.md#fallback-and-break-glass).

## Secrets hygiene

Expand Down
Loading