From 6f0446d0182a94a20cf35d55029f90b34cd971f8 Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 4 Oct 2026 17:25:42 -0700 Subject: [PATCH] docs(ops): point break-glass at -rke2.yaml; split CNPG bucket gap Break-glass RKE2 admin kubeconfigs moved to ~/.kube/-rke2.yaml (#34); mark the Rancher-proxied .yaml issue resolved and track the shared rke2-backups bucket/key for CNPG as an open item. Co-authored-by: Cursor --- docs/OPS_NOTES.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/OPS_NOTES.md b/docs/OPS_NOTES.md index 16a0f0b..bb0e402 100644 --- a/docs/OPS_NOTES.md +++ b/docs/OPS_NOTES.md @@ -7,7 +7,7 @@ Short operational truths that do not belong in the README. - Rancher `v2.15.0` at `https://rancher.dataknife.net`; auth providers **local** + **Keycloak (SAML)** pointed at Authentik (`https://auth.dataknife.net`, on prd-apps). Full guide: [CLUSTER_ACCESS_AND_SSO.md](CLUSTER_ACCESS_AND_SSO.md). - kubectl: `./scripts/setup-rancher-kubeconfig.sh --install-cli --merge --auth-provider keyCloakProvider --break-glass`, then `kubectl --context `. Kubeconfigs embed no tokens; `rancher token` caches one (≤ 90 days) in `~/.rancher/cli2.json`. - SAML site access is **`required`** by design: only principals in `allowedPrincipalIds` may log in via SSO (currently `keycloak_user://4` = akadmin). Allow people via an Authentik group, `keycloak_group://`; single users are `keycloak_user://`, not the username. Local logins are unaffected. See [CLUSTER_ACCESS_AND_SSO.md § Site access](CLUSTER_ACCESS_AND_SSO.md#site-access-required). -- Break-glass: `-local` contexts (Rancher local user), UI **Log in with Local User**, RKE2 admin kubeconfigs `~/.kube/.yaml` (independent of Rancher/Authentik), Authentik recovery key. +- Break-glass: `-local` contexts (Rancher local user), UI **Log in with Local User**, RKE2 admin kubeconfigs `~/.kube/-rke2.yaml` (independent of Rancher/Authentik; client certs expire 2027-01-08, poc-apps 2027-09-19 — re-fetch after RKE2 cert rotation per [RKE2_CERT_ROTATION.md](RKE2_CERT_ROTATION.md)), Authentik recovery key. - Global settings: `kubeconfig-generate-token=false`, `kubeconfig-default-token-ttl-minutes=129600`, `auth-token-max-ttl-minutes=129600` (default) — so **every** Rancher API token, including Terraform's `ttl: 0` one, expires in 90 days. - Unattended jobs: use a dedicated scoped API token or RKE2 admin kubeconfigs, never a personal SSO token cache. @@ -73,11 +73,12 @@ Example layout (mgmt `vmbr0`/`bond0`, storage jumbo `vmbr1`/`bond1`, aux `vmbr2` - **TrueNAS CSI pods restart often on all app clusters.** The TrueNAS API (middlewared, `192.168.9.10`) freezes for ~20–60 s about every 7 minutes; the driver's liveness probe treats the lost API connection as fatal. A driver fix is in progress in the `truenas-csi` fork. Restarts are noisy but mounts recover; don't chase them per pod. - **CNPG replicas stuck on timeline divergence (prd-apps) — fixed 2026-10-04.** `coder-postgres-1` and `high-command-postgres-1` were re-cloned with `kubectl cnpg destroy -n ` (now `-6` and `-4`; both clusters 3/3). coder's dead slot had pinned about 115 GB of WAL by LSN (3.8G on disk thanks to ZFS compression). pg_wal is back to about 20M. Every CNPG cluster now sets `max_slot_wal_keep_size` (1–4GB, sized to the volume). Rebuild any future broken replica the same way. Use a `kubectl-cnpg` plugin that matches the operator version (1.28.x). -- **CNPG backups (since 2026-10-04).** All four clusters (prd-apps: coder, high-command, authentik; nprd-apps: harbor) archive WAL and take daily base backups through the Barman Cloud plugin (gitops-core `cnpg-barman-cloud`) to `s3://rke2-backups/cnpg///`. They use the rustfs S3 API on port **30292**; 30293 is the console. Retention is 14 days. Restore and checks are documented in each repo's README. Gaps: (1) backups share the Rancher-backup bucket and key, because that key can't create buckets; a dedicated `cnpg-backups` bucket and scoped key would be better. (2) Adding or changing the plugin rolls each cluster with an in-place primary restart (`primaryUpdateMethod: restart`). Authentik SSO was down about 4 min (its server pods were restarted by the liveness probe); Coder and high-command lost the DB for under a minute. +- **CNPG backups (since 2026-10-04).** All four clusters (prd-apps: coder, high-command, authentik; nprd-apps: harbor) archive WAL and take daily base backups through the Barman Cloud plugin (gitops-core `cnpg-barman-cloud`) to `s3://rke2-backups/cnpg///`. They use the rustfs S3 API on port **30292**; 30293 is the console. Retention is 14 days. Restore and checks are documented in each repo's README. Adding or changing the plugin rolls each cluster with an in-place primary restart (`primaryUpdateMethod: restart`). Authentik SSO was down about 4 min (its server pods were restarted by the liveness probe); Coder and high-command lost the DB for under a minute. +- **Open: CNPG backups share the `rke2-backups` bucket and key with Rancher backups.** The existing rustfs key can't create buckets. Create a dedicated `cnpg-backups` bucket and a key scoped to it in the rustfs console, then update the `cnpg-backup-rustfs` Secrets (coder, high-command, authentik on prd-apps; harbor on nprd-apps) and each ObjectStore's `destinationPath`. - **high-command-postgres is now Fleet-managed** (gitops-mcp `k8s/high-command/`, adopted with `helm.takeOwnership`). Helm two-way-patches custom resources, so adoption only adds ownership metadata. Spec and Fleet labels were synced to git by hand once. Keep the manifest identical to live, including CNPG-defaulted fields such as `plugins[].enabled`, or Fleet's drift correction loops rollbacks. - **poc-apps wildcard cert expired (2026-04-18).** `cert-manager/cert-sync` (gitops-core, manager) fails for poc-apps: its kubeconfig entry uses the Rancher proxy URL with a deleted token, and the job image fails x509 verification. nprd/prd copies are current. Fix in gitops-core (RKE2 admin kubeconfig entry for poc-apps). The other entries' client certs expire 2027-01-08. See [CLUSTER_ACCESS_AND_SSO.md § Certificate calendar](CLUSTER_ACCESS_AND_SSO.md#certificate-and-expiry-calendar). - **Terraform's Rancher API token is expired** (`config/.rancher-api-token` = `rancher_api_token` in tfvars, HTTP 401), and the registration modules read a hard-coded `/home/lee/git/rancher-deploy/config/.rancher-api-token`. Refresh before any `register_downstream_cluster` apply — [RANCHER_API_TOKEN_CREATION.md](RANCHER_API_TOKEN_CREATION.md). -- **`~/.kube/.yaml` on the main workstation are Rancher-proxied token kubeconfigs**, not the RKE2 admin kubeconfigs Terraform writes (`rancher-manager.yaml`'s token is already expired). Re-pull them before relying on them for break-glass or Terraform add-on steps — see [CLUSTER_ACCESS_AND_SSO.md § Fallback](CLUSTER_ACCESS_AND_SSO.md#fallback-and-break-glass). +- **Break-glass kubeconfigs — resolved 2026-10-04.** Break-glass now lives in `~/.kube/-rke2.yaml` (RKE2 admin, straight to `https://.dataknife.net:6443`; `manager.dataknife.net` for rancher-manager). The old `~/.kube/.yaml` files on the main workstation are Rancher-proxied token kubeconfigs (`rancher-manager.yaml`'s token is expired); don't use them for break-glass. Terraform's `get_kubeconfig` will overwrite them with RKE2 admin kubeconfigs on the next full apply. See [CLUSTER_ACCESS_AND_SSO.md § Fallback](CLUSTER_ACCESS_AND_SSO.md#fallback-and-break-glass). ## Secrets hygiene