infra: Postgres 18.6 on a disk that survives deletion

infra/server/README.md

@@ -58,13 +58,88 @@ run alone, never inside a pasted block.
   `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS`.
 - GHCR package `relay-api`: public, so the cluster pulls it without a password.
 
-## 6. Data (Phase 6)
+## 6. Data: Postgres (step 2.3; backups in Phase 6)
 
-Restore the newest database dump from S3. Until then, a lost server means lost data.
+Postgres runs in namespace `relay` on a disk that is kept even if its claim is deleted
+(StorageClass `local-path-retain`). It's applied by hand, never by the pipeline
+([`k8s/README.md`](../../k8s/README.md)). From the repo root on the laptop, **in this order**,
+each step only after the previous one worked:
+
+| # | What | How | Expected |
+|---|---|---|---|
+| 1 | The StorageClass | `ssh relay 'kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/platform/storageclass-retain.yaml` | `kubectl get sc` lists `local-path-retain … Retain`, and `local-path` stays the default |
+| 2 | The passwords | `ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh` | `created postgres-auth`, `created relay-db` (or `kept …`: it never overwrites) |
+| 3 | The first-start script | `ssh relay 'kubectl --kubeconfig ~/.kube/config -n relay create configmap postgres-init --from-file=10-create-relay.sh=/dev/stdin --dry-run=client -o yaml \| kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/data/initdb/10-create-relay.sh` | `configmap/postgres-init created` |
+| 4 | Postgres | `ssh relay 'kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/data/postgres.yaml`, then `kubectl -n relay rollout status sts/postgres` | `postgres-0` ready; its log says `10-create-relay.sh: created the login relay and its database` |
+
+Check a login **from another pod**, never from inside `postgres-0`: the image trusts the local
+socket and `127.0.0.1` without a password, so a login from inside proves nothing.
+
+Until the backups exist (Phase 6), **a lost server means lost data**. Retain protects against
+`kubectl` mistakes (a deleted claim or namespace), not against losing the disk. Then: restore
+the newest dump from S3.
 
 k3s's own data (the cluster state) isn't backed up on purpose: everything in it is rebuilt
 from git and these scripts. Only the database holds data that can't be rebuilt.
 
+### 6a. Re-attach a kept disk (the claim `data-postgres-0` was deleted)
+
+The disk (a PersistentVolume, PV) shows `Released`, and its folder is still on the server.
+Kubernetes never reuses it by itself. **Postgres must be stopped first.** A claim that a
+running pod uses can't be deleted (it stays `Terminating`), and a running StatefulSet could
+make a new, **empty** disk for `postgres-0` before the old one is back. Never remove a
+`kubernetes.io/pvc-protection` finalizer to force things.
+
+1. Write down which disk it is: the PV name, and its folder (`.spec.hostPath.path` or
+   `.spec.local.path` in `kubectl get pv <pv> -o yaml`).
+2. `kubectl -n relay scale sts postgres --replicas=0`, and wait until `postgres-0` is gone.
+3. If the claim still exists: `kubectl -n relay delete pvc data-postgres-0`, and wait until the PV
+   shows `Released`.
+4. Free the PV for exactly one new claim, by removing only its old binding:
+   `kubectl patch pv <pv> --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'`
+5. Create `data-postgres-0` again: class `local-path-retain`, `ReadWriteOnce`, `10Gi`, **and
+   `volumeName: <pv>`**, so it can only bind to that disk. Wait for `Bound`, and check that its
+   `volumeName` is the PV from step 1.
+6. `kubectl -n relay scale sts postgres --replicas=1`. The data is back.
+
+### 6b. A Secret was lost, but the disk was kept
+
+The volume still has the old password, so the API can't log in. `bootstrap-secrets.sh` makes a
+**new** password, and a new Secret **never** changes the password stored in Postgres. So set the
+stored one to the new value, without the value ever leaving the server:
+
+1. `ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh` creates the missing Secret.
+2. `kubectl -n relay delete pod postgres-0`. The new pod reads the new values into its env.
+3. Set the stored password from the pod's own env. psql's `\getenv` reads it, so it's never on a
+   command line, in shell history or in Postgres's log:
+
+   ```bash
+   ssh relay 'kubectl --kubeconfig ~/.kube/config -n relay exec -i postgres-0 -- psql -X -v ON_ERROR_STOP=1 -U postgres -d postgres' <<'SQL'
+   \getenv pw RELAY_DB_PASSWORD
+   SET log_min_error_statement = panic;
+   ALTER ROLE relay PASSWORD :'pw';
+   SQL
+   ```
+
+   For `postgres-auth`, use `\getenv pw POSTGRES_PASSWORD` and `ALTER ROLE postgres …`. Only admins
+   can `exec`, and the local socket needs no password.
+4. `kubectl -n relay rollout restart deployment/relay-api`, so the API reads the new `url`.
+5. Check a login from another pod (section 6).
+
+Tested on 31 Oct (locally, same image): the new password, containing `'`, `;`, `$` and `"`,
+logs in, the old one is refused, and the password appears 0 times in Postgres's log.
+
+### 6c. The very first start failed halfway
+
+If `10-create-relay.sh` failed (Postgres's log shows an error instead of `created the login
+relay and its database`), the database exists but has no `relay` login. On the next start the
+image sees an existing database and **skips** the script. This can only happen on a brand-new,
+empty volume, so start over (ask the other person first: this deletes the volume):
+1. Scale Postgres to 0.
+2. Delete the PVC and then its PV.
+3. Delete the PV's folder on the server (`sudo rm -rf <path from the PV>`).
+4. Fix the script, then repeat steps 3–4 of section 6.
+
 ## Reboots
 
 `unattended-upgrades` installs security updates every day but never reboots by itself
@@ -74,5 +149,10 @@ defense.
 
 - **Before:** check that the running version is the last deploy, so a reboot can't quietly
   bring back an old image: `curl -s https://api.myrelay.tech/version` must show the same
-  commit as `git rev-parse origin/main`.
-- **After:** `/version` answers again with that same commit.
+  commit as `git rev-parse origin/main`. This is valid because every merge to main is built and
+  deployed (`pipeline.yml` has no path filter on `push`). If a path filter ever comes back,
+  compare with the last *deployed* commit instead, or this check reports a failure after every
+  docs-only merge.
+- **After:** `/version` answers again with that same commit, and `kubectl -n relay get pods`
+  shows `postgres-0` and the API pods ready. Postgres can come up after the API: the API
+  waits for it (from step 2.6).

infra/server/bootstrap-secrets.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# bootstrap-secrets: creates the database passwords as Kubernetes Secrets, on the server (step 2.2).
+#
+# Run it from the laptop, as an admin:
+#   ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh
+# "bash -s" reads the script from SSH, so it runs on the server with your admin kubeconfig and
+# nothing is copied to the server's disk.
+#
+# It creates each Secret ONLY IF IT'S MISSING, and prints only "created <name>" or "kept <name>".
+# The passwords are made here, on the server, and never printed, logged, written to a file or
+# put on a command line (where "ps" would show them to every user on the server).
+#
+# Run it again any time: it never overwrites. That matters, because a Secret is only a copy of
+# the password: Postgres stores its own copy on its volume when it first starts (step 2.3).
+# A new Secret doesn't change that stored password; recovering from a lost Secret is in the runbook.
+set -euo pipefail
+umask 077   # anything bash writes for us (for example a large here-document) is readable only by us
+
+# The namespace can be overridden to test the script in a throwaway namespace:
+#   ssh relay 'NAMESPACE=secrets-test bash -s' < infra/server/bootstrap-secrets.sh
+NAMESPACE="${NAMESPACE:-relay}"
+
+fail() { echo "bootstrap-secrets: $*" >&2; exit 1; }
+
+# kubectl with the admin kubeconfig, always in our namespace.
+kube() { kubectl --kubeconfig "$HOME/.kube/config" -n "$NAMESPACE" "$@"; }
+
+# A new random password: 32 random bytes from the kernel, as 64 characters of 0-9 and a-f.
+# Hex, because base64's "/", "+" and "=" would break the postgres:// URL below.
+# The check makes sure an openssl problem can never turn into an empty or odd password.
+new_password() {
+  local pw
+  pw=$(openssl rand -hex 32) || fail "openssl couldn't make a password"
+  [[ "$pw" =~ ^[0-9a-f]{64}$ ]] || fail "openssl gave an unexpected value"
+  printf '%s' "$pw"
+}
+
+# ensure_secret <name>
+# Creates the Secret <name> from the KEY=VALUE lines on stdin, but only if it doesn't exist yet.
+# Prints "kept <name>" or "created <name>", never a value.
+ensure_secret() {
+  local name="$1" found
+  # --ignore-not-found gives three clear answers: nothing printed = missing, a line = it exists,
+  # a failed command = we couldn't check (no connection, no permission). Never treat "couldn't
+  # check" as "missing": never write when we couldn't read.
+  found=$(kube get secret "$name" --ignore-not-found) || fail "couldn't check for Secret $name"
+  if [[ -n "$found" ]]; then
+    echo "kept $name"
+  else
+    # The values come from stdin, never from the command line (where "ps" would show them).
+    kube create secret generic "$name" --from-env-file=/dev/stdin >/dev/null || fail "couldn't create Secret $name"
+    echo "created $name"
+  fi
+}
+
+
+# Step 1: the superuser's password. Only admins use it, through "kubectl exec".
+# Made first, then handed over: if openssl failed inside the here-document itself, bash would
+# carry on with an empty password instead of stopping.
+superuser_password=$(new_password)
+ensure_secret postgres-auth <<EOF
+password=$superuser_password
+EOF
+unset superuser_password
+
+# Step 2: the app's own login "relay": its password (for Postgres's first-start script, step 2.3)
+# and the full URL with that password in it (for the API and the migrate Job).
+# sslmode=disable: no TLS inside the cluster, because this traffic never leaves the server.
+app_password=$(new_password)
+ensure_secret relay-db <<EOF
+password=$app_password
+url=postgres://relay:$app_password@postgres.$NAMESPACE.svc.cluster.local:5432/relay?sslmode=disable
+EOF
+unset app_password

k8s/README.md

@@ -4,7 +4,8 @@ How each folder gets onto the cluster, and who may apply it.
 
 | Folder | What's in it | Applied by |
 |---|---|---|
-| `k8s/platform/` | Cluster setup: Let's Encrypt issuers, Traefik settings, the `relay` namespace (with its Pod Security level), the `deployer` account and the Service policy for `relay` | One of us, by hand, after the PR is reviewed |
+| `k8s/platform/` | Cluster setup: Let's Encrypt issuers, Traefik settings, the `relay` namespace (with its Pod Security level), the `deployer` account, the Service policy for `relay` and the StorageClass `local-path-retain` | One of us, by hand, after the PR is reviewed |
+| `k8s/data/` (from step 2.3) | Postgres: its Service and StatefulSet (`postgres.yaml`) and its first-start script (`initdb/`). Later also the backup CronJob (Phase 6) | One of us, by hand, after the PR is reviewed. **Never the pipeline**: the `deployer` account can't create StatefulSets or Secrets, so a merged PR can't change the database's version or its disk. Order and recovery: [`infra/server/README.md`](../infra/server/README.md), section 6 |
 | `k8s/app/` (from step 1.7) | Our app in namespace `relay`, the API and the dashboard: Deployments, Services, Ingresses, ConfigMaps | Only the pipeline, through `relay-deploy` |
 
 ## Rules
@@ -37,6 +38,14 @@ How each folder gets onto the cluster, and who may apply it.
   - **Services stay inside the cluster** (`relay-service-policy.yaml`): only type ClusterIP
     (headless included), never `externalIPs`. ufw can't filter ports that Kubernetes
     publishes, so nothing in `relay`, and above all Postgres, may publish one.
+  - **What the guards don't cover**, on purpose, so we know it rather than find out:
+    1. The Service policy only sees **Services**. A Traefik `IngressRouteTCP` could publish
+       a TCP port without one. We have none, and only an admin could create one.
+    2. Both guards only cover **`relay`**. A NodePort Service in another namespace, pointed at
+       a `relay` pod's IP by hand, would still reach that pod from outside.
+
+    The `deployer` account can do neither: it is confined to `relay` and has no rights on
+    Traefik's own objects. Both need an admin doing it deliberately.
 - **The issuers' challenge Services are ClusterIP** (`cluster-issuers.yaml`). cert-manager's
   default is NodePort, which the Service policy would refuse in `relay`. Apply the issuers
   before the policy.

k8s/data/initdb/10-create-relay.sh

@@ -0,0 +1,47 @@
+#!/bin/sh
+# Postgres's first-start script (step 2.3): creates the app's own login "relay" and the
+# database "relay" owned by it. "relay" is NOT a superuser: it can work in its own database,
+# but can't create roles or databases, read the server's files or run programs on it.
+#
+# Runs ONCE: the postgres image starts it from /docker-entrypoint-initdb.d only on an empty
+# volume, while a temporary server listens on the local socket only. Changing this file later
+# changes nothing in an existing database.
+# The same file runs on the laptop (mounted into the container) and in CI (2.6), so tests use
+# the same login as production.
+#
+# The password never appears on a command line, in the output or in Postgres's log.
+set -eu
+
+# Stop with a clear message if the password is missing. Without this, psql's \getenv would
+# silently leave :pw unset and the SQL would fail with a confusing error instead.
+: "${RELAY_DB_PASSWORD:?RELAY_DB_PASSWORD is missing}"
+
+# -X: ignore any .psqlrc (it could turn on echoing). ON_ERROR_STOP: stop at the first error.
+# <<'SQL' (quoted): the shell changes nothing inside, so the SQL arrives exactly as written.
+psql -X -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<'SQL'
+-- Read the password from this process's environment into the psql variable "pw".
+-- (psql -v pw=... would put it on the command line, where "ps" shows it.)
+\getenv pw RELAY_DB_PASSWORD
+
+-- If a statement below fails, Postgres would normally write its full text, password included,
+-- to its log, which the deployer's pods/log right can read. "panic" = only statements that
+-- crash the whole server get logged.
+SET log_min_error_statement = panic;
+
+-- The app's login, with the password from :'pw' (psql quotes it as a value, so a quote or a
+-- semicolon in it can't change the SQL). The NO... flags are the defaults, written out.
+CREATE ROLE relay WITH LOGIN PASSWORD :'pw' NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION;
+CREATE DATABASE relay OWNER relay;
+
+SQL
+
+# Check that it worked, so a mistake fails loudly on the very first start, not later as a
+# confusing login error in the API.
+ok=$(psql -X -At --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" -c "
+  select count(*) from pg_database d join pg_roles r on r.oid = d.datdba
+  where d.datname = 'relay' and r.rolname = 'relay' and r.rolcanlogin and not r.rolsuper")
+if [ "$ok" != 1 ]; then
+  echo "10-create-relay.sh: the login or the database relay is missing or wrong" >&2
+  exit 1
+fi
+echo "10-create-relay.sh: created the login relay and its database"