diff --git a/infra/server/README.md b/infra/server/README.md index 21b63f1..274cd00 100644 --- a/infra/server/README.md +++ b/infra/server/README.md @@ -58,13 +58,88 @@ run alone, never inside a pasted block. `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS`. - GHCR package `relay-api`: public, so the cluster pulls it without a password. -## 6. Data (Phase 6) +## 6. Data: Postgres (step 2.3; backups in Phase 6) -Restore the newest database dump from S3. Until then, a lost server means lost data. +Postgres runs in namespace `relay` on a disk that is kept even if its claim is deleted +(StorageClass `local-path-retain`). It's applied by hand, never by the pipeline +([`k8s/README.md`](../../k8s/README.md)). From the repo root on the laptop, **in this order**, +each step only after the previous one worked: + +| # | What | How | Expected | +|---|---|---|---| +| 1 | The StorageClass | `ssh relay 'kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/platform/storageclass-retain.yaml` | `kubectl get sc` lists `local-path-retain … Retain`, and `local-path` stays the default | +| 2 | The passwords | `ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh` | `created postgres-auth`, `created relay-db` (or `kept …`: it never overwrites) | +| 3 | The first-start script | `ssh relay 'kubectl --kubeconfig ~/.kube/config -n relay create configmap postgres-init --from-file=10-create-relay.sh=/dev/stdin --dry-run=client -o yaml \| kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/data/initdb/10-create-relay.sh` | `configmap/postgres-init created` | +| 4 | Postgres | `ssh relay 'kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/data/postgres.yaml`, then `kubectl -n relay rollout status sts/postgres` | `postgres-0` ready; its log says `10-create-relay.sh: created the login relay and its database` | + +Check a login **from another pod**, never from inside `postgres-0`: the image trusts the local +socket and `127.0.0.1` without a password, so a login from inside proves nothing. + +Until the backups exist (Phase 6), **a lost server means lost data**. Retain protects against +`kubectl` mistakes (a deleted claim or namespace), not against losing the disk. Then: restore +the newest dump from S3. k3s's own data (the cluster state) isn't backed up on purpose: everything in it is rebuilt from git and these scripts. Only the database holds data that can't be rebuilt. +### 6a. Re-attach a kept disk (the claim `data-postgres-0` was deleted) + +The disk (a PersistentVolume, PV) shows `Released`, and its folder is still on the server. +Kubernetes never reuses it by itself. **Postgres must be stopped first.** A claim that a +running pod uses can't be deleted (it stays `Terminating`), and a running StatefulSet could +make a new, **empty** disk for `postgres-0` before the old one is back. Never remove a +`kubernetes.io/pvc-protection` finalizer to force things. + +1. Write down which disk it is: the PV name, and its folder (`.spec.hostPath.path` or + `.spec.local.path` in `kubectl get pv -o yaml`). +2. `kubectl -n relay scale sts postgres --replicas=0`, and wait until `postgres-0` is gone. +3. If the claim still exists: `kubectl -n relay delete pvc data-postgres-0`, and wait until the PV + shows `Released`. +4. Free the PV for exactly one new claim, by removing only its old binding: + `kubectl patch pv --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` +5. Create `data-postgres-0` again: class `local-path-retain`, `ReadWriteOnce`, `10Gi`, **and + `volumeName: `**, so it can only bind to that disk. Wait for `Bound`, and check that its + `volumeName` is the PV from step 1. +6. `kubectl -n relay scale sts postgres --replicas=1`. The data is back. + +### 6b. A Secret was lost, but the disk was kept + +The volume still has the old password, so the API can't log in. `bootstrap-secrets.sh` makes a +**new** password, and a new Secret **never** changes the password stored in Postgres. So set the +stored one to the new value, without the value ever leaving the server: + +1. `ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh` creates the missing Secret. +2. `kubectl -n relay delete pod postgres-0`. The new pod reads the new values into its env. +3. Set the stored password from the pod's own env. psql's `\getenv` reads it, so it's never on a + command line, in shell history or in Postgres's log: + + ```bash + ssh relay 'kubectl --kubeconfig ~/.kube/config -n relay exec -i postgres-0 -- psql -X -v ON_ERROR_STOP=1 -U postgres -d postgres' <<'SQL' + \getenv pw RELAY_DB_PASSWORD + SET log_min_error_statement = panic; + ALTER ROLE relay PASSWORD :'pw'; + SQL + ``` + + For `postgres-auth`, use `\getenv pw POSTGRES_PASSWORD` and `ALTER ROLE postgres …`. Only admins + can `exec`, and the local socket needs no password. +4. `kubectl -n relay rollout restart deployment/relay-api`, so the API reads the new `url`. +5. Check a login from another pod (section 6). + +Tested on 31 Oct (locally, same image): the new password, containing `'`, `;`, `$` and `"`, +logs in, the old one is refused, and the password appears 0 times in Postgres's log. + +### 6c. The very first start failed halfway + +If `10-create-relay.sh` failed (Postgres's log shows an error instead of `created the login +relay and its database`), the database exists but has no `relay` login. On the next start the +image sees an existing database and **skips** the script. This can only happen on a brand-new, +empty volume, so start over (ask the other person first: this deletes the volume): +1. Scale Postgres to 0. +2. Delete the PVC and then its PV. +3. Delete the PV's folder on the server (`sudo rm -rf `). +4. Fix the script, then repeat steps 3–4 of section 6. + ## Reboots `unattended-upgrades` installs security updates every day but never reboots by itself @@ -74,5 +149,10 @@ defense. - **Before:** check that the running version is the last deploy, so a reboot can't quietly bring back an old image: `curl -s https://api.myrelay.tech/version` must show the same - commit as `git rev-parse origin/main`. -- **After:** `/version` answers again with that same commit. + commit as `git rev-parse origin/main`. This is valid because every merge to main is built and + deployed (`pipeline.yml` has no path filter on `push`). If a path filter ever comes back, + compare with the last *deployed* commit instead, or this check reports a failure after every + docs-only merge. +- **After:** `/version` answers again with that same commit, and `kubectl -n relay get pods` + shows `postgres-0` and the API pods ready. Postgres can come up after the API: the API + waits for it (from step 2.6). diff --git a/infra/server/bootstrap-secrets.sh b/infra/server/bootstrap-secrets.sh new file mode 100644 index 0000000..bbad7fd --- /dev/null +++ b/infra/server/bootstrap-secrets.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# bootstrap-secrets: creates the database passwords as Kubernetes Secrets, on the server (step 2.2). +# +# Run it from the laptop, as an admin: +# ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh +# "bash -s" reads the script from SSH, so it runs on the server with your admin kubeconfig and +# nothing is copied to the server's disk. +# +# It creates each Secret ONLY IF IT'S MISSING, and prints only "created " or "kept ". +# The passwords are made here, on the server, and never printed, logged, written to a file or +# put on a command line (where "ps" would show them to every user on the server). +# +# Run it again any time: it never overwrites. That matters, because a Secret is only a copy of +# the password: Postgres stores its own copy on its volume when it first starts (step 2.3). +# A new Secret doesn't change that stored password; recovering from a lost Secret is in the runbook. +set -euo pipefail +umask 077 # anything bash writes for us (for example a large here-document) is readable only by us + +# The namespace can be overridden to test the script in a throwaway namespace: +# ssh relay 'NAMESPACE=secrets-test bash -s' < infra/server/bootstrap-secrets.sh +NAMESPACE="${NAMESPACE:-relay}" + +fail() { echo "bootstrap-secrets: $*" >&2; exit 1; } + +# kubectl with the admin kubeconfig, always in our namespace. +kube() { kubectl --kubeconfig "$HOME/.kube/config" -n "$NAMESPACE" "$@"; } + +# A new random password: 32 random bytes from the kernel, as 64 characters of 0-9 and a-f. +# Hex, because base64's "/", "+" and "=" would break the postgres:// URL below. +# The check makes sure an openssl problem can never turn into an empty or odd password. +new_password() { + local pw + pw=$(openssl rand -hex 32) || fail "openssl couldn't make a password" + [[ "$pw" =~ ^[0-9a-f]{64}$ ]] || fail "openssl gave an unexpected value" + printf '%s' "$pw" +} + +# ensure_secret +# Creates the Secret from the KEY=VALUE lines on stdin, but only if it doesn't exist yet. +# Prints "kept " or "created ", never a value. +ensure_secret() { + local name="$1" found + # --ignore-not-found gives three clear answers: nothing printed = missing, a line = it exists, + # a failed command = we couldn't check (no connection, no permission). Never treat "couldn't + # check" as "missing": never write when we couldn't read. + found=$(kube get secret "$name" --ignore-not-found) || fail "couldn't check for Secret $name" + if [[ -n "$found" ]]; then + echo "kept $name" + else + # The values come from stdin, never from the command line (where "ps" would show them). + kube create secret generic "$name" --from-env-file=/dev/stdin >/dev/null || fail "couldn't create Secret $name" + echo "created $name" + fi +} + + +# Step 1: the superuser's password. Only admins use it, through "kubectl exec". +# Made first, then handed over: if openssl failed inside the here-document itself, bash would +# carry on with an empty password instead of stopping. +superuser_password=$(new_password) +ensure_secret postgres-auth <&2 + exit 1 +fi +echo "10-create-relay.sh: created the login relay and its database" diff --git a/k8s/data/postgres.yaml b/k8s/data/postgres.yaml new file mode 100644 index 0000000..34976cc --- /dev/null +++ b/k8s/data/postgres.yaml @@ -0,0 +1,147 @@ +# Postgres 18.6 for Relay (step 2.3): one instance, on a disk that's kept even if its claim is +# deleted (StorageClass local-path-retain, step 2.1). +# +# Applied by hand by one of us, NEVER by the pipeline: the deployer account can't create +# StatefulSets on purpose, so a merged PR can't change the database's version or its disk. +# Order: k8s/platform/storageclass-retain.yaml → infra/server/bootstrap-secrets.sh → +# ConfigMap postgres-init → this file. The ConfigMap is made from the plain script file, so git +# holds only one copy of it: +# ssh relay 'kubectl --kubeconfig ~/.kube/config -n relay create configmap postgres-init --from-file=10-create-relay.sh=/dev/stdin --dry-run=client -o yaml | kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/data/initdb/10-create-relay.sh +# ssh relay 'kubectl --kubeconfig ~/.kube/config apply -f -' < k8s/data/postgres.yaml + +# The address inside the cluster: postgres.relay.svc.cluster.local:5432. +# Headless (clusterIP: None): DNS answers with the pod's own IP. One pod needs no virtual IP. +# ClusterIP only, never NodePort or LoadBalancer: Postgres must never be reachable from outside +# (namespace relay's Service policy refuses anything else anyway). +apiVersion: v1 +kind: Service +metadata: + name: postgres + namespace: relay + labels: + app.kubernetes.io/name: postgres +spec: + clusterIP: None + selector: + app.kubernetes.io/name: postgres + ports: + - name: postgres + port: 5432 + targetPort: postgres +--- +# StatefulSet: "always exactly one Postgres called postgres-0, and it always gets its own disk +# back". A Deployment's rolling update would start a second Postgres on the same data folder. +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: postgres + namespace: relay + labels: + app.kubernetes.io/name: postgres +spec: + serviceName: postgres + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: postgres + # Deleting the StatefulSet or scaling it down never deletes the disk's claim (the default, + # written out so it's visible). + persistentVolumeClaimRetentionPolicy: + whenDeleted: Retain + whenScaled: Retain + template: + metadata: + labels: + app.kubernetes.io/name: postgres + spec: + automountServiceAccountToken: false # Postgres never talks to Kubernetes + enableServiceLinks: false # no Kubernetes-made POSTGRES_* variables in its env + # Kubernetes asks Postgres to stop with the image's signal, SIGINT ("fast shutdown": end the + # connections, write everything to disk, stop: seconds). 60 s is a generous margin before + # SIGKILL, which would force a crash recovery on the next start. + terminationGracePeriodSeconds: 60 + # Pod Security "restricted": the image's own postgres user (uid/gid 999), never root. + # The disk's folder is made world-writable by local-path, so uid 999 can write in it. + securityContext: + runAsNonRoot: true + runAsUser: 999 + runAsGroup: 999 + fsGroup: 999 + seccompProfile: + type: RuntimeDefault + containers: + - name: postgres + # 18.6, pinned to its exact bytes: a tag can be pushed again with different contents. + image: postgres:18.6@sha256:979bccfde35445db618c65107d27d08c6c21339d8dc3ad4c830c6633947be0e4 + # Written out instead of left as the invisible default (also 100): the connection + # budget (5.1 in the Phase 2 plan) adds up against this number. + args: ["postgres", "-c", "max_connections=100"] + ports: + - name: postgres + containerPort: 5432 + env: + - name: POSTGRES_PASSWORD # the superuser "postgres" (admins only) + valueFrom: + secretKeyRef: + name: postgres-auth + key: password + - name: RELAY_DB_PASSWORD # read once, by the first-start script + valueFrom: + secretKeyRef: + name: relay-db + key: password + # "May this pod get connections?" Asked over TCP (127.0.0.1), not the local socket: + # during the very first start a temporary server answers only on the socket, and + # Postgres would look ready too early. + readinessProbe: + exec: + command: ["pg_isready", "-h", "127.0.0.1", "-p", "5432", "-t", "2"] + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + # No livenessProbe on purpose: it could kill Postgres in the middle of a long crash + # recovery, which would then start over, in a loop. A crashed Postgres exits by + # itself, and Kubernetes restarts it anyway. + resources: + requests: + cpu: 250m + memory: 512Mi + limits: + memory: 1Gi # above this the kernel stops Postgres; it recovers on restart + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true # Postgres may only write to the folders mounted below + capabilities: + drop: ["ALL"] + volumeMounts: + - name: data # the database itself (PGDATA is .../18/docker inside it) + mountPath: /var/lib/postgresql + - name: init # the first-start script, read-only + mountPath: /docker-entrypoint-initdb.d + readOnly: true + - name: socket # Postgres's local socket and lock file + mountPath: /var/run/postgresql + - name: tmp + mountPath: /tmp + volumes: + - name: init + configMap: + name: postgres-init + defaultMode: 0555 # executable: runs as its own program, not sourced + - name: socket + emptyDir: + sizeLimit: 16Mi + - name: tmp + emptyDir: + sizeLimit: 64Mi + # The disk order form: pod postgres-0 gets the claim data-postgres-0 on our Retain class. + # 10Gi is only a label for local-path, which doesn't enforce sizes. + volumeClaimTemplates: + - metadata: + name: data + spec: + storageClassName: local-path-retain + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 10Gi diff --git a/k8s/platform/storageclass-retain.yaml b/k8s/platform/storageclass-retain.yaml new file mode 100644 index 0000000..cf204e4 --- /dev/null +++ b/k8s/platform/storageclass-retain.yaml @@ -0,0 +1,22 @@ +# A second recipe for disks (step 2.1). Identical to k3s's own "local-path", except for one +# clause: when a disk's claim (PVC) is deleted, by mistake or by deleting namespace relay, +# the data folder is kept instead of deleted. Postgres asks for this class by name (step 2.3). +# +# Why not just change local-path? Its reclaimPolicy can't be changed ("field is immutable"), +# and k3s re-applies its own copy when it restarts. Not the default class on purpose: the +# throwaway data of later tools (Prometheus, Loki) shouldn't be kept forever. +# +# The cost of Retain: a kept ("Released") disk is never reused or cleaned up by itself. One of +# us re-attaches it or deletes it on purpose (k8s/README.md). It protects against kubectl +# mistakes, not against losing the server's disk: only backups do that (Phase 6). +# +# Cluster-wide, so it's a platform file, applied by hand by one of us. +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: local-path-retain +provisioner: rancher.io/local-path # the same program that makes local-path's folders +reclaimPolicy: Retain # claim deleted → the PV becomes "Released", the folder stays +volumeBindingMode: WaitForFirstConsumer # make the folder once the pod's node is known +# No size limit here: local-path ignores the requested size, so a disk can grow until the +# server's disk is full. Disk usage is watched in Phase 7.