[devinder-dev]

Rebased onto main after #20 and #21.

What

Steps 2.1–2.3 of Phase 2: Postgres 18.6 on a disk that survives deletion. Nothing here is
applied by the pipeline: the deployer account can't create StorageClasses, Secrets or
StatefulSets, on purpose.

Step	File	What
2.1	k8s/platform/storageclass-retain.yaml	A second StorageClass, local-path-retain: like k3s's local-path, but reclaimPolicy: Retain. Deleting the claim (or the namespace) keeps the data folder instead of rm -rf-ing it
2.2	infra/server/bootstrap-secrets.sh	Creates the Secrets postgres-auth (superuser password) and relay-db (the app's password + its DATABASE_URL) on the server, only if missing, from openssl rand -hex 32. Prints only created/kept; values never on a command line, in output or in a file
2.3	k8s/data/postgres.yaml	Headless Service postgres (ClusterIP only) + StatefulSet: postgres:18.6 pinned by digest, uid 999, Pod Security restricted, read-only root, max_connections=100, readiness pg_isready over TCP, no liveness probe, 60 s grace, a 10Gi claim on local-path-retain
2.3	k8s/data/initdb/10-create-relay.sh	Runs once, on an empty volume: creates the login relay (not a superuser) and the database relay owned by it. Password via psql \getenv + :'pw', and log_min_error_statement = panic so a failing statement can't write the password to Postgres's log

Docs:

• k8s/README.md: a row for k8s/data/ ("never the pipeline"), and the StorageClass in the
  platform row. Also your two limits from infra: guard namespace relay (Pod Security restricted, ClusterIP-only Services) #21: the policy only sees Services (a Traefik
  IngressRouteTCP wouldn't be checked), and both guards cover only relay.
• infra/server/README.md: section 6 becomes the real data section:
  • the apply order (StorageClass → secrets → ConfigMap → Postgres), and why a login check
    must come from another pod;
  • 6a re-attaching a kept disk: scale to 0 first, then re-bind with volumeName; never
    remove finalizers;
  • 6b a lost Secret with a kept disk: ALTER ROLE from the pod's own env with psql
    \getenv. Tested: a password with ', ;, $, " works, the old one is refused, and
    there are 0 lines of it in the log;
  • 6c a first start that failed halfway.
• Your half-sentence from docs: close Phase 1 with the review points and the live test results #20 in "Reboots": the /version == origin/main check is valid because
  every merge is built and deployed.

Why these choices

• Retain, in our own class: local-path can't be changed (reclaimPolicy is immutable, and
  k3s re-applies its own copy). Retain protects against kubectl mistakes, not against losing
  the server's disk; only backups do that (Phase 6).
• A StatefulSet: a Deployment's rolling update would start a second Postgres on the same data
  folder.
• Applied by an admin: a merged PR can't change the database's major version or its disk.
• The app is not a superuser: POSTGRES_USER=relay would make it one, able to run shell
  commands on the database server (COPY … TO PROGRAM).
• No liveness probe: it could kill Postgres in the middle of a crash recovery, in a loop.
• Readiness over TCP: on the first start a temporary server answers only on the local socket.

Tested before the PR

Locally in Docker (the same digest, uid 999, read-only root, no capabilities):

• a password containing ', ;, ", \, $ and an attempted CREATE TABLE pwned(): the
  login works with exactly that password, one character less is refused, and no pwned table
  exists;
• a forced failing CREATE ROLE: the password appears 0 times in Postgres's log. Control
  without the SET: the full statement and password are logged.

bootstrap-secrets.sh in a throwaway namespace: first run created ×2, second run kept ×2 with
unchanged resourceVersions, keys password / password,url, no 64-hex string in any output.
A fake-kubectl test: when the check can't reach the server, it stops before any create.

Postgres in a throwaway namespace pg-test with the same Pod Security labels
(restricted, v1.36), filled by bootstrap-secrets.sh, then deleted:

Check	Result
Pod / PVC / Service	Ready, 0 restarts; Bound; ClusterIP None, no NodePort
Pod Security	No warning on apply, 0 PodSecurity events
Version, checksums, max_connections	18.6, on, 100
Login from another pod with the Secret's url	logged in as relay in relay
Wrong password from another pod	password authentication failed
relay's flags (superuser, createrole, createdb, replication)	all false
COPY … TO PROGRAM as relay	permission denied
Delete the pod	"received fast shutdown request" (the image's SIGINT reaches Postgres), shut down in 26 ms, back with the same data

Found while testing: the image's pg_hba.conf trusts the local socket and 127.0.0.1
without a password; only connections from other pods need one. So login checks run from a
separate pod. Only admins can kubectl exec into postgres-0 (the deployer has no exec).

After merging (an admin, in this order, output posted here)

1. k8s/platform/storageclass-retain.yaml → kubectl get sc shows local-path-retain … Retain,
   local-path still the default.
2. ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh → created ×2.
3. The ConfigMap postgres-init from 10-create-relay.sh (the command is in the header of
   postgres.yaml), then postgres.yaml.
4. The checks above, now in relay, plus from outside: nc -z -G 3 81.88.23.71 443 succeeds and
   nc -z -G 3 81.88.23.71 5432 fails.
5. The deletion drill, before any real data: a marker row survives deleting the pod, the
   StatefulSet, and the PVC (Postgres scaled to 0 first, then the kept PV re-attached with
   volumeName).

For the reviewer

• Anything that should be applied by the pipeline instead of by hand, or the other way round?
• Is "trusted inside the pod, password from outside" acceptable to you?
• The 10Gi is only a label: local-path doesn't enforce sizes. Disk usage is watched in Phase 7.

[AbdikhaniAbdirashid]

Read all six files against docs/CONTEXT.md and the Phase 2 plan, and verified three claims rather than taking them on trust. Approving — the notes below are one wording fix and two questions, none of them blocking.

The digest pin, checked

image: postgres:18.6@sha256:979bccf… — I went looking for a mismatch here and found something worth writing down:

tag 18.6 today, index digest   sha256:3a4264520e12…
your pin,       index digest   sha256:979bccfde354…

Different, which looked wrong for a minute. But your digest answers 200 and is a valid 16-platform OCI index, and the linux/amd64 manifest inside it is

sha256:f4b419e350fd98d4a8429603295208976dea4c8fb8326d818d346d3e29f1db6f

— byte-identical to the amd64 manifest the tag points at today. So the tag has been re-pushed since you pinned it (new attestations or other platforms), the bytes our amd64 server runs have not, and the pin is doing exactly its job. "Pinned to its exact bytes" is literally true for the platform we run.

The ConfigMap is required, and that is the safe choice

This was the failure I went looking for: forget step 3, Postgres starts on an empty volume, the init script never runs, and you get a database with no relay login — section 6c, which costs the volume.

It can't happen. The init volume has no optional: true, so a missing postgres-init leaves the pod in ContainerCreating with a FailedMount event. Nothing is initialised, nothing is lost. 6c is reachable only by the script itself erroring, which is what 6c says.

Worth stating because the apply order has four manual steps and this is the one whose omission would be expensive — and it is the one that can't bite.

enableServiceLinks: false

Non-obvious and right, so: the Service is named postgres, and with service links on, Kubernetes injects POSTGRES_PORT, POSTGRES_SERVICE_HOST and friends into every container in the namespace. The postgres entrypoint reads variables with exactly that prefix. Turning the links off removes a whole class of "why is the entrypoint behaving oddly" before it can happen.

One wording fix

RELAY_DB_PASSWORD is commented # read once, by the first-start script. It is read once by the script, but it stays in the pod's environment for the container's whole life — and section 6b depends on that: \getenv pw RELAY_DB_PASSWORD is how the stored password gets reset without touching a command line. The comment reads as "and then it's gone", which would make someone later think 6b can't work, or worse, try to remove the variable. Something like # the first-start script reads this; 6b reads it again when a Secret is lost would match the runbook.

Two questions

1. 1Gi memory, no liveness probe. Agreed on no liveness — a crash recovery that gets killed and restarted in a loop is worse than waiting. But that also means an OOM kill is the one way the container dies unexpectedly, and Phase 8 is a k6 load test pushing connections against max_connections=100. 100 backends at a few MB of work_mem each plus shared buffers can get close to 1Gi. Is 1Gi the number you want to walk into the load test with, or should Phase 8 revisit it? The VPS has 8 GB and the API asks for 128Mi × 2, so there is room if you want more headroom now.

2. The deletion drill. Step 5 of your post-merge list is the only thing that actually proves Retain works, and nobody has run it yet — everything else in this PR is verified, that one is a promise. Please post the output: the marker row surviving the pod, the StatefulSet and the PVC, and the kept PV re-attached with volumeName. That is also the evidence for my Phase 6 claim that Retain is not a backup, so I would like to be able to point at it on Wednesday.

Order and recovery

The apply table and 6a–6c read well and I could follow them without the code. Three things I checked and agree with: Postgres scaled to 0 before deleting the claim (a running StatefulSet would make a new empty disk for postgres-0), never removing the pvc-protection finalizer, and asking the other person before 6c since it deletes the volume.

And thank you for writing down the two limits of the relay guards in k8s/README.md — having them in the repo rather than in a PR comment is what makes them sayable out loud on Wednesday.

Approving. Ping me when the drill output is up and I will read it.

[devinder-dev]

Applied and drilled, on 1 Nov. Thanks for the careful review, especially the digest check: that's
exactly the kind of thing that looks wrong for good reasons.

Applied (runbook section 6, steps 1–4, from main after the merge), then checked from outside:

• local-path-retain with reclaim Retain, and local-path stays the default; the claim is
  Bound to a Retain disk.
• postgres-0's log: 10-create-relay.sh: created the login relay and its database.
• A login from a separate pod with the Secret's url (never printed) works:
  relay | relay | super=false | createrole=false | createdb=false. The same url with a wrong
  password is refused: FATAL: password authentication failed for user "relay". The check pod
  passed restricted and was deleted afterwards.
• The Service is headless ClusterIP. From outside, 5432 is closed, and 443 answers as the
  control, so the check itself works.

The deletion drill, in the order of runbook 6a, with a marker row in the admin database (not
in relay's). The script stopped at the first surprise; there was none:

== 0. start: 2026-11-01T14:20:50Z
   disk pvc-123cc608-4860-4e67-86a6-ea0677bf9560 (10Gi, ReadWriteOnce), reclaim=Retain
== 1. wrote a marker row in the admin database
   marker survived: "written 2026-11-01T14:20:51Z before the drill"
== 2. delete pod postgres-0
   a new pod (uid 3785cf20, was d36feefe)
   marker survived: "written 2026-11-01T14:20:51Z before the drill"
== 3. delete the StatefulSet, then apply it again
   the claim stayed: Bound, disk pvc-123cc608-4860-4e67-86a6-ea0677bf9560
   marker survived: "written 2026-11-01T14:20:51Z before the drill"
== 4. scale to 0, delete the claim (Retain must keep the disk)
   postgres-0 is gone
   after deleting the claim, the disk is still there: pvc-123cc608-4860-4e67-86a6-ea0677bf9560 Released
   freed it for one new claim (removed only its old claimRef): Available
   new claim: Bound, disk pvc-123cc608-4860-4e67-86a6-ea0677bf9560
   the same disk as in step 0
   marker survived: "written 2026-11-01T14:20:51Z before the drill"
== 5. clean up and final state
   marker table dropped
   sts postgres: 1/1 ready
   disk pvc-123cc608-4860-4e67-86a6-ea0677bf9560: Bound, reclaim=Retain
== done: 2026-11-01T14:21:07Z

Afterwards, the relay login is still there (relay|f), and the marker table is gone. So Retain
survives a deleted pod, a deleted StatefulSet and a deleted claim. As you say for Phase 6, it is
not a backup: it doesn't help if the disk itself is lost.

Your wording fix: you're right, RELAY_DB_PASSWORD stays in the pod's env, and 6b depends on
it. I didn't push it here, so your approval stays valid; it goes into #26 with the comment you
suggested.

Your question about 1Gi: yes, Phase 8 revisits it with numbers, not a guess. Today's budget is
48 connections in normal use (66 worst case) of the 100. With the default shared_buffers
(128 MB) and work_mem (4 MB per sort or hash, per query), that stays well under 1Gi at our
query sizes. The k6 test will measure it (kubectl top and pg_stat_activity during the peak),
and we raise the limit if the peak passes about 70 % (the VPS has room). An OOM kill wouldn't
lose committed data: Postgres replays its WAL on the restart. But it would cut every connection,
so we don't want to find it in the demo.