Repository navigation
infra: guard namespace relay (Pod Security restricted, ClusterIP-only Services) - #21
Merged
Merged
Conversation
AbdikhaniAbdirashid
approved these changes
Oct 31, 2026
AbdikhaniAbdirashid
left a comment
AbdikhaniAbdirashid
left a comment
Collaborator
There was a problem hiding this comment.
Traced the CEL against all seven cases and they behave as described — including the headless one, which step 2.3 needs, and the !has() ordering that your earlier null version got wrong. Worth saying why that version was so dangerous: it compiled, it refused the bad cases, and with failurePolicy: Fail an unset field errors into a deny, so cases 1–3 failed for a reason that looked like the rule working. Only the positive tests could have caught it.
Order is right: issuers before the policy. The other way round, cert-manager's next renewal in relay creates its solver Service, gets refused, and certificates quietly stop renewing.
Checked the current state before approving:
$ kubectl get svc -n relay -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,EXTIP:.spec.externalIPs
NAME TYPE EXTIP
relay-api ClusterIP <none>
One Service, already compliant — so nothing pre-existing is grandfathered in by the CREATE/UPDATE scope.
Two smaller things I liked: serviceType: ClusterIP went on both issuers, not just staging, and enforce-version: v1.36 is pinned rather than latest, so a k3s upgrade can't tighten the rules under us mid-project.
Two limits worth a line in k8s/README.md, neither blocking:
- The policy only sees Services. An
IngressRouteTCP(Traefik's CRD) exposes a TCP port without one, so it wouldn't be evaluated. - It only guards
relay. A NodePort Service in another namespace, pointed at a pod IP by hand, reaches the same pod from outside the guarded room.
The deploy account can reach neither — it's confined to relay and can't touch Namespace labels or policies — so automation can't trip either one. They need a person doing it deliberately. I'd rather we have them written down than discover them under questioning on Wednesday.
Approving. Ping me when the post-merge checks are done and I'll look at the output.
This was referenced Oct 31, 2026
Owner
Author
|
Applied after the merge, in the agreed order, each file read from Checks (server dry runs, nothing saved), each next to a positive control:
Live proof: a throwaway staging Certificate
Then the test Certificate and its Secret were deleted; only the objects from before remain, and Your two limits (IngressRouteTCP, other namespaces) and your runbook half-sentence from #20 go |
devinder-dev
added a commit
that referenced
this pull request
Nov 1, 2026
Rebased onto `main` after #20 and #21. ## What Steps 2.1–2.3 of Phase 2: Postgres 18.6 on a disk that survives deletion. Nothing here is applied by the pipeline: the `deployer` account can't create StorageClasses, Secrets or StatefulSets, on purpose. | Step | File | What | |---|---|---| | 2.1 | `k8s/platform/storageclass-retain.yaml` | A second StorageClass, `local-path-retain`: like k3s's `local-path`, but `reclaimPolicy: Retain`. Deleting the claim (or the namespace) keeps the data folder instead of `rm -rf`-ing it | | 2.2 | `infra/server/bootstrap-secrets.sh` | Creates the Secrets `postgres-auth` (superuser password) and `relay-db` (the app's password + its `DATABASE_URL`) on the server, **only if missing**, from `openssl rand -hex 32`. Prints only `created`/`kept`; values never on a command line, in output or in a file | | 2.3 | `k8s/data/postgres.yaml` | Headless Service `postgres` (ClusterIP only) + StatefulSet: `postgres:18.6` pinned by digest, uid 999, Pod Security `restricted`, read-only root, `max_connections=100`, readiness `pg_isready` over TCP, no liveness probe, 60 s grace, a 10Gi claim on `local-path-retain` | | 2.3 | `k8s/data/initdb/10-create-relay.sh` | Runs once, on an empty volume: creates the login `relay` (not a superuser) and the database `relay` owned by it. Password via psql `\getenv` + `:'pw'`, and `log_min_error_statement = panic` so a failing statement can't write the password to Postgres's log | **Docs:** - `k8s/README.md`: a row for `k8s/data/` ("never the pipeline"), and the StorageClass in the platform row. Also your two limits from #21: the policy only sees Services (a Traefik `IngressRouteTCP` wouldn't be checked), and both guards cover only `relay`. - `infra/server/README.md`: section 6 becomes the real data section: - the apply order (StorageClass → secrets → ConfigMap → Postgres), and why a login check must come from another pod; - **6a** re-attaching a kept disk: scale to 0 first, then re-bind with `volumeName`; never remove finalizers; - **6b** a lost Secret with a kept disk: `ALTER ROLE` from the pod's own env with psql `\getenv`. Tested: a password with `'`, `;`, `$`, `"` works, the old one is refused, and there are 0 lines of it in the log; - **6c** a first start that failed halfway. - Your half-sentence from #20 in "Reboots": the `/version == origin/main` check is valid because every merge is built and deployed. ## Why these choices - **Retain, in our own class:** `local-path` can't be changed (`reclaimPolicy` is immutable, and k3s re-applies its own copy). Retain protects against `kubectl` mistakes, not against losing the server's disk; only backups do that (Phase 6). - **A StatefulSet:** a Deployment's rolling update would start a second Postgres on the same data folder. - **Applied by an admin:** a merged PR can't change the database's major version or its disk. - **The app is not a superuser:** `POSTGRES_USER=relay` would make it one, able to run shell commands on the database server (`COPY … TO PROGRAM`). - **No liveness probe:** it could kill Postgres in the middle of a crash recovery, in a loop. - **Readiness over TCP:** on the first start a temporary server answers only on the local socket. ## Tested before the PR Locally in Docker (the same digest, uid 999, read-only root, no capabilities): - a password containing `'`, `;`, `"`, `\`, `$` and an attempted `CREATE TABLE pwned()`: the login works with exactly that password, one character less is refused, and no `pwned` table exists; - a forced failing `CREATE ROLE`: the password appears **0** times in Postgres's log. Control without the `SET`: the full statement and password are logged. `bootstrap-secrets.sh` in a throwaway namespace: first run `created` ×2, second run `kept` ×2 with unchanged `resourceVersion`s, keys `password` / `password,url`, no 64-hex string in any output. A fake-kubectl test: when the check can't reach the server, it stops before any `create`. Postgres in a throwaway namespace `pg-test` with the **same Pod Security labels** (`restricted`, v1.36), filled by `bootstrap-secrets.sh`, then deleted: | Check | Result | |---|---| | Pod / PVC / Service | Ready, 0 restarts; Bound; ClusterIP `None`, no NodePort | | Pod Security | No warning on apply, 0 PodSecurity events | | Version, checksums, max_connections | 18.6, on, 100 | | Login from **another pod** with the Secret's `url` | `logged in as relay in relay` | | Wrong password from another pod | `password authentication failed` | | `relay`'s flags (superuser, createrole, createdb, replication) | all false | | `COPY … TO PROGRAM` as `relay` | `permission denied` | | Delete the pod | "received fast shutdown request" (the image's SIGINT reaches Postgres), shut down in 26 ms, back with the same data | Found while testing: the image's `pg_hba.conf` trusts the local socket and `127.0.0.1` **without a password**; only connections from other pods need one. So login checks run from a separate pod. Only admins can `kubectl exec` into `postgres-0` (the deployer has no exec). ## After merging (an admin, in this order, output posted here) 1. `k8s/platform/storageclass-retain.yaml` → `kubectl get sc` shows `local-path-retain … Retain`, `local-path` still the default. 2. `ssh relay 'bash -s' < infra/server/bootstrap-secrets.sh` → `created` ×2. 3. The ConfigMap `postgres-init` from `10-create-relay.sh` (the command is in the header of `postgres.yaml`), then `postgres.yaml`. 4. The checks above, now in `relay`, plus from outside: `nc -z -G 3 81.88.23.71 443` succeeds and `nc -z -G 3 81.88.23.71 5432` fails. 5. **The deletion drill, before any real data:** a marker row survives deleting the pod, the StatefulSet, and the PVC (Postgres scaled to 0 first, then the kept PV re-attached with `volumeName`). ## For the reviewer - Anything that should be applied by the pipeline instead of by hand, or the other way round? - Is "trusted inside the pod, password from outside" acceptable to you? - The 10Gi is only a label: local-path doesn't enforce sizes. Disk usage is watched in Phase 7.
AbdikhaniAbdirashid
pushed a commit
that referenced
this pull request
Nov 3, 2026
## What Closes Phase 2 in `docs/CONTEXT.md`'s "Where we are": what is live, and the tests that proved it. ## Phase 2, as merged and applied - **#21:** guards on namespace `relay` (Pod Security `restricted`, ClusterIP-only Services). - **#22:** Postgres 18.6 on a disk that is kept on delete, applied by hand, with the `relay` login. - **#27** (replaces #25): migration 001, and the migrate Job before every rollout (ADR 0006). - **#26:** the API waits for the database before it listens; Postgres in the tests. ## The live evidence, posted on the PRs - **#22:** a login from another pod as `relay`, a wrong password refused, 5432 closed from outside, and the deletion drill: the data survived deleting the pod, the StatefulSet and the claim. - **#27:** the Role checked both ways, the installed script identical to git, and migration 001 applied by its Job. Two negative tests: - a Job that can't start fails within 150 s; - a rollback to Phase 1's release under load: 123,904 requests, 0 failed. - **#26 (live 2 Nov):** both pods log `✅ database answered` before they listen; `/health/ready` answers 200 from outside. Docs only.
AbdikhaniAbdirashid
added a commit
that referenced
this pull request
Nov 4, 2026
## What Step 5.2: the Deployment, Service and Ingress that put the dashboard on `https://cdn.myrelay.tech`. One file in `k8s/app/`, so `relay-deploy` applies it on every merge to main, with the image tag replaced by the commit's SHA. Same shape as the API's manifests, with four differences: | | Why | |---|---| | Port 8080, not 80 | `nginx-unprivileged` runs as uid 101 and can't bind a low port | | Readiness + liveness probes on `/index.html` | a rollout never sends traffic to a pod whose nginx hasn't started | | `cpu: 10m`, `memory: 24Mi` requests, 64Mi limit | the VPS is shared with Postgres, the API and Grafana; serving static files needs almost nothing, and the requests tell the scheduler this pod is cheap | | `cdn.myrelay.tech` + `letsencrypt-prod` | its own host and certificate | Two pods with `maxUnavailable: 0`, and the image tagged by git SHA rather than `latest`, so a rollback can name an exact build. ## Changed since the draft - **Moved from `k8s/dashboard/` into `k8s/app/`.** As written, `relay-deploy` would never have applied it: it only globs `"$REPO"/k8s/app/*.yaml`. The pipeline would have gone green and deployed nothing. - **Image tag `PLACEHOLDER_SHA` -> `__GIT_SHA__`.** `relay-deploy` only substitutes `__GIT_SHA__`, so the tag would have stayed as literal text: `ImagePullBackOff`, the rollout never ready, and the whole commit rolled back — API included. - **Added a 5 s `preStop`.** Removing a pod from the Service and telling it to stop happen at the same moment, but Traefik needs a moment to hear about the removal. Without the pause nginx is already shutting down while Traefik still routes here, and those requests fail. `maxUnavailable: 0` doesn't cover this: it guarantees a replacement is ready before the old pod goes, but says nothing about the old pod's last seconds. The API has the same 5 s. This was the one piece of our four-part zero-downtime recipe the dashboard was missing. `terminationGracePeriodSeconds` stays at its default 30 s, which comfortably holds the 5 s; nginx serves static files, so there is nothing to drain afterwards. - **Labels now follow the API's convention:** `app.kubernetes.io/name: relay-dashboard` instead of `app: relay-dashboard`, on the Deployment, its selector, the pod template and the Service selector. A Deployment's selector is immutable once it exists, so this is free now and a delete-and-recreate later. Checked that all three still agree — a selector that doesn't match the pod labels gives a Service with no endpoints, which looks like a broken app rather than a broken label. ## Pod Security: this would have been refused The file was first written on 26 Oct, before #21. Checked it against the `restricted` level that #21 puts on namespace `relay`, and it was missing two of the four required fields — `runAsNonRoot` and `seccompProfile`. The image already runs as uid 101, but Kubernetes doesn't look inside the image: being non-root has to be declared or the API server refuses the pod. Verified rather than reasoned about, in a throwaway namespace `ps-test` carrying the same labels (`restricted`, `enforce-version: v1.36`), deleted afterwards: | Manifest | Result | |---|---| | This PR's file | `deployment/service/ingress created (server dry run)` — no warnings | | Same file with the two fields removed (control) | `Warning: would violate PodSecurity "restricted:v1.36": runAsNonRoot != true …, seccompProfile …` | The control is the point. A clean dry run on its own doesn't show the check is running — same trap as the `nc -G` check in #17 and the `externalIPs == null` rule in #21. The control names both fields, so the pass means something. One note on Pod Security worth writing down somewhere: it enforces on **Pods**, not on Deployments. Applying a violating Deployment **succeeds** with a warning, and then the pods never appear. So `kubectl apply` returning cleanly is not evidence — the warning line is. Re-checked after the move and the `preStop` addition, against namespace `relay` itself with the commit's real SHA substituted: ``` deployment.apps/relay-dashboard created (server dry run) service/relay-dashboard created (server dry run) ingress.networking.k8s.io/relay-dashboard created (server dry run) ``` ## The image exists #30 is merged, so the pipeline now builds, smoke-tests and pushes `relay-dashboard:<sha>` on every merge to main — not only when the dashboard changed, because the tag has to exist for whatever commit the deploy applies. Confirmed against the registry rather than trusting the green tick: ``` relay-dashboard:c145056 -> HTTP 200 ``` That is also why the image and the manifest are two PRs and not one: the tag has to exist before a manifest references it, or the first rollout fails on a missing image. ## Not in this PR - `readOnlyRootFilesystem` stays `false`: nginx writes its cache and pid file under `/var`, so turning it on means emptyDir mounts for both. `restricted` doesn't require it. Happy to add it if you'd rather — your Postgres does the equivalent. - Nothing else. The label mismatch that used to be listed here is fixed in this PR instead of deferred (see "Changed since the draft"). ## For the reviewer - Are the resource requests sensible next to Postgres, or should I leave more headroom? - Liveness probe on a static file server: worth having, or the same objection as Postgres? - The certificate is new, so expect a minute or two of a self-signed warning on `cdn.myrelay.tech` right after the merge while cert-manager finishes the ACME challenge. --------- Co-authored-by: Abdikhani Abdirashid <Abdikhani_abdirashd@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Step 2.0 of Phase 2: two guards on namespace
relay, before Postgres and its passwords move in.restricted(labels on the Namespace ink8s/platform/deployer.yaml, pinnedto the rules of Kubernetes 1.36): the API server refuses any pod in
relaythat runs as root,is privileged, or uses host folders, network or ports.
k8s/platform/relay-service-policy.yaml, aValidatingAdmissionPolicy plus its binding): a Service in
relaymust be type ClusterIP(headless included) without
externalIPs. Checked on CREATE and UPDATE,failurePolicy: Fail,validationActions: [Deny].k8s/platform/cluster-issuers.yaml).cert-manager's default is NodePort, which opens a port on the server at every certificate
issue or renewal, and which the new policy would refuse.
k8s/README.md: the two guards and the order to apply them.Why
whole server, including every Secret. From Phase 2 the database and its passwords live in
relay.externalIPs, publish ports on the server that ufwcan't filter. Only one.com's firewall would stand between Postgres and the internet, and it
already failed once (29 Oct). The deployer account can't change Namespace labels or policies,
so it can't switch either guard off.
Tested before the PR
no warnings, so the running API pods already comply.
vap-test(its own policy name, boundonly to that namespace, deleted afterwards):
typeleft out (like our API's Service)clusterIP: None, Postgres in step 2.3)externalIPs: []externalIPsAn earlier version of the rule compared
externalIPstonull. It compiled, but refused cases1–3 too: in Kubernetes' CEL an unset field is an error, not null. Without the positive cases
(1–3), it would have looked correct. The rule now checks
has()first.After merging (the one who merges, in this order)
cluster-issuers.yaml(dry run, then apply)relay-service-policy.yaml(dry run first: if the policy is refused, don't apply the file,because
kubectl applywould still create the binding)deployer.yaml(the labels)accepted (dry runs);
externalIPsServices are refused; ClusterIP andheadless are accepted;
validationActions: [Deny].cdn.myrelay.techinrelay. The solverpod must start, with a ClusterIP Service, and the certificate must become Ready. Then the
test certificate and its Secret are deleted.
For the reviewer
relaythat therestrictedlevel would break? The API's pod spec alreadycomplies; cert-manager v1.21.2's solver pod is built to comply (step 5 proves it live).