Skip to content

infra: add the dashboard manifests for cdn.myrelay.tech - #24

Merged
AbdikhaniAbdirashid merged 4 commits into
mainfrom
infra/dashboard-on-cdn
Nov 4, 2026
Merged

AbdikhaniAbdirashid merged 4 commits into
mainfrom
infra/dashboard-on-cdn

Conversation

@AbdikhaniAbdirashid

@AbdikhaniAbdirashid AbdikhaniAbdirashid commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

What

Step 5.2: the Deployment, Service and Ingress that put the dashboard on
https://cdn.myrelay.tech. One file in k8s/app/, so relay-deploy applies it on every merge
to main, with the image tag replaced by the commit's SHA.

Same shape as the API's manifests, with four differences:

Why
Port 8080, not 80 nginx-unprivileged runs as uid 101 and can't bind a low port
Readiness + liveness probes on /index.html a rollout never sends traffic to a pod whose nginx hasn't started
cpu: 10m, memory: 24Mi requests, 64Mi limit the VPS is shared with Postgres, the API and Grafana; serving static files needs almost nothing, and the requests tell the scheduler this pod is cheap
cdn.myrelay.tech + letsencrypt-prod its own host and certificate

Two pods with maxUnavailable: 0, and the image tagged by git SHA rather than latest, so a
rollback can name an exact build.

Changed since the draft

  • Moved from k8s/dashboard/ into k8s/app/. As written, relay-deploy would never have
    applied it: it only globs "$REPO"/k8s/app/*.yaml. The pipeline would have gone green and
    deployed nothing.
  • Image tag PLACEHOLDER_SHA -> __GIT_SHA__. relay-deploy only substitutes
    __GIT_SHA__, so the tag would have stayed as literal text: ImagePullBackOff, the rollout
    never ready, and the whole commit rolled back — API included.
  • Added a 5 s preStop. Removing a pod from the Service and telling it to stop happen at
    the same moment, but Traefik needs a moment to hear about the removal. Without the pause nginx
    is already shutting down while Traefik still routes here, and those requests fail.
    maxUnavailable: 0 doesn't cover this: it guarantees a replacement is ready before the old
    pod goes, but says nothing about the old pod's last seconds. The API has the same 5 s. This
    was the one piece of our four-part zero-downtime recipe the dashboard was missing.
    terminationGracePeriodSeconds stays at its default 30 s, which comfortably holds the 5 s;
    nginx serves static files, so there is nothing to drain afterwards.
  • Labels now follow the API's convention: app.kubernetes.io/name: relay-dashboard instead
    of app: relay-dashboard, on the Deployment, its selector, the pod template and the Service
    selector. A Deployment's selector is immutable once it exists, so this is free now and a
    delete-and-recreate later. Checked that all three still agree — a selector that doesn't match
    the pod labels gives a Service with no endpoints, which looks like a broken app rather than a
    broken label.

Pod Security: this would have been refused

The file was first written on 26 Oct, before #21. Checked it against the restricted level that
#21 puts on namespace relay, and it was missing two of the four required fields —
runAsNonRoot and seccompProfile. The image already runs as uid 101, but Kubernetes doesn't
look inside the image: being non-root has to be declared or the API server refuses the pod.

Verified rather than reasoned about, in a throwaway namespace ps-test carrying the same labels
(restricted, enforce-version: v1.36), deleted afterwards:

Manifest Result
This PR's file deployment/service/ingress created (server dry run) — no warnings
Same file with the two fields removed (control) Warning: would violate PodSecurity "restricted:v1.36": runAsNonRoot != true …, seccompProfile …

The control is the point. A clean dry run on its own doesn't show the check is running — same
trap as the nc -G check in #17 and the externalIPs == null rule in #21. The control names
both fields, so the pass means something.

One note on Pod Security worth writing down somewhere: it enforces on Pods, not on
Deployments. Applying a violating Deployment succeeds with a warning, and then the pods never
appear. So kubectl apply returning cleanly is not evidence — the warning line is.

Re-checked after the move and the preStop addition, against namespace relay itself with the
commit's real SHA substituted:

deployment.apps/relay-dashboard created (server dry run)
service/relay-dashboard created (server dry run)
ingress.networking.k8s.io/relay-dashboard created (server dry run)

The image exists

#30 is merged, so the pipeline now builds, smoke-tests and pushes relay-dashboard:<sha> on
every merge to main — not only when the dashboard changed, because the tag has to exist for
whatever commit the deploy applies. Confirmed against the registry rather than trusting the
green tick:

relay-dashboard:c145056 -> HTTP 200

That is also why the image and the manifest are two PRs and not one: the tag has to exist before
a manifest references it, or the first rollout fails on a missing image.

Not in this PR

  • readOnlyRootFilesystem stays false: nginx writes its cache and pid file under /var, so
    turning it on means emptyDir mounts for both. restricted doesn't require it. Happy to add it
    if you'd rather — your Postgres does the equivalent.
  • Nothing else. The label mismatch that used to be listed here is fixed in this PR instead
    of deferred (see "Changed since the draft").

For the reviewer

  • Are the resource requests sensible next to Postgres, or should I leave more headroom?
  • Liveness probe on a static file server: worth having, or the same objection as Postgres?
  • The certificate is new, so expect a minute or two of a self-signed warning on
    cdn.myrelay.tech right after the merge while cert-manager finishes the ACME challenge.

@devinder-dev

Copy link
Copy Markdown
Owner

Early feedback while it's a draft, so you don't build further on the one part that needs to move.

The Pod Security check is exactly right. Proving the clean dry run with a control that must
be refused (the same file without runAsNonRoot and seccompProfile) is the lesson from #17 and
#21, and I'll use it as an example at the defense. Since #21 merged, relay really enforces
restricted:v1.36, and I tested the cdn path live: a staging certificate for cdn.myrelay.tech
in relay became Ready in 29 s, with its solver Service as ClusterIP. So letsencrypt-prod will
work for your Ingress.

The main change: this belongs in k8s/app/, deployed by the pipeline, not applied by hand.
That's what we agreed in #13 (the dashboard manifests go in k8s/app/, and every merge to main
builds both images and deploys once), and it's the rule in k8s/README.md. Three reasons it
matters:

  • G2/G3: "every merge tested, built and deployed" has to cover the dashboard too. A
    hand-applied dashboard drifts from git, and it's the first thing an examiner would ask about.
  • The deployer can already do it: Deployments, Services and Ingresses in relay are exactly
    its rights. k8s/platform/ and k8s/data/ are hand-applied because they hold things it
    mustn't touch (permissions, the database). The dashboard isn't one of those.
  • Rollback: relay-deploy rolls back manifests and images together by commit. A
    hand-applied dashboard sits outside that.

What that needs, all in this PR:

  1. k8s/app/dashboard.yaml, with namespace: relay in each object's metadata, as the API's
    files have it.
  2. Image tag __GIT_SHA__, not PLACEHOLDER_SHA: relay-deploy replaces exactly that
    string with the commit it deploys.
  3. pipeline.yml builds and pushes relay-dashboard:<sha> on every merge, next to
    relay-api. The rule in k8s/README.md is that every image exists for every commit, so a
    rollback to any commit finds its images. This is the open item from review 2: the image build
    and the manifest must land in the same PR. If the manifest lands first, the next deploy
    pulls an image that doesn't exist, and rolls back.
  4. The Dockerfile from feat: build the four dashboard screens on mock data #9 before or together with this, since the build needs it.

Smaller things, take or leave:

  • preStop: sleep: {seconds: 5}, like the API. Without it a terminating pod can still get a
    request or two while Traefik removes it, which shows up as a few failed requests in the
    zero-downtime test (VG2).
  • runAsUser: 101 written out. runAsNonRoot makes the kubelet check the user when the
    container starts (a dry run can't see that), and naming it doesn't depend on the image's
    USER line, as the API's runAsUser: 1000 does.
  • automountServiceAccountToken: false: nginx never talks to Kubernetes.
  • Labels: the API uses app.kubernetes.io/name: relay-api. app.kubernetes.io/name: relay-dashboard would match, and gives Grafana one label to group by later.
  • readOnlyRootFilesystem: fine to leave false, as you explained. If you try it,
    nginx-unprivileged may only need an emptyDir on /tmp (I haven't verified which paths it
    writes; worth one docker run --read-only test).

Happy to pair on the pipeline.yml part, since that file is mine and it has the deploy job's
time budget in it.

@devinder-dev

Copy link
Copy Markdown
Owner

I had #24 reviewed (against 664345b, with #9's Dockerfile and nginx.conf at d245a42)
and checked the main findings myself. The manifest itself is in good shape: the ports, probes,
TLS wiring and Pod Security fields all line up, and #9's nginx config does the right things (a
CSP that allows api.myrelay.tech, index.html revalidated, hashed assets immutable, and a
missing /assets/*.js answering 404 instead of index.html).

Still open from my earlier comment (no rush, it's a draft): moving it to
k8s/app/dashboard.yaml, __GIT_SHA__, the pipeline building relay-dashboard, namespace: relay on each object, preStop, runAsUser: 101, automountServiceAccountToken: false, and
the app.kubernetes.io/name label.

New, for when you rework it

  1. Check the new image is really public, once. ADR 0004 saw relay-api public from its
    first push, but that doesn't prove it for a second package, and GitHub's docs describe new
    packages as private. Right after the first relay-dashboard push, try an anonymous pull of
    that exact SHA (like ADR 0004 did). If it's private, the first deploy can't pull it and rolls
    back.

  2. A correction to what I wrote on feat: build the four dashboard screens on mock data #9 about assets during a rollout. "The new image also
    carries the previous build's assets" only fixes half of it: old HTML reaching a new pod. New
    HTML reaching an old pod still gets a 404, because the old pod doesn't have the new files.
    So the realistic choices are:

    • (a) a sticky cookie: 5 Traefik annotations on the Service (not the Ingress). Our
      cluster runs Traefik 3.7.13, which supports it. It keeps a browser on one pod, so its
      HTML and assets come from the same build. It reduces the problem, not removes it: the pod
      can still go away mid-session;
    • (b) saying it honestly: a page load during the few seconds of a dashboard rollout can
      fail, and a reload fixes it.

    My suggestion: (b) for Wednesday, and (a) if there's time to test it (a cookie is set, and
    requests stay on one pod while the pods are replaced). The annotations are below. What do
    you think?

    annotations:
      traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
      traefik.ingress.kubernetes.io/service.sticky.cookie.name: relay_dashboard
      traefik.ingress.kubernetes.io/service.sticky.cookie.secure: "true"
      traefik.ingress.kubernetes.io/service.sticky.cookie.httponly: "true"
      traefik.ingress.kubernetes.io/service.sticky.cookie.samesite: lax
  3. relay-deploy: my part. It already waits for every Deployment it applies, so the
    dashboard's rollout is covered. But its test harness never tries "API fine, dashboard
    fails", so nothing proves that triggers the rollback. I'll add those cases when your
    pipeline change lands.

Nice to have

  • The comment about writable paths: the unprivileged image writes its pid and temp files under
    /tmp, not /var. If you try readOnlyRootFilesystem: true, an emptyDir on /tmp is the
    candidate (test it with the real image first).
  • 24Mi/64Mi is probably fine for static files, but worker_processes auto starts 4 workers on
    our 4 vCPUs. Measure the memory once under a few parallel requests, or set
    worker_processes 1;.
  • A rollback to a commit from before the dashboard can't delete it (the deployer can't delete,
    as k8s/README.md says). Fine, just good to know at the defense.

AbdikhaniAbdirashid added a commit that referenced this pull request Nov 3, 2026
Two jobs, so the dashboard gets the same lane in the pipeline the API
already has:

```
dashboard-test ──► dashboard-image ──┐
api-test ──────────► api-image ──────┤──► deploy
deploy-script-test ──────────────────┘
```

`dashboard-test` runs the 35 Playwright tests that until now only ever
ran on my laptop.
`dashboard-image` builds `relay-dashboard`, smoke-tests it, and pushes
it on main only.
`deploy` waits for both image jobs: every image a commit's manifests ask
for has to be on
GHCR before the server is told to pull them.

## Why the smoke test checks two things, not one

A status check on `/index.html` looks sufficient and is not.
`nginx.conf` hands every
unknown path to `index.html` (it has to, or `/deliveries` would 404 on
reload), so that
request answers 200 almost no matter what is in the image.

I built a deliberately broken image — the real one with `assets/`
deleted — to see which
check catches it:

```
/index.html                 → 200   passes
/assets/index-BimGaWWm.js   → 404   fails
```

`location /assets/` has `try_files $uri =404` and no fallback, so the
bundle that
`index.html` names is the request that proves the build is really inside
the image. The
step reads that path out of `index.html` rather than hardcoding it,
because the filename
is content-hashed and changes on every build.

Run against the real image the step exits 0; against the broken one it
exits 1 with
`::error::/assets/index-BimGaWWm.js answered 404: the build is not in
the image`.

## Details worth a second look

- **`cache-from: type=gha,scope=dashboard`.** The scope defaults to
`buildkit` for
everyone, and Docker's docs are explicit: build two images without it
and each
overwrites the other's cache. The API keeps the default, so the two no
longer collide.
Naming the API's scope `api` would be tidier but means touching your
job, so I left it.
- **`npm ci`, Node from `dashboard/.nvmrc`.** The dashboard is npm and
Node 24, not Bun.
`npm install` would be free to resolve versions `package-lock.json`
never pinned.
- **`npx playwright install --with-deps chromium`.** The runner has no
browser. Without
  this line the job fails on its first run with "browser not found".
- **`curl -fs`, not `-fsS`, inside the wait loop.** A refused connection
while nginx
starts is expected, and printing it would make a green run look broken.
- **Comparing `%{http_code}` by hand.** `curl -s` exits 0 on a 404, so
`set -e` does not
  catch it.

## One request, and one ordering note

**Please add `dashboard-test` as a required check** on main. A green job
stops nobody from
merging; only branch protection does, and only you can change it.

The comment in `dashboard-test` names **decision 0005**, which is still
open as #23. Merging
that first keeps the reference from dangling.

## How it was checked

- `npm ci`, `npm run lint`, `npm run build`, `npm run test:e2e` locally:
**35 passed** in 19.5 s.
- The image built and served: `/index.html` 200, `/deliveries` 200
(fallback), uid 101.
- The smoke step extracted straight out of this YAML and run against
both a good and a
  broken image, with the results above.
- `pipeline.yml` parses, and `deploy` really lists all three jobs in
`needs`.

## Not in this PR

- `k8s/app/dashboard.yaml` — #24, which goes out of draft once this is
merged and the first
  `relay-dashboard` image is on GHCR.
- Uploading the Playwright HTML report and traces as artifacts on
failure. The `github`
reporter annotates failing tests on the PR, which is enough until a red
test is hard to read.

Co-authored-by: Abdikhani Abdirashid <Abdikhani_abdirashd@hotmail.com>
@AbdikhaniAbdirashid
AbdikhaniAbdirashid marked this pull request as ready for review November 3, 2026 21:04
@devinder-dev

Copy link
Copy Markdown
Owner

Nice work: the move to k8s/app/, __GIT_SHA__, the preStop and the label change all check out, and the Pod Security control is exactly right. I also checked the image itself (c145056, pulled anonymously). Four small things before I approve, all in this one file:

  1. namespace: relay on each object, like the API's files. Not a deploy breaker (relay-deploy takes the namespace from the deployer's kubeconfig), but it's what the other k8s/app/ files do, and a hand-run kubectl apply with the admin kubeconfig would otherwise land in default.
  2. automountServiceAccountToken: false in the pod spec, like the API: nginx never talks to Kubernetes, so the pod shouldn't carry a token.
  3. The /var comment isn't right for this image. nginx-unprivileged puts the pid file and every temp path under /tmp (pid /tmp/nginx.pid;, client_body_temp_path /tmp/client_temp;, …). I ran c145056 with --read-only --tmpfs /tmp: /index.html and the hashed bundle both answer 200 (the only log line is the IPv6 entrypoint script saying it can't edit the config, at info level). So either fix the comment, or turn it on with one emptyDir on /tmp; your call. With it, the dashboard matches the API's readOnlyRootFilesystem: true.
  4. runAsUser: 101, optional: it works today because the image's USER is numeric, and writing it down doesn't depend on that.

Your three questions:

  • Resources: fine. 400 requests, 20 at a time, peaked at 26 MiB of the 64 MiB limit, on my laptop, where the container saw 10 CPUs (so worker_processes auto starts more workers there than the VPS's 4). kubectl top pod after the deploy gives the real number.
  • Liveness on nginx: keep it. Restarting a stuck static file server is cheap and safe; the objection for Postgres was that a liveness restart under load kills the database and its open transactions, which doesn't apply here.
  • The certificate: fine. The staging test on 31 Oct got a certificate for cdn.myrelay.tech in 29 s.

Still open from my 1 Nov comment, assets during a rollout: for Wednesday I'd say it honestly (a page load during the few seconds of a dashboard rollout can fail, and a reload fixes it), and try the sticky cookie after. OK with you?

On my side: relay-deploy waits up to 180 s per Deployment, so with the dashboard the worst case (migrations, two rollouts, then a rollback) is about 18 minutes. That's inside the job's 40, but past what the 15-minute flock comment promises. I'll fix that and add the harness case "API fine, dashboard fails → rollback" in my own PR.

@AbdikhaniAbdirashid

Copy link
Copy Markdown
Collaborator Author

Ready for review when you get a moment - one file, 116 lines, and it's the last thing before the dashboard is live on cdn. Starting on #31 now; #32 and #33 after it since they're stacked.

@devinder-dev devinder-dev left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving so the dashboard can go live today. None of the four points from last night breaks anything (I checked each one: relay-deploy takes the namespace from the deployer's kubeconfig, the image runs as uid 101 by itself, and nginx only writes to /tmp), so they can come in a small follow-up PR: namespace: relay on each object, automountServiceAccountToken: false, the /tmp comment (or readOnlyRootFilesystem: true with an emptyDir on /tmp), and runAsUser: 101.

Merge whenever you're ready. I'm merging #32 and #33 in between, so the pipeline may deploy them one after the other; it runs main's deploys one at a time.

@AbdikhaniAbdirashid
AbdikhaniAbdirashid merged commit 26e179f into main Nov 4, 2026
6 checks passed
@AbdikhaniAbdirashid
AbdikhaniAbdirashid deleted the infra/dashboard-on-cdn branch November 4, 2026 08:55
devinder-dev added a commit that referenced this pull request Nov 4, 2026
…low-ups (#34)

## What

Follow-ups from the reviews of #24, #32 and #33. One code change (the deploy script's lock
wait), one test change (the deploy harness learns about the dashboard), and three docs changes.

## Changes

- **`test-relay-deploy.sh`: the dashboard is now in the deploy tests.** Until now the fake
  `kubectl apply` only ever reported `relay-api`, so nothing proved that a broken dashboard rolls
  the commit back. The fake now reports `relay-dashboard` too when `k8s/app/dashboard.yaml`
  exists, records each `rollout status` it is asked for, and can fail the dashboard alone.
  - **18.** Two Deployments, both fine → exits 0, waited for each.
  - **19.** The API is fine, the dashboard never gets ready → exits 1, the last good commit is
    deployed again (both Deployments waited for again), last good unchanged, pods printed.
  - Negative control: a copy of `relay-deploy` that waits only for the first Deployment fails 6
    checks, including "exits 0" with a broken dashboard and recording it as last good.
- **`relay-deploy.sh`: lock wait 15 → 30 minutes.** The comment promised 15 minutes was longer
  than the worst case, which stopped being true with a second Deployment: a deploy plus its
  rollback is one migrate Job (up to 180 s) plus up to 180 s per Deployment, twice. That's about
  18 minutes with the API and the dashboard, and about 24 with the worker (Phase 4). The deploy
  job's own limit is 40 minutes.
- **`docs/CONTEXT.md`:** DNS rebinding written down as a known gap with where it closes (the
  worker connects only to the checked address, behind a NetworkPolicy; nothing is sent before
  Phase 4), from the #32 review. The CORS line now names both origins, as `docs/api.md` does,
  from the #33 review. A link to decision 0007.
- **Decision 0007 (new):** idempotent events rely on the unique constraint and READ COMMITTED,
  with the alternatives and the check from 4 Nov (under REPEATABLE READ the race test goes red),
  from the #33 review.
- **Decision 0006:** Proposed → Accepted.

## Tested

- `bash infra/server/test-relay-deploy.sh infra/server/relay-deploy.sh`: 56 passed, 0 failed
  (49 before + 7 new checks).

## After merge

`relay-deploy` is installed by hand and owned by root, so install it again so the server matches
git (`k8s/README.md`):

```
ssh relay 'cat > /tmp/relay-deploy' < infra/server/relay-deploy.sh
ssh -t relay 'sudo install -o root -g root -m 755 /tmp/relay-deploy /usr/local/bin/relay-deploy'
```

Then compare `sha256sum /usr/local/bin/relay-deploy` on the server with `shasum -a 256
infra/server/relay-deploy.sh` on the laptop.
devinder-dev added a commit that referenced this pull request Nov 4, 2026
…n, namespace) (#35)

## What

The four follow-ups from my review of #24, so you only have to review them. One file,
`k8s/app/dashboard.yaml`; the dashboard then matches the API's hardening.

| Change | Why |
|---|---|
| `namespace: relay` on the Deployment, Service and Ingress | as in the API's files; a hand-run `kubectl apply` with the admin kubeconfig would otherwise land in `default` |
| `automountServiceAccountToken: false` | nginx never talks to the Kubernetes API, so the pod carries no token |
| `readOnlyRootFilesystem: true` + an `emptyDir` on `/tmp` (16Mi) | nothing in the image may change at runtime, as for the API. `nginx-unprivileged` writes only under `/tmp` (`pid /tmp/nginx.pid`, `client_body_temp_path /tmp/client_temp`, …), not `/var` as the old comment said |
| `runAsUser: 101` | written out, so it doesn't depend on the image's `USER` line |

## Checked

With the real image (`relay-dashboard:c145056`, pulled anonymously), run the way the pod will run:
`--user 101 --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges`:

```
/index.html                → 200
/assets/index-BimGaWWm.js  → 200
/deliveries (fallback)     → 200
```

The only non-access log line is the IPv6 entrypoint script saying it can't edit the config
(`info:`, then "Configuration complete; ready for start up"). The manifest parses, and all three
objects carry `namespace: relay`.

Server dry run for Pod Security (from the laptop, any commit's image tag works):

```
sed "s/__GIT_SHA__/$(git rev-parse origin/main)/" k8s/app/dashboard.yaml | ssh relay 'kubectl apply --dry-run=server -f -'
```

If the pods ever failed at runtime anyway, `relay-deploy` rolls the commit back, which #34 now
tests for the dashboard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants