Repository navigation
infra: add the dashboard manifests for cdn.myrelay.tech - #24
Conversation
|
Early feedback while it's a draft, so you don't build further on the one part that needs to move. The Pod Security check is exactly right. Proving the clean dry run with a control that must The main change: this belongs in
What that needs, all in this PR:
Smaller things, take or leave:
Happy to pair on the |
|
I had #24 reviewed (against Still open from my earlier comment (no rush, it's a draft): moving it to New, for when you rework it
Nice to have
|
Two jobs, so the dashboard gets the same lane in the pipeline the API
already has:
```
dashboard-test ──► dashboard-image ──┐
api-test ──────────► api-image ──────┤──► deploy
deploy-script-test ──────────────────┘
```
`dashboard-test` runs the 35 Playwright tests that until now only ever
ran on my laptop.
`dashboard-image` builds `relay-dashboard`, smoke-tests it, and pushes
it on main only.
`deploy` waits for both image jobs: every image a commit's manifests ask
for has to be on
GHCR before the server is told to pull them.
## Why the smoke test checks two things, not one
A status check on `/index.html` looks sufficient and is not.
`nginx.conf` hands every
unknown path to `index.html` (it has to, or `/deliveries` would 404 on
reload), so that
request answers 200 almost no matter what is in the image.
I built a deliberately broken image — the real one with `assets/`
deleted — to see which
check catches it:
```
/index.html → 200 passes
/assets/index-BimGaWWm.js → 404 fails
```
`location /assets/` has `try_files $uri =404` and no fallback, so the
bundle that
`index.html` names is the request that proves the build is really inside
the image. The
step reads that path out of `index.html` rather than hardcoding it,
because the filename
is content-hashed and changes on every build.
Run against the real image the step exits 0; against the broken one it
exits 1 with
`::error::/assets/index-BimGaWWm.js answered 404: the build is not in
the image`.
## Details worth a second look
- **`cache-from: type=gha,scope=dashboard`.** The scope defaults to
`buildkit` for
everyone, and Docker's docs are explicit: build two images without it
and each
overwrites the other's cache. The API keeps the default, so the two no
longer collide.
Naming the API's scope `api` would be tidier but means touching your
job, so I left it.
- **`npm ci`, Node from `dashboard/.nvmrc`.** The dashboard is npm and
Node 24, not Bun.
`npm install` would be free to resolve versions `package-lock.json`
never pinned.
- **`npx playwright install --with-deps chromium`.** The runner has no
browser. Without
this line the job fails on its first run with "browser not found".
- **`curl -fs`, not `-fsS`, inside the wait loop.** A refused connection
while nginx
starts is expected, and printing it would make a green run look broken.
- **Comparing `%{http_code}` by hand.** `curl -s` exits 0 on a 404, so
`set -e` does not
catch it.
## One request, and one ordering note
**Please add `dashboard-test` as a required check** on main. A green job
stops nobody from
merging; only branch protection does, and only you can change it.
The comment in `dashboard-test` names **decision 0005**, which is still
open as #23. Merging
that first keeps the reference from dangling.
## How it was checked
- `npm ci`, `npm run lint`, `npm run build`, `npm run test:e2e` locally:
**35 passed** in 19.5 s.
- The image built and served: `/index.html` 200, `/deliveries` 200
(fallback), uid 101.
- The smoke step extracted straight out of this YAML and run against
both a good and a
broken image, with the results above.
- `pipeline.yml` parses, and `deploy` really lists all three jobs in
`needs`.
## Not in this PR
- `k8s/app/dashboard.yaml` — #24, which goes out of draft once this is
merged and the first
`relay-dashboard` image is on GHCR.
- Uploading the Playwright HTML report and traces as artifacts on
failure. The `github`
reporter annotates failing tests on the PR, which is enough until a red
test is hard to read.
Co-authored-by: Abdikhani Abdirashid <Abdikhani_abdirashd@hotmail.com>
664345b to
ede6fdf
Compare
|
Nice work: the move to
Your three questions:
Still open from my 1 Nov comment, assets during a rollout: for Wednesday I'd say it honestly (a page load during the few seconds of a dashboard rollout can fail, and a reload fixes it), and try the sticky cookie after. OK with you? On my side: |
devinder-dev
left a comment
There was a problem hiding this comment.
Approving so the dashboard can go live today. None of the four points from last night breaks anything (I checked each one: relay-deploy takes the namespace from the deployer's kubeconfig, the image runs as uid 101 by itself, and nginx only writes to /tmp), so they can come in a small follow-up PR: namespace: relay on each object, automountServiceAccountToken: false, the /tmp comment (or readOnlyRootFilesystem: true with an emptyDir on /tmp), and runAsUser: 101.
Merge whenever you're ready. I'm merging #32 and #33 in between, so the pipeline may deploy them one after the other; it runs main's deploys one at a time.
…low-ups (#34) ## What Follow-ups from the reviews of #24, #32 and #33. One code change (the deploy script's lock wait), one test change (the deploy harness learns about the dashboard), and three docs changes. ## Changes - **`test-relay-deploy.sh`: the dashboard is now in the deploy tests.** Until now the fake `kubectl apply` only ever reported `relay-api`, so nothing proved that a broken dashboard rolls the commit back. The fake now reports `relay-dashboard` too when `k8s/app/dashboard.yaml` exists, records each `rollout status` it is asked for, and can fail the dashboard alone. - **18.** Two Deployments, both fine → exits 0, waited for each. - **19.** The API is fine, the dashboard never gets ready → exits 1, the last good commit is deployed again (both Deployments waited for again), last good unchanged, pods printed. - Negative control: a copy of `relay-deploy` that waits only for the first Deployment fails 6 checks, including "exits 0" with a broken dashboard and recording it as last good. - **`relay-deploy.sh`: lock wait 15 → 30 minutes.** The comment promised 15 minutes was longer than the worst case, which stopped being true with a second Deployment: a deploy plus its rollback is one migrate Job (up to 180 s) plus up to 180 s per Deployment, twice. That's about 18 minutes with the API and the dashboard, and about 24 with the worker (Phase 4). The deploy job's own limit is 40 minutes. - **`docs/CONTEXT.md`:** DNS rebinding written down as a known gap with where it closes (the worker connects only to the checked address, behind a NetworkPolicy; nothing is sent before Phase 4), from the #32 review. The CORS line now names both origins, as `docs/api.md` does, from the #33 review. A link to decision 0007. - **Decision 0007 (new):** idempotent events rely on the unique constraint and READ COMMITTED, with the alternatives and the check from 4 Nov (under REPEATABLE READ the race test goes red), from the #33 review. - **Decision 0006:** Proposed → Accepted. ## Tested - `bash infra/server/test-relay-deploy.sh infra/server/relay-deploy.sh`: 56 passed, 0 failed (49 before + 7 new checks). ## After merge `relay-deploy` is installed by hand and owned by root, so install it again so the server matches git (`k8s/README.md`): ``` ssh relay 'cat > /tmp/relay-deploy' < infra/server/relay-deploy.sh ssh -t relay 'sudo install -o root -g root -m 755 /tmp/relay-deploy /usr/local/bin/relay-deploy' ``` Then compare `sha256sum /usr/local/bin/relay-deploy` on the server with `shasum -a 256 infra/server/relay-deploy.sh` on the laptop.
…n, namespace) (#35) ## What The four follow-ups from my review of #24, so you only have to review them. One file, `k8s/app/dashboard.yaml`; the dashboard then matches the API's hardening. | Change | Why | |---|---| | `namespace: relay` on the Deployment, Service and Ingress | as in the API's files; a hand-run `kubectl apply` with the admin kubeconfig would otherwise land in `default` | | `automountServiceAccountToken: false` | nginx never talks to the Kubernetes API, so the pod carries no token | | `readOnlyRootFilesystem: true` + an `emptyDir` on `/tmp` (16Mi) | nothing in the image may change at runtime, as for the API. `nginx-unprivileged` writes only under `/tmp` (`pid /tmp/nginx.pid`, `client_body_temp_path /tmp/client_temp`, …), not `/var` as the old comment said | | `runAsUser: 101` | written out, so it doesn't depend on the image's `USER` line | ## Checked With the real image (`relay-dashboard:c145056`, pulled anonymously), run the way the pod will run: `--user 101 --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges`: ``` /index.html → 200 /assets/index-BimGaWWm.js → 200 /deliveries (fallback) → 200 ``` The only non-access log line is the IPv6 entrypoint script saying it can't edit the config (`info:`, then "Configuration complete; ready for start up"). The manifest parses, and all three objects carry `namespace: relay`. Server dry run for Pod Security (from the laptop, any commit's image tag works): ``` sed "s/__GIT_SHA__/$(git rev-parse origin/main)/" k8s/app/dashboard.yaml | ssh relay 'kubectl apply --dry-run=server -f -' ``` If the pods ever failed at runtime anyway, `relay-deploy` rolls the commit back, which #34 now tests for the dashboard.
What
Step 5.2: the Deployment, Service and Ingress that put the dashboard on
https://cdn.myrelay.tech. One file ink8s/app/, sorelay-deployapplies it on every mergeto main, with the image tag replaced by the commit's SHA.
Same shape as the API's manifests, with four differences:
nginx-unprivilegedruns as uid 101 and can't bind a low port/index.htmlcpu: 10m,memory: 24Mirequests, 64Mi limitcdn.myrelay.tech+letsencrypt-prodTwo pods with
maxUnavailable: 0, and the image tagged by git SHA rather thanlatest, so arollback can name an exact build.
Changed since the draft
k8s/dashboard/intok8s/app/. As written,relay-deploywould never haveapplied it: it only globs
"$REPO"/k8s/app/*.yaml. The pipeline would have gone green anddeployed nothing.
PLACEHOLDER_SHA->__GIT_SHA__.relay-deployonly substitutes__GIT_SHA__, so the tag would have stayed as literal text:ImagePullBackOff, the rolloutnever ready, and the whole commit rolled back — API included.
preStop. Removing a pod from the Service and telling it to stop happen atthe same moment, but Traefik needs a moment to hear about the removal. Without the pause nginx
is already shutting down while Traefik still routes here, and those requests fail.
maxUnavailable: 0doesn't cover this: it guarantees a replacement is ready before the oldpod goes, but says nothing about the old pod's last seconds. The API has the same 5 s. This
was the one piece of our four-part zero-downtime recipe the dashboard was missing.
terminationGracePeriodSecondsstays at its default 30 s, which comfortably holds the 5 s;nginx serves static files, so there is nothing to drain afterwards.
app.kubernetes.io/name: relay-dashboardinsteadof
app: relay-dashboard, on the Deployment, its selector, the pod template and the Serviceselector. A Deployment's selector is immutable once it exists, so this is free now and a
delete-and-recreate later. Checked that all three still agree — a selector that doesn't match
the pod labels gives a Service with no endpoints, which looks like a broken app rather than a
broken label.
Pod Security: this would have been refused
The file was first written on 26 Oct, before #21. Checked it against the
restrictedlevel that#21 puts on namespace
relay, and it was missing two of the four required fields —runAsNonRootandseccompProfile. The image already runs as uid 101, but Kubernetes doesn'tlook inside the image: being non-root has to be declared or the API server refuses the pod.
Verified rather than reasoned about, in a throwaway namespace
ps-testcarrying the same labels(
restricted,enforce-version: v1.36), deleted afterwards:deployment/service/ingress created (server dry run)— no warningsWarning: would violate PodSecurity "restricted:v1.36": runAsNonRoot != true …, seccompProfile …The control is the point. A clean dry run on its own doesn't show the check is running — same
trap as the
nc -Gcheck in #17 and theexternalIPs == nullrule in #21. The control namesboth fields, so the pass means something.
One note on Pod Security worth writing down somewhere: it enforces on Pods, not on
Deployments. Applying a violating Deployment succeeds with a warning, and then the pods never
appear. So
kubectl applyreturning cleanly is not evidence — the warning line is.Re-checked after the move and the
preStopaddition, against namespacerelayitself with thecommit's real SHA substituted:
The image exists
#30 is merged, so the pipeline now builds, smoke-tests and pushes
relay-dashboard:<sha>onevery merge to main — not only when the dashboard changed, because the tag has to exist for
whatever commit the deploy applies. Confirmed against the registry rather than trusting the
green tick:
That is also why the image and the manifest are two PRs and not one: the tag has to exist before
a manifest references it, or the first rollout fails on a missing image.
Not in this PR
readOnlyRootFilesystemstaysfalse: nginx writes its cache and pid file under/var, soturning it on means emptyDir mounts for both.
restricteddoesn't require it. Happy to add itif you'd rather — your Postgres does the equivalent.
of deferred (see "Changed since the draft").
For the reviewer
cdn.myrelay.techright after the merge while cert-manager finishes the ACME challenge.