Repository navigation
infra: test the dashboard rollback, and record the Phase 3 review follow-ups - #34
Conversation
AbdikhaniAbdirashid
left a comment
There was a problem hiding this comment.
Read all five files and ran the harness myself rather than taking the count on trust.
$ bash test-relay-deploy.sh relay-deploy.sh
56 passed, 0 failed
And the negative control, because a new test that passes against a broken script tests nothing. I
copied the script with grep -m1 '^deployment' so it waits only for the first Deployment:
50 passed, 6 failed
18. FAIL waits for each one: got 'deployment.apps/relay-api '
19. FAIL exits 1 (CI turns red): got '0'
FAIL puts the last good commit back: got 'bbbb…', expected 'aaaa…'
FAIL last good stays the same: got 'bbbb…'
FAIL waits for both, then both again for the rollback
FAIL prints the pods: got 'no'
So test 19 really pins the thing that matters: with a dashboard that never gets ready, a
first-Deployment-only script exits 0 and records the broken commit as last good. That was
untested until now, and it is my half of the deploy that it protects. The explicit exit 0 at the
end of the fake's apply branch is the detail that keeps tests 1-17 honest, since a missing
dashboard.yaml would otherwise make apply itself fail.
Decision 0007 says what I asked for and more: READ COMMITTED is named as load-bearing, with the
check (REPEATABLE READ turns the race test red) and the four alternatives. The note that keyless
requests never conflict because NULLs are never equal in a UNIQUE constraint is the part I would
not have thought to write down. CONTEXT.md now names both CORS origins and the rebinding gap, so
the three open items from #32 and #33 are closed.
One finding, and it is in a file this PR doesn't touch. The lock wait goes 15 -> 30 minutes, but
.github/workflows/pipeline.yml:268-273 derives the deploy job's timeout-minutes: 40 from the old
value, and still says so:
# success: 15 min lock + 3 migrate + 6 rollout + 9 public check = 33 min
timeout-minutes: 40
With a 30-minute wait that budget is 30 + 3 + 6 + 9 = 48, plus the 5 minutes of overhead the comment
allows for: 53 > 40. A job that waits out most of the lock now gets killed by GitHub after taking
it, partway through a rollout, so the rollback never runs - which is worse than the clean "another
deploy is still running" we had before. Two ways to make the files agree: raise timeout-minutes to
55 (and the Phase 4 note from 45 to 60), or cap the wait at 17 minutes, which still covers your
18-minute worst case minus the public check.
Not blocking it, because it needs a hand-run deploy racing the pipeline to bite and the pipeline
serializes main's deploys. But those two numbers should be derived from each other in one place
before the worker adds a third Deployment. If you'd rather fix it here, push it and I'll re-approve.
…n, namespace) (#35) ## What The four follow-ups from my review of #24, so you only have to review them. One file, `k8s/app/dashboard.yaml`; the dashboard then matches the API's hardening. | Change | Why | |---|---| | `namespace: relay` on the Deployment, Service and Ingress | as in the API's files; a hand-run `kubectl apply` with the admin kubeconfig would otherwise land in `default` | | `automountServiceAccountToken: false` | nginx never talks to the Kubernetes API, so the pod carries no token | | `readOnlyRootFilesystem: true` + an `emptyDir` on `/tmp` (16Mi) | nothing in the image may change at runtime, as for the API. `nginx-unprivileged` writes only under `/tmp` (`pid /tmp/nginx.pid`, `client_body_temp_path /tmp/client_temp`, …), not `/var` as the old comment said | | `runAsUser: 101` | written out, so it doesn't depend on the image's `USER` line | ## Checked With the real image (`relay-dashboard:c145056`, pulled anonymously), run the way the pod will run: `--user 101 --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges`: ``` /index.html → 200 /assets/index-BimGaWWm.js → 200 /deliveries (fallback) → 200 ``` The only non-access log line is the IPv6 entrypoint script saying it can't edit the config (`info:`, then "Configuration complete; ready for start up"). The manifest parses, and all three objects carry `namespace: relay`. Server dry run for Pod Security (from the laptop, any commit's image tag works): ``` sed "s/__GIT_SHA__/$(git rev-parse origin/main)/" k8s/app/dashboard.yaml | ssh relay 'kubectl apply --dry-run=server -f -' ``` If the pods ever failed at runtime anyway, `relay-deploy` rolls the commit back, which #34 now tests for the dashboard.
What
Follow-ups from the reviews of #24, #32 and #33. One code change (the deploy script's lock
wait), one test change (the deploy harness learns about the dashboard), and three docs changes.
Changes
test-relay-deploy.sh: the dashboard is now in the deploy tests. Until now the fakekubectl applyonly ever reportedrelay-api, so nothing proved that a broken dashboard rollsthe commit back. The fake now reports
relay-dashboardtoo whenk8s/app/dashboard.yamlexists, records each
rollout statusit is asked for, and can fail the dashboard alone.deployed again (both Deployments waited for again), last good unchanged, pods printed.
relay-deploythat waits only for the first Deployment fails 6checks, including "exits 0" with a broken dashboard and recording it as last good.
relay-deploy.sh: lock wait 15 → 30 minutes. The comment promised 15 minutes was longerthan the worst case, which stopped being true with a second Deployment: a deploy plus its
rollback is one migrate Job (up to 180 s) plus up to 180 s per Deployment, twice. That's about
18 minutes with the API and the dashboard, and about 24 with the worker (Phase 4). The deploy
job's own limit is 40 minutes.
docs/CONTEXT.md: DNS rebinding written down as a known gap with where it closes (theworker connects only to the checked address, behind a NetworkPolicy; nothing is sent before
Phase 4), from the feat: endpoints with an SSRF check and a 20-endpoint limit (steps 3.2 + 3.3) #32 review. The CORS line now names both origins, as
docs/api.mddoes,from the feat: events with idempotency and fan-out, delivery reads, and CORS (steps 3.4–3.6) #33 review. A link to decision 0007.
with the alternatives and the check from 4 Nov (under REPEATABLE READ the race test goes red),
from the feat: events with idempotency and fan-out, delivery reads, and CORS (steps 3.4–3.6) #33 review.
Tested
bash infra/server/test-relay-deploy.sh infra/server/relay-deploy.sh: 56 passed, 0 failed(49 before + 7 new checks).
After merge
relay-deployis installed by hand and owned by root, so install it again so the server matchesgit (
k8s/README.md):Then compare
sha256sum /usr/local/bin/relay-deployon the server withshasum -a 256 infra/server/relay-deploy.shon the laptop.