Skip to content

infra: test the dashboard rollback, and record the Phase 3 review follow-ups - #34

Merged
devinder-dev merged 1 commit into
mainfrom
infra/phase-3-follow-ups
Nov 4, 2026
Merged

devinder-dev merged 1 commit into
mainfrom
infra/phase-3-follow-ups

Conversation

@devinder-dev

Copy link
Copy Markdown
Owner

What

Follow-ups from the reviews of #24, #32 and #33. One code change (the deploy script's lock
wait), one test change (the deploy harness learns about the dashboard), and three docs changes.

Changes

  • test-relay-deploy.sh: the dashboard is now in the deploy tests. Until now the fake
    kubectl apply only ever reported relay-api, so nothing proved that a broken dashboard rolls
    the commit back. The fake now reports relay-dashboard too when k8s/app/dashboard.yaml
    exists, records each rollout status it is asked for, and can fail the dashboard alone.
    • 18. Two Deployments, both fine → exits 0, waited for each.
    • 19. The API is fine, the dashboard never gets ready → exits 1, the last good commit is
      deployed again (both Deployments waited for again), last good unchanged, pods printed.
    • Negative control: a copy of relay-deploy that waits only for the first Deployment fails 6
      checks, including "exits 0" with a broken dashboard and recording it as last good.
  • relay-deploy.sh: lock wait 15 → 30 minutes. The comment promised 15 minutes was longer
    than the worst case, which stopped being true with a second Deployment: a deploy plus its
    rollback is one migrate Job (up to 180 s) plus up to 180 s per Deployment, twice. That's about
    18 minutes with the API and the dashboard, and about 24 with the worker (Phase 4). The deploy
    job's own limit is 40 minutes.
  • docs/CONTEXT.md: DNS rebinding written down as a known gap with where it closes (the
    worker connects only to the checked address, behind a NetworkPolicy; nothing is sent before
    Phase 4), from the feat: endpoints with an SSRF check and a 20-endpoint limit (steps 3.2 + 3.3) #32 review. The CORS line now names both origins, as docs/api.md does,
    from the feat: events with idempotency and fan-out, delivery reads, and CORS (steps 3.4–3.6) #33 review. A link to decision 0007.
  • Decision 0007 (new): idempotent events rely on the unique constraint and READ COMMITTED,
    with the alternatives and the check from 4 Nov (under REPEATABLE READ the race test goes red),
    from the feat: events with idempotency and fan-out, delivery reads, and CORS (steps 3.4–3.6) #33 review.
  • Decision 0006: Proposed → Accepted.

Tested

  • bash infra/server/test-relay-deploy.sh infra/server/relay-deploy.sh: 56 passed, 0 failed
    (49 before + 7 new checks).

After merge

relay-deploy is installed by hand and owned by root, so install it again so the server matches
git (k8s/README.md):

ssh relay 'cat > /tmp/relay-deploy' < infra/server/relay-deploy.sh
ssh -t relay 'sudo install -o root -g root -m 755 /tmp/relay-deploy /usr/local/bin/relay-deploy'

Then compare sha256sum /usr/local/bin/relay-deploy on the server with shasum -a 256 infra/server/relay-deploy.sh on the laptop.

@AbdikhaniAbdirashid AbdikhaniAbdirashid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read all five files and ran the harness myself rather than taking the count on trust.

$ bash test-relay-deploy.sh relay-deploy.sh
56 passed, 0 failed

And the negative control, because a new test that passes against a broken script tests nothing. I
copied the script with grep -m1 '^deployment' so it waits only for the first Deployment:

50 passed, 6 failed
18. FAIL  waits for each one: got 'deployment.apps/relay-api '
19. FAIL  exits 1 (CI turns red): got '0'
    FAIL  puts the last good commit back: got 'bbbb…', expected 'aaaa…'
    FAIL  last good stays the same: got 'bbbb…'
    FAIL  waits for both, then both again for the rollback
    FAIL  prints the pods: got 'no'

So test 19 really pins the thing that matters: with a dashboard that never gets ready, a
first-Deployment-only script exits 0 and records the broken commit as last good. That was
untested until now, and it is my half of the deploy that it protects. The explicit exit 0 at the
end of the fake's apply branch is the detail that keeps tests 1-17 honest, since a missing
dashboard.yaml would otherwise make apply itself fail.

Decision 0007 says what I asked for and more: READ COMMITTED is named as load-bearing, with the
check (REPEATABLE READ turns the race test red) and the four alternatives. The note that keyless
requests never conflict because NULLs are never equal in a UNIQUE constraint is the part I would
not have thought to write down. CONTEXT.md now names both CORS origins and the rebinding gap, so
the three open items from #32 and #33 are closed.

One finding, and it is in a file this PR doesn't touch. The lock wait goes 15 -> 30 minutes, but
.github/workflows/pipeline.yml:268-273 derives the deploy job's timeout-minutes: 40 from the old
value, and still says so:

#   success: 15 min lock + 3 migrate + 6 rollout + 9 public check = 33 min
timeout-minutes: 40

With a 30-minute wait that budget is 30 + 3 + 6 + 9 = 48, plus the 5 minutes of overhead the comment
allows for: 53 > 40. A job that waits out most of the lock now gets killed by GitHub after taking
it, partway through a rollout, so the rollback never runs - which is worse than the clean "another
deploy is still running" we had before. Two ways to make the files agree: raise timeout-minutes to
55 (and the Phase 4 note from 45 to 60), or cap the wait at 17 minutes, which still covers your
18-minute worst case minus the public check.

Not blocking it, because it needs a hand-run deploy racing the pipeline to bite and the pipeline
serializes main's deploys. But those two numbers should be derived from each other in one place
before the worker adds a third Deployment. If you'd rather fix it here, push it and I'll re-approve.

@devinder-dev
devinder-dev merged commit 5fd9375 into main Nov 4, 2026
6 checks passed
@devinder-dev
devinder-dev deleted the infra/phase-3-follow-ups branch November 4, 2026 10:08
devinder-dev added a commit that referenced this pull request Nov 4, 2026
…n, namespace) (#35)

## What

The four follow-ups from my review of #24, so you only have to review them. One file,
`k8s/app/dashboard.yaml`; the dashboard then matches the API's hardening.

| Change | Why |
|---|---|
| `namespace: relay` on the Deployment, Service and Ingress | as in the API's files; a hand-run `kubectl apply` with the admin kubeconfig would otherwise land in `default` |
| `automountServiceAccountToken: false` | nginx never talks to the Kubernetes API, so the pod carries no token |
| `readOnlyRootFilesystem: true` + an `emptyDir` on `/tmp` (16Mi) | nothing in the image may change at runtime, as for the API. `nginx-unprivileged` writes only under `/tmp` (`pid /tmp/nginx.pid`, `client_body_temp_path /tmp/client_temp`, …), not `/var` as the old comment said |
| `runAsUser: 101` | written out, so it doesn't depend on the image's `USER` line |

## Checked

With the real image (`relay-dashboard:c145056`, pulled anonymously), run the way the pod will run:
`--user 101 --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges`:

```
/index.html                → 200
/assets/index-BimGaWWm.js  → 200
/deliveries (fallback)     → 200
```

The only non-access log line is the IPv6 entrypoint script saying it can't edit the config
(`info:`, then "Configuration complete; ready for start up"). The manifest parses, and all three
objects carry `namespace: relay`.

Server dry run for Pod Security (from the laptop, any commit's image tag works):

```
sed "s/__GIT_SHA__/$(git rev-parse origin/main)/" k8s/app/dashboard.yaml | ssh relay 'kubectl apply --dry-run=server -f -'
```

If the pods ever failed at runtime anyway, `relay-deploy` rolls the commit back, which #34 now
tests for the dashboard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants