infra: test the dashboard rollback, and record the Phase 3 review follow-ups

docs/CONTEXT.md

@@ -48,14 +48,18 @@ Chas Academy's "Avancerad fullstackutveckling, test och DevOps".
   (served on its own port 9090, which no Ingress points to).
 - Endpoint URLs from customers must not point to internal, private or our own VPS addresses
   (SSRF). Checked when an endpoint is created **and** before every delivery attempt; redirects
-  are never followed.
+  are never followed. **Known gap, closed in Phase 4: DNS rebinding.** A name can resolve to a
+  public address at the check and to a private one when the request is sent, so a second check
+  only narrows the window. The worker closes it by connecting only to the address it just
+  checked (Host and SNI kept), behind a NetworkPolicy. Nothing is sent before Phase 4.
 
 ## How Relay works
 
 1. A producer sends `POST /v1/events` with its API key (`Authorization: Bearer relay_live_…`).
 2. The API validates it and, in **one transaction**, saves the event plus one **delivery** row
    per matching endpoint (exact event type, or an endpoint listening to `*`), then answers
-   **202 Accepted**. A repeated `Idempotency-Key` returns the original answer.
+   **202 Accepted**. A repeated `Idempotency-Key` returns the original answer
+   ([decision 0007](decisions/0007-idempotency-unique-constraint.md)).
 3. Worker pods **claim** due deliveries with `FOR UPDATE SKIP LOCKED` and take a 60 s **lease**:
    `attempts + 1`, `next_attempt_at = now() + 60 s`, a fresh `lease_id`, commit. No transaction or
    connection is held during the HTTP call.
@@ -83,7 +87,7 @@ endpoint_id); partial INDEX (next_attempt_at) WHERE status = 'pending') · `atte
 `GET /v1/deliveries/:id` · `POST /v1/deliveries/:id/replay` · `GET /health/live` (no DB check) ·
 `GET /health/ready` (never checks the DB; 503 while shutting down) · `GET /version` (git SHA) ·
 `/metrics` on port 9090 only. Errors: `{"error":{"code","message","request_id"}}`. Another
-owner's id answers 404. Bodies: JSON only, max 256 KiB. CORS: `https://cdn.myrelay.tech` only.
+owner's id answers 404. Bodies: JSON only, max 256 KiB. CORS: `https://cdn.myrelay.tech`, and `http://localhost:4173` (Playwright's preview server), as in `docs/api.md`.
 Owners and API keys are created with a `create-owner` script (no signup page).
 **The full contract, with every request and response shape, is in [`docs/api.md`](api.md).**
 Both the API and the dashboard follow it; changes to it go through a PR first.

docs/decisions/0006-migrations-expand-contract.md

@@ -1,6 +1,6 @@
 # 0006. Database migrations: forward-only, expand/contract
 
-- **Status:** Proposed
+- **Status:** Accepted
 - **Date:** 2026-10-31
 - **Decided by:** Devinder Singh and Abdikhani Mohamed
 

docs/decisions/0007-idempotency-unique-constraint.md

@@ -0,0 +1,44 @@
+# 0007. Idempotent events: a unique constraint, read under READ COMMITTED
+
+- **Status:** Accepted
+- **Date:** 2026-11-04
+- **Decided by:** Devinder Singh and Abdikhani Mohamed
+
+## Context
+
+`POST /v1/events` takes an optional `Idempotency-Key`. A producer that doesn't know whether its
+first request was saved (a 503, a dropped connection, a 500) sends it again with the same key,
+and must get the first answer back without a second event. Two copies of the same request can
+also arrive at the same moment, on different pods.
+
+## Decision
+
+- **The database decides, not our code.** `events` has `UNIQUE (owner_id, idempotency_key)`.
+  The insert is `INSERT … ON CONFLICT (owner_id, idempotency_key) DO NOTHING RETURNING …`, inside
+  the same transaction as the fan-out to `deliveries`.
+- **A replay is read by a separate statement.** When the insert returns no row, the key already
+  exists, and a second statement reads that event and its delivery count. A second request with
+  the same key waits at the unique index until the first commits; under **READ COMMITTED**
+  (Postgres's default) the next statement gets a fresh view and sees the first request's row.
+- **The key wins.** The same key with a different body returns the first answer and saves
+  nothing new (written in `docs/api.md`). Refusing it, as Stripe and Svix do, would need a new
+  error code in the contract; that can come later.
+
+| Not chosen | Why not |
+|---|---|
+| Look the key up first, insert if missing | Two requests both look, both find nothing, both insert: a check-then-act race |
+| An advisory lock on a hash of the key | Works, but adds a lock we have to explain, and the unique constraint is still needed as the real guarantee |
+| `SERIALIZABLE` and retry on `40001` | Correct, but every caller needs retry code for something the constraint already solves |
+| Redis `SETNX` | A second system that has to agree with Postgres: the dual-write problem |
+
+## Consequences
+
+- **READ COMMITTED is load-bearing.** Under REPEATABLE READ the replay `SELECT` would use the
+  transaction's first snapshot, and a request that waited for a concurrent one with the same key
+  would not see it: a 500 instead of the first answer. Checked on 4 Nov: with
+  `sql.begin("isolation level repeatable read", …)` the race test (five requests with one key,
+  held at a real lock) goes red, while the one-after-another retry tests stay green. So don't
+  raise the isolation level for this transaction; the race test is what would catch it.
+- A key is per owner: the same key from two owners gives two events.
+- Requests without a key never conflict (NULLs are never equal in a UNIQUE constraint), so a
+  retry without a key can create a second event. `docs/api.md` tells producers to send a key.

infra/server/relay-deploy.sh

@@ -26,13 +26,14 @@ if ! [[ "$SHA" =~ ^[0-9a-f]{40}$ ]]; then
   fail "invalid commit SHA: $SHA"
 fi
 
-# 2. One deploy at a time: a second deploy waits here for up to 15 minutes. That's longer
-#    than the worst case (deploy + rollback, 180 s per Deployment each). In CI this rarely
-#    waits at all, because the pipeline already runs main's deploys one after the other;
-#    it protects against an admin deploying by hand at the same moment.
-#    The lock is released by itself when the script ends, even if it crashes.
+# 2. One deploy at a time: a second deploy waits here for up to 30 minutes. The worst case is
+#    a deploy and then its rollback, each one migrate Job (up to 180 s) plus up to 180 s per
+#    Deployment: with the API and the dashboard about 18 minutes, with the worker (Phase 4)
+#    about 24. In CI this rarely waits at all, because the pipeline already runs main's
+#    deploys one after the other; it protects against an admin deploying by hand at the same
+#    moment. The lock is released by itself when the script ends, even if it crashes.
 exec 9>/home/deploy/.relay-deploy.lock
-flock --wait 900 9 || fail "another deploy is still running after 15 minutes: nothing was changed, re-run this job"
+flock --wait 1800 9 || fail "another deploy is still running after 30 minutes: nothing was changed, re-run this job"
 
 # 3. Gate 2: the commit must be on main, which means it was merged through a reviewed PR.
 git -C "$REPO" fetch --quiet origin main

infra/server/test-relay-deploy.sh

@@ -25,9 +25,11 @@ esac
 exit 0
 EOF
   # kubectl: every call's first word goes to "calls", so tests can check the order.
-  # apply notes that it ran, and for which commit (applied-commits), and prints one Deployment
-  # (or fails if apply-fails exists); rollout status fails when the checked-out commit is
-  # listed in "broken".
+  # apply notes that it ran, and for which commit (applied-commits), and prints the API's
+  # Deployment, plus the dashboard's when k8s/app/dashboard.yaml exists (or fails if
+  # apply-fails exists). rollout status notes which Deployment it waited for (rollouts) and
+  # fails when the checked-out commit is listed in "broken" (every Deployment of that commit)
+  # or, for the dashboard only, in "broken-dashboard".
   # create (the migrate Job) reads the commit from the Job's label and answers
   # job.batch/migrate-<first 5 characters>, or is refused when create-fails exists.
   # get <that job> answers the Job's true conditions: "SuccessCriteriaMet Complete" unless
@@ -39,8 +41,13 @@ echo "\$1" >> "$T/calls"
 job=""; for a in "\$@"; do case "\$a" in *migrate-*) job="\${a##*/}" ;; esac; done
 case "\$*" in
   apply*) cat > /dev/null; touch "$T/applied"; cat "$T/checked-out" >> "$T/applied-commits"
-          [ -e "$T/apply-fails" ] && exit 1; echo deployment.apps/relay-api ;;
-  "rollout status"*) grep -qx "\$(cat "$T/checked-out")" "$T/broken" && exit 1; exit 0 ;;
+          [ -e "$T/apply-fails" ] && exit 1; echo deployment.apps/relay-api
+          [ -e "$T/home/relay/k8s/app/dashboard.yaml" ] && echo deployment.apps/relay-dashboard
+          exit 0 ;;
+  "rollout status"*) echo "\$3" >> "$T/rollouts"
+          grep -qx "\$(cat "$T/checked-out")" "$T/broken" && exit 1
+          [ "\${3##*/}" = relay-dashboard ] && grep -qx "\$(cat "$T/checked-out")" "$T/broken-dashboard" 2>/dev/null && exit 1
+          exit 0 ;;
   create*) commit=\$(sed -n 's/.*relay.myrelay.tech\/commit: "\([0-9a-f]*\)".*/\1/p')
            [ -e "$T/create-fails" ] && { echo 'Error from server (Forbidden): jobs.batch is forbidden' >&2; exit 1; }
            echo "job.batch/migrate-\${commit:0:5}" ;;
@@ -191,5 +198,25 @@ setup_migrate; echo $GOOD > "$T/home/.relay-last-good"; touch "$T/get-job-fails"
 check "exits 1, never success" "$code" 1
 check "never applies the failing commit" "$(applied_bad)" no
 
+# From Phase 5 the dashboard is a second Deployment in k8s/app/: the script must wait for each,
+# and a dashboard that never gets ready must roll the whole commit back, the API included.
+with_dashboard() { echo "kind: Deployment" > "$T/home/relay/k8s/app/dashboard.yaml"; }
+rollouts() { tr '\n' ' ' < "$T/rollouts" 2>/dev/null; }
+
+echo "18. Two Deployments, both fine"
+setup; with_dashboard; code=$(run $GOOD)
+check "exits 0" "$code" 0
+check "waits for each one" "$(rollouts)" "deployment.apps/relay-api deployment.apps/relay-dashboard "
+
+echo "19. The API is fine, the dashboard never gets ready"
+setup; with_dashboard; echo $GOOD > "$T/home/.relay-last-good"; echo $BAD > "$T/broken-dashboard"
+code=$(run $BAD)
+check "exits 1 (CI turns red)" "$code" 1
+check "puts the last good commit back" "$(cat "$T/checked-out")" $GOOD
+check "last good stays the same" "$(cat "$T/home/.relay-last-good")" $GOOD
+check "waits for both, then both again for the rollback" "$(rollouts)" \
+  "deployment.apps/relay-api deployment.apps/relay-dashboard deployment.apps/relay-api deployment.apps/relay-dashboard "
+check "prints the pods" "$(grep -q -- '--- pods' "$T/out" && echo yes || echo no)" yes
+
 echo; echo "$passed passed, $failed failed"
 [ "$failed" -eq 0 ]