diff --git a/docs/CONTEXT.md b/docs/CONTEXT.md index 1598b8c..37825b6 100644 --- a/docs/CONTEXT.md +++ b/docs/CONTEXT.md @@ -48,14 +48,18 @@ Chas Academy's "Avancerad fullstackutveckling, test och DevOps". (served on its own port 9090, which no Ingress points to). - Endpoint URLs from customers must not point to internal, private or our own VPS addresses (SSRF). Checked when an endpoint is created **and** before every delivery attempt; redirects - are never followed. + are never followed. **Known gap, closed in Phase 4: DNS rebinding.** A name can resolve to a + public address at the check and to a private one when the request is sent, so a second check + only narrows the window. The worker closes it by connecting only to the address it just + checked (Host and SNI kept), behind a NetworkPolicy. Nothing is sent before Phase 4. ## How Relay works 1. A producer sends `POST /v1/events` with its API key (`Authorization: Bearer relay_live_…`). 2. The API validates it and, in **one transaction**, saves the event plus one **delivery** row per matching endpoint (exact event type, or an endpoint listening to `*`), then answers - **202 Accepted**. A repeated `Idempotency-Key` returns the original answer. + **202 Accepted**. A repeated `Idempotency-Key` returns the original answer + ([decision 0007](decisions/0007-idempotency-unique-constraint.md)). 3. Worker pods **claim** due deliveries with `FOR UPDATE SKIP LOCKED` and take a 60 s **lease**: `attempts + 1`, `next_attempt_at = now() + 60 s`, a fresh `lease_id`, commit. No transaction or connection is held during the HTTP call. @@ -83,7 +87,7 @@ endpoint_id); partial INDEX (next_attempt_at) WHERE status = 'pending') · `atte `GET /v1/deliveries/:id` · `POST /v1/deliveries/:id/replay` · `GET /health/live` (no DB check) · `GET /health/ready` (never checks the DB; 503 while shutting down) · `GET /version` (git SHA) · `/metrics` on port 9090 only. Errors: `{"error":{"code","message","request_id"}}`. Another -owner's id answers 404. Bodies: JSON only, max 256 KiB. CORS: `https://cdn.myrelay.tech` only. +owner's id answers 404. Bodies: JSON only, max 256 KiB. CORS: `https://cdn.myrelay.tech`, and `http://localhost:4173` (Playwright's preview server), as in `docs/api.md`. Owners and API keys are created with a `create-owner` script (no signup page). **The full contract, with every request and response shape, is in [`docs/api.md`](api.md).** Both the API and the dashboard follow it; changes to it go through a PR first. diff --git a/docs/decisions/0006-migrations-expand-contract.md b/docs/decisions/0006-migrations-expand-contract.md index 04b88f2..7f5a769 100644 --- a/docs/decisions/0006-migrations-expand-contract.md +++ b/docs/decisions/0006-migrations-expand-contract.md @@ -1,6 +1,6 @@ # 0006. Database migrations: forward-only, expand/contract -- **Status:** Proposed +- **Status:** Accepted - **Date:** 2026-10-31 - **Decided by:** Devinder Singh and Abdikhani Mohamed diff --git a/docs/decisions/0007-idempotency-unique-constraint.md b/docs/decisions/0007-idempotency-unique-constraint.md new file mode 100644 index 0000000..3104a32 --- /dev/null +++ b/docs/decisions/0007-idempotency-unique-constraint.md @@ -0,0 +1,44 @@ +# 0007. Idempotent events: a unique constraint, read under READ COMMITTED + +- **Status:** Accepted +- **Date:** 2026-11-04 +- **Decided by:** Devinder Singh and Abdikhani Mohamed + +## Context + +`POST /v1/events` takes an optional `Idempotency-Key`. A producer that doesn't know whether its +first request was saved (a 503, a dropped connection, a 500) sends it again with the same key, +and must get the first answer back without a second event. Two copies of the same request can +also arrive at the same moment, on different pods. + +## Decision + +- **The database decides, not our code.** `events` has `UNIQUE (owner_id, idempotency_key)`. + The insert is `INSERT … ON CONFLICT (owner_id, idempotency_key) DO NOTHING RETURNING …`, inside + the same transaction as the fan-out to `deliveries`. +- **A replay is read by a separate statement.** When the insert returns no row, the key already + exists, and a second statement reads that event and its delivery count. A second request with + the same key waits at the unique index until the first commits; under **READ COMMITTED** + (Postgres's default) the next statement gets a fresh view and sees the first request's row. +- **The key wins.** The same key with a different body returns the first answer and saves + nothing new (written in `docs/api.md`). Refusing it, as Stripe and Svix do, would need a new + error code in the contract; that can come later. + +| Not chosen | Why not | +|---|---| +| Look the key up first, insert if missing | Two requests both look, both find nothing, both insert: a check-then-act race | +| An advisory lock on a hash of the key | Works, but adds a lock we have to explain, and the unique constraint is still needed as the real guarantee | +| `SERIALIZABLE` and retry on `40001` | Correct, but every caller needs retry code for something the constraint already solves | +| Redis `SETNX` | A second system that has to agree with Postgres: the dual-write problem | + +## Consequences + +- **READ COMMITTED is load-bearing.** Under REPEATABLE READ the replay `SELECT` would use the + transaction's first snapshot, and a request that waited for a concurrent one with the same key + would not see it: a 500 instead of the first answer. Checked on 4 Nov: with + `sql.begin("isolation level repeatable read", …)` the race test (five requests with one key, + held at a real lock) goes red, while the one-after-another retry tests stay green. So don't + raise the isolation level for this transaction; the race test is what would catch it. +- A key is per owner: the same key from two owners gives two events. +- Requests without a key never conflict (NULLs are never equal in a UNIQUE constraint), so a + retry without a key can create a second event. `docs/api.md` tells producers to send a key. diff --git a/infra/server/relay-deploy.sh b/infra/server/relay-deploy.sh index b60a8b9..6b7b08d 100644 --- a/infra/server/relay-deploy.sh +++ b/infra/server/relay-deploy.sh @@ -26,13 +26,14 @@ if ! [[ "$SHA" =~ ^[0-9a-f]{40}$ ]]; then fail "invalid commit SHA: $SHA" fi -# 2. One deploy at a time: a second deploy waits here for up to 15 minutes. That's longer -# than the worst case (deploy + rollback, 180 s per Deployment each). In CI this rarely -# waits at all, because the pipeline already runs main's deploys one after the other; -# it protects against an admin deploying by hand at the same moment. -# The lock is released by itself when the script ends, even if it crashes. +# 2. One deploy at a time: a second deploy waits here for up to 30 minutes. The worst case is +# a deploy and then its rollback, each one migrate Job (up to 180 s) plus up to 180 s per +# Deployment: with the API and the dashboard about 18 minutes, with the worker (Phase 4) +# about 24. In CI this rarely waits at all, because the pipeline already runs main's +# deploys one after the other; it protects against an admin deploying by hand at the same +# moment. The lock is released by itself when the script ends, even if it crashes. exec 9>/home/deploy/.relay-deploy.lock -flock --wait 900 9 || fail "another deploy is still running after 15 minutes: nothing was changed, re-run this job" +flock --wait 1800 9 || fail "another deploy is still running after 30 minutes: nothing was changed, re-run this job" # 3. Gate 2: the commit must be on main, which means it was merged through a reviewed PR. git -C "$REPO" fetch --quiet origin main diff --git a/infra/server/test-relay-deploy.sh b/infra/server/test-relay-deploy.sh index fdca7ba..938da74 100644 --- a/infra/server/test-relay-deploy.sh +++ b/infra/server/test-relay-deploy.sh @@ -25,9 +25,11 @@ esac exit 0 EOF # kubectl: every call's first word goes to "calls", so tests can check the order. - # apply notes that it ran, and for which commit (applied-commits), and prints one Deployment - # (or fails if apply-fails exists); rollout status fails when the checked-out commit is - # listed in "broken". + # apply notes that it ran, and for which commit (applied-commits), and prints the API's + # Deployment, plus the dashboard's when k8s/app/dashboard.yaml exists (or fails if + # apply-fails exists). rollout status notes which Deployment it waited for (rollouts) and + # fails when the checked-out commit is listed in "broken" (every Deployment of that commit) + # or, for the dashboard only, in "broken-dashboard". # create (the migrate Job) reads the commit from the Job's label and answers # job.batch/migrate-, or is refused when create-fails exists. # get answers the Job's true conditions: "SuccessCriteriaMet Complete" unless @@ -39,8 +41,13 @@ echo "\$1" >> "$T/calls" job=""; for a in "\$@"; do case "\$a" in *migrate-*) job="\${a##*/}" ;; esac; done case "\$*" in apply*) cat > /dev/null; touch "$T/applied"; cat "$T/checked-out" >> "$T/applied-commits" - [ -e "$T/apply-fails" ] && exit 1; echo deployment.apps/relay-api ;; - "rollout status"*) grep -qx "\$(cat "$T/checked-out")" "$T/broken" && exit 1; exit 0 ;; + [ -e "$T/apply-fails" ] && exit 1; echo deployment.apps/relay-api + [ -e "$T/home/relay/k8s/app/dashboard.yaml" ] && echo deployment.apps/relay-dashboard + exit 0 ;; + "rollout status"*) echo "\$3" >> "$T/rollouts" + grep -qx "\$(cat "$T/checked-out")" "$T/broken" && exit 1 + [ "\${3##*/}" = relay-dashboard ] && grep -qx "\$(cat "$T/checked-out")" "$T/broken-dashboard" 2>/dev/null && exit 1 + exit 0 ;; create*) commit=\$(sed -n 's/.*relay.myrelay.tech\/commit: "\([0-9a-f]*\)".*/\1/p') [ -e "$T/create-fails" ] && { echo 'Error from server (Forbidden): jobs.batch is forbidden' >&2; exit 1; } echo "job.batch/migrate-\${commit:0:5}" ;; @@ -191,5 +198,25 @@ setup_migrate; echo $GOOD > "$T/home/.relay-last-good"; touch "$T/get-job-fails" check "exits 1, never success" "$code" 1 check "never applies the failing commit" "$(applied_bad)" no +# From Phase 5 the dashboard is a second Deployment in k8s/app/: the script must wait for each, +# and a dashboard that never gets ready must roll the whole commit back, the API included. +with_dashboard() { echo "kind: Deployment" > "$T/home/relay/k8s/app/dashboard.yaml"; } +rollouts() { tr '\n' ' ' < "$T/rollouts" 2>/dev/null; } + +echo "18. Two Deployments, both fine" +setup; with_dashboard; code=$(run $GOOD) +check "exits 0" "$code" 0 +check "waits for each one" "$(rollouts)" "deployment.apps/relay-api deployment.apps/relay-dashboard " + +echo "19. The API is fine, the dashboard never gets ready" +setup; with_dashboard; echo $GOOD > "$T/home/.relay-last-good"; echo $BAD > "$T/broken-dashboard" +code=$(run $BAD) +check "exits 1 (CI turns red)" "$code" 1 +check "puts the last good commit back" "$(cat "$T/checked-out")" $GOOD +check "last good stays the same" "$(cat "$T/home/.relay-last-good")" $GOOD +check "waits for both, then both again for the rollback" "$(rollouts)" \ + "deployment.apps/relay-api deployment.apps/relay-dashboard deployment.apps/relay-api deployment.apps/relay-dashboard " +check "prints the pods" "$(grep -q -- '--- pods' "$T/out" && echo yes || echo no)" yes + echo; echo "$passed passed, $failed failed" [ "$failed" -eq 0 ]