docs: record stack decisions (Fastify, dashboard) and update team context

README.md

@@ -21,15 +21,18 @@ Relay sits in the middle:
 
 | Service | Address | Status |
 |---|---|---|
-| API | https://api.myrelay.tech | In progress |
-| Dashboard (static files) | https://cdn.myrelay.tech | Planned |
-| Metrics and logs | https://grafana.myrelay.tech | Planned |
+| API | https://api.myrelay.tech | In progress (test app live with HTTPS) |
+| Relay dashboard (static files) | https://cdn.myrelay.tech | Planned |
+| Ops dashboard: metrics and logs (Grafana) | https://grafana.myrelay.tech | Planned |
 
 ## Stack
 
-Bun + TypeScript · PostgreSQL (also the job queue, via `SKIP LOCKED`) · k3s on our own VPS ·
-Traefik + cert-manager (Let's Encrypt) · GitHub Actions → GHCR · Prometheus, Grafana, Loki ·
-Terraform for the backup bucket and cluster add-ons.
+Bun 1.4 + TypeScript + Fastify + Zod · React + Vite + Tailwind dashboard ·
+PostgreSQL 18 (also the job queue, via `SKIP LOCKED`) ·
+k3s on our own VPS · Traefik + cert-manager (Let's Encrypt) · GitHub Actions → GHCR ·
+Prometheus, Grafana, Loki · Amazon S3 for backups, with the bucket made by Terraform.
+
+Why we chose what we chose: [`docs/decisions/`](docs/decisions/).
 
 ## Team
 

docs/CONTEXT.md

@@ -10,10 +10,12 @@ Chas Academy's "Avancerad fullstackutveckling, test och DevOps".
 
 - **Team:** Devinder Singh (backend + infrastructure) and Abdikhani Mohamed (frontend + quality).
   Every PR is reviewed by the other person. Both must be able to explain the whole system.
-- **Dates:** Tenta 4 Wed 28 Oct · G complete target Fri 30 Oct · **oral defense Wed 4 Nov
+- **Dates:** Tenta 4 Wed 28 Oct · G target around Sat 31 Oct · **oral defense Wed 4 Nov
   (live demo)** · code submission Sun 8 Nov.
-- **Strategy:** build all G requirements first, then VG (2+ replicas, zero-downtime deploys,
-  autoscaling shown under load).
+- **Strategy:** a **walking skeleton** first (a tiny API that the pipeline deploys to production,
+  already with 2 pods and zero-downtime rollouts), then the features on top. All G requirements
+  before VG. Quality and understanding come before speed: dates are targets, and on 4 Nov we show
+  what is done and explain it well.
 
 ## How we work
 
@@ -28,8 +30,11 @@ Chas Academy's "Avancerad fullstackutveckling, test och DevOps".
   `ssh relay` / `kubectl get` / `gh api`), instead of relying on "done".
 - **Commands that ask questions** (`adduser`, `passwd`, `ufw enable`, a first `sudo`) run
   alone, never inside a pasted block. Pasted lines get swallowed as answers (real incident, 22 Oct).
-- **Docs over memory.** Check current official docs for Bun, Hono, cert-manager, k3s and so on
+- **Docs over memory.** Check current official docs for Bun, Fastify, cert-manager, k3s and so on
   instead of guessing APIs.
+- **By code, not by clicking.** Manifests, scripts, Terraform and `gh` wherever an API exists.
+  Web panels only where none does (one.com VPS, get.tech DNS).
+- **Decisions are written down** in `docs/decisions/` (one short record per choice).
 - **Security scans.** Scan new security-sensitive code (auth, SQL, signing, webhooks, user input)
   with Snyk where available, and fix findings before the PR.
 
@@ -39,53 +44,76 @@ Chas Academy's "Avancerad fullstackutveckling, test och DevOps".
 - Secrets live only in **GitHub Actions secrets** or **Kubernetes Secrets**. Never in git.
 - The kubeconfig is an admin password. It stays on the server in `~/.kube/config`.
 - Ask the other person before any destructive command on the VPS, the database or DNS.
-- Never expose to the internet: Postgres (5432), the Kubernetes API (6443), `/metrics`.
-- Endpoint URLs from customers must not point to internal or private addresses (SSRF).
+- Never expose to the internet: Postgres (5432), the Kubernetes API (6443), `/metrics`
+  (served on its own port 9090, which no Ingress points to).
+- Endpoint URLs from customers must not point to internal, private or our own VPS addresses
+  (SSRF). Checked when an endpoint is created **and** before every delivery attempt; redirects
+  are never followed.
 
 ## How Relay works
 
-1. A producer sends `POST /events` with its API key.
+1. A producer sends `POST /v1/events` with its API key (`Authorization: Bearer relay_live_…`).
 2. The API validates it and, in **one transaction**, saves the event plus one **delivery** row
-   per subscribed endpoint, then answers **202 Accepted**.
-3. Worker pods claim due deliveries with `SELECT … FOR UPDATE SKIP LOCKED`.
-4. The worker signs the body (HMAC-SHA256, header `Relay-Signature: t=<unix>,v1=<hex>`,
-   signed string `t + "." + body`) and POSTs it with a 10 s timeout.
-5. 2xx = delivered. Error or timeout = retry with exponential backoff + jitter
-   (draft: 1 min, 5 min, 30 min, 2 h, 6 h, 12 h). After the last try: **dead**, replayable.
-6. At-least-once delivery: each event has a unique id; producers may send `Idempotency-Key`.
-
-**Data model:** `owners` (id, name, api_key_hash) · `endpoints` (id, owner_id, url, secret,
-event_types[], paused_until, created_at) · `events` (id, owner_id, type, payload jsonb,
-idempotency_key, created_at; UNIQUE (owner_id, idempotency_key)) · `deliveries` (id, event_id,
-endpoint_id, status pending|delivered|dead, attempts, next_attempt_at, last_status_code,
-updated_at; INDEX (status, next_attempt_at)) · `attempts` (id, delivery_id, status_code,
-duration_ms, error, created_at).
-
-**API:** `POST /endpoints` · `POST /events` · `GET /deliveries?status=` · `GET /deliveries/:id`
-· `POST /deliveries/:id/replay` · `GET /health/live` (no DB check) · `GET /health/ready`
-(checks DB) · `GET /metrics` (cluster-internal only).
+   per matching endpoint (exact event type, or an endpoint listening to `*`), then answers
+   **202 Accepted**. A repeated `Idempotency-Key` returns the original answer.
+3. Worker pods **claim** due deliveries with `FOR UPDATE SKIP LOCKED` and take a 60 s **lease**:
+   `attempts + 1`, `next_attempt_at = now() + 60 s`, a fresh `lease_id`, commit. No transaction or
+   connection is held during the HTTP call.
+4. The worker checks the URL again (SSRF), signs with **Standard Webhooks** (headers
+   `webhook-id` = event id, `webhook-timestamp`, `webhook-signature: v1,<base64 HMAC-SHA256 of
+   id.timestamp.body>`), and POSTs with a 10 s timeout. It signs again on **every** attempt.
+5. 2xx = delivered. Error, redirect or timeout = retry with exponential backoff and ±20% jitter:
+   5 s, 1 min, 5 min, 30 min, 2 h, 6 h, 12 h (8 tries, about 21 h). After the last try: **dead**,
+   replayable. Tests and the demo use a shorter schedule from an environment variable.
+6. The result is saved only `WHERE lease_id = <my lease>` (a **fencing token**). A crashed
+   worker's lease runs out and the row becomes due again by itself.
+7. At-least-once delivery: receivers drop duplicates by `webhook-id`.
+
+**Data model v1** (Postgres 18.6, every id is a UUIDv7):
+`owners` (id, name) · `api_keys` (owner_id, prefix, key_hash = SHA-256 UNIQUE, revoked_at) ·
+`endpoints` (owner_id, name, url, signing_secret `whsec_…`, event_types[] default `{*}`,
+disabled_at) · `events` (owner_id, type, payload jsonb object, idempotency_key; UNIQUE (owner_id,
+idempotency_key)) · `deliveries` (owner_id, event_id, endpoint_id, status pending|delivered|dead,
+attempts, next_attempt_at, lease_id, last_status_code, last_error, delivered_at; UNIQUE (event_id,
+endpoint_id); partial INDEX (next_attempt_at) WHERE status = 'pending') · `attempts`
+(delivery_id, started_at, duration_ms, status_code, error, response_excerpt).
+
+**API v1:** `GET /v1/me` · `POST /v1/endpoints` (returns the signing secret once) ·
+`GET /v1/endpoints` · `POST /v1/events` · `GET /v1/deliveries?status=&endpoint_id=&cursor=` ·
+`GET /v1/deliveries/:id` · `POST /v1/deliveries/:id/replay` · `GET /health/live` (no DB check) ·
+`GET /health/ready` (checks DB, 503 while shutting down) · `GET /version` (git SHA) ·
+`/metrics` on port 9090 only. Errors: `{"error":{"code","message","request_id"}}`. Another
+owner's id answers 404. Bodies: JSON only, max 256 KiB. CORS: `https://cdn.myrelay.tech` only.
+Owners and API keys are created with a `create-owner` script (no signup page).
 
 ## Stack and decisions
 
 | Area | Choice | Not chosen |
 |---|---|---|
-| Runtime | Bun 1.4 + TypeScript | Node (more common, but we know Bun) |
+| Runtime | Bun **1.4.4**, pinned everywhere (laptop, CI, Docker). 1.4 is a new rewrite in Rust | Node (more common, but we know Bun; it stays our fallback) |
+| Web framework | **Fastify 5** ([decision 0001](decisions/0001-web-framework-fastify.md)): `buildApp()` + `listen({ host: "0.0.0.0" })`, pino logs, Zod validation, `@fastify/cors`, `inject()` tests | Hono (native on Bun, but new to us), Express, Elysia |
+| Database library | postgres.js behind one file `src/db.ts` | `Bun.sql` (ties the data layer to the new runtime), Drizzle |
+| API code structure | **Zod 4** for bodies, queries and env vars ([decision 0003](decisions/0003-api-code-structure.md)); layers routes → services → repositories; one Fastify error handler with one error shape; small files, one job each | Fastify JSON Schema, `fastify-type-provider-zod`, logic in route handlers |
+| Migrations | dbmate (plain SQL), run by a Kubernetes Job before each rollout. Expand/contract rule | drizzle-kit, an initContainer (would rerun on every scale-up) |
 | Unit + integration tests | `bun test`; integration against a real Postgres (Docker locally, service container in GitHub Actions) | Testcontainers (no official Bun support), mocked DB |
+| Dashboard | **Vite + React + TypeScript, Tailwind CSS v4, React Router, axios** (one client in `src/api.ts`), plain React state with a small refresh loop while a delivery is pending, served by nginx ([decision 0002](decisions/0002-dashboard-stack.md)) | Next.js, plain CSS, `fetch`, TanStack Query |
 | E2E tests | Playwright, runs on Node in the dashboard folder | Cypress |
-| Queue | Postgres `SKIP LOCKED` | RabbitMQ (stretch goal only, as a "doorbell"; dual-write problem) |
-| Database | Postgres in k3s, StatefulSet + local-path volume with Retain | Neon / Supabase (brief: everything on our own VPS) |
+| Queue | Postgres `SKIP LOCKED` + lease with fencing token | RabbitMQ (stretch goal only, as a "doorbell"; dual-write problem) |
+| Signing | Standard Webhooks, re-signed on every attempt | Our own header format |
+| IDs | UUIDv7 (built into Postgres 18). Prefixes only on secrets: `relay_live_`, `whsec_` | bigint counters, prefixed ids |
+| Database | `postgres:18.6` StatefulSet, volume mounted at `/var/lib/postgresql`, our own StorageClass `local-path-retain` (the default deletes data) | Neon / Supabase (brief: everything on our own VPS) |
 | Orchestration | k3s v1.36.5 (Traefik, metrics-server, local-path, secrets encryption on) | Docker Compose |
-| HTTPS | cert-manager v1.21.2 + Let's Encrypt, staging first | certbot by hand |
-| Images | GHCR, tag = git SHA | Docker Hub |
-| CI/CD | GitHub Actions: test → build → push → deploy | Argo CD |
-| Observability | kube-prometheus-stack + Loki (Grafana Cloud free tier as fallback) | Datadog |
-| Backups | Nightly CronJob: pg_dump + gzip → S3-compatible bucket, one tested restore | Volume snapshots |
-| IaC | Kubernetes YAML in git; Terraform for the backup bucket, GitHub settings and Helm add-ons (Phase 4) | Terraform for everything (no provider for one.com or get.tech) |
-| Scaling / zero downtime | HPA on CPU; RollingUpdate maxUnavailable 0, readiness probe, graceful SIGTERM | KEDA (stretch), blue-green |
-
-**Still open:** web framework (Hono, Elysia or Express), database library (`Bun.sql`,
-postgres.js or Drizzle), how Actions reaches the cluster (SSH vs Kubernetes API),
-backup storage (R2, B2 or AWS), final retry schedule, Grafana address.
+| HTTPS | cert-manager v1.21.2 + Let's Encrypt, staging first; HTTP → HTTPS redirect in Traefik with `allowACMEByPass` | certbot by hand |
+| Images | GHCR (public package), tag = git SHA | Docker Hub |
+| CI/CD | GitHub Actions: test → build → push → deploy over SSH as a `deploy` user with a forced command (port 6443 stays closed), then check `/version` | Self-hosted runner, opening 6443, Argo CD |
+| Observability | Slim kube-prometheus-stack + Loki + Alloy (Promtail is end-of-life); Grafana Cloud free tier as fallback | Datadog |
+| Backups | Nightly CronJob: `pg_dump` → **Amazon S3** (eu-north-1), one timed restore | Volume snapshots, R2, B2 |
+| IaC | Kubernetes YAML and scripts in git; **Terraform only for the S3 backup bucket** and its IAM user | Terraform for everything (no provider for one.com or get.tech; Helm via Terraform needs 6443) |
+| Scaling / zero downtime | HPA on the API (2–4 pods, no `replicas:` field); workers fixed at 2; RollingUpdate maxUnavailable 0, readiness probe, preStop sleep, graceful SIGTERM | KEDA (stretch), blue-green |
+
+**Left out on purpose:** rate limiting per IP (Traefik sees every client as `10.42.0.1`),
+encrypting the signing-secret column (we protect the backups instead), OpenAPI tooling
+(`docs/api.md` + shared types instead).
 
 ## Infrastructure facts
 
@@ -96,9 +124,12 @@ backup storage (R2, B2 or AWS), final retry schedule, Grafana address.
 - **Firewalls:** one.com's network firewall + ufw. ufw allows 22, 80, 443 and the k3s networks
   10.42.0.0/16 and 10.43.0.0/16. one.com's firewall allows 22, 80 and 443.
 - **kubectl:** only on the server (`ssh relay`, then `kubectl …`). Port 6443 stays closed.
-- **Domain:** `myrelay.tech`, DNS at get.tech. `api.myrelay.tech` has an A record to the VPS.
-  **No AAAA records**: the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists.
-  Planned: `cdn.myrelay.tech` (dashboard), `grafana.myrelay.tech` (name not final).
+- **Domain:** `myrelay.tech`, DNS at get.tech. A records for `api`, `cdn` (the Relay dashboard)
+  and `grafana` (the ops dashboard: metrics and logs) all point to the VPS. **No AAAA records**:
+  the cluster is IPv4-only, and Let's Encrypt prefers IPv6 when it exists.
+- **HTTP → HTTPS:** `k8s/platform/traefik-config.yaml` (a HelmChartConfig) redirects port 80 to
+  443 for every host with a 308, except `/.well-known/acme-challenge/`, so certificate checks still
+  work over plain HTTP.
 
 ## Git workflow
 
@@ -113,4 +144,12 @@ backup storage (R2, B2 or AWS), final retry schedule, Grafana address.
 
 Phase 0 (server foundation) is done: VPS hardened, k3s, cert-manager with Let's Encrypt
 issuers, and a temporary hello-world app on https://api.myrelay.tech with a trusted
-certificate. Next: Phase 1, the API skeleton (Bun) with tests.
+certificate.
+
+Phase 1 (walking skeleton) has started: HTTP → HTTPS redirect (1.1) and DNS for cdn and
+grafana (1.2) are done. Next: 1.3, the API skeleton in `api/` (Bun 1.4.4 + Fastify) with
+`/health/live`, `/health/ready`, `/version` and the first `bun test`.
+
+The plan in order: 1 walking skeleton · 2 database foundation · 3 core API · 4 worker ·
+5 dashboard on cdn (Abdikhani, in parallel) · 6 backups + Terraform (Abdikhani, in parallel) ·
+7 metrics and logs (Abdikhani, in parallel) · 8 VG under load · 9 docs and defense · 10 submit.

docs/decisions/0001-web-framework-fastify.md

@@ -0,0 +1,52 @@
+# 0001. Web framework: Fastify
+
+- **Status:** Accepted
+- **Date:** 2026-10-25
+- **Decided by:** Devinder Singh and Abdikhani Mohamed
+
+## Context
+
+The Relay API needs an HTTP framework. Our runtime is **Bun 1.4.4**, pinned because Bun 1.4 is the
+first release of Bun rewritten in Rust (August 2026). The API must:
+
+- route requests and read path parameters (`/v1/deliveries/:id`);
+- validate JSON bodies (`POST /v1/events`);
+- write JSON logs with a request id on every line;
+- limit body size (256 KiB) and allow CORS from `https://cdn.myrelay.tech`;
+- shut down gracefully on SIGTERM, which is needed for zero-downtime rollouts;
+- be testable without starting a server.
+
+Both of us are examined individually and must be able to explain every line.
+
+## Options considered
+
+| Option | For | Against |
+|---|---|---|
+| **Fastify 5** | Both of us have used it. JSON logging (pino) with request ids, body validation (JSON Schema), body limit and request ids are built in. `inject()` tests without a server. Runs on Node unchanged. | Built for Node's `http` module, so on Bun it runs through Bun's Node-compatibility layer, which Bun 1.4 rewrote. Not tested by us on 1.4.4 yet. Listens on `localhost` by default. More concepts (plugins, encapsulation, hooks). |
+| Hono 4 | Runs on Bun's native server (`Bun.serve`). Web-standard Request/Response. CORS, body limit and request id built in. `app.request()` tests. | New for both of us. Logging and validation must be added (own logger, Zod). |
+| Express | Most widely known. | Older design without web-standard Request/Response; needs extra packages for the features above. |
+| Elysia | Fast, Bun-first. | Runs only on Bun; more "magic" that is harder to explain. |
+
+## Decision
+
+We use **Fastify 5** on Bun 1.4.4.
+
+Familiarity was the deciding factor: we can write, review and explain Fastify code with
+confidence. Its built-in pino logging (with a request id on every line) also covers a need we
+would otherwise build ourselves.
+
+## Consequences
+
+- **Extra check in step 1.3:** before building on it, we run the app and its tests on Bun 1.4.4 and
+  put it under some load, to confirm Bun's Node-compatibility layer behaves.
+- **Fallback:** if Bun causes problems, the same Fastify code runs on Node without changes.
+- **Listen address:** the server must listen with `host: "0.0.0.0"`, because the default
+  (`localhost`) can't be reached from outside a container.
+- **App structure:** `buildApp()` creates the app (one fresh instance per test, used with `inject()`);
+  `server.ts` only calls `listen()` and handles SIGTERM with `app.close()`.
+- **Logging:** pino, built into Fastify, writing JSON to stdout. We keep our agreed field names
+  (`request_id`, `event_id`, `delivery_id`, `endpoint_id`, `status_code`, `duration_ms`).
+- **Validation:** Zod, called in each route ([decision 0003](0003-api-code-structure.md)). We don't
+  use Fastify's built-in JSON Schema validation: with Zod each body is described once and gives
+  both the runtime check and the TypeScript type.
+- **CORS:** the `@fastify/cors` plugin, limited to `https://cdn.myrelay.tech` (and localhost in CI).