Repository navigation
infra: add API pipeline (test, build, push to GHCR) - #12
Conversation
…unt) (#13) Step 1.6: the path the pipeline will use to deploy. Nothing deploys yet: `k8s/app/` arrives in step 1.7, and the deploy job in the workflow in step 1.8. ## What's in it - `k8s/platform/deployer.yaml`: namespace `relay`, ServiceAccount `deployer`, and a Role + RoleBinding. It may update Deployments, Services, ConfigMaps and Ingresses, and read ReplicaSets and Pods (to follow and undo a rollout). No delete, no Secrets, nothing outside `relay`. - `infra/server/setup-deploy-user.sh`: one-time setup of the Linux user `deploy` (no password, no sudo). Its key is locked with `restrict,command="/usr/local/bin/relay-deploy"`, and it gets a kubeconfig with the `deployer` token and a clone of the repo. - `infra/server/relay-deploy.sh`: the only program the deploy key can run. It runs a lock, then gate 1 (exactly 40 hex characters), then gate 2 (the commit is in main's history). Then it checks out that commit, applies `k8s/app/` with `__GIT_SHA__` replaced, and waits for the rollout, or undoes it. - `k8s/README.md`: which folder is applied how, and by whom. - `docs/CONTEXT.md`: "Where we are", plus the deploy path under infrastructure facts. Already on the server: `deployer.yaml` is applied, the setup script has run, and the deploy script is installed as root. The GitHub environment `production` (main only) holds the secret `DEPLOY_SSH_KEY` and the variables `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS`. ## Why this design Port 6443 stays closed. Three layers each answer one question: - **Who gets in:** the SSH key plus the forced command. - **What gets deployed:** only commits on main. - **Where it can reach:** RBAC, namespace `relay` only. RBAC alone isn't enough, because a Deployment could mount a Secret. That's why the script only applies YAML from main, where every change has been reviewed. ## Tested - `kubectl auth can-i … --as=system:serviceaccount:relay:deployer`: patch deployments → yes. Get secrets, delete deployments, create statefulsets, anything in `kube-system` → no. - The real badge, as the `deploy` user: patch deployments → yes, get secrets → no. - With the deploy key (every case ended with exit 1): - `whoami` → rejected by gate 1. - A valid SHA + `; touch /tmp/pwned` → rejected by gate 1, and the file was never created. - PR #12's commit → rejected by gate 2. - Main's newest commit → passes both gates, then stops at "no k8s/app folder" (expected until 1.7). - The script on the server is identical to the committed file. The laptop copy of the deploy key was deleted after the tests. ## Check it yourself From your laptop, on this branch: ``` ssh relay 'cat /usr/local/bin/relay-deploy' | diff - infra/server/relay-deploy.sh && echo same ssh relay 'kubectl --kubeconfig ~/.kube/config auth can-i get secrets -n relay --as=system:serviceaccount:relay:deployer' ``` ## Please look at - The Role's verbs: enough for `apply` and `rollout undo`, and nothing more? - The two gates in `relay-deploy.sh`.
2db441e to
70257bd
Compare
AbdikhaniAbdirashid
left a comment
There was a problem hiding this comment.
Approving this. It's small, the comments explain the why, and the two jobs do
exactly what the description says. A few things below — none of them blocking.
What I checked locally
bun run typecheckexists inapi/package.json(tsc --noEmit),api/bun.lock
is committed, andapi/Dockerfilereally does takeARG GIT_SHA. So all three
things the workflow calls are actually there..bun-versionis 1.4.4 and the Dockerfile base isoven/bun:1.4.4-slim. Same
version in both places.- I resolved all five pinned action SHAs against the GitHub tag API: checkout
v7.0.1, setup-bun v2.2.0, setup-buildx v4.4.1, login v4.6.0, build-push v7.4.0.
Every one matches its comment. Thanks for pinning these properly — a comment
that doesn't match its SHA is how pinning quietly stops meaning anything.
Two small changes I'd like
-
timeout-minuteson both jobs. Without it a hung step runs to GitHub's 6-hour
default and eats our Actions minutes.timeout-minutes: 10on each would do. -
api-imagebuilds on every pull request, including dashboard-only ones. I
follow the reasoning forapi-testhaving no path filter — a required check
that never runs blocks the merge forever — but the image build doesn't have
that problem, since it won't be a required check. Right now every dashboard PR
I open will build the API image for nothing. Is that deliberate, or worth a
path filter / anifon the image job?
Three things to settle, not necessarily here
-
I looked at the
protect-mainruleset: it hasdeletion,non_fast_forward
andpull_request, but no required status checks at all yet. So the "api-test
will be a required check" reasoning is about a future state. It's in your
follow-up list — just flagging that until that's done, nothing is actually
enforced. -
Making the GHCR package public is a real decision and I don't think it should
live only in a PR comment. A public image means anyone can pull and inspect our
API. The alternative is animagePullSecretin k3s: more setup, image stays
private. I'm fine with public for this project, but can we put it in an ADR
with that trade-off written down? We both have to defend it. -
No Postgres service container here. That's correct today — the three test files
are pure unit tests with no DB. Butdocs/CONTEXT.mdsays integration tests run
against a real Postgres as a service container. When repositories land, does
that go in this workflow, or a separate one? Asking soapi-testdoesn't get
reopened later.
One question for me
- On
main,cancel-in-progressis false, so if two PRs merge a minute apart
both runs finish and we get two images tagged with their own commit SHA. Is the
intent that the deploy step in 1.8 always picks the newest SHA, or does the
ordering of those two runs matter?
|
Thanks for checking the SHAs against the tags yourself, that's exactly the check that keeps pinning meaningful. My answers:
Merging this now as step 1 of the plan, then the image goes public (with ADR 0004 following). |
…15) Step 1.7: the API's Kubernetes manifests in `k8s/app/`. This PR doesn't deploy anything. The pipeline's deploy job comes in 1.8, once #12 (the image) and #14 (rollback) are merged. ## What's in it - **`deployment.yaml`:** - 2 pods (VG1). - Rolling update with `maxSurge: 1, maxUnavailable: 0`, so there are never fewer than 2 ready pods (VG2). - Image `relay-api:__GIT_SHA__`, filled in by `relay-deploy`. - Probes, and a `preStop` sleep of 5 s. - Requests 100m CPU and 128Mi memory, limit 256Mi (the 1.3 load test peaked at 93 MB). - Runs as uid 1000 (not root), with no privilege escalation, all capabilities dropped, and the default seccomp profile. - **`service.yaml`:** port 80 → the pods' port named `http`. - **`ingress.yaml`:** `api.myrelay.tech`, with a `letsencrypt-prod` certificate in namespace `relay`. ## The probes - **Readiness:** `/health/ready` every 5 s. After 3 failures (about 15 s), the pod is taken out of the Service but keeps running. - **Liveness:** `/health/live` every 20 s, starting after 15 s. After 3 failures (about 60 s), the container is restarted. Liveness is the more patient one because a restart is more drastic than losing traffic. It also never checks the database: a database outage must not restart every pod. ## Why `runAsUser: 1000` The image says `USER bun`, which is a name. Kubernetes can only verify `runAsNonRoot` against a number. With just the name, the pod would fail with `CreateContainerConfigError`. ## Tested - **A server-side dry run as the `deployer` account itself** (`sudo -u deploy kubectl … --dry-run=server`): all three files were accepted. A dry run is authorized like a real request, so this proves the pipeline may apply exactly these files. - **The probe paths against the real image.** A loop reads every `path:` from `deployment.yaml` and curls the running container: `200 /health/ready`, `200 /health/live`. The dry run alone can't catch a wrong path: it also passed with `/ready` and `/health`, which both answer 404. ## At the first deploy (1.8) The hello app in `default` also claims `api.myrelay.tech`, so we delete it by hand at that moment, because two Ingresses for one host would compete. cert-manager then issues a new certificate for namespace `relay`. ## Added after an independent review of Phases 0–1 - **`automountServiceAccountToken: false`:** the API never talks to Kubernetes, so its pods get no token at all. - **`readOnlyRootFilesystem: true`:** tested with the real image (`--read-only --cap-drop ALL --user 1000:1000`). `/health/ready`, `/health/live` and `/version` answer 200, and `docker stop` exits 0 in 0.15 s. - **A comment on why there's no CPU limit:** a limit would throttle the API even when the server has CPU to spare. The request is what scheduling, and later the HPA, use. - The server-side dry run still passes.
Step 1.8: every merge to main is deployed, then checked from outside. Rebased onto main now that #12, #14 and #15 are merged. **Merge only after the switch-over below.** ## What changes - **`api.yml` → `pipeline.yml`.** It now deploys too, and the dashboard's jobs will join it later (`needs:` only works inside one workflow file). - **No path filter on `main`.** Every merge is a release: a deploy needs an image tagged with exactly the merged commit, even when only `k8s/app/` or the docs changed. - **New job `deploy-script-test`** on every PR and merge. It runs `infra/server/test-relay-deploy.sh` (relay-deploy's gates and rollback, with fake git and kubectl) and `shellcheck` on the server scripts. The deploy needs it, so a broken deploy script never reaches the server. - **New job `deploy`**, after `api-image` and `deploy-script-test`, only on a merge to main: - **`environment: production`:** the only way to get `DEPLOY_SSH_KEY`, and only on main. - **`permissions: {}`:** no GitHub access at all. - **`timeout-minutes: 30`:** the worst case is the server's lock (15 min, #19) plus a rollout and a rollback. - **Step 1:** writes the key and our server's `known_hosts`, then `ssh … deploy@<vps> "$GITHUB_SHA"` (`StrictHostKeyChecking=yes`, `BatchMode=yes`). The exit code of `relay-deploy` becomes the step's. - **Step 2:** asks `https://api.myrelay.tech/version` until it answers this commit, for up to 36 × 5 s, because the first deploy also waits for a new certificate. - **`timeout-minutes: 10`** on `api-test` and `api-image` (from the #12 review). - **`k8s/hello/` is deleted**, together with its row in `k8s/README.md`. The API's Ingress takes over `api.myrelay.tech`. ## Tested - `actionlint` 1.7.7 (with shellcheck on the `run:` scripts): no findings. - The new job's two steps, run locally: `26 passed, 0 failed`, and `shellcheck` is clean. - The version check, run on its own with `bash -eo pipefail` against a local container: - the right SHA → passes; - an old SHA → keeps asking, then fails; - nothing answering → "no answer" on every try, then fails. - The rebase was rehearsed in a throwaway clone first: the 3 commits replayed onto main without conflicts. ## Already done before merging - #12, #14 and #15 merged. - `deployer.yaml` applied: the deployer can read events, and still can't read Secrets or delete. - `relay-deploy` installed, byte-identical to main. - The image is pullable without credentials (ADR 0004, #18), and the server has already pulled `relay-api:a8274c1`. ## Switch-over (right before merging) 1. Delete the hello app from `default`: the Ingress, Service and Deployment, plus its leftover TLS Secret `api-myrelay-tech-tls`. `api.myrelay.tech` is down for about 1–2 minutes, once. 2. Merge → the pipeline tests, builds, deploys → `/version` answers the commit. ## After the first deploy - **A real rollback:** deploy #15's merge commit `a78934c` by hand. It has no image, so the rollout fails and `relay-deploy` redeploys the last good commit, while a curl loop shows only 200s. - **Zero downtime:** a load loop during a normal deploy (merging #19) should show no failed requests. - Then make `api-test` and `deploy-script-test` required checks.
Abdikhani asked in the review of #12 for the public-image decision to be written down, because we both have to defend it. - **`docs/decisions/0004-public-container-images.md`:** - Our images stay public, so the cluster pulls them without credentials. - **The rule that comes with it:** an image may only contain what's already public in the repo. No secrets, `.env` files or keys, ever. - Not chosen: a private image with an `imagePullSecret` (a long-lived token to rotate and protect, which hides nothing the repo doesn't already show), and Docker Hub (rate limits). - **`docs/CONTEXT.md`:** the Images row links the decision. Checked on 29 Oct: after #12 merged, `relay-api:<commit>` was already pullable without any login (an anonymous pull answered 200, a missing package 403). Our earlier note "GHCR packages start private" turned out to be wrong for a package published from a public repo, so no visibility change was needed. The ADR includes the check command.
Step 1.5: the API pipeline in GitHub Actions (
.github/workflows/api.yml).What it does
api-testinstalls with the frozen lockfile, then runstypecheckandbun test.api-imagethen checks that the Docker image builds forlinux/amd64. Nothing is pushed.mainthat touchesapi/: the same tests, then the image is built and pushed to GHCR asghcr.io/devinder-dev/relay-api:<commit>. Deployment comes in step 1.8.Choices
api-testwill be a required check, and a required check that never runs (for example on a dashboard-only PR) would block the merge forever. The API tests take about 30 seconds.api/.bun-version: one file that CI reads and that documents the version for our laptops.docker/login-actionis v4.6.0, because v4.7.0 was only 3 days old.contents: readby default. Onlyapi-imagegetspackages: write, and it only logs in and pushes onmain.concurrencycancels outdated PR runs. Theghacache reuses unchanged image layers.After the first push to
mainrelay-apipublic, so k3s can pull it without credentials.api-testas a required check in theprotect-mainruleset.Rebased onto
mainafter #11 was merged, so it now contains only the workflow andapi/.bun-version.