Skip to content

infra: add API pipeline (test, build, push to GHCR) - #12

Merged
devinder-dev merged 1 commit into
mainfrom
infra/api-ci
Oct 29, 2026
Merged

devinder-dev merged 1 commit into
mainfrom
infra/api-ci

Conversation

@devinder-dev

@devinder-dev devinder-dev commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Step 1.5: the API pipeline in GitHub Actions (.github/workflows/api.yml).

What it does

  • Every pull request: api-test installs with the frozen lockfile, then runs typecheck and bun test. api-image then checks that the Docker image builds for linux/amd64. Nothing is pushed.
  • Every merge to main that touches api/: the same tests, then the image is built and pushed to GHCR as ghcr.io/devinder-dev/relay-api:<commit>. Deployment comes in step 1.8.

Choices

  • No path filter on pull requests: api-test will be a required check, and a required check that never runs (for example on a dashboard-only PR) would block the merge forever. The API tests take about 30 seconds.
  • Bun 1.4.4 from api/.bun-version: one file that CI reads and that documents the version for our laptops.
  • Actions pinned to full commit SHAs, with the version as a comment, so a moved or hijacked tag can't run new code. docker/login-action is v4.6.0, because v4.7.0 was only 3 days old.
  • Least privilege: contents: read by default. Only api-image gets packages: write, and it only logs in and pushes on main.
  • Images are built on GitHub's amd64 runners, which match the VPS. Images built on a Mac would be arm64.
  • concurrency cancels outdated PR runs. The gha cache reuses unchanged image layers.

After the first push to main

  1. Make the GHCR package relay-api public, so k3s can pull it without credentials.
  2. Add api-test as a required check in the protect-main ruleset.

Rebased onto main after #11 was merged, so it now contains only the workflow and api/.bun-version.

Base automatically changed from feat/api-skeleton to main October 29, 2026 09:19
devinder-dev added a commit that referenced this pull request Oct 29, 2026
…unt) (#13)

Step 1.6: the path the pipeline will use to deploy. Nothing deploys yet:
`k8s/app/` arrives in step 1.7, and the deploy job in the workflow in
step 1.8.

## What's in it
- `k8s/platform/deployer.yaml`: namespace `relay`, ServiceAccount
`deployer`, and a Role + RoleBinding. It may update Deployments,
Services, ConfigMaps and Ingresses, and read ReplicaSets and Pods (to
follow and undo a rollout). No delete, no Secrets, nothing outside
`relay`.
- `infra/server/setup-deploy-user.sh`: one-time setup of the Linux user
`deploy` (no password, no sudo). Its key is locked with
`restrict,command="/usr/local/bin/relay-deploy"`, and it gets a
kubeconfig with the `deployer` token and a clone of the repo.
- `infra/server/relay-deploy.sh`: the only program the deploy key can
run. It runs a lock, then gate 1 (exactly 40 hex characters), then gate
2 (the commit is in main's history). Then it checks out that commit,
applies `k8s/app/` with `__GIT_SHA__` replaced, and waits for the
rollout, or undoes it.
- `k8s/README.md`: which folder is applied how, and by whom.
- `docs/CONTEXT.md`: "Where we are", plus the deploy path under
infrastructure facts.

Already on the server: `deployer.yaml` is applied, the setup script has
run, and the deploy script is installed as root. The GitHub environment
`production` (main only) holds the secret `DEPLOY_SSH_KEY` and the
variables `DEPLOY_HOST` and `DEPLOY_KNOWN_HOSTS`.

## Why this design
Port 6443 stays closed. Three layers each answer one question:
- **Who gets in:** the SSH key plus the forced command.
- **What gets deployed:** only commits on main.
- **Where it can reach:** RBAC, namespace `relay` only.

RBAC alone isn't enough, because a Deployment could mount a Secret.
That's why the script only applies YAML from main, where every change
has been reviewed.

## Tested
- `kubectl auth can-i … --as=system:serviceaccount:relay:deployer`:
patch deployments → yes. Get secrets, delete deployments, create
statefulsets, anything in `kube-system` → no.
- The real badge, as the `deploy` user: patch deployments → yes, get
secrets → no.
- With the deploy key (every case ended with exit 1):
  - `whoami` → rejected by gate 1.
- A valid SHA + `; touch /tmp/pwned` → rejected by gate 1, and the file
was never created.
  - PR #12's commit → rejected by gate 2.
- Main's newest commit → passes both gates, then stops at "no k8s/app
folder" (expected until 1.7).
- The script on the server is identical to the committed file. The
laptop copy of the deploy key was deleted after the tests.

## Check it yourself
From your laptop, on this branch:
```
ssh relay 'cat /usr/local/bin/relay-deploy' | diff - infra/server/relay-deploy.sh && echo same
ssh relay 'kubectl --kubeconfig ~/.kube/config auth can-i get secrets -n relay --as=system:serviceaccount:relay:deployer'
```

## Please look at
- The Role's verbs: enough for `apply` and `rollout undo`, and nothing
more?
- The two gates in `relay-deploy.sh`.

@AbdikhaniAbdirashid AbdikhaniAbdirashid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving this. It's small, the comments explain the why, and the two jobs do
exactly what the description says. A few things below — none of them blocking.

What I checked locally

  • bun run typecheck exists in api/package.json (tsc --noEmit), api/bun.lock
    is committed, and api/Dockerfile really does take ARG GIT_SHA. So all three
    things the workflow calls are actually there.
  • .bun-version is 1.4.4 and the Dockerfile base is oven/bun:1.4.4-slim. Same
    version in both places.
  • I resolved all five pinned action SHAs against the GitHub tag API: checkout
    v7.0.1, setup-bun v2.2.0, setup-buildx v4.4.1, login v4.6.0, build-push v7.4.0.
    Every one matches its comment. Thanks for pinning these properly — a comment
    that doesn't match its SHA is how pinning quietly stops meaning anything.

Two small changes I'd like

  1. timeout-minutes on both jobs. Without it a hung step runs to GitHub's 6-hour
    default and eats our Actions minutes. timeout-minutes: 10 on each would do.

  2. api-image builds on every pull request, including dashboard-only ones. I
    follow the reasoning for api-test having no path filter — a required check
    that never runs blocks the merge forever — but the image build doesn't have
    that problem, since it won't be a required check. Right now every dashboard PR
    I open will build the API image for nothing. Is that deliberate, or worth a
    path filter / an if on the image job?

Three things to settle, not necessarily here

  1. I looked at the protect-main ruleset: it has deletion, non_fast_forward
    and pull_request, but no required status checks at all yet. So the "api-test
    will be a required check" reasoning is about a future state. It's in your
    follow-up list — just flagging that until that's done, nothing is actually
    enforced.

  2. Making the GHCR package public is a real decision and I don't think it should
    live only in a PR comment. A public image means anyone can pull and inspect our
    API. The alternative is an imagePullSecret in k3s: more setup, image stays
    private. I'm fine with public for this project, but can we put it in an ADR
    with that trade-off written down? We both have to defend it.

  3. No Postgres service container here. That's correct today — the three test files
    are pure unit tests with no DB. But docs/CONTEXT.md says integration tests run
    against a real Postgres as a service container. When repositories land, does
    that go in this workflow, or a separate one? Asking so api-test doesn't get
    reopened later.

One question for me

  1. On main, cancel-in-progress is false, so if two PRs merge a minute apart
    both runs finish and we get two images tagged with their own commit SHA. Is the
    intent that the deploy step in 1.8 always picks the newest SHA, or does the
    ordering of those two runs matter?

@devinder-dev

Copy link
Copy Markdown
Owner Author

Thanks for checking the SHAs against the tags yourself, that's exactly the check that keeps pinning meaningful. My answers:

  1. timeout-minutes: agreed. infra: deploy every merge to main and check that it's live #16 already rewrites this file (as pipeline.yml), so I'm adding timeout-minutes: 10 to api-test and api-image there, rather than changing this PR and asking you to approve it again.

  2. api-image on every PR: deliberate for now.

    • Actions minutes are free for public repositories on GitHub's standard runners, so the cost is time, not money.
    • With the build cache, an unchanged image builds in about 20 s.
    • One simple workflow is easier for both of us to explain than per-job path filters.
    • It also proves on every PR that the image still builds.
    • If CI gets slow once the dashboard image joins, we add an if on changed paths.
  3. Required checks: right, nothing is enforced yet. I'll add api-test (and later the dashboard's test job) as required checks last, after infra: roll back to the last good commit and wait for every Deployment #14, infra: add the API manifests (2 pods, probes, zero-downtime rollout) #15 and infra: deploy every merge to main and check that it's live #16 are merged. Adding it now would mean updating every open branch and asking you to approve each one again.

  4. Public image: good point, it's a real decision. I'll write it as ADR 0004 in its own small docs PR. Public means no pull secret to create, rotate or leak, and the code is public anyway. The cost is that anyone can pull and inspect the image, so it must never contain anything that isn't already in the repo. The alternative, a private package plus an imagePullSecret, would be written down there too.

  5. Postgres in CI: in this same workflow. When the first repository lands (step 2.6), api-test gets a services: postgres container plus DATABASE_URL, so api-test stays the one required check for the API.

  6. Two merges close together: the workflow-level concurrency group (<name>-${{ github.ref }}, never cancelling on main) runs main's pipelines one after the other, in merge order. Each deploys its own commit, so the newest merge always deploys last. If several merges wait, GitHub keeps only the newest pending run, and that commit already contains the others. On the server, relay-deploy's flock is a second layer. And its gate only accepts commits on main.

Merging this now as step 1 of the plan, then the image goes public (with ADR 0004 following).

@devinder-dev
devinder-dev merged commit a8274c1 into main Oct 29, 2026
2 checks passed
@devinder-dev
devinder-dev deleted the infra/api-ci branch October 29, 2026 13:39
devinder-dev added a commit that referenced this pull request Oct 29, 2026
…15)

Step 1.7: the API's Kubernetes manifests in `k8s/app/`. This PR doesn't
deploy anything. The pipeline's deploy job comes in 1.8, once #12 (the
image) and #14 (rollback) are merged.

## What's in it
- **`deployment.yaml`:**
  - 2 pods (VG1).
- Rolling update with `maxSurge: 1, maxUnavailable: 0`, so there are
never fewer than 2 ready pods (VG2).
  - Image `relay-api:__GIT_SHA__`, filled in by `relay-deploy`.
  - Probes, and a `preStop` sleep of 5 s.
- Requests 100m CPU and 128Mi memory, limit 256Mi (the 1.3 load test
peaked at 93 MB).
- Runs as uid 1000 (not root), with no privilege escalation, all
capabilities dropped, and the default seccomp profile.
- **`service.yaml`:** port 80 → the pods' port named `http`.
- **`ingress.yaml`:** `api.myrelay.tech`, with a `letsencrypt-prod`
certificate in namespace `relay`.

## The probes
- **Readiness:** `/health/ready` every 5 s. After 3 failures (about 15
s), the pod is taken out of the Service but keeps running.
- **Liveness:** `/health/live` every 20 s, starting after 15 s. After 3
failures (about 60 s), the container is restarted.

Liveness is the more patient one because a restart is more drastic than
losing traffic. It also never checks the database: a database outage
must not restart every pod.

## Why `runAsUser: 1000`
The image says `USER bun`, which is a name. Kubernetes can only verify
`runAsNonRoot` against a number. With just the name, the pod would fail
with `CreateContainerConfigError`.

## Tested
- **A server-side dry run as the `deployer` account itself** (`sudo -u
deploy kubectl … --dry-run=server`): all three files were accepted. A
dry run is authorized like a real request, so this proves the pipeline
may apply exactly these files.
- **The probe paths against the real image.** A loop reads every `path:`
from `deployment.yaml` and curls the running container: `200
/health/ready`, `200 /health/live`. The dry run alone can't catch a
wrong path: it also passed with `/ready` and `/health`, which both
answer 404.

## At the first deploy (1.8)
The hello app in `default` also claims `api.myrelay.tech`, so we delete
it by hand at that moment, because two Ingresses for one host would
compete. cert-manager then issues a new certificate for namespace
`relay`.

## Added after an independent review of Phases 0–1
- **`automountServiceAccountToken: false`:** the API never talks to
Kubernetes, so its pods get no token at all.
- **`readOnlyRootFilesystem: true`:** tested with the real image
(`--read-only --cap-drop ALL --user 1000:1000`). `/health/ready`,
`/health/live` and `/version` answer 200, and `docker stop` exits 0 in
0.15 s.
- **A comment on why there's no CPU limit:** a limit would throttle the
API even when the server has CPU to spare. The request is what
scheduling, and later the HPA, use.
- The server-side dry run still passes.
devinder-dev added a commit that referenced this pull request Oct 29, 2026
Step 1.8: every merge to main is deployed, then checked from outside.
Rebased onto main now that #12, #14 and #15 are merged. **Merge only
after the switch-over below.**

## What changes
- **`api.yml` → `pipeline.yml`.** It now deploys too, and the
dashboard's jobs will join it later (`needs:` only works inside one
workflow file).
- **No path filter on `main`.** Every merge is a release: a deploy needs
an image tagged with exactly the merged commit, even when only
`k8s/app/` or the docs changed.
- **New job `deploy-script-test`** on every PR and merge. It runs
`infra/server/test-relay-deploy.sh` (relay-deploy's gates and rollback,
with fake git and kubectl) and `shellcheck` on the server scripts. The
deploy needs it, so a broken deploy script never reaches the server.
- **New job `deploy`**, after `api-image` and `deploy-script-test`, only
on a merge to main:
- **`environment: production`:** the only way to get `DEPLOY_SSH_KEY`,
and only on main.
  - **`permissions: {}`:** no GitHub access at all.
- **`timeout-minutes: 30`:** the worst case is the server's lock (15
min, #19) plus a rollout and a rollback.
- **Step 1:** writes the key and our server's `known_hosts`, then `ssh …
deploy@<vps> "$GITHUB_SHA"` (`StrictHostKeyChecking=yes`,
`BatchMode=yes`). The exit code of `relay-deploy` becomes the step's.
- **Step 2:** asks `https://api.myrelay.tech/version` until it answers
this commit, for up to 36 × 5 s, because the first deploy also waits for
a new certificate.
- **`timeout-minutes: 10`** on `api-test` and `api-image` (from the #12
review).
- **`k8s/hello/` is deleted**, together with its row in `k8s/README.md`.
The API's Ingress takes over `api.myrelay.tech`.

## Tested
- `actionlint` 1.7.7 (with shellcheck on the `run:` scripts): no
findings.
- The new job's two steps, run locally: `26 passed, 0 failed`, and
`shellcheck` is clean.
- The version check, run on its own with `bash -eo pipefail` against a
local container:
  - the right SHA → passes;
  - an old SHA → keeps asking, then fails;
  - nothing answering → "no answer" on every try, then fails.
- The rebase was rehearsed in a throwaway clone first: the 3 commits
replayed onto main without conflicts.

## Already done before merging
- #12, #14 and #15 merged.
- `deployer.yaml` applied: the deployer can read events, and still can't
read Secrets or delete.
- `relay-deploy` installed, byte-identical to main.
- The image is pullable without credentials (ADR 0004, #18), and the
server has already pulled `relay-api:a8274c1`.

## Switch-over (right before merging)
1. Delete the hello app from `default`: the Ingress, Service and
Deployment, plus its leftover TLS Secret `api-myrelay-tech-tls`.
`api.myrelay.tech` is down for about 1–2 minutes, once.
2. Merge → the pipeline tests, builds, deploys → `/version` answers the
commit.

## After the first deploy
- **A real rollback:** deploy #15's merge commit `a78934c` by hand. It
has no image, so the rollout fails and `relay-deploy` redeploys the last
good commit, while a curl loop shows only 200s.
- **Zero downtime:** a load loop during a normal deploy (merging #19)
should show no failed requests.
- Then make `api-test` and `deploy-script-test` required checks.
devinder-dev added a commit that referenced this pull request Oct 29, 2026
Abdikhani asked in the review of #12 for the public-image decision to be
written down, because we both have to defend it.

- **`docs/decisions/0004-public-container-images.md`:**
- Our images stay public, so the cluster pulls them without credentials.
- **The rule that comes with it:** an image may only contain what's
already public in the repo. No secrets, `.env` files or keys, ever.
- Not chosen: a private image with an `imagePullSecret` (a long-lived
token to rotate and protect, which hides nothing the repo doesn't
already show), and Docker Hub (rate limits).
- **`docs/CONTEXT.md`:** the Images row links the decision.

Checked on 29 Oct: after #12 merged, `relay-api:<commit>` was already
pullable without any login (an anonymous pull answered 200, a missing
package 403). Our earlier note "GHCR packages start private" turned out
to be wrong for a package published from a public repo, so no visibility
change was needed. The ADR includes the check command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants