[devinder-dev]

Step 1.6: the path the pipeline will use to deploy. Nothing deploys yet: k8s/app/ arrives in step 1.7, and the deploy job in the workflow in step 1.8.

What's in it

• k8s/platform/deployer.yaml: namespace relay, ServiceAccount deployer, and a Role + RoleBinding. It may update Deployments, Services, ConfigMaps and Ingresses, and read ReplicaSets and Pods (to follow and undo a rollout). No delete, no Secrets, nothing outside relay.
• infra/server/setup-deploy-user.sh: one-time setup of the Linux user deploy (no password, no sudo). Its key is locked with restrict,command="/usr/local/bin/relay-deploy", and it gets a kubeconfig with the deployer token and a clone of the repo.
• infra/server/relay-deploy.sh: the only program the deploy key can run. It runs a lock, then gate 1 (exactly 40 hex characters), then gate 2 (the commit is in main's history). Then it checks out that commit, applies k8s/app/ with __GIT_SHA__ replaced, and waits for the rollout, or undoes it.
• k8s/README.md: which folder is applied how, and by whom.
• docs/CONTEXT.md: "Where we are", plus the deploy path under infrastructure facts.

Already on the server: deployer.yaml is applied, the setup script has run, and the deploy script is installed as root. The GitHub environment production (main only) holds the secret DEPLOY_SSH_KEY and the variables DEPLOY_HOST and DEPLOY_KNOWN_HOSTS.

Why this design

Port 6443 stays closed. Three layers each answer one question:

• Who gets in: the SSH key plus the forced command.
• What gets deployed: only commits on main.
• Where it can reach: RBAC, namespace relay only.

RBAC alone isn't enough, because a Deployment could mount a Secret. That's why the script only applies YAML from main, where every change has been reviewed.

Tested

• kubectl auth can-i … --as=system:serviceaccount:relay:deployer: patch deployments → yes. Get secrets, delete deployments, create statefulsets, anything in kube-system → no.
• The real badge, as the deploy user: patch deployments → yes, get secrets → no.
• With the deploy key (every case ended with exit 1):
  • whoami → rejected by gate 1.
  • A valid SHA + ; touch /tmp/pwned → rejected by gate 1, and the file was never created.
  • PR infra: add API pipeline (test, build, push to GHCR) #12's commit → rejected by gate 2.
  • Main's newest commit → passes both gates, then stops at "no k8s/app folder" (expected until 1.7).
• The script on the server is identical to the committed file. The laptop copy of the deploy key was deleted after the tests.

Check it yourself

From your laptop, on this branch:

ssh relay 'cat /usr/local/bin/relay-deploy' | diff - infra/server/relay-deploy.sh && echo same
ssh relay 'kubectl --kubeconfig ~/.kube/config auth can-i get secrets -n relay --as=system:serviceaccount:relay:deployer'

Please look at

• The Role's verbs: enough for apply and rollout undo, and nothing more?
• The two gates in relay-deploy.sh.

[AbdikhaniAbdirashid]

Ran your two checks plus a few more rather than taking the list on trust.

The script on the server is byte-identical to the committed file. Every permission is what you say it is: patch deployments, create services, patch ingresses, get replicasets, get pods and patch configmaps all yes; get secrets, delete deployments, create statefulsets, anything in kube-system, namespaces and nodes all no.

The part I'll remember is why RBAC alone isn't enough - a Deployment can mount a Secret the account can't read - and that gate 2 is what actually closes that, by only ever applying YAML that went through a reviewed PR. The three layers each answering one question is a clean way to hold it in your head.

On the Role's verbs

They look right to me for apply and rollout undo, and nothing is there that isn't needed. apply wants get + create + patch, rollout status wants watch on deployments and replicasets, rollout undo patches the deployment and reads the old replicaset. No delete is the right call.

One consequence worth writing down somewhere: without delete, removing a manifest from k8s/app/ doesn't remove the resource from the cluster. It keeps running until someone deletes it by hand. Not a problem, just a thing we'll both forget in three weeks.

Two things on relay-deploy.sh

1. rollout undo only reverts half of what step 4 applied.

Step 4 applies Deployments, Services, ConfigMaps and Ingresses from the new commit. kubectl rollout undo reverts the Deployment's pod template - the image - and nothing else.

So if a ConfigMap change is what breaks the rollout, the undo brings back the old pods and leaves them with the new config, and they fail the same way. The deploy reports a failure and rolls back, but the cluster is now a mix: old image, new everything else.

Does that matter before the pipeline deploys ConfigMaps that can break things? Probably not for 1.7. But it's the kind of thing that is much easier to reason about now than at 2am during the VG demo. A git checkout of the previous commit and a re-apply would be a real rollback; rollout undo is a partial one.

2. deployment/relay-api is hardcoded.

Fine today, since k8s/app/ only has the API. In phase 4 the worker arrives and k8s/app/ has two Deployments. Then:

• apply updates both;
• rollout status only waits for the API, so a worker that never becomes ready is reported as a successful deploy;
• rollout undo only reverts the API, leaving API and worker on different commits.

Worth a note in the script now so phase 4 doesn't inherit it quietly - something like iterating over the Deployments the apply touched, or at least a TODO.

Smaller

flock is taken before gate 1, so an invalid SHA still waits up to 5 minutes for the lock before being rejected. Only reachable by someone who already has the deploy key, so it doesn't matter - but moving gate 1 above the lock costs nothing.

One question from my side

relay-deploy only applies k8s/app/, and my dashboard manifest is in k8s/dashboard/. So as it stands the pipeline deploys your half and nothing deploys mine.

Should the dashboard move into k8s/app/ and ride the same path, or get its own? Either is fine by me - I'd just rather decide it now than find out at 1.8. The G requirement says every merge is deployed automatically, and right now that would only be true for the API.

The deployer Role covers Deployments, Services, ConfigMaps and Ingresses, which is exactly what my manifest needs, so moving it in looks like it would just work.

From my side

The README is the clearest thing in the repo so far - the deploy-path diagram and the "if the deploy key leaks" section are exactly what I'd want to read at the defense, and the by-hand deploy command is good to have.

Approving. Neither point blocks 1.7, I'd just rather we decide on the rollback semantics before the pipeline is the only thing deploying.

[devinder-dev]

Thanks for running the checks yourself, and for the failure case we hadn't tested. You're right on all four points. They'll go into one small follow-up PR before 1.8, so the pipeline never deploys with the old behaviour:

1. Real rollback. Before applying, the script notes which commit is running now. If the rollout fails, it deploys that commit again the same way (checkout, apply, wait). That brings the ConfigMaps, Services and Ingresses back too, not only the image. One limit stays: something that only the failed commit added is left behind, because there's no delete (your point 4).
2. Every Deployment. It waits for each Deployment the apply touched (kubectl apply -o name), not only relay-api. So the worker in phase 4, and your dashboard, are covered.
3. Gate 1 above the lock. Agreed, it costs nothing.
4. No delete. A line in k8s/README.md: removing a file from k8s/app/ doesn't remove it from the cluster, one of us deletes it by hand.

On the dashboard: it goes in k8s/app/, same path and same gates. One commit = one release of the whole system.

That has one consequence for 1.8: every merge to main must build both images with the same SHA. Otherwise a dashboard-only commit would point at a relay-api:<sha> that was never built, and the rollout would fail. So the pipeline becomes: test both → build both → deploy once. The build cache keeps an unchanged image to a few seconds.

When you push your manifest, it should:

• live in k8s/app/ (for example dashboard.yaml);
• use ghcr.io/devinder-dev/relay-dashboard:__GIT_SHA__ as the image;
• have no Namespace object, because the deployer account can't create one and the apply would fail with "forbidden".

#12 (the pipeline) is rebased onto main now: one commit, two files, checks green. It's ready for your review.