[devinder-dev]

Step 1.8: every merge to main is deployed, then checked from outside. Rebased onto main now that #12, #14 and #15 are merged. Merge only after the switch-over below.

What changes

• api.yml → pipeline.yml. It now deploys too, and the dashboard's jobs will join it later (needs: only works inside one workflow file).
• No path filter on main. Every merge is a release: a deploy needs an image tagged with exactly the merged commit, even when only k8s/app/ or the docs changed.
• New job deploy-script-test on every PR and merge. It runs infra/server/test-relay-deploy.sh (relay-deploy's gates and rollback, with fake git and kubectl) and shellcheck on the server scripts. The deploy needs it, so a broken deploy script never reaches the server.
• New job deploy, after api-image and deploy-script-test, only on a merge to main:
  • environment: production: the only way to get DEPLOY_SSH_KEY, and only on main.
  • permissions: {}: no GitHub access at all.
  • timeout-minutes: 30: the worst case is the server's lock (15 min, infra: deploy script follow-ups from the #14 review #19) plus a rollout and a rollback.
  • Step 1: writes the key and our server's known_hosts, then ssh … deploy@<vps> "$GITHUB_SHA" (StrictHostKeyChecking=yes, BatchMode=yes). The exit code of relay-deploy becomes the step's.
  • Step 2: asks https://api.myrelay.tech/version until it answers this commit, for up to 36 × 5 s, because the first deploy also waits for a new certificate.
• timeout-minutes: 10 on api-test and api-image (from the infra: add API pipeline (test, build, push to GHCR) #12 review).
• k8s/hello/ is deleted, together with its row in k8s/README.md. The API's Ingress takes over api.myrelay.tech.

Tested

• actionlint 1.7.7 (with shellcheck on the run: scripts): no findings.
• The new job's two steps, run locally: 26 passed, 0 failed, and shellcheck is clean.
• The version check, run on its own with bash -eo pipefail against a local container:
  • the right SHA → passes;
  • an old SHA → keeps asking, then fails;
  • nothing answering → "no answer" on every try, then fails.
• The rebase was rehearsed in a throwaway clone first: the 3 commits replayed onto main without conflicts.

Already done before merging

• infra: add API pipeline (test, build, push to GHCR) #12, infra: roll back to the last good commit and wait for every Deployment #14 and infra: add the API manifests (2 pods, probes, zero-downtime rollout) #15 merged.
• deployer.yaml applied: the deployer can read events, and still can't read Secrets or delete.
• relay-deploy installed, byte-identical to main.
• The image is pullable without credentials (ADR 0004, docs: decide that our container images stay public (ADR 0004) #18), and the server has already pulled relay-api:a8274c1.

Switch-over (right before merging)

1. Delete the hello app from default: the Ingress, Service and Deployment, plus its leftover TLS Secret api-myrelay-tech-tls. api.myrelay.tech is down for about 1–2 minutes, once.
2. Merge → the pipeline tests, builds, deploys → /version answers the commit.

After the first deploy

• A real rollback: deploy infra: add the API manifests (2 pods, probes, zero-downtime rollout) #15's merge commit a78934c by hand. It has no image, so the rollout fails and relay-deploy redeploys the last good commit, while a curl loop shows only 200s.
• Zero downtime: a load loop during a normal deploy (merging infra: deploy script follow-ups from the #14 review #19) should show no failed requests.
• Then make api-test and deploy-script-test required checks.

[AbdikhaniAbdirashid]

Not reviewing this yet since it's a draft and you've said it changes on the
rebase. But two things are better said before that rebase than after.

1. The handover outage needs a time, not just a step.

Merge plan step 5: "delete the hello app from the cluster by hand.
api.myrelay.tech is down for about 1–2 minutes, once."

That's our only public hostname, and the demo is Wednesday. I'm not arguing
against it — the handover has to happen — but I'd rather we pick the moment
deliberately than discover it's happening while one of us is mid-something.

Two things make me think it could run longer than 1–2 minutes:

• cert-manager has to issue a new certificate for namespace relay. Secrets
  don't cross namespaces, so hello's certificate can't be reused. If Let's Encrypt
  is slow or we hit a rate limit, the site is down until it lands, and the first
  thing we'd want to do is retry — which is exactly what the rate limit punishes.
• Your own version check allows 36 × 5 s ≈ 3 minutes partly because of that
  certificate. So the pipeline already assumes this can take longer than 2 minutes.

Can we do the switch-over together, at a set time, with the staging issuer tried
first? And what's the plan if the certificate doesn't come — do we put hello back?

2. timeout-minutes: 20 is sized for one Deployment.

The comment above it says: worst case is the lock (5 min) + rollout (3) + rollback (3)
= 11 minutes. That holds while k8s/app/ has one Deployment. It will have two as
soon as the dashboard lands, and relay-deploy waits --timeout=180s on each
Deployment it applied, in both the deploy and the rollback:

	1 Deployment	2 (API + dashboard)	3 (+ worker, phase 2)
lock	5	5	5
rollout	3	6	9
rollback	3	6	9
total	11	17	23

At three it goes over 20 and GitHub kills the job — during the rollback, which is
the worst possible moment: the cluster is left half-reverted and the log just says
the job timed out, with none of the show_state output that would tell us why.

This is the same root cause as my line-30 comment on #14, where flock --wait 300
is also sized for one Deployment. Both numbers derive from "180 s × however many
Deployments are in k8s/app/", and right now that relationship isn't written down
anywhere. Worth fixing in one go across both PRs rather than twice.

Keeping this as a plain comment rather than line comments on purpose — you're
rebasing this branch, and inline comments would just go stale.

[AbdikhaniAbdirashid]

Approving. Both things I raised while this was a draft are handled: the lock arithmetic in the comment above timeout-minutes now matches #19's 15 minutes and counts two Deployments, and needs: [image, deploy-script] means a broken deploy script can't reach the server. Adding deploy-script-test to CI is better than what I asked for — I only wanted the numbers to stop drifting, and you made the script's own tests run on every PR instead.

What I checked

• needs: [image, deploy-script] is wired as the comment claims.
• The version-check loop ends with ::error:: and exit 1, so a check that never sees the commit turns the job red rather than passing quietly.
• concurrency on main queues rather than cancels, so a running deploy is never killed halfway.
• The hello Ingress still holds api.myrelay.tech right now — the handover in merge-plan step 5 is still ahead of us.

One line comment on the version-check timing. It's arithmetic, not a design problem.

On merge-plan step 5, repeating what I said before only because it's the part with no undo: a 1–2 minute outage on our only public hostname, with a fresh certificate in the middle of it. Can we do that together, at a time we pick, rather than as a side effect of clicking merge? I'd like us both watching when api.myrelay.tech goes dark.

[AbdikhaniAbdirashid]

36 × 5 s ≈ 3 minutes counts the sleep but not the curl.

Each turn of the loop is curl --max-time 10 plus sleep 5, so the worst case per attempt is 15 s, not 5:

	fast curl	slow curl
per attempt	5 s	15 s
36 attempts	3 min	9 min

Normally curl returns in milliseconds and 3 minutes is right. But the slow case isn't independent of the failing case — they're the same case. If Traefik isn't answering yet during the Ingress handover, connections hang until --max-time fires, and that's also when we'd actually use all 36 attempts.

That matters for the job budget:

deploy succeeds:  lock 15 + rollout 6          = 21 min
                  + version check (worst case)   9 min
                                               = 30 min   <- exactly timeout-minutes: 30

So on the one run where the verification is struggling, the job can be killed by the timeout instead of printing your ::error:: line — and we'd lose the per-attempt log that says what it saw, which is the whole point of the loop.

Easiest fix is probably --max-time 5 plus a shorter sleep, keeping the wall-clock budget but making it predictable. Or state the real worst case in the comment and raise the job timeout. Either is fine; I mainly don't want the number in the comment to be the one we trust when we're debugging the first deploy at 23:00 on Tuesday.

[devinder-dev]

Fixed in the latest push. The deploy timeout is now 35 minutes, and the comments include curl’s timeout: 36 × (10s curl + 5s sleep) = up to 9 minutes, plus overhead.

For two Deployments, the successful path budgets 30 minutes, leaving 5 minutes for overhead. The rollback path budgets 27 minutes and exits without running the public check.

I’ll coordinate the switch-over time with you before removing hello and merging

[AbdikhaniAbdirashid]

Re-approving for the new push — require_last_push_approval is on, so my approval of 67b29dd doesn't carry over to d00063d.

The budget reads right now, and more importantly it shows its working:

path	budget	headroom to 35
success	15 lock + 6 rollout + 9 public check = 30	5 min
rollback	15 + 6 + 6 = 27, no public check	8 min

Naming the curl timeout in the comment is the part that matters. The next person to change --max-time or the attempt count can now see what else moves with it, which is exactly what went wrong the first time round.

Thanks for picking up the switch-over too — I'll be around whenever suits you.