Add a restricted deploy gate and one deploy script for every droplet

[Babissimo]

ClickUp: 1.6 Deploy through a restricted SSH gate with one on-box deploy.sh

Why

CI deploys by logging in to each droplet as root with a key from a repository secret, and it sends the whole deploy script inline. There are three copies of that script (ci.yml, staging-deploy-verify.yml and deploy-test.yml), and they have already drifted. A stolen key is a root shell.

This PR adds the on-box half. The workflows are unchanged. Each droplet switches to ssh <alias> <verb> in its own later PR, once its gate is installed and its new key is authorised: test first, then staging, then production.

What changes

deploy/gate.sh, installed as /usr/local/sbin/retina-deploy-gate

This is the only command a CI key will be authorised to run, as restrict,command="/usr/local/sbin/retina-deploy-gate retina-server".

• Four verbs. It reads SSH_ORIGINAL_COMMAND and accepts deploy, rollback, probe-towers and status. Anything else is refused.
• Character filter. Any request with a character outside letters, digits, ._/@+- and spaces is refused before parsing. The locale is pinned, so a client's LC_* cannot widen that set.
• deploy <run-id> fetches refs/heads/main itself and runs deploy/deploy.sh as it stands in that commit, with a clean environment. Nothing it runs comes from the caller. The ref is spelt out because a bare main also matches a tag.
• deploy <run-id> <branch> <fail-at> is the form on test. It fetches refs/heads/<branch> and nothing else, so a pull request's ref is never fetched. The test key can therefore run any pushed branch as root on test, and the header says so. A branch deploy never replaces the gate.
• probe-towers asks the contract in main's tip, which is what the next deploy ships, not the contract in the tree the box is running. It fetches into a ref of its own.
• Lock. Deploys and rollbacks queue on a lock for up to 15 minutes. A caller that gave up while its call was waiting stops the call before it acts. Git is kept from detaching maintenance that would inherit the lock.
• Output relay. Output reaches the caller through a relay that also logs to .deploy-gate.log on the box and outlives the caller. A dropped connection therefore cannot kill a deploy or rollback mid-step.

One rollback decision

The caller now says which of its jobs failed.

• rollback <run> deploy (the deploy failed or was cancelled) acts only on a marker naming the run, as the inline rollbacks do. It never acts on a finished deploy, so a re-run whose deploy refuses in a pre-flight cannot undo the healthy deploy before it.
• rollback <run> checks (smoke tests or E2E failed) acts on the last deploy record. deploy.sh writes that record once a deploy is healthy. It rolls back when the record names the run, still matches what is checked out, and finished within 90 minutes. Past 90 minutes it refuses loudly, so a broken deploy is not left up behind a green step. A deploy that some later run has replaced is left to that run's own checks.
• Other markers. Any other run's marker is refused, and so is an empty marker.
• Interrupted rollbacks. A rollback marks the box while it runs, and a failed rollback leaves the mark, so the next deploy refuses. The same run can resume it.

deploy/deploy.sh

• What it takes over. It runs the sequence the three inline copies share, plus their per-environment differences: production's secret checks, the origin-pull CA check on production and staging, staging's fleet-container cleanup, and test's retina-edge network and injected failures.
• The record. It removes the last deploy's record before pre-deploy.sh retakes the rollback point.
• Gate refresh. After a healthy deploy of main it refreshes the gate. A failure there only warns, so it cannot turn a good deploy into a rollback.

Supporting changes

• tower-contract.sh gains assert_tower_contract_over_edge, the probe tower-service-contract currently runs inline on staging. Its curl image is pinned by digest.
• setup-server.sh installs main's copy of the gate, whatever tree the box holds. Only a failure there is a warning; it doesn't stop provisioning.
• A shellcheck pre-commit hook covers every shell script at warning level. ONBOARDING's list of what the hook runs now includes it.
• .gitignore and .dockerignore gain the gate's state on the box.

Known limits

• deploy ships main's tip at fetch time, as the inline deploys do today, not the commit the calling run tested.
• deploy.sh runs the checked-out tree's pre-deploy.sh before it moves the tree, as today.
• A holder of a gate key can roll back the latest deploy within the 90-minute window. Run ids are public.
• The gate installed by a healthy deploy is the one that later rollbacks run. Changes to it are guarded by the behavioural tests below.
• Until each droplet's switch PR lands, ci.yml keeps its inline copies of the deploy and of the edge probe, alongside the new scripts.

Verification

• backend/tests/test_deploy_scripts.py (88 tests) runs the real gate and deploy against a scratch clone. Stand-in commands replace hostname, docker, df, openssl, flock and sleep, and the tests pass under bash 3.2 and 5. They cover:
  • the verbs, the character set, and branch, tag and pull-request refs;
  • the deploy order: rollback point, marker, tree move, prune, swap and health;
  • every pre-flight refusal, each of which touches nothing;
  • test's injected failures;
  • the full rollback table, including an interrupted rollback;
  • a caller who disconnects mid-deploy, and one who gives up while waiting for the lock;
  • status, the edge probe, and the install.
• Breaking the scripts in each of 32 ways, one at a time, fails the test that pins that behaviour.
• The shellcheck hook fails a scratch script that carries warnings.
• The workflow tests are unchanged and pass. test_rollback_script.py now also holds deploy.sh's health wait to rollback.sh's.
• Nothing has run on a droplet yet. The per-droplet operator steps are on the ClickUp ticket, and each droplet is exercised by its switch PR: on test, deploy-test.yml with each fail_at; on staging and production, the next real merge.

🤖 Generated with Claude Code

[Babissimo]

Rebased onto main at 14c63f8 and reviewed again. The description is left as it was, so here is what has changed since it was written:

• Health wait. deploy.sh now waits with deploy/wait-for-health.sh from the deployed tree, as the three inline copies have since Poll the deploy health gate every second, from one shared script #615, instead of a loop of its own. test_rollback_script.py is back to main's version, so the verification bullet about it no longer applies. The deploy tests run a stand-in for the wait; test_wait_for_health.py covers the real one.
• Radar key. Staging's deploy now refuses without an effective RADAR_API_KEY in backend/.env, as its inline copy has since Send the radar key with every deploy check that reads the test router #606. Production keeps both of its secret checks, and test needs neither.
• rollback <run> deploy on a finished deploy. When the last deploy record names the run and is still what is checked out, it now refuses loudly (exit 1, nothing touched) instead of reporting nothing to do. A caller that went away while the deploy carried on then shows red, since its checks never ran. A re-run whose deploy refused after an earlier attempt deployed shows red too, which it did not before.
• Record before marker. deploy.sh writes .last-deploy before it removes the marker, so a deploy that dies between the two is still rolled back.
• Clone check. Every verb, status and probe-towers included, now refuses on a deploy directory that is not a git clone. Before, only deploy checked.
• Install. A failed install removes its staged file and refuses, leaving the gate already in place unchanged.
• Edge probe. _edge_curl gains --connect-timeout 10, as the review bot suggested.
• Tests. test_deploy_scripts.py now has 100 tests, including setup-server.sh's install of the gate. Breaking the scripts in each of 45 ways, one at a time, fails the tests that pin them.

ci.yml is still untouched by this PR.

[claude]

Claude finished @jehanazad's task in 14m 40s —— View job

---

Code review

Todo list

• Gather context (CLAUDE.md, PR description, prior reviews, changed files)
• Manually trace deploy/gate.sh, deploy/deploy.sh, deploy/tower-contract.sh, deploy/setup-server.sh against the PR's claims, cross-checked against the unchanged rollback.sh/pre-deploy.sh and the test suite
• Run parallel review agents (line-by-line diff, removed-behavior audit, cross-file tracing, reuse/simplification/efficiency, altitude/conventions)
• Verify candidate findings against the actual code and test suite, synthesize, and post

Summary

This is the third review pass on this PR (two earlier passes found no blocking issues). I re-traced gate.sh/deploy.sh/tower-contract.sh/setup-server.sh line-by-line against the PR's claims, confirmed the behaviors described in the 2026-09-25 update (health-wait delegation, staging's radar-key check, "finished deploy" refusing loudly, record-before-marker ordering, universal clone check, install-failure cleanup, edge-probe timeout) all match the code and are genuinely exercised by test_deploy_scripts.py's 100 real-subprocess tests (verified several directly, e.g. the lock-queue test's flock stub actually blocks via a hold/go file handshake, not just a mock). Six independent review angles (my own pass plus 5 parallel agents covering reuse, efficiency, simplification, altitude, removed-behavior, and cross-file tracing) turned up nothing that breaks a currently-exercised path. No blocking issues.

A few narrow, non-blocking gaps worth a look:

1. deploy/deploy.sh:117 — the .env overlay-identity preflight check (grep -q "^COMPOSE_FILE=.*docker-compose\.${DEPLOY_ENV}\.yml" .env) matches any line in the file, unlike the JWT_SECRET/RADAR_API_KEY checks a few lines below it (and rollback.sh's resync_env_to_tree), which explicitly read only the last COMPOSE_FILE= line because compose honours the last duplicate key. A .env carrying a stale COMPOSE_FILE= line from a previous environment ahead of the real one would pass this identity check even though it shouldn't. Narrow (requires an already-anomalous .env), but inconsistent with the reasoning the surrounding code states out loud. Fix this →

2. deploy/gate.sh:334 — main() trusts an inherited SSH_ORIGINAL_COMMAND whenever it's set in the environment, even for a by-hand invocation. The header comment explicitly invites manual use ("by hand, root may pass the verb as arguments", and several refuse() messages tell an operator to run things "by hand" during an incident). If that operator's own session was itself started as ssh host <command> (so sshd left SSH_ORIGINAL_COMMAND set for the whole session), typing retina-deploy-gate retina-server status would silently run the stale SSH_ORIGINAL_COMMAND instead of status. Narrow and surfaces as a confusing refusal rather than something silent/dangerous, but it's exactly the path an operator reaches for under pressure. Fix this →

3. deploy/tower-contract.sh's _edge_curl (~line 298) — curl's --connect-timeout/--max-time bound the HTTP request inside the container, but nothing bounds the docker run call itself. If curlimages/curl:8.22.0@sha256:... isn't already cached on the box (fresh box, or an evicted cache), a stalled image pull hangs probe-towers indefinitely rather than failing within TOWER_CONTRACT_MAX_TIME. probe-towers deliberately runs without take_lock, so this wouldn't block deploys, but it would hang whatever CI step calls it. Fix this →

Everything else the parallel agents raised turned out to be either already called out explicitly in the PR's own "Known limits" (the 90-minute rollback window and 15-minute lock-wait replacing the old workflows' unbounded waits; the self-installing gate's trust model), the already-acknowledged temporary duplication with the still-live inline ci.yml copies pending each droplet's switch PR, test-harness realism gaps rather than production bugs (the flock/openssl stand-ins not exercising real locking/parsing semantics), or style/premature-abstraction suggestions that cut against this repo's own stated preference for a few similar lines over a new abstraction.

Sandbox limitation: shellcheck and other non-trivial shell invocations required interactive approval unavailable in this environment, so I could not actually execute the shellcheck hook or the pytest suite here — this review is from reading the scripts and the test file's real assertions, not from running them. The two prior review passes noted the same limitation.
· branch