Hold the production lock across production's verification and rollbacks

.github/workflows/deploy-test.yml

@@ -3,13 +3,14 @@ name: Deploy to test droplet
 # Dispatch-only. This exists so the production auto-rollback path can be
 # exercised end to end without breaking production to do it — CI had no route to
 # retina-test at all before this (no TEST_HOST secret, no job referencing it),
-# and `deploy-production` is gated on `push` to main, so a workflow_dispatch run
-# skips it and the rollback job with it.
+# and the production chain is gated on `push` to main, so a workflow_dispatch
+# run skips it and the rollback job with it.
 #
-# The deploy and rollback logic below MIRRORS ci.yml's deploy-production and
-# rollback-production-on-deploy-failure. ci.yml is the source of truth; this is
-# a copy, and a copy can drift. If you change the marker lifecycle or the
-# rollback decisions there, change them here, or this stops verifying anything.
+# The deploy and rollback logic below MIRRORS production-deploy-verify.yml's
+# deploy-production and rollback-production-on-deploy-failure. That file is the
+# source of truth; this is a copy, and a copy can drift. If you change the
+# marker lifecycle or the rollback decisions there, change them here, or this
+# stops verifying anything.
 # staging-deploy-verify.yml carries the same pair for staging; keep all three
 # in step.
 on:
@@ -104,7 +105,7 @@ jobs:
             set -e
             cd ${{ env.APP_DIR }}
 
-            # Identity first, for the reason ci.yml gives: the only check that
+            # Identity first, for the reason production gives: the only check that
             # does not depend on the state of the box. Exact match — retina-test
             # and nothing else, so a mistyped TEST_HOST cannot land a deliberate
             # failure injection on staging or production.
@@ -132,7 +133,7 @@ jobs:
               exit 1
             fi
 
-            # Mirrors ci.yml's .env pre-flight, and matters more here than
+            # Mirrors production's .env pre-flight, and matters more here than
             # there. pre-deploy.sh reads the running image via `docker compose
             # images`, which resolves through whatever ./.env names, so a .env
             # naming another environment would have it snapshot the wrong
@@ -249,7 +250,7 @@ jobs:
     concurrency:
       group: test-deploy
       cancel-in-progress: false
-    # Mirrors ci.yml: keyed to the deploy job's own result, and covering
+    # Mirrors production: keyed to the deploy job's own result, and covering
     # 'cancelled' as well as 'failure'.
     if: >-
       always() && (

.github/workflows/staging-deploy-verify.yml

@@ -79,8 +79,8 @@ jobs:
               exit 1
             fi
 
-            # Same check as ci.yml's production deploy, which has the reasoning:
-            # a marker is a box left mid-deploy, and snapshotting over it would
+            # Same check as the production deploy, which has the reasoning: a
+            # marker is a box left mid-deploy, and snapshotting over it would
             # lose the rollback point.
             if [ -f .deploy-in-progress ]; then
               echo "::error::${{ env.APP_DIR }}/.deploy-in-progress is still present on $(hostname) ($(cat .deploy-in-progress)). An earlier deploy was left mid-flight and never confirmed healthy. Inspect the box and recover it (bash deploy/rollback.sh, or fix forward), then delete the marker and re-run. Refusing to deploy; the running stack is untouched."
@@ -134,7 +134,8 @@ jobs:
             # suite by it and fail closed without one (backend/routes/test.py),
             # so a box with none fails every smoke run and holds production
             # back. The last line is read because it is the one compose passes
-            # on (see the same check in ci.yml's deploy-production).
+            # on (see the same check in production-deploy-verify.yml's
+            # deploy-production).
             if ! grep -E '^RADAR_API_KEY=' backend/.env 2>/dev/null | tail -1 | grep -qE '^RADAR_API_KEY=.+$'; then
               echo "::error::backend/.env on $(hostname) has no effective RADAR_API_KEY. The test router's reads admit the smoke suite only by that key, so the smoke run would fail. Set it to the STAGING_RADAR_API_KEY secret's value and re-run. Refusing to deploy; the running stack is untouched."
               exit 1
@@ -211,7 +212,8 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 20
     needs: [deploy]
-    # Keyed the same way as ci.yml's rollback-production-on-deploy-failure;
+    # Keyed the same way as production-deploy-verify.yml's
+    # rollback-production-on-deploy-failure;
     # see there for why. No concurrency group of its own: the calling job
     # holds `staging-deploy` until every job here has finished (see the
     # header), so a following run's deploy cannot enter the directory while
@@ -244,7 +246,7 @@ jobs:
 
             # No marker: the deploy refused in a pre-flight with the stack
             # untouched, or it had already cleared the marker with the app
-            # healthy on the new build. Neither is a rollback case; ci.yml's
+            # healthy on the new build. Neither is a rollback case; the
             # production pair has the reasoning.
             if [ ! -f .deploy-in-progress ]; then
               echo "No .deploy-in-progress marker: the deploy either refused before touching the box or failed after the app was already healthy on the new build. Nothing to roll back."
@@ -255,7 +257,7 @@ jobs:
 
             # Another run's marker: this run touched nothing and cannot tell
             # a deploy still in progress or an unrecovered outage from a box
-            # already fixed by hand, so stop rather than guess (see ci.yml).
+            # already fixed by hand, so stop rather than guess (see production).
             case "$MARKER" in
               "run ${{ github.run_id }} "*) ;;
               *)

ONBOARDING.md

@@ -319,10 +319,12 @@ CI runs on every PR, on push to `main`, and on demand through
    markdown, and Python whose syntax tree has not moved: a reworded comment or a
    `ruff format` pass ships nothing. `deploy/deploy-scope.py` holds the rules and
    the exceptions, and its tests hold the verdicts.
-   The staging third of it is a called workflow,
-   `.github/workflows/staging-deploy-verify.yml`, invoked from one `Staging`
-   job so that job's concurrency group is held across the deploy and both
-   suites. Adding a staging step means editing that file, not `ci.yml`.
+   Each environment's part of it is a called workflow,
+   `.github/workflows/staging-deploy-verify.yml` and
+   `.github/workflows/production-deploy-verify.yml`, invoked from one `Staging`
+   or `Production` job so that job's concurrency group is held across the
+   deploy, both suites and the rollbacks. Adding a deploy or verification step
+   means editing those files, not `ci.yml`.
    Both deploys take a rollback point first and roll themselves back when
    they fail after it; the runbook's Environments section has the shape.
 
@@ -331,10 +333,11 @@ branch, open a PR, get it green, then merge.
 
 ## Things that will bite you
 
-- **A cancelled `Staging` job on a burst of merges is expected, not a fault.**
-  Only one run may sit pending on the `staging-deploy` group, so when a third
-  merge arrives while one run holds staging and another is queued, the queued
-  one is cancelled. `main` is linear, so the run that replaces it deploys a
+- **A cancelled `Staging` or `Production` job on a burst of merges is expected,
+  not a fault.** Only one run may sit pending on each of the `staging-deploy`
+  and `production-deploy` groups, so when a third merge arrives while one run
+  holds an environment and another is queued for it, the queued one is
+  cancelled. `main` is linear, so the run that replaces it deploys a
   superset of what was dropped. What it does mean is that the cancelled
   commit's own run never reaches production: the following run carries it.
   The last merge in a burst has no successor, so check it landed.