diff --git a/docs/ADRs/022_tag_based_deployment_gate.md b/docs/ADRs/022_tag_based_deployment_gate.md index 37a8c1f..19e030e 100644 --- a/docs/ADRs/022_tag_based_deployment_gate.md +++ b/docs/ADRs/022_tag_based_deployment_gate.md @@ -51,7 +51,7 @@ Normally, `HEAD` points to a branch (e.g., `main`), and the branch points to a c This is **normal and expected on the server**. It means: - The server is looking at a frozen snapshot, not a moving branch. - No `git pull` or `git push` can accidentally change what's checked out. -- The only way to change the running version is to update `~/.views-deploy-tag`. +- Nothing changes the running version except an operator deploying — and deploying is the whole procedure below, not just the tag file. On developer machines, you work on branches as usual. Detached HEAD is only relevant on the deployment server. @@ -112,7 +112,7 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co ### Positive - **Explicit version control.** The operator decides exactly what runs. No surprises from pushed commits. -- **One-line rollback.** `echo 'v1.0.0' > ~/.views-deploy-tag` reverts to the previous release. +- **Cheap rollback.** The previous release is always available — git never deletes tags — so reverting is the same short procedure as deploying, run with the earlier tag. (It is **not** one line; see the amendment below.) - **Audit trail.** The deploy tag file, git log, and provenance ledger together show what version produced what data. - **Safe for multiple maintainers.** Two people can push to `development` without affecting the server. - **No infrastructure dependencies.** No Docker registry, no CI/CD pipeline, no deployment service. Just git tags and a text file. @@ -127,26 +127,24 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co ## Deployment Procedure -**Deploy a new version:** +**On your laptop:** the release ritual in `docs/guides/publishing_to_pypi.md` — bump, promote to `main`, cut a **GitHub Release** (a local `git tag && git push` does not trigger the publish workflow, and both branches have required pull requests since 2026-07-31). -1. On your laptop: `git tag v1.2.0 && git push --tags` -2. On the server: `echo 'v1.2.0' > ~/.views-deploy-tag` -3. Wait for cron (21st of month) or run manually +**On the server:** every block of `docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, `git fetch --tags && git checkout `, `uv sync`, then verification against three sources: the tag file, the checked-out tag, and the **installed distribution version**. A rollback is the same procedure with the earlier tag. -**Roll back:** +The commands are deliberately not reproduced here. This ADR records *why* the gate exists; the quickref is its single operational home, and **if they disagree, the quickref wins.** -1. On the server: `echo 'v1.0.0' > ~/.views-deploy-tag` -2. Run the pipeline manually or wait for cron +### Amendment 2026-10-13 — writing the tag file is not deploying (C-343) -**Verify:** +This section previously gave deploying as three steps — local tag, tag file, wait for cron — and rollback as one. Both were wrong, and the error was not theoretical: on 2026-08-08 the production host ran v1.10.0 for five days while the tag file read v1.11.0, with a green status page, a pinging heartbeat, and no error at any point. v1.11.0 existed solely to raise the `views-frames` floor, and the floor on the box was still the frozen 1.0.0. -```bash -cat ~/.views-deploy-tag # configured version -git describe --tags --exact-match # checked-out version -tail -5 logs/refresh.log # last run output -``` +The gate described in **Decision** is unaffected and still does what it claims: the pipeline refuses to start without a valid tag, and checks that tag out. What was wrong was the inference that the checkout makes the tag file sufficient. It does not, for two reasons established by experiment rather than by reading: + +1. Bash buffers `scripts/refresh_pipeline.sh` before the gate's `git checkout` runs, so a change to that script takes effect on the **next** run — one month, on a monthly cron. +2. The pipeline never invokes `uv sync`, so a dependency change never lands at all, however many times it runs. + +The **Verify** block was also incomplete in the way that mattered: it listed the tag file and `git describe` but not the installed distribution, which is the only one of the three that reflects `uv sync`. In August it was the installed version that was wrong. -Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, section "Deployment and releases." +C-343 is **open**: the three guides and this ADR now agree, but nothing mechanically prevents the procedure from being skipped. The proposed instrument — a pre-flight check comparing the deploy tag against the HEAD the script *started* from, failing loud per ADR-011 rather than self-correcting one run late — is tracked in #424 and not built. --- @@ -154,7 +152,7 @@ Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, se - ADR-011 (Fail Loud, No Stale Data Serving) — justifies fail-loud on misconfiguration - ADR-018 (Operational Resilience Policy) — operator-mediated bounded staleness -- `scripts/refresh_pipeline.sh` lines 76-98 — implementation +- `scripts/refresh_pipeline.sh`, the "Deployment gate" block — implementation - `reports/archive/technical_risk_register_resolved.md` C-98 — risk that motivated this ADR - `docs/guides/hetzner_deployment_guide.md` — operational procedures - `docs/guides/data_serving_guide.md` section 9 — conceptual explanation diff --git a/docs/guides/data_serving_guide.md b/docs/guides/data_serving_guide.md index 71cf3a3..8c38a76 100644 --- a/docs/guides/data_serving_guide.md +++ b/docs/guides/data_serving_guide.md @@ -250,13 +250,22 @@ notice for days. With the tag system: the server keeps running the old working version until you *explicitly* tell it to switch. You test the new version, -tag it, and only then update the server. If the new version breaks, -you write the old tag name back and you're running known-good code -again. - -**The full deployment procedure** (create a tag, push it, update the -server file, rollback if needed) is documented in -`hetzner_deployment_guide.md` under "Deployment and releases". +release it, and only then deploy it. If the new version breaks, you +deploy the previous tag and you're running known-good code again. + +**One thing this explanation makes tempting, and it is wrong.** Because +the pipeline checks the tag out for you, it looks as though changing the +tag file *is* deploying. It is not. The script is already running when it +does that checkout, so a change to the script itself only takes effect +the run after — a month, on a monthly cron — and the script never +installs dependencies, so a dependency change never lands at all. That +misreading left the server on v1.10.0 for five days with every signal +reading v1.11.0 (C-343, August 2026). + +**The deployment procedure** — tag file, checkout, `uv sync`, then verify +three sources agree — lives in `server_quickref.md` under "Deploy a new +version", and only there. A rollback is the same procedure with the +earlier tag. The architectural rationale (why tags instead of branch tracking, Docker, or CI/CD) is in **ADR-022**. diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 6cd8a91..4fc2bc1 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -852,8 +852,11 @@ fixed order: diverge a little further after **every** release, and `git log main..development` starts counting work that already shipped. -5. **Deploy** = update `~/.views-deploy-tag` on the server to the - new tag. The pipeline checks out whatever tag that file contains. +5. **Deploy** = three steps on the server, not one: the tag file, then + `git fetch --tags && git checkout `, then `uv sync`. Writing the + tag file alone is **not** deploying — that mistake left the host on + v1.10.0 for five days while every signal read v1.11.0 (C-343). + `docs/guides/server_quickref.md` holds the commands. **Why this order:** If you tag before bumping, the tag points at code with the old version string — wrong. If you tag on `development` @@ -871,48 +874,51 @@ first, promote second, tag third, back-merge fourth. > corrected, because a second copy of the procedure is what let the two guides drift apart > in the first place (#402). One procedure, one home. -**On the server** (SSH in): - -```bash -# Update the deploy tag file -echo 'vX.Y.Z' > ~/.views-deploy-tag -``` - -That's it. The next cron run (21st of the month) will automatically -use the new tag. If you want to apply it right now instead of waiting: - -```bash -# Run the pipeline manually (same command cron uses) -cd ~/views-datafactory -bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log -``` +**On the server** (SSH in): run **every block** in +`docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, then +`git fetch --tags && git checkout `, then `uv sync`, then the three +verification commands. The commands are there and not here on purpose. + +> **This section used to say "update `~/.views-deploy-tag`. That's it."** It was +> wrong, and it is the sentence C-343 is about: on 2026-08-08 the tag file read +> `v1.11.0`, `git describe` read `v1.10.0`, and the installed `views-frames` was +> still the frozen `1.0.0` — three sources of truth disagreeing, with a green +> status page and a pinging heartbeat and no error anywhere. The in-pipeline +> `git checkout "$DEPLOY_TAG"` does not rescue it: bash has already buffered +> `scripts/refresh_pipeline.sh`, so a change to that script lands one run later — +> a month, on a monthly cron — and `uv sync` is never invoked by the pipeline at +> all, so a dependency change never lands however many times it runs. C-343 was +> corrected in `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` +> in #431 and **this third copy was missed**, which is the same drift #402 is +> about. The commands live in the quickref and nowhere else; if this file and the +> quickref ever disagree, **the quickref wins.** ### How to roll back -If a new version breaks the pipeline: +If a new version breaks the pipeline, re-run the same three deploy steps with the +previous tag — a rollback is a deploy, and skipping `uv sync` on the way back +leaves the dependencies of the version you are abandoning. Tags are never +deleted by git, so every previous version stays available. + +Then run the pipeline manually to apply it immediately rather than waiting for +the next cron run: ```bash -# On the server — point back to the previous version -echo 'vPREVIOUS' > ~/.views-deploy-tag +sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' ``` -The next pipeline run will check out that tag and run its code -instead. The old version is always available — git never deletes tags. - -You can also run the pipeline manually to roll back immediately -instead of waiting for the next cron run. - ### How to check what version is deployed -```bash -# What version is the server configured to run? -cat ~/.views-deploy-tag +**Three sources, and they must agree** — the tag file, the checked-out tag, and +the installed distribution. Checking only the tag file is what C-343 was: it is a +statement of intent, not evidence. The commands are step 4 of +`docs/guides/server_quickref.md` §"Deploy a new version"; they are not repeated +here, because a second copy of them is how this section came to be wrong. -# What version is actually checked out right now? -git describe --tags --exact-match 2>/dev/null || git log --oneline -1 +For when the last pipeline run happened, `tail` the refresh log: -# When did the last pipeline run happen? -tail -5 logs/refresh.log +```bash +sudo -u views-deploy tail -5 ~/views-datafactory/logs/refresh.log ``` ### Log rotation diff --git a/docs/guides/server_operations.md b/docs/guides/server_operations.md index 324b824..3308330 100644 --- a/docs/guides/server_operations.md +++ b/docs/guides/server_operations.md @@ -10,8 +10,8 @@ For initial setup, see `hetzner_deployment_guide.md`. | What | Command | |------|---------| | SSH in | `ssh @204.168.219.108` | -| Check current version | `cat /home/views-deploy/.views-deploy-tag` | -| Set version to deploy | `echo "v1.2.3" \| sudo tee /home/views-deploy/.views-deploy-tag` | +| Check current version | three commands — see `server_quickref.md` §"Deploy a new version" step 4; the tag file alone is intent, not evidence (C-343) | +| Deploy a version | four blocks — see `server_quickref.md` §"Deploy a new version"; writing the tag file is **not** deploying (C-343) | | Run pipeline | `sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'` | | Check pipeline logs | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/refresh.log` | | Check last failure | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/pipeline_failure.json` | @@ -113,24 +113,24 @@ GitHub tag ### Deploy a new version -On your laptop: -```bash -# 1. Tag the release (from main branch) -git tag v1.2.4 -git push origin v1.2.4 -``` - -On the server: -```bash -# 2. SSH in -ssh @204.168.219.108 - -# 3. Set the new version -echo "v1.2.4" | sudo tee /home/views-deploy/.views-deploy-tag - -# 4. Run the pipeline -sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh' -``` +**On your laptop:** the release ritual in `publishing_to_pypi.md` — bump, +promote to `main`, then cut a **GitHub Release**, which is what triggers the +publish workflow. A local `git tag && git push` does not, and both branches have +required pull requests since 2026-07-31, so the sequence that used to sit here +cannot be run any more. + +**On the server:** every block in `server_quickref.md` §"Deploy a new version" — +the tag file, `git fetch --tags && git checkout `, `uv sync`, then the three +verification commands. + +> **This section used to stop at the tag file**, and that is C-343: on 2026-08-08 +> the host ran v1.10.0 for five days while the tag file, and therefore every +> signal an operator checked, read v1.11.0. The pipeline's own `git checkout` +> does not rescue it — bash has already buffered `scripts/refresh_pipeline.sh`, +> so a change to that script lands one run later (a month, on a monthly cron), +> and `uv sync` is never invoked by the pipeline at all. The commands live in the +> quickref and nowhere else; **if this file and the quickref disagree, the +> quickref wins.** ### Run individual pipeline steps manually diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 401aa6e..0b5627c 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -12,17 +12,28 @@ All commands below assume you are on the server. ## Deploy a new version +**All four blocks. Step 1 alone is not a deploy (C-343)** — the pipeline's own +`git checkout` lands a change to `refresh_pipeline.sh` one run later, and never +runs `uv sync` at all. + ```bash -# Set version +# 1. Set version sudo -u views-deploy bash -c 'source ~/.profile && echo v1.2.6 > ~/.views-deploy-tag' -# Pull and checkout (git checkout -- uv.lock discards platform-local lock changes) +# 2. Pull and checkout (git checkout -- uv.lock discards platform-local lock changes) sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && git fetch --tags && git checkout -- uv.lock && git checkout v1.2.6' -# Sync dependencies +# 3. Sync dependencies sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv sync' + +# 4. Verify — all three must name the same version +sudo -u views-deploy cat ~/.views-deploy-tag +sudo -u views-deploy bash -c 'cd ~/views-datafactory && git describe --tags --exact-match 2>/dev/null || git log --oneline -1' +sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv run python -c "import importlib.metadata as m; print(m.version(\"views-datafactory\"))"' ``` +A **rollback is a deploy**: same four blocks, previous tag. + ## Run the pipeline ```bash @@ -77,7 +88,8 @@ sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv ## Check status ```bash -# Current deployed version +# Version the server is CONFIGURED to run (intent, not evidence — +# use all three checks in §"Deploy a new version" step 4 to confirm) sudo -u views-deploy cat ~/.views-deploy-tag # Last run duration diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index ab75ac6..e248cb0 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3241,6 +3241,20 @@ Bash buffers the script and does not re-read it; git writes in place (inode unch **Remediated 2026-08-08** by running the three documented steps; `uv sync` moved views-frames 1.0.0 → 1.10.2 and views-datafactory 1.10.0 → 1.11.0, and `FeatureFrame` was imported on the host afterwards to confirm the ten-minor-version jump is clean. The entry stays **open** because nothing prevents recurrence. + +**Update 2026-10-13 — the fix reached two of six copies, and the four it missed are the ones an operator reads.** Found by the operator asking, before deploying v1.14.0, whether a one-line tag write was really the procedure. It was not, and the reason the question had to be asked is that #431 corrected `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` and stopped there. Four further copies of the one-step claim were still live: + +| Document | What it said | +|---|---| +| `docs/guides/hetzner_deployment_guide.md` §"How to deploy a new version" | the tag file, then *"That's it."* — **and this is the file `refresh_pipeline.sh` itself points to "for full detail"** | +| `docs/guides/server_operations.md` §"Deploy a new version" + the summary table | local `git tag && git push`, then the tag file, then run the pipeline | +| `docs/ADRs/022_tag_based_deployment_gate.md` §Consequences + §Deployment Procedure | *"One-line rollback"* as a claimed **benefit**; three-step deploy with no checkout and no `uv sync`; a Verify block listing the tag file and `git describe` but **not the installed distribution** — the one of the three that was actually wrong in August | +| `docs/guides/data_serving_guide.md` §9 | *"If the new version breaks, you write the old tag name back and you're running known-good code again."* | + +All four corrected, each pointing at `server_quickref.md` as the single operational home with "if they disagree, the quickref wins", and `server_quickref.md` grew the verification block it was missing (step 4: tag file, checked-out tag, **installed distribution version**). ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite — the Decision it records is sound; what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient. + +**What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. + **The instrument, not yet built.** A pre-flight check in `refresh_pipeline.sh` comparing `$DEPLOY_TAG` against the HEAD the script *started* from, and refusing to run when they disagree — fail-loud per ADR-011, rather than silently self-correcting one run late. Proposed for #424; deliberately not built here, because this PR is a register entry and a comment fix. Cross-ref: C-337 (the fix this failed to deliver), C-342 (the other lockfile-versus-reality entry from the same day), C-341 (gates that run nowhere), C-98/ADR-022 (the deployment gate this undermines). Part of the **mechanisms that fail green** cluster. GitHub: #421.