diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index f852129c..caa060b1 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -865,11 +865,15 @@ fixed order: diverge a little further after **every** release, and `git log main..development` starts counting work that already shipped. -5. **Deploy** = three steps on the server, not one: the tag file, then - `git fetch --tags && git checkout `, then `uv sync`. Writing the - tag file alone is **not** deploying — that mistake left the host on - v1.10.0 for five days while every signal read v1.11.0 (C-343). - `docs/guides/server_quickref.md` holds the commands. +5. **Deploy** = four blocks on the server, not one: the tag file, then + `git fetch --tags && git checkout `, then `uv sync`, then the + verification that the tag file, the checked-out tag and the installed + distribution all name the same version. Writing the tag file alone is + **not** deploying — that mistake left the host on v1.10.0 for five days + while every signal read v1.11.0 (C-343), and the fourth block is what + would have caught it. `docs/guides/server_quickref.md` holds the + commands; if this count and the quickref ever disagree, the quickref + wins. **Why this order:** If you tag before bumping, the tag points at code with the old version string — wrong. If you tag on `development` diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index c68683e8..f4c9fd83 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -175,7 +175,7 @@ | C-340 | 3 | Auto-merge fails silently two ways: `gh pr merge --auto --` refuses to change the method on an already-armed PR (nearly put a squash on `main`), and pushing to a branch whose PR already merged orphans the commit with no error | **Before arming a non-default merge method**, read `auto_merge.merge_method` back; **before any follow-up push to a PR branch**, check `merged` first or use a new branch | Operational safety | | ~~C-341~~ | ~~4~~ | ~~Deploy gates only run where someone types pytest — C-320's fix made them skip-with-reason in CI, so they assure only whoever runs the suite at the right moment | **When adding a deploy gate, or relying on one for release assurance:** ~~ | Resolved 2026-08-11 (#424 gave the answerable gates a scheduled runner; #425 deleted the one that could never answer and replaced it with a publish-time guard) | Test infra | | ~~C-342~~ | ~~3~~ | ~~A stale committed `uv.lock` is invisible — `uv sync` (ci.yml:24,42,60,99) rewrites it in place, so CI goes green on a lock that does not match the committed `pyproject.toml` and the stale one stays in git~~ | Resolved 2026-08-11 (#424): `uv lock --check` in the `test` job, before `uv sync`; drilled both directions and the ordering is itself guarded by `tests/test_ci_gates.py` | Dependency policy | -| C-343 | 2 | Writing `~/.views-deploy-tag` is not deploying — the server ran v1.10.0 for five days while the tag file said v1.11.0 and views-frames stayed at the frozen 1.0.0. The in-script `git checkout` cannot fix the running script (bash has buffered it) and never runs `uv sync` | **At the next release** — deploy with all three `server_quickref.md` steps, then verify tag file, `git describe --tags` and installed version all agree | Deployment | +| C-343 | 2 | Writing `~/.views-deploy-tag` is not deploying — the server ran v1.10.0 for five days while the tag file said v1.11.0 and views-frames stayed at the frozen 1.0.0. The in-script `git checkout` cannot fix the running script (bash has buffered it) and never runs `uv sync` | **At the next release** — deploy with all four `server_quickref.md` blocks, the fourth being the verification that tag file, `git describe --tags` and installed distribution all name the same version | Deployment | | C-347 | 3 | The required CI check exercises a different raster decoder than production — since the 3.11 floor the lock forks, so `test` decodes with `imagecodecs 2026.3.6` while the server uses `2026.5.10`. `test-py313` covers the production line but is **not a required check**, and a red nobody must satisfy is ignorable | **Fired and actioned 2026-08-13.** `test-py313` reported green on `development` and on `main`, and was then added to the required-status-check lists on both (readback verified). Re-opens if the job is renamed or the requirement dropped | Test infra | | C-348 | 3 | Nothing asserts which Python the production server runs, and the floor now admits one that installs a **different raster line**. `preflight.py`, `check_health.py` and `refresh_pipeline.sh` contain no `sys.version_info` check of any kind. Created by #443 — this risk did not exist under `>=3.12`. **Measured 2026-08-13: the host runs Python 3.12.3 (Ubuntu 24.04's system interpreter), so it resolves `tifffile 2026.5.15` / `imagecodecs 2026.5.10` — the current fork.** Reassuring, and it is an accident: the guide said "Install Python 3.10+" and this distro happened to hand it 3.12. Still OPEN because one observation is not an assertion | **At the next server provisioning, Python upgrade on the Hetzner host, or any runbook edit that says `apt install python3`** — pin the interpreter explicitly and record which raster fork it resolves | Server hardening | | C-349 | 4 | A config value restated in prose has nothing binding it back — `hetzner_deployment_guide.md` said "Install Python 3.10+" for the three months `pyproject.toml` declared `>=3.12`, an instruction producing an environment where the package could not install. The #444 pin guard binds *workflow* pins to `requires-python`; nothing binds *prose* | **When writing a Python version, or any pyproject value, into a guide or ADR** — link to the declaration instead of restating it, or accept that the copy will not be checked | Documentation drift | @@ -3199,7 +3199,7 @@ Cross-ref: C-337 (the floor-versus-lock concern this sits beside — same file, **Source:** direct observation on the production host, 2026-08-08, during epic #421. Not a code reading — the operator ran the commands and pasted the output. -**Trigger:** **At the next release.** Deploy with all three steps from `server_quickref.md`, then verify the three sources agree — tag file, `git describe --tags`, installed version. A release that changes `scripts/refresh_pipeline.sh` or any dependency is the acute case. +**Trigger:** **At the next release.** Deploy with all four blocks from `server_quickref.md`; the fourth is the verification that the three sources agree — tag file, `git describe --tags`, installed distribution. A release that changes `scripts/refresh_pipeline.sh` or any dependency is the acute case. **Location:** `scripts/refresh_pipeline.sh` (deployment-gate header comment, and the `git checkout "$DEPLOY_TAG"` in the deployment gate), `docs/guides/server_quickref.md` §"Deploy a new version". @@ -3309,6 +3309,39 @@ Corrected by kind rather than uniformly, which is the part worth keeping: the fo **What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing. +**Update 2026-10-20 — the corrected procedure completed, step 4 included, for the first time.** v1.15.0 deployed with all four blocks and all three verification commands agreeing: + +``` +cat ~/.views-deploy-tag → v1.15.0 +git describe --tags --exact-match → v1.15.0 +ls .venv/lib/python*/site-packages | grep views_data… → views_datafactory-1.15.0.dist-info +``` + +`uv sync` again printed the transition — `- views-datafactory==1.14.0` / `+ views-datafactory==1.15.0` — so block 3 is load-bearing twice measured, not once. **What is new is that step 4 ran at all:** on 2026-10-16 its first command died on the `~`-expansion defect, so the 2026-10-13 verification block had never once been executed successfully. The #498 fix is now confirmed by execution rather than by reading, which is the distinction this entry keeps failing on. + +**What this deploy did *not* test, stated so the confirmation is not read as broader than it is.** `scripts/refresh_pipeline.sh` is byte-identical between v1.14.0 and v1.15.0, so the buffered-running-script half of this entry — the reason a `refresh_pipeline.sh` change lands one cron later — was not exercised. Two of the three mechanisms are now confirmed working; the third remains confirmed only by the August incident that created the entry. And nothing yet *refuses* a mismatched deploy: step 4 detects, it does not gate. + +**A fifth instance, one layer up, by the author of the fourth correction.** During this release the instrumentation for #501 was merged to `development` and then described — in session, to the operator — as something the next cron run would pick up "automatically". It would not have: the server runs a tag, and `v1.14.0` did not contain it. *Merging the instrumentation is not deploying the instrumentation* is the same confusion as *writing the tag is not deploying*, displaced from the tag file to the branch, and it was committed three days after this entry's fourth round by whoever wrote that round. Corrected on #501 before any wait was incurred. The same slip recurred twice more in the same session as chat-only paraphrase ("three steps" for the four blocks), each time from recalling the procedure instead of reading it — not registrable, since chat is not a repo artifact, but the frequency is the evidence: five instances across three months, every one from *reasoning about* the procedure rather than *opening* it. + +**Four more copies still said "three steps", and two of them were in this entry.** Writing the update above prompted a grep for the step *count* rather than the step *content* — a dimension none of the four previous rounds had audited. All four were live, and the correction is round five: + +| Location | Said | +|---|---| +| `reports/technical_risk_register.md` — C-343's row in `## Open Items Summary` | *"deploy with all three `server_quickref.md` steps"*, then listed the three checks that constitute step **4** — four things while naming three | +| `reports/technical_risk_register.md` — C-343's own **Trigger:** field | *"Deploy with all three steps from `server_quickref.md`"* | +| `docs/guides/hetzner_deployment_guide.md` §"the release ritual" item 5 | *"**Deploy** = three steps on the server, not one"*, enumerating exactly three | +| `scripts/refresh_pipeline.sh` header comment | *"with ALL THREE steps in docs/guides/server_quickref.md"* | + +The last is the sharpest: that comment's very next sentence says *"two copies of a procedure drifting apart is what C-343 IS, and a fix that adds a third copy would be the same bug"* — and it had drifted, because **a count is a copy too.** The comment carefully avoided duplicating the commands and then duplicated the number, which was the one part that changed. + +Not registrable as a new entry: same entry, same mechanism, fifth round. But it relocates where the mechanism hides. Rounds one through four hunted copies of the *procedure* and found them in guides and ADRs; this round found that the procedure's **cardinality** is independently copyable, and that two of the four stale copies were inside the register entry doing the hunting — including its trigger, the field an operator is meant to act on. #500 audited seven duration figures across the guides and did not check a step count, because nothing suggested a count could rot. + +**Deliberately left as written**, because each is a dated record rather than an instruction, and per #500's precedent a record is amended and never rewritten to look current: ~~C-331~~'s narrative at *"deploying is three steps, not one"* (true when written, and the sentence is about the one-run lag, not the procedure); ADR-022's *"This section previously gave deploying as three steps"* (correct as a description of what it previously said); and `reports/register_changelog.md`'s account of a past redeploy *"run as all three `server_quickref.md` steps"* (it was three at the time). + +`docs/guides/server_quickref.md:136` says *"use all three checks in §'Deploy a new version' step 4"* and is **correct** — step 4 does contain three checks. Four blocks, the fourth holding three commands, is what makes "three" and "four" both appear legitimately in the same procedure, and is a fair part of why the count drifted. + +**What that implies for the instrument below, which is now the entry's only remaining work.** Four rounds of documentation correction have produced a procedure that executes correctly when followed. They have not produced anything that notices when it is not followed, and the failure mode has simply moved — from stale copies, to a stale mental model of where code must be before it runs. A pre-flight gate cannot catch the #501 variant either. What would is the duration/version comparison noted as absent in #502's review: nothing compares what the server *is running* against what the repository *expects*, at any layer. + **The instrument, not yet built.** A pre-flight check in `refresh_pipeline.sh` comparing `$DEPLOY_TAG` against the HEAD the script *started* from, and refusing to run when they disagree — fail-loud per ADR-011, rather than silently self-correcting one run late. Proposed for #424; deliberately not built here, because this PR is a register entry and a comment fix. Cross-ref: C-337 (the fix this failed to deliver), C-342 (the other lockfile-versus-reality entry from the same day), C-341 (gates that run nowhere), C-98/ADR-022 (the deployment gate this undermines). Part of the **mechanisms that fail green** cluster. GitHub: #421. diff --git a/scripts/refresh_pipeline.sh b/scripts/refresh_pipeline.sh index 4aa7bdb5..28151bdb 100755 --- a/scripts/refresh_pipeline.sh +++ b/scripts/refresh_pipeline.sh @@ -48,11 +48,15 @@ # 2. `uv sync` never runs, so a dependency change never lands at # all, no matter how many times the pipeline runs. # -# Deploy — and roll back — with ALL THREE steps in +# Deploy — and roll back — with ALL FOUR BLOCKS in # docs/guides/server_quickref.md §"Deploy a new version": the tag -# file, then the git fetch/checkout, then `uv sync`. Deliberately not +# file, then the git fetch/checkout, then `uv sync`, then the +# verification that all three name the same version. Deliberately not # repeated here: two copies of a procedure drifting apart is what # C-343 IS, and a fix that adds a third copy would be the same bug. +# This comment said THREE until 2026-10-20, five weeks after the +# fourth block was added — a count is a copy too, which is exactly +# the drift the sentence above warns about. # **If this comment and the quickref ever disagree, the quickref # wins.** See docs/guides/hetzner_deployment_guide.md for full detail. #