Skip to content

docs: writing the deploy tag is not deploying — four more copies said it did (C-343) #497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Polichinel merged 1 commit into from
Oct 12, 2026
Merged

docs: writing the deploy tag is not deploying — four more copies said it did (C-343) #497

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 15 additions & 17 deletions docs/ADRs/022_tag_based_deployment_gate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Normally, `HEAD` points to a branch (e.g., `main`), and the branch points to a c
This is **normal and expected on the server**. It means:
- The server is looking at a frozen snapshot, not a moving branch.
- No `git pull` or `git push` can accidentally change what's checked out.
- The only way to change the running version is to update `~/.views-deploy-tag`.
- Nothing changes the running version except an operator deploying — and deploying is the whole procedure below, not just the tag file.

On developer machines, you work on branches as usual. Detached HEAD is only relevant on the deployment server.

Expand Down Expand Up @@ -112,7 +112,7 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co
### Positive

- **Explicit version control.** The operator decides exactly what runs. No surprises from pushed commits.
- **One-line rollback.** `echo 'v1.0.0' > ~/.views-deploy-tag` reverts to the previous release.
- **Cheap rollback.** The previous release is always available — git never deletes tags — so reverting is the same short procedure as deploying, run with the earlier tag. (It is **not** one line; see the amendment below.)
- **Audit trail.** The deploy tag file, git log, and provenance ledger together show what version produced what data.
- **Safe for multiple maintainers.** Two people can push to `development` without affecting the server.
- **No infrastructure dependencies.** No Docker registry, no CI/CD pipeline, no deployment service. Just git tags and a text file.
Expand All @@ -127,34 +127,32 @@ Per ADR-011: the system should crash visibly rather than silently run unknown co

## Deployment Procedure

**Deploy a new version:**
**On your laptop:** the release ritual in `docs/guides/publishing_to_pypi.md` — bump, promote to `main`, cut a **GitHub Release** (a local `git tag && git push` does not trigger the publish workflow, and both branches have required pull requests since 2026-07-31).

1. On your laptop: `git tag v1.2.0 && git push --tags`
2. On the server: `echo 'v1.2.0' > ~/.views-deploy-tag`
3. Wait for cron (21st of month) or run manually
**On the server:** every block of `docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, `git fetch --tags && git checkout <tag>`, `uv sync`, then verification against three sources: the tag file, the checked-out tag, and the **installed distribution version**. A rollback is the same procedure with the earlier tag.

**Roll back:**
The commands are deliberately not reproduced here. This ADR records *why* the gate exists; the quickref is its single operational home, and **if they disagree, the quickref wins.**

1. On the server: `echo 'v1.0.0' > ~/.views-deploy-tag`
2. Run the pipeline manually or wait for cron
### Amendment 2026-10-13 — writing the tag file is not deploying (C-343)

**Verify:**
This section previously gave deploying as three steps — local tag, tag file, wait for cron — and rollback as one. Both were wrong, and the error was not theoretical: on 2026-08-08 the production host ran v1.10.0 for five days while the tag file read v1.11.0, with a green status page, a pinging heartbeat, and no error at any point. v1.11.0 existed solely to raise the `views-frames` floor, and the floor on the box was still the frozen 1.0.0.

```bash
cat ~/.views-deploy-tag # configured version
git describe --tags --exact-match # checked-out version
tail -5 logs/refresh.log # last run output
```
The gate described in **Decision** is unaffected and still does what it claims: the pipeline refuses to start without a valid tag, and checks that tag out. What was wrong was the inference that the checkout makes the tag file sufficient. It does not, for two reasons established by experiment rather than by reading:

1. Bash buffers `scripts/refresh_pipeline.sh` before the gate's `git checkout` runs, so a change to that script takes effect on the **next** run — one month, on a monthly cron.
2. The pipeline never invokes `uv sync`, so a dependency change never lands at all, however many times it runs.

The **Verify** block was also incomplete in the way that mattered: it listed the tag file and `git describe` but not the installed distribution, which is the only one of the three that reflects `uv sync`. In August it was the installed version that was wrong.

Full operational procedures are in `docs/guides/hetzner_deployment_guide.md`, section "Deployment and releases."
C-343 is **open**: the three guides and this ADR now agree, but nothing mechanically prevents the procedure from being skipped. The proposed instrument — a pre-flight check comparing the deploy tag against the HEAD the script *started* from, failing loud per ADR-011 rather than self-correcting one run late — is tracked in #424 and not built.

---

## References

- ADR-011 (Fail Loud, No Stale Data Serving) — justifies fail-loud on misconfiguration
- ADR-018 (Operational Resilience Policy) — operator-mediated bounded staleness
- `scripts/refresh_pipeline.sh` lines 76-98 — implementation
- `scripts/refresh_pipeline.sh`, the "Deployment gate" block — implementation
- `reports/archive/technical_risk_register_resolved.md` C-98 — risk that motivated this ADR
- `docs/guides/hetzner_deployment_guide.md` — operational procedures
- `docs/guides/data_serving_guide.md` section 9 — conceptual explanation
Expand Down
23 changes: 16 additions & 7 deletions docs/guides/data_serving_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,13 +250,22 @@ notice for days.

With the tag system: the server keeps running the old working version
until you *explicitly* tell it to switch. You test the new version,
tag it, and only then update the server. If the new version breaks,
you write the old tag name back and you're running known-good code
again.

**The full deployment procedure** (create a tag, push it, update the
server file, rollback if needed) is documented in
`hetzner_deployment_guide.md` under "Deployment and releases".
release it, and only then deploy it. If the new version breaks, you
deploy the previous tag and you're running known-good code again.

**One thing this explanation makes tempting, and it is wrong.** Because
the pipeline checks the tag out for you, it looks as though changing the
tag file *is* deploying. It is not. The script is already running when it
does that checkout, so a change to the script itself only takes effect
the run after — a month, on a monthly cron — and the script never
installs dependencies, so a dependency change never lands at all. That
misreading left the server on v1.10.0 for five days with every signal
reading v1.11.0 (C-343, August 2026).

**The deployment procedure** — tag file, checkout, `uv sync`, then verify
three sources agree — lives in `server_quickref.md` under "Deploy a new
version", and only there. A rollback is the same procedure with the
earlier tag.

The architectural rationale (why tags instead of branch tracking,
Docker, or CI/CD) is in **ADR-022**.
Expand Down
72 changes: 39 additions & 33 deletions docs/guides/hetzner_deployment_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -852,8 +852,11 @@ fixed order:
diverge a little further after **every** release, and
`git log main..development` starts counting work that already shipped.

5. **Deploy** = update `~/.views-deploy-tag` on the server to the
new tag. The pipeline checks out whatever tag that file contains.
5. **Deploy** = three steps on the server, not one: the tag file, then
`git fetch --tags && git checkout <tag>`, then `uv sync`. Writing the
tag file alone is **not** deploying — that mistake left the host on
v1.10.0 for five days while every signal read v1.11.0 (C-343).
`docs/guides/server_quickref.md` holds the commands.

**Why this order:** If you tag before bumping, the tag points at code
with the old version string — wrong. If you tag on `development`
Expand All @@ -871,48 +874,51 @@ first, promote second, tag third, back-merge fourth.
> corrected, because a second copy of the procedure is what let the two guides drift apart
> in the first place (#402). One procedure, one home.

**On the server** (SSH in):

```bash
# Update the deploy tag file
echo 'vX.Y.Z' > ~/.views-deploy-tag
```

That's it. The next cron run (21st of the month) will automatically
use the new tag. If you want to apply it right now instead of waiting:

```bash
# Run the pipeline manually (same command cron uses)
cd ~/views-datafactory
bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log
```
**On the server** (SSH in): run **every block** in
`docs/guides/server_quickref.md` §"Deploy a new version" — the tag file, then
`git fetch --tags && git checkout <tag>`, then `uv sync`, then the three
verification commands. The commands are there and not here on purpose.

> **This section used to say "update `~/.views-deploy-tag`. That's it."** It was
> wrong, and it is the sentence C-343 is about: on 2026-08-08 the tag file read
> `v1.11.0`, `git describe` read `v1.10.0`, and the installed `views-frames` was
> still the frozen `1.0.0` — three sources of truth disagreeing, with a green
> status page and a pinging heartbeat and no error anywhere. The in-pipeline
> `git checkout "$DEPLOY_TAG"` does not rescue it: bash has already buffered
> `scripts/refresh_pipeline.sh`, so a change to that script lands one run later —
> a month, on a monthly cron — and `uv sync` is never invoked by the pipeline at
> all, so a dependency change never lands however many times it runs. C-343 was
> corrected in `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md`
> in #431 and **this third copy was missed**, which is the same drift #402 is
> about. The commands live in the quickref and nowhere else; if this file and the
> quickref ever disagree, **the quickref wins.**

### How to roll back

If a new version breaks the pipeline:
If a new version breaks the pipeline, re-run the same three deploy steps with the
previous tag — a rollback is a deploy, and skipping `uv sync` on the way back
leaves the dependencies of the version you are abandoning. Tags are never
deleted by git, so every previous version stays available.

Then run the pipeline manually to apply it immediately rather than waiting for
the next cron run:

```bash
# On the server — point back to the previous version
echo 'vPREVIOUS' > ~/.views-deploy-tag
sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log'
```

The next pipeline run will check out that tag and run its code
instead. The old version is always available — git never deletes tags.

You can also run the pipeline manually to roll back immediately
instead of waiting for the next cron run.

### How to check what version is deployed

```bash
# What version is the server configured to run?
cat ~/.views-deploy-tag
**Three sources, and they must agree** — the tag file, the checked-out tag, and
the installed distribution. Checking only the tag file is what C-343 was: it is a
statement of intent, not evidence. The commands are step 4 of
`docs/guides/server_quickref.md` §"Deploy a new version"; they are not repeated
here, because a second copy of them is how this section came to be wrong.

# What version is actually checked out right now?
git describe --tags --exact-match 2>/dev/null || git log --oneline -1
For when the last pipeline run happened, `tail` the refresh log:

# When did the last pipeline run happen?
tail -5 logs/refresh.log
```bash
sudo -u views-deploy tail -5 ~/views-datafactory/logs/refresh.log
```

### Log rotation
Expand Down
40 changes: 20 additions & 20 deletions docs/guides/server_operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ For initial setup, see `hetzner_deployment_guide.md`.
| What | Command |
|------|---------|
| SSH in | `ssh <your-user>@204.168.219.108` |
| Check current version | `cat /home/views-deploy/.views-deploy-tag` |
| Set version to deploy | `echo "v1.2.3" \| sudo tee /home/views-deploy/.views-deploy-tag` |
| Check current version | three commands — see `server_quickref.md` §"Deploy a new version" step 4; the tag file alone is intent, not evidence (C-343) |
| Deploy a version | four blocks — see `server_quickref.md` §"Deploy a new version"; writing the tag file is **not** deploying (C-343) |
| Run pipeline | `sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'` |
| Check pipeline logs | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/refresh.log` |
| Check last failure | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/pipeline_failure.json` |
Expand Down Expand Up @@ -113,24 +113,24 @@ GitHub tag

### Deploy a new version

On your laptop:
```bash
# 1. Tag the release (from main branch)
git tag v1.2.4
git push origin v1.2.4
```

On the server:
```bash
# 2. SSH in
ssh <your-user>@204.168.219.108

# 3. Set the new version
echo "v1.2.4" | sudo tee /home/views-deploy/.views-deploy-tag

# 4. Run the pipeline
sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'
```
**On your laptop:** the release ritual in `publishing_to_pypi.md` — bump,
promote to `main`, then cut a **GitHub Release**, which is what triggers the
publish workflow. A local `git tag && git push` does not, and both branches have
required pull requests since 2026-07-31, so the sequence that used to sit here
cannot be run any more.

**On the server:** every block in `server_quickref.md` §"Deploy a new version" —
the tag file, `git fetch --tags && git checkout <tag>`, `uv sync`, then the three
verification commands.

> **This section used to stop at the tag file**, and that is C-343: on 2026-08-08
> the host ran v1.10.0 for five days while the tag file, and therefore every
> signal an operator checked, read v1.11.0. The pipeline's own `git checkout`
> does not rescue it — bash has already buffered `scripts/refresh_pipeline.sh`,
> so a change to that script lands one run later (a month, on a monthly cron),
> and `uv sync` is never invoked by the pipeline at all. The commands live in the
> quickref and nowhere else; **if this file and the quickref disagree, the
> quickref wins.**

### Run individual pipeline steps manually

Expand Down
20 changes: 16 additions & 4 deletions docs/guides/server_quickref.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,28 @@ All commands below assume you are on the server.

## Deploy a new version

**All four blocks. Step 1 alone is not a deploy (C-343)** — the pipeline's own
`git checkout` lands a change to `refresh_pipeline.sh` one run later, and never
runs `uv sync` at all.

```bash
# Set version
# 1. Set version
sudo -u views-deploy bash -c 'source ~/.profile && echo v1.2.6 > ~/.views-deploy-tag'

# Pull and checkout (git checkout -- uv.lock discards platform-local lock changes)
# 2. Pull and checkout (git checkout -- uv.lock discards platform-local lock changes)
sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && git fetch --tags && git checkout -- uv.lock && git checkout v1.2.6'

# Sync dependencies
# 3. Sync dependencies
sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv sync'

# 4. Verify — all three must name the same version
sudo -u views-deploy cat ~/.views-deploy-tag
sudo -u views-deploy bash -c 'cd ~/views-datafactory && git describe --tags --exact-match 2>/dev/null || git log --oneline -1'
sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv run python -c "import importlib.metadata as m; print(m.version(\"views-datafactory\"))"'
```

A **rollback is a deploy**: same four blocks, previous tag.

## Run the pipeline

```bash
Expand Down Expand Up @@ -77,7 +88,8 @@ sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && uv
## Check status

```bash
# Current deployed version
# Version the server is CONFIGURED to run (intent, not evidence —
# use all three checks in §"Deploy a new version" step 4 to confirm)
sudo -u views-deploy cat ~/.views-deploy-tag

# Last run duration
Expand Down
14 changes: 14 additions & 0 deletions reports/technical_risk_register.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3241,6 +3241,20 @@ Bash buffers the script and does not re-read it; git writes in place (inode unch

**Remediated 2026-08-08** by running the three documented steps; `uv sync` moved views-frames 1.0.0 → 1.10.2 and views-datafactory 1.10.0 → 1.11.0, and `FeatureFrame` was imported on the host afterwards to confirm the ten-minor-version jump is clean. The entry stays **open** because nothing prevents recurrence.


**Update 2026-10-13 — the fix reached two of six copies, and the four it missed are the ones an operator reads.** Found by the operator asking, before deploying v1.14.0, whether a one-line tag write was really the procedure. It was not, and the reason the question had to be asked is that #431 corrected `scripts/refresh_pipeline.sh` and `docs/guides/server_quickref.md` and stopped there. Four further copies of the one-step claim were still live:

| Document | What it said |
|---|---|
| `docs/guides/hetzner_deployment_guide.md` §"How to deploy a new version" | the tag file, then *"That's it."* — **and this is the file `refresh_pipeline.sh` itself points to "for full detail"** |
| `docs/guides/server_operations.md` §"Deploy a new version" + the summary table | local `git tag && git push`, then the tag file, then run the pipeline |
| `docs/ADRs/022_tag_based_deployment_gate.md` §Consequences + §Deployment Procedure | *"One-line rollback"* as a claimed **benefit**; three-step deploy with no checkout and no `uv sync`; a Verify block listing the tag file and `git describe` but **not the installed distribution** — the one of the three that was actually wrong in August |
| `docs/guides/data_serving_guide.md` §9 | *"If the new version breaks, you write the old tag name back and you're running known-good code again."* |

All four corrected, each pointing at `server_quickref.md` as the single operational home with "if they disagree, the quickref wins", and `server_quickref.md` grew the verification block it was missing (step 4: tag file, checked-out tag, **installed distribution version**). ADR-022 carries a dated amendment in the ADR-027 style rather than a rewrite — the Decision it records is sound; what was wrong was the inference that the gate's own `git checkout` makes the tag file sufficient.

**What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing.

**The instrument, not yet built.** A pre-flight check in `refresh_pipeline.sh` comparing `$DEPLOY_TAG` against the HEAD the script *started* from, and refusing to run when they disagree — fail-loud per ADR-011, rather than silently self-correcting one run late. Proposed for #424; deliberately not built here, because this PR is a register entry and a comment fix.

Cross-ref: C-337 (the fix this failed to deliver), C-342 (the other lockfile-versus-reality entry from the same day), C-341 (gates that run nowhere), C-98/ADR-022 (the deployment gate this undermines). Part of the **mechanisms that fail green** cluster. GitHub: #421.
Expand Down
Loading