From 03a18de57c1a049aef379abdcda494857c8b3b86 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Fri, 16 Oct 2026 19:43:50 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20start=20tmux=20before=20a=20pipeline=20?= =?UTF-8?q?run=20=E2=80=94=20the=20rule=20the=20quickref=20didn't=20carry?= =?UTF-8?q?=20(C-343)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-10-13 consolidation made server_quickref.md the single operational home and told every other guide "if they disagree, the quickref wins." hetzner_deployment_guide.md carried a rule the quickref did not — start tmux before a pipeline run, stated at its line 30 with the reason, and backed by a post-mortem section called "tmux saved us from the SSH drop". Promoting the quickref silently demoted that rule. It cost a run the same day. `client_loop: send disconnect: Broken pipe` killed a foreground pipeline after the 37 GB assembly had completed but before the zarr export, leaving grid.npy current and grid.zarr three weeks stale. Every designed defence held: assemble_grid.py renames from grid.npy.tmp so the grid was whole, .exports_required recorded the owed export, flock released with the process, and check_health.py reports the sentinel as PENDING. Nothing was corrupted and nothing was silent. The cost was a re-run. Every interactive refresh_pipeline.sh invocation in the guides now opens with `tmux new -s pipeline`: - server_quickref.md §"Run the pipeline" — the rule, the dated evidence, detach/reattach, and the detail that catches people out: tmux does not change the shell prompt, so only the bottom status bar tells you whether you are inside it - hetzner_deployment_guide.md §2.1, §4.5, and the rollback section added in #497 - server_operations.md summary table and the full-rebuild procedure Cron invocations are deliberately unchanged — cron has no controlling terminal and no SIGHUP to receive. The ~/.profile wrong-vs-right example in server_operations.md is also unchanged; adding tmux would obscure the point it exists to make. C-343 gets the pattern rather than a new entry: a consolidation is only safe if the new authority absorbs everything the demoted copies knew, and checking that was not part of the 2026-10-13 work. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 --- docs/guides/hetzner_deployment_guide.md | 11 ++++++++++- docs/guides/server_operations.md | 5 ++++- docs/guides/server_quickref.md | 25 ++++++++++++++++++++++++- reports/technical_risk_register.md | 8 ++++++++ 4 files changed, 46 insertions(+), 3 deletions(-) diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 431eef1..6f69508 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -181,7 +181,10 @@ source ~/.profile ### 2.1 Run the full pipeline +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline cd ~/views-datafactory bash scripts/refresh_pipeline.sh ``` @@ -434,7 +437,10 @@ crontab -l ### 4.5 Test manually +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline cd ~/views-datafactory bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log ``` @@ -901,9 +907,12 @@ leaves the dependencies of the version you are abandoning. Tags are never deleted by git, so every previous version stays available. Then run the pipeline manually to apply it immediately rather than waiting for -the next cron run: +the next cron run. + +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. ```bash +tmux new -s pipeline sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' ``` diff --git a/docs/guides/server_operations.md b/docs/guides/server_operations.md index 3308330..898efba 100644 --- a/docs/guides/server_operations.md +++ b/docs/guides/server_operations.md @@ -12,7 +12,7 @@ For initial setup, see `hetzner_deployment_guide.md`. | SSH in | `ssh @204.168.219.108` | | Check current version | three commands — see `server_quickref.md` §"Deploy a new version" step 4; the tag file alone is intent, not evidence (C-343) | | Deploy a version | four blocks — see `server_quickref.md` §"Deploy a new version"; writing the tag file is **not** deploying (C-343) | -| Run pipeline | `sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh'` | +| Run pipeline | **start `tmux` first** — see `server_quickref.md` §"Run the pipeline" | | Check pipeline logs | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/refresh.log` | | Check last failure | `sudo -u views-deploy cat /home/views-deploy/views-datafactory/logs/pipeline_failure.json` | | List raw data | `sudo -u views-deploy ls -la /home/views-deploy/views-datafactory/data/raw/ucdp_annual/` | @@ -261,7 +261,10 @@ For JSON output (monitoring integration): add `--json` flag. If you suspect stale data, delete the assembled output and re-run: +> **Run this inside `tmux`.** SSH drops kill a foreground run (SIGHUP). See `server_quickref.md` §"Run the pipeline" — `tmux new -s pipeline` first, every time. + ```bash +tmux new -s pipeline sudo -u views-deploy bash -c 'cd ~/views-datafactory && rm -rf data/assembled data/compiled data/viewpoint data/consolidated' sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh' ``` diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 7364ec9..4aac18b 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -38,10 +38,33 @@ A **rollback is a deploy**: same four blocks, previous tag. ## Run the pipeline +**Start tmux first. Not optional.** The run takes 3-4 hours and SSH will drop +inside that window; without a multiplexer the drop sends SIGHUP and kills the +run. This is not hypothetical — see the note below. + ```bash -# Normal run (output streams to terminal + logs to file) +# 1. ALWAYS start tmux first +tmux new -s pipeline + +# 2. Then run, INSIDE tmux (output streams to terminal + logs to file) sudo -u views-deploy bash -c 'source ~/.profile && cd ~/views-datafactory && bash scripts/refresh_pipeline.sh 2>&1 | tee -a logs/refresh.log' +# Detach: Ctrl-b then d Reattach: tmux attach -t pipeline +# Confirm you are inside tmux: a status bar sits at the BOTTOM of the +# terminal, e.g. "[pipeline] 0:sudo*". The shell prompt does NOT change, +# so the prompt alone never tells you. +``` + +> **2026-10-16: a run was killed this way.** `client_loop: send disconnect: +> Broken pipe` ended a foreground run after the 37 GB assembly had completed +> but before the zarr export. Nothing was corrupted — assembly renames from +> `grid.npy.tmp`, so `grid.npy` was whole, and `.exports_required` recorded the +> owed export — but `grid.zarr` was left three weeks stale while `grid.npy` was +> current, and the work had to be re-run. The lock needs no cleanup (`flock` is +> released by the kernel when the process dies), but the `/start` heartbeat ping +> is left dangling and healthchecks.io will flag it at the ~24h grace timeout. + +```bash # Force re-fetch: delete data first, then run normally # (there is no --force flag; the pipeline skips existing data by default) ``` diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 489cbe2..eba91c7 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3272,6 +3272,14 @@ sudo -u views-deploy cat ~/.../pipeline_failure.json 2>/dev/null || echo "No rec The `||` sat **outside** `bash -c`, so a permission error on the wrong home directory printed **"No recent failure"** — an unreadable path reported as good news, in the command an operator runs to ask whether the last pipeline run broke. Belongs to the **mechanisms that fail green** cluster alongside the rest of this entry. The fallback is now inside the quoted command. +**Addendum 2026-10-16 — consolidating onto one authority moved content *out* of reach, not just into it.** The 2026-10-13 correction made `server_quickref.md` the single operational home and told every other document "if they disagree, the quickref wins." `hetzner_deployment_guide.md` carried a rule the quickref did not: **start `tmux` before a pipeline run**, stated at its line 30 with the reason ("pipeline runs 3-4 hours; SSH will drop without a multiplexer") and backed by a post-mortem section titled *"tmux saved us from the SSH drop"* (`2026-05-25_deployment_v1221.md`). Promoting the quickref silently demoted that rule. + +It cost a run the same day. `client_loop: send disconnect: Broken pipe` killed a foreground pipeline after the 37 GB assembly completed but before the zarr export, leaving `grid.npy` current and `grid.zarr` three weeks stale. **Every designed defence held** — `assemble_grid.py` renames from `grid.npy.tmp` so the grid was whole, `.exports_required` recorded the owed export, `flock` released with the process, and `check_health.py` reports the sentinel as PENDING. Nothing was corrupted and nothing was silent. The cost was a re-run and a window in which served data lagged the grid. + +Every interactive `refresh_pipeline.sh` invocation in the guides now starts with `tmux new -s pipeline`, and the quickref carries the rule, the dated evidence and the detail that catches people out: **tmux does not change the shell prompt**, so only the status bar tells you whether you are inside it. Cron invocations are deliberately left alone — cron has no controlling terminal and no SIGHUP to receive. + +**No new register entry.** The failure is loud, the defences worked, and the trigger would be perpetual ("whenever someone runs the pipeline"), which the trigger-quality gate rejects. What is registrable is the pattern above, and it belongs to this entry: a consolidation is only safe if the authority absorbs everything the demoted copies knew. Checking that was not part of the 2026-10-13 work and should have been. + **The shape worth noticing.** C-343 is now on its third round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. **What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing.