From ccbad36daf6ea097c9e1e843e48018912ce7eae9 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Sun, 18 Oct 2026 18:29:46 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20seven=20places=20claimed=20a=20pipeline?= =?UTF-8?q?=20duration,=20spanning=2024x=20=E2=80=94=20the=20authority's?= =?UTF-8?q?=20figure=20was=20never=20measured=20(C-343)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-10-16 run took 5h50m. The guides claimed 15-30 min, 25-45 min, ~2.5h, 2-3h, 3-4h and ~6h, and the closest figure was in an ADR about something else. `server_quickref.md` was made the single operational authority on 2026-10-13 and given "3-4 hours" — a guess, with no measurement anywhere in the repo, promoted over every document that disagreed. The complement of the tmux finding: consolidation concentrates error as effectively as it concentrates authority. The direction of the error is the hazard. An under-estimate licenses intervention on a healthy run: "25-45 minutes" tells an operator two hours in that the run is hung, and killing a hung run is the reasonable response — reproducing the 2026-10-16 outage from a document instead of a dropped SSH connection. Every stale figure also blamed harvesting; harvest through compile is ~43 min and the zarr export alone is ~5h07m, 88% of the run. Corrected by document kind, not uniformly: - four live guides carry the measured figure and point at the quickref, which owns the per-step breakdown - ADR-041 and the March 2026 deployment log are NOT rewritten — both are records of what was true when written, and falsifying a record to make it current is a different defect from the one being fixed. Both carry dated amendments instead. Also corrected: ADR-041's healthcheck-grace rationale derived 48h from "2-3 hours"; the conclusion survives at ~6h, the reasoning did not. One finding escapes the docs fix and is filed separately: the zarr export was ~57 min in June 2026 and 18,427s (5h07m) on 2026-10-16, a ~5x increase against a store that grew 1.8 -> 3.1 GB. Cause unestablished and deliberately not guessed at. ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not ~103. Verification: ruff clean, validate_docs.sh PASSED, 48 register-reading and citation tests plus 38 deployment-doc guards green. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 --- docs/ADRs/041_content_addressed_skip.md | 15 ++++++++++++++ docs/guides/data_serving_guide.md | 12 +++++++---- docs/guides/hetzner_deployment_guide.md | 17 +++++++++++----- docs/guides/hetzner_deployment_log.md | 6 ++++++ docs/guides/server_quickref.md | 23 ++++++++++++++++++--- reports/technical_risk_register.md | 27 ++++++++++++++++++++++++- 6 files changed, 87 insertions(+), 13 deletions(-) diff --git a/docs/ADRs/041_content_addressed_skip.md b/docs/ADRs/041_content_addressed_skip.md index 34fafb9..58d0c15 100644 --- a/docs/ADRs/041_content_addressed_skip.md +++ b/docs/ADRs/041_content_addressed_skip.md @@ -107,3 +107,18 @@ This ADR covers assembly and export only. It does not cover: - **Output integrity is verified on every skip:** a corrupt grid.npy triggers a rebuild, not silent serving. - **Source set changes are detected:** adding or removing a source (e.g., adding SHDI) prevents false skips. - **Operator override exists:** `--force` provides an escape hatch when data hasn't changed but code has. + +--- + +### Amendment 2026-10-18 — the export cost in Context is 5× low, and the skip is worth more than claimed + +**Context** states that assembly is ~46 min and export ~57 min, and **Consequences** derives "no-change deploys drop from ~103 min to ~30 sec" from that pair. The 2026-10-16 production run measured the export at **18,427s (5h07m)** for a 3.1 GB store — roughly **5× the figure above**, and 88% of a 21,007s (5h50m) run. + +Nothing in this ADR's decision is affected. The skip mechanism works as specified, and the digest gates behave as described. Two statements derived from the stale timings are wrong: + +1. **The export is not comparable in cost to assembly. It dominates the pipeline.** Any future reasoning that treats the two as peers — a decision about which to optimise, which to parallelise, or which to make skippable — starts from a false premise. +2. **This ADR understates its own benefit by about 3.4×.** A no-change deploy avoids roughly **~350 min**, not ~103. The skip is the single largest runtime saving in the pipeline. + +**The cause of the 5× is not established and is deliberately not guessed at here.** The store grew from 1.8 GB (2026-03) to 3.1 GB, which accounts for some of it but not a factor of five. Candidates worth measuring: disk I/O on the CPX32, the `12 × 360 × 720` chunking against a larger store, and compression settings. Tracked as a separate investigation in **#501**, which asks for the three export phases to be timed separately before anything is changed. + +The per-step operational figures live in `docs/guides/server_quickref.md`, which owns them. The numbers in **Context** above are left as written — they were measured in June 2026 and this amendment dates the correction rather than rewriting the record. diff --git a/docs/guides/data_serving_guide.md b/docs/guides/data_serving_guide.md index 8c38a76..b3cd2c1 100644 --- a/docs/guides/data_serving_guide.md +++ b/docs/guides/data_serving_guide.md @@ -174,10 +174,14 @@ Example: "Run the data pipeline on the 21st of every month at midnight" That line means: minute 0, hour 0, day 21, every month, every day-of-week. Run the script. -Your pipeline takes about 15-30 minutes (harvesting is the slow -part — it calls the UCDP and ACLED APIs). After it finishes, the new zarr -and parquet files are in the same directory the web server serves. -Consumers automatically get the new data on their next request. +Your pipeline takes about **six hours**. Harvesting is *not* the slow part, +despite being the part that calls external APIs: harvest through compile is +~43 minutes, and the zarr export alone is ~5 hours. Per-step timings live in +[`server_quickref.md`](server_quickref.md#run-the-pipeline). + +After it finishes, the new zarr and parquet files are in the same directory the +web server serves. Consumers automatically get the new data on their next +request. **No restart needed.** The web server serves whatever files are in the directory. When the pipeline replaces them, the next diff --git a/docs/guides/hetzner_deployment_guide.md b/docs/guides/hetzner_deployment_guide.md index 6f69508..f852129 100644 --- a/docs/guides/hetzner_deployment_guide.md +++ b/docs/guides/hetzner_deployment_guide.md @@ -27,7 +27,7 @@ If the server is already set up and you're just deploying a tag that **already e # 1. SSH into the server (replace with your username) ssh @204.168.219.108 -# 2. Start tmux (pipeline runs 3-4 hours; SSH will drop without a multiplexer) +# 2. Start tmux (the run takes ~6 hours; SSH will drop without a multiplexer) tmux new -s deploy # 3. Deploy and run @@ -189,8 +189,15 @@ cd ~/views-datafactory bash scripts/refresh_pipeline.sh ``` -This takes 25-45 minutes (harvesting calls external APIs and downloads -large raster files). Watch the output — each step prints PASS or FAIL. +This takes about **six hours**. Watch the output — each step prints PASS or +FAIL. Per-step timings are in +[`server_quickref.md`](server_quickref.md#run-the-pipeline), which owns them. + +> **This figure said "25-45 minutes" until 2026-10-18, and that was the +> dangerous direction to be wrong in.** A run at the two-hour mark looks hung +> against a 45-minute estimate, and the reasonable response to a hung run is to +> kill it. The cost is not harvesting — harvest through compile is ~43 minutes. +> **The zarr export alone is ~5 hours and prints nothing while it works.** **Note:** The pipeline includes ACLED (8 features), GHS-POP population data (1 feature), GHS-BUILT-S built-up surface data @@ -463,8 +470,8 @@ See ADR-018 for the rationale. 2. Create a check (or verify the existing one): - **Name:** `views-datafactory pipeline refresh` - **Period:** 30 days (matches the `0 0 21 * *` cron) - - **Grace period:** 48 hours (pipeline runs 2-3 hours; 48h covers - delays, restarts, and weekend non-response) + - **Grace period:** 48 hours (the run takes ~6 hours; 48h still leaves + room for delays, restarts, and weekend non-response) - **Tags:** `views`, `datafactory`, `hetzner` 3. Copy the ping URL: `https://hc-ping.com/` diff --git a/docs/guides/hetzner_deployment_log.md b/docs/guides/hetzner_deployment_log.md index 128be9f..b42ff3e 100644 --- a/docs/guides/hetzner_deployment_log.md +++ b/docs/guides/hetzner_deployment_log.md @@ -295,6 +295,12 @@ or mismatched path. **Total: approximately 2.5 hours for a clean run.** +> **These are March 2026 measurements and are no longer current.** The +> 2026-10-16 run took **5h50m**, with the zarr export alone at ~5h07m against a +> 3.1 GB store (this table has it at ~14 min against 1.8 GB). This is a dated +> log and the figures above are left as measured; current per-step timings live +> in [`server_quickref.md`](server_quickref.md#run-the-pipeline). + Subsequent runs are much faster because most data is cached (only new UCDP versions are fetched). diff --git a/docs/guides/server_quickref.md b/docs/guides/server_quickref.md index 4aac18b..f14ce24 100644 --- a/docs/guides/server_quickref.md +++ b/docs/guides/server_quickref.md @@ -38,9 +38,26 @@ A **rollback is a deploy**: same four blocks, previous tag. ## Run the pipeline -**Start tmux first. Not optional.** The run takes 3-4 hours and SSH will drop -inside that window; without a multiplexer the drop sends SIGHUP and kills the -run. This is not hypothetical — see the note below. +**Start tmux first. Not optional.** The run takes about six hours and SSH will +drop inside that window; without a multiplexer the drop sends SIGHUP and kills +the run. This is not hypothetical — see the note below. + +**Expect ~6 hours, and do not kill a run for "taking too long".** Measured +2026-10-16: **5h50m** (21,007s) with the assembly cached. The zarr export alone +runs for roughly five hours and prints nothing at all while it works — a silent +terminal is the normal state, not a hang. + +| Step | Time | Note | +|------|------|------| +| Harvest → consolidate → viewpoint → compile | ~43 min | all sources, most cached | +| Assemble grid | ~45 min | skipped when inputs are unchanged (ADR-041) | +| **Export zarr** | **~5h07m** | 18,427s for a 3.1 GB store — **88% of the run** | +| Export parquet | ~12 s | 29.5M rows | +| Health check + consumer contract | <1 min | | + +The assembly figure is ADR-041's; every other figure is from the 2026-10-16 run, +which had the assembly cached. A cold run that must rebuild the 37 GB grid is +therefore longer than 5h50m, not shorter. ```bash # 1. ALWAYS start tmux first diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index eba91c7..c68683e 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -3280,7 +3280,32 @@ Every interactive `refresh_pipeline.sh` invocation in the guides now starts with **No new register entry.** The failure is loud, the defences worked, and the trigger would be perpetual ("whenever someone runs the pipeline"), which the trigger-quality gate rejects. What is registrable is the pattern above, and it belongs to this entry: a consolidation is only safe if the authority absorbs everything the demoted copies knew. Checking that was not part of the 2026-10-13 work and should have been. -**The shape worth noticing.** C-343 is now on its third round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. +**Addendum 2026-10-18 — the new authority carried a number nobody had measured.** The 2026-10-16 addendum above records that consolidation can move content *out* of reach. This one records the complement, which is worse: **consolidation concentrates error as effectively as it concentrates authority.** When `server_quickref.md` was promoted on 2026-10-13 it was given "the run takes 3-4 hours." That figure was a guess. No measurement existed for it in the repo, and the document that stated it had just been declared the winner over every document that disagreed. + +Seven places claimed a pipeline duration, and they spanned a factor of 24: + +| Location | Claimed | Kind of document | +|---|---|---| +| `data_serving_guide.md:177` | 15-30 min | live guide | +| `hetzner_deployment_guide.md:192` | 25-45 min | live guide | +| `hetzner_deployment_log.md:296` | ~2.5 h | dated log (March 2026) | +| `hetzner_deployment_guide.md:466` | 2-3 h | live guide (justifies the 48h healthcheck grace) | +| `server_quickref.md:41` | 3-4 h | **the authority, set 2026-10-13** | +| `hetzner_deployment_guide.md:30` | 3-4 h | live guide | +| `ADR-041:12` | ~6 h | ADR | +| **2026-10-16 run, measured** | **5h50m** | — | + +The closest figure was in an **ADR about something else**. ADR-041 got it right incidentally, while arguing for content-addressed skipping. The document whose job is to be correct about it was wrong by 40%, and it was wrong because the correction that created its authority did not come with a measurement. + +**The direction of the error is the operational hazard, not the magnitude.** An over-estimate is harmless; an under-estimate licenses intervention on a healthy run. "25-45 minutes" tells an operator two hours in that the run is hung, and the reasonable response to a hung run is to kill it — reproducing the 2026-10-16 outage from a document instead of a dropped connection. Every stale figure also mis-attributed the cost to harvesting; harvest through compile is ~43 min, and the zarr export alone is ~5h07m, 88% of the run. + +Corrected by kind rather than uniformly, which is the part worth keeping: the four live guides now carry the measured figure and point at the quickref; **ADR-041 and the March deployment log were not rewritten** but carry dated amendments, because both are records of what was true when written and falsifying a record to make it current is a different defect from the one being fixed. + +**No new register entry, again.** The trigger would be perpetual. What is registrable belongs here: a consolidation must verify what it promotes, not only absorb what it demotes. Two rounds of this entry have now found that the 2026-10-13 work was sound in mechanism and unverified in content. + +**One finding escapes this entry and is not a documentation problem.** ADR-041 measured the zarr export at ~57 min in June 2026; it measured **18,427s (5h07m)** on 2026-10-16, a ~5x increase against a store that grew 1.8 GB → 3.1 GB. The cause is unestablished and deliberately not guessed at. Filed as **#501**, because it is a performance question requiring instrumentation rather than a figure requiring correction, and because ADR-041 consequently understates its own benefit by ~3.4x — a no-change deploy avoids ~350 min, not the ~103 min it claims. + +**The shape worth noticing.** C-343 is now on its fourth round of correction, and each round fixed the written procedure while the *instrument that would have caught it* stayed unbuilt. A documented verification step that nobody has executed is not evidence, and this one had been in the quickref §"Check status" since before the entry existed. **What this update changes about the entry's severity, honestly.** It does not raise the tier, but it does weaken the "corrected in the same PR as this entry" claim above: the correction was partial, and the stale copies outlived it by two months and one release. The pattern is #402's — *one procedure, one home* — and C-343 is an instance of it as much as it is a deployment risk. The entry stays **open**, and the still-unbuilt instrument below is now the only thing that would catch a sixth copy appearing.