From d7f9b2b1b24ba47888f8e9005113f74467acad13 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 21:02:56 +0200 Subject: [PATCH 1/6] docs(FAO): the pre-registered landing suite, its anchors, the roadmap, and two register entries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first end-to-end FAO delivery ran today. Six of ten links worked for the first time in this platform's history; the last one refused. This commits what would otherwise exist only in one conversation. **reports/prereg_rusty_bucket_landing_20260929.md** — ten probes with thresholds committed BEFORE the pooling, the publish or the delivery had run, so a disappointing result could not be re-read as a success. It also records the pre-publish baseline (`/provenance/forecast` returning "No forecast prediction files found"), without which nothing afterwards is provable. Results: P1, P2, P3, P7 and P9 PASS; P5 FAIL (refused at ingest); P4, P6, P8, P10 UNRUN. P9 is recorded as a DEFECT IN THE PROBE, not a pass — it assumed only the forecast publish would run, and the postprocessor legitimately rewrites the historical leg too. **..._values.json** — the 25 anchor cells, full sha256 over each draw vector, captured before publishing. Force-added over .gitignore because a pre-registration whose evidence file is not in git is not a pre-registration. These anchors also proved pooling is deterministic: the publish failed once and re-pooled under a new run id, and all 25 came back byte-identical. **reports/fao_delivery_definition_of_done_and_roadmap.md** — six statements defining done that the operator can check without reading code, and six phases to reach them. Written because the operator is a data scientist who said he has no expertise here and needed to know when this is finished. It names explicitly which decisions are his and which must not be handed to him. **Two register entries:** - **C-154 (T3)** — `/workspace` on rented pods silently ignores `chmod`. The command returns success and changes nothing; a credential written there stays mode 666. What saved us was luck of layout: the one credential the runbook names lives on local disk, and nothing says why that matters. - **C-155 (T2)** — an upload can report success having written nothing. Content-hash dedup returns the pre-existing file's id, so the FAO sidecar was never created while the log said "uploaded". It will recur on every delivery, because the sidecar is run-independent by construction. Tier-1-adjacent: `update_document` wrote this run's metadata onto run-0's file, so the only complete run in the partner bucket may carry corrupted provenance. Deduplication note: views-models' C-94 is the datafactory intensive-feature concern and is NOT the upload-success concern that views-faoapi and views-postprocessing call C-94 in their own registers. Register IDs are per-repo; the cross-reference was nearly made to the wrong entry. Filed alongside: #516 (the un_fao environment is unbuildable fresh), #517 (podrun cannot publish, and installing the fix silently reverts the C-151 toolz override), #518 (the RunPod guide's ground rule 5 is wrong — my own text, and it pushed the operator toward an architecture he correctly rejected), #519 (the registry→env bridge does not exist). Cross-repo: views-pipeline-core#551, views-postprocessing#312 and #313. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- ...delivery_definition_of_done_and_roadmap.md | 211 ++++++++++++ .../prereg_rusty_bucket_landing_20260929.md | 246 ++++++++++++++ ..._rusty_bucket_landing_20260929_values.json | 319 ++++++++++++++++++ reports/technical_risk_register.md | 34 +- 4 files changed, 806 insertions(+), 4 deletions(-) create mode 100644 reports/fao_delivery_definition_of_done_and_roadmap.md create mode 100644 reports/prereg_rusty_bucket_landing_20260929.md create mode 100644 reports/prereg_rusty_bucket_landing_20260929_values.json diff --git a/reports/fao_delivery_definition_of_done_and_roadmap.md b/reports/fao_delivery_definition_of_done_and_roadmap.md new file mode 100644 index 00000000..cca00e73 --- /dev/null +++ b/reports/fao_delivery_definition_of_done_and_roadmap.md @@ -0,0 +1,211 @@ +# The FAO delivery — Definition of Done, and the road to it + +**Written 2026-09-29, at the end of the first end-to-end test.** +Author: this session. Audience: Simon, and whoever picks this up next. + +> **The framing that matters:** none of what follows is data science. It is plumbing — packaging, +> credentials, contracts between repositories, and the machinery that moves a file from one place +> to another without lying about whether it arrived. You do not need to become a software engineer +> to own this. You need a definition of done you can check yourself, and a sequence someone can +> work through. That is what this document is. + +--- + +## 1. Definition of Done + +**Done is not "the code is fixed" and not "the test passed".** Done is: *FAO receives real +forecasts, and will keep receiving them.* + +Concretely, all six of these are true at the same time: + +| # | Statement | How YOU check it, without reading code | +|---|---|---| +| **D1** | FAO's API serves a forecast produced by a run we started that day | `GET /provenance/forecast` names today's run id | +| **D2** | It has the right shape | 3 targets (`lr_ged_sb/ns/os`), 36 months, 64,742 cells, `unmapped_count: 0` | +| **D3** | The numbers served are the numbers we produced | spot-check ~25 cells against what came off the machine — a script reports PASS/FAIL | +| **D4** | The whole chain runs from one command, with nothing hand-edited | one command, walk away, come back to a served forecast | +| **D5** | It is repeatable | run it twice from scratch; both complete; the second is identical to the first | +| **D6** | Someone who is not us can do it | a person follows the runbook on a fresh machine and succeeds without asking us | + +**D3 and D5 are the two that matter most and are the easiest to skip.** D3 is the only thing that +proves the numbers were not altered in transit. D5 is the only thing that distinguishes "it worked" +from "it worked once". + +**What is explicitly NOT in the definition of done:** forecast quality. Whether the models are any +good is a separate question, answered by evaluation, not by this. A delivery can be perfectly done +and carry poor forecasts. + +--- + +## 2. Where we actually are + +The 40-lesson test **succeeded at its job**, which was to find the breaks before the real run paid +for them. Six of the ten links worked for the first time in the platform's history: + +``` + 8 models forecast WORKS (first time ever) + pooled to 128 draws WORKS (first time ever) + published to the store WORKS (first time ever, and the #536 fix behaved exactly right) + postprocessor delivers WORKS (first time ever) + lands in FAO's bucket WORKS (first time ever) + FAO ingests and serves REFUSED +``` + +The refusal is correct behaviour. It declined rather than serving something it could not validate. +Had it accepted, we would have learned about it from FAO. + +**Cost of learning all this: about $5 and one day.** The production run is $24. Finding these now +was the cheap outcome. + +--- + +## 3. What is in the way + +Grouped by what they actually block. **The severity ranking is by "would this break the real run", +not by how hard it is to fix.** + +### Group A — blocks the production run. Must be fixed. + +**A1. The sidecar can never be uploaded, on any run, forever.** +The delivery uploads a "sidecar" — a lookup table mapping grid cells to country boundaries. It is +*identical on every run*, because it describes geography, not forecasts. The storage system +deduplicates by content: seeing identical bytes, it declines to store a second copy and reports +success anyway. The manifest then points at a file that was never created, and FAO refuses the run. + +This is not an accident of today. It will recur on **every** future delivery. + +*The deeper problem:* the contract (ADR-013 §4.2) demands one sidecar **per run** for a file that +cannot vary by run. Forcing the upload would satisfy the contract by storing dozens of identical +copies. The contract is what is wrong. + +**A2. An upload can report success having written nothing.** +The cause of A1 being invisible. The upload returned a real file id — for the wrong file — and the +caller took that as proof. Registered on this platform as C-94, and this is a new instance. + +**A3. The delivery's own completeness check does not check the delivery.** +`_assert_delivery_is_findable` verifies two artefacts: the manifest and the historical file. It +does not verify the sidecar or the 108 shards — the things the manifest actually points at. So the +run reported "Completed" while being unservable. + +### Group B — blocks a *repeatable* run. Must be fixed for D5. + +**B1. The environment is not reproducible.** A dependency (`xarray`) floats unpinned. A fresh +install today gets a version incompatible with the platform's pinned pandas. Your laptop works only +because its environment is months old and has never been rebuilt. **Anyone building fresh, today, +fails.** + +**B2. A pinned override silently reverts.** The `toolz` pin (register C-151) is applied after +install; any later install that re-resolves dependencies undoes it, with no warning. + +**B3. The publishing machine needs a package nobody installs.** `pod_run_model.sh` omits the +`appwrite` extra, because it was written for a run type that never publishes. + +**B4. Nobody has decided which machine publishes.** Your laptop lacks the memory, the server is +locked out, and rented hardware was excluded by a rule I wrote badly. **There is currently no +answer**, and every future delivery needs one. + +### Group C — real, but not blocking. Track and schedule. + +- **C1.** The Appwrite key carries 20 permissions and **expires 2026-11-17**. That date forces the + work regardless; it is the moment to narrow rather than renew. +- **C2.** Three API endpoints gave three different answers about the same run. "Did it land?" is + currently unanswerable from outside. *(faoapi's, being handled there.)* +- **C3.** No check anywhere asserts a forecast has all three targets. A one-target delivery would + publish clean and report healthy. *(faoapi's, being handled there.)* +- **C4.** `/workspace` on rented machines silently ignores file permissions. Credentials placed + there cannot be protected. +- **C5.** My own runbook's ground rule 5 is wrong and actively misleading. + +--- + +## 4. The roadmap + +Six phases. Each states what finishes it, so nobody has to guess. + +### Phase 0 — Write it all down · **today** · me · *free* + +File every finding as an issue in the repository that owns it; register the risk-class ones. +Nothing is fixed in this phase — the point is that none of it is carried only in this conversation. + +**Finished when:** every item in §3 has an issue number or a register entry. + +### Phase 1 — Finish the test · **optional, your call** · me · *~$1* + +Use the stopgap faoapi identified (point the manifest at the sidecar that already exists) to get +the run ingested, purely to exercise **the last third of the chain, which has never run**: +do the served numbers match ours (D3), is the coverage right, does it look undertrained. + +**Why it is worth doing:** there may be further breaks behind this one. Finding them now is far +cheaper than during the production run. + +**Why you might decline:** it puts a run in a partner-visible store whose manifest references +another run's file. A deliberate provenance compromise, permanent in the listing. + +**Finished when:** probes P4, P6, P8 and P10 have run and reported. + +### Phase 2 — Fix the blockers · **the real work** · cross-repo · *free, but the longest phase* + +In dependency order: + +1. **A1 — the sidecar contract.** Decide whether the sidecar is per-run or shared. This is a + cross-repo contract change (ADR-013, views-postprocessing, views-pipeline-core, views-faoapi) + and is what the **þing protocol** exists for. *Everything else in this phase is small; this one + is a decision that needs several repos to agree.* +2. **A2 — uploads must verify what they wrote**, by name, not by "did I get an id back". +3. **A3 — the completeness check must check every artefact the manifest names.** +4. **B1–B3 — pin the environment** so a fresh build is identical to a working one. +5. **B4 — decide the publishing machine.** Yours. See §5. +6. **C5 — correct the runbook.** + +**Finished when:** each has a merged fix with a test that fails without it, and the packages are +released (pods install released versions, so an unreleased fix does not exist as far as a run is +concerned). + +### Phase 3 — Prove it, cheaply · me · *~$4* + +Re-run the full chain at 40 lessons, **twice, from a fresh machine**, with no hand-edits. + +**Finished when:** D1, D2, D3, D4 and D5 all hold. This is the gate before spending real money. + +### Phase 4 — The production run · me, on your go · *~$24* + +Eight models at 300 lessons, pooled, published, delivered. + +**Finished when:** D1–D4 hold on a 300-lesson run, and FAO is told it is real. + +### Phase 5 — Make it routine · *ongoing* + +The difference between "we did it" and "it keeps happening": the runbook updated so D6 holds, a +scheduled run, and an alert when a delivery goes stale. Plus the C-group items, with **C1's +2026-11-17 key expiry as a hard date**. + +--- + +## 5. What is yours, and what is not + +**Yours — nobody else can decide these:** + +- **Which machine publishes to FAO** (B4). Not a technical detail: it is about what infrastructure + this platform runs on. A personal laptop is not a defensible answer and you were right to say so. +- **The key**: narrow it, and by when — forced by the 2026-11-17 expiry. +- **Whether to take the Phase 1 stopgap.** +- **Priority between all of this and everything else you have.** +- Money, credentials, releases, and anything that reaches FAO. + +**Not yours — do not let anyone hand these to you:** + +- Which repository an issue belongs in, how a fix is designed, what a test asserts, how the + contract should be restructured. That is engineering, and the sessions working these repos own it. + +**The honest summary:** you are not out of your depth on the decisions above — those are judgement +calls about infrastructure and risk, and you have made several good ones today, including rejecting +the laptop and insisting on establishing behaviour before acting. You are out of your depth on the +mechanisms, and you do not need to be in it. + +--- + +## 6. The one-line version + +> Six of ten links work for the first time ever. One contract is wrong in a way that would break +> every future run, and three guards failed to notice. Fix those, pin the environment, decide where +> this runs — then prove it twice at $4 before spending $24. diff --git a/reports/prereg_rusty_bucket_landing_20260929.md b/reports/prereg_rusty_bucket_landing_20260929.md new file mode 100644 index 00000000..fad35ae2 --- /dev/null +++ b/reports/prereg_rusty_bucket_landing_20260929.md @@ -0,0 +1,246 @@ +# Pre-registration — proving (or falsifying) that `rusty_bucket` landed at FAO + +**Written 2026-09-29 15:30 CEST, BEFORE the pooling, the publish, or the delivery had run.** +Nothing in this document was written with a result in hand. That is the point: every threshold +below is committed in advance so that a disappointing outcome cannot be re-read as a success. + +**Status when written:** 3 of 8 models forecast. Pooling has never run. `rusty_bucket` has never +been published. The chain below has never executed end to end in the history of this platform. + +--- + +## The claim under test + +> The 40-lesson test forecast produced by `rusty_bucket` on 2026-09-29 has landed in the UN FAO +> delivery and is being served to FAO. + +**This is a claim about IDENTITY, not QUALITY.** The models are deliberately undertrained. No +probe below asks whether the numbers are good, and none can. A suite that confirmed "forecasts +are present and plausible" would pass just as happily on someone else's run, on a cached +artefact, or on a partial commit. Every probe is therefore designed to distinguish **our** run +from **any** run. + +--- + +## The baseline, recorded before the fact + +Read live at 2026-09-29 15:28 CEST. **This is the anchor. Without it, nothing below is provable.** + +``` +GET /version + {"version":"1.7.1","deployed_tag":"v1.7.1","served_contract_version":"1.5"} + +GET /provenance/forecast + {"detail":"No forecast prediction files found in the bucket: unfao_bucket"} + +GET /provenance/historical + file_id 6a7d5d960028cdbb1c04 + filename historical_dataset_20260813_080043.parquet + created_at 2026-08-13T06:01:22.142+00:00 + targets lr_ged_sb, lr_ged_ns, lr_ged_os + region land_gaul expected_cell_count 64742 actual 64742 unmapped 0 + file_hash 19a990b52cf07849587c928ac82d553671a7c7fc8d0872d02652ed0406c76e14 +``` + +**Two findings from the baseline that change the test design.** + +1. **There is no stale forecast to displace — there is no forecast at all.** The bucket is empty + of them. So "a forecast exists" is already a strong signal, and any forecast appearing during + this window is overwhelmingly likely to be ours. This makes P5 cheap and decisive, and it means + a *negative* result is unambiguous too. + +2. **The served region is `land_gaul` at 64,742 cells. We produce `land` at 64,818.** A curation + step sits between the publish and what FAO serves — 76 cells are dropped. **Publishing is + therefore NOT landing.** Any probe that stops at the prediction store proves nothing about FAO. + This is the single most likely place for the chain to break silently, because both numbers look + like "global land". + +--- + +## The chain, and where each probe sits + +``` + L1 pod rusty_bucket pools 8 models -> 128 draws P1 + L2 store publish to production_forecasts bucket P2 P3 P4 + L3 delivery un_fao postprocessor curates land -> land_gaul P6 + L4 api FAO serves it P5 P7 + L5 values what FAO serves IS what we produced P8 + control nothing else moved P9 P10 +``` + +A probe may only be reported as passed if it was **executed**. A probe reasoned about but not run +is recorded `UNRUN`, never as a pass. This rule exists because it has been violated on this +platform before. + +--- + +## Probes + +Ordered cheapest-and-most-decisive first. **Each states what result would falsify the claim** — +if a probe cannot fail, it is not a probe and must be struck from this list rather than reported. + +### P1 — the pool is 128 draws, not 16 and not 48 + +**Check.** Pooled artefacts on the pod carry `8 x 16 = 128` draws per cell-month per regression +target, matching `expected_models: 8` and `expected_samples_per_model: 16`. + +**Risky prediction:** exactly **128**. + +**Falsifies if:** 16 (only one constituent found), any multiple below 128 (constituents silently +skipped), or the run completes with fewer than 8 models contributing. A pool that quietly drops a +constituent still produces a plausible forecast — this is the probe that catches it. + +### P2 — exactly THREE targets are published, by name + +**Check.** The store holds manifests for exactly `lr_ged_sb`, `lr_ged_ns`, `lr_ged_os`. + +**Risky prediction:** **3**. Not 6, not 0. + +**Falsifies if:** **6** — the views-pipeline-core#536 fix did not take, and classification channels +went to the partner store. **0** — the publishable set computed empty and the run delivered +nothing while reporting success (the failure mode pipeline-core added a guard for). Any other +count — something is wrong that nobody predicted. + +This probe *is* the #536 fix under test. It has never been exercised against a real store. + +### P3 — no torn commit + +**Check.** For each manifest, every shard it lists resolves to a file that exists in the bucket; +and no shard exists whose manifest is absent. + +**Risky prediction:** shard count == `targets x months`, and manifests present for all of them. + +**Falsifies if:** shards exist without a manifest (the publish died mid-run — consumers are +supposed to ignore unmanifested shards, so this would be invisible to FAO but is still a torn +run), or a manifest references a shard that is not there (worse: a commit marker for an +incomplete delivery). + +### P4 — no silent stem collision + +**Check.** No two published shards resolve to the same consumer series. Then spot-check values +from a published `lr_ged_sb` shard against the pod's **regression** cube — and confirm they do +**not** match the pod's **classification** cube for the same cells. + +**Risky prediction:** three distinct series `sb`, `ns`, `os`; and the served `sb` values match the +regression source, not the classification source. + +**Falsifies if:** values under a fatality name match the probability cube instead. views-faoapi +resolves names by tokenising, so `pred_lr_ged_sb` and `pred_cls_ged_sb` both reduce to `sb` **and +nothing raises**. This is the failure that would put probabilities in front of the UN under a +fatality label, with no error anywhere. It cannot be detected by counting files. + +### P5 — the served forecast is OURS + +**Check.** `GET /provenance/forecast`. + +**Risky prediction:** it no longer returns `"No forecast prediction files found"`. It returns a +record whose `created_at` is **2026-09-29** and whose run identity carries the `rusty_bucket` +forecasting timestamp generated on the pod today. + +**Falsifies if:** still empty (nothing landed), or a record whose date or run id is not today's +(something else landed, or a cached artefact is being served and we are about to congratulate +ourselves for someone else's file). + +**Record the pod-side run id BEFORE the publish**, or this probe degrades into "a forecast exists", +which is not the claim. + +### P6 — the curation happened + +**Check.** The served forecast's coverage. + +**Risky prediction:** `region = land_gaul`, `expected_cell_count = 64742`, `unmapped_count = 0` — +matching the historical record exactly. + +**Falsifies if:** 64,818 (our raw `land` coverage was served uncurated — 76 cells FAO's lookup +cannot place), or `unmapped_count > 0` (the lookup failed for some cells and the delivery shipped +anyway), or any other count. + +This is the probe for the step the baseline revealed. It is the most likely silent failure in the +whole chain, because 64,818 and 64,742 are both "global land" to the naked eye. + +### P7 — the horizon is right + +**Check.** Months served. + +**Risky prediction:** **36** months beginning at **561** — matching what `purple_alien` produced +(verified 561–596 earlier today). + +**Falsifies if:** 37 (the off-by-one already filed as views-models#512 has reached the wire), 12, +or a range that starts before 561 (history leaked into the forecast). + +### P8 — the values served ARE the values we produced + +**The strongest probe, and the only one that cannot be satisfied by coincidence.** + +**Check.** Before the publish, select **20 cell-months at random plus the 5 highest-valued cells** +from the pod's pooled output, and record their values to full precision in this file's companion +log. After the delivery, query the API for those same `(month_id, priogrid_id)` pairs and compare. + +**Risky prediction:** exact agreement on the derived statistic. + +**Procedural caution, agreed in advance so a mismatch is not misread:** the API serves MAP and HDI +summaries (`sb_map`, `sb_hdi90_lower`, …), **not raw draws**. The comparison must therefore be +against the same statistic computed from our 128 draws, using faoapi's own definition — which must +be read from `views-faoapi` **before** comparing, not guessed. A mismatch caused by comparing the +wrong statistic is an error in the probe, not a falsification of the claim, and must be recorded +as such rather than as a failure. + +**Falsifies if:** the values differ beyond floating-point tolerance once the statistic is +correctly matched. That would mean something between the pod and FAO altered the numbers — +a transform, a stale cache, or a different run. + +### P9 — negative control: nothing else moved + +**Check.** `GET /provenance/historical`. + +**Risky prediction:** **unchanged** — `file_hash` still +`19a990b52cf07849587c928ac82d553671a7c7fc8d0872d02652ed0406c76e14`, file still +`historical_dataset_20260813_080043.parquet`. + +**Falsifies if:** it changed. The forecast publish has no business touching historical data. Given +the API key carries write-and-delete across all buckets and files, this is not a theoretical +concern, and it is the only probe here that would catch collateral damage. + +### P10 — adequacy: it must look undertrained + +**Check.** The served forecast's magnitudes against what a 300-lesson run produces. + +**Risky prediction:** recognisably undertrained — `purple_alien` at 40 lessons gave ~45,410 +state-based fatalities over 36 months globally. + +**Falsifies if:** the numbers look like a well-trained run. That would mean we published something +other than what we just produced — an older artefact, or a different model set. A result that +looks *too good* is a failure here, not a success. + +--- + +## What this suite CANNOT prove + +Stated so that nobody, including me, over-reads a green run: + +- **Nothing about forecast quality.** The models are undertrained by design. Every probe tests + identity, coverage, shape and provenance. None tests skill. +- **Nothing about the 300-lesson run.** A green suite says the plumbing carries data correctly at + 40 lessons. It does not say the production run will succeed — that run differs in duration and + memory profile, and those are exactly where this platform has failed before. +- **Nothing about whether FAO can USE it.** P5–P8 prove the data is served and correct. Whether it + is usable is a question for FAO, and the mail sent today explicitly tells them not to try. +- **Nothing about durability.** A forecast that lands and is then evicted, expires, or is + superseded would still pass every probe here at the moment of measurement. + +## Reporting rules, fixed in advance + +1. A probe is **PASS**, **FAIL**, or **UNRUN**. There is no "probably fine". +2. Any probe not executed is reported `UNRUN`, even if its outcome seems obvious. +3. A FAIL is reported as a FAIL. It is not downgraded to a caveat, and the run is not described as + a success with an asterisk. +4. If a probe turns out to be badly designed once run (P8's statistic-matching is the likely + candidate), that is recorded as a **defect in this document**, and the probe is re-run + corrected — not silently dropped, and not counted as a pass. +5. The overall verdict is **LANDED** only if P1–P9 all pass. P10 failing means we published the + wrong artefact, which is also not LANDED. + +--- + +*Companion log for the recorded pre-publish values: `prereg_rusty_bucket_landing_20260929_values.json`, +to be written before the publish and never after.* diff --git a/reports/prereg_rusty_bucket_landing_20260929_values.json b/reports/prereg_rusty_bucket_landing_20260929_values.json new file mode 100644 index 00000000..eb1beabc --- /dev/null +++ b/reports/prereg_rusty_bucket_landing_20260929_values.json @@ -0,0 +1,319 @@ +{ + "captured_utc": "2026-09-29T16:30:33.725087Z", + "pooled_dir": "predictions_forecasting_20260929_162533", + "run_id": "rusty_bucket_forecasting_20260929_162533", + "target": "lr_sb_best", + "pool_shape": [ + 2333448, + 128 + ], + "month_min": 561, + "month_max": 596, + "n_months": 36, + "n_cells": 64818, + "total_mean_collapsed": 25751.3984375, + "random_20": [ + { + "row": 203917, + "month_id": 574, + "priogrid_id": 99585, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 357726, + "month_id": 591, + "priogrid_id": 117043, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 367959, + "month_id": 564, + "priogrid_id": 118003, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 405800, + "month_id": 569, + "priogrid_id": 121933, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 432939, + "month_id": 564, + "priogrid_id": 124778, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 586702, + "month_id": 571, + "priogrid_id": 141316, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 619614, + "month_id": 579, + "priogrid_id": 144391, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 830338, + "month_id": 595, + "priogrid_id": 163092, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 863935, + "month_id": 568, + "priogrid_id": 165341, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 890021, + "month_id": 590, + "priogrid_id": 167192, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 928749, + "month_id": 582, + "priogrid_id": 169680, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 944800, + "month_id": 577, + "priogrid_id": 170773, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1050393, + "month_id": 582, + "priogrid_id": 177254, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1073247, + "month_id": 576, + "priogrid_id": 178707, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1185830, + "month_id": 587, + "priogrid_id": 185487, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1238831, + "month_id": 596, + "priogrid_id": 188457, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1281061, + "month_id": 562, + "priogrid_id": 190930, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1585369, + "month_id": 562, + "priogrid_id": 205414, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 1608378, + "month_id": 567, + "priogrid_id": 206409, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + }, + { + "row": 2264507, + "month_id": 596, + "priogrid_id": 235303, + "n_draws": 128, + "mean": 0.0, + "median": 0.0, + "q95": 0.0, + "max": 0.0, + "n_positive": 0, + "sha256_draws": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" + } + ], + "top_5": [ + { + "row": 652528, + "month_id": 589, + "priogrid_id": 147955, + "n_draws": 128, + "mean": 2354.186767488718, + "median": 0.0, + "q95": 0.0, + "max": 301274.90625, + "n_positive": 3, + "sha256_draws": "777c211cdf0ebf32aac600e21cc6ad8be493e9466d50de5178d37ce817b7e181" + }, + { + "row": 1399073, + "month_id": 566, + "priogrid_id": 196989, + "n_draws": 128, + "mean": 264.68759167566895, + "median": 0.0, + "q95": 10.55000045299524, + "max": 33678.01171875, + "n_positive": 9, + "sha256_draws": "8c0ada28bf14131bf532728f44ffaeadcb64aced70d6467b9bea701c66e1135e" + }, + { + "row": 1414270, + "month_id": 571, + "priogrid_id": 197707, + "n_draws": 128, + "mean": 191.04693603515625, + "median": 0.0, + "q95": 0.0, + "max": 24453.0078125, + "n_positive": 2, + "sha256_draws": "ae22921707af87149cdc066e0d6a73be250d94c9c7459b26352590c072890cd3" + }, + { + "row": 1429896, + "month_id": 573, + "priogrid_id": 198430, + "n_draws": 128, + "mean": 186.3437195122242, + "median": 0.0, + "q95": 0.0, + "max": 23779.99609375, + "n_positive": 6, + "sha256_draws": "e23d26bc9122bd2204384b524b0172d12bcf82445400473777292eae461b4683" + }, + { + "row": 1430028, + "month_id": 561, + "priogrid_id": 198434, + "n_draws": 128, + "mean": 283.81246184557676, + "median": 0.0, + "q95": 1.9499999999999744, + "max": 29535.99609375, + "n_positive": 7, + "sha256_draws": "8e341c4420ba0f590c9f993456a6a2e6198072d1e8ec67ad3ffbf6c58cec1a7b" + } + ] +} \ No newline at end of file diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index a6584411..1e5e085f 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -1,10 +1,10 @@ # Technical Risk Register — views-models -**Last updated:** 2026-09-28 +**Last updated:** 2026-09-29 **Governing ADR:** [ADR-010](../docs/ADRs/010_technical_risk_register.md) -**Total entries:** 161 (152 concerns + 9 disagreements) -**Concerns:** Open 72 | Mitigated 23 | Resolved 46 | Accepted 5 | Partially Resolved 1 | Subsumed 1 | Merged 4 -**Concerns by tier:** T1 6 | T2 48 | T3 65 | T4 25 (4 merge stubs carry no tier) +**Total entries:** 163 (154 concerns + 9 disagreements) +**Concerns:** Open 74 | Mitigated 23 | Resolved 46 | Accepted 5 | Partially Resolved 1 | Subsumed 1 | Merged 4 +**Concerns by tier:** T1 6 | T2 49 | T3 66 | T4 25 (4 merge stubs carry no tier) **Disagreements:** Open 7 | Resolved 1 | Subsumed 1 **Last curated:** 2026-07-31 (`review-rr strategic`, first full pass — tier recalibration, 4 merges, 6 causal clusters identified) @@ -1912,6 +1912,32 @@ --- +### C-154 — `/workspace` on rented pods silently ignores `chmod`; a credential written there stays world-readable however the script sets it + +| Field | Value | +|---|---| +| **Tier** | 3 | +| **Trigger** | **Placing any credential, key or secret on a rented machine under `/workspace`** — the Appwrite publish credential, a `.netrc`, an API token — and relying on `chmod 600` to protect it. Also: writing a runbook step that tells an operator to do so. | +| **Source** | First FAO end-to-end delivery, 2026-09-29 — observed while placing the Appwrite publish credential on `secondary_green_viper` | +| **Status** | Open | +| **Location** | `/workspace` on RunPod (network-backed, `mfs#…runpod.net:9421`); `tools/podrun/pod_run_model.sh` (writes `/root/.netrc`, which is unaffected); `docs/runpod_run_guide.md` Step 2.3 "Place the credential" | +| **Notes** | RunPod mounts `/workspace` from a network filesystem that does not honour POSIX mode bits. `chmod 600` **returns success and changes nothing** — the file was observed at mode `666` after a successful `chmod`, and only reached `600` once moved to `/root/.secrets` on the container's local disk. The failure is silent in the worst way: the command reports success, so a script that sets permissions and checks the exit code passes while leaving the file world-readable. **What saved us today was luck of layout, not design** — the datafactory credential the runbook does tell operators to place lives at `/root/.netrc`, which is local disk, so its `600` holds. Nothing in the runbook says *why* that path matters, so the next credential placed by someone following the pattern rather than the path would land on `/workspace` unprotected. **Tier 3 and not 2:** a RunPod pod is single-tenant and ephemeral, so the realistic exposure is other processes in the same container and anything that later reads a snapshot or detached volume — a real but bounded surface, and the same surface C-153 already accepts for a different reason. Raise to 2 if pods are ever shared, or if a volume outlives a campaign. **Mitigation, cheap and known:** write secrets to `/root/…` on local disk, never `/workspace`; and where a script sets a mode, *verify it afterwards with `stat`* rather than trusting the exit code — the same "check the outcome, not the return value" lesson as C-155. **Exit:** the runbook states the rule and gives the reason, and `pod_run_model.sh` asserts the mode it set. Cross-refs: **C-153** (the same credentials on the same rented hardware, but the transit risk rather than the at-rest one), **C-155** (an operation that reports success without having done the thing), views-models **#518** (the runbook ground rule this sits under). | + +--- + +### C-155 — an upload can report success having written nothing: content-hash dedup returns the pre-existing file's id, and every caller believes it + +| Field | Value | +|---|---| +| **Tier** | 2 | +| **Trigger** | **Any delivery that uploads an artefact whose bytes are identical to one already in the store** — most concretely the FAO gid-to-GAUL sidecar, which is run-independent by construction and will therefore collide on *every* future run, including the 300-lesson production run. Also: any new guard written to confirm a delivery by checking that an upload returned a file id. | +| **Source** | First FAO end-to-end delivery, 2026-09-29 — the delivery reached FAO's bucket and was refused at ingest | +| **Status** | Open | +| **Location** | views-pipeline-core `modules/appwrite/file.py` (`check_file_exists_by_hash` queries `Query.equal("file_hash", …)` alone; the requested *name* is neither in the query nor compared afterwards, and the caller receives `success=True`, `code="METADATA_UPDATED"` and the **pre-existing** file's id); views-postprocessing `unfao/managers/unfao.py::_assert_delivery_is_findable`; consumed by views-faoapi `managers/prediction/manager.py:284-292`, which resolves by `filename` | +| **Notes** | The FAO sidecar was never created, while the log recorded `uploaded rusty_bucket_forecasting_20260929_172325__sidecar.parquet`. Its bytes were identical to run-0's sidecar from August, the uploader matched on content hash, ran `update_document` against **run-0's file**, and returned that file's id. The manifest then referenced a name that did not exist; views-faoapi resolved by name, found nothing, and refused the run (`ingest_failed`). Bucket listing: run-0 110 objects, ours 109. **The consumer refused correctly — the silence is entirely on the producing side.** views-pipeline-core confirmed the mechanism in source and noted the asymmetry that makes it a defect rather than a design choice: the sibling branch handling `FOUND_BY_NAME` with a *different* hash deletes and re-uploads correctly. Name-awareness exists in one direction only. **The sentence worth keeping:** *manifest-last protects against a run that stops early; it does not protect against an upload that returns success having written nothing.* **Why Tier 2 and not 3:** it is structural, the trigger is not hypothetical but certain — it fires on every delivery — and it defeats guards written in the obvious way, since a check for "did I get an id back" passes here. **Why it is Tier-1-adjacent:** `update_document` wrote this run's metadata onto run-0's file, so the only complete run in the partner bucket may now carry corrupted provenance; storage is provably untouched (`createdAt == updatedAt` to the millisecond) but the metadata document is unverified. If that corruption is confirmed, this is silent corruption of a partner-visible record and should be re-tiered. **The deeper question, deliberately not answered here:** ADR-013 §4.2 requires one sidecar *per run* for a file that cannot vary by run, so forcing the upload or making dedup name-aware would satisfy the contract by storing N identical ~894 KB copies — obeying the contract rather than solving the problem. views-faoapi proposes one shared sidecar versioned by GAUL edition instead. That spans four repositories and is a maintainer decision. **Exit:** an upload verifies that the name it was asked to write resolves, and the delivery's completeness check verifies every artefact its manifest references — by name, not by returned id. Cross-refs: **views-pipeline-core#551** (the upload half), the views-postprocessing guard half, **C-154** (the same "reported success, did not happen" shape at the filesystem layer), views-models `reports/prereg_rusty_bucket_landing_20260929.md` (probe P3, which this failed). | + +--- + ## Disagreements ### D-01 — Intentional config duplication vs. DRY principle From 519b9461d853fdb255ff37ea87526033709e59e5 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 21:04:53 +0200 Subject: [PATCH 2/6] =?UTF-8?q?docs(register):=20C-155=20escalated=20to=20?= =?UTF-8?q?Tier=201=20=E2=80=94=20the=20documented=20remedy=20is=20the=20t?= =?UTF-8?q?rigger=20for=20total=20failure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit views-postprocessing established the worst case and it is not 109 of 110 artefacts. It is 0. `contract/wire/naming.py:19-31` embeds the run id in every filename — shards, sidecar and manifest alike. So a re-publish produces NEW filenames over IDENTICAL bytes. Every one of the 110 objects then walks the content-hash dedup path, the store dutifully creates 110 documents, the run logs 110 successful uploads and exits clean with a success alert, and nothing in the bucket is servable. This platform's documented remedy for a torn run is to re-publish (views-postprocessing C-105, C-22). So the prescribed response to a failed delivery is the trigger for the total-failure case of this defect. Today survived only because the sidecar was a first-of-kind collision rather than a repeat. A re-publish WAS proposed on the day as the fix for the torn run, and was not taken only because we stopped to establish the selection behaviour first. It would have produced a silent total failure. Neither seat could have found this alone. views-postprocessing verified the run-id-in-filename half. This seat had measured, accidentally, that pooling is deterministic — 25/25 pre-registered anchor cells byte-identical across two independent pools, full sha256 over each draw vector — which is what turns "the bytes might match" into "the bytes will match". Tier 1 rather than 2 on the register's own criterion: from the producing side there is no error signal whatsoever. The counterargument is recorded in the entry rather than omitted — the consumer does refuse, so FAO receives nothing rather than something wrong, which is an availability failure rather than served corruption. That is why it was first filed Tier 2. It is outweighed by the remedy being the trigger, and by the operator being told, 110 times, that everything worked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- reports/technical_risk_register.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 1e5e085f..66fe5701 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -4,7 +4,7 @@ **Governing ADR:** [ADR-010](../docs/ADRs/010_technical_risk_register.md) **Total entries:** 163 (154 concerns + 9 disagreements) **Concerns:** Open 74 | Mitigated 23 | Resolved 46 | Accepted 5 | Partially Resolved 1 | Subsumed 1 | Merged 4 -**Concerns by tier:** T1 6 | T2 49 | T3 66 | T4 25 (4 merge stubs carry no tier) +**Concerns by tier:** T1 7 | T2 48 | T3 66 | T4 25 (4 merge stubs carry no tier) **Disagreements:** Open 7 | Resolved 1 | Subsumed 1 **Last curated:** 2026-07-31 (`review-rr strategic`, first full pass — tier recalibration, 4 merges, 6 causal clusters identified) @@ -1929,12 +1929,12 @@ | Field | Value | |---|---| -| **Tier** | 2 | -| **Trigger** | **Any delivery that uploads an artefact whose bytes are identical to one already in the store** — most concretely the FAO gid-to-GAUL sidecar, which is run-independent by construction and will therefore collide on *every* future run, including the 300-lesson production run. Also: any new guard written to confirm a delivery by checking that an upload returned a file id. | +| **Tier** | 1 | +| **Trigger** | **Re-publishing a run after a failed delivery** — which is this platform's own documented remedy for a torn run (views-postprocessing C-105, C-22). Also, less severely: any delivery uploading an artefact whose bytes match one already in the store, most concretely the FAO sidecar. Also: any new guard written to confirm a delivery by checking that an upload returned a file id. | | **Source** | First FAO end-to-end delivery, 2026-09-29 — the delivery reached FAO's bucket and was refused at ingest | | **Status** | Open | | **Location** | views-pipeline-core `modules/appwrite/file.py` (`check_file_exists_by_hash` queries `Query.equal("file_hash", …)` alone; the requested *name* is neither in the query nor compared afterwards, and the caller receives `success=True`, `code="METADATA_UPDATED"` and the **pre-existing** file's id); views-postprocessing `unfao/managers/unfao.py::_assert_delivery_is_findable`; consumed by views-faoapi `managers/prediction/manager.py:284-292`, which resolves by `filename` | -| **Notes** | The FAO sidecar was never created, while the log recorded `uploaded rusty_bucket_forecasting_20260929_172325__sidecar.parquet`. Its bytes were identical to run-0's sidecar from August, the uploader matched on content hash, ran `update_document` against **run-0's file**, and returned that file's id. The manifest then referenced a name that did not exist; views-faoapi resolved by name, found nothing, and refused the run (`ingest_failed`). Bucket listing: run-0 110 objects, ours 109. **The consumer refused correctly — the silence is entirely on the producing side.** views-pipeline-core confirmed the mechanism in source and noted the asymmetry that makes it a defect rather than a design choice: the sibling branch handling `FOUND_BY_NAME` with a *different* hash deletes and re-uploads correctly. Name-awareness exists in one direction only. **The sentence worth keeping:** *manifest-last protects against a run that stops early; it does not protect against an upload that returns success having written nothing.* **Why Tier 2 and not 3:** it is structural, the trigger is not hypothetical but certain — it fires on every delivery — and it defeats guards written in the obvious way, since a check for "did I get an id back" passes here. **Why it is Tier-1-adjacent:** `update_document` wrote this run's metadata onto run-0's file, so the only complete run in the partner bucket may now carry corrupted provenance; storage is provably untouched (`createdAt == updatedAt` to the millisecond) but the metadata document is unverified. If that corruption is confirmed, this is silent corruption of a partner-visible record and should be re-tiered. **The deeper question, deliberately not answered here:** ADR-013 §4.2 requires one sidecar *per run* for a file that cannot vary by run, so forcing the upload or making dedup name-aware would satisfy the contract by storing N identical ~894 KB copies — obeying the contract rather than solving the problem. views-faoapi proposes one shared sidecar versioned by GAUL edition instead. That spans four repositories and is a maintainer decision. **Exit:** an upload verifies that the name it was asked to write resolves, and the delivery's completeness check verifies every artefact its manifest references — by name, not by returned id. Cross-refs: **views-pipeline-core#551** (the upload half), the views-postprocessing guard half, **C-154** (the same "reported success, did not happen" shape at the filesystem layer), views-models `reports/prereg_rusty_bucket_landing_20260929.md` (probe P3, which this failed). | +| **Notes** | The FAO sidecar was never created, while the log recorded `uploaded rusty_bucket_forecasting_20260929_172325__sidecar.parquet`. Its bytes were identical to run-0's sidecar from August, the uploader matched on content hash, ran `update_document` against **run-0's file**, and returned that file's id. The manifest then referenced a name that did not exist; views-faoapi resolved by name, found nothing, and refused the run (`ingest_failed`). Bucket listing: run-0 110 objects, ours 109. **The consumer refused correctly — the silence is entirely on the producing side.** views-pipeline-core confirmed the mechanism in source and noted the asymmetry that makes it a defect rather than a design choice: the sibling branch handling `FOUND_BY_NAME` with a *different* hash deletes and re-uploads correctly. Name-awareness exists in one direction only. **The sentence worth keeping:** *manifest-last protects against a run that stops early; it does not protect against an upload that returns success having written nothing.* **Why Tier 2 and not 3:** it is structural, the trigger is not hypothetical but certain — it fires on every delivery — and it defeats guards written in the obvious way, since a check for "did I get an id back" passes here. **Why it is Tier-1-adjacent:** `update_document` wrote this run's metadata onto run-0's file, so the only complete run in the partner bucket may now carry corrupted provenance; storage is provably untouched (`createdAt == updatedAt` to the millisecond) but the metadata document is unverified. If that corruption is confirmed, this is silent corruption of a partner-visible record and should be re-tiered. **The deeper question, deliberately not answered here:** ADR-013 §4.2 requires one sidecar *per run* for a file that cannot vary by run, so forcing the upload or making dedup name-aware would satisfy the contract by storing N identical ~894 KB copies — obeying the contract rather than solving the problem. views-faoapi proposes one shared sidecar versioned by GAUL edition instead. That spans four repositories and is a maintainer decision. **ESCALATED TO TIER 1, 2026-09-29, after views-postprocessing established the worst case.** The failure is not 109 of 110 artefacts — **it is 0 of 110.** `contract/wire/naming.py:19-31` embeds the run id in every filename (`{run_id}__{target}__m{time_id}.arrow.parquet`, `{run_id}__sidecar.parquet`, `{run_id}__manifest.json`), so a re-publish produces **new filenames over identical bytes**. Every one of the 110 objects then walks the dedup path, the store creates 110 documents, the run logs 110 successful uploads and a clean completion, and **nothing in the bucket is servable**. And this platform's documented remedy for a torn run is *re-publish* (views-postprocessing C-105, C-22) — **so the prescribed response to a failed delivery is the trigger for the total-failure case.** Today survived only because it was a first-of-kind collision rather than a repeat; a re-publish was proposed as the fix on the day and would have produced a silent total failure. The two halves were found separately and neither seat could see the other's: views-postprocessing verified the run-id-in-filename half, this seat measured that pooling is deterministic (25/25 anchor cells byte-identical across two independent pools, full sha256 over each draw vector — `reports/prereg_rusty_bucket_landing_20260929_values.json`), which is what makes identical bytes certain rather than possible. **Why Tier 1 rather than 2:** from the producing side there is no error signal at all — 110 successes, a clean exit and a WandB success alert — which is the register's own Tier 1 criterion; and the observed `update_document` against run-0's file means partner-visible provenance may already be corrupted. **The honest counterargument, recorded so a reviewer can push back:** the consumer *does* refuse, so FAO receives nothing rather than something wrong, which is an availability failure rather than served corruption. That counterargument is why this was first filed Tier 2. It is outweighed by the remedy-is-the-trigger property and by the fact that the operator, following documented procedure, is told everything worked. **Exit:** an upload verifies that the name it was asked to write resolves, and the delivery's completeness check verifies every artefact its manifest references — by name, not by returned id. Cross-refs: **views-pipeline-core#551** (the upload half), the views-postprocessing guard half, **C-154** (the same "reported success, did not happen" shape at the filesystem layer), views-models `reports/prereg_rusty_bucket_landing_20260929.md` (probe P3, which this failed). | --- From ee2fa2ae758acfd4aaa8ff7eafe4783f08591904 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 21:30:16 +0200 Subject: [PATCH 3/6] =?UTF-8?q?docs(FAO):=20pre-registration=20v2=20?= =?UTF-8?q?=E2=80=94=20four=20sessions=20found=20what=20one=20seat=20could?= =?UTF-8?q?=20not?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v1 was written alone in a morning. Its most serious defects were all found by someone else, and this rewrite is mostly other people's work. WHAT V1 GOT WRONG, structurally rather than in detail: It tested whether the delivery was WELL-FORMED. It could not have told us whether it was CORRECT. Of ten probes, five tested structure, one tested serving, and ZERO tested value correctness, cross-end agreement, or whether an upload actually wrote what it reported — the last of which is what failed. It also modelled the chain as two hops when there are five, conflating the §3.2 per-target Hop-A manifest with the §4.2 per-run Hop-B one; believed /provenance when forecast_serving_state is authoritative; and reported itself as "5 of 10 passed", which treats a file count and a value check as exchangeable units and is a number that leaves the room. THE BIRTH TEST V1 FAILED: of its ten probes I expected roughly ONE to fail when writing them. Writing probes you expect to pass is this method's p-hacking. v2's probes are written falsifier-first — what observation would let this pass while the system is broken — and a probe whose falsifier is hard to state is too weak to keep. WHAT IS NEW, by contributing seat: views-hydranet — all of Class C, which did not exist. Total mass by horizon is the highest-value number we were not computing: monotone growth is the bloom (Epic #193), measured at ~230x unfixed and ~12x fixed. The gate is not one mechanism — 5 soft, 3 threshold at three DIFFERENT taus — so the pooled zero-fraction is a mixture and is not interpretable as P(event). A spatial translation is invisible to every structural probe (M54: rolling state by 90 cells moves the forecast 90 cells at r~0.90 while skill collapses 48x). And v1's own anchor cell was misread as undertraining when it has the shape of gate/body decoupling. views-datafactory — the criticism that reshaped the document: every v1 probe tested a property of one artefact in isolation, and the failures that matter produce artefacts that are internally perfect. Hence Class B. Also C-353: nine delivered cells carry the WRONG administrative unit right now, because the area-majority join ranks polygons in square degrees rather than area above 55 degrees. unmapped_count: 0 is true for all nine — it is a null-check, not a correctness check, and v1 read it as reassurance. library — sections 0, 1 and 2. The pass-rate framing is the most dangerous thing v1 produced because it is the artefact that leaves the room. Pre-registration borrows credibility from clinical trials, where it prevents fishing; our failure mode is failure of imagination, which the form does not address. views-postprocessing and views-faoapi — the chain model, the authoritative endpoint, and three direct corrections to my claims. ONE THING MEASURED HERE RATHER THAN RELAYED: views-hydranet identified that the cube generator is seeded from torch_seed alone, with no model identity, so blue_stranger and bright_starship (both seed 43, both soft-gated) draw from identical streams. Tested on calibration draws: the shared-seed pair shows 3.42% identical 16-draw masks and +2.70pp agreement over independence; a seeds-44/47 control shows 0.93% and -0.01pp. The mechanism is confirmed and the magnitude is modest — consistent with the NB rejection sampler diverging the streams. 128 pooled draws are not 128 independent draws. The determinism result cannot substitute for this: re-pooling to byte-identical output proves determinism, not non-duplication. Section 6 lists what we will certify as clean while knowing it is wrong — the nine mislabelled cells, the frozen c_id across the forecast horizon, and that lookup_version is a digest of an ingestion rather than of a GAUL edition, which makes views-postprocessing#313's "version by GAUL edition" not currently possible. Section 7 states what the suite still cannot do, including that it is written by the seat that ran the failed delivery and inherits that seat's priors. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- reports/prereg_fao_delivery_v2.md | 297 ++++++++++++++++++++++++++++++ 1 file changed, 297 insertions(+) create mode 100644 reports/prereg_fao_delivery_v2.md diff --git a/reports/prereg_fao_delivery_v2.md b/reports/prereg_fao_delivery_v2.md new file mode 100644 index 00000000..5fbfbd43 --- /dev/null +++ b/reports/prereg_fao_delivery_v2.md @@ -0,0 +1,297 @@ +# Pre-registration v2 — proving (or falsifying) an FAO delivery + +**Written 2026-09-29, after the first end-to-end run failed and after adversarial review by four +other sessions.** Supersedes `prereg_rusty_bucket_landing_20260929.md`, which is retained +unedited as the record of what v1 was and how it did. + +--- + +## 0. What this suite is, and what it cannot be + +**It tests the failure modes we can currently articulate. It does not and cannot test for unknown +unknowns. A finding outside the registered probes is not a lesser finding** — v1's two most +valuable results (that pooling is deterministic, and that uploads can report success having +written nothing) were both accidents, not probes. + +This is stated first because v1 lacked it, and because a pre-registration that makes unexpected +findings feel illegitimate is working against its own purpose. + +**The form's limits, named so they cannot flatter us.** Pre-registration is borrowed from clinical +trials, where it prevents fishing over continuous outcomes. Our failure mode is different: not +post-hoc rationalisation but **failure of imagination**. This platform has both, so the form still +earns its place — but it addresses only the first, and its credibility should not be borrowed for +the second. + +--- + +## 1. How v1 did, honestly + +Not "5 of 10 passed". That framing treats probes as exchangeable units when a file count and a +value check are nothing alike, and it is a number that leaves the room and becomes "the +pre-registered suite passed majority". + +| what was tested | probes | result | +|---|---|---| +| **structure** — counts, spans, alignment, names | 5 | all passed | +| **value correctness** — are the numbers right | **0** | **none existed** | +| **cross-end agreement** — do both ends mean the same thing | **0** | **none existed** | +| **delivery mechanism** — did the upload write what it said | **0** | **none existed, and this is what failed** | +| served to FAO | 1 | FAILED | +| blocked behind the failure | 3 | UNRUN | +| defective premise | 1 | assumed only one component writes | + +**The honest summary: v1 tested whether the delivery was well-formed. It could not have told us +whether it was correct.** + +**The birth test I failed.** Of v1's ten probes, I expected roughly **one** to fail when writing +them. Writing probes you expect to pass is this method's equivalent of p-hacking. v2's probes are +written falsifier-first — *what observation would let this pass while the system is broken?* — and +a probe whose falsifier is hard to state is too weak to keep. + +--- + +## 2. Reporting rules + +1. **PASS / FAIL / UNRUN.** A probe reasoned about but not executed is UNRUN, never a pass. +2. **Never report a bare count of passes.** Report by class (§4), so the reader sees the *shape* of + what was and was not tested. +3. **A FAIL is a FAIL.** Not a caveat, not an asterisk. +4. **A probe that turns out badly designed is recorded as a defect in this document** and re-run + corrected — not dropped, not counted. +5. **Every probe is labelled R (regression — tests something that has already failed) or + D (discovery — tests a failure mode not yet observed).** v2 is deliberately majority-D; a suite + that is all-R only ever catches yesterday. + +--- + +## 3. The chain, correctly modelled + +v1's central structural error: it treated "published" and "landed" as one step. There are five. + +``` + L1 pod 8 models forecast -> rusty_bucket pools -> 128 draws + L2 Hop-A publish to production_forecasts (§3.2 manifest PER (run,target)) + L3 transform un_fao postprocessor: curate land -> land_gaul, re-emit + L4 Hop-B unfao_bucket (§4.2 manifest PER run) + L5 serve views-faoapi ingest -> /data endpoints +``` + +**Two different manifest kinds.** Conflating them cost three sessions an afternoon. + +**Which endpoint is authoritative:** `forecast_serving_state` (`reason`, `fallback_available`). +`/health`'s `status` and `forecast_freshness` read the newest record in the store, **not what is +served** — they reported `healthy, age 0.0 days` while the run was refused. +`/provenance/forecast` answers a legacy resolution path. v1 believed `/provenance`. + +--- + +## 4. The probes + +### Class E — did the delivery mechanism actually do what it reported? (R) + +*The class that did not exist in v1, and the one that failed.* + +**E1 — every artefact the manifest names resolves by name.** (R) +Not "did the upload return an id" — it returned a real id for the wrong file. Resolve each of the +110 filenames. +*Falsifier:* any name the manifest references does not resolve. *Passes-while-broken if:* we check +ids rather than names, or check a count. **Both were true in v1.** + +**E2 — object count at L4 equals L2's, per type.** (R) +Run-0 has 110; our failed run had 109. + +**E3 — a re-publish is verified, not assumed.** (D) +Because pooling is deterministic and filenames carry the run id, a re-publish writes new names over +identical bytes and **all 110 can hit content-hash dedup at once** — 110 successes, nothing +servable. Re-publishing is this platform's own documented remedy for a torn run (C-105/C-22), so +this is the realistic path, not the exotic one. +*Falsifier:* after a re-publish, any name fails to resolve. + +### Class B — do both ends mean the same thing? (mostly D) + +*datafactory's core point: every v1 probe tested a property of the delivery in isolation. A shifted +convention, a stale lookup, a re-derived identifier all produce internally perfect artefacts.* + +**B1 — set equality of cells, not cardinality.** (R) +`set(delivered) == set(load_region_pgids("land_gaul"))`. +*Falsifier:* any difference either way. *Passes-while-broken if:* we compare `len()` — **v1 did +exactly that, and I recorded it as a PASS having predicted this very risk.** `land_gaul ⊂ land` is +verified (64,742 of 64,818, 76 dropped, zero additions), so the count is not vacuous — but it +cannot see 76 *substituted* cells. + +**B2 — geometry round-trip on every delivered id.** (D) +`latlon_to_pgid(pgid_to_latlon(ids)) == ids`. Vectorised, instant. +*Falsifier:* any id fails. Catches reindexing or convention change end to end. + +**B3 — named-place anchors.** (D) +Five cells whose location can be stated in words (e.g. 148759 = Mekelle, Tigray, 13.25N 39.25E). +Assert coordinates *and* admin labels. A convention flip survives every aggregate check and dies +here. + +**B4 — no unmapped sentinel reaches FAO.** (D) +`-1` is the unmapped sentinel (163,622 of 259,200 cells are ocean). Assert independently rather +than trusting the manifest's `unmapped_count` to have computed it. +**`unmapped_count: 0` is a null-check, not a correctness check** — see §6. + +**B5 — both legs share one `lookup_version`.** (D) +The manifest carries **one** `lookup_version`, not one per leg. Nothing compares the historical and +forecast legs. Assert explicitly. + +**B6 — month boundaries at both ends.** (R) +`month_id = (year-1980)*12 + month`, epoch pinned; 561 is 2026-09 everywhere. Assert min, max and +distinct count map to the intended calendar months. +*Note:* the converter returns 0 for Dec-1979 and negatives before, rather than raising — a bad date +arrives as a plausible small integer. + +### Class C — are the numbers right? (all D) + +*hydranet's contribution. v1 had nothing here at all.* + +**C1 — total mass by horizon, h=1..36, per model and pooled.** (D) +**The highest-value number we were not computing.** A healthy gated rollout is flat or mildly +decaying. **Monotone growth is the bloom** (Epic #193, ADR-070) — gate/body decoupling in the +36-step autoregressive rollout, measured at ~230× unfixed and ~12× fixed. All 8 configs set +`rollout_feedback: 'sample'`, which is the fix, **so we should be in the fixed regime — measure it, +do not assume it.** +*Falsifier:* h36 total many times h1. + +**C2 — zero-fraction by horizon.** (D) +Both directions are failures: rising toward 1.0 is zero-collapse; falling is the bloom. Flat is +healthy. + +**C3 — zero-fraction decomposed by model; exactly 3 must be all-or-nothing.** (D) +**The gate is not one mechanism.** 5 models are `soft_gate` (per-draw Bernoulli); 3 are +`threshold_gate` with *different* τ — `violet_visitor` 0.20, `blazing_meteor` 0.16, `bold_comet` +0.14 (verified in the configs). Threshold models emit blocks of 16 that are all-zero or +all-nonzero. +*Falsifier:* the count of all-or-nothing models is not exactly 3 — a config did not load as +believed. +**Consequence to record, not test:** the pooled zero-fraction is a mixture of two mechanisms and is +**not** interpretable as "the ensemble's P(event) = nonzero/128". No single occurrence threshold +exists across the eight. + +**C4 — draw-block ordering, checked by behaviour not metadata.** (D) +Draws 0–15 are model A, 16–31 model B, and so on. Confirm the all-or-nothing blocks land on the +three threshold models. A free axis-order check that uses model behaviour rather than trusting +metadata. + +**C5 — spatial correlation against the last observed month.** (D) +The h=1 forecast's nonzero cells should overlap the truth month heavily. **This repo measured (M54) +that rolling the recurrent state by 90 cells moves the forecast by 90 cells with r≈0.90 while skill +collapses 48×.** A translated forecast is cell-count-perfect and completely wrong; no structural +probe can see it. + +**C6 — order of magnitude against truth, with a directional threshold.** (D) +v1's P10 said "must look undertrained" with no threshold, which is not a test. The threshold: +**the ensemble should undershoot observed totals by roughly 5–20×.** The body is documented as +timid (`size_ratio` measured as low as 0.02). +*Falsifier — and note the direction:* **an ensemble that matches observed totals is the alarming +result, not the good one.** + +**C7 — gate↔body target pairing.** (D) +`to_cube_samples` slices `gate[..., :n_reg]` and applies gate channel *j* to regression target *j*, +positionally. **No validator asserts `classification_targets[j]` is the gate for +`regression_targets[j]`.** A reordered config gates sb's body with ns's gate: right shape, right +cells, right totals, wrong pairing, no error. Assert the two lists are the same targets in the same +order, read from the config the run actually used. + +**C8 — per-model max draw: record, do not gate.** (D) +`M_max` **false-flags** — a single enormous draw once made a clean field look like "blooms +everywhere". Record it; if one model's max is orders above the others, look at that model. Do not +fail on it. *This is the correction to v1's handling of its own anchor cell (§6).* + +### Class D — is the ensemble what it claims to be? (D) + +**D1 — draw independence across models sharing a seed.** (D) +`torch_seed`: pink_pirate 42, violet_visitor 42, blue_stranger 43, bright_starship 43, +purple_alien 44, bold_comet 45, blazing_meteor 46, heavy_freighter 47. The cube generator is seeded +from `torch_seed` **and nothing else** — no model identity — so two models with the same seed draw +from byte-identical streams. + +**Measured on calibration draws, 2026-09-29 (origin_0, `lr_sb_best`):** + +| pair | identical 16-draw mask | per-draw agreement | excess over independence | +|---|---|---|---| +| seed 43 both (`blue_stranger`, `bright_starship`) | **3.42%** | 72.21% (indep. 69.51%) | **+2.70 pp** | +| control (44 / 47) | 0.93% | 69.67% (indep. 69.68%) | **−0.01 pp** | + +The control is exactly independent. The shared-seed pair shows 3.7× the identical-mask rate and a +real excess. **The mechanism is confirmed; the magnitude is modest** — consistent with the NB +rejection sampler diverging the streams after the first target's draws. So 128 pooled draws are not +128 independent draws, though they are much closer than the mechanism alone would suggest. +*Falsifier for a run:* excess over independence materially above the ~3 pp measured here. +**Why the determinism check cannot substitute:** re-pooling and getting byte-identical output proves +*determinism*, not *non-duplication*. That is precisely the gap. + +### Class A — structure (R, and known weak) + +v1's P1, P2, P3, P7 retained unchanged: 128 draws; exactly 3 targets published by name and 3 +withheld; 36 shards per target with a manifest each; months 561–596. + +**Kept because they are cheap and catch gross regressions. Labelled weak because passing them is +nearly uninformative** — they all passed on a delivery that was unservable. + +--- + +## 5. Cross-end, not self-consistent + +The single sharpest criticism received: **every v1 probe tested a property of one artefact.** +`expected_cell_count`, `unmapped_count`, row counts are all self-consistency checks. The failures +that matter produce artefacts that are internally perfect. + +Class B exists entirely to compare the two ends. Where a probe can be written as either a +self-check or a cross-check, **write the cross-check.** + +--- + +## 6. Known-wrong things we will certify anyway + +Stated so that "clean" is never read as "correct". + +- **Nine delivered cells carry the wrong administrative unit, right now.** views-datafactory C-353: + the area-majority join ranks polygons by area in raw EPSG:4326 — square degrees, not area — so + above 55° the ranking can pick the wrong winner. Measured: **9 of 3,581 delivered border cells**; + 8 differ at `gaul2`, 1 also at `gaul1`, none at `gaul0`. All nine are inside our 64,742, and + `unmapped_count: 0` is true for every one. faoapi serves `/gaul2/` endpoints, so the labels are + retrievable. Fix deferred (#471) because applying it republishes the reference layer FAO is + currently served. +- **`c_id` is frozen across the forecast horizon.** `extrapolate_time` repeats the last observed + frame and increments only the time column. `priogrid_gid`, `row`, `col` are time-invariant so + copying is correct; **`c_id` is not.** All 36 forecast months carry the last observed month's + country map. Anything aggregating grid→country by `c_id` uses one month's map for three years. + This may even be intended — but nothing states it and nothing checks it. *v1 could not have seen + this: it checked identifier alignment across targets, and all targets share the same frozen + `c_id`, so they agree and all pass.* +- **`lookup_version` is a digest of datafactory's ingestion, not of a GAUL edition.** It moves when + the harvest changes for any reason, including a re-harvest producing identical geography. + **Consequence for views-postprocessing#313: "version the sidecar by GAUL edition" is not + currently possible** — that field is not carried into the artifact. +- **v1's anchor cell was misread.** 121/128 zeros with one draw of 29,536 was recorded as + "undertrained behaviour". It is the shape of gate/body decoupling: all 7 nonzero draws came from + soft-gate models, implying a gate near 0.09, while the body emitted `log1p ≈ 10.29` against a + training range topping out near 11.6 — a near-maximum value in a cell the gate believes is empty. + **One cell cannot settle it** (M_max false-flags), which is why the diagnostic is C1, not this. + +--- + +## 7. What this suite still cannot do + +- It cannot detect a failure class nobody has articulated. §0. +- It cannot see upstream defects: C-353's nine mislabelled cells are invisible to every probe here. +- It says nothing about forecast **skill**. A delivery can be perfect and the forecasts poor. +- It is written by the seat that ran the failed delivery, so it inherits that seat's priors. + **The standing recommendation is that someone who was not in the room writes probes for the next + revision** — the `opponent` session exists for exactly this. + +--- + +## 8. Acknowledgements, because the corrections were load-bearing + +`views-faoapi` established the refusal mechanism and that coverage curation was correct. +`views-postprocessing` corrected three of my claims, including that xarray is capped rather than +unpinned, and that my proposed findability fix would not have worked. `views-datafactory` supplied +C-353, the subset proof, the epoch, and the cross-end criticism that reshaped this document. +`views-hydranet` supplied all of Class C and the seed mechanism in Class D. The `library` seat +supplied §0, §1, §2 and the falsifier-first method. + +**v1 was written alone in a morning. Its most serious defects were all found by someone else.** From 5d887c58cb9a3ea5e92ae27dea76e6309347ec6e Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 21:49:34 +0200 Subject: [PATCH 4/6] =?UTF-8?q?docs(FAO):=20pre-registration=20v3=20?= =?UTF-8?q?=E2=80=94=20re-indexed=20on=20the=20chain,=20and=20the=20four?= =?UTF-8?q?=20dropped=20probes=20restored?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `opponent` session — the only reviewing seat that was not in the room — found that v2 is a better document that lost the four probes v1 called its strongest. Verified before accepting: P4 (stem collision), P5 (the served run is ours), P8 (served values == produced values) and P9 (the negative control) were all absent from v2, with no successor, sitting beside four EXPLICIT retentions in the same section. THE STRUCTURAL CAUSE, which is the real finding. v1 was indexed by the CHAIN: coverage could be audited by reading down the hops and seeing an empty one. v2 re-indexed on a TAXONOMY of failure classes — good analysis, bad table of contents, because a taxonomy has no empty cells. A class can look full while a hop is empty. Re-adding the four probes would have kept the structure that made them droppable, so v3 re-indexes on the chain and keeps the classes as tags. Every probe is pinned to a hop; an empty hop is visibly empty. AND A BETTER DIAGNOSIS THAN THE ONE V2 WROTE ABOUT ITSELF. v2 blamed "the seat that ran the failed delivery inherits that seat's priors". Wrong. Every reviewer was a repo owner contributing probes about their own repo. NOBODY OWNS THE SEAM — "what the pod produced == what FAO serves" spans two repos and belongs to neither — so no reviewer had it in scope and the one probe covering it vanished unnoticed by five readers. v2's coverage map was the org chart, which is more dangerous than one seat's bias because it looks like coverage. P8 now has its own section. SPLIT INTO TWO DOCUMENTS. Every one of v2's Class C probes is computed pod-side from the raw 128-draw cube, and FAO is served MAP/HDI summaries — so Class C cannot cross the seam and is not a delivery probe at all. It now lives in prereg_model_health_v1.md. Different claim, different falsifiers, different reader; merged only because both were learned on the same day. P8 IS NOW EXECUTABLE, which it was not in v1. v1 flagged that FAO is served MAP/HDI rather than draws and never resolved which statistic to compare, so the probe was a wish. views-faoapi's _compute_single_map delegates to views-frames' tower_point, so v3 imports the same function faoapi uses rather than reimplementing it — removing the "compared the wrong thing" failure mode by construction rather than by care. P9 is REWRITTEN rather than restored: v1's premise was wrong. It assumed only the forecast publish writes, and the un_fao postprocessor legitimately rewrites the historical leg, so v1 would have failed itself for correct behaviour. v3's control is "nothing outside the DECLARED DELIVERY SURFACE moved", including run-0's metadata document — which is marked UNRUN at registration, because it needs the operator console, rather than discovered UNRUN afterwards. R/D RE-LABELLED against a stated scope. v2 claimed "deliberately majority-D" while labelling as D the bloom, M54's translation failure, the timid body and the seed effect — all measured and published. Under the stated scope the delivery suite is majority R and says so. LIMITATIONS: v2 dropped three concrete ones v1 had — durability, the production run differing, whether FAO can use it — for more sophisticated but operationally weaker ones. All three restored, plus two that were missing: the seam has no owner, and `opponent` is now INSIDE the loop, so its review is an event that happened rather than a property this document has. Verified by the document's own mechanism before committing, and it caught two of my own defects: a grouped provenance row (P1/P2/P3/P7 on one line) that was not auditable, which is this document's own thesis applied to itself; and P5 having lost the `created_at` field name that makes it checkable. Both fixed. All ten v1 probes now individually accounted for as restored, superseded-by-a-named-successor, or rewritten-with-a-reason. Measured rather than remembered, since I told opponent v2 was "three times" v1 and it was 1.37x: v1 11,917 chars, v2 17,809, v3 16,698, model-health 11,472. v3 is SMALLER than v2 while restoring four probes, because Class C moved out. v1 and v2 retained unedited. A pre-registration that can be rewritten after the fact is not one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- reports/prereg_fao_delivery_v2.md | 23 +++ reports/prereg_fao_delivery_v3.md | 316 ++++++++++++++++++++++++++++++ reports/prereg_model_health_v1.md | 236 ++++++++++++++++++++++ 3 files changed, 575 insertions(+) create mode 100644 reports/prereg_fao_delivery_v3.md create mode 100644 reports/prereg_model_health_v1.md diff --git a/reports/prereg_fao_delivery_v2.md b/reports/prereg_fao_delivery_v2.md index 5fbfbd43..a587abb0 100644 --- a/reports/prereg_fao_delivery_v2.md +++ b/reports/prereg_fao_delivery_v2.md @@ -1,5 +1,28 @@ # Pre-registration v2 — proving (or falsifying) an FAO delivery +> **SUPERSEDED 2026-09-29 by `prereg_fao_delivery_v3.md` and `prereg_model_health_v1.md`.** +> Retained **unedited** below — a pre-registration that is rewritten after the fact is not one. +> +> **Why it was superseded, recorded here so the record is not only in the successor:** the +> `opponent` session found that v2 silently dropped four of v1's probes — P4 (stem collision), +> P5 (the served run is ours), P8 (served values == produced values) and P9 (the negative +> control) — with no successor and no acknowledgement, alongside four *explicit* retentions in +> the same section. Those were the three v1 itself singled out as its strongest. +> +> The structural cause: **v1 was indexed by the chain and v2 by a taxonomy.** A taxonomy has no +> empty cells, so a class can look full while a hop is empty. v3 re-indexes on the chain and keeps +> the classes as tags. +> +> Two further errors of v2 recorded for the same reason: it reported itself as "5 of 10 passed", +> which treats a file count and a value check as exchangeable units; and its limitations section +> dropped three concrete limitations v1 had (durability, the production run differing, whether FAO +> can use it) in favour of more sophisticated but operationally weaker ones. +> +> What v2 got right and v3 keeps: the five-hop chain model, the two manifest kinds, +> `forecast_serving_state` as the authoritative endpoint, set equality replacing cardinality, a +> directional threshold for the magnitude check, and the refusal to report a bare pass count. + + **Written 2026-09-29, after the first end-to-end run failed and after adversarial review by four other sessions.** Supersedes `prereg_rusty_bucket_landing_20260929.md`, which is retained unedited as the record of what v1 was and how it did. diff --git a/reports/prereg_fao_delivery_v3.md b/reports/prereg_fao_delivery_v3.md new file mode 100644 index 00000000..0770fd45 --- /dev/null +++ b/reports/prereg_fao_delivery_v3.md @@ -0,0 +1,316 @@ +# Pre-registration v3 — does an FAO delivery arrive, intact and ours? + +**Written 2026-09-29 after `opponent` — the only reviewing seat that was not in the room — +established that v2 lost the four probes v1 called its strongest.** Supersedes +`prereg_fao_delivery_v2.md` for delivery verification. Its model-health half moved to +`prereg_model_health_v1.md`; the two are siblings and neither is a superset. + +v1 (`prereg_rusty_bucket_landing_20260929.md`) and v2 are retained **unedited**. A +pre-registration that can be rewritten after the fact is not one. + +--- + +## How to audit this document in ten seconds + +**Read down the hops.** Every probe is pinned to the hop it exercises. **A hop with no probe is a +hole** — you do not have to notice an omission, you have to see an empty row. + +That mechanism is the whole point of v3. v1 was indexed by the chain and its coverage could be +audited this way. v2 re-indexed on a taxonomy of failure classes — good analysis, bad table of +contents, because **a taxonomy has no empty cells**: a class can look full while a hop is empty. +Four probes fell out and none of five reviewers noticed. + +--- + +## The claim + +> A named FAO delivery has arrived, is being served, and the values FAO receives are the values we +> produced. + +Three parts, and v2 could only test the first. **Arrived** is structural. **Served** is a +provenance question. **Ours** is a value question, and it is the one that has never been tested. + +--- + +## The chain + +``` +L1 pod produce and pool P1 +L2 Hop-A publish -> production_forecasts (§3.2 manifest per (run,target)) P2 P3 E3 +L3 transform un_fao postprocessor, land -> land_gaul P6 B1 B4 B5 +L4 Hop-B unfao_bucket (§4.2 manifest per run) E1 E2 +L5 serve faoapi ingest -> /data endpoints P5 P4 P7 B6 +ACROSS served values == produced values P8 <- the spine +CONTROL nothing outside the declared surface moved P9 +``` + +**Two manifest kinds.** Conflating them cost three sessions an afternoon. §3.2 is the Hop-A +producer manifest, one per *(run, target)*. §4.2 is the Hop-B run manifest, one per *run*, with a +`targets` list. + +**Which endpoint is authoritative: `forecast_serving_state`** (`reason`, `fallback_available`). +`/health`'s `status` and `forecast_freshness` read the newest record in the store, **not what is +served** — they reported `healthy, age 0.0 days` while the run was refused. +`/provenance/forecast` answers a *legacy* resolution path. **v1 believed `/provenance`.** + +--- + +## Reporting rules + +1. **PASS / FAIL / UNRUN.** Reasoned about but not executed is UNRUN, never a pass. +2. **Never report a bare count of passes.** Report per hop, so the reader sees the shape of what + was and was not tested. "5 of 10 passed" treats a file count and a value check as exchangeable, + and it is the artefact that leaves the room and becomes "the pre-registered suite passed". +3. **A FAIL is a FAIL.** Not a caveat. +4. **A badly designed probe is recorded as a defect in this document**, re-run corrected — not + dropped, not counted. +5. **R / D labels, against a stated scope.** **R** = tests a failure mode already observed + *anywhere on this platform*. **D** = not yet observed anywhere on this platform. v2 claimed + "majority-D" while labelling as D things that were measured and published (the bloom, M54, the + seed effect). Honest count here: **the delivery suite is majority R, and says so.** + +--- + +## L1 — pod: produce and pool + +**P1 — the pool is 128 draws, not 16 and not 48.** `R` · *v1 P1, retained* +`8 models × 16 = 128` per regression target, matching `expected_models` and +`expected_samples_per_model`. +*Falsifier:* any count below 128, or a run completing with fewer than 8 contributors. A pool that +quietly drops a constituent still produces a plausible forecast. +*Executable:* yes, pod-side. + +--- + +## L2 — Hop-A: publish to the producer store + +**P2 — exactly three targets are published, by name.** `R` · *v1 P2, retained* +Manifests for exactly `lr_ged_sb`, `lr_ged_ns`, `lr_ged_os`. +*Falsifier:* **6** — views-pipeline-core#536's filter did not take, and classification channels +reached the partner store. **0** — the filter computed empty and the run delivered nothing while +reporting success. Any other count. +*Status note:* PASSED on 2026-09-29 against the real store, first exercise ever. + +**P3 — no torn commit.** `R` · *v1 P3, retained* +Every manifested shard resolves; no shard exists whose manifest is absent. +*Falsifier:* shards without a manifest, or a manifest referencing a missing shard. +*Status note:* FAILED on 2026-09-29 (network timeout, 35/36 on one target), then PASSED on retry. + +**E3 — a re-publish is verified, not assumed.** `R` +Pooling is deterministic (measured: 25/25 anchors byte-identical across a re-pool) and filenames +carry the run id, so **a re-publish writes new names over identical bytes and every artefact can +hit content-hash dedup at once** — N successes, nothing servable. Re-publishing is this platform's +documented remedy for a torn run (views-postprocessing C-105/C-22), so this is the realistic path. +*Falsifier:* after a re-publish, any manifested name fails to resolve. +*Executable:* yes, and **this must be run at least once before the production run**, because the +remedy is the trigger. + +--- + +## L3 — transform: the postprocessor and the curation + +**P6 — the curation happened.** `R` · *v1 P6, retained* +`region = land_gaul`, `expected_cell_count = 64742`, `unmapped_count = 0`. +*Status note:* PASSED on 2026-09-29. This was v1's pre-registered "most likely silent failure" and +it did not happen. + +**B1 — set equality of cells, not cardinality.** `D` +`set(delivered) == set(load_region_pgids("land_gaul"))`. +*Falsifier:* any difference either way. +**Why this exists:** P6 checks a *number*. `land_gaul ⊂ land` is verified (64,742 of 64,818, 76 +dropped, zero additions), so the count is not vacuous — **but it cannot see 76 substituted cells.** +v1 predicted the coverage risk, tested cardinality, and recorded a PASS. That is the probe shaped +to confirm rather than to falsify. + +**B4 — no unmapped sentinel reaches FAO.** `D` +`-1` is the unmapped sentinel (163,622 of 259,200 cells are ocean). Assert independently rather +than trusting the manifest's own `unmapped_count`. +**`unmapped_count: 0` is a null-check, not a correctness check** — see "Known wrong" below. + +**B5 — both legs share one `lookup_version`.** `D` +The manifest carries **one**, not one per leg. Nothing compares the historical and forecast legs. +*Falsifier:* they differ, or the field is absent from either. + +--- + +## L4 — Hop-B: the partner bucket + +**E1 — every artefact the manifest names resolves BY NAME.** `R` +Not "did the upload return an id" — on 2026-09-29 it returned a **real id for the wrong file**. +*Falsifier:* any name the manifest references does not resolve. +*Passes-while-broken if:* we check ids, or counts. **Both were true in v1, and this is what +failed.** + +**E2 — object count at L4 equals L2's, per type.** `R` +*Status note:* run-0 has 110 objects; the 2026-09-29 run had **109**. That one-object gap was the +entire incident. + +--- + +## L5 — serve: faoapi ingest and the data endpoints + +**P5 — the served forecast is OURS.** `R` · *v1 P5, restored with the endpoint corrected* +*Falsifier:* `forecast_serving_state` reports a refusal; or the served run id is not the one +recorded pod-side before the publish; or `created_at` is not today's. +**Record the pod-side run id BEFORE the publish**, or this degrades into "a forecast exists", which +is not the claim. +**Endpoint correction:** v1 read `/provenance/forecast`. That is a legacy path and it returned +"No forecast prediction files found" while `/health` said `healthy, age 0.0`. Read +`forecast_serving_state`. + +**P4 — no silent stem collision.** `D` · *v1 P4, restored* +No two published shards resolve to the same consumer series. Spot-check values under a served +`sb_*` column against the pod's **regression** cube, and confirm they do **not** match the +**classification** cube for the same cells. +*Falsifier:* values under a fatality name match the probability cube. +**Now better founded than in v1:** confirmed that faoapi resolves by tokenising, that +`pred_lr_ged_sb` and `pred_cls_ged_sb` both reduce to the stem `sb`, and that **nothing raises** — +`series_of` rejects zero or ambiguous matches *within one name*, and neither of its callers checks +for a repeated stem. **This is the failure that puts probabilities in front of the UN under a +fatality label with no error anywhere.** + +**P7 — the horizon is right.** `R` · *v1 P7, retained* +36 months beginning at 561. +*Falsifier:* 37 (views-models#512's off-by-one reaching the wire), or a range starting before 561. + +**B6 — month boundaries mean the same thing at both ends.** `D` +`month_id = (year-1980)*12 + month`, epoch pinned as a module constant; 561 is 2026-09 everywhere. +Assert min, max and distinct count map to the intended calendar months at **both** ends. +*Note:* the converter returns 0 for Dec-1979 and negatives before it rather than raising, so a bad +date arrives as a plausible small integer. + +--- + +## ACROSS — the seam. The spine of this document. + +**P8 — the values served ARE the values we produced.** `D` · *v1 P8, restored and now executable* + +**The only probe that cannot be satisfied by coincidence, and the one nobody owned.** + +**Check.** Before the publish, record 20 random cell-months plus the 5 highest-valued, to full +precision, in a companion file. After the delivery, query the API for those same +`(month_id, priogrid_id)` pairs and compare. + +**v1 could not execute this.** It flagged that FAO is served MAP and HDI summaries rather than raw +draws, and never resolved which statistic to compare — so the probe was a wish. + +**Resolved.** `views-faoapi/data/handlers/grid_dataset.py::_compute_single_map` delegates to +**views-frames' `tower_point`** estimator. So this probe **imports the same function faoapi uses** +rather than reimplementing the statistic. That removes the "compared the wrong thing" failure mode +*by construction* rather than by care. + +*Falsifier:* values differ beyond floating-point tolerance. +*Carry this caveat:* views-faoapi holds `BUGREPORT_tower_point_degenerate_tip.md`. Do not silently +assume the estimator is sound — a disagreement may be the estimator, not the delivery. + +**Why this hop has its own section.** Every reviewing session was a repo owner and contributed +probes about its own repo. **The seam spans two repos and belongs to neither**, so it was in +nobody's scope and the one probe covering it vanished unremarked. That is not one seat's blind +spot; it is **the union of five seats' priors**, which is more dangerous because it looks like +coverage. + +--- + +## CONTROL — nothing outside the declared surface moved + +**P9 — negative control.** `R` · *v1 P9, rewritten rather than restored* + +**v1's premise was wrong** and this is recorded as a defect in v1, not a pass: it asserted the +historical leg must be unchanged, assuming only the forecast publish writes. The `un_fao` +postprocessor **legitimately** rewrites historical. v1 would have failed itself for correct +behaviour. + +**v3's control:** nothing outside the **declared delivery surface** moved — other buckets, other +collections, and specifically **run-0's metadata document**, which `update_file_metadata` touched +during the 2026-09-29 incident and which **nobody has yet been able to read**. + +*Falsifier:* any write outside the declared surface. +*Executable:* **partially.** Storage side yes — run-0's file still reads `createdAt == updatedAt` +to the millisecond, name unchanged. **The metadata document is not reachable from outside:** +faoapi exposes four `/files/…` routes, all storage, and `/provenance` reports only the served run. +It needs the operator console. **Marked UNRUN at registration rather than discovered UNRUN +afterwards.** + +**Why this is not theoretical:** the Appwrite key carries 20 scopes — full read, write and delete +across every database, table, bucket and file — and both platform keys expire 2026-11-17. + +--- + +## Known wrong, and certified anyway + +So that "clean" is never read as "correct". + +- **Nine delivered cells carry the wrong administrative unit.** views-datafactory C-353: the + area-majority join ranks polygons by area in **raw EPSG:4326 — square degrees, not area** — so + above 55° a cell is a narrow trapezium and the ranking can pick the wrong winner. Measured: + **9 of 3,581 delivered border cells**; 8 differ at `gaul2`, 1 also at `gaul1`, none at `gaul0`. + All nine are inside our 64,742, and **`unmapped_count: 0` is true for every one of them.** faoapi + serves `/gaul2/` endpoints, so the labels are retrievable. Deferred (#471) because the fix + republishes the reference layer FAO is currently served. +- **`c_id` is frozen across the forecast horizon.** `extrapolate_time` repeats the last observed + frame and increments only the time column. `priogrid_gid`, `row`, `col` are time-invariant so + copying is correct; **`c_id` is not.** All 36 forecast months carry one month's country map. + Possibly intended — nothing states it, nothing checks it. *No probe here can see it: every + target shares the same frozen `c_id`, so they agree and all pass.* +- **`lookup_version` is a digest of datafactory's ingestion, not of a GAUL edition.** It moves when + the harvest changes for any reason. **Consequence for views-postprocessing#313: "version the + sidecar by GAUL edition" is not currently possible** — that field is not carried into the + artifact. + +--- + +## What this suite cannot do + +- **It cannot detect a failure class nobody has articulated.** Findings outside these probes are + not lesser findings: v1's two most valuable results — that pooling is deterministic, and that + uploads can report success having written nothing — were both **accidents**. +- **Durability.** A forecast that lands and is then evicted, expired or superseded passes every + probe here at the moment of measurement. *(v1 said this; v2 dropped it.)* +- **The production run differs.** A green suite at 40 lessons says the plumbing carries data + correctly. The 300-lesson run differs in duration and memory profile, and those are exactly where + this platform has failed before. *(v1 said this; v2 dropped it.)* +- **Whether FAO can use it.** A question for FAO. *(v1 said this; v2 dropped it.)* +- **Upstream defects.** C-353's nine cells are invisible to every probe here. +- **Skill.** A delivery can be perfect and the forecasts poor. That is + `prereg_model_health_v1.md`, and it is a different claim. +- **The seam has no owner.** This document exists partly to give it one. That is a statement about + the organisation, not about the code, and no probe closes it. +- **`opponent` is now inside the loop.** v2's limitations section named "someone who was not in the + room" as its remedy. That review has happened, and this document is its result — so it is **a + review that occurred, not a property this document has.** The next revision needs a seat that has + not read this one. + +*A limitation that arrives with a scheduled fix has been converted into a plan and no longer +constrains how the suite should be read now, which is the only job this section has. None of the +above is offered with a remedy attached.* + +--- + +## Probe provenance — every v1 probe accounted for + +v2's failure was four silent deletions sitting beside four explicit retentions in the same section. + +| v1 | v3 | | +|---|---|---| +| P1 | retained unchanged | L1 | +| P2 | retained unchanged | L2 | +| P3 | retained unchanged | L2 | +| P7 | retained unchanged | L5 | +| P4 | **restored**, better founded | L5 | +| P5 | **restored**, endpoint corrected | L5 | +| P6 | retained **and** strengthened by B1 | L3 | +| P8 | **restored**, now executable via `tower_point` | ACROSS | +| P9 | **rewritten** — v1's premise was wrong | CONTROL | +| P10 | **moved** to `prereg_model_health_v1.md` as C6 | — | + +--- + +## Acknowledgements + +`opponent` found the structural defect, the four dropped probes, the org-chart diagnosis, and that +I had misremembered v2's size by a factor of 2.2. `views-faoapi` established the refusal mechanism +and the authoritative endpoint. `views-datafactory` supplied C-353, the subset proof, the epoch, +and the cross-end criticism. `views-hydranet` supplied the model-health suite. `views-postprocessing` +corrected three of my claims. `library` supplied the reporting rules and the falsifier-first method. + +**v1 was written alone. Every serious defect in v1 and v2 was found by someone else.** diff --git a/reports/prereg_model_health_v1.md b/reports/prereg_model_health_v1.md new file mode 100644 index 00000000..31f7ef2d --- /dev/null +++ b/reports/prereg_model_health_v1.md @@ -0,0 +1,236 @@ +# Pre-registration — is the forecast itself sound? + +**Written 2026-09-29.** Sibling to `prereg_fao_delivery_v3.md`. Neither is a superset of the other. + +These probes were first drafted inside `prereg_fao_delivery_v2.md` as "Class C and D". `opponent` +established that they do not belong there, and it is right: **every probe here is computed +pod-side from the raw 128-draw cube, and FAO is served MAP and HDI summaries. None of them can +cross the seam.** Keeping them in a delivery document created the impression that "are the numbers +right?" was covered for the delivery, when what was covered is "are the numbers healthy at the +source" — a different claim, with different falsifiers, for a different reader. + +Substantively this is views-hydranet's work. It is recorded here because views-models runs the +models and owns the decision to ship what they produce. + +--- + +## The claim + +> The pooled forecast is a sound object: the rollout has not diverged, the eight models composed +> the way their configs say, the draws are what they claim to be, and the magnitudes are the ones +> an undertrained ensemble should produce. + +**This says nothing about skill.** Whether the forecasts are *good* is evaluation's question, and +it is answered by metrics against outcomes, not by any probe here. A model can be perfectly healthy +and forecast poorly. + +--- + +## R / D, against a stated scope + +**R** = tests a failure mode already observed *anywhere on this platform*. +**D** = not yet observed anywhere on this platform. + +Applied honestly, **this suite is majority R** — the bloom, the translation failure, the timid +body and `M_max` false-flagging are all measured, published and carry issue numbers. v2 labelled +them D and claimed to be "deliberately majority-D", which inverted the truth. A smaller honest +count is worth more than a larger one that cannot be defended. + +--- + +## Reporting rules + +PASS / FAIL / UNRUN. A probe reasoned about but not executed is UNRUN. Never report a bare count +of passes — report per question, so the reader sees what was and was not tested. A FAIL is a FAIL. +A badly designed probe is recorded as a defect in this document and re-run corrected. + +--- + +## The rollout + +**C1 — total mass by horizon, h = 1…36, per model and pooled.** `R` + +**The highest-value number this platform was not computing.** A healthy gated rollout is flat or +mildly decaying across the horizon. **Monotone growth is the bloom** — gate/body decoupling in the +36-step autoregressive rollout (Epic #193, ADR-070), measured at roughly **230× unfixed** and +**12× fixed**. + +All eight configs set `rollout_feedback: 'sample'`, which is the fix. **So we should be in the +fixed regime — measure it, do not assume it.** + +*Falsifier:* the h=36 total is many times the h=1 total. +*Executable:* yes, pod-side, from the pooled cube. + +**C2 — zero-fraction by horizon.** `R` + +Both directions are failures. Rising toward 1.0 is the zero-collapse this repo has fought for +months; falling is the bloom. **Flat is healthy.** + +--- + +## The composition + +**C3 — zero-fraction decomposed by model; exactly three must be all-or-nothing.** `D` + +**The gate is not one mechanism.** Verified in the configs: + +| gate | models | τ | +|---|---|---| +| `soft_gate` — per-draw Bernoulli | blue_stranger, bright_starship, heavy_freighter, pink_pirate, purple_alien | — | +| `threshold_gate` — deterministic mask | violet_visitor | **0.20** | +| | blazing_meteor | **0.16** | +| | bold_comet | **0.14** | + +`compose_samples` draws `torch.bernoulli(gate)` **independently per draw** for soft gates, and +applies `(gate >= τ)` to the whole cell at once for threshold gates. So the three threshold models +contribute **blocks of 16 that are all-zero or all-nonzero**. + +*Falsifier:* the count of all-or-nothing models is not exactly three — a config did not load the +way we believe. + +**Consequence to record, not test:** the pooled zero-fraction is a **mixture of two mechanisms** +and is **not** interpretable as "the ensemble's P(event) = nonzero/128". τ differs across the three, +so *"the ensemble's implied occurrence threshold"* is not a number that exists. A cell just below +0.14 / 0.16 / 0.20 loses 48 draws to exact zero regardless of how large its body is. + +**C4 — draw-block ordering, checked by behaviour rather than metadata.** `D` + +Draws 0–15 are model A, 16–31 model B, and so on. Confirm the all-or-nothing blocks land on the +three threshold models. **A free axis-order check on the draw axis that uses model behaviour +instead of trusting metadata.** + +**C7 — the gate↔body target pairing.** `D` + +`to_cube_samples` slices `gate[..., :n_reg]` and applies gate channel *j* to regression target *j*, +**positionally**. **No validator asserts that `classification_targets[j]` is the gate for +`regression_targets[j]`** — `config_initializer` checks only a count on `pos_weight`. + +It holds today by convention. A config that reorders one list gets sb's body gated by ns's gate: +right shape, right cells, right range, right totals, **wrong pairing, nothing errors.** + +*Check:* assert the two lists are the same targets in the same order, per model, read from the +config the run actually used. +*Ownership:* the validator belongs in views-hydranet; this probe is the interim guard. + +--- + +## The magnitudes + +**C5 — spatial correlation against the last observed month.** `R` + +The h=1 forecast's nonzero cells should overlap the truth month heavily. + +**Why this matters more than it sounds:** this repo measured (M54) that **rolling the recurrent +state by 90 cells moves the forecast by 90 cells with r ≈ 0.90 while skill collapses 48×.** A +translated forecast is cell-count-perfect, total-perfect, and completely wrong. **No structural +probe anywhere can see a translation.** + +**C6 — order of magnitude against truth, with a direction.** `R` + +v2's predecessor said "it must look undertrained", which is not a test because it has no threshold. + +**The threshold: the ensemble should undershoot observed totals by roughly 5–20×.** The body is +documented as timid; `size_ratio` has been measured as low as **0.02**. + +*Falsifier, and note the direction:* **an ensemble that matches observed totals is the alarming +result, not the good one.** At 40 lessons, matching means something composed wrong. + +*Caveat:* the reference magnitude for observed global state-based fatalities must be taken from our +own truth data, not from recollection. + +**C8 — per-model max draw: record, do not gate.** `R` + +**`M_max` false-flags.** A single enormous draw once made a clean field look like "blooms +everywhere" when there was no bloom. Record it; if one model's maximum is orders above the others, +look at that model. **Do not fail on it.** + +*This is the correction to how v1 handled its own anchor cell — see below.* + +--- + +## The ensemble's identity + +**D1 — draw independence across models sharing a seed.** `R` + +`torch_seed`, verified in the configs: pink_pirate **42**, violet_visitor **42**, blue_stranger +**43**, bright_starship **43**, purple_alien 44, bold_comet 45, blazing_meteor 46, heavy_freighter +47. + +The cube generator is seeded from `torch_seed` **and nothing else** — no model identity — and +`to_cube_samples` derives its per-step seed as `base + pass_index*1_000_003 + tt*10_007`. So two +models with the same seed draw from **byte-identical random streams**. + +The pair that matters: **blue_stranger and bright_starship — same seed, both soft-gated, both +D×K = 4×4.** Their Bernoulli masks are `1{u < gate}` computed from the same `u`. + +**Measured 2026-09-29** on calibration draws (`origin_0`, `lr_sb_best`): + +| pair | identical 16-draw mask | per-draw agreement | excess over independence | +|---|---|---|---| +| seed 43 both | **3.42%** | 72.21% (independent: 69.51%) | **+2.70 pp** | +| control, seeds 44 / 47 | 0.93% | 69.67% (independent: 69.68%) | **−0.01 pp** | + +The control is exactly independent. The shared-seed pair shows 3.7× the identical-mask rate and a +real excess. **The mechanism is confirmed; the magnitude is modest** — consistent with the NB +rejection sampler diverging the streams after the first target's draws. + +**So 128 pooled draws are not 128 independent draws**, though far closer than the mechanism alone +would suggest. + +*Falsifier for a future run:* excess over independence materially above the ~3 pp measured here. + +**Three limits of this measurement, stated because they are easy to omit:** + +1. **Calibration data, not forecasting.** The rejection sampler is the mechanism credited for + divergence, and rejection rates are distribution-dependent — so ~3 pp does not transfer to + forecasting draws by argument, only by assumption. +2. **n = 1 control.** One control pair, one shared-seed pair, no error bar. "Materially above + ~3 pp" inherits an undefined *materially* from a single measurement. +3. **Determinism cannot substitute for this.** Re-pooling to byte-identical output proves + *determinism*, not *non-duplication*. That is exactly the gap, and it is why the determinism + result — however satisfying — does not speak to this at all. + +**D1 does not stand alone in its class.** Seed-sharing is one way the eight configs fail to be +eight independent models; config duplication and shared initialisation are others, untested here. + +**And this compounds with C3, which the v2 taxonomy separated.** C3 says the pooled zero-fraction +is a mixture and not a probability. D1 says the effective draw count is inflated. **Both bear on +any credible interval FAO is served**, and neither is visible from the other's section. + +--- + +## The anchor cell v1 misread — a worked example of C8's rule + +v1 recorded a cell with 121 of 128 draws at zero, one draw at 29,536, mean 283.8, as "undertrained +behaviour" and moved on. + +**It is not undertraining. It has the shape of gate/body decoupling.** 121 zeros means ≤7 nonzero; +the threshold models emit in blocks of 16, so none of those 7 came from them — all came from the +five soft models, implying a gate around **0.09**. Meanwhile `log1p(29536) = 10.29` against a +training input range topping out near **11.6**. The body emitted a near-maximum value in a cell +the gate believes is almost certainly empty. + +**But one cell cannot settle it**, and that is the point of C8: `M_max` false-flags, and v1 handed +itself an `M_max` and read it as reassurance. The diagnostic is **C1**, not this. + +--- + +## What this suite cannot do + +- **Skill.** Not tested here, not testable here. +- **Anything after the pod.** Every probe reads the raw cube. The delivery is + `prereg_fao_delivery_v3.md`, and a green run here says nothing about whether FAO received it. +- **The production run.** A green 40-lesson run says the composition is sound at 40 lessons. The + 300-lesson run differs in duration and memory profile, and those are where this platform has + failed before. +- **Failure modes nobody has articulated.** Findings outside these probes are not lesser findings. +- **Upstream data defects.** views-datafactory C-353's nine mislabelled cells are invisible here. + +--- + +## Acknowledgements + +Every probe in this document originates with the **views-hydranet** session, which was asked what +could be wrong with the numbers that a shape check would pass, and answered with six findings, the +measurement that should replace `M_max`, and the seed mechanism. The seed effect was measured here +rather than relayed. `opponent` established that these probes needed their own document. From 6a49f160b4f8353bcb8e3d778d2c5ffe2cdf5c33 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 22:12:42 +0200 Subject: [PATCH 5/6] =?UTF-8?q?docs(review):=20the=20triplicated=20Appwrit?= =?UTF-8?q?e=20client=20=E2=80=94=20eight=20perspectives,=20and=20a=20date?= =?UTF-8?q?d=20deferral?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commissioned after two defects spread through three copies of the same module in one day, one Tier 1 and one the cause of the first-ever FAO delivery failure. THE MEASUREMENTS, since this review exists because nobody had taken them: views-pipeline-core 4 files 3801 lines 27 public defs (2925-line file.py monolith) views-faoapi 10 files 2343 lines 6 public defs (metadata.py 560 lines) views-crafdapi 10 files 2150 lines 6 public defs (metadata.py 560 lines) faoapi and crafdapi are near-identical twins: same line count, 66 differing lines, the leading differences being import ORDER. All three carry the ancestor's comment `# <-- CHANGED from "FOUND" to "FOUND_BY_HASH"` verbatim, and all three carry the hash-only dedup query it documents. VERDICT: one library is correct; do not start it now. The platform is mid-incident with a UN delivery pending, a Tier 1 fix unreleased and two PRs outstanding. Consolidating an Appwrite client across four repos this week would be the second-largest change on the platform while the largest is unfinished. The value of the document is that it makes the deferral EXPLICIT AND DATED. A duplication nobody has decided about spreads; a duplication with a recorded decision and a named trigger does not. Trigger stated: a fourth consumer, or a third defect propagating, or the FAO delivery reaching monthly cadence — whichever comes first. THE STRONGEST EVIDENCE IN THE REVIEW, and it is not about the bugs: faoapi and crafdapi independently converged on the same 6-method surface from pipeline-core's 27. Two teams arriving at the same number is better evidence about what the interface should be than any argument in the document. When consolidation happens, publish the 6. Four failure modes recorded, of which the one nobody is watching is C-B: crafdapi is UNASSESSED. It carries the identical dedup defect and has a test_serving_isolation.py, but faoapi's tripwire was confirmed green today and crafdapi's was assumed. That is the cheapest regret to prevent and the most likely one to be had. Also recorded: C-C, the re-publish remedy triggers the total-failure case on any repo whose dedup is unfixed — the runbook currently instructs operators into the failure. One thing the review deliberately does not decide: whether this client should exist at all. views-appwrite#339 is a standing epic to evict Appwrite from the platform core. If it lands, the duplication consolidates itself, and nobody should build a shared client without checking that epic first. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- ...pert_review_triplicated_appwrite_client.md | 400 ++++++++++++++++++ 1 file changed, 400 insertions(+) create mode 100644 reports/expert_review_triplicated_appwrite_client.md diff --git a/reports/expert_review_triplicated_appwrite_client.md b/reports/expert_review_triplicated_appwrite_client.md new file mode 100644 index 00000000..650048a9 --- /dev/null +++ b/reports/expert_review_triplicated_appwrite_client.md @@ -0,0 +1,400 @@ +# Multi-expert review — the triplicated Appwrite client + +**2026-09-29.** Commissioned after two defects spread through three copies of the same module in +one day, one of them Tier 1 and one of them the cause of the first-ever FAO delivery failure. + +--- + +## 1. System Summary + +Three independent implementations of an Appwrite storage/metadata client exist on this platform. +`views-pipeline-core/modules/appwrite/` is 4 files and 3,801 lines, dominated by a single +2,925-line `file.py` exposing 27 public methods. `views-faoapi/managers/appwrite/` and +`views-crafdapi/managers/appwrite/` are 10 files each — 2,343 and 2,150 lines — exposing 6 public +methods from a `metadata.py` that is **exactly 560 lines in both**, differing in 66 lines whose +leading entries are **import order**. + +All three descend from one ancestor: each carries the comment +`# <-- CHANGED from "FOUND" to "FOUND_BY_HASH"` verbatim. Consumers are split — +`views-postprocessing` reaches pipeline-core's copy through ports +(`unfao/store_port.py:27`, `crafd/store_port.py:27`), while faoapi and crafdapi each use their own. + +Two defects propagated through all three in a single day: content-hash deduplication that reports +success having written nothing (C-155, Tier 1, broke the 2026-09-29 FAO delivery), and +`get_latest_file_id` promising "newest by creation timestamp" while returning `files_list[0]` from +an unordered query (#555, live in a guard since August). **Both are fixed in one copy and present +in the other two.** The platform delivers to two external partners, the UN FAO and CRAF'd. + +--- + +## 2. Expert Reviews + +### Robert C. Martin — SOLID, boundaries, dependency direction + +**Strengths** + +- The **ports in views-postprocessing** (`unfao/store_port.py:27`, `crafd/store_port.py:27`) are a + correct DIP application: the delivery logic depends on a `latest_file_id(filters)` abstraction, + not on Appwrite. This is why #314 could move its query loop into `delivery/findability.py` and + take a `resolve` callable — the seam already existed. +- **faoapi and crafdapi decomposed where pipeline-core did not**: 10 files against 4, 6 public + methods against 27. Whatever else is wrong, the downstream copies applied ISP to a surface the + upstream never split. + +**Weaknesses** + +- **SRP is violated at the file level in pipeline-core.** `file.py` at 2,925 lines holds upload, + download, dedup, metadata search, container provisioning and pagination. It is the file where + both defects live, and its size is why `Query.limit(1)` at line 1190 and the docstring promise at + `datastore.py:517` could contradict each other without anyone noticing. +- **The dependency direction is inverted for two of three consumers.** faoapi and crafdapi do not + depend on an abstraction; they depend on a *copy*. There is no interface to substitute, so LSP + and DIP are not violated so much as unavailable. +- **The 27-method surface is not segregated.** faoapi needs 6. It took 6 by copying rather than by + depending on a narrower interface, which is ISP solved with a text editor. + +**Improvements** + +1. Extract the 6-method surface faoapi and crafdapi actually use into a published interface, and + have all three depend on it. The evidence that 6 is the right number is that two independent + teams arrived at it. +2. Split `file.py` along its existing seams — metadata search, storage, provisioning, dedup. The + defects clustered in one region of a file too large to review as a unit. + +--- + +### Gang of Four — patterns, and patterns misapplied + +**Strengths** + +- **`OperationResult` is a consistent Result pattern** across all three copies, with `success`, + `code` and `data`. It is the reason the dedup defect was diagnosable at all — `FOUND_BY_HASH` + vs `FOUND_BY_NAME` named the branch that failed. +- **The port/adapter pair in views-postprocessing** is a textbook Adapter, and `findability.verify` + taking a `resolve` callable is a clean Strategy that made the rule testable without Appwrite. + +**Weaknesses** + +- **This is Copy-Paste-as-Inheritance.** Three subclasses of an abstract client that was never + written. The `# <-- CHANGED from "FOUND" to "FOUND_BY_HASH"` comment surviving verbatim in all + three is the marker: it documents a change to an ancestor none of them can reference. +- **No Template Method where one is obvious.** All three implement the same + query → paginate → filter → select sequence, and all three got the *select* step wrong in the + same way. That is precisely the shape Template Method exists to prevent — vary the step, share + the sequence. +- **`check_file_exists_by_hash` is a Strategy with one hard-coded strategy.** Identity is decided + by content hash alone; name-identity is a second strategy the caller cannot select, which is why + the fix had to be a new parameter rather than a different policy object. + +**Improvements** + +1. If the three copies persist, at minimum extract the query→paginate→select sequence as a shared + Template so the select step cannot diverge silently again. +2. Make artefact identity an explicit policy (`by_hash`, `by_name`, `by_both`) rather than an + implicit default. #552 hard-codes `by_both`; a partner-visible store may want to state that. + +--- + +### Michael Feathers — testability, seams, legacy characterisation + +**Strengths** + +- **`test_serving_isolation.py` exists in both faoapi and crafdapi** and is the strongest artefact + in this system. It computes the transitive import closure from the route entrypoint and + AST-scans for producer methods, with a positive control asserting the closure reaches >20 + modules and a tripwire test proving the matcher fires on an injected call. That is a seam test + written by someone who had been burned. +- **`_FakeDatastore` in pipeline-core's publisher tests** is a real in-memory double with + `fail_on=` injection, allowing torn-run behaviour to be exercised without Appwrite. + +**Weaknesses** + +- **The three copies have three independent test suites, and none tests the others.** A + characterisation test proving the three behave identically does not exist, so "they have drifted" + is knowable only by diffing source. 66 differing lines in a 560-line file is a number nobody + computed until today. +- **`file.py` at 2,925 lines has no seam at the query boundary.** The dedup defect required + mutating production code to demonstrate, and pipeline-core reported that a substring mutation + passed 40 tests because fixtures were too dissimilar — a characterisation gap, not a coverage gap. +- **The fixtures do not resemble production.** Real filenames share + `rusty_bucket_forecasting_` and differ only in a timestamp; the fixtures differed wildly. Every + name-comparison test was therefore weaker than it appeared. + +**Improvements** + +1. Write **one** characterisation suite, run against all three implementations, asserting identical + behaviour on the shared 6-method surface. It documents the drift and becomes the acceptance test + for any future consolidation. +2. Move fixtures to production-shaped names. The mutation that survived 40 tests would have died. + +--- + +### Michael T. Nygard — production, failure modes, blast radius + +**Strengths** + +- **Manifest-last as a commit marker** (ADR-013 §3.2) is correct stability design: consumers ignore + unmanifested shards, so a killed run leaves no visible state. +- **The fail-safe on ingest** is right — views-faoapi refused the broken run and served nothing + rather than serving something it could not validate. The system failed *closed* toward a partner. +- **`_classify_storage_presence`** checking both metadata and storage is the kind of + belt-and-braces that catches half-writes. + +**Weaknesses** + +- **The blast radius of a fix does not match the blast radius of the defect.** #552 fixes one of + three copies. An operator reading "uploads can no longer report success having written nothing" + will reasonably conclude a Tier 1 class is closed. It is closed in 33% of the code. +- **Two external partners, three code paths, no single point of truth about behaviour.** Which + client performed an upload determines whether it was correct. That is an operational property + nobody can observe from outside. +- **The remedy is the trigger.** Re-publishing is the documented response to a torn run (C-105, + C-22), and a re-publish writes new names over identical bytes — so every artefact hits dedup at + once. The prescribed fix for a failure triggers the total-failure case. This is the single most + dangerous property in the system. + +**Improvements** + +1. Add a version-and-provenance line to every delivery log stating *which client implementation and + version* performed the upload. Until consolidation, that is the only way to answer "was this run + affected". +2. Make the C-105/C-22 re-publish procedure conditional on the dedup fix being present, or the + runbook is instructing operators into the failure. + +--- + +### Martin Kleppmann — data, consistency, distributed correctness + +**Strengths** + +- **Content hashing is recorded in manifests** (`sha256` per shard), so integrity is verifiable + independently of the store's own bookkeeping. +- **The wire contract separates Hop-A (per-`(run,target)`) from Hop-B (per-run)**, which is a + reasonable staged-commit design and gives the postprocessor a place to curate. + +**Weaknesses** + +- **Content-addressed storage is being used as a name-addressed store.** Dedup keys on bytes; + every consumer resolves by name. These are different identity models and the system holds both + without reconciling them. The sidecar — run-independent by construction — makes the collision + *certain*, not probable. +- **"Latest" is defined by list order, and list order is undefined.** `get_latest_file_id` + (`datastore.py:517`) takes `files_list[0]` from a query with no `order_desc`. This is reading a + distributed store and assuming an ordering the store never promised. It has been correct by luck + since August. +- **The metadata document and the storage object can diverge**, and an `update_document` against + the wrong file wrote one run's provenance onto another's — observed on 2026-09-29 against run-0, + still unverified because no route exposes metadata documents. + +**Improvements** + +1. Make identity explicitly `(name, hash)` everywhere. Either is half an identity; the platform has + been using each as if it were whole, in different places. +2. Never rely on result order without an explicit sort. Sort after the pagination walk — it is free, + since the walk already materialises all pages. + +--- + +### John Ousterhout — complexity, deep modules, information leakage + +**Strengths** + +- **The port interface is a deep module**: `latest_file_id(filters) -> id | None` is a tiny + interface over meaningful work. views-postprocessing's delivery logic knows nothing of Appwrite. +- **`findability.verify(consumer_name, legs, objects, resolve)`** is similarly deep — four + parameters, no store types, and its refusals are testable without infrastructure. + +**Weaknesses** + +- **`file.py` is a shallow module of enormous size.** 27 public methods over 2,925 lines, where + callers must know that `check_file_exists_by_hash` means *by hash only*, that `get_latest_file_id` + does not sort, and that `search_files_by_metadata` pages but does not order. **That is information + leakage as a defining feature**: the interface does not carry what the caller must know. +- **The docstring at `datastore.py:517` is worse than absent.** "returns the file ID of the newest + matching file based on creation timestamp" is a false abstraction — it describes a module deeper + than the one that exists, and two consumers built on the description rather than the behaviour. +- **Three copies triple the complexity without tripling capability.** Each is a separate thing to + understand, and understanding one does not transfer, because the drift is unmapped. + +**Improvements** + +1. Fix the false abstraction first, before the duplication: either make `get_latest_file_id` sort, + or rename it to what it does. A misleading name in three copies is worse than duplication. +2. When consolidating, publish the **6-method** surface, not the 27. The narrow surface is the deep + module; the wide one is the leak. + +--- + +### Rich Hickey — simplicity vs ease, complecting + +**Strengths** + +- **The delivery declaration is data** (`un_fao.py:43`, `targets = (...)`), and the publish filter + derives from a mapping rather than restating a rule. Deriving beats duplicating. +- **`OperationResult` separates outcome from value**, which keeps "did it work" and "what is it" + from being complected into a return type. + +**Weaknesses** + +- **Copying was *easy*; three clients is not *simple*.** The platform chose ease three times and now + holds three artefacts that must be reasoned about together and cannot be. +- **Identity is complected with storage optimisation.** Dedup is a *performance* concern; naming is + an *identity* concern. Binding them means a storage optimisation silently changed what an + artefact *is*. That is the actual root of C-155, not the missing comparison. +- **"Latest" complects ordering with retrieval.** `get_latest_file_id` is a query and a policy in + one name, so the policy could rot without the query noticing. + +**Improvements** + +1. Decomplect identity from storage: an artefact is `(name, hash)`. Dedup may then be a pure + optimisation that cannot change identity, which makes the whole class impossible rather than + guarded. +2. Prefer one library that is *simple* over three that were *easy*, but only once the incident is + closed — consolidating mid-incident complects two changes. + +--- + +### Kent Beck — feedback loops, small steps, working software + +**Strengths** + +- **The feedback loop that found both defects was excellent** — a real delivery, a pre-registered + probe suite, five reviewing seats, mutation testing. Two Tier-1-class defects in one day is a + *healthy* discovery rate. +- **#552 and #314 were both improved by review**, and in both cases the author found a deeper defect + after being pushed one level up. That is the loop working. + +**Weaknesses** + +- **The feedback loop does not cross repo boundaries.** Each copy has its own tests, its own CI, its + own green. Nothing runs that would go red because faoapi's copy still has the defect. +- **A big-bang consolidation is exactly the wrong-sized step** for a platform mid-incident with a + UN delivery pending. Three copies, two external partners, and an unfixed Tier 1 is not the moment + to unify a client. +- **"It has been fine since August" is absence of feedback, not evidence.** The ordering bug + produced no signal because the failure mode is a *false alarm* nobody has yet seen. + +**Improvements** + +1. **Smallest useful step first**: one characterisation test, run against all three, that fails + today. It converts "they have drifted" from an assertion into a red bar. +2. Fix the two known defects in all three copies before consolidating anything. Three small changes + with identical content is the step size this platform can afford this week. + +--- + +### The Maintainer — sufficiency verdict + +**What is worth acting on now:** the two defects in the two unfixed copies, and the false docstring. +Those are bounded, high-value, and do not require an architectural decision. + +**What is not worth acting on now:** consolidation. Eight perspectives above make a strong case that +one library is correct, and I agree — but the platform is mid-incident-recovery with a UN delivery +pending, a Tier 1 defect half-fixed, and two releases outstanding. Consolidating an Appwrite client +across four repos this week would be the second-largest change on the platform whilst the largest is +unfinished. + +**Findings I judge not worth acting on:** + +- *Split `file.py` along its seams* (Martin, Ousterhout). Correct, and it will still be correct in + three months. It is a 2,925-line refactor of the module currently carrying an unreleased Tier 1 + fix. Not now. +- *Make identity an explicit policy object* (GoF). This is designing for a second identity model + nobody has asked for. #552's hard-coded `by_both` is right until something needs otherwise. +- *Template Method for the query sequence* (GoF). If consolidation happens, this is moot. If it does + not, it is a shared abstraction across three repos that cannot import each other — the worst of + both. + +**Sufficiency verdict: the review is sufficient to decide, and the decision is "yes to one library, +no to starting now."** The value of this document is that it makes the deferral *explicit and +dated* rather than implicit. A duplication nobody has decided about spreads; a duplication with a +recorded decision and a trigger does not. + +--- + +## 3. Key Disagreements Between Experts + +**D-1 — Consolidate now, or fix-in-place first?** +Hickey and Martin argue the duplication *is* the defect and everything else is symptom management. +Beck and the Maintainer argue that three small identical fixes is the step size available this week, +and that consolidating mid-incident complects two changes. **Unresolved by evidence; resolved by +timing.** Both are right about different horizons. + +**D-2 — Is the false docstring worse than the duplication?** +Ousterhout says a misleading name replicated three times is worse than duplication, because +consumers built on the description rather than the behaviour — two did exactly that. Kleppmann says +the ordering assumption is the real fault regardless of what the docstring says. **Ousterhout is +right about cause, Kleppmann about consequence.** + +**D-3 — Does the port design vindicate or indict the architecture?** +Martin and Ousterhout praise views-postprocessing's ports as correct DIP and a deep module. Feathers +notes the ports exist only where views-postprocessing built them, and that faoapi and crafdapi have +no equivalent — so the good design is one consumer's local virtue, not the platform's. +**The architecture is not vindicated by one consumer doing it properly.** + +--- + +## 4. Failure Mode Analysis + +**C-A — A fix in one copy is recorded as a platform fix.** *Tier 2.* +Trigger: closing #551/#552 or #555 without naming the other copies. Location: the three +implementations. Already observed in the making — this review exists because it nearly happened. + +**C-B — crafdapi is unassessed.** *Tier 2.* +Trigger: the CRAF'd delivery running with a producer path active. crafdapi carries the identical +dedup defect (verbatim comment, hash-only query) and has a `test_serving_isolation.py`, but nobody +has confirmed the tripwire is green or that its producer paths are dead. faoapi's equivalent was +confirmed today; crafdapi's was assumed. + +**C-C — The re-publish remedy triggers the total-failure case.** *Tier 1.* +Trigger: an operator following C-105/C-22 after a torn run, on any repo whose dedup is unfixed. +Every artefact hits dedup at once; N successes, nothing servable. + +**C-D — Drift is unmeasured and unmeasurable in CI.** *Tier 3.* +Trigger: any change to one copy. 66 differing lines in a 560-line file was computed by hand today +and is not computed anywhere else. + +--- + +## 5. Long-Term Regret Test + +**In one year, what will we regret?** + +- **Not consolidating** — moderate regret, slow-accruing. A fourth consumer appears, copies the + nearest client, and inherits whichever defects that copy holds. +- **Consolidating this week** — high regret, fast-accruing. A four-repo refactor landing on top of + an unreleased Tier 1 fix, with a UN delivery pending, is how an incident becomes an outage. +- **Fixing two copies and forgetting the third** — the most likely regret, and the cheapest to + prevent. crafdapi is the one nobody is watching. +- **Not recording the decision** — the highest regret per unit of effort. This review costs an hour; + rediscovering the duplication after the third defect costs another day like today. + +**What will we be glad of?** That the tripwire tests existed. faoapi's `test_serving_isolation.py` +is the reason its copy is inert rather than live, and it was written before anybody needed it. + +--- + +## 6. Engineering Recommendation + +**One library is the correct end state. Do not start it now.** + +**Now (this week, bounded):** + +1. **Fix the two defects in faoapi and crafdapi** — the same changes as #552 and #555, three small + PRs with identical content. Verify crafdapi's isolation tripwire is green rather than assuming it. +2. **Fix the false docstring at `datastore.py:517`** by making it true, not by softening it. +3. **Record the duplication decision** with a named trigger (below), so the deferral is explicit. + +**The trigger for consolidation, stated now so it is not a judgement call later:** a **fourth** +consumer appearing, **or** a third defect propagating through the copies, **or** the FAO delivery +reaching a steady monthly cadence — whichever comes first. Any of those makes the duplication cost +exceed the consolidation risk. + +**When it happens:** publish the **6-method** surface that faoapi and crafdapi independently +converged on — that number is the strongest evidence in this review about what the interface should +be — not pipeline-core's 27. Begin with the characterisation suite run against all three, which is +useful whether or not consolidation follows. + +**What this review does not decide:** whether the Appwrite client should exist at all, given +views-appwrite#339's standing epic to evict Appwrite from the platform core. If that lands, this +consolidates itself. Nobody should build a shared client without checking that epic's status first. From 763c3577449ee1c0d3128cfa5df317cc0462b87a Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 22:25:58 +0200 Subject: [PATCH 6/6] docs(prereg): P8's import corrected, and P11 added because the consumer asked for it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections and one addition from views-faoapi's readiness review. P8 WOULD NOT HAVE IMPORTED. v3 said the probe imports "views-frames' tower_point". There is no such top-level alias. The real chain, confirmed by the repo that owns it: grid_dataset.py:696 _compute_single_map -> :684 _tower_collapse -> forecast/summarize/estimator.py:54,:103 vfs.tower_point(frame) where vfs is `views_frames_summarize`. So the import is `views_frames_summarize.tower_point`, on views-frames 1.10.2 which is what faoapi pins and runs. The whole point of P8 was that importing the consumer's own function removes the "compared the wrong statistic" failure mode by construction; naming a module that does not exist would have reintroduced it as an ImportError at the worst moment. faoapi also corrected itself here, and it is worth recording: it was carrying a note that it ran views-frames 1.3.0 and therefore predated the tower-tip MAP fix. Stale — measured 1.10.2. Answering from memory would have sent me chasing a version mismatch that does not exist. THE DEGENERATE-TIP CAVEAT IS NOW SPECIFIC. v3 said "do not assume the estimator is sound", which is not actionable. The actual degeneracy on this path is at sample_size == 1, where interval bounds collapse onto the point estimate and the served shape is indistinguishable from a genuine HDI (faoapi register C-265). A 128-draw pool is nowhere near it — but if P8 shows zero-width intervals, that is what to check before suspecting the estimator. P11 IS NEW, AND THE CONSUMER ASKED FOR IT. Nothing in views-faoapi asserts a target count: `value_columns` is checked for EMPTINESS only, in the service, the daily monitor and the smoke test. A manifest declaring one target is internally consistent, so every integrity assert passes and FAO receives one violence series labelled normally. So P11 asserts len(value_columns) == 3 and 27 value columns on the grid parquet. This is the one probe in the suite that is strictly better than what the consumer has, and it exists only because views-faoapi stated plainly what it could not catch. That is worth more than any probe I could have derived from the outside. Also from faoapi's review, recorded in the chain notes rather than as probes: newest-wins is sounder than it hedged earlier — manager.py:179-182 sorts in Python by $createdAt after an exhaustive paged fetch, so the unpinned server order cannot bite; and no approval allowlist is gating us in production, established behaviourally rather than assumed, because the 09-29 run reached ingest_failed and that can only happen after selection. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- reports/prereg_fao_delivery_v3.md | 40 +++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/reports/prereg_fao_delivery_v3.md b/reports/prereg_fao_delivery_v3.md index 0770fd45..49349b82 100644 --- a/reports/prereg_fao_delivery_v3.md +++ b/reports/prereg_fao_delivery_v3.md @@ -39,7 +39,7 @@ L1 pod produce and pool P1 L2 Hop-A publish -> production_forecasts (§3.2 manifest per (run,target)) P2 P3 E3 L3 transform un_fao postprocessor, land -> land_gaul P6 B1 B4 B5 L4 Hop-B unfao_bucket (§4.2 manifest per run) E1 E2 -L5 serve faoapi ingest -> /data endpoints P5 P4 P7 B6 +L5 serve faoapi ingest -> /data endpoints P5 P4 P7 P11 B6 ACROSS served values == produced values P8 <- the spine CONTROL nothing outside the declared surface moved P9 ``` @@ -173,6 +173,20 @@ fatality label with no error anywhere.** 36 months beginning at 561. *Falsifier:* 37 (views-models#512's off-by-one reaching the wire), or a range starting before 561. +**P11 — three targets are actually served, not one or two.** `D` + +**views-faoapi has asked for this probe because its side will not catch it.** Nothing there asserts +a target count: `value_columns` is checked for **emptiness only** — in the service, in the daily +monitor *and* in the smoke test. A manifest declaring one target is internally consistent, so every +integrity assert passes and **FAO receives one violence series, labelled normally.** + +*Check:* `len(value_columns) == 3` against `/health`'s `published.forecast.value_columns`, and +**27 value columns** on the grid parquet's schema (3 series x 9 quantities). +*Falsifier:* any count other than 3 and 27. + +This is the one probe in this suite that is **strictly better than what the consumer has**, and it +exists only because views-faoapi said plainly what it could not catch. + **B6 — month boundaries mean the same thing at both ends.** `D` `month_id = (year-1980)*12 + month`, epoch pinned as a module constant; 561 is 2026-09 everywhere. Assert min, max and distinct count map to the intended calendar months at **both** ends. @@ -194,14 +208,26 @@ precision, in a companion file. After the delivery, query the API for those same **v1 could not execute this.** It flagged that FAO is served MAP and HDI summaries rather than raw draws, and never resolved which statistic to compare — so the probe was a wish. -**Resolved.** `views-faoapi/data/handlers/grid_dataset.py::_compute_single_map` delegates to -**views-frames' `tower_point`** estimator. So this probe **imports the same function faoapi uses** -rather than reimplementing the statistic. That removes the "compared the wrong thing" failure mode -*by construction* rather than by care. +**Resolved.** The chain, traced and then confirmed by views-faoapi: + + grid_dataset.py:696 _compute_single_map + -> :684 _tower_collapse + -> forecast/summarize/estimator.py:54,:103 vfs.tower_point(frame) + +where `vfs` is **`views_frames_summarize`**. So this probe imports **`views_frames_summarize.tower_point`** +— not a `views_frames` top-level alias, which does not exist — and calls the identical function on +the identical version (faoapi pins `views-frames>=1.10.2,<2`, installed 1.10.2). That removes the +"compared the wrong thing" failure mode *by construction* rather than by care. + +`estimator.py:4` defines it as *"median of the narrowest canonical HDI floor"* — the definition to +reach for if a difference ever needs explaining. *Falsifier:* values differ beyond floating-point tolerance. -*Carry this caveat:* views-faoapi holds `BUGREPORT_tower_point_degenerate_tip.md`. Do not silently -assume the estimator is sound — a disagreement may be the estimator, not the delivery. +*Carry this caveat, now specific:* the degeneracy observed on this path is at `sample_size == 1`, +where interval bounds collapse onto the point estimate and the served shape is indistinguishable +from a genuine HDI (views-faoapi register **C-265**). A 128-draw pool is nowhere near that, so it +should not arise — **but if P8 ever shows zero-width intervals, check that before suspecting an +estimator mismatch.** **Why this hop has its own section.** Every reviewing session was a repo owner and contributed probes about its own repo. **The seam spans two repos and belongs to neither**, so it was in