Repository navigation
docs(FAO): the pre-registered landing suite, its anchors, the roadmap, and two register entries - #520
Open
Polichinel wants to merge 6 commits into
Open
Polichinel wants to merge 6 commits into
Polichinel wants to merge 6 commits into
Conversation
…, and two register entries The first end-to-end FAO delivery ran today. Six of ten links worked for the first time in this platform's history; the last one refused. This commits what would otherwise exist only in one conversation. **reports/prereg_rusty_bucket_landing_20260929.md** — ten probes with thresholds committed BEFORE the pooling, the publish or the delivery had run, so a disappointing result could not be re-read as a success. It also records the pre-publish baseline (`/provenance/forecast` returning "No forecast prediction files found"), without which nothing afterwards is provable. Results: P1, P2, P3, P7 and P9 PASS; P5 FAIL (refused at ingest); P4, P6, P8, P10 UNRUN. P9 is recorded as a DEFECT IN THE PROBE, not a pass — it assumed only the forecast publish would run, and the postprocessor legitimately rewrites the historical leg too. **..._values.json** — the 25 anchor cells, full sha256 over each draw vector, captured before publishing. Force-added over .gitignore because a pre-registration whose evidence file is not in git is not a pre-registration. These anchors also proved pooling is deterministic: the publish failed once and re-pooled under a new run id, and all 25 came back byte-identical. **reports/fao_delivery_definition_of_done_and_roadmap.md** — six statements defining done that the operator can check without reading code, and six phases to reach them. Written because the operator is a data scientist who said he has no expertise here and needed to know when this is finished. It names explicitly which decisions are his and which must not be handed to him. **Two register entries:** - **C-154 (T3)** — `/workspace` on rented pods silently ignores `chmod`. The command returns success and changes nothing; a credential written there stays mode 666. What saved us was luck of layout: the one credential the runbook names lives on local disk, and nothing says why that matters. - **C-155 (T2)** — an upload can report success having written nothing. Content-hash dedup returns the pre-existing file's id, so the FAO sidecar was never created while the log said "uploaded". It will recur on every delivery, because the sidecar is run-independent by construction. Tier-1-adjacent: `update_document` wrote this run's metadata onto run-0's file, so the only complete run in the partner bucket may carry corrupted provenance. Deduplication note: views-models' C-94 is the datafactory intensive-feature concern and is NOT the upload-success concern that views-faoapi and views-postprocessing call C-94 in their own registers. Register IDs are per-repo; the cross-reference was nearly made to the wrong entry. Filed alongside: #516 (the un_fao environment is unbuildable fresh), #517 (podrun cannot publish, and installing the fix silently reverts the C-151 toolz override), #518 (the RunPod guide's ground rule 5 is wrong — my own text, and it pushed the operator toward an architecture he correctly rejected), #519 (the registry→env bridge does not exist). Cross-repo: views-pipeline-core#551, views-postprocessing#312 and #313. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
…the trigger for total failure views-postprocessing established the worst case and it is not 109 of 110 artefacts. It is 0. `contract/wire/naming.py:19-31` embeds the run id in every filename — shards, sidecar and manifest alike. So a re-publish produces NEW filenames over IDENTICAL bytes. Every one of the 110 objects then walks the content-hash dedup path, the store dutifully creates 110 documents, the run logs 110 successful uploads and exits clean with a success alert, and nothing in the bucket is servable. This platform's documented remedy for a torn run is to re-publish (views-postprocessing C-105, C-22). So the prescribed response to a failed delivery is the trigger for the total-failure case of this defect. Today survived only because the sidecar was a first-of-kind collision rather than a repeat. A re-publish WAS proposed on the day as the fix for the torn run, and was not taken only because we stopped to establish the selection behaviour first. It would have produced a silent total failure. Neither seat could have found this alone. views-postprocessing verified the run-id-in-filename half. This seat had measured, accidentally, that pooling is deterministic — 25/25 pre-registered anchor cells byte-identical across two independent pools, full sha256 over each draw vector — which is what turns "the bytes might match" into "the bytes will match". Tier 1 rather than 2 on the register's own criterion: from the producing side there is no error signal whatsoever. The counterargument is recorded in the entry rather than omitted — the consumer does refuse, so FAO receives nothing rather than something wrong, which is an availability failure rather than served corruption. That is why it was first filed Tier 2. It is outweighed by the remedy being the trigger, and by the operator being told, 110 times, that everything worked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
…uld not v1 was written alone in a morning. Its most serious defects were all found by someone else, and this rewrite is mostly other people's work. WHAT V1 GOT WRONG, structurally rather than in detail: It tested whether the delivery was WELL-FORMED. It could not have told us whether it was CORRECT. Of ten probes, five tested structure, one tested serving, and ZERO tested value correctness, cross-end agreement, or whether an upload actually wrote what it reported — the last of which is what failed. It also modelled the chain as two hops when there are five, conflating the §3.2 per-target Hop-A manifest with the §4.2 per-run Hop-B one; believed /provenance when forecast_serving_state is authoritative; and reported itself as "5 of 10 passed", which treats a file count and a value check as exchangeable units and is a number that leaves the room. THE BIRTH TEST V1 FAILED: of its ten probes I expected roughly ONE to fail when writing them. Writing probes you expect to pass is this method's p-hacking. v2's probes are written falsifier-first — what observation would let this pass while the system is broken — and a probe whose falsifier is hard to state is too weak to keep. WHAT IS NEW, by contributing seat: views-hydranet — all of Class C, which did not exist. Total mass by horizon is the highest-value number we were not computing: monotone growth is the bloom (Epic #193), measured at ~230x unfixed and ~12x fixed. The gate is not one mechanism — 5 soft, 3 threshold at three DIFFERENT taus — so the pooled zero-fraction is a mixture and is not interpretable as P(event). A spatial translation is invisible to every structural probe (M54: rolling state by 90 cells moves the forecast 90 cells at r~0.90 while skill collapses 48x). And v1's own anchor cell was misread as undertraining when it has the shape of gate/body decoupling. views-datafactory — the criticism that reshaped the document: every v1 probe tested a property of one artefact in isolation, and the failures that matter produce artefacts that are internally perfect. Hence Class B. Also C-353: nine delivered cells carry the WRONG administrative unit right now, because the area-majority join ranks polygons in square degrees rather than area above 55 degrees. unmapped_count: 0 is true for all nine — it is a null-check, not a correctness check, and v1 read it as reassurance. library — sections 0, 1 and 2. The pass-rate framing is the most dangerous thing v1 produced because it is the artefact that leaves the room. Pre-registration borrows credibility from clinical trials, where it prevents fishing; our failure mode is failure of imagination, which the form does not address. views-postprocessing and views-faoapi — the chain model, the authoritative endpoint, and three direct corrections to my claims. ONE THING MEASURED HERE RATHER THAN RELAYED: views-hydranet identified that the cube generator is seeded from torch_seed alone, with no model identity, so blue_stranger and bright_starship (both seed 43, both soft-gated) draw from identical streams. Tested on calibration draws: the shared-seed pair shows 3.42% identical 16-draw masks and +2.70pp agreement over independence; a seeds-44/47 control shows 0.93% and -0.01pp. The mechanism is confirmed and the magnitude is modest — consistent with the NB rejection sampler diverging the streams. 128 pooled draws are not 128 independent draws. The determinism result cannot substitute for this: re-pooling to byte-identical output proves determinism, not non-duplication. Section 6 lists what we will certify as clean while knowing it is wrong — the nine mislabelled cells, the frozen c_id across the forecast horizon, and that lookup_version is a digest of an ingestion rather than of a GAUL edition, which makes views-postprocessing#313's "version by GAUL edition" not currently possible. Section 7 states what the suite still cannot do, including that it is written by the seat that ran the failed delivery and inherits that seat's priors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
…r dropped probes restored The `opponent` session — the only reviewing seat that was not in the room — found that v2 is a better document that lost the four probes v1 called its strongest. Verified before accepting: P4 (stem collision), P5 (the served run is ours), P8 (served values == produced values) and P9 (the negative control) were all absent from v2, with no successor, sitting beside four EXPLICIT retentions in the same section. THE STRUCTURAL CAUSE, which is the real finding. v1 was indexed by the CHAIN: coverage could be audited by reading down the hops and seeing an empty one. v2 re-indexed on a TAXONOMY of failure classes — good analysis, bad table of contents, because a taxonomy has no empty cells. A class can look full while a hop is empty. Re-adding the four probes would have kept the structure that made them droppable, so v3 re-indexes on the chain and keeps the classes as tags. Every probe is pinned to a hop; an empty hop is visibly empty. AND A BETTER DIAGNOSIS THAN THE ONE V2 WROTE ABOUT ITSELF. v2 blamed "the seat that ran the failed delivery inherits that seat's priors". Wrong. Every reviewer was a repo owner contributing probes about their own repo. NOBODY OWNS THE SEAM — "what the pod produced == what FAO serves" spans two repos and belongs to neither — so no reviewer had it in scope and the one probe covering it vanished unnoticed by five readers. v2's coverage map was the org chart, which is more dangerous than one seat's bias because it looks like coverage. P8 now has its own section. SPLIT INTO TWO DOCUMENTS. Every one of v2's Class C probes is computed pod-side from the raw 128-draw cube, and FAO is served MAP/HDI summaries — so Class C cannot cross the seam and is not a delivery probe at all. It now lives in prereg_model_health_v1.md. Different claim, different falsifiers, different reader; merged only because both were learned on the same day. P8 IS NOW EXECUTABLE, which it was not in v1. v1 flagged that FAO is served MAP/HDI rather than draws and never resolved which statistic to compare, so the probe was a wish. views-faoapi's _compute_single_map delegates to views-frames' tower_point, so v3 imports the same function faoapi uses rather than reimplementing it — removing the "compared the wrong thing" failure mode by construction rather than by care. P9 is REWRITTEN rather than restored: v1's premise was wrong. It assumed only the forecast publish writes, and the un_fao postprocessor legitimately rewrites the historical leg, so v1 would have failed itself for correct behaviour. v3's control is "nothing outside the DECLARED DELIVERY SURFACE moved", including run-0's metadata document — which is marked UNRUN at registration, because it needs the operator console, rather than discovered UNRUN afterwards. R/D RE-LABELLED against a stated scope. v2 claimed "deliberately majority-D" while labelling as D the bloom, M54's translation failure, the timid body and the seed effect — all measured and published. Under the stated scope the delivery suite is majority R and says so. LIMITATIONS: v2 dropped three concrete ones v1 had — durability, the production run differing, whether FAO can use it — for more sophisticated but operationally weaker ones. All three restored, plus two that were missing: the seam has no owner, and `opponent` is now INSIDE the loop, so its review is an event that happened rather than a property this document has. Verified by the document's own mechanism before committing, and it caught two of my own defects: a grouped provenance row (P1/P2/P3/P7 on one line) that was not auditable, which is this document's own thesis applied to itself; and P5 having lost the `created_at` field name that makes it checkable. Both fixed. All ten v1 probes now individually accounted for as restored, superseded-by-a-named-successor, or rewritten-with-a-reason. Measured rather than remembered, since I told opponent v2 was "three times" v1 and it was 1.37x: v1 11,917 chars, v2 17,809, v3 16,698, model-health 11,472. v3 is SMALLER than v2 while restoring four probes, because Class C moved out. v1 and v2 retained unedited. A pre-registration that can be rewritten after the fact is not one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
…nd a dated deferral Commissioned after two defects spread through three copies of the same module in one day, one Tier 1 and one the cause of the first-ever FAO delivery failure. THE MEASUREMENTS, since this review exists because nobody had taken them: views-pipeline-core 4 files 3801 lines 27 public defs (2925-line file.py monolith) views-faoapi 10 files 2343 lines 6 public defs (metadata.py 560 lines) views-crafdapi 10 files 2150 lines 6 public defs (metadata.py 560 lines) faoapi and crafdapi are near-identical twins: same line count, 66 differing lines, the leading differences being import ORDER. All three carry the ancestor's comment `# <-- CHANGED from "FOUND" to "FOUND_BY_HASH"` verbatim, and all three carry the hash-only dedup query it documents. VERDICT: one library is correct; do not start it now. The platform is mid-incident with a UN delivery pending, a Tier 1 fix unreleased and two PRs outstanding. Consolidating an Appwrite client across four repos this week would be the second-largest change on the platform while the largest is unfinished. The value of the document is that it makes the deferral EXPLICIT AND DATED. A duplication nobody has decided about spreads; a duplication with a recorded decision and a named trigger does not. Trigger stated: a fourth consumer, or a third defect propagating, or the FAO delivery reaching monthly cadence — whichever comes first. THE STRONGEST EVIDENCE IN THE REVIEW, and it is not about the bugs: faoapi and crafdapi independently converged on the same 6-method surface from pipeline-core's 27. Two teams arriving at the same number is better evidence about what the interface should be than any argument in the document. When consolidation happens, publish the 6. Four failure modes recorded, of which the one nobody is watching is C-B: crafdapi is UNASSESSED. It carries the identical dedup defect and has a test_serving_isolation.py, but faoapi's tripwire was confirmed green today and crafdapi's was assumed. That is the cheapest regret to prevent and the most likely one to be had. Also recorded: C-C, the re-publish remedy triggers the total-failure case on any repo whose dedup is unfixed — the runbook currently instructs operators into the failure. One thing the review deliberately does not decide: whether this client should exist at all. views-appwrite#339 is a standing epic to evict Appwrite from the platform core. If it lands, the duplication consolidates itself, and nobody should build a shared client without checking that epic first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
…er asked for it
Two corrections and one addition from views-faoapi's readiness review.
P8 WOULD NOT HAVE IMPORTED. v3 said the probe imports "views-frames' tower_point". There
is no such top-level alias. The real chain, confirmed by the repo that owns it:
grid_dataset.py:696 _compute_single_map
-> :684 _tower_collapse
-> forecast/summarize/estimator.py:54,:103 vfs.tower_point(frame)
where vfs is `views_frames_summarize`. So the import is
`views_frames_summarize.tower_point`, on views-frames 1.10.2 which is what faoapi pins and
runs. The whole point of P8 was that importing the consumer's own function removes the
"compared the wrong statistic" failure mode by construction; naming a module that does not
exist would have reintroduced it as an ImportError at the worst moment.
faoapi also corrected itself here, and it is worth recording: it was carrying a note that
it ran views-frames 1.3.0 and therefore predated the tower-tip MAP fix. Stale — measured
1.10.2. Answering from memory would have sent me chasing a version mismatch that does not
exist.
THE DEGENERATE-TIP CAVEAT IS NOW SPECIFIC. v3 said "do not assume the estimator is sound",
which is not actionable. The actual degeneracy on this path is at sample_size == 1, where
interval bounds collapse onto the point estimate and the served shape is indistinguishable
from a genuine HDI (faoapi register C-265). A 128-draw pool is nowhere near it — but if P8
shows zero-width intervals, that is what to check before suspecting the estimator.
P11 IS NEW, AND THE CONSUMER ASKED FOR IT. Nothing in views-faoapi asserts a target count:
`value_columns` is checked for EMPTINESS only, in the service, the daily monitor and the
smoke test. A manifest declaring one target is internally consistent, so every integrity
assert passes and FAO receives one violence series labelled normally. So P11 asserts
len(value_columns) == 3 and 27 value columns on the grid parquet.
This is the one probe in the suite that is strictly better than what the consumer has, and
it exists only because views-faoapi stated plainly what it could not catch. That is worth
more than any probe I could have derived from the outside.
Also from faoapi's review, recorded in the chain notes rather than as probes: newest-wins
is sounder than it hedged earlier — manager.py:179-182 sorts in Python by $createdAt after
an exhaustive paged fetch, so the unpinned server order cannot bite; and no approval
allowlist is gating us in production, established behaviourally rather than assumed,
because the 09-29 run reached ingest_failed and that can only happen after selection.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
Polichinel
added a commit
that referenced
this pull request
Sep 29, 2026
The draft said "Registered C-155, Tier 1" with no repository named. The views-pipeline-core
seat read it as pointing at THEIR C-155 — an unrelated Tier 4 documentation-drift row — and
reported the document as citing the wrong entry.
Verified both rather than taking either reading on trust, which is the correction this
session has already needed twice today:
views-models C-155 Tier 1, "an upload can report success having written nothing:
content-hash dedup returns..." — real, but on the UNMERGED #520
branch, which is why it is absent from development
views-pipeline-core C-335 Tier 1, registered 2026-09-29, same defect in their register
(line 820); C-336 beside it is the two-guards-masking finding
So the citation was not wrong, it was AMBIGUOUS — and ambiguity in a cross-repo document is
indistinguishable from wrong to the reader who has to resolve it. Register IDs are per-repo
here and collide constantly; this register already records that views-models' C-94 is a
different concern from views-postprocessing's C-94.
Both entries now named with their repository, and the §9 footnote too. A short note in §3.1
records why, since the failure took hours to surface and will recur otherwise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first end-to-end FAO delivery ran on 2026-09-29. Six of ten links worked for the first time in this platform's history; the last one refused. This PR commits what would otherwise exist only in one conversation.
No code changes. Documents and register entries only.
What ran, and what it found
The refusal is correct behaviour: FAO declined rather than serving something it could not validate. Cost of learning this: ~$5 and one day, against a $24 production run.
The pre-registration
reports/prereg_rusty_bucket_landing_20260929.md— ten probes with thresholds and falsifiers committed before the pooling, the publish or the delivery had run, so a disappointing outcome could not be re-read afterwards as a success.It also records the pre-publish baseline —
/provenance/forecastreturning "No forecast prediction files found" — without which nothing measured afterwards is provable.P9 is recorded as a defect in the probe, not a pass. I wrote it assuming only the forecast publish would run; the postprocessor legitimately rewrites the historical leg too. The reporting rules fixed in advance require saying that rather than quietly counting it.
The anchors, and an accident worth keeping
..._values.json— 25 cells with full sha256 over each draw vector, captured before publishing. Force-added over.gitignore, because a pre-registration whose evidence file is not in git is not a pre-registration.These also proved something unplanned: the publish failed once and re-pooled under a new run id, and all 25 anchors came back byte-identical. Pooling is deterministic. That both validated the anchors for the published run and — as I argued to the
postprocessingsession — means a re-run produces byte-identical shards, so today's sidecar collision is the general case arriving early rather than a special case.The roadmap
reports/fao_delivery_definition_of_done_and_roadmap.md— six statements defining done that the operator can check without reading code, and six phases to reach them.Written because the operator is a data scientist who said plainly he has no expertise here. It names explicitly which decisions are his (which machine publishes, the key, priority) and which must not be handed to him (repo ownership, fix design, contract restructuring).
Register
/workspaceon rented pods silently ignoreschmod. Returns success, changes nothing; the file was observed at666after a successfulchmod 600. What saved us was luck of layout — the one credential the runbook names sits on local disk, and nothing says why that matters.uploaded. It will recur on every delivery, because the sidecar is run-independent by construction. Tier-1-adjacent:update_documentwrote this run's metadata onto run-0's file.Deduplication note worth reading: views-models' C-94 is the datafactory intensive-feature concern and is not the upload-success concern that views-faoapi and views-postprocessing call C-94 in their registers. Register IDs are per-repo. The cross-reference was nearly made to the wrong entry.
Corrections I accepted from other sessions
Recorded because the review genuinely changed the content, not just polished it:
postprocessingcorrected my xarray diagnosis. I wrote "floats unpinned"; it is capped (views-datafactory/pyproject.toml:51,xarray>=2024.1,<2026) and 2025.12.0 is inside the cap. A fresh un_fao environment is unbuildable: datafactory's xarray cap permits a release needing pandas>=2.1, the viewser chain caps pandas below 2 #516 is retitled and rewritten. It also showed my "sidecar must land before manifest" note was a non-finding — the ordering is already implemented and already in ADR-013 — and that adding the sidecar to the findability dict would not have caught this, because the lookup key itself is wrong.faoapiestablished that the refusal happens at artifact resolution before any shard is read, confirmed our coverage curation is correct (64742,land_gaul,unmapped_count: 0— the failure I had pre-registered as most likely did not happen), and argued the contract is the deeper problem.pipeline-coreverified the dedup mechanism in source and found the asymmetry that makes it a defect rather than a design choice.Companion issues
views-models #516, #517, #518, #519 · views-pipeline-core#551 · views-postprocessing#312, #313
🤖 Generated with Claude Code
https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K