From 098939a71d63d87780a566d023d552367729a4ce Mon Sep 17 00:00:00 2001 From: Polichinl Date: Fri, 31 Jul 2026 04:48:48 +0200 Subject: [PATCH] test(gaul): C-43 forward-check as a standing guard + fail-loud lookup build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-011 swapped runtime geopandas mapping for a precomputed lookup, and the plan's required old-vs-new value diff (#20 Stage 2 / #23) was signed off without being produced. The comparator was then deleted in PR #42, so the backward check is unrecoverable — and run-0 shipped 64,742 cells to FAO on 2026-07-27 with no gate anywhere asserting that a *present* GAUL value was the *right* one. The check that remains available runs forwards, against the producer. Run this session, zero mismatches on all four links: gid->lat/lon formula vs PRIO-GRID priogrid_cell.dbf, all 259,200 cells -> max abs error 0.00e+00 lookup values vs the 7 datafactory gaul_admin parquets -> 0 mismatches / 64,742 cells lookup gid set vs land_gaul_pgids.json -> exactly equal, all unique delivered run-0 sidecar bytes vs the lookup -> 0 mismatches, sha256 matches the manifest The leading hypothesis going in — a flipped or off-by-one coordinate formula putting wrong-but-non-null coordinates on every cell, invisible to every gate — was falsified. The formula is exact across the entire global grid. This commit turns that session result into a standing guarantee. 18 tests split by dependency: always-on checks on the committed artifact (key uniqueness, no nulls, no -1 sentinels, region count, coordinate formula against a committed 219-cell PRIO-GRID ground-truth sample) and skipif checks against the views-datafactory sibling (all 7 columns, full 259,200-cell formula sweep). build_gaul_lookup.py: the three bare `assert`s become explicit LookupBuildError raises (python -O silently strips asserts, writing an unvalidated artifact that looks identical), plus a new index-uniqueness guard. Rebuild verified value-identical to the committed artifact. Two entries I registered earlier today are corrected here on mutation evidence rather than left overstated: - C-59 tier 2 -> 3. A duplicate cannot reach the artifact on the production path: Index.intersection de-duplicates first. The guard is reachable only via --region all. Kept anyway — that de-duplication is accidental pandas behaviour, not a declared guard, and silently absorbing upstream duplication hides a producer defect. - C-61 exposure corrected. The primary protection against -1 is the completeness filter, which is plain code and survives -O; the strippable-assert argument never applied to the -1 case. The real defect was stating invariants in a form the interpreter can delete. C-43 scope stated explicitly and deliberately: this proves transcription fidelity, not assignment correctness. If views-datafactory's area-majority join puts a cell in the wrong country, every test here still passes. That is filed as views-datafactory#387 — it had been resolved here on 2026-06-24 as "tracked there" and was never actually opened there, leaving it untracked platform-wide for five weeks. Register: 61 concerns, 24 open. Cluster K added. ruff clean; 296 passed (5 known local pyarrow byte-parity failures, CI authoritative). Co-Authored-By: Claude Fable 5 --- reports/technical_risk_register.md | 111 ++- scripts/build_gaul_lookup.py | 32 +- .../priogrid_centres_sample.json | 883 ++++++++++++++++++ tests/test_gaul_lookup_fidelity.py | 386 ++++++++ 4 files changed, 1407 insertions(+), 5 deletions(-) create mode 100644 tests/fixtures/priogrid_geometry/priogrid_centres_sample.json create mode 100644 tests/test_gaul_lookup_fidelity.py diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 78c2661..59a17b7 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -5,8 +5,8 @@ | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-07-31 | -| Total Concerns | 58 | -| Open Concerns | 21 | +| Total Concerns | 61 | +| Open Concerns | 24 | | Resolved Concerns | 37 | --- @@ -58,6 +58,13 @@ covered a single open entry (see Historical clusters below). **Resolution scope:** Partial (process, not code). **Newly acute 2026-07-31:** every entry here was written conditionally — "*if* wrong data ever reaches FAO." Run-0 shipped 108 arrow shards, a sidecar, a manifest and 28.3M historical rows to `unfao_bucket`, and its integrity verification is still open. The conditional is spent. +### Cluster K: The committed lookup artifact is trusted but unverified in CI +**Root cause:** `views_postprocessing/data/gaul_lookup.parquet` is a 64,742-row binary that the enricher, the historical builder and the wire sidecar all treat as ground truth — yet no test in the suite checks it against views-datafactory, and the build script's own guarantees are enforced by strippable `assert`s. Everything downstream verifies *presence* (nulls, counts, gid-set equality); nothing verified *values* until the 2026-07-31 forward-check. +**Entries:** C-59, C-61 (build-time guarantees unenforced), C-60 (provenance stamp can silently degrade), C-46 (the one cross-repo check is CI-skipped on a hardcoded path), C-43 (the value-correctness debt this cluster's fix discharges) +**Highest tier:** 3 (C-59, C-60, C-61 — C-59 recalibrated 2→3 on 2026-07-31 mutation evidence) +**Fix strategy:** one test file — `tests/test_gaul_lookup_fidelity.py` — split into an always-on half (gid uniqueness, region-set equality, coordinate formula against a committed ground-truth sample, no nulls, no `-1` codes) and a `skipif`-gated half comparing all 7 GAUL columns against the datafactory sibling. Plus two one-line hardenings in `scripts/build_gaul_lookup.py`: assert index uniqueness, and convert the bare `assert`s to explicit raises. +**Resolution scope:** Full for C-59/C-61/C-43's residual guard; partial for C-60 (needs the flat declared `lookup_version` key) and C-46 (needs the hardcoded path removed). + ### Historical clusters (mapper era — all resolved or moot) Clusters **A** (cache architecture never unified), **B** (silent error-hiding @@ -388,6 +395,29 @@ This is the most important consequence of the run-0 cluster (Cluster H). Run-0 d **Recommended action (unchanged, now overdue rather than pre-emptive):** forward-check a sample of delivered `land_gaul` cell assignments directly against the datafactory GAUL parquet — cheap, and it is the mitigation this entry proposed from the start. Folds naturally into #131 q1 (run-0 delivery-integrity verification). +--- + +**TRANSCRIPTION FIDELITY DISCHARGED 2026-07-31 (`expert-code-review`) — the forward-check was run, offline, against committed artifacts. Four checks, zero mismatches:** + +| Link in the chain | Ground truth | Result | +|---|---|---| +| Coordinate formula (`gaul_schema.xcoord`/`ycoord`) | views-datafactory `data/raw/priogrid/shapefile/priogrid_cell.dbf` — **all 259,200 cells** | **max abs error 0.00e+00**, 0 mismatches | +| Lookup values, all 7 GAUL columns | the 7 `data/raw/gaul_admin/*.parquet` | **0 mismatches** across 64,742 cells | +| Lookup gid set + key uniqueness | `src/datafactory_query/land_gaul_pgids.json` | **exactly equal**; 64,742 unique of 64,742 | +| **The delivered run-0 sidecar** (`rusty_bucket_forecasting_20260727_095355__sidecar.parquet` — the real bytes on FAO's shelf) | the lookup | **0 mismatches** on all 9 columns; SHA-256 matches the manifest's declaration | + +The chain producer → lookup → delivered bytes is verified end to end. Note the leading hypothesis going in — that the gid→lat/lon formula might be flipped or off-by-one, producing wrong-but-non-null coordinates on *every* cell, invisible to every existing gate — was **falsified**: the formula is exact for the entire global grid. + +**SCOPE — what this does and does not prove** (the Kleppmann-vs-Nygard split in the 2026-07-31 review, adjudicated to *both, scoped*)**.** It proves **transcription fidelity**: this repo faithfully carries the producer's area-majority GAUL assignment through to the partner. It does **not** prove **assignment correctness** — if views-datafactory's area-majority join puts a cell in the wrong country, every check above still passes and FAO still receives a confidently wrong label. That is a separate concern belonging to **views-datafactory** — the degree-based (square-degree) area math its area-majority join uses, which distorts by up to ~2× at 60°N and could flip the winning polygon for high-latitude border cells now that the region is global. This distinction must survive retelling: C-43 was registered as *a lost old-vs-new cross-check inside this repo*, and that is what has been discharged. + +**⚠ CORRECTION, same day (2026-07-31).** This paragraph originally asserted the upstream half was *"a separate, already-registered concern (C-08, relocated to views-datafactory)."* **That was false and is corrected here.** Verified by direct inspection: views-datafactory's register carries 32 concerns and mentions "area-majority" nine times, but has **no entry** for the degree-based area calculation. C-08 was resolved *here* on 2026-06-24 with the note *"Tracked there, not here"* — and nobody ever opened it there. **The concern has been untracked platform-wide since that date**, and run-0 shipped the affected high-latitude cells to FAO on 2026-07-27. + +Filed upstream as **views-platform/views-datafactory#387** so it is tracked where the code and the geopandas toolchain actually live. This repo cannot verify it: the forward-check above confirms faithful *transcription* of the producer's answer and is structurally incapable of judging whether that answer is right. + +This is a textbook instance of **C-42**'s registered hazard (acting on a mis-stated cross-repo state) and of **Cluster I** — and it was reproduced *while writing the very paragraph describing it*. Concrete lesson for the "relocated" convention added to the Register Conventions this same day: **relocation is not complete until the destination issue or entry exists and is cited by number.** A relocation note naming only a repo is an assumption, not a handoff. + +**Residual (why this entry stays open):** the verification was a one-off session result, not a standing guarantee. Nothing in CI re-runs it, so a future lookup rebuild against a wrong or stale datafactory would ship silently exactly as before. **C-43 closes when `tests/test_gaul_lookup_fidelity.py` is committed and green** — the entry should then cite the test, not the session. Tracked as **Cluster K**; the same test discharges C-59 and C-61. + See also C-03 (the sibling enrich→validate test-coverage gap), C-22 (no post-delivery correction/recall process — **now acute: the consequence path is live**), C-39 / C-31 / C-23 (the resolved mapper-deletion cluster this emerged from), C-30 (coverage — discharged by the same run that left this standing), C-32 / C-34 (RESOLVED — the go-global scale risks that fired cleanly), D-08 (the swap-to-lookup-first decision whose verification debt this is), #131 (run-0 delivery-integrity verification). --- @@ -490,6 +520,74 @@ Cross-refs: C-57 (registry drift — the most likely way a coordinate goes wrong --- +### C-59: The GAUL lookup build asserts no key uniqueness — a duplicate gid silently inflates the legacy delivery + +| Field | Value | +|-------|-------| +| ID | C-59 | +| Tier | 3 — **recalibrated from 2 the same day, see the correction below.** A duplicated key would multiply rows through the legacy pandas merge invisibly to every gate, but reaching the artifact requires `--region all`: the production `--region land_gaul` path de-duplicates first. Unguarded fragility on a non-default code path, not present corruption. | +| Source | `expert-code-review` (2026-07-31) — Kleppmann lens; verified empirically in the same pass | +| Trigger | When views-datafactory regenerates the `gaul_admin` parquets, or when `build_gaul_lookup.py` is re-run against a new datafactory version — verify the resulting lookup index is unique before committing the artifact; nothing checks it today | +| Location | `scripts/build_gaul_lookup.py:146-154` (the invariant block, which checks nulls and `-1` but never uniqueness); consumed at `views_postprocessing/unfao/enrichment.py:117` (pandas left-merge — the inflating path), `unfao/historical.py:60` and `unfao/wire/sidecar.py:56` (deterministic-pick paths) | + +`build(...)` sets `df.index = df.index.astype("int64")`, names it `priogrid_gid`, sorts, and then asserts only that no nulls and no `-1` sentinels survive. It never asserts `df.index.is_unique`. The seven source parquets are joined via `pd.DataFrame({...})` over gid-indexed Series (`build_gaul_lookup.py:59-73`), so uniqueness is inherited from upstream data rather than enforced here. + +Downstream, `GaulLookupEnricher.enrich_dataframe_with_pg_info` does `base.merge(self._lookup, left_on=pg_id_col, right_index=True, how="left")`. A duplicated key produces **N rows per affected cell**. The delivery then carries more rows than cells, with every metadata value present and correct — invisible to the null gate, invisible to the distinct-cell coverage gate, and invisible to the `country_iso_a3` proxy at `enrichment.py:122`. + +**⚠ TIER RECALIBRATED 2 → 3, same day (2026-07-31), on empirical evidence.** Registering this at Tier 2 assumed a duplicate could reach the committed artifact through the normal build. Mutation-testing the builder showed it cannot, on the production path: `build()` applies `src.loc[src.index.intersection(sorted(region_gids))]` (`build_gaul_lookup.py:114-116`), and pandas' `Index.intersection` **de-duplicates**, so an injected duplicate is silently removed before the invariant block ever sees it. The guard is reachable only with `--region all`, which bypasses that filter — verified: it raises there, and raises under `python -O` too. + +Two consequences, both kept: the explicit raise still earns its place, because the de-duplication is an *accidental pandas behaviour* rather than a declared guard (and silently absorbing upstream duplication is itself undesirable — it hides a producer defect); and the tier drops to 3, because the realistic exposure is a non-default flag, not routine regeneration. *Recalibrated during the same session that registered it — the original Tier 2 rationale was written from code reading before the mutation test was run.* + +**Mitigation — landed 2026-07-31:** explicit `LookupBuildError` on a non-unique index in `build_gaul_lookup.py` (not `assert`, per C-61), pinned by `tests/test_gaul_lookup_fidelity.py::test_builder_rejects_a_duplicate_gid`, plus a standing uniqueness check on the committed artifact (`test_lookup_key_is_unique`). + +Cross-refs: C-43 (the value-correctness debt this shares a fix with), C-61 (the same invariant block's strippable asserts), C-30 (the distinct-cell coverage gate that cannot see this), C-40 (the legacy pandas path whose deletion would remove the inflating consumer), **Cluster K**. + +--- + +### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except + +| Field | Value | +|-------|-------| +| ID | C-60 | +| Tier | 3 | +| Source | `expert-code-review` (2026-07-31) — Ousterhout lens (information leakage / silent degradation) | +| Trigger | When views-datafactory renames or restructures its ingestion-ledger entries (`dataset` key, `content_digest` field, or the `land_gaul_region` entry name) — verify `lookup_version` still resolves to a real value rather than the string `"unknown"`; nothing fails if it does not | +| Location | `views_postprocessing/unfao/enrichment.py:61-79` (`_read_version`); written at `scripts/build_gaul_lookup.py:89-107` (`_provenance`) and `:163-164`; consumed as the C-15 provenance field via the manager's delivery description | + +`_read_version` reconstructs the stamp by traversing three levels of the producer's schema — parquet metadata → `source_provenance` JSON → `land_gaul_region` → `content_digest` — and wraps the traversal in `except (ValueError, AttributeError): pass`, returning `"unknown"` when anything along the path is absent. This is a **declare-don't-infer violation at the consumer**: the value that ties a delivered artifact to the exact lookup build (C-15's traceability provenance) can silently become a placeholder, and no gate notices. + +No wrong data results — this is a traceability failure, not a correctness one → **Tier 3**. But it defeats the specific question C-15 exists to answer *after* a suspect delivery ("which lookup produced this?"), and C-22 has no correction procedure that could compensate. + +**Mitigation:** have `build_gaul_lookup.py` write a **flat, declared `lookup_version` key** into the parquet metadata, and have `_read_version` read that one key and **raise** if absent. The consumer stops knowing the producer's nested ledger schema, and the stamp stops being able to vanish quietly. Separately worth stamping `lookup_version` into the sidecar's own parquet metadata so a delivered artifact is self-describing without the store document. + +Cross-refs: C-15 (the provenance this field serves), C-22 (the recall process that would need it), C-57 (the same class — a cross-repo fact this repo reads without a way to detect drift), **Cluster K**. + +--- + +### C-61: The lookup build's hard invariants are bare `assert`s — stripped under `python -O`, and `-1` sentinels are caught nowhere else + +| Field | Value | +|-------|-------| +| ID | C-61 | +| Tier | 3 | +| Source | `expert-code-review` (2026-07-31) — Feathers lens | +| Trigger | When `build_gaul_lookup.py` is run under `python -O` (or from a wheel/CI step that sets `PYTHONOPTIMIZE`), or when the build is wrapped in any tooling that optimizes bytecode — verify the invariant block still executed; a stripped run writes an unvalidated lookup that looks identical | +| Location | `scripts/build_gaul_lookup.py:152-154` (`assert df.isna().sum().sum() == 0`, `assert (df[c] != -1).all()`) | + +The builder's docstring and the enricher both rely on the lookup being "clean by construction" — no nulls, no `-1` sentinels. That guarantee is enforced by three bare `assert` statements, which Python removes entirely under `-O`. + +The asymmetry the original registration leaned on: **nulls have a downstream backstop** (`_validate`, `historical.assert_metadata_complete`) but **`-1` codes have none** — `-1` is non-null, so it would pass every delivery gate, which is the resolved **C-35** defect (invalid country codes shipped to FAO for Somaliland cells) returning through a different door. + +**⚠ EXPOSURE CORRECTED, same day (2026-07-31), on empirical evidence.** That framing overstated the risk. Mutation-testing the builder showed the **primary protection against `-1` is not the `assert` at all** — it is the completeness filter at `build_gaul_lookup.py:125-131` (`complete &= df[c].notna() & (df[c] != -1)`), which drops sentinel rows outright. That filter is **plain code, untouched by `python -O`**, so the strippable-assert exposure never applied to the `-1` case. In practice the `assert` was unreachable: no ordinary input can get past the filter to reach it. + +What remains true, and why the entry stays open at Tier 3: the invariant block was the only *explicit statement* of "this artifact is clean," it was strippable, and the same block also carried the null and (now) uniqueness checks where the argument does bite. Stating invariants in a form the interpreter can delete is the defect; the `-1` severity was not. + +**Mitigation — landed 2026-07-31:** all three invariants converted from bare `assert` to explicit `LookupBuildError` raises with diagnostic messages. The `-1` raise is retained deliberately as a backstop should the filter ever change, and is **deliberately left untested** — reaching it requires stubbing pandas internals, and a test that fragile is worse than the invariant it guards. What *is* pinned is the behaviour that actually protects the partner: `tests/test_gaul_lookup_fidelity.py::test_a_sentinel_code_is_dropped_rather_than_shipped` (the cell is excluded, so it later fails loud as *absent* rather than shipping as wrong-but-non-null) and `test_lookup_carries_no_sentinel_codes` on the committed artifact. + +Cross-refs: C-35 (RESOLVED — the `-1` defect class this guards against), C-59 (same invariant block), C-43 (the fidelity test that would catch a bad artifact regardless), **Cluster K**. + +--- + ## Disagreements ### D-12: Post-Run-0 infrastructure & naming intents — repo rename, internal-store transport, compute co-location @@ -527,7 +625,14 @@ The pandas→views-frames migration (epic #85) deliberately swaps each seam by a **Decision: A**, consistent with the maintainer's WET-before-DRY rule and the "migration not coexistence" reality. **Re-open trigger (the one acute case):** S6 (#91) introduces a temporary **dual-write** (legacy parquet + arrow sample-frame) for parity during the faoapi cutover — *if that coexistence proves long-lived* (rather than a brief cutover window), a small abstraction may then earn its place; revisit only then. Until then, concrete-and-delete stands. -**SETTLED 2026-07-31 (review-rr — the re-open trigger resolved without firing).** The sole re-open condition was a long-lived S6/#91 dual-write. **#91 is CLOSED**, and the contract path shipped frame-native end-to-end (PRs #115–#129, wire delivered in run-0) without a durable dual-representation window. Position A is vindicated by events: concrete siblings + delete is what actually happened, the pandas path is confined to the legacy branch pending its named post-run-0 deletion, and no polymorphic representation port was ever needed. No live tension remains — this entry is kept as decision provenance for the remaining seam work (#89, #90) rather than as an open question. +**RE-INSTANTIATED 2026-07-31 (`expert-code-review`) — same tension, new subject; merged here rather than registered as its own disagreement entry.** The review surfaced the identical disagreement one level down, about the **keyed gather** rather than the representation seam. "Given gids, fetch their lookup rows" now exists **three times, all simultaneously live**: a pandas left-merge (`unfao/enrichment.py:117`), an argsort/searchsorted gather (`unfao/historical.py:57-68`), and an isin/filter/sort (`unfao/wire/sidecar.py:47-57`). + +- **Position A (GoF, Hickey) — unify now.** Three implementations of one join is a correctness surface, not a migration artifact. They differ in failure semantics: two raise in place naming the absent gids; the pandas one yields NaN and defers to a downstream gate. That divergence is invisible at the call sites. +- **Position B (Feathers, Beck) — defer to #89.** The differing failure semantics are *deliberate*, each implementation is separately tested, and unification means changing three call sites while the legacy branch is still live. The duplication is scheduled to die with that branch. + +**Adjudication: B — but conditionally, and the condition is now named.** B is only correct *if the legacy forecast branch is actually deleted*. It currently has no scheduled deletion PR — only a "named post-run-0 follow-up" (C-40). If it lingers, three-way duplication becomes the permanent shape and Position A wins retroactively. **Re-open trigger: if the legacy forecast branch still exists when #89 (numpy/pyarrow keyed gather) lands, unify all three consumers onto one fail-loud gather primitive as part of that story rather than deferring again.** Cross-refs C-59 (the duplicate-key hazard that lands differently in each of the three), C-40, #89. + +**SETTLED 2026-07-31 (review-rr — the original re-open trigger resolved without firing).** The sole re-open condition was a long-lived S6/#91 dual-write. **#91 is CLOSED**, and the contract path shipped frame-native end-to-end (PRs #115–#129, wire delivered in run-0) without a durable dual-representation window. Position A is vindicated by events: concrete siblings + delete is what actually happened, the pandas path is confined to the legacy branch pending its named post-run-0 deletion, and no polymorphic representation port was ever needed. No live tension remains — this entry is kept as decision provenance for the remaining seam work (#89, #90) rather than as an open question. See also C-40 (the inheritance/representation coupling this migration unwinds), #85 (the migration epic), #45 (the faoapi wire / S6 dual-write). diff --git a/scripts/build_gaul_lookup.py b/scripts/build_gaul_lookup.py index 0c1af03..18cb20b 100644 --- a/scripts/build_gaul_lookup.py +++ b/scripts/build_gaul_lookup.py @@ -43,6 +43,10 @@ ) +class LookupBuildError(ValueError): + """The lookup cannot be built as declared — never write a half-valid artifact.""" + + def _resolve_datafactory() -> Path: """Locate the views-datafactory checkout without a machine-specific path. @@ -149,9 +153,33 @@ def build(datafactory: Path, region: str, out: Path) -> pd.DataFrame: df = df.sort_index() # Hard invariants — the lookup must be clean by construction. - assert df.isna().sum().sum() == 0, "lookup contains nulls" + # + # These are explicit raises, NOT `assert`: `python -O` strips assert statements + # entirely, and a stripped run would write an unvalidated lookup that looks + # identical on disk (register C-61). The -1 check in particular has no downstream + # backstop — -1 is non-null, so every gate in the delivery chain would pass it + # straight through to FAO, which is exactly the resolved C-35 defect recurring. + if not df.index.is_unique: + dupes = df.index[df.index.duplicated()].unique().tolist() + raise LookupBuildError( + f"{len(dupes)} duplicate gid(s) in the lookup index: {dupes[:10]}. " + "A duplicated key multiplies rows through the enricher's left-merge with " + "every value non-null, so no downstream gate can see it (C-59)." + ) + n_null = int(df.isna().sum().sum()) + if n_null: + raise LookupBuildError( + f"lookup contains {n_null} null value(s); only fully-complete cells may " + "enter it, so that an ABSENT cell (never a partial one) is what fails " + "downstream." + ) for c in CODE_COLS: - assert (df[c] != -1).all(), f"{c} contains -1 sentinel" + n_sentinel = int((df[c] == -1).sum()) + if n_sentinel: + raise LookupBuildError( + f"{c} contains {n_sentinel} -1 sentinel(s). -1 is non-null, so it would " + "reach FAO through every gate as a country/admin code (cf. C-35)." + ) out.parent.mkdir(parents=True, exist_ok=True) table = pa.Table.from_pandas(df, preserve_index=True) diff --git a/tests/fixtures/priogrid_geometry/priogrid_centres_sample.json b/tests/fixtures/priogrid_geometry/priogrid_centres_sample.json new file mode 100644 index 0000000..619dd08 --- /dev/null +++ b/tests/fixtures/priogrid_geometry/priogrid_centres_sample.json @@ -0,0 +1,883 @@ +{ + "cells": { + "1": { + "xcoord": -179.75, + "ycoord": -89.75 + }, + "101167": { + "xcoord": 3.25, + "ycoord": -19.75 + }, + "102464": { + "xcoord": -68.25, + "ycoord": -18.75 + }, + "103761": { + "xcoord": -139.75, + "ycoord": -17.75 + }, + "10377": { + "xcoord": -31.75, + "ycoord": -82.75 + }, + "105058": { + "xcoord": 148.75, + "ycoord": -17.25 + }, + "106355": { + "xcoord": 77.25, + "ycoord": -16.25 + }, + "107652": { + "xcoord": 5.75, + "ycoord": -15.25 + }, + "108949": { + "xcoord": -65.75, + "ycoord": -14.25 + }, + "110246": { + "xcoord": -137.25, + "ycoord": -13.25 + }, + "111543": { + "xcoord": 151.25, + "ycoord": -12.75 + }, + "112840": { + "xcoord": 79.75, + "ycoord": -11.75 + }, + "114137": { + "xcoord": 8.25, + "ycoord": -10.75 + }, + "115434": { + "xcoord": -63.25, + "ycoord": -9.75 + }, + "116731": { + "xcoord": -134.75, + "ycoord": -8.75 + }, + "11674": { + "xcoord": -103.25, + "ycoord": -81.75 + }, + "118028": { + "xcoord": 153.75, + "ycoord": -8.25 + }, + "119325": { + "xcoord": 82.25, + "ycoord": -7.25 + }, + "120622": { + "xcoord": 10.75, + "ycoord": -6.25 + }, + "121919": { + "xcoord": -60.75, + "ycoord": -5.25 + }, + "123216": { + "xcoord": -132.25, + "ycoord": -4.25 + }, + "124513": { + "xcoord": 156.25, + "ycoord": -3.75 + }, + "125810": { + "xcoord": 84.75, + "ycoord": -2.75 + }, + "127107": { + "xcoord": 13.25, + "ycoord": -1.75 + }, + "128404": { + "xcoord": -58.25, + "ycoord": -0.75 + }, + "129600": { + "xcoord": 179.75, + "ycoord": -0.25 + }, + "129601": { + "xcoord": -179.75, + "ycoord": 0.25 + }, + "129701": { + "xcoord": -129.75, + "ycoord": 0.25 + }, + "12971": { + "xcoord": -174.75, + "ycoord": -80.75 + }, + "1298": { + "xcoord": 108.75, + "ycoord": -89.25 + }, + "130320": { + "xcoord": 179.75, + "ycoord": 0.25 + }, + "130321": { + "xcoord": -179.75, + "ycoord": 0.75 + }, + "130998": { + "xcoord": 158.75, + "ycoord": 0.75 + }, + "132295": { + "xcoord": 87.25, + "ycoord": 1.75 + }, + "133592": { + "xcoord": 15.75, + "ycoord": 2.75 + }, + "134889": { + "xcoord": -55.75, + "ycoord": 3.75 + }, + "136186": { + "xcoord": -127.25, + "ycoord": 4.75 + }, + "137483": { + "xcoord": 161.25, + "ycoord": 5.25 + }, + "138780": { + "xcoord": 89.75, + "ycoord": 6.25 + }, + "140077": { + "xcoord": 18.25, + "ycoord": 7.25 + }, + "141374": { + "xcoord": -53.25, + "ycoord": 8.25 + }, + "142671": { + "xcoord": -124.75, + "ycoord": 9.25 + }, + "14268": { + "xcoord": 113.75, + "ycoord": -80.25 + }, + "143968": { + "xcoord": 163.75, + "ycoord": 9.75 + }, + "1440": { + "xcoord": 179.75, + "ycoord": -89.25 + }, + "1441": { + "xcoord": -179.75, + "ycoord": -88.75 + }, + "145265": { + "xcoord": 92.25, + "ycoord": 10.75 + }, + "146562": { + "xcoord": 20.75, + "ycoord": 11.75 + }, + "147859": { + "xcoord": -50.75, + "ycoord": 12.75 + }, + "149156": { + "xcoord": -122.25, + "ycoord": 13.75 + }, + "150453": { + "xcoord": 166.25, + "ycoord": 14.25 + }, + "151750": { + "xcoord": 94.75, + "ycoord": 15.25 + }, + "153047": { + "xcoord": 23.25, + "ycoord": 16.25 + }, + "154344": { + "xcoord": -48.25, + "ycoord": 17.25 + }, + "155641": { + "xcoord": -119.75, + "ycoord": 18.25 + }, + "15565": { + "xcoord": 42.25, + "ycoord": -79.25 + }, + "156938": { + "xcoord": 168.75, + "ycoord": 18.75 + }, + "158235": { + "xcoord": 97.25, + "ycoord": 19.75 + }, + "159532": { + "xcoord": 25.75, + "ycoord": 20.75 + }, + "160829": { + "xcoord": -45.75, + "ycoord": 21.75 + }, + "162126": { + "xcoord": -117.25, + "ycoord": 22.75 + }, + "163423": { + "xcoord": 171.25, + "ycoord": 23.25 + }, + "164720": { + "xcoord": 99.75, + "ycoord": 24.25 + }, + "166017": { + "xcoord": 28.25, + "ycoord": 25.25 + }, + "167314": { + "xcoord": -43.25, + "ycoord": 26.25 + }, + "168611": { + "xcoord": -114.75, + "ycoord": 27.25 + }, + "16862": { + "xcoord": -29.25, + "ycoord": -78.25 + }, + "169908": { + "xcoord": 173.75, + "ycoord": 27.75 + }, + "171205": { + "xcoord": 102.25, + "ycoord": 28.75 + }, + "172502": { + "xcoord": 30.75, + "ycoord": 29.75 + }, + "173799": { + "xcoord": -40.75, + "ycoord": 30.75 + }, + "175096": { + "xcoord": -112.25, + "ycoord": 31.75 + }, + "176393": { + "xcoord": 176.25, + "ycoord": 32.25 + }, + "177690": { + "xcoord": 104.75, + "ycoord": 33.25 + }, + "178987": { + "xcoord": 33.25, + "ycoord": 34.25 + }, + "180284": { + "xcoord": -38.25, + "ycoord": 35.25 + }, + "181581": { + "xcoord": -109.75, + "ycoord": 36.25 + }, + "18159": { + "xcoord": -100.75, + "ycoord": -77.25 + }, + "182470": { + "xcoord": -25.25, + "ycoord": 36.75 + }, + "182878": { + "xcoord": 178.75, + "ycoord": 36.75 + }, + "184175": { + "xcoord": 107.25, + "ycoord": 37.75 + }, + "185472": { + "xcoord": 35.75, + "ycoord": 38.75 + }, + "186769": { + "xcoord": -35.75, + "ycoord": 39.75 + }, + "186778": { + "xcoord": -31.25, + "ycoord": 39.75 + }, + "188066": { + "xcoord": -107.25, + "ycoord": 40.75 + }, + "189363": { + "xcoord": -178.75, + "ycoord": 41.75 + }, + "190660": { + "xcoord": 109.75, + "ycoord": 42.25 + }, + "191957": { + "xcoord": 38.25, + "ycoord": 43.25 + }, + "193254": { + "xcoord": -33.25, + "ycoord": 44.25 + }, + "194551": { + "xcoord": -104.75, + "ycoord": 45.25 + }, + "19456": { + "xcoord": -172.25, + "ycoord": -76.25 + }, + "195848": { + "xcoord": -176.25, + "ycoord": 46.25 + }, + "197145": { + "xcoord": 112.25, + "ycoord": 46.75 + }, + "198442": { + "xcoord": 40.75, + "ycoord": 47.75 + }, + "199739": { + "xcoord": -30.75, + "ycoord": 48.75 + }, + "2": { + "xcoord": -179.25, + "ycoord": -89.75 + }, + "201036": { + "xcoord": -102.25, + "ycoord": 49.75 + }, + "202333": { + "xcoord": -173.75, + "ycoord": 50.75 + }, + "203630": { + "xcoord": 114.75, + "ycoord": 51.25 + }, + "204927": { + "xcoord": 43.25, + "ycoord": 52.25 + }, + "206224": { + "xcoord": -28.25, + "ycoord": 53.25 + }, + "207521": { + "xcoord": -99.75, + "ycoord": 54.25 + }, + "20753": { + "xcoord": 116.25, + "ycoord": -75.75 + }, + "208818": { + "xcoord": -171.25, + "ycoord": 55.25 + }, + "210115": { + "xcoord": 117.25, + "ycoord": 55.75 + }, + "211412": { + "xcoord": 45.75, + "ycoord": 56.75 + }, + "212709": { + "xcoord": -25.75, + "ycoord": 57.75 + }, + "214006": { + "xcoord": -97.25, + "ycoord": 58.75 + }, + "215303": { + "xcoord": -168.75, + "ycoord": 59.75 + }, + "216600": { + "xcoord": 119.75, + "ycoord": 60.25 + }, + "217897": { + "xcoord": 48.25, + "ycoord": 61.25 + }, + "219194": { + "xcoord": -23.25, + "ycoord": 62.25 + }, + "220491": { + "xcoord": -94.75, + "ycoord": 63.25 + }, + "22050": { + "xcoord": 44.75, + "ycoord": -74.75 + }, + "221788": { + "xcoord": -166.25, + "ycoord": 64.25 + }, + "223085": { + "xcoord": 122.25, + "ycoord": 64.75 + }, + "224382": { + "xcoord": 50.75, + "ycoord": 65.75 + }, + "225679": { + "xcoord": -20.75, + "ycoord": 66.75 + }, + "226976": { + "xcoord": -92.25, + "ycoord": 67.75 + }, + "228273": { + "xcoord": -163.75, + "ycoord": 68.75 + }, + "229570": { + "xcoord": 124.75, + "ycoord": 69.25 + }, + "230867": { + "xcoord": 53.25, + "ycoord": 70.25 + }, + "232164": { + "xcoord": -18.25, + "ycoord": 71.25 + }, + "233461": { + "xcoord": -89.75, + "ycoord": 72.25 + }, + "23347": { + "xcoord": -26.75, + "ycoord": -73.75 + }, + "234758": { + "xcoord": -161.25, + "ycoord": 73.25 + }, + "236055": { + "xcoord": 127.25, + "ycoord": 73.75 + }, + "237352": { + "xcoord": 55.75, + "ycoord": 74.75 + }, + "238649": { + "xcoord": -15.75, + "ycoord": 75.75 + }, + "239946": { + "xcoord": -87.25, + "ycoord": 76.75 + }, + "241243": { + "xcoord": -158.75, + "ycoord": 77.75 + }, + "242540": { + "xcoord": 129.75, + "ycoord": 78.25 + }, + "243837": { + "xcoord": 58.25, + "ycoord": 79.25 + }, + "245134": { + "xcoord": -13.25, + "ycoord": 80.25 + }, + "246431": { + "xcoord": -84.75, + "ycoord": 81.25 + }, + "24644": { + "xcoord": -98.25, + "ycoord": -72.75 + }, + "247728": { + "xcoord": -156.25, + "ycoord": 82.25 + }, + "249025": { + "xcoord": 132.25, + "ycoord": 82.75 + }, + "250322": { + "xcoord": 60.75, + "ycoord": 83.75 + }, + "251619": { + "xcoord": -10.75, + "ycoord": 84.75 + }, + "252916": { + "xcoord": -82.25, + "ycoord": 85.75 + }, + "254213": { + "xcoord": -153.75, + "ycoord": 86.75 + }, + "255510": { + "xcoord": 134.75, + "ycoord": 87.25 + }, + "256807": { + "xcoord": 63.25, + "ycoord": 88.25 + }, + "258104": { + "xcoord": -8.25, + "ycoord": 89.25 + }, + "258480": { + "xcoord": 179.75, + "ycoord": 89.25 + }, + "258481": { + "xcoord": -179.75, + "ycoord": 89.75 + }, + "259199": { + "xcoord": 179.25, + "ycoord": 89.75 + }, + "259200": { + "xcoord": 179.75, + "ycoord": 89.75 + }, + "25941": { + "xcoord": -169.75, + "ycoord": -71.75 + }, + "2595": { + "xcoord": 37.25, + "ycoord": -88.25 + }, + "27238": { + "xcoord": 118.75, + "ycoord": -71.25 + }, + "28535": { + "xcoord": 47.25, + "ycoord": -70.25 + }, + "29832": { + "xcoord": -24.25, + "ycoord": -69.25 + }, + "31129": { + "xcoord": -95.75, + "ycoord": -68.25 + }, + "32426": { + "xcoord": -167.25, + "ycoord": -67.25 + }, + "33723": { + "xcoord": 121.25, + "ycoord": -66.75 + }, + "35020": { + "xcoord": 49.75, + "ycoord": -65.75 + }, + "36317": { + "xcoord": -21.75, + "ycoord": -64.75 + }, + "37614": { + "xcoord": -93.25, + "ycoord": -63.75 + }, + "38911": { + "xcoord": -164.75, + "ycoord": -62.75 + }, + "3892": { + "xcoord": -34.25, + "ycoord": -87.25 + }, + "40208": { + "xcoord": 123.75, + "ycoord": -62.25 + }, + "41505": { + "xcoord": 52.25, + "ycoord": -61.25 + }, + "42802": { + "xcoord": -19.25, + "ycoord": -60.25 + }, + "44099": { + "xcoord": -90.75, + "ycoord": -59.25 + }, + "45396": { + "xcoord": -162.25, + "ycoord": -58.25 + }, + "46693": { + "xcoord": 126.25, + "ycoord": -57.75 + }, + "47990": { + "xcoord": 54.75, + "ycoord": -56.75 + }, + "49287": { + "xcoord": -16.75, + "ycoord": -55.75 + }, + "50584": { + "xcoord": -88.25, + "ycoord": -54.75 + }, + "51078": { + "xcoord": 158.75, + "ycoord": -54.75 + }, + "51881": { + "xcoord": -159.75, + "ycoord": -53.75 + }, + "5189": { + "xcoord": -105.75, + "ycoord": -86.25 + }, + "53178": { + "xcoord": 128.75, + "ycoord": -53.25 + }, + "54475": { + "xcoord": 57.25, + "ycoord": -52.25 + }, + "55772": { + "xcoord": -14.25, + "ycoord": -51.25 + }, + "57069": { + "xcoord": -85.75, + "ycoord": -50.25 + }, + "58366": { + "xcoord": -157.25, + "ycoord": -49.25 + }, + "59663": { + "xcoord": 131.25, + "ycoord": -48.75 + }, + "60960": { + "xcoord": 59.75, + "ycoord": -47.75 + }, + "62257": { + "xcoord": -11.75, + "ycoord": -46.75 + }, + "62356": { + "xcoord": 37.75, + "ycoord": -46.75 + }, + "63554": { + "xcoord": -83.25, + "ycoord": -45.75 + }, + "64851": { + "xcoord": -154.75, + "ycoord": -44.75 + }, + "6486": { + "xcoord": -177.25, + "ycoord": -85.25 + }, + "66148": { + "xcoord": 133.75, + "ycoord": -44.25 + }, + "67445": { + "xcoord": 62.25, + "ycoord": -43.25 + }, + "68742": { + "xcoord": -9.25, + "ycoord": -42.25 + }, + "70039": { + "xcoord": -80.75, + "ycoord": -41.25 + }, + "71336": { + "xcoord": -152.25, + "ycoord": -40.25 + }, + "720": { + "xcoord": 179.75, + "ycoord": -89.75 + }, + "721": { + "xcoord": -179.75, + "ycoord": -89.25 + }, + "722": { + "xcoord": -179.25, + "ycoord": -89.25 + }, + "72633": { + "xcoord": 136.25, + "ycoord": -39.75 + }, + "73930": { + "xcoord": 64.75, + "ycoord": -38.75 + }, + "75227": { + "xcoord": -6.75, + "ycoord": -37.75 + }, + "76524": { + "xcoord": -78.25, + "ycoord": -36.75 + }, + "77821": { + "xcoord": -149.75, + "ycoord": -35.75 + }, + "7783": { + "xcoord": 111.25, + "ycoord": -84.75 + }, + "79118": { + "xcoord": 138.75, + "ycoord": -35.25 + }, + "80415": { + "xcoord": 67.25, + "ycoord": -34.25 + }, + "81712": { + "xcoord": -4.25, + "ycoord": -33.25 + }, + "83009": { + "xcoord": -75.75, + "ycoord": -32.25 + }, + "84306": { + "xcoord": -147.25, + "ycoord": -31.25 + }, + "85603": { + "xcoord": 141.25, + "ycoord": -30.75 + }, + "86900": { + "xcoord": 69.75, + "ycoord": -29.75 + }, + "88197": { + "xcoord": -1.75, + "ycoord": -28.75 + }, + "89494": { + "xcoord": -73.25, + "ycoord": -27.75 + }, + "90791": { + "xcoord": -144.75, + "ycoord": -26.75 + }, + "9080": { + "xcoord": 39.75, + "ycoord": -83.75 + }, + "92088": { + "xcoord": 143.75, + "ycoord": -26.25 + }, + "93385": { + "xcoord": 72.25, + "ycoord": -25.25 + }, + "94682": { + "xcoord": 0.75, + "ycoord": -24.25 + }, + "94776": { + "xcoord": 47.75, + "ycoord": -24.25 + }, + "95979": { + "xcoord": -70.75, + "ycoord": -23.25 + }, + "97276": { + "xcoord": -142.25, + "ycoord": -22.25 + }, + "98573": { + "xcoord": 146.25, + "ycoord": -21.75 + }, + "99870": { + "xcoord": 74.75, + "ycoord": -20.75 + } + }, + "n_cells_in_source": 259200, + "note": "Authoritative PRIO-GRID cell-centre coordinates. Committed so the gid->lat/lon formula stays verifiable in CI without the datafactory checkout.", + "source": "views-datafactory data/raw/priogrid/shapefile/priogrid_cell.dbf" +} diff --git a/tests/test_gaul_lookup_fidelity.py b/tests/test_gaul_lookup_fidelity.py new file mode 100644 index 0000000..578d22f --- /dev/null +++ b/tests/test_gaul_lookup_fidelity.py @@ -0,0 +1,386 @@ +"""Fidelity guards for the committed ADR-011 GAUL lookup (register C-43 / C-59 / C-61). + +``views_postprocessing/data/gaul_lookup.parquet`` is a 64,742-row binary that the +enricher, the historical builder and the wire sidecar all treat as ground truth. +Until 2026-07-31 nothing verified it: every downstream gate checks **presence** +(nulls, cell counts, gid-set equality) and none checked **values**. The migration +plan's own old-vs-new diff (#20 Stage 2 / #23) was signed off without being +produced, and the comparator was deleted in PR #42 — so the backward check is +unrecoverable and C-43 stood open with global data already delivered to FAO. + +The check that *is* available runs **forwards**: the lookup is a transcription of +views-datafactory's area-majority GAUL parquets plus a closed-form gid->lat/lon +formula, so both halves can be re-derived from the producer. That was run by hand +on 2026-07-31 (zero mismatches on all four links, including the delivered run-0 +sidecar bytes); this module makes it a standing guarantee instead of a session +result. C-43 closes on these tests being green, not on that transcript. + +**Scope, deliberately.** These tests prove the lookup faithfully *transcribes* the +producer's assignment. They cannot prove the assignment is *correct* — if +views-datafactory puts a cell in the wrong country, every test here still passes. +That is views-datafactory#387 (square-degree area math at high latitudes), and it +must not be conflated with what this file guarantees. + +Split by dependency, on purpose: + * **always-on** — self-consistency of the committed artifact, plus the coordinate + formula against a committed PRIO-GRID sample. Runs anywhere, including CI. + * **skipif** — full value comparison against the views-datafactory sibling + checkout. Stronger, but absent on most machines (cf. C-46, where a hardcoded + path made a cross-repo gate invisible; here the skip is explicit and the + always-on half still guards regressions). +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import numpy as np +import pyarrow.parquet as pq +import pytest + +from views_postprocessing.delivery import coverage +from views_postprocessing.unfao.gaul_schema import ( + CODE_COLS, + COORD_COLS, + METADATA_COLS, + NAME_COLS, + SOURCE_RENAME, + xcoord, + ycoord, +) + +_REPO = Path(__file__).resolve().parent.parent +_LOOKUP = _REPO / "views_postprocessing" / "data" / "gaul_lookup.parquet" +_GROUND_TRUTH = _REPO / "tests" / "fixtures" / "priogrid_geometry" / "priogrid_centres_sample.json" + +# The lookup is built for this region; its count is pinned in delivery/coverage.py. +_REGION = "land_gaul" + +_DATAFACTORY = _REPO.parent / "views-datafactory" +_HAS_DATAFACTORY = (_DATAFACTORY / "data" / "raw" / "gaul_admin").is_dir() +_needs_datafactory = pytest.mark.skipif( + not _HAS_DATAFACTORY, + reason=( + "views-datafactory sibling checkout not present — the producer-comparison half " + "cannot run here. The always-on tests in this module still guard the artifact." + ), +) + + +@pytest.fixture(scope="module") +def lookup(): + assert _LOOKUP.exists(), f"the ADR-011 lookup is missing: {_LOOKUP}" + return pq.read_table(_LOOKUP) + + +@pytest.fixture(scope="module") +def lookup_meta(): + raw = pq.read_metadata(_LOOKUP).metadata or {} + return {k.decode(): v.decode() for k, v in raw.items() if k != b"pandas"} + + +@pytest.fixture(scope="module") +def gids(lookup): + return lookup.column("priogrid_gid").to_numpy() + + +# ── always-on: the committed artifact must be self-consistent ──────────────── + + +def test_lookup_key_is_unique(gids): + """C-59: a duplicated gid silently inflates the legacy pandas delivery. + + ``enrichment.py`` left-merges on this key, so a duplicate multiplies rows for + the affected cell — with every metadata value present and non-null, so the + null gate passes, and with the distinct-cell count unchanged, so the coverage + gate passes too. Nothing downstream can see it; it has to be caught here. + """ + unique = np.unique(gids) + assert unique.size == gids.size, ( + f"the GAUL lookup has {gids.size - unique.size} duplicate gid(s) — the legacy " + "merge would inflate rows invisibly to every downstream gate (C-59)." + ) + + +def test_lookup_has_no_nulls(lookup): + for col in METADATA_COLS: + assert lookup.column(col).null_count == 0, ( + f"lookup column {col!r} has nulls; the lookup must contain only complete " + "cells so that an absent cell (not a partial one) is what fails downstream." + ) + + +def test_lookup_carries_no_sentinel_codes(lookup): + """C-61: ``-1`` is non-null, so no downstream gate would ever catch it. + + Shipping ``-1`` as a country/admin code is the resolved C-35 defect returning + through a different door. The builder guards this with a bare ``assert``, + which ``python -O`` strips — so the artifact is re-checked here. + """ + for col in CODE_COLS: + values = lookup.column(col).to_numpy(zero_copy_only=False).astype("float64") + assert not np.any(values == -1), ( + f"lookup column {col!r} contains the -1 sentinel — this is C-35 (invalid " + "codes delivered to FAO) recurring, and it is non-null so nothing downstream " + "would reject it (C-61)." + ) + + +def test_lookup_names_are_non_empty(lookup): + for col in NAME_COLS: + values = lookup.column(col).to_pylist() + empty = sum(1 for v in values if v is None or str(v).strip() == "") + assert empty == 0, f"lookup column {col!r} has {empty} empty name(s)." + + +def test_lookup_columns_are_exactly_the_contract(lookup): + present = [c for c in lookup.column_names if not c.startswith("__")] + assert set(present) == {"priogrid_gid", *METADATA_COLS}, ( + f"lookup schema drifted from the 9-column contract: {sorted(present)}" + ) + + +def test_lookup_cell_count_matches_the_pinned_region_count(gids): + """The artifact and the delivery's coverage contract must agree. + + If these diverge, either the lookup was rebuilt for a different region or the + pin in ``delivery/coverage.py`` is stale — both ship wrong coverage. + """ + expected = coverage.expected_for(_REGION) + assert expected is not None, f"{_REGION} is unpinned in delivery/coverage.py" + assert gids.size == expected, ( + f"lookup has {gids.size} cells but delivery/coverage.py pins {expected} for " + f"{_REGION!r}. One of the two is stale." + ) + + +def test_lookup_excludes_the_gaul_uncovered_cells(gids): + """The excluded cells have no GAUL assignment; they must not be in the lookup.""" + leaked = set(gids.tolist()) & set(coverage.excluded_for(_REGION)) + assert not leaked, ( + f"{len(leaked)} GAUL-uncovered cell(s) are present in the lookup: " + f"{sorted(leaked)[:10]}. These would attribute partner rows to a non-country." + ) + + +def test_coordinate_formula_matches_priogrid_ground_truth(): + """The gid->lat/lon formula against authoritative PRIO-GRID cell centres. + + This is the highest-leverage check in the module. ``gaul_schema.xcoord/ycoord`` + derive coordinates arithmetically rather than reading them from the producer, + so a transposed row/col, an off-by-one, or a hemisphere flip would put a plausible + wrong coordinate on **every** cell — non-null, correctly typed, and invisible to + every gate in the delivery chain. + + The fixture samples grid corners, row boundaries, the equator crossing and an + even spread, so the failure modes above cannot all pass by coincidence. + """ + truth = json.loads(_GROUND_TRUTH.read_text())["cells"] + bad = [] + for gid_s, want in truth.items(): + gid = int(gid_s) + if abs(xcoord(gid) - want["xcoord"]) > 1e-9: + bad.append((gid, "xcoord", want["xcoord"], xcoord(gid))) + if abs(ycoord(gid) - want["ycoord"]) > 1e-9: + bad.append((gid, "ycoord", want["ycoord"], ycoord(gid))) + assert not bad, f"coordinate formula diverged from PRIO-GRID ground truth: {bad[:5]}" + + +def test_lookup_coordinates_were_built_with_that_formula(lookup, gids): + """Every stored coordinate must equal the formula applied to its own gid.""" + want_x = np.array([xcoord(int(g)) for g in gids]) + want_y = np.array([ycoord(int(g)) for g in gids]) + got_x = lookup.column("pg_xcoord").to_numpy(zero_copy_only=False) + got_y = lookup.column("pg_ycoord").to_numpy(zero_copy_only=False) + assert np.abs(got_x - want_x).max() == 0.0, "stored pg_xcoord disagrees with the formula" + assert np.abs(got_y - want_y).max() == 0.0, "stored pg_ycoord disagrees with the formula" + + +def test_lookup_declares_its_provenance(lookup_meta): + """C-60: the traceability stamp must be present rather than degrading silently. + + ``enrichment._read_version`` falls back to ``"unknown"`` when this metadata is + absent or reshaped, so the field that ties a delivery to its lookup build can + vanish without any gate noticing. At minimum the build must declare itself. + """ + for key in ("adr", "region", "n_cells"): + assert key in lookup_meta, f"lookup parquet metadata is missing {key!r}" + assert lookup_meta["region"] == _REGION + assert int(lookup_meta["n_cells"]) == coverage.expected_for(_REGION) + + +def test_lookup_version_stamp_resolves(lookup): + """The stamp the delivery provenance carries (C-15) must not be 'unknown'.""" + from views_postprocessing.unfao.enrichment import GaulLookupEnricher + + version = GaulLookupEnricher(_LOOKUP).lookup_version + assert version != "unknown", ( + "lookup_version resolved to 'unknown' — the delivery would ship untraceable " + "provenance (C-60). The lookup's embedded source_provenance is absent or reshaped." + ) + assert version.startswith(f"{_REGION}@"), f"unexpected lookup_version: {version!r}" + + +# ── skipif: the full comparison against the producer ───────────────────────── + + +@_needs_datafactory +def test_lookup_values_match_the_producer_parquets(lookup, gids): + """C-43, the core forward-check: every value against views-datafactory. + + This is what the deleted #23 diff was for, run forwards against the producer + instead of backwards against the retired mapper. All 7 GAUL columns, all cells. + """ + mismatches = {} + for src, dst in SOURCE_RENAME.items(): + table = pq.read_table(_DATAFACTORY / "data" / "raw" / "gaul_admin" / f"{src}.parquet") + source = dict( + zip( + (int(g) for g in table.column("gid").to_pylist()), + table.column("value").to_pylist(), + ) + ) + got = lookup.column(dst).to_pylist() + bad = [ + (int(g), source.get(int(g)), v) + for g, v in zip(gids, got) + if source.get(int(g)) != v + ] + if bad: + mismatches[dst] = bad[:5] + assert not mismatches, ( + f"the committed lookup diverges from the producer's GAUL parquets: {mismatches}. " + "Rebuild it with scripts/build_gaul_lookup.py against the current datafactory." + ) + + +@_needs_datafactory +def test_lookup_gid_set_equals_the_declared_region(gids): + region_file = _DATAFACTORY / "src" / "datafactory_query" / f"{_REGION}_pgids.json" + region = set(json.loads(region_file.read_text())) + got = set(int(g) for g in gids) + assert got == region, ( + f"lookup gid set != {_REGION} region: {len(region - got)} missing, " + f"{len(got - region)} extra." + ) + + +@_needs_datafactory +def test_coordinate_formula_matches_every_priogrid_cell(): + """The committed fixture samples 219 cells; the sibling lets us check all 259,200.""" + import struct + + dbf = _DATAFACTORY / "data" / "raw" / "priogrid" / "shapefile" / "priogrid_cell.dbf" + if not dbf.exists(): + pytest.skip("PRIO-GRID shapefile not present in the datafactory checkout") + with dbf.open("rb") as fh: + header = fh.read(32) + n_records = struct.unpack("float64 wire cast losslessly (sidecar/historical §5.1).""" + for col in CODE_COLS: + values = lookup.column(col).to_numpy(zero_copy_only=False) + assert np.all(np.abs(values) < 2**53), ( + f"{col} exceeds exact float64 integer range; the §5.1 float64 wire cast " + "would silently lose precision." + ) + for col in COORD_COLS: + assert lookup.schema.field(col).type == "double", f"{col} must be float64"