S5 (#186) — the lookup declares its own version instead of the consumer inferring it

docs/CICs/GaulLookupEnricher.md

@@ -62,9 +62,16 @@ views-datafactory area-majority join), so this class does only a table join.
 - Output: the input frame (or, with `only_metadata=True`, just `pg_id_col` +
   `time_id_col`) left-merged with the 9 metadata columns.
 - Public attribute: `lookup_version` — a short, stampable id read from the
-  lookup's embedded provenance at construction (`<region>@<short digest>`, or
-  `"unknown"` if the lookup carries none). The manager stamps it on each
-  delivery so a delivery is traceable to the exact lookup build.
+  lookup's **declared** `lookup_version` metadata key at construction
+  (`<region>@<8-char source digest>`). Delegates to
+  `contract.gaul_lookup.version`. The manager stamps it on each delivery so a
+  delivery is traceable to the exact lookup build.
+  **It does not degrade.** An artifact carrying no declared key raises
+  `gaul_lookup.LookupVersionError` (logged at ERROR first, per ADR-008) rather
+  than returning a placeholder. Until S5 (#186) this returned the string
+  `"unknown"` whenever views-datafactory's ingestion-ledger shape moved under
+  it — silently, in the one field register C-15 exists to answer *after* a
+  suspect delivery. See register **C-60**.
 - Side effects: logs the lookup size + version at construction (INFO); logs a
   WARNING with the count and sample of unmatched cell ids when any occur; logs
   ignored mapper-only kwargs at DEBUG. No file writes, no network.

reports/technical_risk_register.md

@@ -6,8 +6,8 @@
 | Owner             | Dylan Pinheiro / PRIO MD&D Team      |
 | Last Updated      | 2026-08-02                           |
 | Total Concerns    | 74                                   |
-| Open Concerns     | 21                                   |
-| Resolved Concerns | 53                                   |
+| Open Concerns     | 20                                   |
+| Resolved Concerns | 54                                   |
 
 ---
 
@@ -71,7 +71,7 @@ covered a single open entry (see Historical clusters below).
 
 **The lesson, and it generalises past this cluster.** C-43's residual said the forward-check *"was a one-off session result, not a standing guarantee"* — and the fix was to attach it to something the interpreter runs. The entries then reproduced the identical error one level up: they stated their closing conditions in prose and attached them to nothing. S2 therefore added a closing-condition check to `test_register_integrity.py`. **A guarantee needs a check, and that applies to the register's own guarantees too.**
 
-**Remaining in this cluster:** **C-60** (the flat declared `lookup_version` key — S5 / #186) and **C-46** (the hardcoded datafactory path — S7 / #188). Both unblocked, both in epic #181.
+**C-60 closed 2026-08-02 (S5 / #186)** — the flat declared key landed and the committed artifact was rebuilt, metadata-only. **Remaining in this cluster: C-46 alone** (the hardcoded datafactory path — S7 / #188), which is the last thing keeping the cluster open.
 
 ### Cluster L: The won migration was never cleaned up
 **Root cause:** the frame-native contract path replaced the pandas path and **won** — run-0 delivered global-land on 2026-07-27 and FAO has been served from it since. The replaced path was deliberately kept behind a config fork "until run 0 proves the contract path live" (C-40) and was then never removed. Everything below is residue of that one omission, not independent defects.
@@ -454,26 +454,6 @@ Cross-refs: C-57 (registry drift — the most likely way a coordinate goes wrong
 
 ---
 
-### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except
-
-| Field | Value |
-|-------|-------|
-| ID | C-60 |
-| Tier | 3 |
-| Source | `expert-code-review` (2026-07-31) — Ousterhout lens (information leakage / silent degradation) |
-| Trigger | When views-datafactory renames or restructures its ingestion-ledger entries (`dataset` key, `content_digest` field, or the `land_gaul_region` entry name) — verify `lookup_version` still resolves to a real value rather than the string `"unknown"`; nothing fails if it does not |
-| Location | `views_postprocessing/unfao/enrichment.py:61-79` (`_read_version`); written at `scripts/build_gaul_lookup.py:89-107` (`_provenance`) and `:163-164`; consumed as the C-15 provenance field via the manager's delivery description |
-
-`_read_version` reconstructs the stamp by traversing three levels of the producer's schema — parquet metadata → `source_provenance` JSON → `land_gaul_region` → `content_digest` — and wraps the traversal in `except (ValueError, AttributeError): pass`, returning `"unknown"` when anything along the path is absent. This is a **declare-don't-infer violation at the consumer**: the value that ties a delivered artifact to the exact lookup build (C-15's traceability provenance) can silently become a placeholder, and no gate notices.
-
-No wrong data results — this is a traceability failure, not a correctness one → **Tier 3**. But it defeats the specific question C-15 exists to answer *after* a suspect delivery ("which lookup produced this?"), and C-22 has no correction procedure that could compensate.
-
-**Mitigation:** have `build_gaul_lookup.py` write a **flat, declared `lookup_version` key** into the parquet metadata, and have `_read_version` read that one key and **raise** if absent. The consumer stops knowing the producer's nested ledger schema, and the stamp stops being able to vanish quietly. Separately worth stamping `lookup_version` into the sidecar's own parquet metadata so a delivered artifact is self-describing without the store document.
-
-Cross-refs: C-15 (the provenance this field serves), C-22 (the recall process that would need it), C-57 (the same class — a cross-repo fact this repo reads without a way to detect drift), **Cluster K**.
-
----
-
 ### C-62: The pinned pipeline-core release still installs geopandas and torch into a repo that architecturally excised them
 
 | Field | Value |
@@ -694,6 +674,38 @@ See also C-40 (the inheritance/representation coupling this migration unwinds),
 
 ## Resolved Concerns
 
+### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except — RESOLVED
+
+| Field | Value |
+|-------|-------|
+| ID | C-60 |
+| Resolved | 2026-08-02 |
+| Resolution | **Closed by S5 (#186).** The builder composes the stamp and writes one flat `lookup_version` key; `contract/gaul_lookup.version` reads that key and **raises** `LookupVersionError` (logged first, ADR-008) when it is absent. The three-level traversal of views-datafactory's ledger shape, the bare `except … pass` and the `"unknown"` branch are all gone — pinned by `tests/test_gaul_lookup_access.py`, which asserts by **AST** rather than by word-matching that the module imports no `json` and holds zero exception handlers. An untraceable *build* now fails at build time, where a human is present, instead of surfacing as a placeholder in production.
+
+**The committed artifact was rebuilt, metadata-only.** 64,742 rows, column order and dtypes unchanged, a content hash over every column's values identical, and the stamp resolves to `land_gaul@f74d3b2b` — the same string the old traversal produced, so no delivered provenance changes meaning. ADR-013 untouched: `contract_version` 1.5, `tests/fixtures/` byte-identical (`build_sidecar` constructs a fresh table, so lookup metadata cannot reach the wire — verified empirically, sidecar schema metadata is `[]`).
+
+**⚠ The fix reintroduced this entry's own defect class, caught in review before merge.** The first draft read `provenance.get("land_gaul_region") or provenance.get("gaul_admin_area_majority")`. Since `_provenance` takes no region, `land_gaul_region` is present for *every* build — so `--region all` would have been stamped `all@f74d3b2b`, the land_gaul region definition's digest, on a global artifact: authoritative-looking, wrong, silent. Replaced by a declared `stamp_dataset(region)` with no fallback; a region whose ledger entry is absent is **refused**, not substituted. Verified: `all@272cdb01`, `land_gaul@f74d3b2b`, `africa_me_legacy` → raises. Worth keeping visible — the reflex that produced C-60 reappeared while fixing C-60.
+
+**Residual, deliberately deferred:** stamping `lookup_version` into the **sidecar's own** parquet metadata, so a delivered artifact is self-describing without the store document. Not done here because it changes wire bytes and therefore `contract_version` — an ADR-013 amendment with three repos to notify. Named trigger: **the next ADR-013 version bump**, whatever prompts it.
+
+**Also recorded from review:** the ingestion ledger is append-only and the **last entry per dataset wins** (14 `gaul_admin_area_majority` entries, 2 `land_gaul_region`). Harmless while provenance was decorative; load-bearing now that the stamp raises, so it is stated in `_provenance` rather than left implicit — the same shape **C-73** was opened for on the forecast path, intended here and now said. `docs/CICs/GaulLookupEnricher.md` still documented the `"unknown"` degrade and was corrected. |
+| Tier | 3 |
+| Source | `expert-code-review` (2026-07-31) — Ousterhout lens (information leakage / silent degradation) |
+| Trigger | When views-datafactory renames or restructures its ingestion-ledger entries (`dataset` key, `content_digest` field, or the `land_gaul_region` entry name) — verify `lookup_version` still resolves to a real value rather than the string `"unknown"`; nothing fails if it does not |
+| Location | `views_postprocessing/unfao/enrichment.py:61-79` (`_read_version`); written at `scripts/build_gaul_lookup.py:89-107` (`_provenance`) and `:163-164`; consumed as the C-15 provenance field via the manager's delivery description |
+
+`_read_version` reconstructs the stamp by traversing three levels of the producer's schema — parquet metadata → `source_provenance` JSON → `land_gaul_region` → `content_digest` — and wraps the traversal in `except (ValueError, AttributeError): pass`, returning `"unknown"` when anything along the path is absent. This is a **declare-don't-infer violation at the consumer**: the value that ties a delivered artifact to the exact lookup build (C-15's traceability provenance) can silently become a placeholder, and no gate notices.
+
+No wrong data results — this is a traceability failure, not a correctness one → **Tier 3**. But it defeats the specific question C-15 exists to answer *after* a suspect delivery ("which lookup produced this?"), and C-22 has no correction procedure that could compensate.
+
+**Mitigation:** have `build_gaul_lookup.py` write a **flat, declared `lookup_version` key** into the parquet metadata, and have `_read_version` read that one key and **raise** if absent. The consumer stops knowing the producer's nested ledger schema, and the stamp stops being able to vanish quietly. Separately worth stamping `lookup_version` into the sidecar's own parquet metadata so a delivered artifact is self-describing without the store document.
+
+Cross-refs: C-15 (the provenance this field serves), C-22 (the recall process that would need it), C-57 (the same class — a cross-repo fact this repo reads without a way to detect drift), **Cluster K**.
+
+---
+
+---
+
 ### C-03: Test coverage gaps in manager validation and the enrich→validate path — RESOLVED
 
 | Field | Value |

scripts/build_gaul_lookup.py

@@ -90,8 +90,79 @@ def _region_gids(datafactory: Path, region: str) -> set[int] | None:
     return set(json.loads(path.read_text()))
 
 
-def _provenance(datafactory: Path) -> dict:
-    """Pull the land_gaul ledger entry for traceability (best-effort)."""
+#: The ledger dataset naming the whole GAUL area-majority join, used to stamp an
+#: unregionalised (``--region all``) build.
+AREA_MAJORITY_DATASET = "gaul_admin_area_majority"
+
+#: Length of the digest carried in a stamp. Declared so producer and consumer agree.
+DIGEST_CHARS = 8
+
+
+def stamp_dataset(region: str) -> str:
+    """DECLARED: which ledger dataset's digest stamps this region's build.
+
+    One rule, no fallback. ``--region all`` is stamped by the area-majority join that
+    produced every value; a regional build is stamped by that region's own definition
+    entry, because the region determines which rows exist and a change to it changes
+    the artifact.
+
+    **Why not "whichever entry happens to be present" (register C-60, review of #186).**
+    The first draft read ``land_gaul_region or gaul_admin_area_majority``. Since
+    ``_provenance`` does not take a region, ``land_gaul_region`` is present for *every*
+    build — so ``--region all`` would have been stamped ``all@f74d3b2b``, the land_gaul
+    region definition's digest, on a global artifact. Authoritative-looking and wrong,
+    silently: the exact defect class C-60 exists to close, reintroduced one level up.
+    A region with no ledger entry is now refused, not substituted.
+    """
+    return AREA_MAJORITY_DATASET if region == "all" else f"{region}_region"
+
+
+def _lookup_version(region: str, provenance: dict) -> str:
+    """Compose the DECLARED build stamp: ``<region>@<source digest>``.
+
+    This is the value written as a flat ``lookup_version`` key and read back verbatim
+    by ``contract.gaul_lookup.version`` — register **C-60**. Composing it here is the
+    point: the builder is the only thing holding both the region and the producer's
+    ledger, so the consumer stops needing to know views-datafactory's schema in order
+    to answer "which lookup produced this delivery?".
+
+    **Refuses an untraceable build.** ``_provenance`` is best-effort by design — a
+    missing or reshaped ledger yields ``{}``. Previously that produced the string
+    ``"unknown"`` at *delivery* time, silently, in the one field C-15 exists to answer
+    after a suspect delivery. Failing here moves the error to the moment a human is
+    present to fix it: the build.
+    """
+    dataset = stamp_dataset(region)
+    digest = str((provenance.get(dataset) or {}).get("content_digest", ""))
+    if len(digest) < DIGEST_CHARS:
+        raise LookupBuildError(
+            f"cannot compose a lookup_version for region {region!r}: the datafactory "
+            f"ingestion ledger has no usable content_digest for {dataset!r} "
+            f"(provenance/gaul_admin/ingestion_ledger.jsonl; need at least "
+            f"{DIGEST_CHARS} characters, got {len(digest)}). An artifact without a "
+            "declared version cannot be traced back to the build that made it, which "
+            "is what register C-15 needs after a suspect delivery. Rebuild against a "
+            "datafactory checkout whose ledger declares that dataset — this build is "
+            "NOT stamped from some other region's entry."
+        )
+    return f"{region}@{digest[:DIGEST_CHARS]}"
+
+
+def _provenance(datafactory: Path, *, datasets: tuple[str, ...]) -> dict:
+    """Pull the named ledger entries for traceability (best-effort).
+
+    Kept richer than the stamp — it carries timestamps and the upstream GAUL digest,
+    which are worth having. What changed in C-60 is that the STAMP is no longer
+    derived from it by the consumer; see ``_lookup_version``.
+
+    **The ledger is append-only and the LAST entry per dataset wins.** The real file
+    carries 14 ``gaul_admin_area_majority`` entries and 2 ``land_gaul_region`` ones;
+    this loop overwrites, so the most recent ingestion is what stamps the build. That
+    was harmless while provenance was decorative. Since C-60 the stamp *raises* on its
+    absence, so the selection rule is load-bearing and is stated here rather than left
+    to be inferred from the loop. (Recency-over-a-broad-match is the shape register
+    **C-73** was opened for on the forecast path; here it is intended, and now said.)
+    """
     ledger = (datafactory / "provenance" / "gaul_admin"
               / "ingestion_ledger.jsonl")
     out = {}
@@ -101,8 +172,7 @@ def _provenance(datafactory: Path) -> dict:
                 e = json.loads(line)
             except json.JSONDecodeError:
                 continue
-            if e.get("dataset") in ("land_gaul_region",
-                                    "gaul_admin_area_majority"):
+            if e.get("dataset") in datasets:
                 out[e["dataset"]] = {
                     k: e[k] for k in
                     ("content_digest", "source_gaul_digest", "timestamp")
@@ -188,8 +258,11 @@ def build(datafactory: Path, region: str, out: Path) -> pd.DataFrame:
     meta[b"region"] = region.encode()
     meta[b"n_cells"] = str(len(df)).encode()
     meta[b"n_dropped_incomplete"] = str(dropped).encode()
-    meta[b"source_provenance"] = json.dumps(
-        _provenance(datafactory)).encode()
+    prov = _provenance(datafactory, datasets=(AREA_MAJORITY_DATASET, stamp_dataset(region)))
+    # The DECLARED stamp: one flat key, composed here, read verbatim by the consumer
+    # (C-60). Key order in parquet metadata carries no meaning; this is a dict.
+    meta[b"lookup_version"] = _lookup_version(region, prov).encode()
+    meta[b"source_provenance"] = json.dumps(prov).encode()
     table = table.replace_schema_metadata(meta)
     pq.write_table(table, out)