S5 (#186) — the lookup declares its own version instead of the consumer inferring it

[Polichinel]

Closes #186. Closes register C-60. Part of epic #181.

The defect

gaul_lookup.version() produced the stamp that ties a delivered artifact to the lookup build that made it — C-15's traceability field, the thing you would need after a suspect delivery to answer "which lookup produced this?"

It got that stamp by reaching three levels into views-datafactory's ingestion-ledger shape:

prov = json.loads(meta.get("source_provenance", "{}"))
digest = (prov.get("land_gaul_region", {}).get("content_digest", "?"))[:8]
except (ValueError, AttributeError):
    pass
return "unknown" if region == "?" and digest == "?" else f"{region}@{digest}"

A rename upstream makes every delivery untraceable, silently. The provenance record is still written; it just stops meaning anything, and no gate can tell the difference.

The fix

The builder composes the stamp — it is the only thing holding both the region and the ledger — and writes one flat lookup_version key. The consumer reads that key and raises. The json import and every exception handler are gone from the module; a test asserts that by AST rather than by word-matching.

An untraceable build now fails at build time, where a human is present, instead of surfacing as a placeholder in production.

Review caught the fix reintroducing its own defect

The first draft read:

entry = provenance.get("land_gaul_region") or provenance.get("gaul_admin_area_majority") or {}

_provenance does not take a region, so land_gaul_region is present for every build. --region all would have been stamped all@f74d3b2b — the land_gaul region definition's digest, on a global artifact. Authoritative-looking, wrong, silent: C-60's own defect class, one level up.

Replaced by a declared rule with no fallback:

stamp_dataset("all")              -> gaul_admin_area_majority
stamp_dataset("land_gaul")        -> land_gaul_region
stamp_dataset("africa_me_legacy") -> africa_me_legacy_region

Verified against the real ledger:

region	stamp
all	all@272cdb01	the area-majority join that produced every value
land_gaul	land_gaul@f74d3b2b	unchanged from before this story
africa_me_legacy	—	refused, not substituted, though another digest sits right there in the blob

The rebuilt artifact — metadata-only, proven

	before	after
rows	64,742	64,742
column order		same
dtypes		same
content hash over every column's values		IDENTICAL
file bytes	902,066	902,187 (the new key)
stamp	land_gaul@f74d3b2b	land_gaul@f74d3b2b

The stamp is the same string the old traversal produced, so no delivered provenance changes meaning.

The reproducible evidence is pytest -q tests/test_gaul_lookup_fidelity.py — 26 tests comparing the artifact value-for-value against views-datafactory's seven source parquets, plus gid-set equality, key uniqueness, the coordinate formula, and the absence of nulls and -1 sentinels. That is a stronger claim than a byte hash because it re-runs on every push.

The wire is untouched

contract_version stays 1.5; git diff origin/main -- tests/fixtures/ is empty. Verified empirically before scoping the story: build_sidecar constructs a fresh pa.table(columns), so lookup parquet metadata cannot reach the wire — the sidecar's schema metadata is [].

Other review findings

F2 — docs/CICs/GaulLookupEnricher.md still documented the "unknown" degrade. Corrected to state the raise and name LookupVersionError. Third story running where the CIC lagged; noted for S9 — it is in no story's default file list, so it is only updated when someone remembers.

F3 — the ledger is append-only (14 gaul_admin_area_majority entries, 2 land_gaul_region) and the last per dataset wins. Harmless while provenance was decorative; load-bearing now that the stamp raises. Stated in _provenance rather than left implicit in the loop.

F4 — [:8] would have emitted land_gaul@abcd from a 4-char digest: accepted by the builder, rejected by the test. Digest length is now declared and enforced on both sides.

F5 — dropped a comment implying parquet metadata key order is meaningful.

Verification

ruff check .                                     # clean
pytest -q                                        # 295 passed, 41 xfailed, exactly the 5 known pyarrow (C-72)
pytest -q tests/test_gaul_lookup_fidelity.py     # 26 passed
git diff --stat origin/main -- tests/fixtures/   # empty

Register: 74 concerns, 20 open after C-60.

🤖 Generated with Claude Code