Repository navigation
S5 — build_gaul_lookup.py: pyarrow-native build #90
Description
Epic: #85 · S5 · [UNILATERAL — lowest priority]
Background
scripts/build_gaul_lookup.py builds the GAUL lookup parquet from views-datafactory's source parquets using pandas + pyarrow. It runs offline, outside the delivery path, so it touches nothing interconnected and has zero samples relevance. Pandas-free for completeness only.
Work
- Make the builder pandas-free (pure
pyarrow): the join/filter/dtype/sort steps currently done with pandas (build_gaul_lookup.py~:71-166) reimplemented on arrow tables; keep the embedded provenance metadata that_read_versionreads.
Acceptance criteria
- The builder produces a lookup parquet identical to the current one from the same datafactory inputs (table contents + embedded provenance metadata).
- No pandas import in the script.
Parity / validation
Build the lookup both ways from the same inputs; assert identical arrow table and identical provenance metadata. (Downstream tests/test_enrichment.py lookup-integrity assertions must still pass against the rebuilt file.)
Dependencies
Independent; lowest urgency. Keep in sync with S4 and gaul_schema.py (the 9-column contract + rename map).
Files
scripts/build_gaul_lookup.py.
Activity
Polichinel commentedon Aug 2, 2026
Collision with epic #181's S5 (#186) — read before starting this.
Both stories rewrite
scripts/build_gaul_lookup.py, and #186 lands first.This issue's acceptance criterion is "produces a lookup parquet identical to the current one from the same datafactory inputs (table contents + embedded provenance metadata)". #186 deliberately changes the embedded metadata — it adds a flat declared
lookup_versionkey so the provenance stamp can no longer silently degrade to the string"unknown"(register C-60), and rebuilds the committed artifact. Table contents stay identical; metadata does not.Also stale here: this issue says "keep the embedded provenance metadata that
_read_versionreads"._read_versionmoved out ofenrichment.pytocontract/gaul_lookup.version()in #152, and #186 rewrites it to read one declared key and raise when it is absent.Re-baseline this issue's parity target against the tree after #186 merges.
Polichinel commentedon Aug 2, 2026
#186 has now merged (#201) — this issue's parity target needs re-baselining.
The builder's embedded metadata changed as predicted: it now writes a flat, declared
lookup_versionkey, and refuses a build whose ledger yields no digest for the region's declared stamp dataset (stamp_dataset(region)— area-majority forall,<region>_regionotherwise, no fallback).So this issue's "produces a lookup parquet identical to the current one … + embedded provenance metadata" should now read: table contents identical, metadata including the
lookup_versionkey the pandas-free builder must reproduce byte-for-byte.Also:
_provenancenow takes a requireddatasetskeyword, and_read_versionno longer exists anywhere — the reader iscontract/gaul_lookup.version(), which reads one key and raises.feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop the pan…
feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop the pan…
feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop pandas …
Merge pull request #220 from views-platform/feat/90-pyarrow-native-lo…
Polichinel commentedon Aug 4, 2026
Done in #220 — the builder is arrow-native.
Proven against the real producer rather than asserted. Rebuilt
land_gaulfrom the views-datafactory checkout and compared to the committed artifact: 64,742 rows both sides, identical column order, identical schema types includingdictionary<string>on the four name columns, all ten columns value-identical, and every declared metadata key byte-for-byte —adr,region,n_cells,n_dropped_incomplete,source_provenance, andlookup_version(land_gaul@f74d3b2b).That is this issue's re-baselined acceptance criterion exactly: contents identical, flat declared
lookup_versionreproduced byte-for-byte.One deliberate difference: no
pandasschema-metadata blob. It described an index arrow does not have, the only consumer reads the file as arrow (contract.gaul_lookup.load), and reproducing it would have meant keeping pandas' metadata format in the one script this change removes pandas from.The committed artifact is not regenerated. The builder is what this issue asked for; rewriting a delivered artifact is a separate act with no delivery benefit.
Two register entries moved with it: C-76 closed (the builder refuses a zero-row result — and the guard failed its own first mutation test, because on an empty table
pa.array([True] * 0)infers NULL type andpc.and_raised before the check could speak), and C-75 decided: retireGaulLookupEnricher— zero production callers, its build/verification justification spent, holding the package's last pandas reference, duplicating a shipping gather that four test files cover independently.That retirement is the next change, not this one — it touches ten files including a CIC and ADR-012, and mixing it with a builder rewrite is what epic #148's S5 refused to do.
#85 and #93 stay open until it lands — their claim only becomes true when that module is gone.