Repository navigation
S4 — enrichment.py: numpy/pyarrow keyed gather (drop pandas merge) #89
Description
Epic: #85 · S4 · [UNILATERAL — low priority]
Background
GaulLookupEnricher is OWN-CHOICE pandas: it loads the GAUL lookup via pd.read_parquet (enrichment.py:48) and attaches metadata with base.merge(left_on=pg_id_col, right_index=True, how="left") (enrichment.py:117-119). This is a keyed metadata-attach join, not frame algebra. It does not block samples (it attaches geo columns to whatever rows exist), so this is low priority.
Work
- Replace
pd.read_parquet(lookup)+.mergewith a pandas-free keyed gather: load the lookup once into apriogrid_gid → rownumpy/dict structure (viapyarrow.parquet, already imported for_read_version), then gather metadata by cell id, producing NaN/sentinel for misses to preserve the fail-loud null behaviour. - Explicitly do NOT push this into views-frames —
PredictionFramecarries values + aSpatioTemporalIndex, not arbitrary GAUL columns; modelling a 9-column geographic attach as a frame op would distort the value-object contract. The target is plain numpy/arrow keyed gather.
Acceptance criteria
- Enricher no longer imports/uses pandas; output is column-for-column identical to the merge path, including NaN positions for unmapped cells.
- The
country_iso_a3.isna()warning path fires identically.
Parity / validation
tests/test_enrichment.py + tests/test_append_metadata.py as the oracle: feed identical inputs to old merge vs new gather, assert equality incl. NaN positions and dtypes.
Dependencies
Independent of S1–S3 (can land any time). Keep gaul_schema.py (the 9-column contract) as the single source of truth; coordinate with S5.
Files
views_postprocessing/unfao/enrichment.py, tests/test_enrichment.py, tests/test_append_metadata.py.
Activity
Polichinel commentedon Jul 27, 2026
Scope update (2026-07-27): the historical half of this issue is delivered by #126/PR #129 — geography attaches via a pyarrow searchsorted gather from the ADR-011 lookup (
unfao/historical.py), no pandas merge. Remaining scope here: the legacy forecast path'senrichment.pymerge only, which is slated for deletion with that path post-run-0 — at which point this issue likely closes as moot rather than implemented. Re-assess after run 0.🤖 Generated with Claude Code
docs: S8 — epic #148 closeout; Cluster L closed (#166)
Polichinel commentedon Aug 2, 2026
Stale paths — re-read against the tree before starting.
Epic #148's #153 moved this module:
unfao/enrichment.py→views_postprocessing/contract/enrichment.py.And
_read_version, which this issue cites as "already imported for" the pyarrow dependency, moved out of the enricher entirely in #152 — it lives incontract/gaul_lookup.pyas the publicversion(), and epic #181's S5 (#186) rewrites it again.GaulLookupEnricher.lookup_versionnow delegates there.The substance of this story is unaffected — the pandas merge at what is now
contract/enrichment.pyis still a keyed metadata-attach join. Only the coordinates changed.feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop the pan…
feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop the pan…
feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop pandas …
Polichinel commentedon Aug 3, 2026
Landed in #210.
The enricher's lookup side is pandas-free: arrow read, one
takesort,np.searchsortedgather. This unblocks #90 — the old path didpd.read_parquet+right_index=True, which depends on the artifact carrying pandas index metadata that a pyarrow-native builder will not write. Doing #90 first would have broken the merge silently.Two full
/code-review maxpasses, eleven findings, all fixed. The ones worth remembering:development, found because the parity proof usedcheck_dtype=Falseandreset_index, which hide exactly what a merge→assign rewrite changes;54220.000000001matched real cell 54220. The pandas merge returns null there and even warns;2**63-1is odd and unrepresentable in float64, so comparing against it rounds up and admitted exactly what it excluded;timedelta64slipped throughisinstance(..., np.integer), which is True for it and False fordatetime64;test_validation.pytwo days earlier, committed again in the PR boasting about catching its cousin;39 tests, every guard mutation-proven. Registered C-75 (this class has no production caller and now holds a second copy of the delivery path's gather) and C-76 (the builder writes an empty lookup without complaint).