Epic: FAO delivery input-integrity — complete-coverage contract + no silent bad data

[Polichinel]

Parent: umbrella #20. Source: the input-integrity cluster (register C-25/C-26/C-30/C-34/C-15) hardened by two expert-code-reviews + a /falsify pass (2026-06-25/26).

Problem

The FAO delivery trusts its upstream inputs with no completeness or identity contract. The only gate (unfao.py:_validate, L147-172) checks the 9 GAUL metadata columns for nulls — nothing else. So bad data reaches FAO silently: fabricated zeros from missing months (C-26, Tier 1), the wrong forecast file (C-25), unassigned cells at global scale (C-30, Tier 1), partial coverage from a wrong region (C-34), and no provenance to audit any of it (C-15).

Why it matters

Two Tier-1 risks deliver wrong numbers a partner acts on, with no error signal — and especially before the imminent global switch (REGION africa_me_legacy → land_gaul, views-models#127). The fail-loud mechanism already exists (the enricher left-merges → NaN → crash, and logs unmapped counts); this epic adds the missing contracts, built right.

Desired end state

• Every delivered cell is verified complete for the configured region; observed history is never fabricated; the forecast input's identity is verified; the delivery carries structured provenance.
• All guards are representation-agnostic invariants in a reusable views_postprocessing/delivery/ package; pandas is isolated to one views_postprocessing/unfao/extraction.py seam; guards are called by — never methods of — the manager.
• The pipeline is ready for the land_gaul global switch with no silent-bad-data paths.

Design contract — NON-NEGOTIABLE (every story obeys it; enforced by tests/test_input_integrity_design_contract.py)

① Two homes. Representation-free invariants + constants in delivery/ (partner-agnostic, reusable by the coming UN agencies); the pandas→primitives seam in unfao/extraction.py (FAO-local). Contract and representation never share a module.
② Primitives are the abstraction. Invariants take set[int] / np.ndarray / scalars / dicts (DIP). Extraction isolated in one module; pandas→frames swap = a new extraction module, invariants untouched (OCP). No premature Extractor Protocol (YAGNI/ISP) — it's a migration, not a coexistence.
③ Called, never inherited. Guards live outside the inherited class; the manager does extract → call pure guards → raise. The manager's LSP/SDP/SAP violation is DEFERRED to C-40 — this epic does not claim to fix it.

Scope

✅ In: the delivery/ scaffold + unfao/extraction.py seam; the five guards (coverage, observed-range, identity, land_gaul-pin, provenance); package hygiene (delete orphaned unfao/frames.py + dead unfao/mapping/; fix the "frame" name overload); end-to-end tests.
❌ Out: the pandas→views-frames migration of the delivery (C-40 / pipeline-core-gated); the manager de-inheritance (C-40); the datafactory NaN refactor; the per-cell coverage mask (Case B — deferred behind a tripwire); timeouts (C-13/#11, C-28) and the multi-store refactor (C-33) — different clusters.

Stories (implementation order)

• S0 — Scaffold + package hygiene (new — foundational)
• S1 — Coverage contract (S1 — Coverage contract: complete-coverage invariant + cell-count logging (C-34) #53, C-34)
• S2 — Observed-range / no fabricated zeros (S2 — Observed-range guard: no fabricated zeros, clip to last_valid_month_id (C-26) #52, C-26 — decided splint; companion-blocked)
• S3 — Forecast-file identity (S3 — Forecast-file identity invariant (C-25) #54, C-25)
• S4 — land_gaul completeness pin (S4 — land_gaul completeness pin: 64,736 + 82 excluded gids + FAO disclosure (C-30) #55, C-30 — after S1)
• S5 — Structured provenance (S5 — Structured delivery provenance invariant (C-15, supersedes #12) #56, C-15)
• S6 — End-to-end input-integrity tests (S6 — End-to-end input-integrity test suite (through the manager seam) #57)

Dependencies

S0 ──► S1 ──► S4
   ├─► S2 (also blocked on a pipeline-core companion: surface last_valid_month_id)
   ├─► S3
   └─► S5
{S1..S5} ──► S6

Epic acceptance criteria

• delivery/ holds representation-free invariants (no import pandas anywhere under it); unfao/extraction.py is the only representation-aware code; guards are called, not inherited.
• The five checks are enforced: complete coverage (S1), no fabricated observed months (S2), forecast identity (S3), land_gaul 64,736 + 82 (S4), structured provenance (S5).
• Package hygiene done (orphaned frames.py + dead mapping/ gone; no "frame" name overload).
• tests/test_input_integrity_design_contract.py stubs have flipped from xfail to real assertions; full suite green; ready for the land_gaul switch.

[Polichinel]

Stories created (2026-06-24): #52 (S1, C-26, decide) · #53 (S2, C-34, coverage) · #54 (S3, C-25, forecast identity) · #55 (S4, C-30, land_gaul pin) · #56 (S5, C-15, provenance) · #57 (S6, tests). Tracking checklist: #58. Branch: sprint/fao-input-integrity.

[Polichinel]

Design contract for the sprint (post-/falsify, 2026-06-26)

/falsify showed the first cut of the "primitives + adapter" direction only covered half the rubric (silent on file-structure; overclaimed host alignment). The corrected contract — every guard in this epic (#52–#57) must follow it:

① Two homes — separate the rule from the representation.

• views_postprocessing/delivery/ — partner-agnostic, representation-free: the invariants as pure functions over primitives (set[int], np.ndarray, scalars, dicts) + constants (region→count, the 82 gids). Reusable by FAO and the coming UN agencies. (CRP/REP/SRP/screaming)
• views_postprocessing/unfao/extraction.py — FAO-local, representation-specific: the thin pandas→primitives seam. A representation change touches only this file. (CCP)
• Contract and representation never share a module.

② Primitives are the abstraction — no premature Protocol.
Invariants take primitives (DIP); extraction isolated in one module; pandas→frames swap = a new extraction module, invariants untouched (OCP). A formal Extractor Protocol only if pandas+frames must coexist — they won't (migration). YAGNI.

③ Called, never inherited.
Guards are called by the manager, never methods of it — they live outside the inherited class, so the C-40 de-inheritance won't touch them. The manager's LSP/SDP/SAP violation is explicitly DEFERRED to C-40; this sprint does not claim to fix it.

Housekeeping (fold into the sprint): fix the "frame" name overload in enrichment.py; delete the dead unfao/mapping/ husk.

Enforcement: tests/test_input_integrity_design_contract.py (xfail-strict) encodes all of the above; each stub flips RED when its piece is built correctly, forcing the contract to become a real assertion.

[Polichinel]

Input-integrity sprint S0–S6: implemented + green on `sprint/fao-input-integrity`

All guards built to the design contract — representation-free invariants in `delivery/`, fed by the `unfao/extraction.py` seam, called by the manager (never inherited; C-40 de-inheritance stays deferred). Full suite 126 passed, 44 xfailed; the only remaining input-integrity xfail is the C-40 manager-inheritance contract.

Story	Guard	Commit
S1 / #53	coverage contract (C-34)	`72fda59`
S2 / #52	observed-range clip (C-26)	`72fda59`
S3 / #54	forecast-file identity (C-25)	`f325cf3`
S4 / #55	land_gaul exclusion manifest (C-30)	`0bb03b3`
S5 / #56	structured provenance (C-15)	`c4317dc`
S6 / #57	e2e suite through the manager seam	`cac5f0d`

Corrections surfaced during build:

• S4: the pinned `64,736 + 82` was stale — live producer (datafactory v1.4.0) is 64,742 + 76 (datafactory refactor(structure): S5 — separate partner declarations from delivery machinery #163 / ADR-043 supplemented 6 Azorean cells). vpp's own lookup already shipped 64,742. Corrected everywhere; S4 — land_gaul completeness pin: 64,736 + 82 excluded gids + FAO disclosure (C-30) #55 has detail.
• S5: `upload_data` has no structured field, so provenance is JSON-encoded into `description` for now; a dedicated field is requested at pipeline-core S4: delete the unfiltered arrival check; keep the row differential #245.

Residuals (carried in the register, not blockers): C-25 & C-43 name/equivalence checks verify at the first live run; C-40 de-inheritance deferred.

Next: PR `sprint/fao-input-integrity` → `development`. The stories close on that merge.

[Polichinel]

Delivered and merged to development via #64 (input-integrity sprint S0–S6). ruff clean, 126 passed / 44 xfailed. Closing — the auto-close keyword did not fire because #64 merged into development, not the default branch. Live-run validation of the residual contracts (C-25 identity name/loa, C-43 enrichment equivalence) is tracked in the risk register and exercised by the go-global run (views-models#127).