Reconciliation phase 2, slice 1: numpy proportional reconciler + parity

docs/CICs/ReconciliationModule.md

@@ -0,0 +1,118 @@
+# Class Intent Contract: ReconciliationModule
+
+**Status:** Active
+**Owner:** PRIO MD&D Team
+**Last reviewed:** 2026-06-24
+**Related ADRs:** ADR-003 (fail loud), views-frames ADR-014 (injected cross-level mapping); epic #31, migration `docs/reconciliation_migration.md`, origin #3 / views-reporting#72
+
+---
+
+## 1. Purpose
+
+> Make PRIO-GRID-month (pgm) forecasts consistent with country-month (cm) totals: within each `(time, country)`, scale the country's grid cells so their per-draw sum equals the country forecast, preserving each cell's relative share and its zeros.
+
+It is the frames-native, numpy-only home of the reconciliation that previously lived in views-reporting (`ForecastReconciler` + `ReconciliationModule`), ported parity-preserving.
+
+---
+
+## 2. Non-Goals (Explicit Exclusions)
+
+- Does **not** embed or fetch geography. The `(time, priogrid_gid) -> country_id` mapping is **injected** at construction (views-frames ADR-014); the class never queries viewser, shapefiles, or the GAUL lookup.
+- Does **not** change the algorithm. It is **top-down proportional scaling per posterior draw** (FPP3 forecast proportions), a faithful port — **not** principled joint probabilistic reconciliation (the upgrade is **C-37**, deferred).
+- Does **not** load, save, or upload data; does **not** depend on pandas, torch, viewser, or wandb.
+- Does **not** mutate its inputs.
+
+---
+
+## 3. Responsibilities and Guarantees
+
+- Validates inputs fail-loud before any work (level, sample-count, time coverage, country coverage).
+- For each `(time, country)`: scales grid cells so their per-draw sum **equals** the country forecast; preserves zero cells; clamps to non-negative.
+- Returns a **new** pgm `PredictionFrame` with the same index/metadata as the input grid frame (de-mutation, C-184).
+- Bit-for-bit reproduces the frozen views-reporting pipeline on the parity fixture.
+
+---
+
+## 4. Inputs and Assumptions
+
+- Constructed with `map_keys` `(M, 2)` `(time, priogrid_gid)` and `map_vals` `(M,)` `country_id` covering every grid row.
+- `reconcile(cm_frame, pgm_frame)`: a CM-level `PredictionFrame` (`country_id` units) and a PGM-level one (`priogrid_gid` units), **same** sample count `S` and **same** set of times. One target per call (multi-target = call per target).
+- Time identifiers are opaque integers (`month_id`); country totals are authoritative.
+
+---
+
+## 5. Outputs and Side Effects
+
+- Output: a new pgm `PredictionFrame` `(N, S)`, reconciled. **No** side effects (no I/O, no logging of data, no global state).
+- **Memory ∝ frame size.** Grouping is `O(N log N)` (group-by-sort; register C-38), but the whole frame is held in memory at once — peak ≈ input + output ≈ `2·N·S·4` bytes. At global volume (`land` region) the **caller must chunk by time**: reconciliation is independent across months, so call `reconcile` per month-slice and write each result out rather than materialising the global frame. (C-38; verified on a global dry-run at S7, #39.)
+- **Approximate where flagged:** for a draw in which *all* of a country's grid cells are zero, there are no proportions to distribute, so those cells stay zero and that draw's total is not conserved (the algorithm's documented edge case). Uncertainty is reconciled per-draw, which is a pragmatic approximation (C-37).
+
+---
+
+## 6. Failure Modes and Loudness
+
+Raises `ValueError` (never silently degrades) when:
+- `map_keys` is not `(M, 2)` or `map_vals` is not length `M` (constructor);
+- a frame is at the wrong `SpatialLevel`;
+- cm and pgm sample counts differ;
+- cm and pgm cover different time steps;
+- a grid row's `(time, priogrid_gid)` is absent from the mapping (raised by `cross_level_align`);
+- a `(time, country)` group has no matching country forecast in `cm_frame`.
+
+Aligns with ADR-003: ambiguity fails loud, before computation.
+
+---
+
+## 7. Boundaries and Interactions
+
+- **Trusts:** the leaf `reconcile_proportional` (the math), `grouping.reconcile_pgm_to_cm` (the cross-level grouping/scatter), `validation` (the guards), `frames` (array↔frame I/O), and `views_frames` (`PredictionFrame`, `SpatioTemporalIndex`, `cross_level_align`).
+- **Must not depend on:** pandas, torch, viewser, wandb, the unfao delivery code, or any geography source.
+- The injected mapping is treated as opaque, caller-owned truth.
+
+---
+
+## 8. Examples of Correct Usage
+
+```python
+from views_postprocessing.reconciliation import ReconciliationModule
+
+rm = ReconciliationModule(map_keys, map_vals)   # injected (time, pgid) -> country_id
+reconciled_pgm = rm.reconcile(cm_frame, pgm_frame)   # one target; new frame
+```
+
+Multi-target: call `rm.reconcile(cm_t, pgm_t)` once per target.
+
+---
+
+## 9. Examples of Incorrect Usage
+
+- Constructing it and expecting it to *derive* the country mapping (it never does — inject it).
+- Passing a pgm frame where a cm frame is expected, or frames with different sample counts / times (raises, by design — do not pre-pad or coerce to silence it).
+- Reusing it as a generic disaggregator for non-reconciliation tasks.
+
+---
+
+## 10. Test Alignment
+
+- **Parity (gate):** `tests/test_reconciliation_e2e_parity.py` — the module reproduces the frozen oracle (`tests/fixtures/reconciliation_e2e_parity.npz`) bit-for-bit on every target.
+- **Unit:** `tests/test_reconciliation_{frames,grouping,validation}.py` — adapters, grouping core, and each fail-loud guard.
+- **Leaf parity:** `tests/test_reconciliation_parity.py` — `reconcile_proportional` vs the torch oracle.
+- **Scale:** `tests/test_reconciliation_scale.py` — conservation holds across thousands of `(time, country)` groups (guards the group-by-sort logic; C-38).
+- Regression-protected: bit-exact parity, zero-preservation, de-mutation, every `ValueError` guard, and grouping correctness at scale.
+
+---
+
+## 11. Evolution Notes
+
+- **Stable:** the injected-mapping contract, the fail-loud guards, the de-mutated return.
+- **Expected to change:** the *algorithm* — the principled probabilistic upgrade (**C-37**) will arrive as a sibling method behind this same interface (OCP); when it does, this contract's §2/§5 approximation notes must be revisited.
+- The production mapping **source** (viewser-derived `country_id` vs the GAUL lookup) is decided at wiring time (S7, #39) and does not change this class's contract.
+
+---
+
+## End of Contract
+
+This document defines the **intended meaning** of `ReconciliationModule`.
+
+Changes to behavior that violate this intent are bugs.
+Changes to intent must update this contract.

docs/reconciliation_migration.md

@@ -0,0 +1,64 @@
+# Reconciliation migration — notes & decisions
+
+Tracks the migration of forecast reconciliation from views-reporting into this
+repo. Epic: **#31**; tracking checklist: **#41**; origin: **#3** / views-reporting#72.
+Story S6 (#38) formalises this into the `ReconciliationModule` CIC.
+
+## Status
+
+- **Slice 1 (PR #30) — done.** Leaf algorithm `reconcile_proportional`
+  (`views_postprocessing/reconciliation/proportional.py`), pure numpy, **bit-exact**
+  parity vs the views-reporting torch oracle (`tests/test_reconciliation_parity.py`).
+- **S0 (#32) — done.** End-to-end oracle fixture captured offline
+  (`tests/fixtures/reconciliation_e2e_parity.npz`, via
+  `scripts/gen_reconciliation_e2e_fixture.py`). Decisions below.
+- **S1–S5 (#33–#37) — done.** The frames-native module is complete and
+  **end-to-end parity-proven**: `cm/pgm` adapters (`reconciliation/frames.py`),
+  the `cross_level_align` grouping core (`reconciliation/grouping.py`), fail-loud
+  validation (`reconciliation/validation.py`), and the public
+  `ReconciliationModule` (`reconciliation/module.py`) — which reproduces the
+  frozen views-reporting pipeline **bit-for-bit** on every target
+  (`tests/test_reconciliation_e2e_parity.py`). No torch / pandas / viewser / wandb.
+- **S6 (#38) — done.** CIC at `docs/CICs/ReconciliationModule.md`.
+- **C-38 (scale) — compute fixed.** The grouping is now `O(N log N)` (group-by-sort
+  in `reconciliation/grouping.py`); parity stays bit-exact and a scale guard
+  (`tests/test_reconciliation_scale.py`) protects it. Residual: global peak memory
+  is bounded by caller-side **chunk-by-time** (documented in the CIC), verified at S7.
+- **In-repo migration complete.** Remaining: S7 (#39) pipeline-core repoint and
+  S8 (#40) views-reporting phase-out — both cross-repo, **blocked** on this
+  landing; and the principled-algorithm upgrade (**C-37**), a separate epic.
+
+## D-R1 — Group by injected VIEWS `country_id`, not GAUL (for parity)
+
+The frozen oracle groups grid cells by **VIEWS `country_id`** (from viewser's
+`country_month` LOA). This repo's GAUL lookup (`data/gaul_lookup.parquet`) numbers
+countries by **`admin1_gaul0_code`** — a *different* id system. The migration is
+**parity-preserving**, so the frames-native module groups by the **same VIEWS
+`country_id`**, **injected** by the caller (the leaf never embeds geography —
+views-frames ADR-014). The fixture bypasses viewser by pre-setting
+`pg_ds._country_to_grids_cache`.
+
+> **Deferred (S7, #39):** whether the *production* mapping should eventually come
+> from our GAUL lookup instead of viewser is a separate decision, taken at wiring
+> time. It does **not** affect parity and is out of scope until the migration is
+> wired and proven.
+
+## D-R2 — The fixture deliberately includes the all-zero-country-draw edge case
+
+When *every* grid cell of a country is zero for a posterior draw, proportional
+scaling has no proportions to distribute, so the oracle leaves those cells **zero**
+(country total not conserved for that draw — the algorithm's documented edge case).
+The S0 fixture's sparsity (~30% zeros, small countries) produces such draws
+(178 across the battery), captured verbatim. The frames-native module must
+**reproduce this behaviour** (parity, not "correctness"); improving it belongs to
+the principled-reconciliation upgrade (**C-37**), not this migration.
+
+## Parity oracle (how the fixture is made)
+
+`scripts/gen_reconciliation_e2e_fixture.py` builds a realistic cm + pgm sample
+(5 countries of varying size, 3 months, 2 targets, 100 samples), injects the
+`country_id` mapping, runs the **untouched** views-reporting `ReconciliationModule`
+on CPU with WandB patched out, and freezes `(cm, pg, pg_country, recon)` to npz.
+It needs pipeline-core + views-reporting + torch **only at generation time** (the
+`views_pipeline` conda env); the committed fixture is consumed offline (numpy only),
+so CI needs none of them.

reports/technical_risk_register.md

@@ -4,9 +4,9 @@
 |-------------------|--------------------------------------|
 | Project           | views-postprocessing                 |
 | Owner             | Dylan Pinheiro / PRIO MD&D Team      |
-| Last Updated      | 2026-06-22                           |
-| Total Concerns    | 36                                   |
-| Open Concerns     | 33                                   |
+| Last Updated      | 2026-06-24                           |
+| Total Concerns    | 38                                   |
+| Open Concerns     | 35                                   |
 | Resolved Concerns | 3                                    |
 
 ---
@@ -617,6 +617,42 @@ See also C-01 (null-validation re-enabled — but it does not check code *validi
 
 ---
 
+### C-37: Reconciliation uses a pragmatic per-draw approximation, not principled probabilistic reconciliation
+
+| Field | Value |
+|-------|-------|
+| ID | C-37 |
+| Tier | 3 |
+| Source | `manual` (2026-06-24) — phase-2 reconciliation migration |
+| Trigger | When reconciliation is wired into a delivery and its uncertainty is consumed (intervals, scores), verify the method is the principled one — the current per-draw scaling can distort the joint predictive distribution |
+| Location | `views_postprocessing/reconciliation/proportional.py` |
+
+`reconcile_proportional` is a faithful numpy port of views-reporting's `ForecastReconciler.reconcile_forecast`: **top-down disaggregation using forecast proportions** (FPP3), applied **per posterior draw**. It rescales each marginal draw independently to hit that draw's country total, which implicitly assumes the grid and country samples are index-aligned joint draws. This is a pragmatic approximation, **not** principled joint probabilistic reconciliation (the IJF paper, PII `S0169207023001097` — exact title TBC; cf. FPP3 §reconciliation), under which the reconciled draws would be coherent samples from a single reconciled joint distribution (e.g. MinT-style projection on samples). The migration deliberately preserves the existing method first (parity proven bit-for-bit against the untouched views-reporting oracle, `tests/test_reconciliation_parity.py`); the upgrade is **gated behind** completing the move and wiring (slices 2-3) so behaviour change and relocation never mix. Until then, treat reconciled uncertainty as approximate.
+
+See also the migration plan (reconciliation slices 2-4) and views-reporting issue #72 (the relocation).
+
+**Update 2026-06-24 (expert-code-review, Kleppmann lens):** the per-draw index-pairing is only *valid* if the production cm and pgm forecasts are the **same joint posterior draws**. If they come from independent models (separate posteriors), pairing draw *s* of the grid with draw *s* of the country is arbitrary and the reconciled uncertainty is meaningless — and the parity fixture cannot detect this, because it manufactures aligned draws. **At S7 (#39) wiring, verify the sample-alignment assumption against the real pipeline as a hard precondition** (or escalate the C-37 upgrade). This is a correctness precondition distinct from the "is the method principled" question.
+
+---
+
+### C-38: Reconciliation grouping is O(groups × N) and materializes the whole grid frame — won't scale to global volume
+
+| Field | Value |
+|-------|-------|
+| ID | C-38 |
+| Tier | 2 |
+| Source | `expert-code-review` (2026-06-24) |
+| Trigger | Before the first global / `land`-region reconciliation run — i.e. before wiring at S7 (#39) — benchmark `ReconciliationModule.reconcile` runtime and peak memory on global-volume frames; nothing above the 39-row fixture has been measured |
+| Location | `views_postprocessing/reconciliation/grouping.py:69-79` (per-group `np.nonzero(inverse == gi)`); `views_postprocessing/reconciliation/module.py` (holds the full pgm frame; `np.empty_like` copy) |
+
+`reconcile_pgm_to_cm` groups grid rows with `np.unique` (good) but then loops over unique `(time, country)` groups doing `np.nonzero(inverse == gi)` **per group** — an O(groups × N_pg) full-array scan. At global scale (~86k groups × ~28M pgm rows) that is ~10¹² comparisons plus 86k full-size boolean masks. Separately, the module holds the **entire** pgm frame at once (28M rows × S samples × 4 bytes ≈ 11 GB at S=100, **>100 GB at S=1000**) and `np.empty_like` doubles it — the original views-reporting code processed per-country subsets, never materialized the global frame, and used `ProcessPoolExecutor` for exactly this scale. **Parity is unaffected** (the result is identical); only runtime/memory blow up. Mitigation: replace the per-group `nonzero` with a single `argsort(inverse)` + contiguous slices (O(N log N)); budget/measure peak memory on a global-volume dry run and chunk by time or country if needed — both **before** wiring. Same enumerable-vs-discovered-at-scale pattern as C-31/C-32.
+
+Tier 2: structural fragility under the realistic change of wiring to global, with a clear trigger; not Tier 1 (no silent corruption — parity is exact; this is a runtime/memory failure). See also C-31 (mapper scale), C-32 (enricher memory), C-37 (the algorithm), epic #31 / views-reporting#72.
+
+**Update 2026-06-24 — compute RESOLVED.** The per-group `np.nonzero(inverse == gi)` was replaced with **group-by-sort** (`argsort(inverse)` + contiguous slices from `np.unique` counts, O(N log N), one index array). Parity stays **bit-exact** (`tests/test_reconciliation_grouping.py`, `test_reconciliation_e2e_parity.py` → 0.0) and a scale guard (`tests/test_reconciliation_scale.py`) protects against regression. **Residual (still open):** the module holds the whole pgm frame in memory at once; at global volume the **caller must chunk by time** (reconciliation is independent across months) — the chunk-by-time contract is documented in the CIC (`docs/CICs/ReconciliationModule.md` §5), to be **verified on a global-volume dry-run at S7 (#39)**. This entry stays open until that verification.
+
+---
+
 ## Disagreements
 
 ### D-01: Cache strategy refactoring — extract now vs. characterize first