From e94e3be34f9e1673022dbd045d113346743c5aa8 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Wed, 24 Jun 2026 07:39:16 +0200 Subject: [PATCH] docs(register): reframe C-37/C-38 as near-term preconditions for probabilistic reconciliation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-repo investigation re-aimed the reconciliation effort: FAO (rusty_bucket) sums by construction and needs no reconciliation, but an imminent UN-agency deliverable is a frames-based grid ensemble that DOES reconcile against a CM model with ~1024 pooled draws — i.e. probabilistic, not pgm_cm_point. The reconciler is already probabilistic-ready, so: - C-37 (sample-alignment): now an active design gate for that deliverable (grid draws -> country point total vs aligned country draws), gating pipeline-core#200. - C-38 (memory residual): the chunk-by-time obligation is now near-term, to be honored + dry-run-measured by pipeline-core#200's PFE consumer. Append-only notes; counts unchanged (41/24/17). Co-Authored-By: Claude Opus 4.8 --- reports/technical_risk_register.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index f66318f..5fcdc48 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -433,6 +433,8 @@ See also the migration plan (reconciliation slices 2-4) and views-reporting issu **Update 2026-06-24 (expert-code-review, Kleppmann lens):** the per-draw index-pairing is only *valid* if the production cm and pgm forecasts are the **same joint posterior draws**. If they come from independent models (separate posteriors), pairing draw *s* of the grid with draw *s* of the country is arbitrary and the reconciled uncertainty is meaningless — and the parity fixture cannot detect this, because it manufactures aligned draws. **At S7 (#39) wiring, verify the sample-alignment assumption against the real pipeline as a hard precondition** (or escalate the C-37 upgrade). This is a correctness precondition distinct from the "is the method principled" question. +**Update 2026-06-24 (reframe — now near-term, not deferred):** this is no longer a someday concern. FAO (`rusty_bucket`) sidesteps reconciliation entirely (pure-grid ensemble aggregated *up* — sums by construction), but the **next UN-agency deliverable** is an FAO-like grid ensemble that **does** reconcile against a CM model, with **full pooled draws (~1024)** → **probabilistic** reconciliation. The reconciler is already probabilistic-ready (fully vectorized over samples), so the open question is purely C-37's: does that deliverable reconcile grid draws to a country **point total** (well-defined, no alignment needed) or to country **draws** (needs a defined draw-alignment — and today CM models are point-only / independently trained, so no aligned draws exist)? **This decision gates the probabilistic-reconciliation-on-`PredictionFrameEnsembleManager` work** (pipeline-core#200, under epic #193); resolve it once the UN models / CM target are defined. + --- ### C-38: Reconciliation grouping is O(groups × N) and materializes the whole grid frame — won't scale to global volume @@ -451,6 +453,8 @@ Tier 2: structural fragility under the realistic change of wiring to global, wit **Update 2026-06-24 — compute RESOLVED.** The per-group `np.nonzero(inverse == gi)` was replaced with **group-by-sort** (`argsort(inverse)` + contiguous slices from `np.unique` counts, O(N log N), one index array). Parity stays **bit-exact** (`tests/test_reconciliation_grouping.py`, `test_reconciliation_e2e_parity.py` → 0.0) and a scale guard (`tests/test_reconciliation_scale.py`) protects against regression. **Residual (still open):** the module holds the whole pgm frame in memory at once; at global volume the **caller must chunk by time** (reconciliation is independent across months) — the chunk-by-time contract is documented in the CIC (`docs/CICs/ReconciliationModule.md` §5), to be **verified on a global-volume dry-run at S7 (#39)**. This entry stays open until that verification. +**Update 2026-06-24 (reframe — residual now near-term):** the memory residual is no longer "verify someday." The upcoming UN-agency deliverable reconciles **frames with ~1024 pooled draws** — squarely in the >100 GB-at-global regime. When pipeline-core's `PredictionFrameEnsembleManager` wires probabilistic reconciliation (pipeline-core#200, under epic #193), the **caller must chunk by time** (reconciliation is independent across months) and **measure peak memory on a global-volume dry-run** as part of that work. The reconciler code itself is unchanged (compute already O(N log N)); this is a consumer-side obligation. + --- ### C-40: FAO delivery logic fused to pipeline-core via double inheritance + interleaved infrastructure