Reconciliation phase 2, slice 1: numpy proportional reconciler + parity by Polichinel · Pull Request #30 · views-platform/views-postprocessing

Polichinel · 2026-06-23T22:45:22Z

Phase 2, slice 1 — bring reconciliation home (the leaf math, parity-proven)

First slice of relocating forecast reconciliation from views-reporting → views-postprocessing (views-reporting issue #72), where it belongs. This is not the full migration — it is the inner algorithm only, ported faithfully and proven bit-for-bit against the frozen original.

What's here

views_postprocessing/reconciliation/proportional.py — reconcile_proportional, a pure-numpy float32 port of views-reporting's ForecastReconciler.reconcile_forecast (torch). Per-draw top-down proportional scaling (FPP3 forecast proportions): zero-preserving, non-negative, handles point and probabilistic inputs, same fail-loud ValueError. No torch, no pandas — so torch leaves with it (resolves views-reporting C-24).
Parity, bit-for-bit. scripts/gen_reconciliation_parity_fixture.py runs the untouched views-reporting oracle over its full test battery + independent-country / negatives / all-zero-draw edge cases and freezes inputs→outputs to a committed .npz. tests/test_reconciliation_parity.py reproduces every case exactly (max abs diff = 0.0) and needs neither torch nor views-reporting in CI.
C-37 logged: this is the pragmatic per-draw method, not principled joint probabilistic reconciliation — the upgrade is deliberately deferred until the move is wired.
Second commit: promotes test_version_bumped_past_latest_tag — views-datafactory#224 was resolved (v1.4.0), the xfail(strict) guard flipped, so it's now a live guard.

What's deliberately NOT here (next slices)

Slice 2: the orchestration (ReconciliationModule), country↔grid mapping, and end-to-end parity vs the frozen views-reporting pipeline on a real fixture (e.g. red_ranger).
Slice 3: wiring / pipeline-core repoint.
Slice 4: phase out reconciliation in views-reporting; upgrade to principled probabilistic reconciliation (C-37).

Risk / impact

Not wired into any live path → zero delivery impact. views-reporting is untouched (it stays the parity oracle until slice 4).

🤖 Generated with Claude Code

…, slice 1) Bring the forecast-reconciliation algorithm home from views-reporting (their issue #72), starting with the leaf math as pure numpy — no torch, no pandas. reconcile_proportional is a faithful float32 port of ForecastReconciler. reconcile_forecast: per-draw top-down proportional scaling (FPP3 forecast proportions), zero-preserving, non-negative, point + probabilistic. Parity is proven bit-for-bit against the *untouched* views-reporting torch oracle: scripts/gen_reconciliation_parity_fixture.py captures inputs->outputs over their full test battery + independent-country / negatives / all-zero edge cases; tests/test_reconciliation_parity.py reproduces them exactly (max abs diff 0.0) and needs neither torch nor views-reporting (fixture committed). Not wired to any live path (zero delivery impact). Orchestration, country<->grid mapping, end-to-end parity, and the pipeline-core repoint/phase-out are slices 2-4. C-37 records that this is the pragmatic per-draw method, to be upgraded to principled probabilistic reconciliation only after the move is wired. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…solved) datafactory 6f7f4ec bumped to v1.4.0 (untagged), resolving #224. The xfail(strict) guard flipped to a pass, so remove the marker and promote the test to a live guard against a future version<->tag collision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Freeze the whole views-reporting reconciliation pipeline as a committed fixture: build a realistic cm+pgm sample (5 countries, 3 months, 2 targets, 100 samples), run the *untouched* oracle fully offline (inject the country->grids mapping so no viewser; patch WandB; CPU), and save (cm, pg, pg_country, reconciled pgm). The frames-native module must reproduce it (the parity gate). Records D-R1 (group by injected VIEWS country_id, GAUL deferred to S7) and D-R2 (all-zero-draw edge case captured) in docs/reconciliation_migration.md. Closes #32. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Array -> PredictionFrame builders at SpatialLevel.CM (country_id) and PGM (priogrid_gid); predictions carry a real sample axis so values are (N, S). The reconciliation package's own I/O (CRP: distinct from the unfao pandas-scalar adapters). Contract-checked against the S0 fixture; inputs never mutated. Closes #33. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ty (S2, #34) The heart of the migration: label grid rows by country with views-frames cross_level_align (fail-loud on unmapped rows), group by (time, country) via np.unique (no per-row loop -> scales to the full grid), and reconcile each group with the parity-proven leaf reconcile_proportional. Returns a new pgm frame (de-mutated, C-184). Reproduces the S0 oracle fixture BIT-FOR-BIT (max abs diff 0.0) on both targets. Closes #34. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Polichinel · 2026-06-24T00:02:44Z

Extended beyond slice 1: this branch now carries the in-repo migration stories #32 (S0) offline e2e oracle fixture, #33 (S1) cm/pgm PredictionFrame adapters, and #34 (S2) the cross_level_align grouping core — which reproduces the frozen views-reporting pipeline bit-for-bit (max abs diff 0.0) on both targets. S3 (validation), S4 (orchestration), S5 (parity gate), S6 (CIC/docs) to follow on this branch. Tracking: #41.

) Port views-reporting ReconciliationModule.__init__'s guards as small, testable helpers (SRP): spatial-level, sample-count, identical time coverage, and country-coverage checks — each raising a clear ValueError before any work (ADR-003). The "different time units" check is subsumed by the level guard. Closes #35. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…S4/S5, #36 #37) Assemble the proven pieces into the public ReconciliationModule: it holds the injected (time, priogrid_gid)->country_id mapping, validates, and reconciles a cm/pgm frame pair to a NEW pgm frame (de-mutated). SRP orchestration only; wandb and ProcessPoolExecutor dropped (numpy is fast, no GPU; parallelism can later sit behind the same interface, OCP). The module reproduces the frozen views-reporting pipeline BIT-FOR-BIT on every target, with module-level property tests (sum constraint on active draws, all-zero draws stay zero, de-mutation, fail-loud). Closes #36. Closes #37. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…#38) CIC at docs/CICs/ReconciliationModule.md (purpose, non-goals, guarantees, fail-loud failure modes, boundaries, usage, test alignment, evolution incl. the C-37 upgrade) matching the actual module behaviour. Migration doc updated: in-repo migration complete and end-to-end parity-proven; S7/S8 (cross-repo) remain. Closes #38. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Polichinel · 2026-06-24T00:12:38Z

In-repo migration complete. S3 (validation guards, #35), S4 (ReconciliationModule, #36), S5 (end-to-end parity gate, #37), and S6 (CIC + docs, #38) have landed. The public ReconciliationModule reproduces the frozen views-reporting pipeline bit-for-bit on every target, with no torch/pandas/viewser/wandb in the path. Suite: 146 passed / 42 xfailed. Remaining is cross-repo and blocked: S7 (#39) pipeline-core repoint, S8 (#40) views-reporting phase-out.

…condition From the expert-code-review of the reconciliation migration (PR #30): - C-38 (Tier 2, new): reconcile_pgm_to_cm grouping is O(groups x N) (per-group np.nonzero) and materializes the whole pgm frame -> won't scale to global volume. Parity unaffected; fix before wiring (S7). Cross-refs C-31/C-32. - C-37 (merged): added the Kleppmann sample-alignment precondition -- at S7, verify cm/pgm samples are the same joint draws (the parity fixture can't). - Header: 37 -> 38 total, 34 -> 35 open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… (C-38) Address C-38 (expert-code-review) so PR #30 merges scale-ready. reconcile_pgm_to_cm replaced the per-group `np.nonzero(inverse == gi)` (O(groups x N), ~86k full-array scans + masks at global volume) with group-by-sort: one `argsort(inverse)` + contiguous slices from `np.unique` counts (O(N log N), one index array). Parity is unchanged -- BIT-EXACT (tests/test_reconciliation_{grouping,e2e_parity}.py -> 0.0); a new scale guard (tests/test_reconciliation_scale.py) reconciles 2000 groups and checks conservation. Memory: the module holds the whole frame at once (inherent to a whole-frame reconcile); at global volume the caller chunks by time -- documented in the CIC (memory contract) and migration doc, to be verified on a global dry-run at S7. C-38 compute marked RESOLVED; memory residual narrowed to the S7 gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Polichinel · 2026-06-24T00:55:23Z

Scale-ready (C-38 addressed). Grouping rewritten to group-by-sort (argsort + contiguous slices) — O(groups × N) → O(N log N), with parity unchanged (bit-exact, 0.0) and a new scale guard (tests/test_reconciliation_scale.py). Memory is documented as a caller chunk-by-time contract (CIC §5), to be verified on a global dry-run at S7. C-38 compute is RESOLVED in the register. Suite: 147 passed / 42 xfailed, ruff clean.

…as an unmerged branch (#198) S3 (#184/#195) pinned the Appwrite Seam Contract at a commit resolved with `git -C ../views-appwrite rev-parse HEAD`. That checkout was sitting on the unmerged `feat/s1-single-writer-rule` branch, so the pin captured a commit that never reached `main` and has now been withdrawn (views-appwrite #30, #27). It declared registry v1.4.0; main carries the ratified v1.3.0. Verified before repinning, not taken on the issue's word: the withdrawn commit is reachable only from that feature branch, 47172af is origin/main's tip, its registry declares version = "1.3.0" (RATIFIED — þing-02), and both cited files exist at it. Four references repointed — appwrite_env.py's docstring and ADR-013 §7d's two links — plus the version labels, the §7d correction paragraph, and Erratum E2, all of which asserted a 1.3.0 -> 1.4.0 move that never landed. The withdrawn sha is deliberately not repeated anywhere, including in the prose narrating its withdrawal, so that grepping for it finds nothing. #196's acceptance criteria are met literally. The lesson is narrow and now recorded in both places a future contributor will look: resolve a cross-repo pin from the tip of the other repo's main, never from whatever its working copy has checked out. A pin is a claim about what the contract says; HEAD answers a different question — what someone was working on. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Polichinel and others added 2 commits June 24, 2026 00:44

Polichinel and others added 3 commits June 24, 2026 02:01

Polichinel and others added 3 commits June 24, 2026 02:12

Polichinel and others added 2 commits June 24, 2026 02:33

Polichinel merged commit 2f2d5fd into development Jun 24, 2026
4 checks passed

Polichinel deleted the feature/reconciliation-numpy-core branch June 26, 2026 21:38

Polichinel mentioned this pull request Aug 2, 2026

#196 — repin views-appwrite to main's tip; the first pin was an unmerged branch #198

Merged

Navigation Menu

Reconciliation phase 2, slice 1: numpy proportional reconciler + parity - #30

Reconciliation phase 2, slice 1: numpy proportional reconciler + parity#30
Polichinel merged 10 commits into
developmentfrom
feature/reconciliation-numpy-core

Polichinel commented Jun 23, 2026

Uh oh!

Polichinel commented Jun 24, 2026

Uh oh!

Polichinel commented Jun 24, 2026

Uh oh!

Polichinel commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Polichinel commented Jun 23, 2026

Phase 2, slice 1 — bring reconciliation home (the leaf math, parity-proven)

What's here

What's deliberately NOT here (next slices)

Risk / impact

Uh oh!

Polichinel commented Jun 24, 2026

Uh oh!

Polichinel commented Jun 24, 2026

Uh oh!

Polichinel commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant