Skip to content

[Scoping] Second prediction store / partner — sample-bearing, multi-target, cm+pgm with reconciliation #97

Description

@Polichinel

Context

A second prediction store / partner (beyond UN FAO) is coming soon. Per WET-before-DRY (maintainer, explicit): build it as its own partner track — a sibling package to unfao/, deliberately duplicated and quirky. Do NOT generalize delivery/ or abstract a partner framework yet. Generalize only after a 3rd/4th store reveals the real commonality. This issue scopes the requirements + the open architectural decisions; it is not a generalization epic.

This also makes the rename (#96) more obviously right: a second partner = a multi-partner delivery layer, not "postprocessing".

Same as FAO (reuse, don't reinvent)

Different from FAO

  1. Uncertainty products: the delivery must carry exceedance probabilities + HDI + MAP (the views_frames_summarize outputs), not just point estimates.
  2. Two LOAs with dedicated models + reconciliation: FAO's country numbers are aggregated up from PGM. The new store has dedicated country (cm) models AND dedicated grid (pgm) models, plus reconciliation between them so they map. views-frames already supports both levels (SpatialLevel.CM / SpatialLevel.PGM) and reconciliation lives in views_frames_reconcile.

Open architectural decisions (the crux)

  • D1 — where does cm↔pgm reconciliation run?
    Options: (a) upstream, in the forecasting/ensemble stage, so the store holds already-reconciled forecasts (consistent with reconciliation having been moved OUT of this repo into views_frames_reconcile); (b) here, invoking views_frames_reconcile on the delivered cm+pgm frames; (c) the API.
    Recommendation: (a) upstream if the pipeline allows; if the store delivers cm+pgm unreconciled, prefer (b) here over (c) the API — reconciliation is a producer-side data op (D-07), not a serving concern. Either way the algorithm is views_frames_reconcile; the question is the call site, not a re-implementation.

  • D2 — where does summarization (HDI/MAP/exceedance) run?
    Default platform contract (draws-preservation, FAO delivery path is point/DataFrame-based — won't carry rusty_bucket's pooled draws #45) = deliver draws uncollapsed, summarize at the edge (the consuming API). Decide per the new partner's contract: does it ingest draws (like faoapi → API summarizes, this repo unaffected) or want pre-summarized data delivered (→ this repo calls views_frames_summarize before upload)?

Things to do now (WET-aware — no premature abstraction)

Refs

C-33 / D-09 (store identity / DeliveryProfile), #45 (draws / API summarization), epic #85 (#86 S1 done, #87 S2 done — pandas→frames), views-models#143 (rusty_bucket draws), views_frames_reconcile + views_frames_summarize (the algorithms), #96 (rename).

Activity

  1. added
    documentationImprovements or additions to documentation
    needs-decisionRequires a human decision before proceeding
    on Jun 29, 2026
  2. Polichinel commented on Jun 29, 2026

    @Polichinel
    CollaboratorAuthor

    D1 resolved by investigation (2026-06-29): reconciliation is upstream — not this repo's job.

    Sniffed out pipeline-core: cm↔pgm reconciliation runs in the ensemble/forecasting stage, before the prediction store is written:

    • views_pipeline_core/managers/ensemble/prediction_frame_ensemble.py:599 — reconcile_frames(self._reconciler, cm_frame, agg_pf) (frames-native, draws-aware — rescales every grid draw to the cm totals per time step), then save_predictions(...).
    • legacy path: dataframe_ensemble.py via reconcile_datasets (the injected Reconciler port, Decision K).

    So the store holds already-reconciled forecasts; this delivery repo reads two consistent frames and delivers them — no reconciliation here (consistent with reconciliation having been moved out to views_frames_reconcile). The only condition is that the new store's dedicated cm/pgm models run through the frames ensemble manager with a reconciler injected (views-frames ≥1.8) — a views-models / pipeline-core wiring concern, not this repo's.

    Remaining open: D2 only (does the new partner ingest draws and summarize at the edge, or want HDI/MAP/exceedance delivered pre-summarized).

  3. added a commit that references this issue on Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationneeds-decisionRequires a human decision before proceeding

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions