Repository navigation
S2 — Observed-range guard: no fabricated zeros, clip to last_valid_month_id (C-26) #52
Description
Activity
DECISION (2026-06-26): splint now, cure later — "missing is not zero"
The problem in one line
Missing upstream data silently becomes
0("no conflict") and reaches FAO with no signal. It is structural, not occasional: the historical range is requested up to_current_month_id() - 1(a clock boundary,views-models config_partitions.py), but UCDP's real data ends earlier (reporting lag) atlast_valid_month_id. So every run ships the most-recent months as fabricated zeros — exactly the months FAO cares about most.Root cause (why this isn't just "add a check")
The FAO delivery-of-record is produced by model-training machinery whose missing-data rule is the opposite of delivery's: a training loader is supposed to fill blanks with
0(pipeline-core dataloaders.py:1208— correct for fitting models); a record-of-truth must never invent a value. The manager even is-aForecastingModelManager(unfao.py:26) and reads withvalidate=False(unfao.py:49-54). This is the same root as C-40 (delivery fused to the modeling framework by inheritance), seen from the data-semantics side.Decision — two tracks
1. Splint — now, this repo (stops the Tier-1 bleed):
Make delivery enforce one invariant — do not ship "observed" months beyond the real data boundary (last_valid_month_id), and treat missing as missing (fail, don't zero). Concretely: clip/validate the historical delivery to≤ last_valid_month_id; if asked to deliver beyond it, fail loud. This covers Case A (requested range beyond observed) — the common, every-run case.- Dependency — small pipeline-core companion: we need
last_valid_month_idsurfaced to us. datafactory's loader already returns it (dataset.py:217) and pipeline-core receives and discards it (dataloaders.py). So the companion is "stop discardinglast_valid_month_id; pass it through" (+ ideally log a fill count). To be opened when the splint starts.
2. Cure — later, deliberate (NOT delivery week) = C-40:
Give delivery its own data read that treats missing as missing, instead of inheriting the model-training loader (compose a thin observed-record reader + forecast reader + FAO writer; stop being aForecastingModelManager). This dissolves the root for FAO and the 2–3 UN agencies coming next, so we stop re-patching per partner. The splint is removed by the cure.Explicitly NOT doing
- Not the big datafactory "use NaN everywhere" refactor — correct for delivery but breaks every modeling consumer relying on dense 0-padding. Wrong blast radius.
- Not a per-cell coverage mask yet. That addresses Case B (a single unharvested cell inside the valid range) — undetectable downstream today (no mask exists), but rare by how datafactory assembles. Defer behind a cheap tripwire (log per-month zero-fraction) to gather evidence before building mask infrastructure.
Status
- ✅ Strategy decided (splint + cure) —
needs-decisioncleared. - ◻ Splint implementation + the pipeline-core companion: start when the FAO input-integrity sprint (Epic: FAO delivery input-integrity — complete-coverage contract + no silent bad data #51) kicks off; this issue tracks it.
- ◻ Cure tracked under C-40 (register) — scheduled deliberately, with this fabricated-zeros problem as its explicit goal.
Grounded in a cross-repo survey + two expert-code-reviews (2026-06-25/26). Same root as C-40; both are the symptom of delivery built on modeling machinery.
- Dependency — small pipeline-core companion: we need
- changed the title
[-]S1 — Decide the no-fabricated-zeros guard for FAO feature columns (C-26)[/-][+]S2 — Observed-range guard: no fabricated zeros, clip to last_valid_month_id (C-26)[/+]on Jun 26, 2026 UNBLOCKED + built via Option B (2026-06-26)
S2 was never hard-blocked — it only looked blocked because the delivery routes its data through pipeline-core, which receives
last_valid_month_idfrom datafactory and discards it. But the fact is published by the producer: datafactory shipsget_last_valid_month_id(zarr_url)(reads its own.zattrs, 10s timeout). So we read it straight from datafactory, not pipeline-core.Architectural principle recorded (maintainer): data-related facts — data, metadata, validity dates, country/admin codes — come from the producer (views-datafactory, or viewser until phased out), not routed through pipeline-core. pipeline-core is orchestration, not a data pass-through; depending on it for data facts couples the delivery to the unstable hub (SDP/ADP). (Relates to register D-07 — this is the decision for it.)
Built (all green: 120 passed / 44 xfailed, ruff clean)
delivery/observed_range.py— representation-free invariant (fabricated_months,is_observed) + 4 unit tests.unfao/source_metadata.py— the single, named place the delivery reads producer facts from datafactory (lazy import). Currently serveslast_valid_month_id; the S1 region cell-count SSOT belongs here too once datafactory publishes it.unfao/extraction.py—drop_months_above(pandas clip) + tests.unfao.py:_clip_observed_history— wired into_read_historical_data: reads the boundary from datafactory, clips only the historical (observed) frame (forecast is future-dated, untouched), logs the clip count. If the attr is absent → log-skip; a network failure fails loud (datafactory is already a hard dep for the data itself).
Note
The pipeline-core "stop discarding
last_valid_month_id" change is no longer a prerequisite — if it ever lands, we swap the one line insource_metadata/extraction (that's why the seam exists). No pipeline-core companion required.Delivered and merged to
developmentvia #64 (input-integrity sprint S0–S6). ruff clean, 126 passed / 44 xfailed. Closing — the auto-close keyword did not fire because #64 merged intodevelopment, not the default branch. Live-run validation of the residual contracts (C-25 identity name/loa, C-43 enrichment equivalence) is tracked in the risk register and exercised by the go-global run (views-models#127).Note from views-pipeline-core, following an audit of our own fill behaviour (pipeline-core#366, registered as our C-278). This is information for your C-26, not a request to close it — see the last paragraph.
The FeatureFrame path adds no fill.
feature_frame_path.py:19-21deliberately does notfillna, and that is stated in the source. So the frame route through pipeline-core is not a source of fabricated zeros.The legacy DataFrame path did, silently.
dataloaders.py_fetch_data_from_datafactorycalleddf.fillna(0.0)with no record that a substitution happened — and only on that route; the viewser and synthetic routes keep their NaN. That asymmetry was undocumented. It now logs the count, the model and the per-column breakdown before filling. We did not remove the fill, because the zeros mostly arrive already-zeroed from datafactory's zarr and removing it would change behaviour without changing what is knowable.Do not close C-26 on the strength of the first paragraph. Two reasons:
- The zeros you are worried about are produced upstream of both of us, in datafactory's assembly, by design. Their ADR-047 says a filled month is "structurally indistinguishable from months where the source observed zero events". Removing every downstream fill does not recover the distinction — it was never in the data.
- Your
last_valid_month_idclip guards the trailing edge only. A coverage lapse in the middle of a series, or a source that starts late for one region, is not caught by it.
What did change that helps you. views-datafactory ships a
UserWarningfromload_dataset()for pre-coverage queries (their #269). Our generated model scaffolding was discarding it:templates/{model,ensemble}/template_main.pyemitted a barewarnings.filterwarnings("ignore")into every generated main, so the warning was suppressed in exactly the process that consumed the data — about 130 scaffolded scripts per consuming repo. That is now narrowed toDeprecationWarningandFutureWarningby name, and our pytest config no longer ignoresUserWarning. If you scaffold from our templates, regenerate and you will start seeing that signal.The open question — whether the schema should be able to express "nobody looked" at all — is now filed as views-datafactory#420 with pipeline-core's position recorded. It is their decision to take; we are not proposing an answer.
Epic: #51 · S2 · depends on #61 (S0) · blocked on a pipeline-core companion ·
storyimplementationblockedProblem (C-26, Tier 1)
Missing upstream data becomes literal
0("no conflict") and ships to FAO with no signal. Structural: the historical range is requested to_current_month_id()-1(clock), but UCDP data ends earlier atlast_valid_month_id— so every run fabricates the most-recent months.Build shape (per the epic design contract)
delivery/observed_range.py(representation-free):clip_to_observed(months, last_valid_month_id)/assert_no_fabrication_inside_range(...)over primitives. No pandas.unfao/extraction.py:months_of(df) -> np.ndarray.≤ last_valid_month_id; fails loud on a gap inside the valid range. Covers Case A (range beyond observed).Dependency — pipeline-core companion (blocking)
last_valid_month_idis not surfaced to this repo: datafactory's loader returns it (dataset.py:217) and pipeline-core discards it (dataloaders.py). Open a companion: "stop discardinglast_valid_month_id; pass it through" (+ log a fill count). This story is blocked until that lands.Acceptance criteria
delivery/observed_range.pypure + pandas-free; unit-tested on month arrays + a scalar boundary.last_valid_month_id; fails loud on inside-range gaps.Out of scope
Case B (a single unharvested cell inside the valid range) — no per-cell mask exists; defer behind a tripwire (log per-month zero-fraction).
Files
views_postprocessing/delivery/observed_range.py(new),unfao/extraction.py(extend),unfao/managers/unfao.py(call),tests/.