From b7bc6c13b74a118d1b29936460fa3de570937851 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Sat, 19 Sep 2026 04:13:52 +0200 Subject: [PATCH] fix(seam): correct the eval-seam contract for vpc 3.3.0's per-model source; floors vpc>=3.3.0, views-evaluation>=2.0.0 (#287) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit views-pipeline-core 3.3.0 (PyPI 2026-09-19) fixes C-215's root: its reporting stage now constructs PerModelMetricFrameSource — one of OUR MetricFrameFileSources per model, each rooted at THAT model's own data/generated (vpc #485, PR #522, their C-328) — instead of a single source rooted at the subject, which is why every baseline/constituent row vanished from every ensemble report. Our locked on-disk layout is untouched; only the root per model changed. Our half, in the same window as the register's sequencing note required: the seam test that had faithfully locked the DEFECTIVE subject-rooted call (test_stage_constructs_source_with_our_init_contract) is replaced by two real guards: - the stage constructs PerModelMetricFrameSource with EXACTLY {primary_model, primary_root=.data_generated, run_type, target} — no root_of= override, so the C-215 shape cannot be re-expressed through the new class (verified against five mutant stages, all fail); - a real on-disk per-model round-trip: frames saved under TWO roots resolve through the public metric_frame() with values read back exactly, and the subject-rooted shape provably returns None for the other model — never a shared root, the fixture blind spot that hid C-215 for months. Pins: views-pipeline-core>=3.3.0 (a <3.3.0 stage would silently blank rows again); views-evaluation>=2.0.0 — the floor held on 09-18 lands now because vpc 3.3.0 itself requires >=2.0.0, so the shared env co-resolves. Lock 129→126 packages (stepshift, xarray, views-transformation-library left; vpc 3.3.0 dropped the last). Suite green on the RELEASED 3.3.0 + 2.0.0: 443. Register: C-215 → Resolved (with what it does NOT do: C-216/C-217 configs and pins, C-223's one-time baseline evaluation, C-219/C-222 announce and vintage layers, and the end-to-end proof still owed); C-192 correction note closed; Cluster I updated honestly — the C-220/C-225 "land WITH the fix" constraint was NOT honoured (upstream shipped alone), so C-220 is now ARMED and its scrub is owed before C-223's evaluation run; Cluster H: eval floor landed, and the next trigger (views-frames 2.0.0, gated by TWO ceilings — ours and views-evaluation's [frames] extra, eval#91) recorded. No vpc ADR records the per-model rule yet; our 08-21 draft should be filed. CICs (evaluation_source, evaluation_report_template) updated for the injected source. 91 concerns (69 resolved, 22 open). Closes #287 item 1. Items 2-4 (announce layer, arm-on-fix traps, vintage semantics) remain open. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01N7YBtiS2kSVj27spMKDYfh --- .../CICs/cic_evaluation_report_template.md | 4 +- documentation/CICs/cic_evaluation_source.md | 11 +- pyproject.toml | 22 ++-- reports/technical_risk_register.md | 38 ++++--- tests/test_vpc_seam_contract.py | 103 +++++++++++++++--- uv.lock | 64 ++--------- 6 files changed, 143 insertions(+), 99 deletions(-) diff --git a/documentation/CICs/cic_evaluation_report_template.md b/documentation/CICs/cic_evaluation_report_template.md index c8985a8..dbee006 100644 --- a/documentation/CICs/cic_evaluation_report_template.md +++ b/documentation/CICs/cic_evaluation_report_template.md @@ -12,7 +12,7 @@ > **What is this class for?** -EvaluationReportTemplate **renders** self-contained HTML evaluation reports for VIEWS forecasting models and ensembles from an **injected `EvaluationSource`** — a typed `MetricFrame` per model (ADR-018 / C-108) — rather than acquiring metrics itself at render time. It assembles the comparative metric tables, optionally renders historical-vs-predicted line graphs from on-disk predictions (parquet or numpy PredictionFrame per ADR-012), and delegates all HTML assembly to `ReportModule`. This class does **not** touch WandB: it renders purely from the injected `EvaluationSource` (a `MetricFrame` per model), which the caller (pipeline-core's reporting stage) constructs as a `MetricFrameFileSource` reading the persisted frame. +EvaluationReportTemplate **renders** self-contained HTML evaluation reports for VIEWS forecasting models and ensembles from an **injected `EvaluationSource`** — a typed `MetricFrame` per model (ADR-018 / C-108) — rather than acquiring metrics itself at render time. It assembles the comparative metric tables, optionally renders historical-vs-predicted line graphs from on-disk predictions (parquet or numpy PredictionFrame per ADR-012), and delegates all HTML assembly to `ReportModule`. This class does **not** touch WandB: it renders purely from the injected `EvaluationSource` (a `MetricFrame` per model), which the caller (pipeline-core's reporting stage) constructs as an `EvaluationSource` (since pipeline-core 3.3.0 its `PerModelMetricFrameSource`, one `MetricFrameFileSource` per model) reading the persisted frame. --- @@ -35,7 +35,7 @@ EvaluationReportTemplate **renders** self-contained HTML evaluation reports for - **Canonical metric selection (ADR-017).** The metric *set* shown is the **central canonical standard** in `ReportingConfig.canonical_report_metrics` (keyed by `{regression,classification}×{point,sample}`), **not** the model's own list. A model occupies a cell when its `__metrics` config key is present & non-empty (declared, not inferred — ADR-003); one labelled canonical table is rendered per active cell. Values come from each model's `MetricFrame` via `mean_metric_value` (the `group_id="mean"` row); a canonical metric the frame lacks (absent row or NaN) is shown with an explicit "not calculated — add `` to ``" note (ADR-008), never dropped silently. **C-116:** more than one matching mean row raises `AmbiguousMetric` and renders a visible "ambiguous — multiple matching keys" cell rather than a silently-picked value. - **Baseline model inclusion.** Collects baseline model names from tier-specific config keys (`regression_point_baselines`, `regression_sample_baselines`, `classification_point_baselines`, `classification_sample_baselines`) and includes them in the comparative metric tables (lines 173-181). - **Constituent model verification.** For ensemble reports, the template asks `source.metric_frame(model)` for each declared constituent and verifies `level`/`partition` consistency across the resolved frames (via `unique_axis_value` over the frame axes), raising `ValueError` on mismatch (C-48 — a single ensemble row must not mix constituents from different levels/partitions). -- **No run selection in the template (C-48 resolved).** The template does not select WandB runs; it only consumes `source.metric_frame(model)`. The 22/25 production blanking is resolved structurally: pipeline-core persists the evaluation-of-record `MetricFrame` and injects a `MetricFrameFileSource`, so there is no render-time run to mis-select. The interim metric-aware selection seam (`evaluation_run_resolver`) has been deleted (B2 / C-108); the same-partition stale-relog concern moves upstream to the producer (C-110, gated on #220 provenance). +- **No run selection in the template (C-48 resolved).** The template does not select WandB runs; it only consumes `source.metric_frame(model)`. The 22/25 production blanking is resolved structurally: pipeline-core persists the evaluation-of-record `MetricFrame` and injects an `EvaluationSource` (one `MetricFrameFileSource` per model via `PerModelMetricFrameSource` since pipeline-core 3.3.0), so there is no render-time run to mis-select. The interim metric-aware selection seam (`evaluation_run_resolver`) has been deleted (B2 / C-108); the same-partition stale-relog concern moves upstream to the producer (C-110, gated on #220 provenance). - **Constituent resolution — degrade-and-announce (#105, honoring the #177 contract).** A declared constituent is never silently dropped. The source reporting **ABSENT** (`metric_frame(model)` returns `None`) records the model and **announces** it under "Model Metrics" as missing. A `metric_frame` call that **raises** (a **transient** failure) is **retried once**; if it still fails the model is **marked degraded** and announced distinctly (a retrieval error, not a confirmed absence). Absent and degraded models contribute no metrics row. Opt-in **strict mode** (`config["strict_constituents"] = True`) instead raises `ValueError` listing the absent/degraded models. (ADR-008 fail-loud / make-degradation-visible.) The subject model is always rendered (an absent subject shows "not calculated" cells). - **Metric table sorting.** Sorts combined metric DataFrames by MSLE first, then CRPS, then the first available metric (lines 265-282). - **Prediction sample graphs (non-fatal).** `_add_prediction_sample_graphs()` is wrapped in a `try/except` at the caller (lines 297-302). A failure in graph generation does not invalidate the metrics tables already written to the report. diff --git a/documentation/CICs/cic_evaluation_source.md b/documentation/CICs/cic_evaluation_source.md index f492d7e..95f40d4 100644 --- a/documentation/CICs/cic_evaluation_source.md +++ b/documentation/CICs/cic_evaluation_source.md @@ -8,7 +8,7 @@ --- -> **Scope note.** Covers the `EvaluationSource` *interface* (`typing.Protocol`), its single concrete adapter (`MetricFrameFileSource` — durable, loads the persisted `MetricFrame`), the `EvaluationProvenance` value object, and the pure value queries (`mean_metric_value`, `unique_axis_value`, `AmbiguousMetric`). This is the evaluation-side counterpart of the prediction `loaders/` Ingestion surface (C-108 / #173). +> **Scope note.** Covers the `EvaluationSource` *interface* (`typing.Protocol`), its single concrete adapter in this repo (`MetricFrameFileSource` — durable, loads the persisted `MetricFrame`), the `EvaluationProvenance` value object, and the pure value queries (`mean_metric_value`, `unique_axis_value`, `AmbiguousMetric`). This is the evaluation-side counterpart of the prediction `loaders/` Ingestion surface (C-108 / #173). Sources: `views_reporting/sources/_protocol.py`, `metric_frame_file_source.py`, `evaluation_provenance.py`, `metric_value.py`, `__init__.py`. @@ -35,7 +35,7 @@ Sources: `views_reporting/sources/_protocol.py`, `metric_frame_file_source.py`, - **Port (`EvaluationSource`, a `typing.Protocol`).** Provides `metric_frame(model: str) -> MetricFrame | None` and `provenance() -> EvaluationProvenance`. A source is bound to one `target` at construction (a report is per-target), so only `model` varies per call. The render code is statically checkable against the interface (LSP/ISP); a new source is a new class, not a change here (OCP). `views_evaluation` is imported only under `TYPE_CHECKING` in the port so it stays import-light (SDP/SAP). - **Failure taxonomy (the #105/#177 contract).** `metric_frame` returns **`None` = absent** (no evaluation for this model → the report degrades-and-announces, never a silent drop); **raises = transient** (a retrieval hiccup → the report retries once, then marks the model degraded). -- **`MetricFrameFileSource` (durable — the only implementation).** `metric_frame(model)` loads the persisted frame for `(model, run_type, target)` or returns `None` when its directory is absent; a corrupt/unreadable frame propagates (transient). `provenance()` is read from the subject model's frame metadata (run_id / evaluation_timestamp / data_version / scoring_code_version; no WandB url/owner). +- **`MetricFrameFileSource` (durable — the only implementation in this repo; pipeline-core's `PerModelMetricFrameSource` composes it per model).** `metric_frame(model)` loads the persisted frame for `(model, run_type, target)` or returns `None` when its directory is absent; a corrupt/unreadable frame propagates (transient). `provenance()` is read from the subject model's frame metadata (run_id / evaluation_timestamp / data_version / scoring_code_version; no WandB url/owner). - **Value query (`mean_metric_value`).** Reads the `group_id="mean"` row for `(eval_type, target, metric)`: `None` when absent or NaN ("not calculated"); raises `AmbiguousMetric` on >1 matching mean row (C-116; ADR-008); else the `float`. - **Axis query (`unique_axis_value`).** The single distinct value of an axis (`level`/`partition`) across a frame, for the cross-constituent consistency guard; raises if the axis is non-uniform within a frame. - **`EvaluationProvenance`.** A frozen presentation DTO (`run_id` + optional `run_url`/`owner`/`run_date`/`data_version`/`scoring_code_version`) rendered with None-omission, so each source supplies only what it knows. `MetricFrameFileSource` populates `run_id` and the frame provenance (`data_version`, `scoring_code_version`); `run_url`/`owner` remain generic optional fields that no current source populates. @@ -84,7 +84,10 @@ Nothing is silently dropped or silently guessed. ## 8. Examples of Correct Usage ```python -# pipeline-core's reporting stage constructs the source and calls generate: +# Direct construction (a test, a script, or a caller that owns ONE root). Since +# pipeline-core 3.3.0 the reporting stage instead injects its +# PerModelMetricFrameSource, which composes one of these per model, each at +# that model's own data/generated (vpc #485 / register C-215): source = MetricFrameFileSource(root, run_type="calibration", target="lr_ged_sb", primary_model="first_love") template.generate(source=source, target="lr_ged_sb") @@ -117,7 +120,7 @@ except AmbiguousMetric: ## 11. Evolution Notes -- The interim WandB scrape has been removed: pipeline-core now persists frames and injects a `MetricFrameFileSource`, the only source implementation. The render path imports no WandB, and the clickable WandB link/owner have fallen away (no source populates `run_url`/`owner`; they remain generic optional fields on `EvaluationProvenance` for a future source). +- The interim WandB scrape has been removed: pipeline-core persists frames and injects an `EvaluationSource` built on this class — one `MetricFrameFileSource` per model since pipeline-core 3.3.0 (`PerModelMetricFrameSource`, vpc #485; register C-215 records why a single subject-rooted source blanked every comparison row before that). The render path imports no WandB, and the clickable WandB link/owner have fallen away (no source populates `run_url`/`owner`; they remain generic optional fields on `EvaluationProvenance` for a future source). - The on-disk layout is a locked cross-repo contract with pipeline-core's producer (C-192/C-202), executable-pinned in `tests/test_vpc_seam_contract.py`. - The cross-constituent consistency check moved onto frame axes (`level`/`partition`); when run-resolved partition windows are plumbed into the producer (#220), they sharpen. diff --git a/pyproject.toml b/pyproject.toml index 2cc7d06..8e3dc2d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,12 @@ readme = "README.md" # (risk register C-36). requires-python = ">=3.11,<3.15" dependencies = [ - "views-pipeline-core>=3.0.0,<4.0.0", + # Floor 3.3.0 (#287, 2026-09-19): the first release whose reporting stage + # resolves every comparison model's MetricFrame from THAT model's own + # data/generated (PerModelMetricFrameSource, vpc #485) — the fix for + # baseline/constituent rows vanishing from every ensemble report (C-215). + # Our seam test locks that contract; <3.3.0 stages would silently blank them. + "views-pipeline-core>=3.3.0,<4.0.0", # views-frames: the leaf data contract — PredictionFrame / TargetFrame + # SpatioTemporalIndex / SpatialLevel, plus the views_frames_summarize sibling # (MAP/HDI/quantiles), which ships in the SAME distribution (one line). numpy-only, @@ -43,12 +48,15 @@ dependencies = [ # `[frames]` extra survives; the only merged changes to metric_frame.py since # 1.0.0 are to the provenance stamp's VALUE — the `+g` suffix and S1's # bounding of it — which we read as opaque text). The span keeps - # co-resolution with views-pipeline-core across its move to ^2.0.0. A >=2.0.0 - # FLOOR is a separate step, gated on 2.0.0 actually reaching PyPI — a floor - # naming an unpublished version is the C-213(a) mistake (Cluster H). - "views-evaluation[frames]>=1.0.0,<3.0.0", - # views-transformation-library removed: never imported here and already - # pulled transitively via views-pipeline-core (falsify P3). + # co-resolution with views-pipeline-core across its move to ^2.0.0. + # Floor 2.0.0 (2026-09-19): raised only once BOTH gates held — 2.0.0 on PyPI + # (2026-09-18) AND vpc 3.3.0 itself requiring >=2.0.0 (its to_dataframe + # egress removed) — so the shared env co-resolves. Re-qualified: suite green + # on 2.0.0, a 2.0.0-emitted frame round-trips through our source exactly. + "views-evaluation[frames]>=2.0.0,<3.0.0", + # views-transformation-library removed: never imported here (falsify P3); + # it was transitive via views-pipeline-core until 3.3.0 dropped it — now + # simply absent, which is correct. # viewser RETIRED from runtime dependencies (C-22 Resolved, epic #204): the render # path reads the bundled metadata parquets (metadata/data/); only # scripts/build_entity_metadata.py uses viewser, at dev time — see the diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index caaae40..eec6cbf 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -1,8 +1,8 @@ # Technical Risk Register -**Last updated:** 2026-09-17 +**Last updated:** 2026-09-19 **Governing ADR:** ADR-010 (Technical Risk Register) -**Entry count:** 91 concerns (68 resolved, 23 open) + 5 disagreements (4 resolved) +**Entry count:** 91 concerns (69 resolved, 22 open) + 5 disagreements (4 resolved) --- @@ -30,8 +30,8 @@ Root causes shared by multiple concerns. Resolving the root tends to dissolve or | **E — Legacy transform machinery** | RESOLVED (2026-06-20, #119) — `DatasetTransformationModule` removed + direct `polars` declaration dropped (polars stays transitive via pipeline-core) | C-25 ✓ (+ resolved C-10, C-04, C-02) | ✓ Resolved | | **F — Value-correctness & contract assurance** | The load → compute → render → reconcile chain is tested for shape / does-not-crash, not for value equality, contract conformance, or input completeness | C-29 ✓ (render fidelity), C-35 ✓ (MAP/HDI correctness — render path + PosteriorDistributionAnalyzer both on the views-frames tower + law tests; RESOLVED, ADR-019 / #157), C-185 (`*_map` is a tower tip, not a MAP — naming debt), C-39 ✓ (metadata accessors untested — resolved, `tests/test_metadata_accessors.py`), C-41 ✓ (canonical-token contract test), C-116 ✓ (multi-match → silent wrong value — RESOLVED: fail-loud default + visible "ambiguous" cell + collision contract test), C-111 ✓ (input completeness — RESOLVED 2026-07-31, S4 #266), C-113 (actuals provenance), C-112 (bundled-data staleness — forward, pairs with C-22), C-186 ✓ (views-frames version → forecast-output drift), C-208 ✓ (gapped-lattice cell stretch/misplacement — rendering-geometry faithfulness, RESOLVED 2026-07-16), C-192 ✓ (cross-repo eval contract untested — RESOLVED 2026-07-31, executable seam test), C-227 ✓ (CM choropleth silently drew the dead country on duplicate ISO codes — RESOLVED 2026-09-17, loud guard at the merge chokepoint; latent, never observed) (+ resolved C-01, C-11) | Open — highest latent severity; mostly "write the missing correctness/contract test" (an assurance sprint). Phase 1 (correctness/contract core) landed 2026-06-28: C-29 ✓ (render==source fidelity test), C-41 ✓ (canonical-metric contract test), C-111 values-half guarded. C-186 ✓ (tower behavioural-regime tests, Phase 2, 2026-06-28). Remaining: C-112 (guards live — watch the regen cadence), C-113 / C-185 (non-test); C-111 ✓ resolved (coverage checks, S4 #266); C-192 ✓ resolved (seam test landed, S2 #264); C-39 ✓ resolved (accessor tests landed, 2026-07). | | **G — Partner-deliverable readiness** | Reports are built for internal preview, not yet hardened as a standalone, traceable, decision-appropriate *partner artifact* | C-28 ✓ (offline / self-contained — RESOLVED, vendored Tailwind, #132), C-34 ✓ (provenance / auditability — RESOLVED, footer stamp, #131), C-187 ✓ (vendored-CSS class coverage — RESOLVED, #132), C-188 ✓ (machine-readable provenance — RESOLVED, #132), C-109 (decision-appropriate uncertainty), C-214 ✓ (plotly-injection heuristics — RESOLVED 2026-07-31, S3 #265) | Open — C-109 (decision-appropriate uncertainty, partially delivered 2026-07: the #230 P(any) exceedance map layer) is the live remainder; C-214 ✓ resolved (S3 #265); offline + provenance now shipped | -| **H — Published-state qualification gap (dev-tandem ↔ PyPI seam)** | Contracts, pins, and names are qualified against dev branches moving in tandem, never against **published** artifacts — the coordinated publish (#179) is the single event where every deferred qualification fires at once | C-213 ✓ (root — vpc pin outran PyPI; import surface canonicalized + guarded 2026-07-31 S1 #263; ordering half resolved 2026-08-03 when vpc 3.0.0 published pre-qualified), C-192 ✓ (cross-repo seam untested — resolved S2 #264), C-36 (upstream transitive pins — re-probed at vpc's 2026-08-03 publish: envelope matches ours, ingester3==2.1.1 unchanged so the levenshtein wall persists), C-185 (rename waits on a coordinated cross-repo change), C-186 ✓ residual (b) (exact-pin views-frames for release), C-211 (partial — cross-repo decision, not publish-gated) | Open (added review-rr 2026-07-31) — mitigation is the #179 release-gate runbook (largely written); C-192's executable contract test landed (S2 #264); C-36 stays upstream-blocked regardless. **SECOND CONFIRMED FIRING (2026-08-02, v0.3.3):** 0.3.2's published `requires_dist` pin `views-evaluation>=0.4.0,<1.0.0` — qualified only against the dev-tandem git source — excluded views-evaluation **1.0.0** the day it hit PyPI, breaking platform co-resolution at the coordinated publish (caught by a sibling-repo agent). Fixed same-day: pin `>=1.0.0,<2.0.0`, interim git-source retired (its documented trigger fired), suite green against the RELEASED 1.0.0. The root cause has now bitten in production twice (vpc pin C-213(a) + this); every remaining dev-tandem git source (vpc, until their 3.0.0 publishes) carries the same latent defect until re-qualified at its publish. **CYCLE CLOSED (2026-08-03, #279):** vpc 3.0.0 published to PyPI carrying the pre-qualified pins (the vpc#313 warning worked — no third firing); the last dev-tandem git source retired, `views-reporting==0.3.3` co-resolves from PyPI alone, suite green against the released 3.0.0. **No dev-tandem git sources remain anywhere in pyproject/uv.lock** — the cluster's generating condition is discharged for this release cycle; it re-arises only if a future unreleased-dependency tandem is reintroduced. **THIRD EXPOSURE — of the published-metadata facet only, caught as a prediction (2026-09-17, #289):** no dev-tandem was reintroduced and the `<2.0.0` ceiling WAS correctly qualified against the released 1.0.0; what is Cluster-H-shaped is the second half below. views-evaluation's pandas-free 2.0.0 (their #66/#72) and vpc's coming `^2.0.0` pin (vpc#515) would make our `<2.0.0` ceiling unsatisfiable in the shared env. Source widened to a SPAN pin `>=1.0.0,<3.0.0` — deliberately NOT the requested `>=2.0.0`, because 2.0.0 was unpublished at fix time (PyPI latest 1.0.0) and a floor naming an unpublished version is exactly C-213(a)'s mistake. Safety evidence: everything we import lives in `evaluation.metric_frame` (+ `metric_catalog` in one test), and no 2.0.0 story changes the MetricFrame format or the names we import (S8's gate: "MetricFrame format unchanged"; `[frames]` survives; the only merged changes to that module since 1.0.0 are to the provenance stamp's value — the `+g` suffix and S1's bounding of it — which we treat as opaque text). **Not yet pre-empted at the seam that matters:** the shared env resolves from PyPI, where 0.3.3's immutable metadata still says `<2.0.0` — vpc#515's "after views-reporting has widened" precondition is met only by a **0.3.4 publish**, which collides with the publish-last rule and is therefore a decision, not a mechanical step. Re-qualification trigger: views-evaluation 2.0.0 on PyPI → re-lock, suite, then the `>=2.0.0` floor + 0.3.4. **TRIGGER FIRED 2026-09-18 — re-qualified, floor deliberately NOT raised:** 2.0.0 published (21:29). Our suite is green on it (442) in isolation and a 2.0.0-emitted frame round-trips through `MetricFrameFileSource` + `mean_metric_value` exactly. But `uv lock --upgrade-package views-evaluation` resolves only 1.1.0: **views-pipeline-core 3.2.0 pins `views-evaluation<2.0.0`**, and its evaluation stage still calls `report.to_dataframe()` (stage.py:375-377) — their ceiling is protecting a live call, not stale. A `>=2.0.0` floor here today would make views-reporting + vpc mutually unsatisfiable (the C-115-shape shared-env trap). Sequencing is therefore fixed: vpc#515 lands (removing their `to_dataframe` egress, vpc#512) → our floor `>=2.0.0` → our 0.3.4 publish. The lock stays on 1.0.0 until then; the span admits 2.0.0 the moment vpc's ceiling lifts. | -| **I — Post-B2 eval-report comparison seam (the baselines-vanishing braid)** | The B2 inversion shipped with its production wiring unverified end-to-end: consumer and producer disagree on frame location, declarations/targets/pins/envs each independently blank comparison rows, and every failure mode is silenced or mislabeled — so the deliverable degraded for months with green CI | C-215 (root, Tier 1 — subject-rooted source + seam test pinning it), C-216 (target drift + stranded fixes), C-217 (pins/envs make the seam unreachable), C-218 (Tier 1 — zero-report silence), C-219 (declaration/announce layer), C-220 (post-fix detonation), C-221 (corruption path), C-222 (vintage semantics dropped), C-223 (cache short-circuit / no baseline orchestrator), C-224 (3.0.0 classification drop), C-225/C-226 (ride-alongs) | Open (registered 2026-08-17 from /code-review max on the baselines-vanishing mystery; reconstruction VERIFIED). Fix sequencing constraint: C-220/C-225(AP-sort) must land WITH the C-215/C-216 resolution fix; C-192's locked seam contract must be corrected in the same change. Cross-repo: vpc stage + views-models configs/branches own most mechanisms; this repo owns the template/source/announce layers and the seam test. | +| **H — Published-state qualification gap (dev-tandem ↔ PyPI seam)** | Contracts, pins, and names are qualified against dev branches moving in tandem, never against **published** artifacts — the coordinated publish (#179) is the single event where every deferred qualification fires at once | C-213 ✓ (root — vpc pin outran PyPI; import surface canonicalized + guarded 2026-07-31 S1 #263; ordering half resolved 2026-08-03 when vpc 3.0.0 published pre-qualified), C-192 ✓ (cross-repo seam untested — resolved S2 #264), C-36 (upstream transitive pins — re-probed at vpc's 2026-08-03 publish: envelope matches ours, ingester3==2.1.1 unchanged so the levenshtein wall persists), C-185 (rename waits on a coordinated cross-repo change), C-186 ✓ residual (b) (exact-pin views-frames for release), C-211 (partial — cross-repo decision, not publish-gated) | Open (added review-rr 2026-07-31) — mitigation is the #179 release-gate runbook (largely written); C-192's executable contract test landed (S2 #264); C-36 stays upstream-blocked regardless. **SECOND CONFIRMED FIRING (2026-08-02, v0.3.3):** 0.3.2's published `requires_dist` pin `views-evaluation>=0.4.0,<1.0.0` — qualified only against the dev-tandem git source — excluded views-evaluation **1.0.0** the day it hit PyPI, breaking platform co-resolution at the coordinated publish (caught by a sibling-repo agent). Fixed same-day: pin `>=1.0.0,<2.0.0`, interim git-source retired (its documented trigger fired), suite green against the RELEASED 1.0.0. The root cause has now bitten in production twice (vpc pin C-213(a) + this); every remaining dev-tandem git source (vpc, until their 3.0.0 publishes) carries the same latent defect until re-qualified at its publish. **CYCLE CLOSED (2026-08-03, #279):** vpc 3.0.0 published to PyPI carrying the pre-qualified pins (the vpc#313 warning worked — no third firing); the last dev-tandem git source retired, `views-reporting==0.3.3` co-resolves from PyPI alone, suite green against the released 3.0.0. **No dev-tandem git sources remain anywhere in pyproject/uv.lock** — the cluster's generating condition is discharged for this release cycle; it re-arises only if a future unreleased-dependency tandem is reintroduced. **THIRD EXPOSURE — of the published-metadata facet only, caught as a prediction (2026-09-17, #289):** no dev-tandem was reintroduced and the `<2.0.0` ceiling WAS correctly qualified against the released 1.0.0; what is Cluster-H-shaped is the second half below. views-evaluation's pandas-free 2.0.0 (their #66/#72) and vpc's coming `^2.0.0` pin (vpc#515) would make our `<2.0.0` ceiling unsatisfiable in the shared env. Source widened to a SPAN pin `>=1.0.0,<3.0.0` — deliberately NOT the requested `>=2.0.0`, because 2.0.0 was unpublished at fix time (PyPI latest 1.0.0) and a floor naming an unpublished version is exactly C-213(a)'s mistake. Safety evidence: everything we import lives in `evaluation.metric_frame` (+ `metric_catalog` in one test), and no 2.0.0 story changes the MetricFrame format or the names we import (S8's gate: "MetricFrame format unchanged"; `[frames]` survives; the only merged changes to that module since 1.0.0 are to the provenance stamp's value — the `+g` suffix and S1's bounding of it — which we treat as opaque text). **Not yet pre-empted at the seam that matters:** the shared env resolves from PyPI, where 0.3.3's immutable metadata still says `<2.0.0` — vpc#515's "after views-reporting has widened" precondition is met only by a **0.3.4 publish**, which collides with the publish-last rule and is therefore a decision, not a mechanical step. Re-qualification trigger: views-evaluation 2.0.0 on PyPI → re-lock, suite, then the `>=2.0.0` floor + 0.3.4. **TRIGGER FIRED 2026-09-18 — re-qualified, floor deliberately NOT raised:** 2.0.0 published (21:29). Our suite is green on it (442) in isolation and a 2.0.0-emitted frame round-trips through `MetricFrameFileSource` + `mean_metric_value` exactly. But `uv lock --upgrade-package views-evaluation` resolves only 1.1.0: **views-pipeline-core 3.2.0 pins `views-evaluation<2.0.0`**, and its evaluation stage still calls `report.to_dataframe()` (stage.py:375-377) — their ceiling is protecting a live call, not stale. A `>=2.0.0` floor here today would make views-reporting + vpc mutually unsatisfiable (the C-115-shape shared-env trap). Sequencing is therefore fixed: vpc#515 lands (removing their `to_dataframe` egress, vpc#512) → our floor `>=2.0.0` → our 0.3.4 publish. The lock stays on 1.0.0 until then; the span admits 2.0.0 the moment vpc's ceiling lifts. **LANDED 2026-09-19:** vpc 3.3.0 requires `views-evaluation>=2.0.0` (their #515/#512 shipped); floor raised to `>=2.0.0,<3.0.0` in the same PR as the 3.3.0 bump; lock resolves 2.0.0 (129→126 packages: views-evaluation dropped its scikit-learn dep, but sklearn stays via viewser→pyod; the three that left are stepshift, xarray and views-transformation-library — vpc 3.3.0 dropped the latter). The 0.3.4 publish remains the open decision. **NEXT TRIGGER, already visible (2026-09-19): views-frames 2.0.0 is on PyPI** and cannot reach a reporting env until TWO ceilings move — ours (`views-frames<2.0.0`, deliberate: C-186's characterization pins must re-qualify a major of the leaf data contract) and views-evaluation's `[frames]` extra (`<2`, theirs — views-evaluation#91). Not a rider on the 3.3.0 bump; a separate qualification PR when views-evaluation lifts theirs. | +| **I — Post-B2 eval-report comparison seam (the baselines-vanishing braid)** | The B2 inversion shipped with its production wiring unverified end-to-end: consumer and producer disagree on frame location, declarations/targets/pins/envs each independently blank comparison rows, and every failure mode is silenced or mislabeled — so the deliverable degraded for months with green CI | C-215 ✓ (root, Tier 1 — subject-rooted source + seam test pinning it; RESOLVED 2026-09-19), C-216 (target drift + stranded fixes), C-217 (pins/envs make the seam unreachable), C-218 (Tier 1 — zero-report silence), C-219 (declaration/announce layer), C-220 (post-fix detonation), C-221 (corruption path), C-222 (vintage semantics dropped), C-223 (cache short-circuit / no baseline orchestrator), C-224 (3.0.0 classification drop), C-225/C-226 (ride-alongs) | Open — ROOT RESOLVED 2026-09-19 (C-215 ✓: vpc 3.3.0 per-model source + our seam-test correction, in step). Remaining members: C-216/C-217 (views-models configs/pins), C-218 (silence), C-219 (announce), C-220 (post-fix detonation — NOW ARMED: rows CAN resolve, so once C-223's one-time baseline evaluation writes a cm-baseline frame, a pgm subject declaring it hits the level guard and aborts the whole report; the cross-level scrub is C-220's own remediation on views-models, not #405's vocabulary fix), C-221, C-222, C-223 (baselines still need a one-time evaluation run), C-225. End-to-end proof (a real ensemble report with baseline rows) still owed. Sequencing note, updated: the original constraint (C-220/C-225 land WITH the C-215 fix) was NOT honoured — upstream shipped the root fix alone in 3.3.0 and we corrected the seam test in step; C-220's scrub and C-225's AP-sort are now owed BEFORE C-223's baseline evaluation run, which is the step that would actually arm them. C-192's seam contract correction: done. Cross-repo: vpc stage + views-models configs/branches own most mechanisms; this repo owns the template/source/announce layers and the seam test. | C-34 (provenance) and C-28 (offline) now anchor **Cluster G** (partner-deliverable readiness) rather than standing alone; the C-108 inversion does not fix C-28 (the exported HTML's view-time CDN dependency), which is why C-28 moved out of Cluster A. @@ -171,18 +171,6 @@ C-34 (provenance) and C-28 (offline) now anchor **Cluster G** (partner-deliverab | Narrative | The viewser pgm loa is regional (Africa+ME) while the platform's data supply has moved to the GAUL-coded datafactory — so the global bundle (#231) is built by an **explicitly interim** crosswalk: GAUL `iso3_code` → VIEWS `isoab` → `country_id`, duplicate isoab resolved to the most-recently-observed entity (active-active collisions fail loud, #242). Coverage is declared, not silent, **with denominators** (#248, all figures reconcile with the stamp): of the **64,818-cell forecast land region**, 99.50% crosswalks — 322 cells (disputed/non-VIEWS territories: Western Sahara, Kashmir, Abyei…) carry no VIEWS country and degrade via the visible-NaN path; **grid-wide**, 29,373 of 95,578 GAUL-coded land cells are unmatched, dominated by Antarctica (25,444) and Greenland (3,038) — bulk out-of-scope regions, not disputed footnotes. ~157 Africa+ME border cells differ from the retired viewser assignment (GAUL 2024 vs VIEWS DB borders, e.g. the Aouzou strip). **Declared absorption (#249):** GAUL 2024 has no Kosovo unit — its 10 cells carry `SRB` and crosswalk to Serbia (country_id 233) while VIEWS Kosovo (232) receives zero cells; because the code is valid this never enters the unmatched accounting, so the stamp declares it explicitly (`known_gaul_absorptions`, validated against the built table at regen). The structural risk is the **transition itself**: two coding systems now coexist in the platform with reporting owning the only bridge. Containment: runtime accessors depend ONLY on the bundle schema contract (`priogrid_id → country_id`) — swapping/adding codings is a build-script-only change (DIP), and a future multi-coding bundle would ADD coding columns rather than rewrite accessors (OCP). The eventual coding standard is deliberately undecided (GAUL is FAO's coding; other partner codings may follow) — the deep-dive/decision is filed cross-repo as views-datafactory#341 + views-postprocessing#123, which will announce any general solution back here. | | Cross-refs | C-112 (bundle staleness observability — the stamp this entry extends; reciprocal); C-192 (sibling cross-repo contract-assurance theme); epic #230; views-datafactory#341 / views-postprocessing#123 (coding-decision venue); ADR-021 (documents the interim status + the handover path). | -### C-215: Post-B2 cross-model metric resolution is structurally broken — the source is rooted at the SUBJECT's data_generated while the producer writes per-model, and our own seam test pins the defective binding - -| Field | Value | -|-------|-------| -| ID | C-215 | -| Tier | 1 — model-output incorrectness with no error signal in the common case: every post-B2 evaluation report omits ALL baseline/constituent comparison rows; for subjects declaring no baselines the report reads as complete (no banner, log-only warning), and CI is green throughout because every fixture writes multiple models' frames under ONE shared root — a state production never produces. Observed (the baselines-vanishing incident), not latent. | -| Source | /code-review max (baselines-vanishing investigation, 2026-08-17); verdict: the session reconstruction VERIFIED with amendments | -| Trigger | The next `--evaluate --report` run on any subject with declared baselines under vpc>=3.0.0 (rows structurally absent); becomes acutely visible the moment a developer fixes resolution to per-model paths — `tests/test_vpc_seam_contract.py:112` AST-locks `root=` as a LOCKED contract, so the CORRECT fix fails CI | -| Location | vpc 3.0.0 `managers/reporting/stage.py:220` (roots `MetricFrameFileSource` at the subject's own `data/generated`); producer `managers/evaluation/stage.py:439-445` (`_save_metric_frame` writes only under each model's own `data/generated`); our pinning surfaces: `tests/test_vpc_seam_contract.py:112`, `tests/test_eval_report_from_source.py` (dict stub), e2e fixtures, `scripts/generate_demo_eval_reports.py:133` (all share one root) | -| Narrative | The B2 inversion's consumer and producer disagree about WHERE frames live: the reporting stage probes `/data/generated///metricframe_` for every comparison model, but nothing ever writes another model's frame under the subject's tree — no copy/symlink mechanism exists anywhere in vpc, views-reporting, or views-models (exhaustively searched); all three ensemble managers share the defect. So `metric_frame(model)` returns None for every baseline/constituent → "absent" → zero comparison rows, regardless of target names, pins, or environment. This is the FIRST gate of the baselines-vanishing braid (C-216/C-217 are independently sufficient co-causes). The assurance layer actively protects the defect: the C-192 seam test locks the subject-rooted binding, and every green suite exercises a shared-root layout production cannot produce — CI proves a working multi-model report while production structurally cannot resolve one (the C-46 false-confidence class, third confirmed firing). Remediation direction (not prescribed here): resolve each comparison model via its OWN path manager (or persist frames to a shared metrics root), then CORRECT the seam-test contract in the same change. | -| Cross-refs | C-48/C-108 ✓ (their "end-to-end complete" resolutions now carry dated correction notes pointing here); C-192 ✓ (the seam test that pins the wrong contract); C-46 (CI-green-while-production-broken sibling); C-218 (the silence layer that let this run unnoticed); C-219 (declaration layer); Cluster I (root); views-r2darts2#23 / Slack (Dylan's observed symptom). | - ### C-216: Baseline target vocabulary on views-models development is stale (lr_*_best vs lr_ged_*) and the fixes are stranded on two side branches — frame lookup is exact-name equality, so mismatch alone blanks baseline rows | Field | Value | @@ -236,7 +224,7 @@ C-34 (provenance) and C-28 (offline) now anchor **Cluster G** (partner-deliverab | Field | Value | |-------|-------| | ID | C-220 | -| Tier | 2 — dormant today ONLY because rows never resolve (C-215); becomes an acute whole-report-loss failure the moment the resolution defects are fixed. Registered now so the fix sequencing accounts for it. | +| Tier | 2 — was dormant only because rows never resolved (C-215, fixed 2026-09-19); now ARMED — it fires the moment a cm-baseline frame exists on disk for a pgm subject that declares it (i.e. after C-223's baseline evaluation run). Registered so the fix sequencing accounts for it. | | Source | /code-review max (2026-08-17) | | Trigger | Fixing C-215/C-216/C-217 (baseline rows first resolve) without first scrubbing cross-level baseline declarations and hardening the guard's comparison semantics | | Location | `views_reporting/templates/reports/evaluation.py:404` (level equality + partition compared as `str(partition_dict[run_type])`, stamped at vpc `managers/evaluation/stage.py:434`; ValueError propagates through the whole report); views-models: 17 pgm-level configs on BOTH branches declaring `average_cmbaseline`/`zero_cmbaseline`/`locf_cmbaseline` (including the pgm baselines declaring their cm counterparts) | @@ -376,6 +364,20 @@ C-34 (provenance) and C-28 (offline) now anchor **Cluster G** (partner-deliverab ## Resolved Concerns +### C-215: Post-B2 cross-model metric resolution is structurally broken — the source is rooted at the SUBJECT's data_generated while the producer writes per-model, and our own seam test pins the defective binding — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-215 | +| Tier | 1 — model-output incorrectness with no error signal in the common case: every post-B2 evaluation report omits ALL baseline/constituent comparison rows; for subjects declaring no baselines the report reads as complete (no banner, log-only warning), and CI is green throughout because every fixture writes multiple models' frames under ONE shared root — a state production never produces. Observed (the baselines-vanishing incident), not latent. | +| Source | /code-review max (baselines-vanishing investigation, 2026-08-17); verdict: the session reconstruction VERIFIED with amendments | +| Trigger | The next `--evaluate --report` run on any subject with declared baselines under vpc>=3.0.0 (rows structurally absent); becomes acutely visible the moment a developer fixes resolution to per-model paths — `tests/test_vpc_seam_contract.py:112` AST-locks `root=` as a LOCKED contract, so the CORRECT fix fails CI | +| Location | vpc 3.0.0 `managers/reporting/stage.py:220` (roots `MetricFrameFileSource` at the subject's own `data/generated`); producer `managers/evaluation/stage.py:439-445` (`_save_metric_frame` writes only under each model's own `data/generated`); our pinning surfaces: `tests/test_vpc_seam_contract.py:112`, `tests/test_eval_report_from_source.py` (dict stub), e2e fixtures, `scripts/generate_demo_eval_reports.py:133` (all share one root) | +| Narrative | The B2 inversion's consumer and producer disagree about WHERE frames live: the reporting stage probes `/data/generated///metricframe_` for every comparison model, but nothing ever writes another model's frame under the subject's tree — no copy/symlink mechanism exists anywhere in vpc, views-reporting, or views-models (exhaustively searched); all three ensemble managers share the defect. So `metric_frame(model)` returns None for every baseline/constituent → "absent" → zero comparison rows, regardless of target names, pins, or environment. This is the FIRST gate of the baselines-vanishing braid (C-216/C-217 are independently sufficient co-causes). The assurance layer actively protects the defect: the C-192 seam test locks the subject-rooted binding, and every green suite exercises a shared-root layout production cannot produce — CI proves a working multi-model report while production structurally cannot resolve one (the C-46 false-confidence class, third confirmed firing). Remediation direction (not prescribed here): resolve each comparison model via its OWN path manager (or persist frames to a shared metrics root), then CORRECT the seam-test contract in the same change. | +| Cross-refs | C-48/C-108 ✓ (their "end-to-end complete" resolutions now carry dated correction notes pointing here); C-192 ✓ (the seam test that pins the wrong contract); C-46 (CI-green-while-production-broken sibling); C-218 (the silence layer that let this run unnoticed); C-219 (declaration layer); Cluster I (root); views-r2darts2#23 / Slack (Dylan's observed symptom). | +| Resolved | 2026-09-19 (vpc 3.3.0 on PyPI + our #287 seam correction, same window) | +| Resolution | **Both halves of the seam landed in step, as the C-192 correction note required** (the separate C-220/C-225 constraint was not — see Cluster I). Producer/consumer agreement: views-pipeline-core 3.3.0's reporting stage now constructs `PerModelMetricFrameSource` — one of OUR `MetricFrameFileSource`s per model, each rooted at THAT model's own `data/generated` (vpc #485, PR #522, their C-328) — so baseline/constituent frames are looked up where the producer writes them. Our locked layout `root///metricframe_` is untouched; only the root per model changed. Our half: the seam test that had faithfully locked the defective subject-rooted call (`test_stage_constructs_source_with_our_init_contract`) is replaced by two — one pins the new construction (`primary_root=.data_generated`), one is a REAL on-disk round-trip (frames saved under two different roots, resolved via the public `metric_frame()` of the stage's source CLASS with an injected test resolver (production's default `model_data_generated` resolver is exercised by vpc's own C-328 tests, not here), values read back exactly; the subject-rooted shape provably returns None for the other model) — and test 1 also asserts NO `root_of=` override in the stage call, so the C-215 shape cannot be re-expressed through the new class. The shared-root fixtures elsewhere (`test_e2e_eval_report.py`, `scripts/generate_demo_eval_reports.py`) still exist and are now the residual of this blind spot, not its closure. Pin raised to `views-pipeline-core>=3.3.0` (a <3.3.0 stage would silently blank rows again); suite green against the RELEASED 3.3.0 (443). No vpc ADR records this decision yet (their ADR-062 is unrelated — retire-by-refusal); the per-model rule exists as vpc #485 + PR #522 + C-328 and our draft ADR text of 2026-08-21, which should now be filed as the record. **What this does NOT do:** frames still only exist for models that were actually evaluated at the report's run_type (C-223 — the one-time baseline re-evaluation remains an operational step), C-216's config drift and C-217's pins on views-models are unchanged, and the announce/vintage layers (C-219/C-222) are still open in #287 items 2-4. The end-to-end proof — a real ensemble report with baseline rows — is still owed and depends on those. | + ### C-227: CM choropleth silently draws the DEAD country when a frame carries two country_ids under one ISO code — RESOLVED | Field | Value | @@ -444,7 +446,7 @@ C-34 (provenance) and C-28 (offline) now anchor **Cluster G** (partner-deliverab | Narrative | The B2 inversion (C-108) made the cross-repo eval contract an *injected* one: pipeline-core's reporting stage builds a `MetricFrameFileSource` and calls `generate(source=, target=)`, and its evaluation stage persists the frame at the agreed path. No executable test in **this** repo exercises that real seam — the suite mocks pipeline-core, so an API/semantic drift surfaces only at runtime. This was a **near-miss (2026-06-28):** this repo's `uv.lock` was pinned to a pre-epic pipeline-core dev commit that still called the deleted `generate(wandb_run=)` signature, and green CI hid it because nothing executes the seam; the instance was resolved by realigning the lock (#181) to pipeline-core's eval-of-record HEAD. **Durable residual:** the assurance gap itself — the contract is not pinned by an executable test on our side. **Mitigations already in place:** (a) pipeline-core's reporting stage has a **fail-loud preflight** that raises a clear upgrade message if the installed views-reporting lacks `MetricFrameFileSource` (catches a missing *consumer*, though not a *signature* change); (b) the on-disk frame path is a **locked cross-repo contract** documented on both sides (pipeline-core `managers/evaluation/stage.py` ↔ our `MetricFrameFileSource._frame_dir`) — this is what retired the earlier "provisional path" worry. Remediation: a contract test that imports the real `views_pipeline_core.managers.reporting.stage` and asserts it calls `generate(source=, target=)` (skip if absent), and/or a shared `MetricFrame` round-trip fixture both repos pin. Fails **loud** at runtime today (TypeError / preflight), not silent → Tier 3, **elevate if a silent semantic drift in the consumed `MetricFrame` is ever observed.** | | Cross-refs | C-46 (sibling "mocked seam → false confidence" / CI-status gap — same theme, different mechanism); C-108 (the inversion that created this injected contract); C-34 (provenance the contract also carries); Cluster F (value-correctness & contract assurance); GitHub #179 (the publish-last coordination this contract gates); C-213 (the release-coupling entry gating the same publish event — the moment this un-tested seam becomes acute; reciprocal). | | Resolved | 2026-07-31 (epic #262 S2, #264) | -| Resolution | **`tests/test_vpc_seam_contract.py` executes against the REAL pipeline-core stage** (skip-if-absent, C-46 contract): AST-pins the stage's single `evaluation_template.generate(source=, target=)` call (keyword-only, exactly those two); binds our `generate` signature against that call; pins the `MetricFrameFileSource(...)` construction kwargs + `root=`; pins the on-disk layout by **executable equality against the producer's own exported `METRICFRAME_DIR_PREFIX`** constant; bonus: both `forecast_template.generate(...)` call sites bind our forecast signature. RED demonstrated for a signature rename and a path-suffix drift. Rider: the stale "provisional layout" docstring in `metric_frame_file_source.py` corrected to the locked-contract language (the producer had long declared it C-202-locked). **CORRECTION NOTE (2026-08-17, C-215): the AST pin faithfully locks the REAL stage call — but the real call is defective (subject-rooted source vs per-model producer), so this test now PROTECTS the defect: the correct per-model fix fails this test. When C-215 is fixed, the locked contract must be corrected in the same change, not worked around.** | +| Resolution | **`tests/test_vpc_seam_contract.py` executes against the REAL pipeline-core stage** (skip-if-absent, C-46 contract): AST-pins the stage's single `evaluation_template.generate(source=, target=)` call (keyword-only, exactly those two); binds our `generate` signature against that call; pins the `MetricFrameFileSource(...)` construction kwargs + `root=`; pins the on-disk layout by **executable equality against the producer's own exported `METRICFRAME_DIR_PREFIX`** constant; bonus: both `forecast_template.generate(...)` call sites bind our forecast signature. RED demonstrated for a signature rename and a path-suffix drift. Rider: the stale "provisional layout" docstring in `metric_frame_file_source.py` corrected to the locked-contract language (the producer had long declared it C-202-locked). **CORRECTION NOTE (2026-08-17, C-215): the AST pin faithfully locks the REAL stage call — but the real call is defective (subject-rooted source vs per-model producer), so this test now PROTECTS the defect: the correct per-model fix fails this test. When C-215 is fixed, the locked contract must be corrected in the same change, not worked around.** **CLOSED 2026-09-19: corrected in the same window as vpc 3.3.0 — the test now pins `PerModelMetricFrameSource(primary_root=.data_generated)` and a real per-model-layout fixture (see C-215 Resolution).** | ### C-189: Spatial aggregation of forecast cells can misrepresent the quantity (MAUP / wrong operator) — RESOLVED (superseded) diff --git a/tests/test_vpc_seam_contract.py b/tests/test_vpc_seam_contract.py index 46bea17..6d7d09d 100644 --- a/tests/test_vpc_seam_contract.py +++ b/tests/test_vpc_seam_contract.py @@ -8,15 +8,21 @@ stage code, so a drift on EITHER side fails loud here — before the coordinated release (#179) can ship it. -Contract surfaces pinned (all verified against vpc 3.0.0 dev): +Contract surfaces pinned (verified against the RELEASED vpc 3.3.0): - ``ReportingStage.generate_evaluation_report`` calls - ``evaluation_template.generate(source=…, target=…)`` (stage.py:236) — + ``evaluation_template.generate(source=…, target=…)`` — call shape via AST (survives reformatting; a kwarg rename fails). - Our ``EvaluationReportTemplate.generate`` binds exactly that call. -- The stage's ``MetricFrameFileSource(root=…, run_type=…, target=…, - primary_model=…)`` construction binds our ``__init__``, with - ``root=context.model_path.data_generated``. +- The stage constructs pipeline-core's ``PerModelMetricFrameSource`` (one of + OUR ``MetricFrameFileSource``s per model, each at that model's own + ``data_generated`` — vpc #485 / our C-215) with ``primary_root= + context.model_path.data_generated`` and NO ``root_of`` override — the + default per-model resolver must be in force. Until vpc 3.3.0 this file + faithfully locked the old single subject-rooted source, i.e. the defect. +- A real per-model round-trip: frames saved under TWO different roots resolve + through the stage's source via our public ``metric_frame()`` — never a + shared root (the fixture blind spot that hid C-215). - The on-disk layout — "LOCKED cross-repo path contract" (their C-202, our C-192): our ``_frame_dir`` must equal ``root / model / run_type / METRICFRAME_DIR_PREFIX + target`` using the @@ -33,13 +39,18 @@ import ast import inspect +from pathlib import Path import pytest reporting_stage = pytest.importorskip("views_pipeline_core.managers.reporting.stage") evaluation_stage = pytest.importorskip("views_pipeline_core.managers.evaluation.stage") +metric_frame_source = pytest.importorskip( + "views_pipeline_core.managers.reporting.metric_frame_source" +) from views_reporting.sources import MetricFrameFileSource # noqa: E402 +from views_reporting.sources.metric_value import mean_metric_value # noqa: E402 from views_reporting.templates.reports.evaluation import ( # noqa: E402 EvaluationReportTemplate, ) @@ -97,22 +108,84 @@ def test_our_eval_generate_binds_their_call(): sig.bind(None, source=object(), target="sb") # raises TypeError on drift -def test_stage_constructs_source_with_our_init_contract(): - """The stage's ``MetricFrameFileSource(…)`` construction binds our - ``__init__``, and roots it at ``context.model_path.data_generated``.""" - calls = _ctor_calls("MetricFrameFileSource") - assert calls, "stage no longer constructs MetricFrameFileSource" - sig = inspect.signature(MetricFrameFileSource.__init__) +def test_stage_constructs_the_per_model_source_rooted_at_the_subject(): + """The stage builds pipeline-core's per-model ``EvaluationSource``; the SUBJECT + is rooted at ``context.model_path.data_generated``, every other model at its own + (vpc #485 / our C-215). + + History: until vpc 3.3.0 the stage built ONE ``MetricFrameFileSource`` rooted at + the subject, and this test faithfully locked that call — which meant it locked + the defect that blanked every baseline/constituent row (C-215). Corrected in + the same window as their fix, per the register's sequencing note. + """ + calls = _ctor_calls("PerModelMetricFrameSource") + assert calls, "stage no longer constructs PerModelMetricFrameSource" + sig = inspect.signature(metric_frame_source.PerModelMetricFrameSource.__init__) for call in calls: assert not call.args kwargs = _keywords(call) sig.bind(None, **{k: object() for k in kwargs}) - (root_kw,) = [kw for kw in call.keywords if kw.arg == "root"] + # Exactly these, and in particular NO `root_of=`: an override could + # re-express the C-215 defect (every model at the subject's root) + # through the new class; production must use the default per-model + # resolver. + assert kwargs == {"primary_model", "primary_root", "run_type", "target"}, ( + f"stage constructs the per-model source with {sorted(kwargs)}" + ) + (root_kw,) = [kw for kw in call.keywords if kw.arg == "primary_root"] assert isinstance(root_kw.value, ast.Attribute) - assert root_kw.value.attr == "data_generated", ( - "the locked path contract roots at " - "(their C-202 / our C-192)" + assert root_kw.value.attr == "data_generated" + + +def test_per_model_source_resolves_frames_from_each_models_own_root(tmp_path): + """A REAL per-model on-disk round-trip through the public port (#287): + two models, two roots, frames written where the producer writes them + (our locked layout under EACH model's own ``data_generated``), resolved + through the stage's source class via ``metric_frame()`` (test resolver + injected via ``root_of``; production's default resolver is vpc's to test). + Never a shared root — + the fixture blind spot that let C-215 pass CI for months. A source that + regressed to subject-rooting resolves ``m1`` as None here.""" + import sys + + sys.path.insert(0, str(Path(__file__).parent)) + from _eval_source_doubles import make_metric_frame + + prefix = evaluation_stage.METRICFRAME_DIR_PREFIX + roots = { + "ens": tmp_path / "ensembles" / "ens" / "data" / "generated", + "m1": tmp_path / "models" / "m1" / "data" / "generated", + } + expected = {"ens": 0.42, "m1": 0.93} + for model, value in expected.items(): + frame_dir = roots[model] / model / "calibration" / f"{prefix}sb" + frame_dir.parent.mkdir(parents=True) + make_metric_frame({"MSLE": value}, target="sb").save(frame_dir) + + source = metric_frame_source.PerModelMetricFrameSource( + primary_model="ens", + primary_root=roots["ens"], + run_type="calibration", + target="sb", + root_of=lambda name: roots[name], + ) + for model, value in expected.items(): + frame = source.metric_frame(model) + assert frame is not None, f"{model} did not resolve from its own root" + got = mean_metric_value( + frame, eval_type="time-series-wise", target="sb", metric="MSLE" ) + assert got == pytest.approx(value) + + # Both rooted at the subject (the C-215 shape): the other model vanishes. + regressed = metric_frame_source.PerModelMetricFrameSource( + primary_model="ens", + primary_root=roots["ens"], + run_type="calibration", + target="sb", + root_of=lambda name: roots["ens"], + ) + assert regressed.metric_frame("m1") is None def test_frame_path_contract_uses_producers_own_prefix(tmp_path): diff --git a/uv.lock b/uv.lock index 4bf09e2..df341a0 100644 --- a/uv.lock +++ b/uv.lock @@ -2254,18 +2254,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0f/36/4d44f7035ab3c0b2b6a4c4ebb98dedf36246ccbc1b3e2f51ebcd7ac83abb/statsmodels-0.14.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a518d3f9889ef920116f9fa56d0338069e110f823926356946dae83bc9e33e19", size = 10363350, upload-time = "2025-12-05T23:13:53.08Z" }, ] -[[package]] -name = "stepshift" -version = "2.2.6" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "pandas", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "pymonad", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "toolz", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "xarray", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/c8/fa/29c737319bffc9defb1d86a2d0a4caa8b2780ace0abcf11ee2f5ee1d8126/stepshift-2.2.6.tar.gz", hash = "sha256:398681a585c44eb056612b009ad0e3f6d48eb6a9be36f7fc911c8c0dd91af048", size = 166243, upload-time = "2024-08-30T09:28:51.172Z" } - [[package]] name = "strconv" version = "0.4.2" @@ -2358,16 +2346,15 @@ wheels = [ [[package]] name = "views-evaluation" -version = "1.0.0" +version = "2.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/5f/16/984c3c98a3510acf0e9805777ed548c814e9debd9373d609cd01bd3ea17a/views_evaluation-1.0.0.tar.gz", hash = "sha256:8bff41285f3c6455e75e232e95e633140c72d38c15954b6998103634db57d093", size = 38239, upload-time = "2026-08-02T12:48:53.254Z" } +sdist = { url = "https://files.pythonhosted.org/packages/05/1a/ec601a60486acbde59525b1c7f536a91f723db2ae59ecb748091a59dec9c/views_evaluation-2.0.0.tar.gz", hash = "sha256:bde3feac02bf998948412c7807f514bcc03c6ec5e53ba0351a2db3d85d083869", size = 45706, upload-time = "2026-09-18T21:29:24.227Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/b7/38/a6a5d2e26bf0895723fb73b952312903169a8551bcd3d5a93933bb1a37b3/views_evaluation-1.0.0-py3-none-any.whl", hash = "sha256:b5b0b2e49e4dad25eebf0919ad587328ada52b8a9aff001775b7123c36f99c7e", size = 39875, upload-time = "2026-08-02T12:48:54.31Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f7/8268b2aaa5746b82e287776e332ed5ac3871a0505d61850536c705378e31/views_evaluation-2.0.0-py3-none-any.whl", hash = "sha256:f18c3427c6c0d57cb37badbc7c2eb21df685798d0f0d8a477f91e3ebb370f487", size = 47225, upload-time = "2026-09-18T21:29:25.058Z" }, ] [package.optional-dependencies] @@ -2389,22 +2376,24 @@ wheels = [ [[package]] name = "views-pipeline-core" -version = "3.0.0" +version = "3.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "art", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "ingester3", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, + { name = "pandas", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "polars", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, + { name = "pyarrow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "pyprojroot", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, + { name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "views-evaluation", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "views-frames", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "views-transformation-library", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "viewser", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, { name = "wandb", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/be/3a/9a3086c6ec40050aa9f2886919f63e9ac6f570354e6656e376522f6109f2/views_pipeline_core-3.0.0.tar.gz", hash = "sha256:3da2b1c98b45f612ba136dda9374543beb6a89f3e0b58791ab73c69cefdb5e4f", size = 311623, upload-time = "2026-08-03T02:06:23.379Z" } +sdist = { url = "https://files.pythonhosted.org/packages/d1/5d/f004a02c76d20cae90c9778063c2012a25ab49a5cd1465ce91f6736906e3/views_pipeline_core-3.3.0.tar.gz", hash = "sha256:8edf80123d35a626ce282aae8bac94c9b6a56cdf3b0163c0d98c9ddbfb5aa7d1", size = 369635, upload-time = "2026-09-19T01:23:32.565Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/e4/f9/91c3239ed68698a697c1a8f5a763c7ef5feea7d8faab7da0a6b2b4e27123/views_pipeline_core-3.0.0-py3-none-any.whl", hash = "sha256:400aeff47b8b56b443165e587db706596f9129763e1dc792e6dbff817289ff3c", size = 380953, upload-time = "2026-08-03T02:06:24.4Z" }, + { url = "https://files.pythonhosted.org/packages/60/ae/fa74678ee63bd9ab79f8c7de147b241c13cc347f651f19413a545a74bd8a/views_pipeline_core-3.3.0-py3-none-any.whl", hash = "sha256:7c0c7535ee9de7a74528bb64393621e379d57556ee125785d4d149a745dab515", size = 448026, upload-time = "2026-09-19T01:23:34.338Z" }, ] [[package]] @@ -2446,9 +2435,9 @@ requires-dist = [ { name = "pyarrow", specifier = ">=16.0.0,<22.0.0" }, { name = "scipy", specifier = ">=1.15.1,<2.0.0" }, { name = "seaborn", specifier = ">=0.13.2,<0.14.0" }, - { name = "views-evaluation", extras = ["frames"], specifier = ">=1.0.0,<3.0.0" }, + { name = "views-evaluation", extras = ["frames"], specifier = ">=2.0.0,<3.0.0" }, { name = "views-frames", specifier = ">=1.10.2,<2.0.0" }, - { name = "views-pipeline-core", specifier = ">=3.0.0,<4.0.0" }, + { name = "views-pipeline-core", specifier = ">=3.3.0,<4.0.0" }, ] [package.metadata.requires-dev] @@ -2503,23 +2492,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c8/20/0fcf6a0832dfc2e334fddcf0be4ca3d9ce6f16f8a1cbe0dd9f00a235f3bd/views_tensor_utilities-1.1.3-py3-none-any.whl", hash = "sha256:95fa32c3470cf377cceabd3f4daf48625fe1062141d84bf72c3f9718184559aa", size = 15199, upload-time = "2025-10-17T07:28:17.895Z" }, ] -[[package]] -name = "views-transformation-library" -version = "2.7.2" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "ingester3", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "pandas", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "stepshift", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "xarray", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/3f/a4/a22983e3ff6dd153e6d847fa3f10c63ea302c3c1f9eaa7de1a6d640f421c/views_transformation_library-2.7.2.tar.gz", hash = "sha256:1ce9d7c405d74bd6e609ad0c789d27be97f52007809b40f35fc09006b3ce4173", size = 33764, upload-time = "2023-11-21T13:33:25.439Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/db/87/b8ba859534c32cb0ec1b04b64530e6c085f765c59450ef913a381b8c915d/views_transformation_library-2.7.2-py3-none-any.whl", hash = "sha256:8869ebb508ae8b397fa0c9daff8a2c448bf233baa88cd296b599b2978fd87b5f", size = 39357, upload-time = "2023-11-21T13:33:23.759Z" }, -] - [[package]] name = "viewser" version = "6.6.4" @@ -2600,17 +2572,3 @@ sdist = { url = "https://files.pythonhosted.org/packages/2c/41/aa4bf9664e4cda14c wheels = [ { url = "https://files.pythonhosted.org/packages/34/db/b10e48aa8fff7407e67470363eac595018441cf32d5e1001567a7aeba5d2/websocket_client-1.9.0-py3-none-any.whl", hash = "sha256:af248a825037ef591efbf6ed20cc5faa03d3b47b9e5a2230a529eeee1c1fc3ef", size = 82616, upload-time = "2025-10-07T21:16:34.951Z" }, ] - -[[package]] -name = "xarray" -version = "2024.3.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "packaging", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, - { name = "pandas", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/db/ac/9db13d2a6d2e9f72d4d6305ed7796a45ea9cd58f0a4a47d434a276830adf/xarray-2024.3.0.tar.gz", hash = "sha256:5c1db19efdde61db7faedad8fc944f4e29698fb6fbd578d352668b63598bd1d8", size = 3674424, upload-time = "2024-03-29T17:09:00.691Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/f7/fe/c4d15ac730b2bcdd530e4bc6491958c53237eb573dba4eec3ad31ff0519a/xarray-2024.3.0-py3-none-any.whl", hash = "sha256:ca2bc4da2bf2e7879e15862a7a7c3fc76ad19f6a08931d030220cef39a29118d", size = 1134478, upload-time = "2024-03-29T17:08:57.554Z" }, -]