Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions documentation/CICs/cic_evaluation_report_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

> **What is this class for?**

EvaluationReportTemplate **renders** self-contained HTML evaluation reports for VIEWS forecasting models and ensembles from an **injected `EvaluationSource`** β€” a typed `MetricFrame` per model (ADR-018 / C-108) β€” rather than acquiring metrics itself at render time. It assembles the comparative metric tables, optionally renders historical-vs-predicted line graphs from on-disk predictions (parquet or numpy PredictionFrame per ADR-012), and delegates all HTML assembly to `ReportModule`. This class does **not** touch WandB: it renders purely from the injected `EvaluationSource` (a `MetricFrame` per model), which the caller (pipeline-core's reporting stage) constructs as a `MetricFrameFileSource` reading the persisted frame.
EvaluationReportTemplate **renders** self-contained HTML evaluation reports for VIEWS forecasting models and ensembles from an **injected `EvaluationSource`** β€” a typed `MetricFrame` per model (ADR-018 / C-108) β€” rather than acquiring metrics itself at render time. It assembles the comparative metric tables, optionally renders historical-vs-predicted line graphs from on-disk predictions (parquet or numpy PredictionFrame per ADR-012), and delegates all HTML assembly to `ReportModule`. This class does **not** touch WandB: it renders purely from the injected `EvaluationSource` (a `MetricFrame` per model), which the caller (pipeline-core's reporting stage) constructs as an `EvaluationSource` (since pipeline-core 3.3.0 its `PerModelMetricFrameSource`, one `MetricFrameFileSource` per model) reading the persisted frame.

---

Expand All @@ -35,7 +35,7 @@ EvaluationReportTemplate **renders** self-contained HTML evaluation reports for
- **Canonical metric selection (ADR-017).** The metric *set* shown is the **central canonical standard** in `ReportingConfig.canonical_report_metrics` (keyed by `{regression,classification}Γ—{point,sample}`), **not** the model's own list. A model occupies a cell when its `<task>_<pred_type>_metrics` config key is present & non-empty (declared, not inferred β€” ADR-003); one labelled canonical table is rendered per active cell. Values come from each model's `MetricFrame` via `mean_metric_value` (the `group_id="mean"` row); a canonical metric the frame lacks (absent row or NaN) is shown with an explicit "not calculated β€” add `<metric>` to `<key>`" note (ADR-008), never dropped silently. **C-116:** more than one matching mean row raises `AmbiguousMetric` and renders a visible "ambiguous β€” multiple matching keys" cell rather than a silently-picked value.
- **Baseline model inclusion.** Collects baseline model names from tier-specific config keys (`regression_point_baselines`, `regression_sample_baselines`, `classification_point_baselines`, `classification_sample_baselines`) and includes them in the comparative metric tables (lines 173-181).
- **Constituent model verification.** For ensemble reports, the template asks `source.metric_frame(model)` for each declared constituent and verifies `level`/`partition` consistency across the resolved frames (via `unique_axis_value` over the frame axes), raising `ValueError` on mismatch (C-48 β€” a single ensemble row must not mix constituents from different levels/partitions).
- **No run selection in the template (C-48 resolved).** The template does not select WandB runs; it only consumes `source.metric_frame(model)`. The 22/25 production blanking is resolved structurally: pipeline-core persists the evaluation-of-record `MetricFrame` and injects a `MetricFrameFileSource`, so there is no render-time run to mis-select. The interim metric-aware selection seam (`evaluation_run_resolver`) has been deleted (B2 / C-108); the same-partition stale-relog concern moves upstream to the producer (C-110, gated on #220 provenance).
- **No run selection in the template (C-48 resolved).** The template does not select WandB runs; it only consumes `source.metric_frame(model)`. The 22/25 production blanking is resolved structurally: pipeline-core persists the evaluation-of-record `MetricFrame` and injects an `EvaluationSource` (one `MetricFrameFileSource` per model via `PerModelMetricFrameSource` since pipeline-core 3.3.0), so there is no render-time run to mis-select. The interim metric-aware selection seam (`evaluation_run_resolver`) has been deleted (B2 / C-108); the same-partition stale-relog concern moves upstream to the producer (C-110, gated on #220 provenance).
- **Constituent resolution β€” degrade-and-announce (#105, honoring the #177 contract).** A declared constituent is never silently dropped. The source reporting **ABSENT** (`metric_frame(model)` returns `None`) records the model and **announces** it under "Model Metrics" as missing. A `metric_frame` call that **raises** (a **transient** failure) is **retried once**; if it still fails the model is **marked degraded** and announced distinctly (a retrieval error, not a confirmed absence). Absent and degraded models contribute no metrics row. Opt-in **strict mode** (`config["strict_constituents"] = True`) instead raises `ValueError` listing the absent/degraded models. (ADR-008 fail-loud / make-degradation-visible.) The subject model is always rendered (an absent subject shows "not calculated" cells).
- **Metric table sorting.** Sorts combined metric DataFrames by MSLE first, then CRPS, then the first available metric (lines 265-282).
- **Prediction sample graphs (non-fatal).** `_add_prediction_sample_graphs()` is wrapped in a `try/except` at the caller (lines 297-302). A failure in graph generation does not invalidate the metrics tables already written to the report.
Expand Down
11 changes: 7 additions & 4 deletions documentation/CICs/cic_evaluation_source.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

---

> **Scope note.** Covers the `EvaluationSource` *interface* (`typing.Protocol`), its single concrete adapter (`MetricFrameFileSource` β€” durable, loads the persisted `MetricFrame`), the `EvaluationProvenance` value object, and the pure value queries (`mean_metric_value`, `unique_axis_value`, `AmbiguousMetric`). This is the evaluation-side counterpart of the prediction `loaders/` Ingestion surface (C-108 / #173).
> **Scope note.** Covers the `EvaluationSource` *interface* (`typing.Protocol`), its single concrete adapter in this repo (`MetricFrameFileSource` β€” durable, loads the persisted `MetricFrame`), the `EvaluationProvenance` value object, and the pure value queries (`mean_metric_value`, `unique_axis_value`, `AmbiguousMetric`). This is the evaluation-side counterpart of the prediction `loaders/` Ingestion surface (C-108 / #173).

Sources: `views_reporting/sources/_protocol.py`, `metric_frame_file_source.py`, `evaluation_provenance.py`, `metric_value.py`, `__init__.py`.

Expand All @@ -35,7 +35,7 @@ Sources: `views_reporting/sources/_protocol.py`, `metric_frame_file_source.py`,

- **Port (`EvaluationSource`, a `typing.Protocol`).** Provides `metric_frame(model: str) -> MetricFrame | None` and `provenance() -> EvaluationProvenance`. A source is bound to one `target` at construction (a report is per-target), so only `model` varies per call. The render code is statically checkable against the interface (LSP/ISP); a new source is a new class, not a change here (OCP). `views_evaluation` is imported only under `TYPE_CHECKING` in the port so it stays import-light (SDP/SAP).
- **Failure taxonomy (the #105/#177 contract).** `metric_frame` returns **`None` = absent** (no evaluation for this model β†’ the report degrades-and-announces, never a silent drop); **raises = transient** (a retrieval hiccup β†’ the report retries once, then marks the model degraded).
- **`MetricFrameFileSource` (durable β€” the only implementation).** `metric_frame(model)` loads the persisted frame for `(model, run_type, target)` or returns `None` when its directory is absent; a corrupt/unreadable frame propagates (transient). `provenance()` is read from the subject model's frame metadata (run_id / evaluation_timestamp / data_version / scoring_code_version; no WandB url/owner).
- **`MetricFrameFileSource` (durable β€” the only implementation in this repo; pipeline-core's `PerModelMetricFrameSource` composes it per model).** `metric_frame(model)` loads the persisted frame for `(model, run_type, target)` or returns `None` when its directory is absent; a corrupt/unreadable frame propagates (transient). `provenance()` is read from the subject model's frame metadata (run_id / evaluation_timestamp / data_version / scoring_code_version; no WandB url/owner).
- **Value query (`mean_metric_value`).** Reads the `group_id="mean"` row for `(eval_type, target, metric)`: `None` when absent or NaN ("not calculated"); raises `AmbiguousMetric` on >1 matching mean row (C-116; ADR-008); else the `float`.
- **Axis query (`unique_axis_value`).** The single distinct value of an axis (`level`/`partition`) across a frame, for the cross-constituent consistency guard; raises if the axis is non-uniform within a frame.
- **`EvaluationProvenance`.** A frozen presentation DTO (`run_id` + optional `run_url`/`owner`/`run_date`/`data_version`/`scoring_code_version`) rendered with None-omission, so each source supplies only what it knows. `MetricFrameFileSource` populates `run_id` and the frame provenance (`data_version`, `scoring_code_version`); `run_url`/`owner` remain generic optional fields that no current source populates.
Expand Down Expand Up @@ -84,7 +84,10 @@ Nothing is silently dropped or silently guessed.
## 8. Examples of Correct Usage

```python
# pipeline-core's reporting stage constructs the source and calls generate:
# Direct construction (a test, a script, or a caller that owns ONE root). Since
# pipeline-core 3.3.0 the reporting stage instead injects its
# PerModelMetricFrameSource, which composes one of these per model, each at
# that model's own data/generated (vpc #485 / register C-215):
source = MetricFrameFileSource(root, run_type="calibration", target="lr_ged_sb",
primary_model="first_love")
template.generate(source=source, target="lr_ged_sb")
Expand Down Expand Up @@ -117,7 +120,7 @@ except AmbiguousMetric:

## 11. Evolution Notes

- The interim WandB scrape has been removed: pipeline-core now persists frames and injects a `MetricFrameFileSource`, the only source implementation. The render path imports no WandB, and the clickable WandB link/owner have fallen away (no source populates `run_url`/`owner`; they remain generic optional fields on `EvaluationProvenance` for a future source).
- The interim WandB scrape has been removed: pipeline-core persists frames and injects an `EvaluationSource` built on this class β€” one `MetricFrameFileSource` per model since pipeline-core 3.3.0 (`PerModelMetricFrameSource`, vpc #485; register C-215 records why a single subject-rooted source blanked every comparison row before that). The render path imports no WandB, and the clickable WandB link/owner have fallen away (no source populates `run_url`/`owner`; they remain generic optional fields on `EvaluationProvenance` for a future source).
- The on-disk layout is a locked cross-repo contract with pipeline-core's producer (C-192/C-202), executable-pinned in `tests/test_vpc_seam_contract.py`.
- The cross-constituent consistency check moved onto frame axes (`level`/`partition`); when run-resolved partition windows are plumbed into the producer (#220), they sharpen.

Expand Down
22 changes: 15 additions & 7 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@ readme = "README.md"
# (risk register C-36).
requires-python = ">=3.11,<3.15"
dependencies = [
"views-pipeline-core>=3.0.0,<4.0.0",
# Floor 3.3.0 (#287, 2026-09-19): the first release whose reporting stage
# resolves every comparison model's MetricFrame from THAT model's own
# data/generated (PerModelMetricFrameSource, vpc #485) β€” the fix for
# baseline/constituent rows vanishing from every ensemble report (C-215).
# Our seam test locks that contract; <3.3.0 stages would silently blank them.
"views-pipeline-core>=3.3.0,<4.0.0",
# views-frames: the leaf data contract β€” PredictionFrame / TargetFrame +
# SpatioTemporalIndex / SpatialLevel, plus the views_frames_summarize sibling
# (MAP/HDI/quantiles), which ships in the SAME distribution (one line). numpy-only,
Expand All @@ -43,12 +48,15 @@ dependencies = [
# `[frames]` extra survives; the only merged changes to metric_frame.py since
# 1.0.0 are to the provenance stamp's VALUE β€” the `+g<sha>` suffix and S1's
# bounding of it β€” which we read as opaque text). The span keeps
# co-resolution with views-pipeline-core across its move to ^2.0.0. A >=2.0.0
# FLOOR is a separate step, gated on 2.0.0 actually reaching PyPI β€” a floor
# naming an unpublished version is the C-213(a) mistake (Cluster H).
"views-evaluation[frames]>=1.0.0,<3.0.0",
# views-transformation-library removed: never imported here and already
# pulled transitively via views-pipeline-core (falsify P3).
# co-resolution with views-pipeline-core across its move to ^2.0.0.
# Floor 2.0.0 (2026-09-19): raised only once BOTH gates held β€” 2.0.0 on PyPI
# (2026-09-18) AND vpc 3.3.0 itself requiring >=2.0.0 (its to_dataframe
# egress removed) β€” so the shared env co-resolves. Re-qualified: suite green
# on 2.0.0, a 2.0.0-emitted frame round-trips through our source exactly.
"views-evaluation[frames]>=2.0.0,<3.0.0",
# views-transformation-library removed: never imported here (falsify P3);
# it was transitive via views-pipeline-core until 3.3.0 dropped it β€” now
# simply absent, which is correct.
# viewser RETIRED from runtime dependencies (C-22 Resolved, epic #204): the render
# path reads the bundled metadata parquets (metadata/data/); only
# scripts/build_entity_metadata.py uses viewser, at dev time β€” see the
Expand Down
Loading
Loading