Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/run_pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,11 @@ jobs:
assert u.find_spec('sklearn') is None, 'scikit-learn still installed'
assert u.find_spec('pandas') is None, 'pandas still installed'
ids = {k: np.array(v) for k, v in {'time': [1, 1], 'unit': [1, 2], 'origin': [0, 0], 'step': [1, 1]}.items()}
cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point'], 'evaluation_profile': 'base'}
cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point']}
report = NativeEvaluator(cfg).evaluate(EvaluationFrame(np.array([0., 1.]), np.array([[0.2], [0.8]]), ids, metadata={'target': 't'}))
mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp='2026-09-18T00:00:00')
assert mf.n_rows == 12, mf.n_rows # 2 metrics x (1 group + mean) x 3 schemas
mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp=1758153600, evaluation_timestamp='2026-09-18T00:00:00')
assert isinstance(mf.metadata.provenance.timestamp, int), 'timestamp is int in the cross-repo contract (C-47)'
assert mf.n_rows == len(cfg['classification_point_metrics']) * 2 * 3, mf.n_rows # metrics x (group + mean) x schemas
print('runtime-only install: evaluate + to_metric_frame OK, rows =', mf.n_rows)
PY
poetry install --all-extras
Expand Down
61 changes: 60 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,19 @@ provided they were announced here.

## [Unreleased]

The next release is **2.0.0** (MAJOR, ADR-022 §5): it removes surface that 1.1.0 deprecated.
_Nothing yet._

---

## [2.0.0] — 2026-09-18

**MAJOR** (ADR-022 §5): this release removes the surface 1.1.0 deprecated, one release
cycle later as §2 requires. It is the end of the pandas-free epic (#66): `import
views_evaluation` loads neither pandas nor scikit-learn, no public method returns a
pandas object, and the runtime dependencies are numpy, scipy and the optional `frames`
extra. `MetricFrame` (on disk and in memory), `to_dict()` and `get_schema_results()` are
unchanged from 1.1.0: every test on them passes unmodified. Stories S7 (#63) and S8
(#72); tracking #73.

### Removed

Expand Down Expand Up @@ -51,6 +63,53 @@ The next release is **2.0.0** (MAJOR, ADR-022 §5): it removes surface that 1.1.
pandas is absent, keeps something to prove. The CI runtime-only install step now asserts
pandas is *absent* alongside scikit-learn.

### Release checklist (ADR-022 §7)

- [x] **Does this release do anything rule 2 governs — remove an `__all__` symbol, remove
a supported config key, narrow an accepted input, or change a raised exception type?
If so, did a `DeprecationWarning` ship at least one release ago?** It removes a
documented method of an `__all__` symbol (the removed `EvaluationReport.to_dataframe()`)
and the helper behind it. Yes: the `DeprecationWarning` naming `to_dict()` as the replacement
shipped in **1.1.0** (published 2026-09-18; see its *Deprecated* entry), one full release
cycle before this one, and both release notes record it (§2 rule 3). Said plainly: §2
counts cycles in releases, not days, and both releases were cut on the same date, so a
consumer who never installed 1.1.0 never saw the warning at runtime. The release notes
and the pre-tag notifications below are that consumer's notice; the maintainer chose
two same-day releases over one so the warning exists in a published artifact and the
§2 record is honest rather than skipped. The three
`make_*_evaluation_dict` factories were never documented behaviour of any `__all__`
symbol (§1) and had no caller since 0.4.0; they go as private cleanup and are listed
above for completeness, not because rule 2 reached them. No config key is removed, no
accepted input narrowed, no raised exception type changed.
- [x] **Does this release make previously-accepted input fail? If so, are the release
notes explicit, and have known consumers been notified?** Yes — a call to
`to_dataframe()` (any schema, including `'raw'`) now raises `AttributeError`, and
`pip install views-evaluation[dataframe]` no longer selects an extra. The migration is
above. Known consumers, measured 2026-09-18: **views-pipeline-core** — their released
3.2.0 still calls the removed `report.to_dataframe()` in `managers/evaluation/stage.py`;
their `development` branch does not (their #513) and keeps a deliberate tripwire test
that calls it; their pin `^1.0.0` excludes this release until they move (their #515).
**views-reporting** — `main` and `development` pin `>=1.0.0,<3.0.0`; their released
v0.3.3 still pins `<2.0.0`, so an environment built from that tag excludes this
release too; they call nothing that changed. Both were
notified **before the tag** on 2026-09-18: views-platform/views-pipeline-core#515
(comment 5733334619) and views-platform/views-reporting#289 (comment 5733334868);
the comments are the committed artifact.
- [x] **Does the version bump match the change class (rule 5)?** `1.1.0` → `2.0.0`,
MAJOR. Removing a documented method is breaking under §1/§5; nothing here is argued
MINOR-eligible.
- [x] **Do the release notes list every breaking change with its migration?** Yes — the
one breaking change is the *Removed* entry, with the `to_dict()` recipe, the three ways
it differs from what the old method returned, and the `schema='raw'` replacement. The
dependency change (pandas to dev-only) removes nothing any consumer imported through
this package: the `dataframe` extra was the only way pandas arrived, and it is the
removed surface.
- [x] **Does `MetricFrame`'s format or axis vocabulary change? If so, has it been agreed
with views-reporting and views-pipeline-core?** No. Axes, `MEAN_GROUP_ID`,
`SCHEMA_TO_EVAL_TYPE`, dtype and the `save`/`load` layout are unchanged; the CI
runtime-only step emits a frame from the release commit with pandas and scikit-learn
removed.

---

## [1.1.0] — 2026-09-18
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ The evaluation ontology has been updated to be more explicit and task-specific.

The **VIEWS Evaluation** repository provides a standardized framework for **assessing time-series forecasting models** used in the **VIEWS conflict prediction pipeline**. It ensures consistent, robust, and interpretable evaluations through **metrics tailored to conflict-related data**, which often exhibit **right-skewness and zero-inflation**.

The library is built as a pure NumPy + SciPy core (ADR-011's Level 0) plus a thin emit layer — `EvaluationReport` and `MetricFrame`, the "emit path" of the logging standard §5.1 — with orchestration left to the caller (ADR-011's Level 2, external). All mathematical evaluation logic is independent of Pandas or any other data-frame library — nothing in the package imports one.
The library is built as a pure NumPy + SciPy core (ADR-011's Level 0, which includes `EvaluationReport`) plus a thin emit layer — `MetricFrame` and `EvaluationReport.to_metric_frame()`, the emit path of the logging standard §5.1 — with orchestration left to the caller (ADR-011's Level 2, external). All mathematical evaluation logic is independent of Pandas or any other data-frame library — nothing in the package imports one.

---

Expand Down
10 changes: 5 additions & 5 deletions documentation/CICs/EvaluationReport.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@

**Status:** Active
**Owner:** Evaluation Core
**Last reviewed:** 2026-08-02
**Last reviewed:** 2026-09-18 (2.0.0: DataFrame surface removed)
**Related ADRs:** ADR-010 (Ontology), ADR-041 (Output Schema), views-frames ADR-020 (MetricFrame contract home)

---

## 1. Purpose

A structured, framework-agnostic container for evaluation results. It decouples the mathematical outcomes from their final presentation format (DataFrames, JSON, etc.).
A structured, framework-agnostic container for evaluation results. It decouples the mathematical outcomes from their final presentation format (nested dictionaries for JSON, typed dataclasses, and the `MetricFrame` evaluation-of-record; no DataFrame since 2.0.0).

---

Expand All @@ -26,7 +26,7 @@ A structured, framework-agnostic container for evaluation results. It decouples
- **Multi-Schema Storage**: Guarantees storage of results across Month, Sequence, and Step schemas.
- **Traceability**: Maintains metadata about the target, task type, and prediction type.
- **Representation Agnosticism**: Provides a standard internal representation that can be converted to external formats (dictionary or `MetricFrame`).
- **Field Validation**: Guarantees that computed metric names match dataclass fields. Raises `ValueError` with an actionable message if a metric is computed but has no corresponding field in the typed container (FM1 guard).
- **Field Validation** (`get_schema_results()` only): guarantees that computed metric names match dataclass fields on the typed path. Raises `ValueError` with an actionable message if a metric is computed but has no corresponding field in the typed container (FM1 guard). `to_dict()` and `to_metric_frame()` carry every metric the results hold unvalidated; for catalog metrics the membership check in `NativeEvaluator` and `test_metric_membership_matches_dataclass_fields` close the gap.

---

Expand All @@ -53,7 +53,7 @@ A structured, framework-agnostic container for evaluation results. It decouples
## 6. Failure Modes and Loudness

- Raises `KeyError` if a requested schema is not found in the report.
- Raises `ValueError` if a computed metric name has no corresponding field in the typed metrics dataclass (FM1 guard against silent metric loss).
- `get_schema_results()` raises `ValueError` if a computed metric name has no corresponding field in the typed metrics dataclass (FM1 guard against silent metric loss); nothing is logged.
- Fails loud if the input result structure is inconsistent.
- `to_metric_frame()` raises `ImportError` (loud, actionable) if the optional `views-frames` dependency is not installed.
- `to_metric_frame()` raises `ValueError` if the report contains **no metric values for any schema** (ADR-015 ruling 6, risk register C-30). Such an emit previously produced a structurally valid **zero-row** MetricFrame that satisfied `assert_frame_envelope` and persisted to disk as a legitimate-looking audit artifact recording nothing. The message names the target and which schemas were present-but-empty. A **partial** report — at least one metric value in any schema — still emits normally.
Expand Down Expand Up @@ -91,7 +91,7 @@ schema = report.get_schema_results("month") # dict → typed metrics datacla

- **Green:** `tests/test_evaluation_report.py` — construction, schema access, to_dict. `tests/test_metric_frame.py` — `to_metric_frame()` for all 4 cells, envelope conformance, save/load round-trip, provenance split.
- **Beige:** `tests/test_evaluation_report.py` — empty schemas, single-entry schemas. `tests/test_metric_frame.py` — empty/fully-empty schema, NaN values, run_id None at emit.
- **Red:** `tests/test_evaluation_report.py` — missing schema keys, field mismatch (FM1 guard), the removed DataFrame surface staying removed. `tests/test_metric_frame.py` — fail-loud MetricFrame construction.
- **Red:** `tests/test_evaluation_report.py` — missing schema keys, field mismatch (FM1 guard, asserted silent), the public surface pinned as an allowlist (nothing removed in 2.0.0 comes back under any name). `tests/test_adversarial_inputs.py::TestOptionalExtraAbsentRed` — `to_metric_frame()` with views-frames absent raises `ImportError` naming the `frames` extra and logs it on the emit-path logger. `tests/test_metric_frame.py` — fail-loud MetricFrame construction.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ The following must be logged:
> **Scope note (revised 2026-08-02):** logging responsibility in this repository splits by architectural level (ADR-011). The split is stated positively below so it cannot be re-derived by guesswork as new components are added — the previous wording named three classes and predated `MetricFrame`, which left Level 1 uncovered by omission rather than by decision.
>
> **Level 0 — exempt. Must NOT maintain loggers.**
> `evaluation_frame.py`, `native_evaluator.py`, `evaluation_report.py`, `metric_catalog.py`, `native_metric_calculators.py`.
> `evaluation_frame.py`, `native_evaluator.py`, `evaluation_report.py`, `metric_catalog.py`, `native_metric_calculators.py`, `metrics.py`, `config_schema.py` (the last two added 2026-09-18 so this list, the logging guard and the import-purity guard classify the same files).
> These are pure math and pure registry. They compute and validate; they do not act on the world. They rely on exception propagation per ADR-013, and logging responsibility sits at the orchestration layer (e.g. `views-pipeline-core`). Adding a logger here is a violation, not an improvement: it would duplicate what the orchestrator already records and put I/O in the numeric core.
>
> **Level 1 — must log at `ERROR` before raising.**
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "views_evaluation"
version = "1.1.0"
version = "2.0.0"
description = ""
authors = [
"Xiaolong Sun <xiaolong.sun@pcr.uu.se>",
Expand Down
Loading
Loading