Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 31 additions & 15 deletions .github/workflows/run_pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,37 +33,53 @@ jobs:

- name: Install dependencies
run: |
# --all-extras is REQUIRED, not tidiness. `views-frames` and `pandas` are
# optional extras, and tests/test_metric_frame.py and
# tests/test_evaluation_report.py both guard with a MODULE-LEVEL
# pytest.importorskip. Without the extras those files are skipped whole and
# --all-extras is REQUIRED, not tidiness. `views-frames` is an optional
# extra, and tests/test_metric_frame.py guards with a MODULE-LEVEL
# pytest.importorskip. Without the extra that file is skipped whole and
# CI reports green without ever touching the MetricFrame cross-repo contract
# (ADR-022 §1). Guarded by tests/test_falsification_ci_coverage_gap.py.
poetry install --all-extras

# Nothing else ever exercises "works without the dev group": the local env and the
# steps below always carry scikit-learn. `poetry sync --only main` removes the dev
# group (scikit-learn among it) and keeps the extras; the probe then asserts that
# scikit-learn is genuinely ABSENT — not merely unloaded, which would be vacuous —
# and that importing the package does not load pandas, which the extras install
# (ADR-011; C-05). The reinstall restores the dev group; the verify step below
# proves the extras are present, and the pytest step needs the dev group or it
# cannot collect.
# steps below always carry scikit-learn and pandas. `poetry sync --only main`
# removes the dev group (both among it) and keeps the extras; the probe then
# asserts that scikit-learn and pandas are genuinely ABSENT — not merely unloaded,
# which would be vacuous — and then evaluates a frame and emits a MetricFrame,
# because the emit path imports everything it needs at call time and an
# import-only probe would not execute a line of it (ADR-011; C-05, C-40). The
# "does not LOAD pandas when it is present" half is the purity guard in
# tests/test_metric_calculators.py, which fails rather than skips if pandas is
# missing from the dev group. The reinstall restores the dev group; the verify
# step below proves the extra is present, and the pytest step needs the dev group
# or it cannot collect.
- name: Prove the runtime-only install works
run: |
set -e
poetry sync --only main --all-extras
poetry run python -c "import sys, importlib.util as u, views_evaluation, views_evaluation.evaluation.metric_frame; assert u.find_spec('sklearn') is None, 'scikit-learn still installed'; assert 'pandas' not in sys.modules, 'import loaded pandas'"
poetry run python - <<'PY'
import importlib.util as u
import numpy as np
from views_evaluation import EvaluationFrame, NativeEvaluator
assert u.find_spec('sklearn') is None, 'scikit-learn still installed'
assert u.find_spec('pandas') is None, 'pandas still installed'
ids = {k: np.array(v) for k, v in {'time': [1, 1], 'unit': [1, 2], 'origin': [0, 0], 'step': [1, 1]}.items()}
cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point']}
report = NativeEvaluator(cfg).evaluate(EvaluationFrame(np.array([0., 1.]), np.array([[0.2], [0.8]]), ids, metadata={'target': 't'}))
mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp=1758153600, evaluation_timestamp='2026-09-18T00:00:00')
assert isinstance(mf.metadata.provenance.timestamp, int), 'timestamp is int in the cross-repo contract (C-47)'
assert mf.n_rows == len(cfg['classification_point_metrics']) * 2 * 3, mf.n_rows # metrics x (group + mean) x schemas
print('runtime-only install: evaluate + to_metric_frame OK, rows =', mf.n_rows)
PY
poetry install --all-extras

# Proves the extras ARRIVED, not merely that --all-extras was requested.
# Proves the extra ARRIVED, not merely that --all-extras was requested.
# `poetry install` exits 0 even if it resolves without an optional package, and
# every downstream guard is a module-level pytest.importorskip that turns absence
# the downstream guard is a module-level pytest.importorskip that turns absence
# into silence. Without this step a partial install silently shrinks the suite
# and CI still reports green. Guarded by
# tests/test_falsification_extras_actually_installed.py.
- name: Verify optional extras are installed
run: poetry run python -c "import views_frames, pandas"
run: poetry run python -c "import views_frames"

- name: Run tests
run: |
Expand Down
91 changes: 91 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,97 @@ _Nothing yet._

---

## [2.0.0] — 2026-09-18

**MAJOR** (ADR-022 §5): this release removes the surface 1.1.0 deprecated, one release
cycle later as §2 requires. It is the end of the pandas-free epic (#66): `import
views_evaluation` loads neither pandas nor scikit-learn, no public method returns a
pandas object, and the runtime dependencies are numpy, scipy and the optional `frames`
extra. `MetricFrame` (on disk and in memory), `to_dict()` and `get_schema_results()` are
unchanged from 1.1.0: every test on them passes unmodified. Stories S7 (#63) and S8
(#72); tracking #73.

### Removed

- **Removed: `EvaluationReport.to_dataframe()`** and, with it, `BaseEvaluationMetrics.evaluation_dict_to_dataframe()`,
the three `make_*_evaluation_dict` factories (unused since 0.4.0 removed `EvaluationManager`,
their only caller; never documented, never warned on — they go as private cleanup), and
the **`dataframe` extra** (`pip install views-evaluation[dataframe]` is no longer a thing
to install). Breaking. The `DeprecationWarning` for the removed `to_dataframe()` and
`evaluation_dict_to_dataframe()` shipped in 1.1.0 (see that section's *Deprecated*
entry). views-pipeline-core, the one production caller: its `development` branch stopped
calling it (their #513, observed 2026-09-16), but as of 2026-09-18 that is in no release
of theirs — their released 3.2.0 still calls the removed `to_dataframe()`, and their
`^1.0.0` pin excludes this release until they move (their #515, after views-reporting).
**Migration:**
build the table in the caller —
`pd.DataFrame.from_dict(report.to_dict()['schemas'][schema], orient='index')`. Same
values; not byte-identical, as the 1.1.0 entry spells out: the old method dropped any
column that was `nan` in every group (register **C-40**, closed by this removal —
`to_dict()` and `to_metric_frame()` now are the only export paths and agree on the
metric set by construction), ordered columns by dataclass field, and kept an empty
group as a `nan` row. `schema='raw'` callers use `to_dict()['schemas']`.
`get_schema_results()` and the four typed dataclasses' fields are unchanged (the
dataclasses lose the four inherited helpers named above and nothing else).
- `examples/benchmark_probabilistic_scaling.py`, a relic that never imported this package.

### Changed

- **Nothing under `views_evaluation/` imports pandas any more**, lazily or otherwise; the
package's runtime dependencies are numpy, scipy and the optional `frames` extra. pandas
moves to the dev group so the import-purity guard, which fails rather than skips when
pandas is absent, keeps something to prove. The CI runtime-only install step now asserts
pandas is *absent* alongside scikit-learn.

### Release checklist (ADR-022 §7)

- [x] **Does this release do anything rule 2 governs — remove an `__all__` symbol, remove
a supported config key, narrow an accepted input, or change a raised exception type?
If so, did a `DeprecationWarning` ship at least one release ago?** It removes a
documented method of an `__all__` symbol (the removed `EvaluationReport.to_dataframe()`)
and the helper behind it. Yes: the `DeprecationWarning` naming `to_dict()` as the replacement
shipped in **1.1.0** (published 2026-09-18; see its *Deprecated* entry), one full release
cycle before this one, and both release notes record it (§2 rule 3). Said plainly: §2
counts cycles in releases, not days, and both releases were cut on the same date, so a
consumer who never installed 1.1.0 never saw the warning at runtime. The release notes
and the pre-tag notifications below are that consumer's notice; the maintainer chose
two same-day releases over one so the warning exists in a published artifact and the
§2 record is honest rather than skipped. The three
`make_*_evaluation_dict` factories were never documented behaviour of any `__all__`
symbol (§1) and had no caller since 0.4.0; they go as private cleanup and are listed
above for completeness, not because rule 2 reached them. No config key is removed, no
accepted input narrowed, no raised exception type changed.
- [x] **Does this release make previously-accepted input fail? If so, are the release
notes explicit, and have known consumers been notified?** Yes — a call to
`to_dataframe()` (any schema, including `'raw'`) now raises `AttributeError`, and
`pip install views-evaluation[dataframe]` no longer selects an extra. The migration is
above. Known consumers, measured 2026-09-18: **views-pipeline-core** — their released
3.2.0 still calls the removed `report.to_dataframe()` in `managers/evaluation/stage.py`;
their `development` branch does not (their #513) and keeps a deliberate tripwire test
that calls it; their pin `^1.0.0` excludes this release until they move (their #515).
**views-reporting** — `main` and `development` pin `>=1.0.0,<3.0.0`; their released
v0.3.3 still pins `<2.0.0`, so an environment built from that tag excludes this
release too; they call nothing that changed. Both were
notified **before the tag** on 2026-09-18: views-platform/views-pipeline-core#515
(comment 5733334619) and views-platform/views-reporting#289 (comment 5733334868);
the comments are the committed artifact.
- [x] **Does the version bump match the change class (rule 5)?** `1.1.0` → `2.0.0`,
MAJOR. Removing a documented method is breaking under §1/§5; nothing here is argued
MINOR-eligible.
- [x] **Do the release notes list every breaking change with its migration?** Yes — the
one breaking change is the *Removed* entry, with the `to_dict()` recipe, the three ways
it differs from what the old method returned, and the `schema='raw'` replacement. The
dependency change (pandas to dev-only) removes nothing any consumer imported through
this package: the `dataframe` extra was the only way pandas arrived, and it is the
removed surface.
- [x] **Does `MetricFrame`'s format or axis vocabulary change? If so, has it been agreed
with views-reporting and views-pipeline-core?** No. Axes, `MEAN_GROUP_ID`,
`SCHEMA_TO_EVAL_TYPE`, dtype and the `save`/`load` layout are unchanged; the CI
runtime-only step emits a frame from the release commit with pandas and scikit-learn
removed.

---

## [1.1.0] — 2026-09-18

The first release cut through the gated publish path (S2, #68): `publish_package.yml` now
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ The evaluation ontology has been updated to be more explicit and task-specific.

The **VIEWS Evaluation** repository provides a standardized framework for **assessing time-series forecasting models** used in the **VIEWS conflict prediction pipeline**. It ensures consistent, robust, and interpretable evaluations through **metrics tailored to conflict-related data**, which often exhibit **right-skewness and zero-inflation**.

The library is built on a **three-layer architecture** with a framework-agnostic NumPy core, ensuring that all mathematical evaluation logic is independent of Pandas or any other data-frame library.
The library is built as a pure NumPy + SciPy core (ADR-011's Level 0, which includes `EvaluationReport`) plus a thin emit layer — `MetricFrame` and `EvaluationReport.to_metric_frame()`, the emit path of the logging standard §5.1 — with orchestration left to the caller (ADR-011's Level 2, external). All mathematical evaluation logic is independent of Pandas or any other data-frame library — nothing in the package imports one.

---

Expand Down Expand Up @@ -119,8 +119,8 @@ report = evaluator.evaluate(ef)
# 3. Access results
report.to_dict() # {'target', 'task', 'pred_type', 'schemas': {'step': {...}, ...}}
report.get_schema_results("month") # typed metrics dataclass
# to_dataframe() is deprecated and removed in 2.0.0; build a DataFrame from
# to_dict()['schemas'][schema] in the caller if you need one.
# Need a DataFrame? Build it in the caller:
# pd.DataFrame.from_dict(report.to_dict()['schemas']['month'], orient='index')
```

> For the full walkthrough including input formatting and sample evaluation, see [`documentation/integration_guide.md`](documentation/integration_guide.md).
Expand Down Expand Up @@ -257,7 +257,7 @@ pip install views_evaluation
---
## 🏗 **Architecture**

The library follows a strict three-layer architecture (ADR-011):
The library follows the strict layering of ADR-011; the first two levels live in this repository:

```
Level 0 — Pure Core (NumPy + SciPy; no dataframe libraries)
Expand All @@ -268,7 +268,7 @@ Level 0 — Pure Core (NumPy + SciPy; no dataframe libraries)
Profiles Named hyperparameter sets (base, hydranet_ucdp, ...)

Level 1 — Bridge / Emit
EvaluationReport Results container with dict / MetricFrame export (DataFrame export deprecated)
EvaluationReport Results container with dict / MetricFrame export
MetricFrame Typed, provenance-stamped evaluation-of-record (views-frames ADR-020)

Level 2 — Orchestration
Expand Down
2 changes: 1 addition & 1 deletion documentation/ADRs/041_evaluation_output_schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The 2025 version of this ADR asserted that *"views-evaluation returns a structur

- `to_dict()` — nested plain dict; the pandas-free, extra-free path.
- `get_schema_results(schema)` — mapped onto the typed metrics dataclasses.
- `to_dataframe(schema)` — pandas DataFrame; requires the optional `dataframe` extra. **Deprecated; removed in 2.0.0** (ADR-022 §2): build the DataFrame from `to_dict()` in the caller.
- `to_dataframe(schema)` — *removed in 2.0.0* after a 1.1.0 `DeprecationWarning` (ADR-022 §2). It returned a pandas DataFrame behind the optional `dataframe` extra; a caller builds one from `to_dict()` now.

`views-evaluation` does **not** serialise these to JSON or render HTML. That remains the orchestrator's job, and the original rationale still holds for them: formatting and run context belong where the run is orchestrated.

Expand Down
Loading
Loading