diff --git a/.github/workflows/run_pytest.yml b/.github/workflows/run_pytest.yml index c67e6e6..e8dd1b2 100644 --- a/.github/workflows/run_pytest.yml +++ b/.github/workflows/run_pytest.yml @@ -63,10 +63,11 @@ jobs: assert u.find_spec('sklearn') is None, 'scikit-learn still installed' assert u.find_spec('pandas') is None, 'pandas still installed' ids = {k: np.array(v) for k, v in {'time': [1, 1], 'unit': [1, 2], 'origin': [0, 0], 'step': [1, 1]}.items()} - cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point'], 'evaluation_profile': 'base'} + cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point']} report = NativeEvaluator(cfg).evaluate(EvaluationFrame(np.array([0., 1.]), np.array([[0.2], [0.8]]), ids, metadata={'target': 't'})) - mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp='2026-09-18T00:00:00') - assert mf.n_rows == 12, mf.n_rows # 2 metrics x (1 group + mean) x 3 schemas + mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp=1758153600, evaluation_timestamp='2026-09-18T00:00:00') + assert isinstance(mf.metadata.provenance.timestamp, int), 'timestamp is int in the cross-repo contract (C-47)' + assert mf.n_rows == len(cfg['classification_point_metrics']) * 2 * 3, mf.n_rows # metrics x (group + mean) x schemas print('runtime-only install: evaluate + to_metric_frame OK, rows =', mf.n_rows) PY poetry install --all-extras diff --git a/CHANGELOG.md b/CHANGELOG.md index d0fafd8..f362b77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,19 @@ provided they were announced here. ## [Unreleased] -The next release is **2.0.0** (MAJOR, ADR-022 §5): it removes surface that 1.1.0 deprecated. +_Nothing yet._ + +--- + +## [2.0.0] — 2026-09-18 + +**MAJOR** (ADR-022 §5): this release removes the surface 1.1.0 deprecated, one release +cycle later as §2 requires. It is the end of the pandas-free epic (#66): `import +views_evaluation` loads neither pandas nor scikit-learn, no public method returns a +pandas object, and the runtime dependencies are numpy, scipy and the optional `frames` +extra. `MetricFrame` (on disk and in memory), `to_dict()` and `get_schema_results()` are +unchanged from 1.1.0: every test on them passes unmodified. Stories S7 (#63) and S8 +(#72); tracking #73. ### Removed @@ -51,6 +63,53 @@ The next release is **2.0.0** (MAJOR, ADR-022 §5): it removes surface that 1.1. pandas is absent, keeps something to prove. The CI runtime-only install step now asserts pandas is *absent* alongside scikit-learn. +### Release checklist (ADR-022 §7) + +- [x] **Does this release do anything rule 2 governs — remove an `__all__` symbol, remove + a supported config key, narrow an accepted input, or change a raised exception type? + If so, did a `DeprecationWarning` ship at least one release ago?** It removes a + documented method of an `__all__` symbol (the removed `EvaluationReport.to_dataframe()`) + and the helper behind it. Yes: the `DeprecationWarning` naming `to_dict()` as the replacement + shipped in **1.1.0** (published 2026-09-18; see its *Deprecated* entry), one full release + cycle before this one, and both release notes record it (§2 rule 3). Said plainly: §2 + counts cycles in releases, not days, and both releases were cut on the same date, so a + consumer who never installed 1.1.0 never saw the warning at runtime. The release notes + and the pre-tag notifications below are that consumer's notice; the maintainer chose + two same-day releases over one so the warning exists in a published artifact and the + §2 record is honest rather than skipped. The three + `make_*_evaluation_dict` factories were never documented behaviour of any `__all__` + symbol (§1) and had no caller since 0.4.0; they go as private cleanup and are listed + above for completeness, not because rule 2 reached them. No config key is removed, no + accepted input narrowed, no raised exception type changed. +- [x] **Does this release make previously-accepted input fail? If so, are the release + notes explicit, and have known consumers been notified?** Yes — a call to + `to_dataframe()` (any schema, including `'raw'`) now raises `AttributeError`, and + `pip install views-evaluation[dataframe]` no longer selects an extra. The migration is + above. Known consumers, measured 2026-09-18: **views-pipeline-core** — their released + 3.2.0 still calls the removed `report.to_dataframe()` in `managers/evaluation/stage.py`; + their `development` branch does not (their #513) and keeps a deliberate tripwire test + that calls it; their pin `^1.0.0` excludes this release until they move (their #515). + **views-reporting** — `main` and `development` pin `>=1.0.0,<3.0.0`; their released + v0.3.3 still pins `<2.0.0`, so an environment built from that tag excludes this + release too; they call nothing that changed. Both were + notified **before the tag** on 2026-09-18: views-platform/views-pipeline-core#515 + (comment 5733334619) and views-platform/views-reporting#289 (comment 5733334868); + the comments are the committed artifact. +- [x] **Does the version bump match the change class (rule 5)?** `1.1.0` → `2.0.0`, + MAJOR. Removing a documented method is breaking under §1/§5; nothing here is argued + MINOR-eligible. +- [x] **Do the release notes list every breaking change with its migration?** Yes — the + one breaking change is the *Removed* entry, with the `to_dict()` recipe, the three ways + it differs from what the old method returned, and the `schema='raw'` replacement. The + dependency change (pandas to dev-only) removes nothing any consumer imported through + this package: the `dataframe` extra was the only way pandas arrived, and it is the + removed surface. +- [x] **Does `MetricFrame`'s format or axis vocabulary change? If so, has it been agreed + with views-reporting and views-pipeline-core?** No. Axes, `MEAN_GROUP_ID`, + `SCHEMA_TO_EVAL_TYPE`, dtype and the `save`/`load` layout are unchanged; the CI + runtime-only step emits a frame from the release commit with pandas and scikit-learn + removed. + --- ## [1.1.0] — 2026-09-18 diff --git a/README.md b/README.md index f6a7441..fea074c 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ The evaluation ontology has been updated to be more explicit and task-specific. The **VIEWS Evaluation** repository provides a standardized framework for **assessing time-series forecasting models** used in the **VIEWS conflict prediction pipeline**. It ensures consistent, robust, and interpretable evaluations through **metrics tailored to conflict-related data**, which often exhibit **right-skewness and zero-inflation**. -The library is built as a pure NumPy + SciPy core (ADR-011's Level 0) plus a thin emit layer — `EvaluationReport` and `MetricFrame`, the "emit path" of the logging standard §5.1 — with orchestration left to the caller (ADR-011's Level 2, external). All mathematical evaluation logic is independent of Pandas or any other data-frame library — nothing in the package imports one. +The library is built as a pure NumPy + SciPy core (ADR-011's Level 0, which includes `EvaluationReport`) plus a thin emit layer — `MetricFrame` and `EvaluationReport.to_metric_frame()`, the emit path of the logging standard §5.1 — with orchestration left to the caller (ADR-011's Level 2, external). All mathematical evaluation logic is independent of Pandas or any other data-frame library — nothing in the package imports one. --- diff --git a/documentation/CICs/EvaluationReport.md b/documentation/CICs/EvaluationReport.md index 6d4ade5..6894652 100644 --- a/documentation/CICs/EvaluationReport.md +++ b/documentation/CICs/EvaluationReport.md @@ -2,14 +2,14 @@ **Status:** Active **Owner:** Evaluation Core -**Last reviewed:** 2026-08-02 +**Last reviewed:** 2026-09-18 (2.0.0: DataFrame surface removed) **Related ADRs:** ADR-010 (Ontology), ADR-041 (Output Schema), views-frames ADR-020 (MetricFrame contract home) --- ## 1. Purpose -A structured, framework-agnostic container for evaluation results. It decouples the mathematical outcomes from their final presentation format (DataFrames, JSON, etc.). +A structured, framework-agnostic container for evaluation results. It decouples the mathematical outcomes from their final presentation format (nested dictionaries for JSON, typed dataclasses, and the `MetricFrame` evaluation-of-record; no DataFrame since 2.0.0). --- @@ -26,7 +26,7 @@ A structured, framework-agnostic container for evaluation results. It decouples - **Multi-Schema Storage**: Guarantees storage of results across Month, Sequence, and Step schemas. - **Traceability**: Maintains metadata about the target, task type, and prediction type. - **Representation Agnosticism**: Provides a standard internal representation that can be converted to external formats (dictionary or `MetricFrame`). -- **Field Validation**: Guarantees that computed metric names match dataclass fields. Raises `ValueError` with an actionable message if a metric is computed but has no corresponding field in the typed container (FM1 guard). +- **Field Validation** (`get_schema_results()` only): guarantees that computed metric names match dataclass fields on the typed path. Raises `ValueError` with an actionable message if a metric is computed but has no corresponding field in the typed container (FM1 guard). `to_dict()` and `to_metric_frame()` carry every metric the results hold unvalidated; for catalog metrics the membership check in `NativeEvaluator` and `test_metric_membership_matches_dataclass_fields` close the gap. --- @@ -53,7 +53,7 @@ A structured, framework-agnostic container for evaluation results. It decouples ## 6. Failure Modes and Loudness - Raises `KeyError` if a requested schema is not found in the report. -- Raises `ValueError` if a computed metric name has no corresponding field in the typed metrics dataclass (FM1 guard against silent metric loss). +- `get_schema_results()` raises `ValueError` if a computed metric name has no corresponding field in the typed metrics dataclass (FM1 guard against silent metric loss); nothing is logged. - Fails loud if the input result structure is inconsistent. - `to_metric_frame()` raises `ImportError` (loud, actionable) if the optional `views-frames` dependency is not installed. - `to_metric_frame()` raises `ValueError` if the report contains **no metric values for any schema** (ADR-015 ruling 6, risk register C-30). Such an emit previously produced a structurally valid **zero-row** MetricFrame that satisfied `assert_frame_envelope` and persisted to disk as a legitimate-looking audit artifact recording nothing. The message names the target and which schemas were present-but-empty. A **partial** report — at least one metric value in any schema — still emits normally. @@ -91,7 +91,7 @@ schema = report.get_schema_results("month") # dict → typed metrics datacla - **Green:** `tests/test_evaluation_report.py` — construction, schema access, to_dict. `tests/test_metric_frame.py` — `to_metric_frame()` for all 4 cells, envelope conformance, save/load round-trip, provenance split. - **Beige:** `tests/test_evaluation_report.py` — empty schemas, single-entry schemas. `tests/test_metric_frame.py` — empty/fully-empty schema, NaN values, run_id None at emit. -- **Red:** `tests/test_evaluation_report.py` — missing schema keys, field mismatch (FM1 guard), the removed DataFrame surface staying removed. `tests/test_metric_frame.py` — fail-loud MetricFrame construction. +- **Red:** `tests/test_evaluation_report.py` — missing schema keys, field mismatch (FM1 guard, asserted silent), the public surface pinned as an allowlist (nothing removed in 2.0.0 comes back under any name). `tests/test_adversarial_inputs.py::TestOptionalExtraAbsentRed` — `to_metric_frame()` with views-frames absent raises `ImportError` naming the `frames` extra and logs it on the emit-path logger. `tests/test_metric_frame.py` — fail-loud MetricFrame construction. --- diff --git a/documentation/standards/logging_and_observability_standard.md b/documentation/standards/logging_and_observability_standard.md index 203ef33..13ef2e2 100644 --- a/documentation/standards/logging_and_observability_standard.md +++ b/documentation/standards/logging_and_observability_standard.md @@ -128,7 +128,7 @@ The following must be logged: > **Scope note (revised 2026-08-02):** logging responsibility in this repository splits by architectural level (ADR-011). The split is stated positively below so it cannot be re-derived by guesswork as new components are added — the previous wording named three classes and predated `MetricFrame`, which left Level 1 uncovered by omission rather than by decision. > > **Level 0 — exempt. Must NOT maintain loggers.** -> `evaluation_frame.py`, `native_evaluator.py`, `evaluation_report.py`, `metric_catalog.py`, `native_metric_calculators.py`. +> `evaluation_frame.py`, `native_evaluator.py`, `evaluation_report.py`, `metric_catalog.py`, `native_metric_calculators.py`, `metrics.py`, `config_schema.py` (the last two added 2026-09-18 so this list, the logging guard and the import-purity guard classify the same files). > These are pure math and pure registry. They compute and validate; they do not act on the world. They rely on exception propagation per ADR-013, and logging responsibility sits at the orchestration layer (e.g. `views-pipeline-core`). Adding a logger here is a violation, not an improvement: it would duplicate what the orchestrator already records and put I/O in the numeric core. > > **Level 1 — must log at `ERROR` before raising.** diff --git a/pyproject.toml b/pyproject.toml index 166cd29..6b497f4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "views_evaluation" -version = "1.1.0" +version = "2.0.0" description = "" authors = [ "Xiaolong Sun ", diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index ee7e754..4f15561 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -14,6 +14,32 @@ > review then found four gaps in the rewrite (linked-worktree `commondir`, locale-dependent > reads, unchecked ref contents, an unpinned name gate) — all fixed and tested before merge. 18 → 17. +> **2026-09-18 — release 2.0.0 (S8, #72).** The MAJOR that ends the pandas-free epic: `to_dataframe()` +> and the `dataframe` extra gone one release cycle after 1.1.0's `DeprecationWarning` (ADR-022 §2). +> §3.2 notifications posted before the tag and cited by comment ID in the checklist (views-pipeline-core +> #515, views-reporting #289), the second release to do so. Consumer state at the cut, measured: +> views-pipeline-core's released 3.2.0 still calls the removed method and pins `^1.0.0`, so 2.0.0 +> reaches them only when S10 raises the pin; views-reporting's `main` span admits it (their released +> v0.3.3 still pins `<2.0.0`) and calls nothing removed. The release review found fifteen more guard +> defects, all fixed on the release branch and each re-verified red: the S7 import guard keyed its +> allowlist by file stem (so `profiles/__init__.py` inherited the root package's allowance) and read +> a string literal under ANY call (a `nan` package on sys.path would have turned `float("nan")` into a +> red build); both guards are now one walker with one path-keyed allowlist set to the measured census, +> reading literals only under import machinery and its aliases, with `exec`/`eval` forbidden everywhere +> and the runtime probe also exercising `save`/`load`. The doc guard now scans the CHANGELOG section of +> the release being cut (it froze it as history), uses word-anchored removal framings, and lost the +> five block-wide 2.0-era markers that had widened the older guards' exemption. `to_dict()`'s live-dict +> identity, `get_schema_results()` on all three schemas, the report's public surface as an allowlist, +> FM1's silence and the CI probe's `timestamp` type (an ISO string had been passed into the contract's +> `int` field) are now pinned. The independent guard audit then ran 64 mutations and found five weak +> guards; all closed and re-verified red: the runtime probe now evaluates EVERY implemented kernel in +> all four cells (a spec-loaded or `getattr`-imported pandas in an uncalled kernel was invisible), the +> Level-0 crossing check sees `from views_evaluation.evaluation import metric_frame`, the public-surface +> guard probes the removed names behaviourally and bans `__getattr__` (a shim with a split token beat +> the `dir()` listing), the logging guard checks for Logger objects rather than a text pattern, the +> dependency guard handles both pyproject layouts, and the doc guard scans fenced code in the section +> being cut with real line numbers. Residuals stated in the docstrings. Count unchanged. + > **2026-09-18 — story S7 (#63) closed C-40 and Cluster A's second residue.** `to_dataframe()`, its > DataFrame helper and the three dead factories are gone; the `dataframe` extra is deleted and > pandas is a dev-only dependency (the purity guard needs it present). C-15's closed-row rationale @@ -362,7 +388,7 @@ Moved out of the register (no correctness/reliability dimension) — tracked in | C-05 | 3 | sklearn/scipy in pure-math core | `AP` and `MTD` reimplemented in numpy as transcriptions of scikit-learn 1.7.2 (S4, #70: kernels alongside + parity suite, seen red on 26 injected defects; S5, #64: the switch). scikit-learn moved to the dev group as the parity oracle. `TestLevelZeroImportPurity` asserts a fresh `import views_evaluation` loads neither scikit-learn nor pandas and fails (not skips) if either is uninstalled; a CI step proves the runtime-only install. scipy stays (ADR-011 permits it). The empty-group AP case was ruled on the way (D-01, ADR-015 R9). | #70, #64 | | C-44 | 4 | `to_dataframe()` without pandas raises a bare `ModuleNotFoundError`, unlike its guarded sibling | `find_spec("pandas")` gate raising `ModuleNotFoundError(name="pandas")` that names `pip install views-evaluation[dataframe]` — same type as the bare import, so no caller's `except` changes; no log, since Level 0 does not log (standard §5.1). Test `tests/test_adversarial_inputs.py::TestOptionalExtraAbsentRed` (outside the pandas-gated module so it runs where the case applies; observed red on the bare import first). The entry's second cited site, `metrics.py`'s `evaluation_dict_to_dataframe`, is deprecated with the method and deleted in S7 rather than guarded. Same change deprecates the method (S3, #69). | #69 | | C-39 | 2 | `scoring_code_version` stamps the consumer's git SHA when the wheel is installed inside the consumer's checkout | `_source_git_sha` bounded to the directory containing `views_evaluation/` (`parents[2]`, flat layout), accepted only when a `pyproject.toml` there declares this distribution (parsed with `tomllib`, PEP 503-normalised); reads UTF-8; follows a `.git` file's `gitdir:` relative to the file and a linked worktree's `commondir`; rejects non-hash ref contents; logs at WARNING when `.git` is ours but unreadable. Tests `TestSourceGitShaBoundary{Red,Green,Beige}` in `tests/test_metric_frame.py` (consumer venv, vendored copy, unreadable state, loose/packed refs, hyphenated name, linked worktree, submodule); the consumer-venv and relative-`gitdir:` cases were observed failing on the old walker first. CHANGELOG, CIC MetricFrame.md and CIC EvaluationReport.md corrected. Epic #66, story #67; the `/code-review max` of that PR found the `commondir`, locale, hex-check and name-gate gaps and they were fixed before merge. | #67 | -| C-40 | 3 | `to_dataframe()` silently erases any metric that is `nan` in every group | `to_dataframe()`, `BaseEvaluationMetrics.evaluation_dict_to_dataframe()` (the `df.loc[:, df.notna().any()]` filter) and the three unused `make_*_evaluation_dict` factories deleted; pandas moved from the `dataframe` extra (deleted) to the dev group so the import-purity guard keeps something to prove; `test_removed_dataframe_surface_stays_removed` fails on a same-name reintroduction (class or `__getattr__` shim); the package-wide forbidden-import guard fails on a pandas import under any name. Ships in 2.0.0 after 1.1.0's `DeprecationWarning`; closed 2026-09-18. The remaining two export paths, `to_dict()` and `to_metric_frame()`, agree on the metric set by construction. | #63 | +| C-40 | 3 | `to_dataframe()` silently erases any metric that is `nan` in every group | `to_dataframe()`, `BaseEvaluationMetrics.evaluation_dict_to_dataframe()` (the `df.loc[:, df.notna().any()]` filter) and the three unused `make_*_evaluation_dict` factories deleted; pandas moved from the `dataframe` extra (deleted) to the dev group so the import-purity guard keeps something to prove; `test_removed_dataframe_surface_stays_removed` fails on a same-name reintroduction (class or `__getattr__` shim); the package-wide import guard (one allowlist per file, over import statements and string literals handed to import machinery under any alias) fails on a pandas import however it is spelled short of string concatenation. Ships in 2.0.0 after 1.1.0's `DeprecationWarning`; closed 2026-09-18. The remaining two export paths, `to_dict()` and `to_metric_frame()`, agree on the metric set by construction. | #63 | ### Closed by the Fail-Loud Doctrine epic (#26), 2026-08-02 diff --git a/tests/test_documentation_contracts.py b/tests/test_documentation_contracts.py index 0b7b5f9..8cde229 100644 --- a/tests/test_documentation_contracts.py +++ b/tests/test_documentation_contracts.py @@ -68,9 +68,14 @@ "make_step_wise_evaluation_dict", "make_month_wise_evaluation_dict", ) +# Word-anchored phrases that predicate removal on the thing named. Ordinary +# collocations (`used to`, `was the`, `deleted`) exempted "is used to build the table" +# and an unanchored `removed` matched `removed_cols = ...` (release review, 2026-09-18); +# `\b` on both sides is what keeps `removed_cols` out (`_` is a word character). _REMOVAL_FRAMING = re.compile( - r"removed|remove[sd]? in|gone in|until 2\.|deprecated|deleted|was the|were the|" - r"used to|no longer|left in|retired|purged|lost the", re.I, + r"\b(?:removed|removal|gone in \d|until \d+\.\d|no longer|deprecated|" + r"retired|purged|lost the|left in \d|now raises)\b", + re.I, ) # Files where a historical mention is legitimate: change logs, decision records and @@ -93,9 +98,6 @@ # or add the new phrasing here — otherwise the check fails for a purely cosmetic edit. _HISTORICAL_MARKERS = ( "until 0.", "prior to 0.", "before 0.", - # 2.0-era framing, added with the 2.0.0 removals: a dated CHANGELOG section written - # before 2.0.0 says "removed in 2.0.0"; a docstring written after says "until 2.0.0". - "until 2.", "removed in 2.", "gone in 2.", "since 2.", "before 2.", "was removed", "were removed", "was deleted", "were deleted", "removed in phase 3", "deleted in phase 3", "replaced by", "were replaced", "renamed", "breaking rename", @@ -277,16 +279,18 @@ def test_no_doc_or_docstring_names_a_deleted_class_as_current(self): def test_no_doc_or_docstring_presents_a_removed_method_as_current(self): """`to_dataframe()` left in 2.0.0. Eight documents were hand-edited for it; this is what makes the ninth a red build instead of a C-29 repeat. A mention is exempt - only with a removal framing within a line of it, which is how the metrics.py + only with a removal framing ON THE SAME LINE, which is how the metrics.py docstring ("Until 2.0.0 it also carried") and the CIC's Evolution Notes pass; - dated CHANGELOG sections are frozen and skipped. Seen red on + dated CHANGELOG sections of published releases are frozen and skipped, the + section being cut is scanned. Seen red on `report.to_dataframe('month')` injected into the integration guide, the `[Unreleased]` section, a docstring, a CIC, an example, and — after the guard audit — with a space before the paren, as a bare table token, as an attribute, and with "no longer" one line away. Known residual: a single LINE that both frames a removal and advertises the method ("X is no longer shipped; call - to_dataframe()") passes; no text heuristic distinguishes that from a - legitimate "to_dataframe() is no longer available".""" + to_dataframe()"), or an identifier named `removed`, or a `# deprecated` + trailer, passes; no text heuristic distinguishes those from a legitimate + "to_dataframe() is no longer available".""" import views_evaluation pkg_root = Path(views_evaluation.__file__).parent offenders = [] @@ -294,20 +298,35 @@ def test_no_doc_or_docstring_presents_a_removed_method_as_current(self): # (so the CIC of the very class that lost the method is NOT exempt here), and # `examples/` is scanned by nothing else. ADRs and reports stay history. sources = ([README, CHANGELOG] + sorted(DOCS.rglob("*.md")) + sorted(pkg_root.rglob("*.py")) - + sorted((REPO_ROOT / "examples").glob("*.py"))) + + sorted((REPO_ROOT / "examples").rglob("*.py")) + sorted((REPO_ROOT / "examples").rglob("*.md"))) pattern = re.compile(r"\b(" + "|".join(_DELETED_METHODS) + r")\b") for path in sources: rel = path.relative_to(REPO_ROOT).as_posix() - if rel.startswith(("documentation/ADRs", "documentation/contributor_protocols", "reports")): - continue + if rel.startswith(("documentation/ADRs", "documentation/contributor_protocols")): + continue # history by nature (the CICs are NOT skipped: framed line by line) text = path.read_text(encoding="utf-8") - if path == CHANGELOG: - # A dated `## [x.y.z]` section is a frozen record of what that release - # did and legitimately names what it deprecated; only `[Unreleased]` - # (everything above the first dated heading) makes a current claim. - text = re.split(r"^## \[\d+\.\d+\.\d+\]", text, maxsplit=1, flags=re.M)[0] lines = text.splitlines() + scan = set(range(len(lines))) + if path == CHANGELOG: + # A dated section of an already-published release is a frozen record and + # legitimately names what it deprecated. `[Unreleased]` AND the section + # for the version pyproject declares — the release being cut, which the + # checklist points consumers at — make current claims and are scanned, + # fenced code included (a fence-stripped body hid a "how to" example in + # the [2.0.0] section; guard audit, 2026-09-18). Headings are located + # with the shared parser's regex on the RAW text so line numbers are real. + TestChangelogCoversTheDeclaredVersion._released_sections(text) # structural asserts + pyproject = tomllib.loads((REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")) + version = pyproject.get("project", {}).get("version") or pyproject["tool"]["poetry"]["version"] + heading = TestChangelogCoversTheDeclaredVersion._HEADING + starts = [i for i, line in enumerate(lines) if heading.match(line)] + scan = set(range(starts[0] if starts else len(lines))) + for k, i in enumerate(starts): + if heading.match(lines[i]).group("version") == version: + scan |= set(range(i, starts[k + 1] if k + 1 < len(starts) else len(lines))) for i, line in enumerate(lines): + if i not in scan: + continue m = pattern.search(line) if not m: continue @@ -513,20 +532,31 @@ def test_changelog_has_a_section_for_the_pyproject_version(self): class TestLoggingScopeContract: """The Level-0/Level-1 split from logging standard §5.1, asserted against code.""" + # Must agree with logging standard §5.1 and with `TestLevelZeroImportPurity._LEVEL_0` + # (tests/test_metric_calculators.py): `metrics` and `config_schema` were classified + # Level 0 by the import guard on 2026-09-18 and belong here too. _LEVEL_0 = [ "evaluation_frame", "native_evaluator", "evaluation_report", - "metric_catalog", "native_metric_calculators", + "metric_catalog", "native_metric_calculators", "metrics", "config_schema", ] @pytest.mark.parametrize("module_name", _LEVEL_0) def test_level_zero_modules_declare_no_module_logger(self, module_name): import importlib + import logging module = importlib.import_module(f"views_evaluation.evaluation.{module_name}") - src = inspect.getsource(module) - # A module-level `logger = logging.getLogger(__name__)` is the violation. - assert not re.search(r"^logger\s*=\s*logging\.getLogger", src, re.MULTILINE), ( + # Behavioural, not textual: any module-level binding that IS a Logger, whatever + # it is called and however it was obtained (`_logger = ...`, `from logging import + # getLogger`) — a regex on `^logger = logging.getLogger` missed both (guard audit, + # 2026-09-18). The text check stays for the canonical spelling in a docstring. + loggers = sorted(k for k, v in vars(module).items() if isinstance(v, (logging.Logger, logging.LoggerAdapter))) + assert not loggers, ( f"{module_name} is Level 0 and must not maintain a module logger " - f"(logging standard §5.1)" + f"(logging standard §5.1); found {loggers}" + ) + src = inspect.getsource(module) + assert not re.search(r"^\w+\s*=\s*(?:logging\.)?getLogger\(", src, re.MULTILINE), ( + f"{module_name} binds a logger at module level (logging standard §5.1)" ) def test_standard_names_both_levels_explicitly(self): diff --git a/tests/test_evaluation_report.py b/tests/test_evaluation_report.py index 53e74ae..43e5e6b 100644 --- a/tests/test_evaluation_report.py +++ b/tests/test_evaluation_report.py @@ -11,6 +11,8 @@ without the `dataframe` extra every test here was silently skipped. It now runs in every environment; the report has no pandas surface left. """ +import logging + import pytest from views_evaluation.evaluation.evaluation_report import EvaluationReport @@ -96,6 +98,36 @@ def test_get_schema_results_regression_point_returns_correct_cls(self): assert schema['month100'].MSE == 1.0 assert schema['month101'].MSE == 2.0 + @pytest.mark.parametrize("schema, groups", [ + ("month", {"month100": 1.0, "month101": 2.0}), + ("time_series", {"ts00": 1.5}), + ("step", {"step01": 1.0, "step02": 2.0}), + ]) + def test_get_schema_results_maps_every_schema(self, schema, groups): + """All three schemas, not only `month`: until 2.0.0 the `time_series` and `step` + paths were exercised only through the deleted to_dataframe tests, so a + `get_schema_results` that only looked up `month` would have gone unnoticed + (release review, 2026-09-18).""" + report = EvaluationReport('t', 'regression', 'point', _regression_point_results()) + mapped = report.get_schema_results(schema) + assert {k: v.MSE for k, v in mapped.items()} == groups + assert all(isinstance(v, RegressionPointEvaluationMetrics) for v in mapped.values()) + + def test_to_dict_schemas_is_the_live_structure_not_a_copy(self): + """Documented identity (1.0.0 behaviour): `to_dict()['schemas']` IS the internal + dict — the same object across calls and the object `to_metric_frame()` reads. + Pinned so a silent switch to a copy (or a silent switch of the emit path to a + different object) is a red build rather than an invisible behaviour change.""" + results = _regression_point_results() + report = EvaluationReport('t', 'regression', 'point', results) + first = report.to_dict()['schemas'] + assert first is report.to_dict()['schemas'] + assert first is results + first['month']['month100']['MSE'] = 42.0 + mf = report.to_metric_frame() + row = (mf.identifiers["group_id"] == "month100") & (mf.identifiers["metric"] == "MSE") + assert mf.values[row].item() == 42.0, "the emit path reads the same live structure" + def test_get_schema_results_regression_sample_returns_correct_cls(self): report = EvaluationReport('t', 'regression', 'sample', _regression_sample_results()) schema = report.get_schema_results('month') @@ -164,31 +196,51 @@ def test_non_dict_schema_value_fails_at_access(self): with pytest.raises(AttributeError): report.get_schema_results('month') - def test_removed_dataframe_surface_stays_removed(self): + def test_public_surface_is_exactly_the_documented_one(self): """2.0.0 removed to_dataframe() and the evaluation_dict_to_dataframe helper after 1.1.0's DeprecationWarning (ADR-022 §2; register C-40), and the three - make_*_evaluation_dict factories as private cleanup (unused since 0.4.0). This - pins those five NAMES, on the class and on an instance (a `__getattr__` shim - serves instances only). A helper under another name is not caught here; the - import-purity guard catches any pandas import whatever the name.""" + make_*_evaluation_dict factories as private cleanup (unused since 0.4.0). Pinned + as an ALLOWLIST of public names (a helper under a new name — `to_records`, + `as_rows` — is a red build and must go through the CIC) PLUS a behavioural probe + of the removed names on instances and a ban on `__getattr__`, because `dir()` + does not list what `__getattr__` serves (release review and guard audit, + 2026-09-18). The import-purity guard separately catches any pandas import + whatever the name.""" + import dataclasses report = EvaluationReport('t', 'regression', 'point', _regression_point_results()) - assert not hasattr(report, "to_dataframe") - assert not hasattr(type(report), "to_dataframe") + + def public(obj): + return sorted(n for n in dir(obj) if not n.startswith("_")) + + removed = ("to_dataframe", "evaluation_dict_to_dataframe", "make_time_series_wise_evaluation_dict", + "make_step_wise_evaluation_dict", "make_month_wise_evaluation_dict") + assert public(EvaluationReport) == ["get_schema_results", "to_dict", "to_metric_frame"] + assert public(report) == ["get_schema_results", "pred_type", "target", "task", "to_dict", "to_metric_frame"] + # `dir()` never lists what a `__getattr__` serves, so the removed names are ALSO + # probed behaviourally on an instance (guard audit, 2026-09-18: a shim survived + # the listing alone), and no `__getattr__` may exist at all. + for name in removed: + assert not hasattr(report, name), f"EvaluationReport().{name} is served (a shim?)" + assert not hasattr(EvaluationReport, "__getattr__") for cls in (RegressionPointEvaluationMetrics, RegressionSampleEvaluationMetrics, ClassificationPointEvaluationMetrics, ClassificationSampleEvaluationMetrics): - for name in ("evaluation_dict_to_dataframe", "make_time_series_wise_evaluation_dict", - "make_step_wise_evaluation_dict", "make_month_wise_evaluation_dict"): - assert not hasattr(cls, name), f"{cls.__name__}.{name} is back" - assert not hasattr(cls(), name), f"{cls.__name__}().{name} is served by a shim" - - def test_metric_with_no_dataclass_field_raises(self): + fields = sorted(f.name for f in dataclasses.fields(cls)) + assert public(cls) == fields, f"{cls.__name__} carries non-field public attributes" + assert public(cls()) == fields, f"{cls.__name__}() serves non-field attributes (a shim?)" + for name in removed: + assert not hasattr(cls(), name), f"{cls.__name__}().{name} is served (a shim?)" + assert not hasattr(cls, "__getattr__") + + def test_metric_with_no_dataclass_field_raises(self, caplog): """FM1 guard: a computed metric that has no field on the typed dataclass raises instead of being dropped. The CIC has claimed a Red test for this since before 2.0.0; there was none, and the guard could be replaced by `continue` unnoticed.""" report = EvaluationReport('t', 'regression', 'point', { 'month': {'month100': {'MSE': 1.0, 'BOGUS': 9.9}}, 'time_series': {}, 'step': {}}) - with pytest.raises(ValueError, match="no field exists"): - report.get_schema_results('month') + with caplog.at_level(logging.DEBUG): + with pytest.raises(ValueError, match="no field exists"): + report.get_schema_results('month') + assert not caplog.records, "Level 0 does not log (logging standard §5.1); only the emit path does" def test_get_schema_results_unknown_schema_raises_key_error(self): report = EvaluationReport('t', 'regression', 'point', {}) diff --git a/tests/test_falsification_extras_actually_installed.py b/tests/test_falsification_extras_actually_installed.py index 75cfae3..fe11048 100644 --- a/tests/test_falsification_extras_actually_installed.py +++ b/tests/test_falsification_extras_actually_installed.py @@ -457,10 +457,11 @@ class TestPublishGateIsReal: assert u.find_spec('sklearn') is None, 'scikit-learn still installed' assert u.find_spec('pandas') is None, 'pandas still installed' ids = {k: np.array(v) for k, v in {'time': [1, 1], 'unit': [1, 2], 'origin': [0, 0], 'step': [1, 1]}.items()} - cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point'], 'evaluation_profile': 'base'} + cfg = {'steps': [1], 'classification_targets': ['t'], 'classification_point_metrics': ['AP', 'Brier_cls_point']} report = NativeEvaluator(cfg).evaluate(EvaluationFrame(np.array([0., 1.]), np.array([[0.2], [0.8]]), ids, metadata={'target': 't'})) - mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp='2026-09-18T00:00:00') - assert mf.n_rows == 12, mf.n_rows # 2 metrics x (1 group + mean) x 3 schemas + mf = report.to_metric_frame(model_id='ci', run_type='test', timestamp=1758153600, evaluation_timestamp='2026-09-18T00:00:00') + assert isinstance(mf.metadata.provenance.timestamp, int), 'timestamp is int in the cross-repo contract (C-47)' + assert mf.n_rows == len(cfg['classification_point_metrics']) * 2 * 3, mf.n_rows # metrics x (group + mean) x schemas print('runtime-only install: evaluate + to_metric_frame OK, rows =', mf.n_rows) PY poetry install --all-extras diff --git a/tests/test_metric_calculators.py b/tests/test_metric_calculators.py index a25d337..b1be561 100644 --- a/tests/test_metric_calculators.py +++ b/tests/test_metric_calculators.py @@ -1814,183 +1814,244 @@ class TestLevelZeroImportPurity: pandas in `sys.modules` — the "zero knowledge of external data frameworks" claim was true of the source and false of the interpreter (C-05). - Three guards, because an audit showed two half-guards leave a seam. The static one - walks every import statement — top-level or lazy — in each Level-0 module, requires - its root to be allowed, forbids importing the Level-1 module (`metric_frame`) from - the pure modules, and forbids dynamic-import machinery (`__import__`, - `importlib.import_module`, `exec`) outright, since a call-time import spelled as a - call is invisible to an AST walk over import statements. The dynamic one imports the - package in a fresh interpreter AND exercises a kernel of each kind and one full - evaluation, then asserts that neither pandas nor scikit-learn was loaded; it is + Three kinds of guard, because an audit showed two half-guards leave a seam. The + static one walks every file in the package and every import in it — statement or + string literal handed to import machinery, top-level or lazy — and requires each + root to be on that file's allowlist; Level 0 files may name only numpy and scipy + and may not import the Level-1 module; `exec`/`eval` and non-literal machinery + calls are forbidden everywhere. The dynamic one imports the package in a fresh + interpreter AND exercises a kernel of each kind, one full evaluation, one emit and + one save/load, then asserts that neither pandas nor scikit-learn was loaded; it is meaningful only if both are installed and not loaded, so it fails, not skips, when - they are missing. The third pins the declaration: scikit-learn is in the dev group - only. + they are missing. The third pins the declarations: the runtime dependency set and + the extras, exactly, under either pyproject layout. """ - _PURE_LEVEL_0 = ["evaluation_frame", "native_evaluator", "metric_catalog", "native_metric_calculators", - "metrics", "config_schema"] - # Package-wide allowlist, by module stem: what each file may import from outside the - # standard library and this package, statically OR dynamically. Level 0 gets numpy and - # scipy (ADR-011); the two Level-1 bridge modules add views_frames. Every other file - # (`__init__`, `profiles/*`, `adapters/*`, `config_schema`, `metrics`) gets nothing. - # An allowlist, not a denylist: `import xarray` in metric_frame.py is as wrong as - # `import pandas`, and a denylist can only name the frameworks it has met. - _PACKAGE_WIDE_ALLOWED = { - "evaluation_frame": {"numpy", "scipy"}, "native_evaluator": {"numpy", "scipy"}, - "metric_catalog": {"numpy", "scipy"}, "native_metric_calculators": {"numpy", "scipy"}, - "evaluation_report": {"numpy", "views_frames"}, "metric_frame": {"numpy", "views_frames"}, - "__init__": {"views_frames"}, # the find_spec gate that exposes MetricFrame + # ONE declaration, keyed by path relative to the package: what each file may import + # from outside the standard library and this package, statically or through import + # machinery. Set to the measured census (2026-09-18) — no dead allowances, so a + # module that stops importing scipy must drop it here too. Level 0 modules may name + # only numpy and scipy (ADR-011; pinned by `test_the_allowlist_is_adr_011s`); the + # two Level-1 bridge modules add views_frames (the root `__init__` only PROBES for it + # with `find_spec`, which imports nothing). Every other file gets nothing. An allowlist, not a + # denylist: `import xarray` in metric_frame.py is as wrong as `import pandas`, and a + # denylist can only name the frameworks it has met. Keyed by path, not stem: a stem + # key `__init__` gave `profiles/__init__.py` the root package's allowance (release + # review, 2026-09-18). + _ALLOWED_BY_PATH = { + "evaluation/evaluation_frame.py": {"numpy"}, + "evaluation/native_evaluator.py": {"numpy"}, + "evaluation/native_metric_calculators.py": {"numpy", "scipy"}, + "evaluation/evaluation_report.py": {"numpy", "views_frames"}, + "evaluation/metric_frame.py": {"numpy", "views_frames"}, } - # ADR-011 §Layering: "No external imports except numpy and scipy". The package's own - # modules are allowed by name below, not by root, so intra-package topology is checked. + # ADR-011 §Layering: "No external imports except numpy and scipy" for Level 0. _ALLOWED_EXTERNAL = {"numpy", "scipy"} + _LEVEL_0 = ["evaluation/evaluation_frame.py", "evaluation/native_evaluator.py", + "evaluation/metric_catalog.py", "evaluation/native_metric_calculators.py", + "evaluation/metrics.py", "evaluation/config_schema.py"] _LEVEL_1_MODULES = {"metric_frame"} - # evaluation_report.py is Level 0 by the logging standard but carries one lazy - # Level-1 bridge by design (`to_metric_frame` → views_frames and metric_frame). Held - # to the same rule plus that. (`to_dataframe` → pandas was the second bridge until - # 2.0.0 removed it; pandas is no longer allowed anywhere in the package.) - _BRIDGE_EXTERNAL = {"views_frames"} + # The import machinery, by the names it is bound to. A string literal handed to one + # of these (positionally or as `name=`) is an import. Only these callees are read — + # matching a literal under ANY call made `float("nan")` an import of a `nan` package + # whenever one was installed or a `nan/` directory sat on sys.path (release review, + # 2026-09-18). Aliases (`from importlib import import_module as _load`) are resolved + # per file. `exec`/`eval` are forbidden everywhere: an import spelled inside a string + # is invisible to any AST walk. + _IMPORT_MACHINERY = {"__import__", "import_module"} + _FORBIDDEN_CALLS = {"exec", "eval"} @staticmethod - def _imports_of(module_name): - """(external roots, internal module names, dynamic-import call names) for a module.""" + def _package_root(): + from pathlib import Path + import views_evaluation + return Path(views_evaluation.__file__).parent + + @classmethod + def _imports_of(cls, rel_path): + """(external roots, internal module names, dynamic-import call names) for the + package file at `rel_path`.""" + return cls._imports_in((cls._package_root() / rel_path).read_text(encoding="utf-8")) + + @classmethod + def _imports_in(cls, source): + """The same, from source text: import statements at any depth, plus string + literals passed to import machinery (by any local alias), plus the names of + forbidden calls.""" import ast - import importlib - import inspect - - module = importlib.import_module(f"views_evaluation.evaluation.{module_name}") - tree = ast.parse(inspect.getsource(module)) + tree = ast.parse(source) + machinery = set(cls._IMPORT_MACHINERY) + for node in ast.walk(tree): # local aliases of the machinery + if isinstance(node, ast.ImportFrom) and node.module == "importlib": + machinery |= {a.asname or a.name for a in node.names if a.name in cls._IMPORT_MACHINERY} external, internal, dynamic = set(), set(), set() + + def root_of(dotted): + head = dotted.split(".")[0] + (internal if head == "views_evaluation" else external).add( + dotted.split(".")[-1] if head == "views_evaluation" else head) + for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: - root = alias.name.split(".")[0] - (internal if root == "views_evaluation" else external).add(alias.name.split(".")[-1] if root == "views_evaluation" else root) + root_of(alias.name) elif isinstance(node, ast.ImportFrom): if node.level > 0 or (node.module or "").split(".")[0] == "views_evaluation": internal.add((node.module or "").split(".")[-1] or "") - internal |= {alias.name for alias in node.names} if node.level > 0 else set() + # The imported NAMES too, for absolute imports as well as relative: + # `from views_evaluation.evaluation import metric_frame` recorded + # only `evaluation` and the Level-1 crossing check never saw + # `metric_frame` (guard audit, 2026-09-18). + internal |= {alias.name for alias in node.names} elif node.module: external.add(node.module.split(".")[0]) elif isinstance(node, ast.Call): fn = node.func name = fn.id if isinstance(fn, ast.Name) else (fn.attr if isinstance(fn, ast.Attribute) else None) - if name in {"__import__", "import_module", "exec", "eval"}: + if name in cls._FORBIDDEN_CALLS: dynamic.add(name) + elif name in machinery: + literal = next((a for a in node.args[:1]), None) or next( + (kw.value for kw in node.keywords if kw.arg == "name"), None) + if isinstance(literal, ast.Constant) and isinstance(literal.value, str): + root_of(literal.value) + dynamic.add(name) + else: + dynamic.add(f"{name}()") return external, internal, dynamic - @pytest.mark.parametrize("module_name", _PURE_LEVEL_0 + ["evaluation_report"]) - def test_every_import_in_a_level_zero_module_is_allowed(self, module_name): + @pytest.mark.parametrize("rel_path", sorted(str(p) for p in []) or [ + "__init__.py", "adapters/__init__.py", "evaluation/__init__.py", + "evaluation/config_schema.py", "evaluation/evaluation_frame.py", + "evaluation/evaluation_report.py", "evaluation/metric_catalog.py", + "evaluation/metric_frame.py", "evaluation/metrics.py", "evaluation/native_evaluator.py", + "evaluation/native_metric_calculators.py", "profiles/__init__.py", "profiles/base.py", + "profiles/hydranet_ucdp.py", + ]) + def test_every_import_in_the_package_is_on_its_files_allowlist(self, rel_path): + """Every file, static and machinery imports alike. Seen red (2026-09-18) on: + `import pandas` in metrics.py and profiles/base.py; `from importlib import + import_module as _load; _load("pandas")`; `importlib.import_module("pandas")` + and `import xarray` in metric_frame.py; `__import__("pandas")` in + profiles/base.py; `import_module(name="pandas")`; a hard `import views_frames` + in profiles/__init__.py; and `exec("import pandas")` anywhere. A statically + allowed root spelled through machinery is still red: the package imports by + statement only.""" import sys stdlib = sys.stdlib_module_names - external, internal, dynamic = self._imports_of(module_name) - allowed = self._ALLOWED_EXTERNAL | (self._BRIDGE_EXTERNAL if module_name == "evaluation_report" else set()) + external, internal, dynamic = self._imports_of(rel_path) + allowed = self._ALLOWED_BY_PATH.get(rel_path, set()) offending = sorted(r for r in external if r not in stdlib and r not in allowed) assert not offending, ( - f"{module_name} imports {offending}; Level 0 may import only the standard " - f"library, numpy and scipy (ADR-011; C-05)" + f"{rel_path} imports {offending}, which its allowlist does not name " + f"(ADR-011; C-05, C-40). Level 0 may import only the standard library, numpy and scipy." ) - if module_name != "evaluation_report": + if rel_path in self._LEVEL_0: + assert allowed <= self._ALLOWED_EXTERNAL, f"{rel_path} is Level 0 (ADR-011: numpy and scipy only)" crossing = sorted(internal & self._LEVEL_1_MODULES) assert not crossing, ( - f"{module_name} imports Level-1 {crossing}; Level 0 must not depend on the " + f"{rel_path} imports Level-1 {crossing}; Level 0 must not depend on the " f"emit layer (ADR-011 layering) — that would also make the core unimportable " f"without the `frames` extra" ) + forbidden = sorted(d for d in dynamic if d in self._FORBIDDEN_CALLS or d.endswith("()")) + assert not forbidden, ( + f"{rel_path} uses {forbidden}; an import spelled as a string or built at run time " + f"is invisible to this guard and forbidden anywhere in the package" + ) assert not dynamic, ( - f"{module_name} uses {sorted(dynamic)}; dynamic imports are invisible to this " - f"guard and forbidden in Level 0" + f"{rel_path} uses import machinery {sorted(dynamic)}; the package imports by statement " + f"only (`find_spec` is a probe, not machinery, and is allowed)" ) - def test_the_static_guard_is_not_vacuous(self): - """It must see numpy (an `import`) AND scipy (an `import from`) in the kernels, - or one of its two branches is scanning nothing.""" - external, internal, _ = self._imports_of("native_metric_calculators") - assert {"numpy", "scipy"} <= external - external, internal, _ = self._imports_of("native_evaluator") - assert "evaluation_frame" in internal, "the internal-import branch sees nothing" - - def test_no_module_in_the_package_imports_outside_its_allowlist(self): - """Package-wide rglob, AST-parsed, static AND dynamic: `import pandas` in - `metrics.py` (the file 2.0.0 purged of it) passed the per-module guard because - that guard listed five of the package's modules; then the first version of THIS - guard, a three-name denylist over `Import` nodes only, let through - `from importlib import import_module as _load; _load("pandas")` in any file, - `importlib.import_module("pandas")` in `metric_frame.py`, `__import__("pandas")` - in `profiles/base.py`, and `import xarray` in `metric_frame.py` (guard audit, - 2026-09-18). So: every non-stdlib, non-package root imported anywhere must be on - the module's allowlist, and every string literal that names such a root as the - argument of any call — whatever the callee is called — counts as an import.""" - import ast + def test_the_parametrisation_lists_every_package_file(self): + """Vacuity: a new module must appear above or it is guarded by nothing.""" + actual = sorted(p.relative_to(self._package_root()).as_posix() for p in self._package_root().rglob("*.py")) + listed = sorted(self.test_every_import_in_the_package_is_on_its_files_allowlist.pytestmark[0].args[1]) + assert actual == listed, f"package files {actual} vs parametrised {listed}" + + def test_the_allowlist_matches_the_census_exactly(self): + """Positive control and dead-allowance check in one: the set of files that import + anything external, as the walker sees them, must equal the allowlist's keys, and + each file's measured roots must equal its allowance — so the walker is proven to + see numpy (an `import`) and scipy and views_frames (`import from`), and an + allowance nothing uses cannot linger.""" import sys - from pathlib import Path - import views_evaluation - pkg = Path(views_evaluation.__file__).parent stdlib = sys.stdlib_module_names - hits = {} - for path in sorted(pkg.rglob("*.py")): - allowed = self._PACKAGE_WIDE_ALLOWED.get(path.stem, set()) - roots = set() - for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))): - if isinstance(node, ast.Import): - roots |= {a.name.split(".")[0] for a in node.names} - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - roots.add(node.module.split(".")[0]) - elif isinstance(node, ast.Call): - # Any call whose first argument is a dotted-name string literal that - # resolves to a real, importable, non-stdlib distribution. - for arg in node.args[:1]: - if isinstance(arg, ast.Constant) and isinstance(arg.value, str): - root = arg.value.split(".")[0] - if root.isidentifier() and root not in stdlib and root != "views_evaluation": - import importlib.util - if importlib.util.find_spec(root) is not None: - roots.add(root) - offending = sorted(r for r in roots if r not in stdlib and r != "views_evaluation" and r not in allowed) - if offending: - hits[str(path.relative_to(pkg))] = offending - assert not hits, f"imports outside the module allowlist: {hits}" - - def test_the_package_wide_allowlist_covers_every_module_that_imports_anything(self): - """Vacuity check: the allowlist must name the modules that DO import numpy, or a - renamed module would silently get the empty allowance and fail — good — while a - module renamed INTO the dict would inherit an allowance it should not have.""" - import views_evaluation - from pathlib import Path - stems = {p.stem for p in Path(views_evaluation.__file__).parent.rglob("*.py")} - assert set(self._PACKAGE_WIDE_ALLOWED) <= stems, set(self._PACKAGE_WIDE_ALLOWED) - stems + measured = {} + for path in self._package_root().rglob("*.py"): + rel = path.relative_to(self._package_root()).as_posix() + external, _, _ = self._imports_of(rel) + ext = {r for r in external if r not in stdlib} + if ext: + measured[rel] = ext + assert measured == self._ALLOWED_BY_PATH, (measured, self._ALLOWED_BY_PATH) + assert measured["evaluation/native_metric_calculators.py"] == {"numpy", "scipy"} + _, internal, _ = self._imports_of("evaluation/native_evaluator.py") + assert "evaluation_frame" in internal, "the internal-import branch sees nothing" + + def test_the_machinery_branch_sees_every_spelling_it_claims_to(self): + """Positive control for the branch no package file exercises today: an aliased + `import_module`, a keyword `name=`, a dotted `importlib.import_module`, a bare + `__import__`, a non-literal argument, and `exec`.""" + external, _, dynamic = self._imports_in( + "import importlib\n" + "from importlib import import_module as _load\n" + "def f(n):\n" + " _load('pandas'); __import__(name='polars'); importlib.import_module('sklearn.metrics')\n" + " import_module(n); exec('import modin')\n" + ) + assert external - {"importlib"} == {"pandas", "polars", "sklearn"}, external + assert {"exec", "import_module()"} <= dynamic, dynamic def test_the_allowlist_is_adr_011s(self): - """The declaration is the guard; an edit here must be a visible two-place change.""" + """The declaration is the guard; an edit here must be a visible change.""" assert self._ALLOWED_EXTERNAL == {"numpy", "scipy"}, "ADR-011: numpy and scipy only" def test_runtime_dependencies_are_exactly_the_declared_set(self): - """The declaration is the guard: the runtime set is python, numpy, scipy and the - optional views-frames — nothing else, by name. scikit-learn (C-05) and pandas - (C-40) live in the dev group, where the parity suite and the purity guard need - them present. Re-adding either to runtime, as a hard dependency or as an extra, - is a red build here rather than a post-push CI probe.""" + """The declaration is the guard: the hard runtime set is numpy and scipy, the + declared set adds the optional views-frames — nothing else, by name, under either + pyproject layout (a hard views-frames in a PEP 621 `[project] dependencies` list + survived a check that only looked at the poetry table; guard audit, 2026-09-18). scikit-learn + (C-05) and pandas (C-40) live in the dev group, where the parity suite and the + purity guard need them present. Re-adding either to runtime, as a hard dependency + or inside an extra, is a red build here rather than a post-push CI probe.""" + import re import tomllib from pathlib import Path - data = tomllib.loads((Path(__file__).resolve().parents[1] / "pyproject.toml").read_text(encoding="utf-8")) - runtime = data["tool"]["poetry"]["dependencies"] - dev = data["tool"]["poetry"]["group"]["dev"]["dependencies"] - assert set(runtime) == {"python", "numpy", "scipy", "views-frames"}, sorted(runtime) - # Contents, not just names, and both TOML layouts: `frames = ["views-frames", - # "pandas"]` and a `[project.optional-dependencies]` table both passed a - # names-only check under one layout (guard audit, 2026-09-18). - assert data["tool"]["poetry"].get("extras") == {"frames": ["views-frames"]} - assert "optional-dependencies" not in data.get("project", {}), ( - "extras are declared under [tool.poetry.extras] in this repo; a PEP 621 table " - "would be read first by poetry >= 2 and bypass this guard" - ) + from tests.test_falsification_extras_actually_installed import _extras_table + text = (Path(__file__).resolve().parents[1] / "pyproject.toml").read_text(encoding="utf-8") + data = tomllib.loads(text) + poetry = data["tool"]["poetry"] + + def dist_name(spec): # "views-frames[frames]>=1.10,<2 ; python>='3.11'" -> "views-frames" + return re.split(r"[\s\[<>=!~;@]", spec.strip(), 1)[0].lower() + + # Hard runtime dependencies, under either layout. Poetry-table entries are hard + # unless `optional = true`; a PEP 621 `[project] dependencies` entry is always + # hard (an optional one belongs under optional-dependencies). views-frames must + # be optional wherever it is declared. + poetry_deps = {k: v for k, v in poetry.get("dependencies", {}).items() if k != "python"} + hard = {k for k, v in poetry_deps.items() if not (isinstance(v, dict) and v.get("optional") is True)} + hard |= {dist_name(d) for d in data.get("project", {}).get("dependencies", [])} + assert hard == {"numpy", "scipy"}, f"hard runtime dependencies must be numpy and scipy only; got {sorted(hard)}" + declared = set(poetry_deps) | {dist_name(d) for d in data.get("project", {}).get("dependencies", [])} + assert declared == {"numpy", "scipy", "views-frames"}, sorted(declared) + # Extras: names AND members, version specifiers stripped, from whichever table + # is authoritative (the shared parser reads PEP 621 first, as poetry does). + extras = {m.group(1): sorted(dist_name(d) for d in re.findall(r"""["']([^"']+)["']""", m.group(2))) + for m in re.finditer(r'^(\w[\w-]*)\s*=\s*\[([^\]]*)\]', _extras_table(text), re.M)} + assert extras == {"frames": ["views-frames"]}, extras + dev = poetry["group"]["dev"]["dependencies"] for dist, why in (("scikit-learn", "the dev-only parity oracle (C-05)"), ("pandas", "what the purity guard proves is not loaded (C-40)")): assert dist in dev, f"{dist} must stay in the dev group as {why}" def test_importing_and_exercising_the_package_loads_neither_pandas_nor_sklearn(self): """Import-time AND call-time: a kernel that imported pandas only when called would - pass an import-only probe.""" + pass an import-only probe, and so would a `MetricFrame.save()` that did — so + every implemented kernel in the catalog, one emit per cell, and one save/load + are exercised, and the script asserts nothing in the catalog was skipped.""" import importlib.util import os import subprocess @@ -2009,12 +2070,28 @@ def test_importing_and_exercising_the_package_loads_neither_pandas_nor_sklearn(s import sys, numpy as np import views_evaluation, views_evaluation.evaluation.metric_frame from views_evaluation import EvaluationFrame, NativeEvaluator -from views_evaluation.evaluation.native_metric_calculators import calculate_ap_native, calculate_mtd_native -calculate_ap_native(np.array([1., 0., 1.]), np.array([[.9], [.1], [.8]])) -calculate_mtd_native(np.array([1., 2., 3.]), np.array([[1.5], [2.5], [2.5]]), power=1.5) -ids = {'time': np.array([1, 1]), 'unit': np.array([1, 2]), 'origin': np.array([0, 0]), 'step': np.array([1, 1])} -ef = EvaluationFrame(np.array([0., 1.]), np.array([[.1], [.9]]), ids, metadata={'target': 't'}) -NativeEvaluator({'steps': [1], 'regression_targets': ['t'], 'regression_point_metrics': ['MSE', 'MTD']}).evaluate(ef).to_metric_frame() +from views_evaluation.evaluation.metric_catalog import METRIC_CATALOG, METRIC_MEMBERSHIP +# EVERY implemented kernel, through the evaluator, in all four cells: a lazy import the +# static walker cannot see (a getattr'd import_module, a module loaded by spec) is +# caught only on a path this probe walks, so it walks them all (guard audit, 2026-09-18). +rs = np.random.RandomState(0) +n, S = 8, 5 +ids = {'time': np.repeat([1, 2], 4), 'unit': np.tile([1, 2, 3, 4], 2), 'origin': np.zeros(n, int), 'step': np.repeat([1, 2], 4)} +ran = set() +for (task, ptype), names in METRIC_MEMBERSHIP.items(): + names = sorted(m for m in names if METRIC_CATALOG[m].implemented) + y = rs.randint(0, 2, n).astype(float) if task == 'classification' else rs.gamma(2.0, 1.0, n) + 0.1 + cols = S if ptype == 'sample' else 1 + pred = rs.uniform(0.05, 0.95, (n, cols)) if task == 'classification' else rs.gamma(2.0, 1.0, (n, cols)) + 0.1 + cfg = {'steps': [1, 2], f'{task}_targets': ['t'], f'{task}_{ptype}_metrics': names} + mf = NativeEvaluator(cfg).evaluate(EvaluationFrame(y, pred, ids, metadata={'target': 't'})).to_metric_frame() + ran |= set(names) +missing = sorted({m for m, spec in METRIC_CATALOG.items() if spec.implemented} - ran) +assert not missing, f'kernels the sweep never ran: {missing}' +import tempfile, pathlib +from views_evaluation.evaluation.metric_frame import MetricFrame +with tempfile.TemporaryDirectory() as d: + mf.save(pathlib.Path(d) / 'mf'); MetricFrame.load(pathlib.Path(d) / 'mf') print(sorted(m for m in ('pandas', 'sklearn') if m in sys.modules)) """ out = subprocess.run([sys.executable, "-c", script], capture_output=True, text=True, cwd=repo, env=env) diff --git a/views_evaluation/evaluation/evaluation_report.py b/views_evaluation/evaluation/evaluation_report.py index 3f21dd8..501758b 100644 --- a/views_evaluation/evaluation/evaluation_report.py +++ b/views_evaluation/evaluation/evaluation_report.py @@ -229,7 +229,14 @@ def _emit(eval_type: str, metric: str, group_id: str, value: float) -> None: return MetricFrame(values=values_arr, identifiers=identifiers, metadata=metadata) def to_dict(self) -> Dict[str, Any]: - """Converts the entire report into a nested dictionary.""" + """The report as a nested dictionary. + + ``schemas`` is the report's LIVE internal structure — the same object every call + returns, the object ``to_metric_frame()`` reads, and (as constructed) the caller's + own ``results`` argument — not a copy. Mutating it changes what is subsequently + emitted. That has been the behaviour since 1.0.0 and is pinned by a test; a copy + would be a behaviour change and is not made here. Treat the result as read-only. + """ return { "target": self.target, "task": self.task,