From 78adb47799154564852c4e84e39c9405d6bf552c Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 17:32:56 -0700 Subject: [PATCH 01/18] chore(studio): studio/dev becomes the integration branch (#60) Studio tasks branch off studio/dev and PR into it; studio/dev merges into main at phase boundaries. main sees Studio in reviewed batches while the model and viz work continues there untouched. CI runs on pushes to studio/dev as well as main, so a merge is verified and not only the PR that preceded it -- a PR is tested against its own head, and a stale one can go green and still break the branch it lands on. Amends the "trunk-based" line in studio/CLAUDE.md rather than leaving the documented workflow disagreeing with the actual one, and records the two things that cost time while landing #59: gh pr create defaults to the repository default branch, so --base studio/dev must be explicit; and a conflicted PR is not merely blocked but silently untested, because GitHub cannot build the merge ref and so runs no workflow at all. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/studio-ci.yml | 6 +++++- docs/studio/PROGRESS.md | 22 ++++++++++++++++++++ studio/CLAUDE.md | 36 +++++++++++++++++++++++++++------ 3 files changed, 57 insertions(+), 7 deletions(-) diff --git a/.github/workflows/studio-ci.yml b/.github/workflows/studio-ci.yml index 41eb718..ea10877 100644 --- a/.github/workflows/studio-ci.yml +++ b/.github/workflows/studio-ci.yml @@ -13,7 +13,11 @@ name: studio-ci on: push: - branches: ["main"] + # `studio/dev` is the integration branch every Studio task PRs into; main sees Studio in + # reviewed batches. Running on pushes to both means a merge into either is verified, not just + # the PR that preceded it -- a PR is tested against its own head, and a stale one can go green + # and still break the branch it lands on. + branches: ["main", "studio/dev"] paths: - "studio/**" - "docs/studio/**" diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 33fa62e..730db11 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -28,6 +28,28 @@ Nothing is built on top of `studio/schema` until 0.2 is reviewed and merged. --- +### 2026-08-13 — Branching: `studio/dev` becomes the integration branch + +Not a task; a workflow decision taken after task 0.1 merged (#59). + +Studio tasks now branch off **`studio/dev`** and PR into it; `studio/dev` merges into `main` at +phase boundaries. `main` therefore sees Studio in reviewed batches rather than one task at a time, +while the model and viz work continues on `main` untouched. `studio-ci.yml` runs on pushes to both +branches, so a merge is verified and not just the PR that preceded it. + +This amends the "trunk-based" line in `studio/CLAUDE.md` rather than leaving the documented workflow +disagreeing with the actual one. Task 0.1 pre-dates the change and went straight into `main`. + +Two operational notes, both learned the hard way while landing #59: + +- `gh pr create` defaults to the repository's default branch. Studio PRs must pass + `--base studio/dev` explicitly. +- **Merge `main` into `studio/dev` regularly.** A conflicted PR is not merely blocked, it is + silently *untested*: GitHub cannot build the merge ref, so no workflow runs at all and the PR + shows no checks rather than a failure. + +--- + ### 2026-08-13 — Task 0.1 (cont.): toolchain, CI and the lockfile Closes 0.1. Still no runnable app code — the point of this half is that the next task's code has diff --git a/studio/CLAUDE.md b/studio/CLAUDE.md index 3471074..fc29c14 100644 --- a/studio/CLAUDE.md +++ b/studio/CLAUDE.md @@ -100,17 +100,41 @@ just documented in [`CAVEATS.md`](../docs/studio/CAVEATS.md): ## Git workflow -- Trunk-based, short-lived branches: `feat/-`, `fix/…`, `docs/…`, `chore/…`. -- Every branch corresponds to an issue. Labels: `phase-0`…`phase-n`, `science`, `blocking`, - `architecture`, `frontend`, `backend`, `data`, `testing`, `docs`; a milestone per phase. +**`studio/dev` is the integration branch. Studio PRs target it, never `main`.** Branch off +`studio/dev`, and pass the base explicitly — `gh pr create --base studio/dev` — because `gh` +defaults to the repository's default branch, which is `main`. + +``` +main ────●────────────●──────────● viz + model work, plus batched Studio merges + \ \ / +studio/dev ●──●──●────────●──●──● integration; CI runs on pushes here + \ \ \ + task 0.2 ─● \ \ one task → one branch → one PR → studio/dev + task 0.3 ────● \ + task 0.4 ────────────────────● +``` + +- `studio/dev` → `main` at phase boundaries, or sooner when something there is needed by the model + side. **Merge `main` into `studio/dev` regularly** — the model and viz work moves independently, + and a long-lived branch that never pulls is how you get a conflicted merge nobody wants to do. + A conflicted PR is also silently untested: GitHub cannot build the merge ref, so no workflow runs + at all. +- Task 0.1 pre-dates this and went straight into `main` (#59). Everything from 0.2 on goes through + `studio/dev`. +- Short-lived task branches: `feat/-`, `fix/…`, `docs/…`, `chore/…`. +- Every branch corresponds to an issue. Labels: `studio` plus `science`, `blocking`, `architecture`, + `frontend`, `backend`, `data`, `testing`, `docs`. **Do not use the `phase-N` labels** — those are + the coupled model's phases (issues #11–#28), not Studio's; name the Studio phase in the text. - Conventional commits. Small, coherent commits; no "wip" on shared branches. - **One task, one PR.** No scope creep. -- **Model-side changes to `coupled/` go in their own PR**, with their own tests — never buried inside - an app feature. -- Annotated tag at each phase completion (`v0.1.0-phase0`) with release notes. +- **Model-side changes to `coupled/` go in their own PR against `main`**, with their own tests — + never buried inside an app feature, and never routed through `studio/dev`. +- Annotated tag at each phase completion (`v0.1.0-phase0`) with release notes, cut from `main` after + the phase's `studio/dev` → `main` merge. ## PR checklist +- [ ] Base branch is `studio/dev` (not `main`) - [ ] Linked issue; scope matches - [ ] New/changed schema fields carry unit, range, description, default, provenance - [ ] Any new `[ASSUMPTION]` added to `docs/studio/ASSUMPTIONS.md` From 90419e12e6d46dedd4ed19f24ffabd155ef0b033 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 18:00:48 -0700 Subject: [PATCH 02/18] =?UTF-8?q?studio:=20schema=20v0=20=E2=80=94=20SciFi?= =?UTF-8?q?eld,=20RunConfig,=20RunSet=20and=20the=20config=20hash=20(#62)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit studio: schema v0 -- SciField, RunConfig, RunSet and the config hash Task 0.2 (#61), the review gate. studio/schema/ in seven modules: a closed canonical unit registry, the SciField metadata carrier, the enums, RunConfig and its ten groups, RunSet with GRID/ZIP/LIST axes, canonical JSON + SHA-256, and the JSON Schema export. 41 leaf fields, 61 Tier-A tests. Provenance is required, and its rules are enforced when the module is imported rather than when a run produces a quietly wrong number: MODEL_DEFAULT and PAPER_ENSEMBLE must cite a source, LITERATURE must cite a reference, DERIVED must declare its inputs and must not ship a value. A default whose origin nobody recorded cannot be declared at all. Defaults are the paper ensemble's rather than the model's where the two differ (ASSUMPTION-5) -- ion_pair_rate is the visible case, 30 cm^-3 s^-1 against the model's 0.0, which would disable ion-induced nucleation entirely. Both are recorded on the field. RunConfig() with no arguments is therefore the golden case, which is the form's opening state and the base of every RunSet. Identity carries only what changes the result: no label, notes or output_dir field, so two runs differing in name are one computation. The hash is pinned by a test and checked across four PYTHONHASHSEEDs in fresh interpreters, because a hash that drifts silently invalidates every cache and fixture at once and reads as a performance problem. RunSet reproduces the 810-run ensemble -- same count, same order, same case IDs, checked at the golden case. That is why LIST exists: the site axis covaries latitude, T, p and H2O, and its cross product is not physically meaningful. Model validation is mirrored where it is cheap (DT/dt_couple divisibility, the 40/80/160 bin grids, background_evolves accepting only false), each mirror citing the model line it copies. ADR-002's third motivating problem was that a form cannot learn what is valid without importing most of the model; this answers it. No physics: plume_volume_cm3 and so2_initial_pptv are declared with derived_from and left unresolved for tasks 0.3 and 0.5. Two divergences from the plan are recorded in PROGRESS.md rather than absorbed: max_sim_time is optional, and DilutionRegime.CONSTANT is spelled "constant" where the model spells it "". Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/GLOSSARY.md | 14 +- docs/studio/PROGRESS.md | 64 +- docs/studio/plan/PHASE_0.md | 8 +- pyproject.toml | 2 +- studio/schema/__init__.py | 108 +++- studio/schema/config.py | 741 ++++++++++++++++++++++ studio/schema/enums.py | 104 +++ studio/schema/export.py | 76 +++ studio/schema/fields.py | 169 +++++ studio/schema/hashing.py | 99 +++ studio/schema/runset.py | 283 +++++++++ studio/schema/units.py | 80 +++ studio/tests/unit/test_config_hash.py | 157 +++++ studio/tests/unit/test_runset.py | 344 ++++++++++ studio/tests/unit/test_schema_export.py | 150 +++++ studio/tests/unit/test_schema_metadata.py | 263 ++++++++ 16 files changed, 2646 insertions(+), 16 deletions(-) create mode 100644 studio/schema/config.py create mode 100644 studio/schema/enums.py create mode 100644 studio/schema/export.py create mode 100644 studio/schema/fields.py create mode 100644 studio/schema/hashing.py create mode 100644 studio/schema/runset.py create mode 100644 studio/schema/units.py create mode 100644 studio/tests/unit/test_config_hash.py create mode 100644 studio/tests/unit/test_runset.py create mode 100644 studio/tests/unit/test_schema_export.py create mode 100644 studio/tests/unit/test_schema_metadata.py diff --git a/docs/studio/GLOSSARY.md b/docs/studio/GLOSSARY.md index 1e1311a..d00dcd4 100644 --- a/docs/studio/GLOSSARY.md +++ b/docs/studio/GLOSSARY.md @@ -15,9 +15,17 @@ uses, plus the repository-specific names that are otherwise unguessable. **RunSet** — the primary user-facing object: a base `RunConfig` plus zero or more **axes**. A single run is a RunSet with zero axes, so there is no separate code path for N = 1. -**Axis** — a schema path marked as varying, either `{path, values: [...]}` or -`{path, range: {start, stop, n, spacing}}`. Expanded by **GRID** (Cartesian product), **ZIP** -(paired), or **LIST** (explicit configs). +**Axis** — one dimension of a sweep: a name, a kind, and **points**. Each point is a short `label` +plus the field `assignments` it stands for. Kinds: **GRID** (crossed with the other GRID/LIST axes), +**ZIP** (advanced in lockstep with the other ZIP axes, the group then crossed with the rest), and +**LIST** (crossed, but each point sets *several* fields at once — a covarying group, e.g. the paper +ensemble's site axis, where latitude, T, p and H₂O move together). Expansion order is +`itertools.product`: the last axis varies fastest, which is what reproduces the existing ensemble's +case order. Assignments name **leaf** paths only; a whole group has no unit, provenance or DAG node. + +**Axis point label** — the short token that becomes part of the run label, e.g. `sabr220`, `a1p0`. +Joined by `__` across axes to give the **case ID**, which is how the existing ensemble names its +directories. **Derived field** — a value computed from other fields, declared via `derived_from` metadata. Each carries a state: **auto** (recomputed silently when an upstream field changes) or **user_override** diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 730db11..c882616 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -15,7 +15,7 @@ with full provenance, and the golden tests pass. | Task | Status | |---|---| | 0.1 Repo, CI, docs skeleton | **done** | -| 0.2 `studio/schema` v0 — **review gate** | not started | +| 0.2 `studio/schema` v0 — **review gate** | **awaiting review** (#61) | | 0.3 Dependency-graph engine + override semantics | not started | | 0.4 `studio/modelio` seam + `RunSummary` | not started | | 0.5 `studio/science` derivations | not started | @@ -28,6 +28,68 @@ Nothing is built on top of `studio/schema` until 0.2 is reviewed and merged. --- +### 2026-08-13 — Task 0.2: `studio/schema` v0 · **awaiting review** (issue #61) + +The review gate. Nothing is built on top of this until it is reviewed and merged. + +**Added** — `studio/schema/`: `units.py` (closed canonical registry), `fields.py` (`SciField`, +`Provenance`), `enums.py`, `config.py` (`RunConfig` and its ten groups), `runset.py` (`RunSet`, +`Axis`, expansion), `hashing.py` (canonical JSON + SHA-256), `export.py` (JSON Schema + flat field +catalogue). 41 leaf fields, every one carrying unit, description, range and provenance. Four test +modules, 61 tests, all Tier A. + +**The load-bearing decisions** + +- **Provenance is required and its rules are enforced at import time.** `MODEL_DEFAULT` and + `PAPER_ENSEMBLE` must give a `source`; `LITERATURE` must give a `cite`; `DERIVED` must give + `derived_from` and must *not* give a value. A field whose default has no recorded origin cannot be + declared — which is the one failure mode `studio/CLAUDE.md` is most emphatic about, made + structural rather than aspirational. +- **Defaults are the paper ensemble's, not the model's**, where they differ (ASSUMPTION-5). The + visible case is `ion_pair_rate`: the model defaults to 0.0, which disables ion-induced nucleation + entirely, while the ensemble uses 30 cm⁻³ s⁻¹. Both are recorded, with the divergence stated on + the field. +- **`RunConfig()` with no arguments is the golden case.** That is not a convenience: it is the + form's opening state and the base of every RunSet. +- **Identity contains only what changes the result.** No `label`, `notes` or `output_dir` field — + two runs differing only in a name are the same computation. Labels live on `ExpandedRun`. +- **The hash is pinned by a test**, not merely asserted self-consistent, and checked across four + `PYTHONHASHSEED`s in fresh interpreters. Canonical form: sorted keys, no padding, `allow_nan=False` + (NaN raises rather than emitting a token no other parser reads back). +- **`RunSet` reproduces the 810-run ensemble** — same count, same order, same case IDs, verified + against the golden case at index 121. This is the strongest available evidence that the axis model + is faithful to what this project actually does, and it is why `LIST` exists: the site axis covaries + latitude, T, p and H₂O, and its cross product is not physically meaningful. +- **Model validation is mirrored where it is cheap** — the `DT`/`dt_couple` divisibility rule, the + 40/80/160 bin grids, `background_evolves` accepting only `false` (SCIENCE-5). Each mirror cites the + model line it copies. ADR-002's third motivating problem was that a form cannot learn what is valid + without importing most of the model; this is the answer to it. + +**Divergences from the plan, stated rather than absorbed** + +- `max_sim_time` is **optional**, not required. Task 0.6 says both limits are required; but simulated + time is already bounded by `schedule.duration_days`, so a required second copy would be redundant, + and inventing a default ceiling would be a fabricated number. It is an optional *lower* ceiling. + `max_wall_time_s` is required and defaults to 3600 s (ASSUMPTION-4). +- `DilutionRegime.CONSTANT` is spelled `"constant"`, while the model spells it `""`. An empty string + is not a usable dropdown key. This is the **only** enum value that is not the model's own string, + it is flagged at the point of deviation, and task 0.4's equivalence test must cover it explicitly. + +**Deliberately not done** — no physics: `plume_volume_cm3` and `so2_initial_pptv` are declared with +`derived_from` and left unresolved (0.3 resolves, 0.5 derives). No species-name validation: the +species list belongs to the model, so `studio/modelio` validates at the seam. No preset library: the +paper ensemble's axes live in the test, and earn a home in `studio/` when 0.4 or 0.7 needs them. + +**Found while writing it:** `resolve_path` initially accepted a path naming a whole group. The test +caught it. Groups are now rejected — they have no unit, no provenance and no node in the dependency +graph 0.3 builds from leaf paths. + +**One interpreter-level assumption**, recorded in `hashing.py`: float formatting via `repr` has been +the shortest round-tripping decimal since Python 3.1, so the canonical form is stable across the +versions this project supports. The pinned-hash test is what would catch that changing. + +--- + ### 2026-08-13 — Branching: `studio/dev` becomes the integration branch Not a task; a workflow decision taken after task 0.1 merged (#59). diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index f85d69e..7341c94 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -27,7 +27,7 @@ Delivered as above, with two things worth stating plainly rather than leaving im --- -## 0.2 — `studio/schema` v0 · **review gate** +## 0.2 — `studio/schema` v0 · **review gate** · *implemented, awaiting review* (issue #61) - `SciField(unit=…, range=…, provenance=…, cite=…, derived_from=[…])` over `Field(json_schema_extra=…)`. @@ -45,6 +45,12 @@ Delivered as above, with two things worth stating plainly rather than leaving im **Request review explicitly before building on it.** +Delivered as specified, with two divergences recorded in `PROGRESS.md` rather than absorbed +silently: `max_sim_time` is optional (simulated time is already bounded by `schedule.duration_days`, +and a required second bound would need an invented default), and `DilutionRegime.CONSTANT` is +spelled `"constant"` where the model spells it `""` — the only enum value that is not the model's +own string, and one the 0.4 equivalence test must cover explicitly. + --- ## 0.3 — Dependency-graph engine and override semantics diff --git a/pyproject.toml b/pyproject.toml index cb697dd..db272af 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -80,7 +80,7 @@ markers = [ [tool.ruff] line-length = 100 target-version = "py311" -src = ["studio"] +src = ["."] extend-exclude = ["studio/web"] [tool.ruff.lint] diff --git a/studio/schema/__init__.py b/studio/schema/__init__.py index 6b05083..fd4aeaa 100644 --- a/studio/schema/__init__.py +++ b/studio/schema/__init__.py @@ -5,23 +5,111 @@ Defined once here in Pydantic v2, exported as JSON Schema, consumed by the web client for form generation and validation. Nothing in the UI may invent a field that does not exist here. -Contents (task 0.2, not yet implemented): +Contents: -* ``SciField`` -- the field-metadata carrier: canonical unit, valid range or enum, label, - description, default, PROVENANCE of that default, citation, and ``derived_from``. -* ``RunConfig`` -- one validated simulation description; versioned, canonically serialisable, - hashable. -* ``RunSet`` + axes (GRID / ZIP / LIST) -- the primary user-facing object. A single run is a RunSet - with zero axes; there is no separate N=1 code path. -* Canonical JSON serialisation and stable SHA-256 hashing (ADR-006). +* ``SciField`` (``fields.py``) -- the field-metadata carrier: canonical unit, valid range, label, + description, default, PROVENANCE of that default, citation, ``derived_from``, and any caveat that + must travel with the value. Inconsistent metadata raises at import time. +* ``RunConfig`` (``config.py``) -- one validated simulation description; versioned, canonically + serialisable, hashable. Minimum viable for the Phase-0 slice: what the golden case + ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` needs, plus the provenance and derivation inputs the + model has no field for. +* ``RunSet`` + axes (``runset.py``) -- the primary user-facing object. A single run is a RunSet with + zero axes; there is no separate N = 1 code path. +* Canonical JSON and stable SHA-256 (``hashing.py``) -- run identity and cache key (ADR-006). +* JSON Schema export and the flat field catalogue (``export.py``). -Canonical units are the MODEL's native units (mbar, ppm, pptv, K, s, um^2/cm^3), not SI -- see +Canonical units are the MODEL's native units (mbar, ppmv, pptv, K, s, um^2/cm^3), not SI -- see ADR-003 and ASSUMPTION-1. ``pint`` is used for display conversion at the presentation boundary and for property tests, never inside the model interface. +**No physics happens here.** Derived fields declare what they are computed from and stay unset; the +dependency-graph engine is task 0.3 and the derivations are task 0.5. + This package must not import ``coupled``, the API, or the database (see ``studio/__init__.py``). """ from __future__ import annotations -__all__: list[str] = [] +from studio.schema.config import ( + PAPER_BACKGROUND_GAS_PPTV, + SCHEMA_VERSION, + Background, + Chemistry, + Dilution, + Injection, + Microphysics, + Numerics, + ProcessSwitches, + RunConfig, + Schedule, + SchemaModel, + Site, + Termination, +) +from studio.schema.enums import AxisKind, BackgroundAerosol, DilutionRegime, PhotolysisMode +from studio.schema.export import ( + SCHEMA_ID, + field_catalogue, + iter_leaf_fields, + run_config_json_schema, +) +from studio.schema.fields import EXTENSION_KEY, Provenance, SciField, field_metadata +from studio.schema.hashing import ( + CANONICAL_FORM_VERSION, + canonical_json, + canonical_payload, + config_hash, + short_hash, +) +from studio.schema.runset import ( + Axis, + AxisPoint, + ExpandedRun, + RunSet, + apply_assignments, + resolve_path, +) +from studio.schema.units import PINT_EXPRESSION, Unit + +__all__ = [ + "CANONICAL_FORM_VERSION", + "EXTENSION_KEY", + "PAPER_BACKGROUND_GAS_PPTV", + "PINT_EXPRESSION", + "SCHEMA_ID", + "SCHEMA_VERSION", + "Axis", + "AxisKind", + "AxisPoint", + "Background", + "BackgroundAerosol", + "Chemistry", + "Dilution", + "DilutionRegime", + "ExpandedRun", + "Injection", + "Microphysics", + "Numerics", + "PhotolysisMode", + "ProcessSwitches", + "Provenance", + "RunConfig", + "RunSet", + "Schedule", + "SchemaModel", + "SciField", + "Site", + "Termination", + "Unit", + "apply_assignments", + "canonical_json", + "canonical_payload", + "config_hash", + "field_catalogue", + "field_metadata", + "iter_leaf_fields", + "resolve_path", + "run_config_json_schema", + "short_hash", +] diff --git a/studio/schema/config.py b/studio/schema/config.py new file mode 100644 index 0000000..957f0ce --- /dev/null +++ b/studio/schema/config.py @@ -0,0 +1,741 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunConfig`` -- one validated simulation description (ADR-002). + +Deliberately MINIMUM VIABLE: it covers exactly what the Phase-0 slice needs, which is everything +``run_ensemble.build_scenario()`` sets for the golden case +``30N_20km__sabr220__D2med__a1p0__nuc1__cg1``, plus the provenance and derivation inputs the model +has no field for. Stages 4-7 of the spec are NOT modelled here; doing that before anything runs is +how a config layer ends up describing a model that does not exist. + +Defaults are the paper ensemble's configuration, not the model's, wherever the two differ -- Phase +0's job is to reproduce existing trusted runs (ASSUMPTION-5). Each such field records which it is, +so "why is this 30 and not 0?" has an answer in the schema rather than in someone's memory. + +Three things this module deliberately does NOT do: + +* **No physics.** Fields marked ``DERIVED`` declare what they are computed from and stay unset. The + dependency-graph engine is task 0.3 and the derivations are task 0.5; a plausible number computed + here would be exactly the failure mode ``studio/CLAUDE.md`` forbids. +* **No species-name validation.** Whether ``"HCl"`` is a species is a question for the model's own + ``IDX``, and ``studio.schema`` may not import the model. ``studio/modelio`` validates names at the + seam, where the answer actually lives. +* **No unit conversion.** Values are in canonical (= model-native) units already (ADR-003). + +Validation that the model performs in ``CoupledScenario.__post_init__`` is MIRRORED here where it is +cheap to do so -- the DT/dt_couple divisibility rule especially -- because ADR-002's third +motivating problem is that a form cannot today learn what is valid without importing most of the +model. +Mirrored rules cite the model line they mirror; if the model's rule changes, the citation is how you +find this one. +""" + +from __future__ import annotations + +from typing import Annotated, Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from studio.schema.enums import BackgroundAerosol, DilutionRegime, PhotolysisMode +from studio.schema.fields import Provenance, SciField +from studio.schema.units import Unit + +#: The schema's own version. Bumped on any change to field names, semantics or defaults, because +#: those change the config hash and therefore run identity (ADR-006). Old configs are never silently +#: reinterpreted under new semantics. +SCHEMA_VERSION = "0.1.0" + +#: Stratospheric background gas composition [pptv] used by the 810-run ensemble +#: (``run_ensemble.py:56-57``). Module-level so the default is one object with one source, and so a +#: test can compare against it without reaching into a field default. +PAPER_BACKGROUND_GAS_PPTV: dict[str, float] = { + "O2": 2.1e11, + "O3": 1.18e6, + "OH": 0.5, + "HO2": 3.0, + "NO": 450.0, + "NO2": 450.0, + "HCl": 777.0, + "ClONO2": 127.0, + "HNO3": 5000.0, +} + + +class SchemaModel(BaseModel): + """Base for every schema model: frozen, and unknown keys are an error. + + ``frozen`` because a submitted config is immutable (ADR-004) -- an edit produces a new config + and a new run, which is what makes ``config_hash`` a meaningful identity. ``extra="forbid"`` + because a typo'd key that is silently accepted is a config that does not describe the run. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + +class Site(SchemaModel): + """Where the box is, and the thermodynamic state it sits in. + + Phase 0 takes T and p as user input. Phase 1 derives them from climatology at a chosen + (lat, lon, altitude-or-tropopause-relative) point -- which is why they are primary fields now + and become ``derived_from`` targets later, not the other way round. + """ + + latitude_deg: float = SciField( + default=30.0, + unit=Unit.DEGREE, + ge=-90.0, + le=90.0, + label="Latitude", + description="Box latitude; drives the solar zenith angle and therefore photolysis.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:62 (LAT_ALT '30N_20km')", + examples=[30.0, 60.0], + ) + longitude_deg: float = SciField( + default=0.0, + unit=Unit.DEGREE, + ge=-180.0, + le=180.0, + label="Longitude", + description=( + "Box longitude. Only affects the solar zenith angle via local solar time; at 0 deg, " + "UTC and local solar time coincide, which is why the ensemble's 00:00 local release " + "is expressed as start_utc_hour = 0." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:98", + ) + temperature_k: float = SciField( + default=210.0, + unit=Unit.KELVIN, + gt=0.0, + label="Temperature", + description=( + "Box temperature. Isobaric and isothermal unless the heating switch is on; see the " + "caveat on switches.heating_to_t and SCIENCE-4 (issue #56)." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Site: 210 K, 55 hPa)", + examples=[210.0, 213.0], + ) + pressure_mbar: float = SciField( + default=55.0, + unit=Unit.MBAR, + gt=0.0, + label="Pressure", + description=( + "Box pressure, the model's native pressure unit (mbar == hPa). Also sets the box " + "altitude used to place the aerosol in the TUV-x radiation column." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Site: 210 K, 55 hPa)", + examples=[55.0, 120.0], + ) + h2o_ppmv: float = SciField( + default=6.9104, + unit=Unit.PPMV, + ge=0.0, + label="Water vapour", + description=( + "Water vapour mixing ratio. The ensemble value is RH = 3% precomputed at 210 K / " + "55 hPa; it is a decimal in native units on purpose (ADR-003) -- do not round-trip it " + "through SI." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:62 (LAT_ALT WTR column, RH = 3%)", + caveat=( + "Reanalysis stratospheric water vapour is biased dry, so ERA5 is not the recommended " + "source for this field when Phase 1 lands (BLOCKING-5)." + ), + ) + + +class Schedule(SchemaModel): + """When the release happens and how long the box is integrated.""" + + day_of_year: int = SciField( + default=172, + unit=Unit.DIMENSIONLESS, + ge=1, + le=366, + label="Day of year", + description="Day of year of the release; sets the solar declination.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (day 172, ~21 June)", + ) + start_utc_hour: float = SciField( + default=0.0, + unit=Unit.HOUR, + ge=0.0, + lt=24.0, + label="Release hour (UTC)", + description=( + "UTC hour of release. The paper ensemble releases at 00:00 LOCAL SOLAR time and sets " + "this to 0 with longitude 0, where the two coincide. At any other longitude they do " + "not, and the distinction is the user's to make." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:98", + ) + duration_days: int = SciField( + default=10, + unit=Unit.DAY, + ge=1, + label="Duration", + description=( + "Simulated duration. Measured cost: ~3-5 min for 10 days at 80 bins, ~30-40 min for " + "60 days (BLOCKING-4)." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:99", + examples=[10, 60], + ) + + +class Injection(SchemaModel): + """What is released, and into what volume. + + ASSUMPTION-5, and it must be said in the UI too: **V0 does not enter the dynamics**. The model + is intensive and volume-invariant (``coupled/tests/test_boxvol_invariance.py``); the geometry + below exists only to turn an injected mass into an initial concentration. Presenting it as a + plume shape that the physics responds to would be a lie of layout. + + What t = 0 means -- engine exit plane or post-vortex-breakup -- is SCIENCE-2 (issue #54) and is + the most consequential open question in the project, because it moves the initial concentration + by orders of magnitude. + """ + + so2_mass_kg: float = SciField( + default=1000.0, + unit=Unit.KILOGRAM, + gt=0.0, + label="SO2 released", + description=( + "Mass of SO2 released into the initial plume volume. The ensemble uses 1 tonne." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Injection: 1 t)", + ) + plume_length_m: float = SciField( + default=15000.0, + unit=Unit.METRE, + gt=0.0, + label="Plume length", + description="Along-track length of the initial plume volume.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:45 (10 m x 10 m x 15 km)", + caveat="Only sets the initial concentration; the dynamics are volume-invariant.", + examples=[15000.0, 30000.0], + ) + plume_width_m: float = SciField( + default=10.0, + unit=Unit.METRE, + gt=0.0, + label="Plume width", + description="Cross-track width of the initial plume volume.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:45 (10 m x 10 m x 15 km)", + caveat="Only sets the initial concentration; the dynamics are volume-invariant.", + ) + plume_height_m: float = SciField( + default=10.0, + unit=Unit.METRE, + gt=0.0, + label="Plume height", + description="Vertical extent of the initial plume volume.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:45 (10 m x 10 m x 15 km)", + caveat="Only sets the initial concentration; the dynamics are volume-invariant.", + ) + plume_volume_cm3: float | None = SciField( + default=None, + unit=Unit.CM3, + label="Plume volume V0", + description=( + "Initial plume volume. Computed, not entered: the ensemble's 10 m x 10 m x 15 km gives " + "1.5e12 cm^3. Left unresolved by the schema -- task 0.5 owns the one cited " + "implementation, replacing the five copies that exist today with two different values." + ), + provenance=Provenance.DERIVED, + derived_from=[ + "injection.plume_length_m", + "injection.plume_width_m", + "injection.plume_height_m", + ], + ) + so2_initial_pptv: float | None = SciField( + default=None, + unit=Unit.PPTV, + label="Initial SO2", + description=( + "Initial plume SO2 mixing ratio. The ensemble fixes the NUMBER DENSITY " + "(6.27e15 molec cm^-3) and lets the pptv follow from the air density at this site, so " + "this depends on temperature and pressure as well as on mass and volume." + ), + provenance=Provenance.DERIVED, + derived_from=[ + "injection.so2_mass_kg", + "injection.plume_volume_cm3", + "site.temperature_k", + "site.pressure_mbar", + ], + ) + + +class Background(SchemaModel): + """The air the plume is diluted into, and the aerosol it entrains.""" + + aerosol: BackgroundAerosol = SciField( + default=BackgroundAerosol.SABR_220, + unit=Unit.DIMENSIONLESS, + label="Background aerosol", + description=( + "Background aerosol size distribution seeded into the initial TOMAS state and " + "entrained thereafter. The lognormal sets are digitized from source plots, not " + "published parameters." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_dilution_parameters.md (Background: SABRE aged air)", + ) + so2_pptv: float = SciField( + default=20.0, + unit=Unit.PPTV, + ge=0.0, + label="Background SO2", + description=( + "SO2 mixing ratio of the entrained background air. The plume's SO2 relaxes toward this " + "value rather than toward zero." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_dilution_parameters.md (Background: SO2 20 pptv)", + examples=[20.0, 100.0], + ) + gas_pptv: dict[str, float] = SciField( + default_factory=lambda: dict(PAPER_BACKGROUND_GAS_PPTV), + unit=Unit.PPTV, + label="Background gas composition", + description=( + "Initial plume gas composition, which is also the entrained background composition. " + "Species omitted start at zero. Names are validated at the model seam " + "(studio/modelio), not here, because the species list belongs to the model." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:56-57 (_BG_GAS_PPT)", + caveat=( + "Standing decision (Ali, 2026-07-08, run_ensemble.py:49-55): production runs should " + "initialise from a SPUN-UP control run, not this static list, which is retained only " + "to reproduce the existing 810-run ensemble." + ), + ) + + +class Dilution(SchemaModel): + """Plume expansion and entrainment of background air.""" + + regime: DilutionRegime = SciField( + default=DilutionRegime.D2, + unit=Unit.DIMENSIONLESS, + label="Dilution regime", + description=( + "Volume-expansion regime V(t)/V0 (Schumann et al. 1998 form). CONSTANT uses " + "dilution.rate_per_s instead of a curve." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_dilution_parameters.md (Med Kz (D2, default))", + ) + rate_per_s: float = SciField( + default=1.157e-6, + unit=Unit.PER_SECOND, + ge=0.0, + label="Constant dilution rate", + description=( + "First-order relaxation rate toward the background. IGNORED unless regime is CONSTANT; " + "the model ignores it silently, so a UI must grey it out rather than imply it applies." + ), + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:102", + ) + zero_species: tuple[str, ...] = SciField( + default=(), + unit=Unit.DIMENSIONLESS, + label="Species zeroed in the background", + description=( + "Gas species set to zero in the entrained background air; all others keep their " + "initial value. Empty means the background is the full initial composition." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:100 (dilution_zero_species=())", + ) + background_overrides_pptv: dict[str, float] = SciField( + default_factory=dict, + unit=Unit.PPTV, + label="Background overrides", + description=( + "Explicit background mixing ratios applied AFTER zero_species (an override wins over a " + "zero). background.so2_pptv is merged in here by studio/modelio, so SO2 need not be " + "repeated; this field is for any OTHER species." + ), + provenance=Provenance.CONVENTION, + ) + background_evolves: Literal[False] = SciField( + default=False, + unit=Unit.DIMENSIONLESS, + label="Background evolves", + description=( + "Whether the entrained background reservoir evolves photochemically. The model is " + "one-box with a STATIC background (driver.py:261-276), so False is the only accepted " + "value and any other fails validation rather than being quietly ignored." + ), + provenance=Provenance.CONVENTION, + caveat="Whether one box with a spun-up IC suffices is SCIENCE-5 (issue #57).", + ) + + +class Microphysics(SchemaModel): + """TOMAS sectional microphysics: resolution and the three sensitivity multipliers.""" + + n_bins: Literal[40, 80, 160] = SciField( + default=80, + unit=Unit.COUNT, + label="Size bins", + description=( + "TOMAS size resolution over a FIXED dry Dp range of 1.7 nm - 17.5 um. The mass " + "ratio is 2**(40/n_bins), so the top boundary is pinned; d_min/d_max/mass_doubling " + "are not selectable, whatever the spec implies." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (TOMAS, 80 size bins)", + ) + condensation_alpha: float = SciField( + default=1.0, + unit=Unit.DIMENSIONLESS, + gt=0.0, + le=1.0, + label="Condensation alpha", + description="Fuchs-Sutugin mass-accommodation coefficient for H2SO4 condensation.", + provenance=Provenance.PAPER_ENSEMBLE, + source=( + "coupled/paper_ensemble/TABLE_microphysics_parameters.md (Varied: alpha 0.5, **1.0**)" + ), + examples=[0.5, 1.0], + ) + nucleation_rate_scale: float = SciField( + default=1.0, + unit=Unit.DIMENSIONLESS, + ge=0.0, + label="Nucleation rate scale", + description=( + "Free multiplier on the Dunne et al. (2016) binary H2SO4-H2O nucleation rate (neutral " + "and ion-induced channels alike). 0 disables nucleation." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source=( + "coupled/paper_ensemble/TABLE_microphysics_parameters.md " + "(Varied: 0.01x, **1x**, 100x)" + ), + examples=[0.01, 1.0, 100.0], + ) + coag_kernel_scale: float = SciField( + default=1.0, + unit=Unit.DIMENSIONLESS, + ge=0.0, + label="Coagulation kernel scale", + description=( + "Free multiplier on the Brownian coagulation kernel (Fuchs non-continuum corrected)." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Varied: 0.5x, **1x**, 2x)", + examples=[0.5, 1.0, 2.0], + ) + ion_pair_rate: float = SciField( + default=30.0, + unit=Unit.PER_CM3_PER_S, + ge=0.0, + label="Ion pair production rate", + description=( + "Ion-pair production rate feeding the Dunne (2016) ion-induced nucleation channels. " + "0 disables the ion-induced channels; the neutral ones are unaffected." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source=( + "coupled/paper_ensemble/TABLE_microphysics_parameters.md " + "(30 ion pairs cm^-3 s^-1, galactic cosmic rays at ~20 km)" + ), + caveat=( + "The MODEL defaults this to 0.0, which disables ion-induced nucleation entirely; the " + "ensemble value of 30 is used here instead. It is a bare constant with no derivation: " + "a cited function of altitude, latitude and solar-cycle phase is task 0.5, and until " + "one is agreed it stays a fixed number rather than a computed-looking one." + ), + ) + + +class Chemistry(SchemaModel): + """Gas-phase chemistry and photolysis.""" + + photolysis: PhotolysisMode = SciField( + default=PhotolysisMode.TUVX, + unit=Unit.DIMENSIONLESS, + label="Photolysis", + description=( + "Photolysis driver. Also gates the sulfur chain in the model today: SO2->SO3->H2SO4 is " + "active only when this is not REFERENCE." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:99 (photolysis='tuvx')", + ) + so2_ho2_rate: float = SciField( + default=1.0e-18, + unit=Unit.CM3_PER_MOLEC_PER_S, + ge=0.0, + label="SO2 + HO2 rate constant", + description="Rate constant for SO2 + HO2 -> SO3 + OH. Active only in the sulfur chain.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:130", + cite="JPL 19-5, reaction I34", + caveat=( + "JPL gives only an UPPER LIMIT (~1e-18) for this reaction and recommends no products, " + "so this is a deliberate sensitivity knob, not a measured rate. 0 removes the channel; " + "1e-18/1e-17/1e-16 scan the plausible range." + ), + examples=[0.0, 1e-18, 1e-17, 1e-16], + ) + + +class Numerics(SchemaModel): + """Time stepping. The two steps are coupled by a divisibility rule the model enforces.""" + + output_dt_s: float = SciField( + default=600.0, + unit=Unit.SECOND, + gt=0.0, + label="Output step", + description=( + "Interval at which state is recorded. NOTE for anyone reading results: outer intervals " + "snap to the terminator, so the ACTUAL mean step is ~592 s against this nominal 600 s. " + "Never reconstruct the time axis as i * dt; use the stored t." + ), + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:80", + ) + couple_dt_s: float = SciField( + default=600.0, + unit=Unit.SECOND, + gt=0.0, + label="Coupling step", + description=( + "Outer operator-split step: the cadence at which TUV-x J and aerosol optics are " + "recomputed and frozen. Within it, the gas/TOMAS/dilution coupling is resolved on an " + "adaptive micro-step, so this does not have to be small." + ), + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:85", + examples=[300.0, 600.0], + ) + + @model_validator(mode="after") + def _check_step_divisibility(self) -> Numerics: + """Mirror of ``coupled/coupled_scenario.py:200-204``. + + Mirrored rather than deferred so a form can reject the combination without importing the + model (ADR-002). If the model's rule changes, this citation is how you find this copy. + """ + if self.couple_dt_s > self.output_dt_s: + raise ValueError( + f"couple_dt_s ({self.couple_dt_s}) must be <= output_dt_s ({self.output_dt_s})" + ) + ratio = self.output_dt_s / self.couple_dt_s + if abs(ratio - round(ratio)) > 1e-9: + raise ValueError( + f"output_dt_s ({self.output_dt_s}) must be an integer multiple of couple_dt_s " + f"({self.couple_dt_s}); got a ratio of {ratio}" + ) + return self + + +class ProcessSwitches(SchemaModel): + """Per-process on/off flags. Mirrors ``coupled.coupled_scenario.Switches``.""" + + sulfur: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Sulfur chain", + description="Gas-phase SO2 -> SO3 -> H2SO4 chain.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:42", + ) + nucleation: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Nucleation", + description="TOMAS binary H2SO4-H2O nucleation.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:43", + ) + condensation: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Condensation", + description="TOMAS condensation of H2SO4 onto existing particles.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:44", + ) + coagulation: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Coagulation", + description="TOMAS Brownian coagulation.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:45", + ) + aerosol_to_j: bool = SciField( + default=False, + unit=Unit.DIMENSIONLESS, + label="Aerosol -> photolysis", + description="Feed the box aerosol's optics back into the TUV-x radiation field.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:106 (aerosol_to_j=False)", + caveat=( + "The MODEL defaults this on; the ensemble runs it off. The aerosol radiator in the " + "TUV-x port is approximate -- an exact one is deferred (see the repository's " + "validation status)." + ), + ) + heating_to_t: bool = SciField( + default=False, + unit=Unit.DIMENSIONLESS, + label="Radiative heating -> T", + description="Let radiative heating change the box temperature.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:106 (heating_to_t=False)", + caveat=( + "The heating term is SHORTWAVE-ONLY -- no longwave cooling (AD-5.4) -- so switching it " + "on gives a one-sided ~+1.2 K / 10 d warm drift, not an energy balance. Every science " + "script leaves it off. The UI must warn on enable rather than silently drifting." + ), + ) + dilution: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Dilution", + description="Plume dilution and entrainment of background gas and aerosol.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:106 (dilution=True)", + ) + + +class Termination(SchemaModel): + """Limits that stop a run. A run stopped by one is never presented as converged.""" + + max_wall_time_s: float = SciField( + default=3600.0, + unit=Unit.SECOND, + gt=0.0, + label="Max wall time", + description=( + "Wall-clock cap enforced by the runner (task 0.6). 3600 s covers the measured 3-5 min " + "for a 10-day/80-bin case and 30-40 min for 60 days (BLOCKING-4), with headroom." + ), + provenance=Provenance.CONVENTION, + source="docs/studio/ASSUMPTIONS.md ASSUMPTION-4", + ) + max_sim_time_days: float | None = SciField( + default=None, + unit=Unit.DAY, + gt=0.0, + label="Max simulated time", + description=( + "Optional early stop in SIMULATED time. None means the run is bounded by " + "schedule.duration_days alone, which is the normal case -- this is not a second copy " + "of the duration, it is a lower ceiling for a run you expect to cut short." + ), + provenance=Provenance.CONVENTION, + caveat=( + "The model's stop_condition callback receives only (t1_seconds, wet_SA), so " + "multi-quantity termination criteria (e.g. on SO2 or number) are NOT available and " + "must raise rather than be approximated. See the capability register in " + "OPEN_QUESTIONS.md." + ), + ) + + +def _group(model: type[SchemaModel], description: str) -> Any: + """A field holding a nested group of scientific fields. + + Groups carry no unit, range or provenance of their own -- their leaves do -- so they use plain + ``Field``. The metadata-completeness test knows this and checks the leaves. + + Every group is defaultable, which is what makes ``RunConfig()`` with no arguments the paper + ensemble's golden case rather than a validation error. That property is load-bearing: it is the + starting point a form opens on, and the base of a RunSet. + """ + return Field(default_factory=model, description=description) + + +class RunConfig(SchemaModel): + """One simulation, fully described. + + Identity is the SHA-256 of the canonical JSON of THIS object (ADR-006), so anything that changes + the result belongs here and anything that does not must stay out. In particular there is no + ``label``, ``notes`` or ``output_dir`` field: a run's name is a property of the run, not of the + physics, and two runs whose only difference is a name are the same computation. + """ + + schema_version: Literal["0.1.0"] = SciField( + default=SCHEMA_VERSION, + unit=Unit.DIMENSIONLESS, + label="Schema version", + description=( + "Version of this schema. Part of the hashed identity: old configs are never silently " + "reinterpreted under new semantics." + ), + provenance=Provenance.CONVENTION, + ) + site: Site = _group(Site, "Location and thermodynamic state of the box.") + schedule: Schedule = _group(Schedule, "Release time and simulated duration.") + injection: Injection = _group(Injection, "What is released, and into what initial volume.") + background: Background = _group( + Background, "Composition and aerosol of the air being entrained." + ) + dilution: Dilution = _group(Dilution, "Plume expansion and entrainment.") + microphysics: Microphysics = _group( + Microphysics, "TOMAS resolution and sensitivity multipliers." + ) + chemistry: Chemistry = _group(Chemistry, "Gas-phase chemistry and photolysis.") + numerics: Numerics = _group(Numerics, "Time stepping.") + switches: ProcessSwitches = _group(ProcessSwitches, "Per-process on/off flags.") + termination: Termination = _group(Termination, "Limits that stop a run.") + + def canonical_json(self) -> str: + """Canonical JSON serialisation. See ``studio.schema.hashing``.""" + from studio.schema.hashing import canonical_json + + return canonical_json(self) + + def config_hash(self) -> str: + """Stable SHA-256 over the canonical JSON -- this config's identity (ADR-006).""" + from studio.schema.hashing import config_hash + + return config_hash(self) + + +#: Convenience alias for annotating "a path into a RunConfig", e.g. ``"microphysics.n_bins"``. +ConfigPath = Annotated[str, "dotted path into RunConfig"] + +__all__ = [ + "PAPER_BACKGROUND_GAS_PPTV", + "SCHEMA_VERSION", + "Background", + "Chemistry", + "ConfigPath", + "Dilution", + "Injection", + "Microphysics", + "Numerics", + "ProcessSwitches", + "RunConfig", + "Schedule", + "SchemaModel", + "Site", + "Termination", +] diff --git a/studio/schema/enums.py b/studio/schema/enums.py new file mode 100644 index 0000000..ec60d97 --- /dev/null +++ b/studio/schema/enums.py @@ -0,0 +1,104 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Closed choice sets, with the model's own string values wherever one exists. + +Where a member's value equals the model's string, ``studio/modelio`` passes it through unchanged and +there is nothing to get wrong. Exactly ONE member deviates -- ``DilutionRegime.CONSTANT`` -- and it +says so at the point of deviation, because that is where the 0.4 equivalence test has to account +for it. +""" + +from __future__ import annotations + +from enum import StrEnum + + +class PhotolysisMode(StrEnum): + """Photolysis driver. Values match ``coupled.coupled_scenario.PHOTOLYSIS_MODES`` exactly. + + Note the sulfur chain is gated on this being non-``reference`` in the model today + (``Env.sulfur_chain = photolysis != "reference"``), so it is not purely a radiation choice. + """ + + #: Fixed 45-degree J, on by day / off by night. Reproduces the original MATLAB model. + REFERENCE = "reference" + #: Reference J scaled by the real solar zenith angle. + SZA = "sza" + #: Absolute per-reaction J from the TUV-x port at the box altitude. The paper ensemble's choice. + TUVX = "tuvx" + + +class DilutionRegime(StrEnum): + """Plume volume-expansion regime (Schumann et al. 1998 form; ``coupled/dilution.py:39``). + + The four constant-Kz regimes share ``V(t)/V0 = max(1, t^0.8)`` for t <= 1e4 s and + ``1585 * exp[k (t - 1e4)^(3/2)]`` after, differing only in the turbulent-growth coefficient k. + ``BURST`` replaces the single exponential with a three-stage sequence (Kz = 10 m^2 s^-1 for + ~14 h), continuous at the breakpoints. See ``TABLE_dilution_parameters.md``. + """ + + #: Constant first-order rate from ``dilution.rate_per_s`` instead of a V(t) curve. + #: THE ONE VALUE THAT IS NOT THE MODEL'S STRING: the model spells this ``""`` (empty + #: ``dilution_regime``), which cannot be a sane dropdown key. ``studio/modelio`` maps + #: ``CONSTANT -> ""``, and the 0.4 equivalence test must cover this case explicitly. + CONSTANT = "constant" + #: Low Kz, k = 2.811e-9. + D1 = "D1" + #: Medium Kz, k = 8.89e-9. The paper ensemble's default regime. + D2 = "D2" + #: High Kz, k = 2.811e-8. + D3 = "D3" + #: Very high Kz, k = 5.33e-8. + D5 = "D5" + #: Transient burst of turbulence; three-stage, continuous at the breakpoints. + BURST = "burst" + + +class BackgroundAerosol(StrEnum): + """Background aerosol size distribution seeded into the initial TOMAS state. + + Values match ``coupled.tomas_bridge.BACKGROUND_MODES`` keys, plus ``redcircles`` (the tabulated + loader, which is the model's default and is not in that dict). + + The lognormal mode sets are DIGITIZED from source plots, and the number concentrations are + chosen so each mode's peak dN/dlogDp matches the value read off the plot -- the most reliable + digitized feature. They are approximations of a figure, not published parameters, which is why + every member below carries where it came from. + + Wet-vs-dry matters here and is per-dataset rather than declared: ``AER_GEO`` and ``CESM_G6_AMB`` + are specified at AMBIENT conditions and skip the STP->ambient factor on seeding, the others are + at STP. Making that an explicit field is SCIENCE-3 (issue #55). + """ + + #: Marianna's tabulated distribution. The model's default. + REDCIRCLES = "redcircles" + #: SABRE young air (high N2O), one mode, peak dN/dlogDp ~1000 cm^-3. + SABR_330 = "sabr_330" + #: SABRE mid air (310-320 ppbv N2O), peak ~320 cm^-3. + SABR_310 = "sabr_310" + #: SABRE aged air (220-230 ppbv N2O), Dg = 0.12 um, sigma_g = 1.6. The paper ensemble's clean + #: background, and the one used by the golden case. + SABR_220 = "sabr_220" + #: CESM G6 SAI, three modes, read at STP. + CESM_G6 = "cesm_g6" + #: CESM G6 with the source plot read as AMBIENT. Kept separate so ``CESM_G6`` stays + #: reproducible. + CESM_G6_AMB = "cesm_g6_amb" + #: AER 2D geoengineered stratosphere (Pierce et al., 5 Mt-S/yr, 95 nm case), ambient basis. + AER_GEO = "aer_geo" + + +class AxisKind(StrEnum): + """How a ``RunSet`` axis combines with the others. See ``studio/schema/runset.py``.""" + + #: Crossed with every other GRID/LIST axis (Cartesian product). + GRID = "grid" + #: Advanced in lockstep with the other ZIP axes; the zipped group is then crossed with the rest. + ZIP = "zip" + #: Like GRID, but each point sets SEVERAL fields at once -- a covarying group, e.g. the paper + #: ensemble's site axis, where latitude, T, p and H2O move together and only certain + #: combinations are physically meaningful. + LIST = "list" + + +__all__ = ["AxisKind", "BackgroundAerosol", "DilutionRegime", "PhotolysisMode"] diff --git a/studio/schema/export.py b/studio/schema/export.py new file mode 100644 index 0000000..1b1f271 --- /dev/null +++ b/studio/schema/export.py @@ -0,0 +1,76 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""JSON Schema export and the flat field catalogue. + +The web client generates its form from the exported JSON Schema (ADR-002) -- **nothing in the UI may +invent a field that does not exist here**. Pydantic emits the structure; ``SciField``'s metadata +rides along under ``x-studio`` because that is how ``json_schema_extra`` works, so the export needs +no parallel serialisation path that could drift from the models. + +``field_catalogue()`` is the flat view: dotted path -> metadata, for every leaf. It is what a +"what does this parameter mean, and where did its default come from?" panel reads, what the +dependency-graph engine (task 0.3) will walk to build the DAG from ``derived_from``, and what the +metadata-completeness test iterates. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +from pydantic import BaseModel + +from studio.schema.config import SCHEMA_VERSION, RunConfig +from studio.schema.fields import field_metadata + +#: Stable identifier for the exported schema. Versioned with the schema itself so a client can tell +#: which one it is holding. +SCHEMA_ID = f"https://reflective.org/studio/schemas/run-config/{SCHEMA_VERSION}.json" + + +def iter_leaf_fields( + model_cls: type[BaseModel] = RunConfig, prefix: str = "" +) -> Iterator[tuple[str, Any]]: + """Yield ``(dotted_path, FieldInfo)`` for every leaf field, descending into nested groups.""" + for name, info in model_cls.model_fields.items(): + path = f"{prefix}{name}" + annotation = info.annotation + if isinstance(annotation, type) and issubclass(annotation, BaseModel): + yield from iter_leaf_fields(annotation, prefix=f"{path}.") + else: + yield path, info + + +def field_catalogue(model_cls: type[BaseModel] = RunConfig) -> dict[str, dict[str, Any]]: + """Flat ``{dotted path: metadata}`` for every leaf field. + + The metadata is ``SciField``'s ``x-studio`` block plus the field's ``description``, ``default`` + and ``required`` flag -- everything needed to render and explain one input. + """ + catalogue: dict[str, dict[str, Any]] = {} + for path, info in iter_leaf_fields(model_cls): + entry = field_metadata(info) + entry["description"] = info.description + entry["required"] = info.is_required() + if not info.is_required(): + default = info.get_default(call_default_factory=True, validated_data=None) + entry["default"] = default.value if hasattr(default, "value") else default + catalogue[path] = entry + return catalogue + + +def run_config_json_schema() -> dict[str, Any]: + """The JSON Schema the web client consumes. + + ``by_alias=False`` because the schema has no aliases and the field names ARE the paths used by + ``RunSet`` axes; a client that reads a path here can use it there unchanged. + """ + schema = RunConfig.model_json_schema(by_alias=False, mode="serialization") + schema["$schema"] = "https://json-schema.org/draft/2020-12/schema" + schema["$id"] = SCHEMA_ID + schema["title"] = "Plume Studio run configuration" + schema["x-studio-schema-version"] = SCHEMA_VERSION + return schema + + +__all__ = ["SCHEMA_ID", "field_catalogue", "iter_leaf_fields", "run_config_json_schema"] diff --git a/studio/schema/fields.py b/studio/schema/fields.py new file mode 100644 index 0000000..ef93121 --- /dev/null +++ b/studio/schema/fields.py @@ -0,0 +1,169 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``SciField`` -- the field-metadata carrier the whole schema is built from (ADR-002). + +A schema field is not just a type and a default. It is a scientific quantity, and the thing that +makes it usable in a form -- or defensible in a paper -- is the metadata around it: what unit it is +in, what range is valid, and above all WHERE ITS DEFAULT CAME FROM. + +That last part is the point. The defaults in this project are not arbitrary: they are the paper +ensemble's configuration, recorded in ``TABLE_microphysics_parameters.md`` and +``TABLE_dilution_parameters.md``, or the model's own defaults in ``coupled/coupled_scenario.py``, or +values with a literature citation. A default with no recorded source is exactly the failure mode +``studio/CLAUDE.md`` exists to prevent, so ``provenance`` is REQUIRED and its consistency rules are +enforced at import time -- a bad field definition raises when the module is imported, not when a run +produces a quietly wrong number. + +Metadata lands in the exported JSON Schema under the ``x-studio`` key, which is what the web client +reads to generate a form. Everything below is data; nothing here computes physics. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from enum import StrEnum +from typing import Any, Final + +from pydantic import Field +from pydantic.fields import FieldInfo + +from studio.schema.units import Unit + + +class Provenance(StrEnum): + """Where a field's default came from. Required on every field; no default value. + + The distinction that matters is between a value someone MEASURED or PUBLISHED and a value + someone CHOSE. Both are legitimate; conflating them is not. + """ + + #: The model's own default, unchanged. ``source`` cites the file:line. + MODEL_DEFAULT = "model_default" + #: The paper ensemble's configuration. ``source`` cites the TABLE_*.md row or the runner line. + PAPER_ENSEMBLE = "paper_ensemble" + #: From the literature. ``cite`` is required. + LITERATURE = "literature" + #: A chosen convention with no external source -- an interface decision, not a scientific claim. + CONVENTION = "convention" + #: Computed from other fields. ``derived_from`` is required; the value is not supplied by hand. + DERIVED = "derived" + #: No defensible default exists. The user must supply it; there is no fallback (ADR-005). + USER_REQUIRED = "user_required" + + +#: JSON Schema extension key holding Studio's metadata. Namespaced with the conventional `x-` +#: prefix so a generic JSON Schema validator ignores it. +EXTENSION_KEY: Final = "x-studio" + +_SENTINEL: Final = object() + + +def SciField( + *, + unit: Unit, + description: str, + provenance: Provenance, + default: Any = _SENTINEL, + default_factory: Any = None, + label: str | None = None, + source: str | None = None, + cite: str | None = None, + derived_from: Sequence[str] = (), + caveat: str | None = None, + ge: float | None = None, + le: float | None = None, + gt: float | None = None, + lt: float | None = None, + examples: Sequence[Any] | None = None, +) -> Any: + """A pydantic field carrying Studio's scientific metadata. + + Args: + unit: Canonical unit (ADR-003). Use ``Unit.DIMENSIONLESS`` for pure scale factors. + description: What the quantity IS -- enough for someone who has not read the model. + provenance: Where the default came from. See ``Provenance``. + default: The default value. Omit for a required field. + default_factory: For mutable defaults (dicts, tuples), as in pydantic. + label: Short human-readable name for a form. Defaults to the field name at export time. + source: File:line or document reference backing the default. + cite: Literature citation. Required when ``provenance`` is ``LITERATURE``. + derived_from: Dotted paths this field is computed from. Required when ``DERIVED``, and + forbidden otherwise -- it is what the dependency graph in task 0.3 is built from. + caveat: A warning that must travel with the value into the UI (e.g. a one-sided physics + approximation). Surfaced, never hidden. + ge, le, gt, lt: Validity bounds, passed to pydantic AND recorded in the metadata. + examples: Illustrative values, e.g. the levels this field takes in the paper ensemble. + + Raises: + ValueError: If the metadata is internally inconsistent. Raised at import time, on purpose. + """ + if provenance is Provenance.LITERATURE and not cite: + raise ValueError("provenance=LITERATURE requires `cite`; a citation is the whole claim") + if provenance in (Provenance.MODEL_DEFAULT, Provenance.PAPER_ENSEMBLE) and not source: + raise ValueError( + f"provenance={provenance.value} requires `source` (file:line or TABLE_*.md row) -- " + f"the point of these two values is that the default is traceable" + ) + if provenance is Provenance.DERIVED: + if not derived_from: + raise ValueError("provenance=DERIVED requires `derived_from`") + if default is not _SENTINEL and default is not None: + raise ValueError( + "a DERIVED field must not carry a hand-written default; it is computed from " + f"{list(derived_from)} by the dependency-graph engine (task 0.3)" + ) + elif derived_from: + raise ValueError( + f"`derived_from` is only meaningful with provenance=DERIVED, got {provenance.value}" + ) + if provenance is Provenance.USER_REQUIRED and ( + default is not _SENTINEL or default_factory is not None + ): + raise ValueError( + "provenance=USER_REQUIRED means there is no defensible default, so it must not have one" + ) + if not description.strip(): + raise ValueError("description is required and must not be blank") + + extra: dict[str, Any] = { + "unit": unit.value, + "provenance": provenance.value, + "derived_from": list(derived_from), + } + for key, value in (("label", label), ("source", source), ("cite", cite), ("caveat", caveat)): + if value is not None: + extra[key] = value + bounds = { + name: v for name, v in (("ge", ge), ("le", le), ("gt", gt), ("lt", lt)) if v is not None + } + if bounds: + extra["range"] = bounds + + kwargs: dict[str, Any] = { + "description": description, + "json_schema_extra": {EXTENSION_KEY: extra}, + **bounds, + } + if examples is not None: + kwargs["examples"] = list(examples) + if default_factory is not None: + kwargs["default_factory"] = default_factory + elif default is not _SENTINEL: + kwargs["default"] = default + return Field(**kwargs) + + +def field_metadata(info: FieldInfo) -> dict[str, Any]: + """Studio metadata for a pydantic field, or ``{}`` if it was not declared with ``SciField``. + + Used by the metadata-completeness test and by the JSON Schema export; a caller that gets ``{}`` + is looking at a field that bypassed ``SciField``, which is a bug rather than a special case. + """ + extra = info.json_schema_extra + if not isinstance(extra, dict): + return {} + meta = extra.get(EXTENSION_KEY, {}) + return dict(meta) if isinstance(meta, dict) else {} + + +__all__ = ["EXTENSION_KEY", "Provenance", "SciField", "field_metadata"] diff --git a/studio/schema/hashing.py b/studio/schema/hashing.py new file mode 100644 index 0000000..6d81291 --- /dev/null +++ b/studio/schema/hashing.py @@ -0,0 +1,99 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Canonical serialisation and the config hash (ADR-006). + +A run's identity is the SHA-256 of the canonical JSON of its ``RunConfig``. That hash is also the +cache key, the golden-fixture key and half of the idempotence check, so **a hash that drifts +silently invalidates everything at once** -- and does it quietly, which is worse. Hence a canonical +form pinned by explicit rules and a test that asserts a known hash rather than merely asserting +self-consistency. + +Canonical form: + +1. ``model_dump(mode="json")`` -- enums become their string values, tuples become lists. +2. ``sort_keys=True`` -- insertion order cannot leak into identity. This is what makes two configs + built by different code paths hash the same. +3. ``separators=(",", ":")`` -- no incidental whitespace. +4. ``ensure_ascii=False`` with a UTF-8 encode -- one representation per string, not two. +5. ``allow_nan=False`` -- ``NaN``/``Infinity`` are not JSON and are not a valid configuration + either. This RAISES rather than emitting a non-standard token (ADR-005). + +Float formatting is Python's ``repr``, which has produced the shortest round-tripping decimal since +3.1 and is therefore stable across the versions this project supports. That is the one assumption +here that is a property of the interpreter rather than of this module, and the pinned-hash test is +what would catch it changing. + +There is no hashing anywhere else in the repository -- this is the first -- so nothing constrains +the choice except the need for it to never change silently. +""" + +from __future__ import annotations + +import hashlib +import json +from typing import Any + +from pydantic import BaseModel + +#: Named so a future change is a visible migration rather than an invisible one. If the canonical +#: form ever has to change, bump this, bump the schema version, and re-baseline the fixtures on +#: purpose -- never quietly. +CANONICAL_FORM_VERSION = 1 + + +def canonical_payload(model: BaseModel) -> dict[str, Any]: + """The JSON-mode dict that gets serialised. + + Exposed for tests, and for debugging a hash change: diffing two payloads says which field moved. + """ + payload = model.model_dump(mode="json") + if not isinstance(payload, dict): # pragma: no cover -- pydantic models always dump to a dict + raise TypeError(f"expected a dict from model_dump, got {type(payload).__name__}") + return payload + + +def canonical_json(model: BaseModel) -> str: + """Canonical JSON string for ``model``. + + Raises: + ValueError: If the config contains NaN or Infinity, which are neither valid JSON nor a valid + configuration. Failing here beats writing a token no other parser will read back. + """ + try: + return json.dumps( + canonical_payload(model), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ) + except ValueError as exc: + raise ValueError( + f"{type(model).__name__} is not canonically serialisable: {exc}. NaN and Infinity are " + f"not valid JSON and not a valid configuration; fix the value rather than the encoder." + ) from exc + + +def config_hash(model: BaseModel) -> str: + """Stable SHA-256 (hex) over the canonical JSON of ``model``.""" + return hashlib.sha256(canonical_json(model).encode("utf-8")).hexdigest() + + +def short_hash(model: BaseModel, length: int = 12) -> str: + """First ``length`` hex characters of the config hash, for display and directory names. + + Display only. Twelve hex characters is ~48 bits, fine for a human reading a list and not fine as + an identity; equality checks use the full hash. + """ + if not 4 <= length <= 64: + raise ValueError(f"length must be in [4, 64], got {length}") + return config_hash(model)[:length] + + +__all__ = [ + "CANONICAL_FORM_VERSION", + "canonical_json", + "canonical_payload", + "config_hash", + "short_hash", +] diff --git a/studio/schema/runset.py b/studio/schema/runset.py new file mode 100644 index 0000000..6778089 --- /dev/null +++ b/studio/schema/runset.py @@ -0,0 +1,283 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSet`` -- the primary user-facing object. A single run is the N = 1 case of it. + +**There is no separate single-run code path.** A run with no sweep is a ``RunSet`` with zero axes, +and it goes through exactly the same expansion. Bolting sweeps on later would mean rewriting the +config layer, the results schema and every comparison view, which is why this exists in the first +version of the schema rather than the third. + +The shape is taken from what the paper ensemble already does. Its 810 cases are +``itertools.product`` over six axes (``run_ensemble.py:82``), and its case IDs are the axis LABELS +joined by ``__`` (``:84``) -- e.g. ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1``. That naming is +genuinely good design for a fixed factorial and it is preserved here: an axis point carries a label, +and the expanded run's label is the join. Identity is still the config hash (ADR-006); the label is +for people, and for the existing directory layout. + +Three axis kinds: + +* ``GRID`` -- crossed with every other GRID/LIST axis. +* ``LIST`` -- also crossed, but each point assigns SEVERAL fields at once. The paper ensemble's site + axis is exactly this: ``("30N_20km", 30.0, 210.0, 55.0, 6.9104)`` moves latitude, T, p and H2O + together, and the intermediate combinations are not physically meaningful. +* ``ZIP`` -- advanced in lockstep with the other ZIP axes; the zipped group is then crossed with the + GRID/LIST axes as a single pseudo-axis, positioned where the first ZIP axis was declared. + +Ordering is deterministic and matches ``itertools.product``: axes vary right-to-left, the LAST axis +fastest. This is not an implementation detail -- it is what makes an expansion reproducible and what +lets a RunSet reproduce the existing ensemble's case order exactly. +""" + +from __future__ import annotations + +import itertools +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from studio.schema.config import RunConfig, SchemaModel +from studio.schema.enums import AxisKind + + +def resolve_path(model_cls: type[BaseModel], path: str) -> None: + """Validate that ``path`` names a real LEAF field, or raise. + + Fails loud and early on both ways of getting it wrong: an axis over ``"microphysics.n_bin"`` is + a typo that would otherwise surface as a sweep whose axis silently never varied, and an axis + over ``"microphysics"`` would assign a whole group at once -- which the model would accept but + which has no unit, no provenance and no place in the dependency graph task 0.3 builds from leaf + paths. Groups are containers; values live on leaves. + """ + parts = path.split(".") + if not all(parts): + raise ValueError(f"malformed path {path!r}") + current: type[BaseModel] = model_cls + for i, part in enumerate(parts): + fields = getattr(current, "model_fields", None) + if fields is None or part not in fields: + known = sorted(fields) if fields else [] + where = ".".join(parts[:i]) or model_cls.__name__ + raise ValueError(f"unknown field {part!r} in {where}; known fields: {known}") + annotation = fields[part].annotation + is_group = isinstance(annotation, type) and issubclass(annotation, BaseModel) + if i < len(parts) - 1: + if not is_group: + raise ValueError( + f"{'.'.join(parts[: i + 1])} is a leaf field; {path!r} tries to descend into it" + ) + current = annotation + elif is_group: + raise ValueError( + f"{path!r} is a group of fields, not a leaf; assign its leaves individually " + f"({', '.join(f'{path}.{name}' for name in sorted(annotation.model_fields))})" + ) + + +def apply_assignments(config: RunConfig, assignments: dict[str, Any]) -> RunConfig: + """Return a new ``RunConfig`` with ``assignments`` applied. + + Re-validates through the model rather than mutating: configs are frozen (ADR-004), and an axis + value that violates a bound or a cross-field rule must fail here, at expansion time, rather than + at submit time for run 407 of 810. + """ + payload = config.model_dump(mode="python") + for path, value in assignments.items(): + resolve_path(RunConfig, path) + parts = path.split(".") + cursor: dict[str, Any] = payload + for part in parts[:-1]: + cursor = cursor[part] + cursor[parts[-1]] = value + return RunConfig.model_validate(payload) + + +class AxisPoint(SchemaModel): + """One level of an axis: a short label and the field assignments it stands for.""" + + label: str = Field( + description=( + "Short token used to build the run label, e.g. 'sabr220' or 'a1p0'. Kept terse because " + "it becomes part of a directory name, following the existing ensemble's convention." + ), + min_length=1, + ) + assignments: dict[str, Any] = Field( + description="Dotted RunConfig paths to values, applied together as one point.", + min_length=1, + ) + + +class Axis(SchemaModel): + """One dimension of a sweep.""" + + name: str = Field(description="Axis name, for display and for error messages.", min_length=1) + kind: AxisKind = Field(default=AxisKind.GRID, description="How this axis combines with others.") + points: tuple[AxisPoint, ...] = Field(description="The levels of this axis.", min_length=1) + + @model_validator(mode="after") + def _check_points(self) -> Axis: + labels = [point.label for point in self.points] + duplicates = sorted({label for label in labels if labels.count(label) > 1}) + if duplicates: + raise ValueError( + f"axis {self.name!r} has duplicate point labels {duplicates}; labels become run " + f"labels and directory names, so they must be unique within an axis" + ) + for point in self.points: + for path in point.assignments: + resolve_path(RunConfig, path) + if self.kind is not AxisKind.LIST: + multi = [p.label for p in self.points if len(p.assignments) > 1] + if multi: + raise ValueError( + f"axis {self.name!r} is {self.kind.value.upper()} but points {multi} assign " + f"more than one field; a covarying group is what LIST is for" + ) + paths = {path for point in self.points for path in point.assignments} + if len(paths) > 1: + raise ValueError( + f"axis {self.name!r} varies {sorted(paths)}; a {self.kind.value.upper()} axis " + f"varies exactly one field. Use LIST for a covarying group." + ) + return self + + @classmethod + def over( + cls, + name: str, + path: str, + levels: dict[str, Any], + kind: AxisKind = AxisKind.GRID, + ) -> Axis: + """Build a single-field axis from ``{label: value}``. The common case, spelled short.""" + return cls( + name=name, + kind=kind, + points=tuple( + AxisPoint(label=label, assignments={path: value}) for label, value in levels.items() + ), + ) + + +class ExpandedRun(BaseModel): + """One concrete run produced by expanding a ``RunSet``.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + #: Axis point labels joined by ``__``, matching the existing ensemble's case IDs. Empty for a + #: RunSet with no axes -- identity is the hash, so an unlabelled run is not an anonymous one. + label: str + #: The axis point label per axis name, so a comparison view can group by axis without re-parsing + #: the label string (``_tokens()`` in make_paper_candidate_plots.py:92 exists because that + #: re-parsing is otherwise necessary). + coordinates: dict[str, str] + config: RunConfig + + @property + def config_hash(self) -> str: + """This run's identity (ADR-006).""" + return self.config.config_hash() + + +class RunSet(SchemaModel): + """A base configuration plus the axes to sweep over it. + + ``expand()`` is the only way to get runs out, including when there are no axes. + """ + + base: RunConfig = Field( + default_factory=RunConfig, description="The configuration every run starts from." + ) + axes: tuple[Axis, ...] = Field( + default=(), description="Sweep axes; empty means a single run (N = 1)." + ) + + @model_validator(mode="after") + def _check_axes(self) -> RunSet: + names = [axis.name for axis in self.axes] + duplicates = sorted({name for name in names if names.count(name) > 1}) + if duplicates: + raise ValueError(f"duplicate axis names {duplicates}") + zipped = [axis for axis in self.axes if axis.kind is AxisKind.ZIP] + if zipped: + lengths = {axis.name: len(axis.points) for axis in zipped} + if len(set(lengths.values())) > 1: + raise ValueError( + f"ZIP axes are advanced in lockstep and must have equal length, got {lengths}" + ) + assigned: dict[str, str] = {} + for axis in self.axes: + for path in {p for point in axis.points for p in point.assignments}: + if path in assigned and assigned[path] != axis.name: + raise ValueError( + f"axes {assigned[path]!r} and {axis.name!r} both assign {path!r}; the " + f"result would depend on axis order, so it is rejected rather than ordered" + ) + assigned[path] = axis.name + return self + + def size(self) -> int: + """Number of runs ``expand()`` will produce, WITHOUT building any of them. + + The existing runners' ``plan`` verb prints a count before committing compute; this is the + equivalent, and it stays cheap no matter how large the sweep is. + """ + crossed = [axis for axis in self.axes if axis.kind is not AxisKind.ZIP] + zipped = [axis for axis in self.axes if axis.kind is AxisKind.ZIP] + total = 1 + for axis in crossed: + total *= len(axis.points) + if zipped: + total *= len(zipped[0].points) + return total + + def expand(self) -> list[ExpandedRun]: + """Every run in this set, in a deterministic order (last axis varies fastest).""" + groups, order = self._axis_groups() + runs: list[ExpandedRun] = [] + for combination in itertools.product(*groups): + assignments: dict[str, Any] = {} + coordinates: dict[str, str] = {} + for axes_in_group, points in zip(order, combination, strict=True): + for axis, point in zip(axes_in_group, points, strict=True): + assignments.update(point.assignments) + coordinates[axis.name] = point.label + label = "__".join( + coordinates[axis.name] for axis in self.axes if axis.name in coordinates + ) + runs.append( + ExpandedRun( + label=label, + coordinates=coordinates, + config=apply_assignments(self.base, assignments) if assignments else self.base, + ) + ) + return runs + + def _axis_groups(self) -> tuple[list[list[tuple[AxisPoint, ...]]], list[list[Axis]]]: + """Axes as product operands, preserving declaration order. + + Each operand is a list of "point tuples": one point per axis in that group. Crossed axes + form single-axis groups; all ZIP axes form ONE group whose points advance together, placed + where the first ZIP axis was declared. + """ + zipped = [axis for axis in self.axes if axis.kind is AxisKind.ZIP] + groups: list[list[tuple[AxisPoint, ...]]] = [] + order: list[list[Axis]] = [] + zip_emitted = False + for axis in self.axes: + if axis.kind is AxisKind.ZIP: + if zip_emitted: + continue + zip_emitted = True + groups.append( + [tuple(points) for points in zip(*(a.points for a in zipped), strict=True)] + ) + order.append(zipped) + else: + groups.append([(point,) for point in axis.points]) + order.append([axis]) + return groups, order + + +__all__ = ["Axis", "AxisPoint", "ExpandedRun", "RunSet", "apply_assignments", "resolve_path"] diff --git a/studio/schema/units.py b/studio/schema/units.py new file mode 100644 index 0000000..77502af --- /dev/null +++ b/studio/schema/units.py @@ -0,0 +1,80 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The canonical unit registry. + +Canonical units are the MODEL's native units, not SI (ADR-003, ASSUMPTION-1): mbar, ppmv, pptv, K, +s, um^2 cm^-3, cm^-3 s^-1. The reason is float identity at the model seam -- the existing ensemble's +constants are decimals in native units (``WTR = 6.9104`` ppm at ``run_ensemble.py:62``) and +round-tripping them through SI is not guaranteed to return the same float64. Reproducing trusted +runs is worth more than SI purity here. + +Every dimensioned schema field declares one of these symbols. The set is CLOSED: a unit that is not +listed cannot be used, which turns a typo (``"ppvt"``) into an import-time error instead of a +silently unconvertible field. + +``pint`` is used for display conversion at the presentation boundary and for round-trip property +tests -- never inside the model interface, where ``studio/modelio`` hands ``CoupledScenario`` plain +floats already in native units. + +Three units are deliberately NOT pint-parseable, and say so rather than being faked: + +* ``ppmv`` / ``pptv`` are mixing ratios by VOLUME (mole fraction). pint would treat a bare + ``1e-12`` as dimensionless, losing the by-volume convention, and the conversion to a number + density depends on T and p -- which is a derivation (``studio/science``), not a unit conversion. +* ``molec`` is a count of molecules. pint has no such unit; ``cm^3 molec^-1 s^-1`` is the standard + bimolecular rate-constant unit and is carried symbolically. +""" + +from __future__ import annotations + +from enum import StrEnum +from typing import Final + + +class Unit(StrEnum): + """Canonical unit symbols. The string value is what appears in the exported JSON Schema.""" + + DIMENSIONLESS = "1" + KELVIN = "K" + MBAR = "mbar" + PPMV = "ppmv" + PPTV = "pptv" + DEGREE = "degree" + SECOND = "s" + DAY = "d" + HOUR = "h" + METRE = "m" + KILOGRAM = "kg" + CM3 = "cm^3" + PER_CM3_PER_S = "cm^-3 s^-1" + CM3_PER_MOLEC_PER_S = "cm^3 molec^-1 s^-1" + UM2_PER_CM3 = "um^2 cm^-3" + PER_SECOND = "s^-1" + COUNT = "count" + + +#: Canonical symbol -> the equivalent ``pint`` expression, or ``None`` where no faithful one exists. +#: A ``None`` here is a statement that the quantity carries a convention pint cannot represent, not +#: an omission -- see the module docstring. Checked exhaustively by the unit tests, so adding a +#: member to ``Unit`` without adding it here is a test failure rather than a runtime surprise. +PINT_EXPRESSION: Final[dict[Unit, str | None]] = { + Unit.DIMENSIONLESS: "dimensionless", + Unit.KELVIN: "kelvin", + Unit.MBAR: "millibar", + Unit.PPMV: None, # mole fraction x 1e6; by-volume convention, T/p-dependent to a number density + Unit.PPTV: None, # mole fraction x 1e12; likewise + Unit.DEGREE: "degree", + Unit.SECOND: "second", + Unit.DAY: "day", + Unit.HOUR: "hour", + Unit.METRE: "meter", + Unit.KILOGRAM: "kilogram", + Unit.CM3: "centimeter ** 3", + Unit.PER_CM3_PER_S: "1 / centimeter ** 3 / second", + Unit.CM3_PER_MOLEC_PER_S: None, # `molec` is a molecule count; pint has no such unit + Unit.UM2_PER_CM3: "micrometer ** 2 / centimeter ** 3", + Unit.PER_SECOND: "1 / second", + Unit.COUNT: None, # a plain count of things (size bins); dimensionless but not a ratio +} + +__all__ = ["PINT_EXPRESSION", "Unit"] diff --git a/studio/tests/unit/test_config_hash.py b/studio/tests/unit/test_config_hash.py new file mode 100644 index 0000000..9a3bb57 --- /dev/null +++ b/studio/tests/unit/test_config_hash.py @@ -0,0 +1,157 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The config hash must not drift. Ever, silently. + +``config_hash`` is run identity, the cache key and the golden-fixture key at once (ADR-006). If it +changes for a reason nobody noticed, every cached result stops matching and every fixture starts +missing -- and the symptom is "everything recomputes", which reads as a performance problem rather +than a correctness one. + +So the pinned hash below is a deliberate tripwire. If a change to the schema alters it, that is +correct and expected: bump ``SCHEMA_VERSION`` and update the constant IN THE SAME COMMIT, with the +reason in the message. What must never happen is the value changing without anyone deciding it +should. +""" + +from __future__ import annotations + +import json +import math +import subprocess +import sys +import textwrap + +import pytest + +from studio.schema import RunConfig, canonical_json, canonical_payload, config_hash, short_hash +from studio.schema.hashing import CANONICAL_FORM_VERSION + +#: SHA-256 of the canonical JSON of ``RunConfig()`` -- the paper ensemble's golden case, which is +#: also the schema's default configuration. Tied to SCHEMA_VERSION 0.1.0 and canonical form 1. +GOLDEN_DEFAULT_HASH = "629fc801779ca43a4a7ae43d74c43f3221b213d78e36c1dfce1e94f17d46cbe3" + + +@pytest.mark.tier_a +def test_default_config_hash_is_pinned() -> None: + """Tolerance: exact. A hash is either the same or it is a different config (ADR-006).""" + assert CANONICAL_FORM_VERSION == 1, "canonical form changed; the pinned hash must be re-derived" + assert config_hash(RunConfig()) == GOLDEN_DEFAULT_HASH, ( + "the default RunConfig's hash changed. If you meant to change the schema, bump " + "SCHEMA_VERSION and update GOLDEN_DEFAULT_HASH in this commit, with the reason in the " + "message. If you did not, something altered a default silently." + ) + + +@pytest.mark.tier_a +def test_hash_is_independent_of_dict_insertion_order() -> None: + """Two configs built by different code paths must hash the same. + + The API builds a config from a JSON body, the CLI from a YAML file, a sweep from + ``apply_assignments`` -- three insertion orders for the same run. If order leaked into the hash, + the same computation would get three identities and the cache would never hit. + """ + forward = RunConfig() + payload = forward.model_dump(mode="python") + shuffled = {key: payload[key] for key in reversed(list(payload))} + shuffled["background"] = { + key: shuffled["background"][key] for key in reversed(list(shuffled["background"])) + } + shuffled["background"]["gas_pptv"] = { + key: shuffled["background"]["gas_pptv"][key] + for key in reversed(list(shuffled["background"]["gas_pptv"])) + } + reversed_config = RunConfig.model_validate(shuffled) + + assert canonical_json(reversed_config) == canonical_json(forward) + assert config_hash(reversed_config) == config_hash(forward) + + +@pytest.mark.tier_a +def test_hash_is_stable_across_interpreters_and_hash_seeds() -> None: + """Run in fresh interpreters with different PYTHONHASHSEEDs and compare. + + Python randomises string hashing per process by default. Any dependence of the canonical form on + set or dict iteration influenced by that randomisation would make the hash vary between runs -- + invisible in one process, and the reason this check spawns real ones. It is also the closest a + single-version CI can get to the "stable across Python versions" requirement; the pinned + constant above covers the rest. + """ + probe = textwrap.dedent(""" + from studio.schema import RunConfig, config_hash + print(config_hash(RunConfig())) + """) + hashes = set() + for seed in ("0", "1", "12345", "random"): + proc = subprocess.run( + [sys.executable, "-c", probe], + capture_output=True, + text=True, + env={"PYTHONHASHSEED": seed, "PATH": "/usr/bin:/bin"}, + cwd=str(__import__("pathlib").Path(__file__).resolve().parents[3]), + ) + assert proc.returncode == 0, proc.stderr + hashes.add(proc.stdout.strip()) + assert hashes == {GOLDEN_DEFAULT_HASH}, f"hash varied across hash seeds: {hashes}" + + +@pytest.mark.tier_a +def test_a_changed_value_changes_the_hash() -> None: + """The other half of identity: different configs must not collide. + + Uses the smallest change that matters scientifically -- a nucleation scale of 1 vs 1.0000001 -- + because a hash that only notices large edits is worse than none. + """ + base = RunConfig() + tweaked = base.model_copy( + update={ + "microphysics": base.microphysics.model_copy( + update={"nucleation_rate_scale": 1.0000001} + ) + } + ) + assert config_hash(tweaked) != config_hash(base) + + +@pytest.mark.tier_a +def test_canonical_json_is_sorted_compact_and_parseable() -> None: + """The canonical form's rules, asserted rather than assumed.""" + text = canonical_json(RunConfig()) + assert ", " not in text and '": ' not in text, "canonical JSON must not contain padding spaces" + parsed = json.loads(text) + assert list(parsed) == sorted(parsed), "top-level keys must be sorted" + assert list(parsed["site"]) == sorted(parsed["site"]), "nested keys must be sorted too" + assert parsed == canonical_payload(RunConfig()) + + +@pytest.mark.tier_a +def test_enums_serialise_as_their_model_string() -> None: + """The hashed payload carries the model's own strings, so a config is readable as what it is.""" + parsed = json.loads(canonical_json(RunConfig())) + assert parsed["chemistry"]["photolysis"] == "tuvx" + assert parsed["background"]["aerosol"] == "sabr_220" + assert parsed["dilution"]["regime"] == "D2" + + +@pytest.mark.tier_a +def test_non_finite_values_raise_rather_than_serialising() -> None: + """NaN is not JSON, and it is not a configuration either (ADR-005). + + ``json.dumps`` would happily emit the non-standard ``NaN`` token, which then fails to parse in + every other language. Better to refuse at the boundary. + """ + base = RunConfig() + nan_config = base.model_copy( + update={"site": base.site.model_copy(update={"temperature_k": math.nan})} + ) + with pytest.raises(ValueError, match="not canonically serialisable"): + canonical_json(nan_config) + + +@pytest.mark.tier_a +def test_short_hash_is_a_prefix_and_bounded() -> None: + """Display-only, and it says so by refusing silly lengths.""" + config = RunConfig() + assert config_hash(config).startswith(short_hash(config)) + assert len(short_hash(config)) == 12 + with pytest.raises(ValueError, match=r"\[4, 64\]"): + short_hash(config, length=2) diff --git a/studio/tests/unit/test_runset.py b/studio/tests/unit/test_runset.py new file mode 100644 index 0000000..c7ba508 --- /dev/null +++ b/studio/tests/unit/test_runset.py @@ -0,0 +1,344 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSet`` expansion, checked against the sweep this project actually ran. + +The strongest available evidence that the axis model is faithful is that it reproduces the paper +ensemble: 810 cases, the same order, and the same case-ID labels that name the directories on disk +today. That is what ``test_reproduces_the_paper_ensemble`` does, and it is why the axis kinds are +shaped the way they are -- ``LIST`` exists because the site axis covaries latitude, T, p and H2O. + +The axis definitions live here rather than in the package: task 0.2 is the schema, not a library of +presets. If task 0.4 or 0.7 needs them, that is when they earn a home in ``studio/``. +""" + +from __future__ import annotations + +import pytest + +from studio.schema import ( + Axis, + AxisKind, + AxisPoint, + BackgroundAerosol, + DilutionRegime, + RunConfig, + RunSet, + apply_assignments, +) + +#: The six axes of ``coupled/paper_ensemble/run_ensemble.py:61-76``, in declaration order. +#: Labels are the ensemble's own tokens, so the expanded labels are its case IDs verbatim. +PAPER_AXES = ( + Axis( + name="site", + # latitude, T, p and H2O move together; the cross product is not physically meaningful + kind=AxisKind.LIST, + points=( + AxisPoint( + label="30N_20km", + assignments={ + "site.latitude_deg": 30.0, + "site.temperature_k": 210.0, + "site.pressure_mbar": 55.0, + "site.h2o_ppmv": 6.9104, + }, + ), + AxisPoint( + label="60N_15km", + assignments={ + "site.latitude_deg": 60.0, + "site.temperature_k": 210.0, + "site.pressure_mbar": 120.0, + "site.h2o_ppmv": 3.1673, + }, + ), + AxisPoint( + label="30N_20km_213K", + assignments={ + "site.latitude_deg": 30.0, + "site.temperature_k": 213.0, + "site.pressure_mbar": 55.0, + "site.h2o_ppmv": 10.1834, + }, + ), + ), + ), + Axis( + name="background", + kind=AxisKind.LIST, # aerosol distribution and background SO2 co-vary + points=( + AxisPoint( + label="sabr330", + assignments={ + "background.aerosol": BackgroundAerosol.SABR_330, + "background.so2_pptv": 20.0, + }, + ), + AxisPoint( + label="sabr220", + assignments={ + "background.aerosol": BackgroundAerosol.SABR_220, + "background.so2_pptv": 20.0, + }, + ), + AxisPoint( + label="cesm", + assignments={ + "background.aerosol": BackgroundAerosol.CESM_G6, + "background.so2_pptv": 100.0, + }, + ), + ), + ), + Axis.over( + "dilution", + "dilution.regime", + { + "D1low": DilutionRegime.D1, + "D2med": DilutionRegime.D2, + "D3high": DilutionRegime.D3, + "burst": DilutionRegime.BURST, + "D5vhigh": DilutionRegime.D5, + }, + ), + Axis.over("sticking", "microphysics.condensation_alpha", {"a0p5": 0.5, "a1p0": 1.0}), + Axis.over( + "nucleation", + "microphysics.nucleation_rate_scale", + {"nuc0p01": 0.01, "nuc1": 1.0, "nuc100": 100.0}, + ), + Axis.over("coag", "microphysics.coag_kernel_scale", {"cg0p5": 0.5, "cg1": 1.0, "cg2": 2.0}), +) + +#: The case the golden tests key on, and its index in the ensemble's ordering +#: (site 0, background 1, dilution 1, sticking 1, nucleation 1, coag 1 under itertools.product). +GOLDEN_CASE_ID = "30N_20km__sabr220__D2med__a1p0__nuc1__cg1" +GOLDEN_CASE_INDEX = 121 + + +@pytest.mark.tier_a +def test_a_single_run_is_a_runset_with_no_axes() -> None: + """N = 1 goes through the same expansion as N = 810. There is no second code path.""" + runs = RunSet().expand() + assert len(runs) == 1 + assert runs[0].config == RunConfig() + assert runs[0].label == "" + assert runs[0].coordinates == {} + + +@pytest.mark.tier_a +def test_reproduces_the_paper_ensemble() -> None: + """810 cases, in the ensemble's order, with the ensemble's case IDs. + + Compared against the labels rather than against ``run_ensemble.all_cases()`` directly: importing + that module pulls in ``coupled`` and therefore JAX, which Tier A must stay clear of. The tokens + below are copied from ``run_ensemble.py:61-76``, so a divergence in either direction shows up. + """ + runset = RunSet(axes=PAPER_AXES) + assert runset.size() == 810 == 3 * 3 * 5 * 2 * 3 * 3 + runs = runset.expand() + assert len(runs) == runset.size(), "size() must agree with expand() without building anything" + + assert runs[0].label == "30N_20km__sabr330__D1low__a0p5__nuc0p01__cg0p5" + assert runs[-1].label == "30N_20km_213K__cesm__D5vhigh__a1p0__nuc100__cg2" + assert len({run.label for run in runs}) == 810, "case IDs must be unique" + + golden = runs[GOLDEN_CASE_INDEX] + assert golden.label == GOLDEN_CASE_ID + assert golden.coordinates == { + "site": "30N_20km", + "background": "sabr220", + "dilution": "D2med", + "sticking": "a1p0", + "nucleation": "nuc1", + "coag": "cg1", + } + + +@pytest.mark.tier_a +def test_the_golden_case_resolves_to_the_ensembles_values() -> None: + """Spot-check the resolved config against ``run_ensemble.build_scenario`` for the golden case. + + This is not the equivalence proof -- that is task 0.4, field-for-field against a real + ``CoupledScenario``. It is the cheap version that catches an axis wired to the wrong path now, + rather than after the model seam exists. + """ + config = RunSet(axes=PAPER_AXES).expand()[GOLDEN_CASE_INDEX].config + assert config.site.latitude_deg == 30.0 + assert config.site.temperature_k == 210.0 + assert config.site.pressure_mbar == 55.0 + assert config.site.h2o_ppmv == 6.9104 + assert config.background.aerosol is BackgroundAerosol.SABR_220 + assert config.background.so2_pptv == 20.0 + assert config.dilution.regime is DilutionRegime.D2 + assert config.microphysics.condensation_alpha == 1.0 + assert config.microphysics.nucleation_rate_scale == 1.0 + assert config.microphysics.coag_kernel_scale == 1.0 + # unswept values stay at the ensemble's fixed configuration + assert config.microphysics.n_bins == 80 + assert config.microphysics.ion_pair_rate == 30.0 + assert config.schedule.day_of_year == 172 + assert config.chemistry.so2_ho2_rate == 1e-18 + assert config.switches.aerosol_to_j is False + assert config.switches.heating_to_t is False + + +@pytest.mark.tier_a +def test_expanded_runs_have_distinct_identities() -> None: + """Different parameters, different hashes -- on a real sweep, not a two-element toy.""" + runs = RunSet( + axes=( + Axis.over( + "nuc", "microphysics.nucleation_rate_scale", {"lo": 0.01, "mid": 1.0, "hi": 100.0} + ), + Axis.over("coag", "microphysics.coag_kernel_scale", {"a": 0.5, "b": 1.0, "c": 2.0}), + ) + ).expand() + assert len({run.config_hash for run in runs}) == len(runs) == 9 + + +@pytest.mark.tier_a +def test_grid_order_varies_the_last_axis_fastest() -> None: + """Ordering is part of the contract: it is what makes an expansion reproducible.""" + runs = RunSet( + axes=( + Axis.over("a", "microphysics.condensation_alpha", {"a1": 0.5, "a2": 1.0}), + Axis.over("b", "microphysics.coag_kernel_scale", {"b1": 0.5, "b2": 1.0, "b3": 2.0}), + ) + ).expand() + assert [run.label for run in runs] == [ + "a1__b1", + "a1__b2", + "a1__b3", + "a2__b1", + "a2__b2", + "a2__b3", + ] + + +@pytest.mark.tier_a +def test_zip_axes_advance_in_lockstep_and_cross_with_the_grid() -> None: + """ZIP pairs values instead of crossing them; the pair is then crossed with GRID axes.""" + runs = RunSet( + axes=( + Axis.over( + "site_t", "site.temperature_k", {"cold": 210.0, "warm": 213.0}, kind=AxisKind.ZIP + ), + Axis.over( + "site_p", "site.pressure_mbar", {"low": 55.0, "high": 120.0}, kind=AxisKind.ZIP + ), + Axis.over("nuc", "microphysics.nucleation_rate_scale", {"n1": 1.0, "n2": 100.0}), + ) + ).expand() + assert [run.label for run in runs] == [ + "cold__low__n1", + "cold__low__n2", + "warm__high__n1", + "warm__high__n2", + ] + assert (runs[0].config.site.temperature_k, runs[0].config.site.pressure_mbar) == (210.0, 55.0) + assert (runs[2].config.site.temperature_k, runs[2].config.site.pressure_mbar) == (213.0, 120.0) + + +@pytest.mark.tier_a +def test_zip_axes_of_unequal_length_are_rejected() -> None: + """Silently truncating to the shorter axis would drop runs the user asked for.""" + with pytest.raises(ValueError, match="equal length"): + RunSet( + axes=( + Axis.over("t", "site.temperature_k", {"a": 210.0, "b": 213.0}, kind=AxisKind.ZIP), + Axis.over("p", "site.pressure_mbar", {"x": 55.0}, kind=AxisKind.ZIP), + ) + ) + + +@pytest.mark.tier_a +def test_two_axes_assigning_the_same_field_are_rejected() -> None: + """The result would depend on axis order, so it is refused rather than silently ordered.""" + with pytest.raises(ValueError, match="both assign"): + RunSet( + axes=( + Axis.over("a", "microphysics.n_bins", {"lo": 40}), + Axis.over("b", "microphysics.n_bins", {"hi": 80}), + ) + ) + + +@pytest.mark.tier_a +def test_a_grid_axis_may_not_covary_two_fields() -> None: + """Covariation is what LIST is for; allowing it on GRID would make the kind meaningless.""" + with pytest.raises(ValueError, match="LIST"): + Axis( + name="site", + kind=AxisKind.GRID, + points=( + AxisPoint( + label="a", assignments={"site.temperature_k": 210.0, "site.pressure_mbar": 55.0} + ), + ), + ) + + +@pytest.mark.tier_a +def test_duplicate_point_labels_are_rejected() -> None: + """Labels become directory names; two runs cannot share one.""" + with pytest.raises(ValueError, match="duplicate point labels"): + Axis( + name="nuc", + points=( + AxisPoint(label="x", assignments={"microphysics.nucleation_rate_scale": 1.0}), + AxisPoint(label="x", assignments={"microphysics.nucleation_rate_scale": 2.0}), + ), + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("path", "message"), + [ + ("microphysics.n_bin", "unknown field"), + ("microphysics", "is a group of fields"), # a group has no unit, provenance or DAG node + ("site.temperature_k.value", "leaf field"), + ("nonexistent.thing", "unknown field"), + ], +) +def test_bad_axis_paths_are_rejected_at_construction(path: str, message: str) -> None: + """A typo'd path would otherwise produce a sweep whose axis silently never varied.""" + with pytest.raises(ValueError, match=message): + Axis(name="typo", points=(AxisPoint(label="x", assignments={path: 1.0}),)) + + +@pytest.mark.tier_a +def test_axis_values_are_validated_at_expansion_time() -> None: + """Fail on run 1 of 810, not on run 407. + + An out-of-range level is a mistake in the sweep definition; discovering it hours in, after + compute has been spent, is the expensive way to find out. + """ + runset = RunSet( + axes=(Axis.over("alpha", "microphysics.condensation_alpha", {"ok": 1.0, "bad": 1.5}),) + ) + with pytest.raises(ValueError, match="condensation_alpha"): + runset.expand() + + +@pytest.mark.tier_a +def test_apply_assignments_leaves_the_base_untouched() -> None: + """Configs are frozen and expansion must not alias them (ADR-004).""" + base = RunConfig() + changed = apply_assignments(base, {"microphysics.n_bins": 40}) + assert changed.microphysics.n_bins == 40 + assert base.microphysics.n_bins == 80 + assert changed.config_hash() != base.config_hash() + + +@pytest.mark.tier_a +def test_cross_field_rules_still_apply_to_swept_values() -> None: + """A sweep cannot slip past validation that a hand-written config would hit. + + ``couple_dt_s`` must divide ``output_dt_s`` (mirroring coupled_scenario.py:200-204), including + when it arrives from an axis. + """ + runset = RunSet(axes=(Axis.over("dt", "numerics.couple_dt_s", {"bad": 450.0}),)) + with pytest.raises(ValueError, match="integer multiple"): + runset.expand() diff --git a/studio/tests/unit/test_schema_export.py b/studio/tests/unit/test_schema_export.py new file mode 100644 index 0000000..fa6fa68 --- /dev/null +++ b/studio/tests/unit/test_schema_export.py @@ -0,0 +1,150 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""JSON Schema export and round-tripping. + +The export is the contract with the web client (ADR-002): the form is generated from it, so a field +that loses its metadata on the way out is a field the UI cannot explain. Round-tripping is the +contract with everything else -- the CLI reads YAML, the API reads JSON, the runner writes the +resolved config next to the results, and all three must reconstruct the same object with the same +hash. +""" + +from __future__ import annotations + +import json + +import pytest + +from studio.schema import ( + EXTENSION_KEY, + SCHEMA_ID, + SCHEMA_VERSION, + RunConfig, + config_hash, + field_catalogue, + run_config_json_schema, +) + + +@pytest.mark.tier_a +def test_export_is_json_serialisable_and_identifies_itself() -> None: + """A client must be able to tell which schema version it is holding.""" + schema = run_config_json_schema() + json.dumps(schema) # raises if anything in the export is not JSON + assert schema["$id"] == SCHEMA_ID + assert SCHEMA_VERSION in schema["$id"] + assert schema["x-studio-schema-version"] == SCHEMA_VERSION + assert schema["$schema"].startswith("https://json-schema.org/") + + +@pytest.mark.tier_a +def test_metadata_survives_the_export() -> None: + """Unit, provenance and source must reach the client, or the form cannot explain a field.""" + schema = run_config_json_schema() + site = schema["$defs"]["Site"]["properties"] + temperature = site["temperature_k"][EXTENSION_KEY] + assert temperature["unit"] == "K" + assert temperature["provenance"] == "paper_ensemble" + assert "TABLE_microphysics_parameters.md" in temperature["source"] + assert temperature["range"] == {"gt": 0.0} + assert site["temperature_k"]["description"] + + +@pytest.mark.tier_a +def test_every_leaf_field_appears_in_the_catalogue_with_a_default() -> None: + """``RunConfig()`` must be constructible with no arguments -- it is the form's opening state. + + The only fields without a default are the derived ones, which are unresolved by design. + """ + catalogue = field_catalogue() + assert len(catalogue) >= 40 + missing_default = [path for path, meta in catalogue.items() if "default" not in meta] + assert missing_default == [], f"fields with no default: {missing_default}" + derived_unset = [ + path + for path, meta in catalogue.items() + if meta["provenance"] == "derived" and meta["default"] is not None + ] + assert derived_unset == [] + + +@pytest.mark.tier_a +def test_catalogue_paths_are_the_same_paths_runset_axes_use() -> None: + """One vocabulary. A path read from the schema must be usable as an axis path unchanged.""" + from studio.schema import resolve_path + + for path in field_catalogue(): + resolve_path(RunConfig, path) + + +@pytest.mark.tier_a +def test_json_round_trip_preserves_identity() -> None: + """Serialise, parse, revalidate: same object, same hash.""" + original = RunConfig() + restored = RunConfig.model_validate_json(original.model_dump_json()) + assert restored == original + assert config_hash(restored) == config_hash(original) + + +@pytest.mark.tier_a +def test_round_trip_survives_a_non_default_config() -> None: + """Defaults round-tripping proves little; a config with every group altered proves more.""" + original = RunConfig.model_validate( + { + "site": { + "latitude_deg": -60.0, + "longitude_deg": 175.0, + "temperature_k": 213.0, + "pressure_mbar": 120.0, + "h2o_ppmv": 3.1673, + }, + "schedule": {"day_of_year": 355, "start_utc_hour": 13.5, "duration_days": 60}, + "injection": {"so2_mass_kg": 2500.0, "plume_length_m": 30000.0}, + "background": {"aerosol": "cesm_g6", "so2_pptv": 100.0, "gas_pptv": {"O3": 1.2e6}}, + "dilution": {"regime": "constant", "rate_per_s": 2.0e-6, "zero_species": ("SO2",)}, + "microphysics": {"n_bins": 160, "condensation_alpha": 0.5, "ion_pair_rate": 0.0}, + "chemistry": {"photolysis": "sza", "so2_ho2_rate": 1e-16}, + "numerics": {"output_dt_s": 1200.0, "couple_dt_s": 300.0}, + "switches": {"heating_to_t": True}, + "termination": {"max_wall_time_s": 60.0, "max_sim_time_days": 5.0}, + } + ) + restored = RunConfig.model_validate_json(original.model_dump_json()) + assert restored == original + assert config_hash(restored) == config_hash(original) + assert config_hash(restored) != config_hash(RunConfig()) + + +@pytest.mark.tier_a +def test_unknown_fields_are_rejected() -> None: + """A typo'd key is a config that does not describe the run; accepting it silently is worse.""" + payload = RunConfig().model_dump() + payload["site"]["temprature_k"] = 210.0 + with pytest.raises(ValueError, match="temprature_k"): + RunConfig.model_validate(payload) + + +@pytest.mark.tier_a +def test_configs_are_immutable() -> None: + """Identity is a hash of the content, so content that can change under it is a bug (ADR-004).""" + config = RunConfig() + with pytest.raises(ValueError, match="frozen"): + config.site.temperature_k = 250.0 # type: ignore[misc] + + +@pytest.mark.tier_a +def test_background_evolves_accepts_only_false() -> None: + """SCIENCE-5: the model's background is static, so True must fail rather than be ignored.""" + payload = RunConfig().model_dump() + payload["dilution"]["background_evolves"] = True + with pytest.raises(ValueError, match="background_evolves"): + RunConfig.model_validate(payload) + + +@pytest.mark.tier_a +def test_bin_count_is_restricted_to_the_grids_the_model_has() -> None: + """40/80/160 are the only TOMAS grids; 100 must fail here, not inside tomas_bridge.""" + payload = RunConfig().model_dump() + payload["microphysics"]["n_bins"] = 100 + with pytest.raises(ValueError, match="n_bins"): + RunConfig.model_validate(payload) diff --git a/studio/tests/unit/test_schema_metadata.py b/studio/tests/unit/test_schema_metadata.py new file mode 100644 index 0000000..18605db --- /dev/null +++ b/studio/tests/unit/test_schema_metadata.py @@ -0,0 +1,263 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Field metadata is a guarantee, not an intention. + +The PR checklist says "new/changed schema fields carry unit, range, description, default, +provenance". A checklist is a request for attention; this module makes it a build failure, so a +field added in six months without a recorded source cannot merge. + +The provenance rules themselves are enforced inside ``SciField`` at import time. What is tested here +is that they FIRE -- a validation rule nobody has ever seen reject anything is a rule you do not +know works. +""" + +from __future__ import annotations + +import pytest +from pint import UndefinedUnitError, UnitRegistry + +from studio.schema import ( + PINT_EXPRESSION, + Provenance, + RunConfig, + SciField, + Unit, + field_catalogue, + iter_leaf_fields, +) +from studio.schema.fields import field_metadata + +CATALOGUE = field_catalogue() + +#: Units carrying a convention pint cannot express. Kept here, spelled out, rather than as "anything +#: mapping to None" -- so that mapping a unit to None BY MISTAKE fails this test instead of joining +#: an exemption list silently. See studio/schema/units.py for why each one is here. +NON_PINT_UNITS = {Unit.PPMV, Unit.PPTV, Unit.CM3_PER_MOLEC_PER_S, Unit.COUNT} + + +@pytest.mark.tier_a +def test_every_leaf_field_has_complete_metadata() -> None: + """Unit, description and provenance on every field; no exceptions, no exemption list.""" + incomplete = {} + for path, info in iter_leaf_fields(): + meta = field_metadata(info) + missing = [key for key in ("unit", "provenance") if not meta.get(key)] + if not info.description: + missing.append("description") + if missing: + incomplete[path] = missing + assert incomplete == {}, ( + f"fields missing metadata: {incomplete}. Every schema field is declared with SciField, " + f"which requires unit, description and provenance -- see studio/schema/fields.py." + ) + + +@pytest.mark.tier_a +def test_declared_units_are_in_the_canonical_registry() -> None: + """A unit string outside the registry is a typo; the registry is closed on purpose (ADR-003).""" + known = {unit.value for unit in Unit} + unknown = {path: meta["unit"] for path, meta in CATALOGUE.items() if meta["unit"] not in known} + assert unknown == {}, f"units outside the canonical registry: {unknown}; known: {sorted(known)}" + + +@pytest.mark.tier_a +def test_pint_parses_every_unit_that_claims_to_be_parseable() -> None: + """The registry's pint expressions must actually parse, and the exemptions must be deliberate. + + pint is a declared dependency precisely so display conversion is possible; an expression that + does not parse would only be discovered at the presentation boundary, in front of a user. + """ + registry = UnitRegistry() + assert set(PINT_EXPRESSION) == set(Unit), ( + "every Unit member needs an entry in PINT_EXPRESSION (a pint expression, or None with a " + "stated reason); missing: " + f"{sorted(u.value for u in set(Unit) - set(PINT_EXPRESSION))}" + ) + for unit, expression in PINT_EXPRESSION.items(): + if expression is None: + assert unit in NON_PINT_UNITS, ( + f"{unit.value} maps to None but is not one of the documented non-pint units " + f"{sorted(u.value for u in NON_PINT_UNITS)}. If it genuinely cannot be expressed, " + f"say why in units.py and add it there." + ) + continue + try: + registry.Unit(expression) + except UndefinedUnitError as exc: # pragma: no cover -- the failure path is the point + pytest.fail(f"pint cannot parse {expression!r} for unit {unit.value!r}: {exc}") + + +@pytest.mark.tier_a +def test_sourced_provenance_carries_a_source() -> None: + """MODEL_DEFAULT and PAPER_ENSEMBLE mean "traceable"; without a source they mean nothing.""" + unsourced = { + path: meta["provenance"] + for path, meta in CATALOGUE.items() + if meta["provenance"] in {Provenance.MODEL_DEFAULT.value, Provenance.PAPER_ENSEMBLE.value} + and not meta.get("source") + } + assert unsourced == {}, f"defaults claiming a source but not giving one: {unsourced}" + + +@pytest.mark.tier_a +def test_derived_fields_declare_their_inputs_and_stay_unresolved() -> None: + """Derived fields are declarations, not computations (task 0.3 resolves them, 0.5 derives them). + + A derived field arriving with a value would mean physics happened in the schema layer, which is + the single thing studio/CLAUDE.md is most emphatic about. + """ + derived = {p: m for p, m in CATALOGUE.items() if m["provenance"] == Provenance.DERIVED.value} + assert derived, "expected at least the V0 and initial-concentration derivations to be declared" + for path, meta in derived.items(): + assert meta["derived_from"], f"{path} is DERIVED but declares no inputs" + assert meta.get("default") is None, ( + f"{path} is DERIVED but ships a value ({meta['default']!r}); it must stay unresolved " + f"until the dependency-graph engine computes it" + ) + for source_path in meta["derived_from"]: + assert source_path in CATALOGUE, ( + f"{path} derives from {source_path!r}, which is not a field. The dependency graph " + f"in task 0.3 is built from these paths, so a stale one silently drops an edge." + ) + + +@pytest.mark.tier_a +def test_non_derived_fields_declare_no_inputs() -> None: + """``derived_from`` on a primary field would put a phantom edge in the task-0.3 DAG.""" + strays = { + path: meta["derived_from"] + for path, meta in CATALOGUE.items() + if meta["provenance"] != Provenance.DERIVED.value and meta["derived_from"] + } + assert strays == {}, f"non-derived fields declaring derived_from: {strays}" + + +@pytest.mark.tier_a +def test_bounded_quantities_declare_their_range() -> None: + """Physical quantities that cannot take any float must say so, and the bound must be enforced. + + Checked against a hand-listed set rather than "all floats": some quantities genuinely are + unbounded, and a test that demanded bounds everywhere would be satisfied by meaningless ones. + """ + must_be_bounded = { + "site.latitude_deg", + "site.longitude_deg", + "site.temperature_k", + "site.pressure_mbar", + "site.h2o_ppmv", + "schedule.day_of_year", + "schedule.start_utc_hour", + "schedule.duration_days", + "injection.so2_mass_kg", + "microphysics.condensation_alpha", + "microphysics.nucleation_rate_scale", + "microphysics.ion_pair_rate", + "chemistry.so2_ho2_rate", + "numerics.output_dt_s", + "numerics.couple_dt_s", + "termination.max_wall_time_s", + } + unbounded = {path for path in must_be_bounded if not CATALOGUE[path].get("range")} + assert unbounded == set(), f"quantities with no declared range: {sorted(unbounded)}" + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("path", "value"), + [ + ("site.latitude_deg", 91.0), + ("site.temperature_k", 0.0), + ("site.pressure_mbar", -1.0), + ("schedule.day_of_year", 367), + ("schedule.start_utc_hour", 24.0), + ("microphysics.condensation_alpha", 1.5), + ("microphysics.nucleation_rate_scale", -1.0), + ("chemistry.so2_ho2_rate", -1e-18), + ], +) +def test_declared_ranges_are_actually_enforced(path: str, value: float) -> None: + """A declared range that pydantic does not enforce is documentation, not validation.""" + group, field = path.split(".") + base = RunConfig() + payload = base.model_dump() + payload[group][field] = value + with pytest.raises(ValueError, match=field): + RunConfig.model_validate(payload) + + +@pytest.mark.tier_a +def test_caveats_survive_into_the_catalogue() -> None: + """A caveat exists to reach the user; losing it in export would defeat the point. + + The heating switch is the case that matters most: shortwave-only heating produces a one-sided + warm drift, and enabling it without that warning is how someone reports a temperature trend as + a result. + """ + assert "one-sided" in CATALOGUE["switches.heating_to_t"]["caveat"] + assert "UPPER LIMIT" in CATALOGUE["chemistry.so2_ho2_rate"]["caveat"] + assert "spun-up" in CATALOGUE["background.gas_pptv"]["caveat"].lower() + + +@pytest.mark.tier_a +class TestSciFieldRejectsInconsistentMetadata: + """The import-time rules in ``SciField``, exercised. Each of these once looked reasonable.""" + + def test_literature_without_citation(self) -> None: + with pytest.raises(ValueError, match="requires `cite`"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.LITERATURE, + default=1.0, + ) + + def test_model_default_without_source(self) -> None: + with pytest.raises(ValueError, match="requires `source`"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.MODEL_DEFAULT, + default=1.0, + ) + + def test_derived_without_inputs(self) -> None: + with pytest.raises(ValueError, match="requires `derived_from`"): + SciField(unit=Unit.KELVIN, description="x", provenance=Provenance.DERIVED) + + def test_derived_with_a_hand_written_default(self) -> None: + with pytest.raises(ValueError, match="must not carry a hand-written default"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.DERIVED, + derived_from=["site.temperature_k"], + default=210.0, + ) + + def test_derived_from_on_a_primary_field(self) -> None: + with pytest.raises(ValueError, match="only meaningful with provenance=DERIVED"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.CONVENTION, + derived_from=["site.temperature_k"], + default=1.0, + ) + + def test_user_required_with_a_default(self) -> None: + with pytest.raises(ValueError, match="must not have one"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.USER_REQUIRED, + default=1.0, + ) + + def test_blank_description(self) -> None: + with pytest.raises(ValueError, match="description is required"): + SciField( + unit=Unit.KELVIN, + description=" ", + provenance=Provenance.CONVENTION, + default=1.0, + ) From 39ca343db80df19b48da596d090224d0ef7c9395 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 18:35:39 -0700 Subject: [PATCH 03/18] =?UTF-8?q?studio:=20science=20derivations=20?= =?UTF-8?q?=E2=80=94=20plume,=20size=20distribution,=20air=20density,=20GC?= =?UTF-8?q?R=20(#65)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit studio: science derivations -- plume, size distribution, air density, GCR Task 0.5 (#64), taken before 0.3 so the dependency-graph engine has real derivations to resolve. studio/science/ in five modules, 40 new Tier-A tests (101 total). Consolidation first, new code second -- and the consolidation corrected two claims the plan made from memory. Both are now fixed in plan/PHASE_0.md: There are SIX copies of the V0 / initial-concentration derivation, not five (the missed one is make_rf_runs.py:44), and run_dilution_d1_clean.py lives at coupled/, not coupled/paper_ensemble/. The divergence between them is real and consequential: the ensemble uses a 15 km track and the D1 flagship a 30 km one, so the same injected mass gives half the concentration. Which is right depends on SCIENCE-2 (#54). Separately, run_60day.py:37's hard-coded 6.273063291666667e15 is bit-identical to what run_ensemble.py:46 computes -- a frozen copy, not a variant. The "two different mid-point expressions" are one expression. 10**(0.5*(log a + log b)) and sqrt(a*b) are algebraically identical, as are log b - log a and log(b/a); measured agreement on an 80-bin grid is to a few ULP (<= 7e-16 relative). The test measures this rather than asserting it, because "there are two conventions in the code" is the kind of belief people work around. air_number_density is a MIRROR of the model's, not a fork: studio.science may not import the model (ADR-001), so the test runs the model's own implementation in a subprocess and asserts exact agreement at the four T-p corners the runs use. Note the asymmetry -- CI does not check out the private submodules, so that check skips there and only really runs on a developer machine. gcr.py computes nothing. ion_pair_production_rate raises NotImplementedError naming SCIENCE-6 (#63), and PAPER_ENSEMBLE_ION_PAIR_RATE = 30.0 carries its provenance instead. A test asserts it refuses even at ~20 km / 30N, where the uncited 30.0 came from: returning the known value at the known point and raising elsewhere would look like a working function with gaps. Two molar masses are deliberately the model's rounded values (SO2 64.0 vs 64.066, H2SO4 98.0 vs 98.079), recorded as such in constants.py along with the ~0.036% Avogadro seam at the gas/TOMAS boundary. Studio inherits the model's constants so Phase 0 can reproduce its runs; a silent correction would make a Studio bug indistinguishable from a model change. Studio's Python floor is now stated as 3.12 in ruff, black and mypy: numpy's bundled stubs use 3.12-only type statements, so mypy --strict could not check this package against 3.11 at all. The lockfile and CI were already 3.12 and no dependency changed. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/OPEN_QUESTIONS.md | 21 +++ docs/studio/PROGRESS.md | 61 ++++++- docs/studio/plan/PHASE_0.md | 16 +- pyproject.toml | 9 +- studio/science/__init__.py | 93 ++++++++--- studio/science/air.py | 38 +++++ studio/science/constants.py | 83 +++++++++ studio/science/gcr.py | 77 +++++++++ studio/science/plume.py | 140 ++++++++++++++++ studio/science/size_distribution.py | 98 +++++++++++ studio/tests/unit/test_science_air.py | 101 +++++++++++ studio/tests/unit/test_science_gcr.py | 67 ++++++++ studio/tests/unit/test_science_plume.py | 158 ++++++++++++++++++ .../unit/test_science_size_distribution.py | 123 ++++++++++++++ 14 files changed, 1052 insertions(+), 33 deletions(-) create mode 100644 studio/science/air.py create mode 100644 studio/science/constants.py create mode 100644 studio/science/gcr.py create mode 100644 studio/science/plume.py create mode 100644 studio/science/size_distribution.py create mode 100644 studio/tests/unit/test_science_air.py create mode 100644 studio/tests/unit/test_science_gcr.py create mode 100644 studio/tests/unit/test_science_plume.py create mode 100644 studio/tests/unit/test_science_size_distribution.py diff --git a/docs/studio/OPEN_QUESTIONS.md b/docs/studio/OPEN_QUESTIONS.md index 47765ba..55db52b 100644 --- a/docs/studio/OPEN_QUESTIONS.md +++ b/docs/studio/OPEN_QUESTIONS.md @@ -204,6 +204,27 @@ required. Until then `background_evolves` is a schema field whose only accepted --- +### SCIENCE-6 — GCR ion-pair production rate has no derivation · **OPEN** · Phase 0/4 · [#63](https://github.com/reflective-org/SANDBOX/issues/63) +*What is the ion-pair production rate as a function of altitude, latitude and solar-cycle phase?* + +Raised by task 0.5. The two values available in the repository are an **uncited constant** and a +value that **switches off a physical process**: the paper ensemble uses a bare `30.0` cm⁻³ s⁻¹ +(`run_ensemble.py:102`, described in `TABLE_microphysics_parameters.md` as "galactic cosmic rays at +~20 km"), and the model defaults to `0.0`, which disables ion-induced nucleation entirely +(`coupled/coupled_scenario.py:117`). + +It feeds the ion-induced channels of Dunne et al. (2016) nucleation — the most sensitive part of this +system. GCR ionisation varies by roughly a factor of two over the solar cycle and strongly with +latitude and altitude, so one number is wrong nearly everywhere except where it was read off. + +`studio/science/gcr.py` therefore raises `NotImplementedError` rather than interpolating an uncited +number, and exposes `PAPER_ENSEMBLE_ION_PAIR_RATE = 30.0` as a constant with its provenance attached. + +**Answered when** either a citable parameterisation is agreed and implemented with its reference, or +the decision is recorded that the fixed value stands, with its sensitivity quantified. + +--- + ## Register of capabilities the spec assumes but the model does not have Not open questions — settled facts, listed here because the spec's stage descriptions imply diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index c882616..f2d0be6 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -15,20 +15,73 @@ with full provenance, and the golden tests pass. | Task | Status | |---|---| | 0.1 Repo, CI, docs skeleton | **done** | -| 0.2 `studio/schema` v0 — **review gate** | **awaiting review** (#61) | +| 0.2 `studio/schema` v0 — **review gate** | **done** (#62, reviewed) | | 0.3 Dependency-graph engine + override semantics | not started | | 0.4 `studio/modelio` seam + `RunSummary` | not started | -| 0.5 `studio/science` derivations | not started | +| 0.5 `studio/science` derivations | **done** (#64) | | 0.6 `studio/runner` + job lifecycle | not started | | 0.7 Golden-file harness (two tiers) | not started | | 0.8 Four contained fixes in `coupled/` | not started | | 0.9 Vertical slice: CLI + API + minimal UI | not started | -Nothing is built on top of `studio/schema` until 0.2 is reviewed and merged. +Task order note: 0.5 was taken **before 0.3**, so the dependency-graph engine has real +derivations to resolve rather than fixtures. --- -### 2026-08-13 — Task 0.2: `studio/schema` v0 · **awaiting review** (issue #61) +### 2026-08-13 — Task 0.5: `studio/science` derivations (issue #64) + +Taken **before 0.3** at Ali's direction, so the dependency-graph engine has real derivations to +resolve rather than fixtures. + +**Added** — `studio/science/`: `constants.py`, `air.py`, `plume.py`, `size_distribution.py`, +`gcr.py`. 40 new Tier-A tests (101 total). + +**What the consolidation actually found.** The plan described this task from memory, and two of its +claims did not survive contact with the code. Both are corrected in `plan/PHASE_0.md`: + +- **Six copies of the V₀ / initial-concentration derivation, not five** — the missed one is + `make_rf_runs.py:44` — and `run_dilution_d1_clean.py` is at `coupled/`, not + `coupled/paper_ensemble/`. The divergence is real and it matters: `run_ensemble.py` uses a **15 km** + track, the D1 flagship a **30 km** one. Same injected mass, half the concentration. Which is right + depends on SCIENCE-2 (#54). Also verified: `run_60day.py:37`'s hard-coded `6.273063291666667e15` + is **bit-identical** to what `run_ensemble.py:46` computes, so it is a frozen copy, not a variant. +- **The "two different mid-point expressions" are one expression.** + `10**(0.5*(log a + log b))` and `sqrt(a*b)` are algebraically identical, as are + `log b - log a` and `log(b/a)`. Measured difference on an 80-bin grid: ≤ 7e-16 (mid-point) and + ≤ 5e-15 (dlog10Dp) relative — a few ULP of float64. Consolidating is still worth doing; believing + there were two conventions was not. `test_the_repositorys_two_spellings_are_the_same_quantity` + measures it rather than asserting it, because that belief would otherwise get worked around. + +**Decisions** + +- **`air_number_density` is a MIRROR, not a fork.** `studio.science` may not import the model + (ADR-001), so this one relation is duplicated — and `test_science_air.py` runs the model's own + implementation in a subprocess and asserts **exact** agreement at the four T–p corners the runs + use. That is what makes a duplicate acceptable. Note the asymmetry: CI does not check out the + private submodules, so this check *skips* in CI and only really runs on a developer machine. +- **`gcr.py` computes nothing.** `ion_pair_production_rate` raises `NotImplementedError` naming + SCIENCE-6; `PAPER_ENSEMBLE_ION_PAIR_RATE = 30.0` is available as a constant with its provenance. + A test asserts it refuses **even at ~20 km / 30°N**, where the uncited 30.0 came from — returning + the known value at the known point and raising elsewhere is the most tempting version of this + mistake, because it looks like a working function with gaps. +- **Two constants are deliberately the model's rounded values**, recorded as such in `constants.py`: + SO₂ at 64.0 g/mol (true 64.066, a 0.10 % difference) and H₂SO₄ at 98.0 (true 98.079). Studio + inherits them so Phase 0 reproduces the golden runs; a silent correction would shift every derived + initial concentration and make a Studio bug indistinguishable from a model change. The ~0.036 % + Avogadro seam at the gas/TOMAS boundary is likewise recorded (`AVOGADRO_GAS_MODEL`) and not used. +- **Reused, not rewritten**: `coupled.dilution.volume_ratio` / `kdil_from_regime`, + `coupled.aerosol_props`, `coupled.units`. They are already tested in the model, and Studio reaches + them through `studio/modelio` rather than keeping a second copy. + +**Toolchain** — Studio's Python floor is now stated as **3.12** in all three tools. numpy's bundled +type stubs use 3.12-only `type` statements, so `mypy --strict` could not check `studio/science` +against 3.11 at all; the lockfile and CI were already 3.12. No dependency changed, so the lockfile +is untouched. + +--- + +### 2026-08-13 — Task 0.2: `studio/schema` v0 · **merged** (#62) The review gate. Nothing is built on top of this until it is reviewed and merged. diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index 7341c94..4b54f53 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -83,16 +83,20 @@ construction. It is a data-model problem, not a UI problem. --- -## 0.5 — `studio/science` derivations +## 0.5 — `studio/science` derivations · *done* (issue #64) One cited, tested implementation of each derivation that currently exists several times over. **Consolidate:** -- V₀ and mass → initial concentration — five copies today (`run_ensemble.py:41-46`, - `run_dilution_d1_clean.py:61`, `run_60day.py:37`, `viz/bake_plume_dynamics.py:63`, `run_boxsize.py`), - with two different V₀ values. -- dN/dlogDp and bin edges — four copies, two mid-point expressions - (`run_ensemble.py:146-149` vs `run_dilution_d1_clean.py:588-590`). +- V₀ and mass → initial concentration — **six** copies today, not five: `run_ensemble.py:41-46,95`, + `run_60day.py:37`, `make_rf_runs.py:44`, `viz/bake_plume_dynamics.py:62-63`, + `coupled/run_dilution_d1_clean.py:61,130` (note: at `coupled/`, not `coupled/paper_ensemble/`), + and `run_boxsize.py:43`. Two different V₀ values — a 15 km track and a 30 km one, a factor of two. +- dN/dlogDp and bin edges — four copies, written two ways. **Corrected by 0.5:** the two + expressions (`run_ensemble.py:147-149` vs `coupled/run_dilution_d1_clean.py:587-589`) are + *algebraically identical*, not two conventions — `10**(0.5*(log a + log b)) == sqrt(a*b)` — and + measured agreement on an 80-bin grid is to a few ULP (≤ 7e-16 relative). The consolidation is + still worth doing; the divergence claim was not accurate. **Reuse, do not rewrite:** `coupled/dilution.py:62` `volume_ratio` and `:74` `kdil_from_regime` (both tested); `air_number_density` (`stratchem-jax/config.py:55`); `coupled/aerosol_props.py`; diff --git a/pyproject.toml b/pyproject.toml index db272af..df9b25f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -79,7 +79,10 @@ markers = [ # ---------------------------------------------------------------------------------------------- [tool.ruff] line-length = 100 -target-version = "py311" +# Studio's floor is 3.12, not the model's >=3.10: the lockfile is compiled for 3.12, CI runs 3.12, +# and numpy's bundled type stubs use 3.12-only `type` statements, so mypy cannot check this package +# against anything older. All three tools state the same floor deliberately. +target-version = "py312" src = ["."] extend-exclude = ["studio/web"] @@ -101,11 +104,11 @@ ignore = ["ANN401"] [tool.black] line-length = 100 -target-version = ["py311"] +target-version = ["py312"] extend-exclude = "studio/web" [tool.mypy] -python_version = "3.11" +python_version = "3.12" # --strict is applied to studio/schema and studio/science on the command line and in CI; the settings # here are the baseline for everything else under studio/. files = ["studio"] diff --git a/studio/science/__init__.py b/studio/science/__init__.py index c67e544..4f13ee7 100644 --- a/studio/science/__init__.py +++ b/studio/science/__init__.py @@ -2,33 +2,86 @@ # SPDX-License-Identifier: Apache-2.0 """Scientific derivations: plume volume, initial concentration, size-distribution reductions, GCR. -Purpose is consolidation as much as new code. Several of these derivations already exist in the -repository three to five times over, with drifting constants -- one implementation each, cited and -tested (task 0.5): +Consolidation as much as new code. Each derivation below existed several times over in this +repository, and this is the one cited, tested implementation: -* V0 and injected mass -> initial concentration. Five copies today, with two different V0 values - (``run_ensemble.py:41-46`` uses a 15 km track, ``run_dilution_d1_clean.py:61`` a 30 km one). -* dN/dlogDp and bin diameter edges. Four copies, two mid-point expressions. +* **Plume volume and injected mass -> initial concentration** (``plume.py``). Six copies today, and + they disagree: the ensemble uses a 15 km track and the D1 flagship a 30 km one -- a factor of two + in V0 for the same injected mass. Which is right depends on what t = 0 means (SCIENCE-2, #54). +* **dN/dlogDp and bin mid-points** (``size_distribution.py``). Four copies, written two ways which + turn out to be algebraically identical -- measured agreement to a few ULP. +* **Air number density** (``air.py``). A deliberate MIRROR of the model's, because this package may + not import the model; a test asserts they agree exactly. +* **GCR ion-pair rate** (``gcr.py``). Raises ``NotImplementedError``: the value in use is uncited + and the honest thing is to say so (SCIENCE-6, #63). -Reused rather than rewritten -- these are already correct and tested: +**Reused, not rewritten.** These are already implemented and tested in the model, so Studio reaches +them through ``studio/modelio`` (the only package allowed to import ``coupled``) rather than keeping +a second copy here: -* ``coupled.dilution.volume_ratio`` / ``kdil_from_regime`` (plume expansion V(t)/V0 and k_dil) -* ``config.air_number_density`` (stratchem-jax) -* ``coupled.aerosol_props`` (surface area, effective wet radius, H2SO4 weight percent) -* ``coupled.units`` (mass <-> number density; note the DELIBERATE ~0.036% Avogadro mismatch at the - gas/TOMAS seam documented at ``coupled/units.py:18-22`` -- inherited, not silently "corrected") +* ``coupled.dilution.volume_ratio`` / ``kdil_from_regime`` -- plume expansion V(t)/V0 and k_dil +* ``coupled.aerosol_props`` -- surface area, effective wet radius, H2SO4 weight percent +* ``coupled.units`` -- number density <-> mass per grid cell -Genuinely new: a galactic-cosmic-ray ion-pair parameterisation. Today ``ion_pair_rate`` is a bare -30.0 with no derivation. The default should not be zero and should depend on altitude, latitude and -solar-cycle phase -- but absent an agreed citation it raises ``NotImplementedError`` rather than -returning a plausible number (ADR-005). - -No magic numbers: physical constants live in one module with sources, and any numeric literal in -scientific code needs a named constant and a citation. +Every physical constant lives in ``constants.py`` with its source, including the two that are +deliberately the model's rounded values rather than the best-known ones -- Studio inherits the +model's constants so that Phase 0 can reproduce its runs, and says so at the point of use. This package must not import ``coupled``, the API, or the database (see ``studio/__init__.py``). """ from __future__ import annotations -__all__: list[str] = [] +from studio.science.air import air_number_density +from studio.science.constants import ( + AIR_NUMBER_DENSITY_COEFF, + AVOGADRO, + AVOGADRO_GAS_MODEL, + AVOGADRO_SEAM_RELATIVE_DIFFERENCE, + CM3_PER_M3, + G_PER_KG, + H2SO4_MOLAR_MASS_G_PER_MOL, + MBAR_TO_TORR, + PPMV_PER_MOLE_FRACTION, + PPTV_PER_MOLE_FRACTION, + SO2_MOLAR_MASS_G_PER_MOL, +) +from studio.science.gcr import ( + MODEL_DEFAULT_ION_PAIR_RATE, + PAPER_ENSEMBLE_ION_PAIR_RATE, + ion_pair_production_rate, +) +from studio.science.plume import ( + initial_mixing_ratio_pptv, + injected_number_density, + number_density_to_pptv, + plume_volume_cm3, + pptv_to_number_density, +) +from studio.science.size_distribution import bin_midpoints_um, dlog10_dp, dn_dlogdp + +__all__ = [ + "AIR_NUMBER_DENSITY_COEFF", + "AVOGADRO", + "AVOGADRO_GAS_MODEL", + "AVOGADRO_SEAM_RELATIVE_DIFFERENCE", + "CM3_PER_M3", + "G_PER_KG", + "H2SO4_MOLAR_MASS_G_PER_MOL", + "MBAR_TO_TORR", + "MODEL_DEFAULT_ION_PAIR_RATE", + "PAPER_ENSEMBLE_ION_PAIR_RATE", + "PPMV_PER_MOLE_FRACTION", + "PPTV_PER_MOLE_FRACTION", + "SO2_MOLAR_MASS_G_PER_MOL", + "air_number_density", + "bin_midpoints_um", + "dlog10_dp", + "dn_dlogdp", + "initial_mixing_ratio_pptv", + "injected_number_density", + "ion_pair_production_rate", + "number_density_to_pptv", + "plume_volume_cm3", + "pptv_to_number_density", +] diff --git a/studio/science/air.py b/studio/science/air.py new file mode 100644 index 0000000..5877ab9 --- /dev/null +++ b/studio/science/air.py @@ -0,0 +1,38 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Air number density. + +This is the one function here that duplicates a model function rather than reusing it, and the +duplication is forced: ``studio.science`` may not import ``coupled`` or the model's flat modules +(ADR-001), because the packages must be usable from a bare Python session and importing the model +costs a JAX import. + +So it is a MIRROR, not a fork. It is one line, it cites the line it mirrors, and +``studio/tests/unit/test_science_air.py`` asserts that the two agree exactly whenever the +``stratchem-jax`` submodule is checked out. If the model's relation ever changes, that test fails -- +which is the property that makes a mirror acceptable and a quiet copy not. +""" + +from __future__ import annotations + +from studio.science.constants import AIR_NUMBER_DENSITY_COEFF, MBAR_TO_TORR + + +def air_number_density(pressure_mbar: float, temperature_k: float) -> float: + """Air number density M [molec cm^-3] at ``pressure_mbar`` and ``temperature_k``. + + Mirrors ``stratchem-jax/config.py:55`` (``M = 9.65e18*P/T*conv``), itself a port of the MATLAB + ``runconcs_het.m``. The coefficient folds in the mbar->torr conversion. + + Raises: + ValueError: On a non-positive pressure or temperature. There is no sensible fallback: a + zero temperature is a division by zero and a negative pressure is not a state (ADR-005). + """ + if pressure_mbar <= 0.0: + raise ValueError(f"pressure must be > 0 mbar, got {pressure_mbar}") + if temperature_k <= 0.0: + raise ValueError(f"temperature must be > 0 K, got {temperature_k}") + return AIR_NUMBER_DENSITY_COEFF * pressure_mbar / temperature_k * MBAR_TO_TORR + + +__all__ = ["air_number_density"] diff --git a/studio/science/constants.py b/studio/science/constants.py new file mode 100644 index 0000000..f5ac1ae --- /dev/null +++ b/studio/science/constants.py @@ -0,0 +1,83 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Physical constants, in one place, each with its source. + +``studio/CLAUDE.md``: *no magic numbers -- physical constants live in one module with sources, and +any numeric literal in scientific code needs a named constant and a citation.* + +Two of the values here are deliberately NOT their best-known values, and that is the interesting +part of this module. The model uses rounded constants in places, and Studio's job in Phase 0 is to +reproduce the model's existing runs exactly. A silent "correction" here would shift every derived +initial concentration by a small amount and make it impossible to tell a Studio bug from a model +change -- the same reasoning that made the canonical units the model's native ones (ADR-003). + +Each such value says so, in place, so the choice is visible at the point of use rather than +discoverable by whoever eventually diffs a result. +""" + +from __future__ import annotations + +from typing import Final + +#: Avogadro constant [molecules / mol]. CODATA 2019 exact value, and what ``coupled/units.py:23`` +#: uses so its gas<->TOMAS bridge is self-consistent with TOMAS. The paper ensemble uses this same +#: value for the injected-mass conversion (``run_ensemble.py:39``), so Studio must too. +AVOGADRO: Final = 6.02214076e23 + +#: The GAS-PHASE model's Avogadro constant -- rounded (``mechanism.khet``, +#: ``config.air_number_density``). Recorded, never used here. Molecule<->mole conversions therefore +#: differ by ~0.036 % between the two halves of the model. ``coupled/units.py:18-22`` documents this +#: as a conscious seam, not drift; Studio INHERITS it and does not silently reconcile it. Anything +#: that needs the gas side's convention must say so explicitly. +AVOGADRO_GAS_MODEL: Final = 6.02e23 + +#: Relative size of that seam, as a fraction. Stated as a number so a tolerance can cite it rather +#: than hard-coding "about 4e-4" somewhere downstream. +AVOGADRO_SEAM_RELATIVE_DIFFERENCE: Final = (AVOGADRO - AVOGADRO_GAS_MODEL) / AVOGADRO + +#: Molar mass of SO2 [g / mol] AS THE MODEL USES IT (``run_ensemble.py:44``). The true value is +#: 64.066 g/mol; the ensemble's 64.0 is a 0.10 % difference. Kept rounded because the golden runs +#: were produced with it -- see the module docstring. +SO2_MOLAR_MASS_G_PER_MOL: Final = 64.0 + +#: Molar mass of H2SO4 [g / mol] as the model uses it (``coupled/run_dilution_d1_clean.py:598``). +#: True value 98.079; the 0.08 % difference is inherited for the same reason. +H2SO4_MOLAR_MASS_G_PER_MOL: Final = 98.0 + +#: Coefficient of the air-number-density relation, [molec K / (cm^3 torr)]. From the MATLAB +#: ``runconcs_het.m`` line ``M = 9.65e18*P/T*conv``, ported at ``stratchem-jax/config.py:55``. +#: It is the ideal-gas law with the torr/kelvin units folded in; it is NOT independently derived +#: here, because the whole point is to match what the gas model uses. +AIR_NUMBER_DENSITY_COEFF: Final = 9.65e18 + +#: Millibar -> torr, the ``conv`` factor inside that same MATLAB expression +#: (``stratchem-jax/config.py:52``). +MBAR_TO_TORR: Final = 760.0 / 1013.25 + +#: Cubic centimetres per cubic metre. Named because plume geometry is entered in metres and the +#: model's volumes are in cm^3, and that factor of 1e6 is exactly the kind of literal that ends up +#: wrong once. +CM3_PER_M3: Final = 1.0e6 + +#: Parts per trillion by volume per unit mole fraction. +PPTV_PER_MOLE_FRACTION: Final = 1.0e12 + +#: Parts per million by volume per unit mole fraction. +PPMV_PER_MOLE_FRACTION: Final = 1.0e6 + +#: Grams per kilogram. +G_PER_KG: Final = 1000.0 + +__all__ = [ + "AIR_NUMBER_DENSITY_COEFF", + "AVOGADRO", + "AVOGADRO_GAS_MODEL", + "AVOGADRO_SEAM_RELATIVE_DIFFERENCE", + "CM3_PER_M3", + "G_PER_KG", + "H2SO4_MOLAR_MASS_G_PER_MOL", + "MBAR_TO_TORR", + "PPMV_PER_MOLE_FRACTION", + "PPTV_PER_MOLE_FRACTION", + "SO2_MOLAR_MASS_G_PER_MOL", +] diff --git a/studio/science/gcr.py b/studio/science/gcr.py new file mode 100644 index 0000000..57c32de --- /dev/null +++ b/studio/science/gcr.py @@ -0,0 +1,77 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Galactic-cosmic-ray ion-pair production rate. + +**This module deliberately does not compute anything.** It is the clearest case in the project of +the rule in ``studio/CLAUDE.md``: a number that is needed, that has no agreed source, and that would +be trivially easy to fabricate convincingly. + +What exists today is a bare ``30.0`` cm^-3 s^-1 in the paper ensemble (``run_ensemble.py:102``), +described in ``TABLE_microphysics_parameters.md`` as "galactic cosmic rays at ~20 km" with no +citation; and the model's own default of ``0.0``, which switches off ion-induced nucleation +altogether (``coupled/coupled_scenario.py:117``). So the two available values are an uncited +constant and a value that disables a physical process. + +The rate genuinely varies -- roughly a factor of two over the solar cycle, and strongly with +latitude (geomagnetic cutoff) and altitude. A function of those three arguments is the right shape. +Inventing its coefficients is not, because the result would arrive with the same confidence as a +computed one and feed the most sensitive part of the system: the ion-induced channels of Dunne et +al. (2016) nucleation. + +So :func:`ion_pair_production_rate` raises, and :data:`PAPER_ENSEMBLE_ION_PAIR_RATE` is available +for anyone who wants the ensemble's constant *as a constant*, with its provenance attached. + +Tracked as SCIENCE-6, issue #63. +""" + +from __future__ import annotations + +from typing import Final + +#: The paper ensemble's fixed value [ion pairs cm^-3 s^-1] (``run_ensemble.py:102``; +#: ``TABLE_microphysics_parameters.md``, "30 ion pairs cm^-3 s^-1, galactic cosmic rays at ~20 km"). +#: Uncited. Use it to reproduce the ensemble, not as a general-purpose value -- it is a single +#: number for a quantity that varies with altitude, latitude and solar-cycle phase. +PAPER_ENSEMBLE_ION_PAIR_RATE: Final = 30.0 + +#: The MODEL's default, which disables the ion-induced nucleation channels entirely +#: (``coupled/coupled_scenario.py:117``). Recorded so that "the default is 0" is discoverable here +#: rather than surprising someone whose nucleation quietly lost a channel. +MODEL_DEFAULT_ION_PAIR_RATE: Final = 0.0 + +_SCIENCE_6 = ( + "SCIENCE-6 (issue #63): the GCR ion-pair production rate has no agreed parameterisation. " + "The paper ensemble uses a fixed, uncited 30.0 cm^-3 s^-1 at ~20 km and the model defaults to " + "0.0, which disables ion-induced nucleation entirely." +) + + +def ion_pair_production_rate( + altitude_km: float, latitude_deg: float, solar_cycle_phase: float +) -> float: + """Ion-pair production rate [cm^-3 s^-1]. **Not implemented, by decision.** + + Args: + altitude_km: Box altitude. + latitude_deg: Geographic latitude; the geomagnetic cutoff rigidity, and therefore the + ionisation rate, is a strong function of it. + solar_cycle_phase: Phase in [0, 1], 0 at solar minimum (maximum GCR flux). + + Raises: + NotImplementedError: Always. Absent a citable parameterisation, returning a plausible number + would be worse than failing: it would look computed. Use + :data:`PAPER_ENSEMBLE_ION_PAIR_RATE` explicitly if you want the ensemble's constant. + """ + raise NotImplementedError( + f"ion_pair_production_rate(altitude_km={altitude_km}, latitude_deg={latitude_deg}, " + f"solar_cycle_phase={solar_cycle_phase}) is not implemented. {_SCIENCE_6} Set " + f"microphysics.ion_pair_rate explicitly -- studio.science.gcr." + f"PAPER_ENSEMBLE_ION_PAIR_RATE is that constant with its provenance attached." + ) + + +__all__ = [ + "MODEL_DEFAULT_ION_PAIR_RATE", + "PAPER_ENSEMBLE_ION_PAIR_RATE", + "ion_pair_production_rate", +] diff --git a/studio/science/plume.py b/studio/science/plume.py new file mode 100644 index 0000000..eeaccd3 --- /dev/null +++ b/studio/science/plume.py @@ -0,0 +1,140 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Initial plume volume, and injected mass -> initial concentration. + +**The most-duplicated derivation in the repository.** Six copies today, and they do not agree: + +=================================================== ========================================== +``coupled/paper_ensemble/run_ensemble.py:41-46,95`` V0 = 10 m x 10 m x **15 km**; 1 t SO2 -> + 6.273063291666667e15 molec cm^-3 -> pptv +``coupled/paper_ensemble/run_60day.py:37`` the same number, HARD-CODED (bit-identical, + verified) rather than derived +``coupled/paper_ensemble/make_rf_runs.py:44`` V0 = 1.5e12 cm^3, restated +``coupled/viz/bake_plume_dynamics.py:62-63`` V0 = 1.5e12 cm^3, restated, used to convert + concentration back to SO2-equivalent tonnes +``coupled/run_dilution_d1_clean.py:61,130`` V0 = 10 m x 10 m x **30 km** -- twice the + volume -- and the injection is specified + pptv-FIRST (2.9e9 pptv, "~1.7 t"), with the + mass computed back from it +``coupled/paper_ensemble/run_boxsize.py:43`` scales the initial concentration by a volume + factor, which is how the volume-invariance + sweep is done +=================================================== ========================================== + +The divergence that matters is the **15 km vs 30 km track**: a factor of two in V0, and therefore a +factor of two in initial concentration for the same injected mass. Which is right depends on what +t = 0 means, which is SCIENCE-2 (issue #54) and unresolved. Phase 0 follows the 810-run ensemble +(ASSUMPTION-5); this module implements the derivation, not the choice of inputs. + +**V0 does not enter the dynamics.** The model is intensive and volume-invariant +(``coupled/tests/test_boxvol_invariance.py``); the box-size sweep works purely by scaling the +initial concentration. So this derivation exists to turn a mass into a concentration and for no +other reason, and a UI must not imply that plume geometry feeds the physics. +""" + +from __future__ import annotations + +from studio.science.air import air_number_density +from studio.science.constants import ( + AVOGADRO, + CM3_PER_M3, + G_PER_KG, + PPTV_PER_MOLE_FRACTION, +) + + +def plume_volume_cm3(length_m: float, width_m: float, height_m: float) -> float: + """Initial plume volume V0 [cm^3] from a rectangular track. + + The ensemble's 10 m x 10 m x 15 km gives 1.5e12 cm^3 (``run_ensemble.py:45``). + + Raises: + ValueError: On a non-positive dimension -- a zero-volume plume divides by zero downstream. + """ + for name, value in (("length", length_m), ("width", width_m), ("height", height_m)): + if value <= 0.0: + raise ValueError(f"plume {name} must be > 0 m, got {value}") + return length_m * width_m * height_m * CM3_PER_M3 + + +def injected_number_density( + mass_kg: float, molar_mass_g_per_mol: float, volume_cm3: float +) -> float: + """Number density [molec cm^-3] of ``mass_kg`` of a species spread through ``volume_cm3``. + + ``n = (mass / M_w) * N_A / V``. Mirrors ``run_ensemble.py:46`` exactly, including its use of the + CODATA Avogadro constant rather than the gas model's rounded one -- see + ``studio/science/constants.py`` on that seam. + + Raises: + ValueError: On non-positive mass, molar mass or volume. + """ + if mass_kg <= 0.0: + raise ValueError(f"injected mass must be > 0 kg, got {mass_kg}") + if molar_mass_g_per_mol <= 0.0: + raise ValueError(f"molar mass must be > 0 g/mol, got {molar_mass_g_per_mol}") + if volume_cm3 <= 0.0: + raise ValueError(f"plume volume must be > 0 cm^3, got {volume_cm3}") + return mass_kg * G_PER_KG / molar_mass_g_per_mol * AVOGADRO / volume_cm3 + + +def number_density_to_pptv( + number_density_molec_cm3: float, air_number_density_molec_cm3: float +) -> float: + """Number density [molec cm^-3] -> mixing ratio [pptv] at a given air number density. + + Raises: + ValueError: On a non-positive air number density, or a negative number density. + """ + if air_number_density_molec_cm3 <= 0.0: + raise ValueError( + f"air number density must be > 0 molec/cm^3, got {air_number_density_molec_cm3}" + ) + if number_density_molec_cm3 < 0.0: + raise ValueError(f"number density must be >= 0, got {number_density_molec_cm3}") + return number_density_molec_cm3 / air_number_density_molec_cm3 * PPTV_PER_MOLE_FRACTION + + +def pptv_to_number_density(pptv: float, air_number_density_molec_cm3: float) -> float: + """Mixing ratio [pptv] -> number density [molec cm^-3]. + + Inverse of :func:`number_density_to_pptv`. Needed because one existing run specifies its + injection pptv-FIRST and computes the mass back (``coupled/run_dilution_d1_clean.py:130``); + both directions are real workflows. + """ + if air_number_density_molec_cm3 <= 0.0: + raise ValueError( + f"air number density must be > 0 molec/cm^3, got {air_number_density_molec_cm3}" + ) + if pptv < 0.0: + raise ValueError(f"mixing ratio must be >= 0 pptv, got {pptv}") + return pptv / PPTV_PER_MOLE_FRACTION * air_number_density_molec_cm3 + + +def initial_mixing_ratio_pptv( + *, + mass_kg: float, + molar_mass_g_per_mol: float, + volume_cm3: float, + pressure_mbar: float, + temperature_k: float, +) -> float: + """The full chain: injected mass -> initial mixing ratio [pptv]. Mirrors ``run_ensemble.py:95``. + + Note what the ensemble actually fixes: the injected **number density** is the same at every + site, and the pptv follows from the local air density. So the same 1 t release is a different + mixing ratio at 55 hPa than at 120 hPa, and the mixing ratio is the derived quantity -- not the + other way round. Keyword-only because five positional floats in a row is a units bug waiting to + happen. + """ + number_density = injected_number_density(mass_kg, molar_mass_g_per_mol, volume_cm3) + return number_density_to_pptv(number_density, air_number_density(pressure_mbar, temperature_k)) + + +__all__ = [ + "initial_mixing_ratio_pptv", + "injected_number_density", + "number_density_to_pptv", + "plume_volume_cm3", + "pptv_to_number_density", +] diff --git a/studio/science/size_distribution.py b/studio/science/size_distribution.py new file mode 100644 index 0000000..13dbbda --- /dev/null +++ b/studio/science/size_distribution.py @@ -0,0 +1,98 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Size-distribution reductions: bin mid-points, dlog10Dp, and dN/dlogDp. + +Four copies exist in the repository, written two ways: + +* ``run_ensemble.py:147-149`` and ``analyses/.../make_background_overlays.py:41-44`` + -- ``dp_mid = 10**(0.5*(log10(edges[:-1]) + log10(edges[1:])))``, + ``dlogdp = log10(edges)[1:] - log10(edges)[:-1]`` +* ``coupled/run_dilution_d1_clean.py:587-589`` + -- ``dp_mid = sqrt(edges[:-1]*edges[1:])``, ``dlogdp = log10(edges[1:]/edges[:-1])`` + +**These are the same quantity.** ``10**(0.5*(log a + log b)) == sqrt(a*b)`` and +``log b - log a == log(b/a)`` identically; the plan's note that the repository uses "two different +mid-point expressions" is, on inspection, two spellings of one expression. Measured on an 80-bin +TOMAS-like grid they differ by <= 7e-16 (mid-point) and <= 5e-15 (dlog10Dp) relative -- a few ULP of +float64 rounding, not a modelling difference. That is worth stating plainly, because "there are two +conventions in the code" would otherwise become a thing people believe and work around. + +This module implements the geometric-mean form (``sqrt(a*b)``): fewer operations, and no +intermediate logarithm to round. The equivalence is asserted by test, so the choice cannot +silently start mattering. + +The real trap in this area is not the formula. It is that **``dp_mid_um`` and ``dNdlogDp`` in +``state.npz`` are DRY diameters** while ``SA`` and ``radius_cm`` in the same file are WET -- see +``docs/studio/CAVEATS.md``. This module computes numbers; it does not know which basis its inputs +are on, so its callers must, and ``RunSummary`` (task 0.4) has to declare it per array. +""" + +from __future__ import annotations + +import numpy as np +import numpy.typing as npt + +FloatArray = npt.NDArray[np.float64] + + +def _validated_edges(edges_um: npt.ArrayLike) -> FloatArray: + """Bin edges as a float array, or raise. Strictly increasing and positive.""" + edges = np.asarray(edges_um, dtype=np.float64) + if edges.ndim != 1: + raise ValueError(f"bin edges must be 1-D, got shape {edges.shape}") + if edges.size < 2: + raise ValueError(f"need at least 2 bin edges to define a bin, got {edges.size}") + if not np.all(np.isfinite(edges)): + raise ValueError("bin edges must all be finite") + if np.any(edges <= 0.0): + raise ValueError("bin edges must be > 0 um (the grid is logarithmic in diameter)") + if not np.all(np.diff(edges) > 0.0): + raise ValueError("bin edges must be strictly increasing") + return edges + + +def bin_midpoints_um(edges_um: npt.ArrayLike) -> FloatArray: + """Geometric mid-point diameter of each bin [um]. + + Geometric, not arithmetic: the TOMAS grid is logarithmic in mass (ratio ``2**(40/n_bins)``), so + the arithmetic mean of two edges is not the centre of the bin on the axis these are plotted on. + """ + edges = _validated_edges(edges_um) + return np.sqrt(edges[:-1] * edges[1:]) + + +def dlog10_dp(edges_um: npt.ArrayLike) -> FloatArray: + """Width of each bin in log10(diameter): the ``dlogDp`` a size distribution is normalised by.""" + edges = _validated_edges(edges_um) + return np.log10(edges[1:] / edges[:-1]) + + +def dn_dlogdp(number_per_cm3: npt.ArrayLike, edges_um: npt.ArrayLike) -> FloatArray: + """Normalise per-bin number concentration [cm^-3] to dN/dlogDp [cm^-3]. + + Accepts a single spectrum ``(n_bins,)`` or a time series ``(n_times, n_bins)``; the last axis is + the size axis, matching ``state.npz``'s ``n_cm3``. + + Normalising is what makes bins comparable across grids: raw per-bin counts on a 40-bin grid and + an 80-bin grid are not the same curve, and only the normalised form is. + + Raises: + ValueError: If the bin count does not match the edges. This is the mistake worth catching -- + passing ``n_bins + 1`` edges is right and passing ``n_bins`` is a silent off-by-one that + numpy would broadcast into a plausible-looking wrong answer. + """ + counts = np.asarray(number_per_cm3, dtype=np.float64) + widths = dlog10_dp(edges_um) + if counts.ndim not in (1, 2): + raise ValueError(f"number concentration must be 1-D or 2-D, got shape {counts.shape}") + if counts.shape[-1] != widths.size: + raise ValueError( + f"got {counts.shape[-1]} bins but {widths.size + 1} edges define {widths.size} bins; " + f"edges must have exactly one more element than bins" + ) + if np.any(counts < 0.0): + raise ValueError("number concentration must be >= 0 cm^-3") + return counts / widths + + +__all__ = ["FloatArray", "bin_midpoints_um", "dlog10_dp", "dn_dlogdp"] diff --git a/studio/tests/unit/test_science_air.py b/studio/tests/unit/test_science_air.py new file mode 100644 index 0000000..30fc8fd --- /dev/null +++ b/studio/tests/unit/test_science_air.py @@ -0,0 +1,101 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``air_number_density`` is a mirror of the model's. This is what makes that acceptable. + +``studio.science`` may not import the model (ADR-001), so this one relation is duplicated. A +duplicate is only safe if divergence is detectable, so the test below imports the model's own +implementation and asserts EXACT agreement across the parameter range the runs actually use. + +It skips -- rather than fails -- when the ``stratchem-jax`` submodule is not checked out, because +its absence says nothing about the code under test. CI does not check out the private submodules +(see ``.github/workflows/studio-ci.yml``), so in CI this is a skip and locally it is a real check. +That asymmetry is deliberate but worth knowing: a divergence would be caught on a developer machine, +not by CI. +""" + +from __future__ import annotations + +import subprocess +import sys +import textwrap +from pathlib import Path + +import pytest + +from studio.science import air_number_density + +#: The T-p corners the ensemble and the flagship runs use: 55 hPa / 210 K and 213 K (30N 20 km), +#: 120 hPa / 210 K (60N 15 km), 55 hPa / 215 K (the D1 clean run). +RUN_CONDITIONS = [(55.0, 210.0), (55.0, 213.0), (120.0, 210.0), (55.0, 215.0)] + + +@pytest.mark.tier_a +def test_known_value_at_the_golden_site() -> None: + """55 hPa, 210 K -> 1.8956916099773243e18 molec cm^-3. + + Tolerance 1e-12 relative: the value is one multiply-divide chain on exact constants, so the only + admissible difference is float64 rounding. Quoted to full precision on purpose -- a rounded + literal here would have passed while hiding a wrong constant. + """ + assert air_number_density(55.0, 210.0) == pytest.approx(1.8956916099773243e18, rel=1e-12) + + +@pytest.mark.tier_a +def test_scales_as_the_ideal_gas_law() -> None: + """Linear in pressure, inverse in temperature -- the property, not just a value.""" + assert air_number_density(110.0, 210.0) == pytest.approx( + 2.0 * air_number_density(55.0, 210.0), rel=1e-15 + ) + assert air_number_density(55.0, 420.0) == pytest.approx( + 0.5 * air_number_density(55.0, 210.0), rel=1e-15 + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize(("pressure", "temperature"), [(0.0, 210.0), (-1.0, 210.0)]) +def test_non_physical_pressure_raises(pressure: float, temperature: float) -> None: + with pytest.raises(ValueError, match="pressure must be > 0"): + air_number_density(pressure, temperature) + + +@pytest.mark.tier_a +@pytest.mark.parametrize(("pressure", "temperature"), [(55.0, 0.0), (55.0, -3.0)]) +def test_non_physical_temperature_raises(pressure: float, temperature: float) -> None: + """Zero kelvin would be a division by zero; ``inf`` is not a useful answer (ADR-005).""" + with pytest.raises(ValueError, match="temperature must be > 0"): + air_number_density(pressure, temperature) + + +@pytest.mark.tier_a +def test_agrees_exactly_with_the_models_own_implementation(repo_root: Path) -> None: + """The mirror check. Tolerance: exact -- same formula and constants, so any difference is real. + + Runs in a subprocess with ``stratchem-jax`` on ``sys.path``: importing the model's flat + ``config`` module in-process would leave a ``config`` in ``sys.modules`` for every test that + follows, which is exactly the kind of cross-test contamination the import-boundary test warns + about. + """ + stratchem = repo_root / "stratchem-jax" + if not (stratchem / "config.py").is_file(): + pytest.skip( + f"stratchem-jax submodule not checked out at {stratchem} " + f"(`git submodule update --init`); the mirror check needs the model's own version" + ) + probe = textwrap.dedent(f""" + import sys + sys.path.insert(0, {str(stratchem)!r}) + from config import air_number_density as model_impl + print("\\n".join(repr(model_impl(p, t)) for p, t in {RUN_CONDITIONS!r})) + """) + proc = subprocess.run( + [sys.executable, "-c", probe], capture_output=True, text=True, cwd=str(repo_root) + ) + assert proc.returncode == 0, f"could not evaluate the model's implementation:\n{proc.stderr}" + + model_values = [float(line) for line in proc.stdout.split()] + studio_values = [air_number_density(p, t) for p, t in RUN_CONDITIONS] + assert model_values == studio_values, ( + "studio.science.air_number_density has diverged from stratchem-jax/config.py:55. " + "It is a deliberate mirror (studio.science may not import the model); if the model's " + "relation changed, change this one in the same commit and say so." + ) diff --git a/studio/tests/unit/test_science_gcr.py b/studio/tests/unit/test_science_gcr.py new file mode 100644 index 0000000..66609f5 --- /dev/null +++ b/studio/tests/unit/test_science_gcr.py @@ -0,0 +1,67 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The GCR module refuses to compute, and these tests hold it to that. + +An unimplemented derivation is only safe if it STAYS unimplemented until someone supplies a +citation. The obvious future failure is a well-meaning change that makes +``ion_pair_production_rate`` return "something reasonable" so a form stops erroring -- these tests +are what would fail in that PR, with the reason attached. +""" + +from __future__ import annotations + +import pytest + +from studio.science import ( + MODEL_DEFAULT_ION_PAIR_RATE, + PAPER_ENSEMBLE_ION_PAIR_RATE, + ion_pair_production_rate, +) + + +@pytest.mark.tier_a +def test_the_parameterisation_refuses_to_guess() -> None: + """No citation, no number (ADR-005). The error must say why and where it is tracked.""" + with pytest.raises(NotImplementedError) as excinfo: + ion_pair_production_rate(altitude_km=20.0, latitude_deg=30.0, solar_cycle_phase=0.5) + message = str(excinfo.value) + assert "SCIENCE-6" in message + assert "#63" in message + assert "PAPER_ENSEMBLE_ION_PAIR_RATE" in message, ( + "the error must point at the constant that DOES have provenance, or the next person " + "will invent a value rather than find it" + ) + + +@pytest.mark.tier_a +def test_it_refuses_for_every_input_including_the_ensembles_own_conditions() -> None: + """~20 km / 30N is exactly where the uncited 30.0 came from, and it is not special-cased. + + Returning the known value at the known point and raising elsewhere would be the most tempting + version of this mistake: it would look like a working function with gaps. + """ + for altitude, latitude, phase in [(20.0, 30.0, 0.0), (15.0, 60.0, 1.0), (20.0, 30.0, 0.5)]: + with pytest.raises(NotImplementedError): + ion_pair_production_rate( + altitude_km=altitude, latitude_deg=latitude, solar_cycle_phase=phase + ) + + +@pytest.mark.tier_a +def test_the_two_documented_constants_are_what_the_code_uses() -> None: + """The ensemble's value and the model's default, both recorded, neither silently preferred.""" + assert PAPER_ENSEMBLE_ION_PAIR_RATE == 30.0 + assert MODEL_DEFAULT_ION_PAIR_RATE == 0.0 + + +@pytest.mark.tier_a +def test_the_schema_default_matches_the_ensemble_constant() -> None: + """One value, two places: the schema default and this constant must not drift apart. + + ``microphysics.ion_pair_rate`` defaults to the ensemble's 30.0 (ASSUMPTION-5) rather than the + model's 0.0, and that choice is only defensible while both sides agree on what the ensemble + used. + """ + from studio.schema import RunConfig + + assert RunConfig().microphysics.ion_pair_rate == PAPER_ENSEMBLE_ION_PAIR_RATE diff --git a/studio/tests/unit/test_science_plume.py b/studio/tests/unit/test_science_plume.py new file mode 100644 index 0000000..7e29530 --- /dev/null +++ b/studio/tests/unit/test_science_plume.py @@ -0,0 +1,158 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The injected-mass -> initial-concentration chain, against the numbers the ensemble actually ran. + +Tolerance for the golden values: **exact**. These are not measurements being approximated, they are +the same arithmetic on the same constants, and the ensemble's own code hard-codes one of them +(``run_60day.py:37``) -- so anything other than bit-equality means Studio's chain differs from the +model's, which is precisely what this task exists to prevent. +""" + +from __future__ import annotations + +import pytest + +from studio.science import ( + SO2_MOLAR_MASS_G_PER_MOL, + air_number_density, + initial_mixing_ratio_pptv, + injected_number_density, + number_density_to_pptv, + plume_volume_cm3, + pptv_to_number_density, +) + +#: ``run_ensemble.py:45`` -- 10 m x 10 m x 15 km. +ENSEMBLE_V0_CM3 = 1.5e12 + +#: ``run_ensemble.py:46`` computes this, and ``run_60day.py:37`` hard-codes it. Verified +#: bit-identical between the two, which is why it can be asserted exactly. +ENSEMBLE_SO2_NUMBER_DENSITY = 6.273063291666667e15 + +#: 1 tonne, the ensemble's release (``TABLE_microphysics_parameters.md``). +ENSEMBLE_SO2_MASS_KG = 1000.0 + + +@pytest.mark.tier_a +def test_plume_volume_matches_the_ensemble_geometry() -> None: + """10 m x 10 m x 15 km == 1.5e12 cm^3, exactly.""" + assert plume_volume_cm3(15000.0, 10.0, 10.0) == ENSEMBLE_V0_CM3 + + +@pytest.mark.tier_a +def test_injected_number_density_reproduces_the_ensemble_exactly() -> None: + """The number the whole 810-run ensemble was initialised with.""" + assert ( + injected_number_density(ENSEMBLE_SO2_MASS_KG, SO2_MOLAR_MASS_G_PER_MOL, ENSEMBLE_V0_CM3) + == ENSEMBLE_SO2_NUMBER_DENSITY + ) + + +@pytest.mark.tier_a +def test_the_full_chain_reproduces_the_golden_cases_initial_so2() -> None: + """Mass -> number density -> pptv at the golden case's 210 K / 55 hPa. + + What the ensemble fixes is the number DENSITY, so the mixing ratio is the derived quantity and + differs between sites for the same injected mass. That is asserted below, because it is the + part people get backwards. + """ + pptv = initial_mixing_ratio_pptv( + mass_kg=ENSEMBLE_SO2_MASS_KG, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=ENSEMBLE_V0_CM3, + pressure_mbar=55.0, + temperature_k=210.0, + ) + expected = ENSEMBLE_SO2_NUMBER_DENSITY / air_number_density(55.0, 210.0) * 1e12 + assert pptv == expected + assert pptv == pytest.approx(3.309115922996412e9, rel=1e-15) + + +@pytest.mark.tier_a +def test_the_same_mass_gives_a_different_mixing_ratio_at_a_different_site() -> None: + """60N / 15 km (120 hPa) is denser air, so 1 t of SO2 is a SMALLER mixing ratio there.""" + at_20km = initial_mixing_ratio_pptv( + mass_kg=ENSEMBLE_SO2_MASS_KG, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=ENSEMBLE_V0_CM3, + pressure_mbar=55.0, + temperature_k=210.0, + ) + at_15km = initial_mixing_ratio_pptv( + mass_kg=ENSEMBLE_SO2_MASS_KG, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=ENSEMBLE_V0_CM3, + pressure_mbar=120.0, + temperature_k=210.0, + ) + assert at_15km < at_20km + assert at_20km / at_15km == pytest.approx(120.0 / 55.0, rel=1e-12) + + +@pytest.mark.tier_a +def test_the_two_track_lengths_in_the_repository_differ_by_exactly_two() -> None: + """The one divergence that matters, pinned as a fact rather than left as a memory. + + ``run_ensemble.py:45`` uses a 15 km track and ``coupled/run_dilution_d1_clean.py:61`` a 30 km + one. Same injected mass, half the concentration. Which is correct depends on SCIENCE-2 (#54). + """ + ensemble = plume_volume_cm3(15000.0, 10.0, 10.0) + d1_flagship = plume_volume_cm3(30000.0, 10.0, 10.0) + assert d1_flagship == 2.0 * ensemble + assert injected_number_density(1000.0, SO2_MOLAR_MASS_G_PER_MOL, d1_flagship) == 0.5 * ( + injected_number_density(1000.0, SO2_MOLAR_MASS_G_PER_MOL, ensemble) + ) + + +@pytest.mark.tier_a +def test_pptv_round_trips_through_number_density() -> None: + """Both directions are real workflows: the D1 run specifies pptv and computes the mass back.""" + m_air = air_number_density(55.0, 215.0) + original = 2.9e9 # coupled/run_dilution_d1_clean.py:69 + assert pptv_to_number_density(original, m_air) / m_air * 1e12 == pytest.approx( + original, rel=1e-15 + ) + assert number_density_to_pptv(pptv_to_number_density(original, m_air), m_air) == pytest.approx( + original, rel=1e-15 + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("kwargs", "message"), + [ + ({"length_m": 0.0, "width_m": 10.0, "height_m": 10.0}, "length must be > 0"), + ({"length_m": 15000.0, "width_m": -1.0, "height_m": 10.0}, "width must be > 0"), + ({"length_m": 15000.0, "width_m": 10.0, "height_m": 0.0}, "height must be > 0"), + ], +) +def test_degenerate_geometry_raises(kwargs: dict[str, float], message: str) -> None: + """A zero-volume plume is a division by zero one step later; catch it where it is meaningful.""" + with pytest.raises(ValueError, match=message): + plume_volume_cm3(**kwargs) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("mass", "molar_mass", "volume", "message"), + [ + (0.0, 64.0, 1.5e12, "mass must be > 0"), + (-1.0, 64.0, 1.5e12, "mass must be > 0"), + (1000.0, 0.0, 1.5e12, "molar mass must be > 0"), + (1000.0, 64.0, 0.0, "volume must be > 0"), + ], +) +def test_injection_inputs_are_validated( + mass: float, molar_mass: float, volume: float, message: str +) -> None: + with pytest.raises(ValueError, match=message): + injected_number_density(mass, molar_mass, volume) + + +@pytest.mark.tier_a +def test_conversions_reject_a_non_physical_air_density() -> None: + """No default, no fallback: an air density of zero has no meaningful mixing ratio (ADR-005).""" + with pytest.raises(ValueError, match="air number density must be > 0"): + number_density_to_pptv(1e15, 0.0) + with pytest.raises(ValueError, match="air number density must be > 0"): + pptv_to_number_density(1e9, -1.0) diff --git a/studio/tests/unit/test_science_size_distribution.py b/studio/tests/unit/test_science_size_distribution.py new file mode 100644 index 0000000..bb3e898 --- /dev/null +++ b/studio/tests/unit/test_science_size_distribution.py @@ -0,0 +1,123 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Size-distribution reductions, and the claim that the repository's "two conventions" are one. + +The headline test here is ``test_the_repositorys_two_spellings_are_the_same_quantity``: it measures +the difference between the two forms in the code rather than asserting they are equivalent on paper, +because "these are the same" is the kind of statement that is easy to believe and expensive to be +wrong about. +""" + +from __future__ import annotations + +import numpy as np +import pytest + +from studio.science import bin_midpoints_um, dlog10_dp, dn_dlogdp + +#: A TOMAS-like 80-bin grid: dry Dp from 1.7 nm to 17.5 um, geometric (the real grid's mass ratio is +#: 2**(40/n_bins), which is geometric in diameter too). Built here rather than imported from the +#: model because Tier A must not pay a JAX import; the property under test is grid-shape-independent +#: and is also checked on a deliberately irregular grid below. +EDGES_UM = np.geomspace(1.7e-3, 17.5, 81) + + +@pytest.mark.tier_a +def test_the_repositorys_two_spellings_are_the_same_quantity() -> None: + """``10**(0.5*(log a + log b))`` vs ``sqrt(a*b)``, and ``log b - log a`` vs ``log(b/a)``. + + Tolerance 1e-14 relative, against a measured ~7e-16 (mid-point) and ~5e-15 (dlog10Dp) on this + grid: a few ULP of float64, i.e. rounding, not a modelling difference. The plan's note that the + repository carries "two different mid-point expressions" is a misreading of two spellings of one + expression, and this is the evidence for saying so. + """ + log_edges = np.log10(EDGES_UM) + run_ensemble_mid = 10 ** (0.5 * (log_edges[:-1] + log_edges[1:])) # run_ensemble.py:148 + run_ensemble_dlog = log_edges[1:] - log_edges[:-1] # run_ensemble.py:148 + + np.testing.assert_allclose(bin_midpoints_um(EDGES_UM), run_ensemble_mid, rtol=1e-14, atol=0.0) + np.testing.assert_allclose(dlog10_dp(EDGES_UM), run_ensemble_dlog, rtol=1e-14, atol=0.0) + + +@pytest.mark.tier_a +def test_equivalence_holds_on_an_irregular_grid_too() -> None: + """Not an artefact of a perfectly geometric grid: same check where bin widths vary wildly.""" + edges = np.array([1e-3, 2e-3, 5e-3, 1e-2, 3e-1, 1.0, 17.5]) + log_edges = np.log10(edges) + np.testing.assert_allclose( + bin_midpoints_um(edges), 10 ** (0.5 * (log_edges[:-1] + log_edges[1:])), rtol=1e-14 + ) + np.testing.assert_allclose(dlog10_dp(edges), log_edges[1:] - log_edges[:-1], rtol=1e-14) + + +@pytest.mark.tier_a +def test_midpoint_is_geometric_not_arithmetic() -> None: + """On a log axis the arithmetic mean is not the centre, and the difference is not small. + + For a bin spanning a decade the two differ by ~28 %, which would be visible as a shifted mode + diameter in every size-distribution figure. + """ + edges = np.array([0.1, 1.0]) + assert bin_midpoints_um(edges)[0] == pytest.approx(np.sqrt(0.1), rel=1e-15) + arithmetic = 0.55 + assert abs(bin_midpoints_um(edges)[0] - arithmetic) / arithmetic > 0.25 + + +@pytest.mark.tier_a +def test_dn_dlogdp_normalises_by_bin_width() -> None: + """The defining property: equal counts in unequal bins are NOT an equal dN/dlogDp.""" + edges = np.array([1.0, 10.0, 1000.0]) # widths 1 and 2 in log10 + result = dn_dlogdp(np.array([100.0, 100.0]), edges) + np.testing.assert_allclose(result, [100.0, 50.0], rtol=1e-15) + + +@pytest.mark.tier_a +def test_dn_dlogdp_handles_a_time_series() -> None: + """``state.npz`` stores ``n_cm3`` as (n_times, n_bins); the size axis is last.""" + counts = np.tile(np.linspace(1.0, 80.0, 80), (5, 1)) + result = dn_dlogdp(counts, EDGES_UM) + assert result.shape == (5, 80) + np.testing.assert_allclose(result[0], counts[0] / dlog10_dp(EDGES_UM), rtol=1e-15) + + +@pytest.mark.tier_a +def test_integrating_dn_dlogdp_recovers_the_total_number() -> None: + """sum(dN/dlogDp * dlogDp) == sum(N) -- the conservation check behind the normalisation.""" + counts = np.linspace(0.5, 40.0, 80) + recovered = float(np.sum(dn_dlogdp(counts, EDGES_UM) * dlog10_dp(EDGES_UM))) + assert recovered == pytest.approx(float(counts.sum()), rel=1e-14) + + +@pytest.mark.tier_a +def test_off_by_one_in_the_edge_count_raises() -> None: + """The mistake worth catching: n edges for n bins broadcasts into a plausible wrong answer.""" + with pytest.raises(ValueError, match="edges must have exactly one more element"): + dn_dlogdp(np.ones(80), EDGES_UM[:-1]) + with pytest.raises(ValueError, match="edges must have exactly one more element"): + dn_dlogdp(np.ones(79), EDGES_UM) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("edges", "message"), + [ + (np.array([1.0]), "at least 2 bin edges"), + (np.array([[1.0, 2.0], [3.0, 4.0]]), "must be 1-D"), + (np.array([0.0, 1.0]), "must be > 0 um"), + (np.array([-1.0, 1.0]), "must be > 0 um"), + (np.array([1.0, 0.5, 2.0]), "strictly increasing"), + (np.array([1.0, 1.0]), "strictly increasing"), + (np.array([1.0, np.nan]), "must all be finite"), + ], +) +def test_malformed_edges_raise(edges: np.ndarray, message: str) -> None: + """A logarithmic grid has preconditions; violating them silently yields NaN, not an error.""" + with pytest.raises(ValueError, match=message): + bin_midpoints_um(edges) + + +@pytest.mark.tier_a +def test_negative_counts_raise() -> None: + """A negative number concentration is a corrupted input, not a small one.""" + with pytest.raises(ValueError, match="must be >= 0"): + dn_dlogdp(np.array([1.0, -1.0]), np.array([1.0, 2.0, 3.0])) From fb2e99affbfd664ecc05fa9610639dda2669c264 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 18:49:12 -0700 Subject: [PATCH 04/18] studio: dependency graph and override semantics (#67) Task 0.3 (#66). New package studio/resolve: graph.py (the DAG built from derived_from), registry.py (which function computes which field), resolver.py (resolution, overrides, staleness). 47 new Tier-A tests, 148 total. A fourth pure package rather than a module inside studio/schema, because schema is data and stays free of computation while science is computation and stays free of the config model; resolution is the composition of the two, and naming it keeps that layering visible. It joins schema and science in the import-boundary test and under mypy --strict: the API resolves on every keystroke, so a JAX import on that path would be unaffordable. Staleness is defined against a fingerprint. Setting an override records the upstream values at that moment, and the field is stale when they no longer match. That makes staleness a property of the config alone -- no edit history, no ordering assumptions -- and it is what lets the UI present the old value, the newly-derived value and what changed between them rather than a bare warning. accept_derived and keep_override are the two ways out, both explicit; a kept override re-anchors and goes stale again on the next change rather than being silenced for good. require_consistent() raises on any stale field and the stale list survives serialisation, so a persisted config cannot lose the fact that it is inconsistent. The trap that guards: a stale config still has a hash, which would be a stable identity for numbers that do not follow from each other. The load-bearing test captures every field before and after an edit and asserts the set that moved is exactly the edited field plus its downstream closure. Both directions fail silently otherwise -- too little leaves a stale number that reaches the model, too much discards a value the user chose -- and it covers three fields with no dependents, where the expected change set is the edited field alone. The registry is checked against the schema rather than trusted: every DERIVED field has a derivation, no derivation exists for a field the schema does not derive, and declared inputs equal derived_from exactly. Cycles raise at construction rather than iterating to a fixed point or breaking an arbitrary edge, both of which would make the result depend on where the engine started. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/studio-ci.yml | 4 +- docs/studio/PROGRESS.md | 64 ++++- docs/studio/plan/PHASE_0.md | 2 +- studio/CLAUDE.md | 8 +- studio/resolve/__init__.py | 63 +++++ studio/resolve/graph.py | 160 ++++++++++++ studio/resolve/registry.py | 123 +++++++++ studio/resolve/resolver.py | 275 ++++++++++++++++++++ studio/tests/unit/test_import_boundaries.py | 15 +- studio/tests/unit/test_resolve_graph.py | 131 ++++++++++ studio/tests/unit/test_resolve_registry.py | 106 ++++++++ studio/tests/unit/test_resolve_resolver.py | 257 ++++++++++++++++++ 12 files changed, 1193 insertions(+), 15 deletions(-) create mode 100644 studio/resolve/__init__.py create mode 100644 studio/resolve/graph.py create mode 100644 studio/resolve/registry.py create mode 100644 studio/resolve/resolver.py create mode 100644 studio/tests/unit/test_resolve_graph.py create mode 100644 studio/tests/unit/test_resolve_registry.py create mode 100644 studio/tests/unit/test_resolve_resolver.py diff --git a/.github/workflows/studio-ci.yml b/.github/workflows/studio-ci.yml index ea10877..7c92197 100644 --- a/.github/workflows/studio-ci.yml +++ b/.github/workflows/studio-ci.yml @@ -83,10 +83,10 @@ jobs: - name: black run: .venv/bin/black --check studio/ - # --strict on the two packages that must stay pure; the pyproject baseline covers the rest. + # --strict on the three packages that must stay pure; the pyproject baseline covers the rest. - name: mypy run: | - .venv/bin/mypy --strict studio/schema studio/science + .venv/bin/mypy --strict studio/schema studio/science studio/resolve .venv/bin/mypy # Tier B (full-case golden reproduction against the archived ensemble) is NOT run here: the diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index f2d0be6..12df06c 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -16,7 +16,7 @@ with full provenance, and the golden tests pass. |---|---| | 0.1 Repo, CI, docs skeleton | **done** | | 0.2 `studio/schema` v0 — **review gate** | **done** (#62, reviewed) | -| 0.3 Dependency-graph engine + override semantics | not started | +| 0.3 Dependency-graph engine + override semantics | **done** (#66) | | 0.4 `studio/modelio` seam + `RunSummary` | not started | | 0.5 `studio/science` derivations | **done** (#64) | | 0.6 `studio/runner` + job lifecycle | not started | @@ -29,6 +29,68 @@ derivations to resolve rather than fixtures. --- +### 2026-08-13 — Task 0.3: dependency graph and override semantics (issue #66) + +**New package: `studio/resolve/`** — `graph.py` (the DAG), `registry.py` (which function computes +which field), `resolver.py` (resolution, overrides, staleness). 47 new Tier-A tests (148 total). + +**Why a fourth pure package rather than a module in `studio/schema`.** `studio/schema` is data and +stays free of computation; `studio/science` is computation and stays free of the config model. +Resolution is the composition of the two. Naming it keeps that layering visible — and keeps schema +and science usable, and testable, without it. It joins schema and science in the import-boundary +test and under `mypy --strict`, because the API resolves a config on every keystroke and a JAX +import on that path would be unaffordable. + +**The semantics** + +- **auto** → recomputed silently whenever anything upstream changes. +- **user_override** → never overwritten by a recomputation. +- **user_override + stale** → an override whose inputs have moved since it was set. + +Staleness is defined against a **fingerprint**: setting an override records the upstream values at +that moment. Stale means those recorded values differ from the current ones. That makes staleness a +property of the config alone — no edit history, no ordering assumptions — and it is what lets the UI +show the old value, the newly-derived value, *and what changed between them* rather than a bare +warning. Two explicit ways out, both the user's call: `accept_derived` (drop the override) or +`keep_override` (keep the value, re-anchor the fingerprint; it goes stale again on the next change +rather than being permanently silenced). + +`ResolvedConfig.require_consistent()` raises on any stale field, and the stale list survives +serialisation — a persisted config cannot lose the fact that it is inconsistent. The trap it guards: +a stale config still *has* a hash, which would be a stable identity for numbers that do not follow +from each other. + +**The load-bearing test** is `test_an_edit_changes_exactly_the_downstream_closure`: capture every +field before and after an edit, assert the set that moved is **exactly** the edited field plus its +closure. Both directions are silent failures — recomputing too little leaves a stale number that +reaches the model, recomputing too much discards something the user chose. It runs over eight edits +including three fields with no dependents, where the expected change set is the edited field alone. + +**Decisions** + +- **Editing a derived field IS an override.** A user typing into a computed box means "I want this + value", not "recompute me away on the next edit". +- **The registry is checked against the schema, not trusted.** Every `DERIVED` field must have a + derivation, no derivation may exist for a field the schema does not derive, and each derivation's + declared inputs must equal the field's `derived_from` exactly. Without that, a field could declare + an input its derivation ignores (the UI reports a change that did not happen) or read one the + graph does not know about (the stale result reaches the model). +- **Cycles raise at graph construction.** Not fixed-point iteration, not breaking an arbitrary edge: + both would produce numbers that depend on where the engine started. Tested on synthetic graphs, + since the real schema has no cycle to exhibit. +- **Topological order, ties broken alphabetically** — deterministic because resolution order is + observable through which error surfaces first. +- The graph tests run mostly on **hand-built graphs**: the schema has exactly one chain of length + two today, and an engine tested only against the shape it currently meets breaks the first time + the schema grows. + +**Corrected while writing the tests:** a test asserted that a zero plume dimension would fail inside +the derivation. It fails earlier, at the schema's `gt=0` bound — which is the better layer, because +the error names the field the user typed rather than a function they have never heard of. The +derivation's own check stays as the guard for callers that do not come through the schema. + +--- + ### 2026-08-13 — Task 0.5: `studio/science` derivations (issue #64) Taken **before 0.3** at Ali's direction, so the dependency-graph engine has real derivations to diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index 4b54f53..30222e3 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -53,7 +53,7 @@ own string, and one the 0.4 equivalence test must cover explicitly. --- -## 0.3 — Dependency-graph engine and override semantics +## 0.3 — Dependency-graph engine and override semantics · *done* (issue #66) The mechanism that makes "go back and edit stage 1 without losing your stage 6 choices" correct by construction. It is a data-model problem, not a UI problem. diff --git a/studio/CLAUDE.md b/studio/CLAUDE.md index fc29c14..79a198d 100644 --- a/studio/CLAUDE.md +++ b/studio/CLAUDE.md @@ -36,13 +36,13 @@ looks right, presented with the same confidence as a computed result, is worse t - **No shortcuts around the main code path.** Do not special-case tests, do not bypass validation, do not stub the model to make a UI demo work. If a demo needs fake data it is clearly labelled synthetic and lives in a fixture. -- **Type hints everywhere.** `mypy --strict` on `studio/schema` and `studio/science`. +- **Type hints everywhere.** `mypy --strict` on `studio/schema`, `studio/science` and `studio/resolve`. - **Known bugs are issues, not TODO comments.** If a TODO is unavoidable it references an issue number. ## Package boundaries — enforced by tests -- `studio/schema` and `studio/science` import nothing from the API, the database, the web layer, or - `coupled`. They must be usable from a bare Python session. Importing `coupled` pulls in JAX. +- `studio/schema`, `studio/science` and `studio/resolve` import nothing from the API, the database, + the web layer, or `coupled`. They must be usable from a bare Python session. Importing `coupled` pulls in JAX. - **`studio/modelio` is the only package permitted to import `coupled`.** - Enforced by `studio/tests/unit/test_import_boundaries.py`, both at runtime and statically. @@ -73,7 +73,7 @@ cp studio/.env.example .env # then edit; .env i ```bash pytest studio/tests -m tier_a # fast; what CI runs pytest studio/tests -m tier_b # full-case golden reproduction; nightly/manual -mypy --strict studio/schema studio/science +mypy --strict studio/schema studio/science studio/resolve ruff check studio/ && black --check studio/ ``` diff --git a/studio/resolve/__init__.py b/studio/resolve/__init__.py new file mode 100644 index 0000000..253e718 --- /dev/null +++ b/studio/resolve/__init__.py @@ -0,0 +1,63 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The dependency graph and override semantics (task 0.3). + +What makes "go back and edit stage 1 without losing your stage 6 choices" correct by construction. +It is a data-model problem, not a UI problem, so it lives here and the UI merely renders the result. + +Three pieces: + +* ``graph.py`` -- the DAG, built from the schema's ``derived_from`` metadata. Pure graph work: no + values, no derivations, no physics. +* ``registry.py`` -- which function in ``studio/science`` computes which derived field, with the + binding checked against the schema rather than trusted. +* ``resolver.py`` -- resolution in topological order, the auto / user_override / stale states, and + the two explicit ways to settle a stale field. + +**Why a package of its own.** ``studio/schema`` is data and stays free of computation; +``studio/science`` is computation and stays free of the config model. Resolution is the composition +of the two, and giving it a name keeps that layering visible -- schema and science remain usable, +and testable, without it. + +Like schema and science, this package must not import ``coupled``, the API or the database: the API +resolves a config on every keystroke, and a JAX import on that path would be unaffordable. Enforced +by ``studio/tests/unit/test_import_boundaries.py``. +""" + +from __future__ import annotations + +from studio.resolve.graph import CyclicDependencyError, DependencyGraph, schema_derived_fields +from studio.resolve.registry import DERIVATIONS, Derivation, derivation_for +from studio.resolve.resolver import ( + ChangedInput, + InconsistentConfigError, + OverrideRecord, + ResolvedConfig, + StaleField, + accept_derived, + apply_change, + downstream_of, + keep_override, + resolve, + set_override, +) + +__all__ = [ + "DERIVATIONS", + "ChangedInput", + "CyclicDependencyError", + "DependencyGraph", + "Derivation", + "InconsistentConfigError", + "OverrideRecord", + "ResolvedConfig", + "StaleField", + "accept_derived", + "apply_change", + "derivation_for", + "downstream_of", + "keep_override", + "resolve", + "schema_derived_fields", + "set_override", +] diff --git a/studio/resolve/graph.py b/studio/resolve/graph.py new file mode 100644 index 0000000..d890266 --- /dev/null +++ b/studio/resolve/graph.py @@ -0,0 +1,160 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The dependency graph, built from ``derived_from`` metadata. + +Pure graph work: no config values, no derivation functions, no physics. Given the schema's +``derived_from`` edges it answers three questions -- what depends on this, in what order must things +be computed, and is the graph even acyclic. + +Kept separate from the resolver because the two fail differently and at different times. A malformed +graph is a **schema** bug that exists the moment the metadata is written, and should be found by a +test that never touches a config; a wrong recomputation is a **resolution** bug that needs values to +show up. Mixing them would mean a cycle in the metadata first manifesting as a hang while resolving +someone's run. + +Edges point from a dependency to its dependent (``site.temperature_k`` -> +``injection.so2_initial_pptv``), so "downstream" means "what must be recomputed when this changes", +which is the only direction the override machinery ever asks about. +""" + +from __future__ import annotations + +from collections import deque +from collections.abc import Iterable, Mapping + +from studio.schema import RunConfig, field_catalogue +from studio.schema.fields import Provenance + + +class CyclicDependencyError(ValueError): + """A ``derived_from`` cycle. Raised at graph construction, never worked around. + + A cycle means the schema claims a field is computed from something that is computed from it. No + resolution order exists, and the plausible-looking alternatives -- iterate to a fixed point, or + break the cycle at an arbitrary edge -- would both produce numbers that depend on where the + engine happened to start. + """ + + +class DependencyGraph: + """Immutable DAG over dotted field paths. + + Built from ``{node: (its dependencies)}``. Nodes with no dependencies are primary fields; the + rest are derived. + """ + + def __init__(self, dependencies: Mapping[str, Iterable[str]]) -> None: + self._dependencies: dict[str, tuple[str, ...]] = { + node: tuple(deps) for node, deps in dependencies.items() + } + self._dependents: dict[str, list[str]] = {node: [] for node in self._dependencies} + for node, deps in self._dependencies.items(): + for dep in deps: + if dep not in self._dependencies: + raise ValueError( + f"{node!r} depends on {dep!r}, which is not a node in the graph. Every " + f"path named in derived_from must be a real field." + ) + self._dependents[dep].append(node) + self._order = self._topological_order() + + @classmethod + def from_schema(cls, model_cls: type[RunConfig] = RunConfig) -> DependencyGraph: + """Build the graph from the schema's own metadata -- the only source of edges (ADR-002).""" + catalogue = field_catalogue(model_cls) + return cls( + { + path: tuple(meta["derived_from"]) if meta["derived_from"] else () + for path, meta in catalogue.items() + } + ) + + @property + def nodes(self) -> tuple[str, ...]: + """Every field path, in topological order (dependencies before dependents).""" + return self._order + + def dependencies_of(self, path: str) -> tuple[str, ...]: + """The paths ``path`` is computed from. Empty for a primary field.""" + self._check_known(path) + return self._dependencies[path] + + def dependents_of(self, path: str) -> tuple[str, ...]: + """The paths computed DIRECTLY from ``path``. One hop only; see :meth:`downstream_of`.""" + self._check_known(path) + return tuple(self._dependents[path]) + + def downstream_of(self, paths: Iterable[str]) -> tuple[str, ...]: + """Every field reachable from ``paths``, in topological order. The recompute set. + + Transitive on purpose: editing a plume dimension changes ``plume_volume_cm3``, which + changes ``so2_initial_pptv``. A one-hop answer would leave the second stale and silently + wrong -- and it is the second one that reaches the model. + + The seeds themselves are NOT included; the result is what must be recomputed, and a field + the user just set is not recomputed from itself. + """ + seeds = list(paths) + for path in seeds: + self._check_known(path) + seen: set[str] = set() + queue = deque(seeds) + while queue: + for dependent in self._dependents[queue.popleft()]: + if dependent not in seen: + seen.add(dependent) + queue.append(dependent) + return tuple(node for node in self._order if node in seen) + + def derived_nodes(self) -> tuple[str, ...]: + """Fields with at least one dependency, in topological order.""" + return tuple(node for node in self._order if self._dependencies[node]) + + def _check_known(self, path: str) -> None: + if path not in self._dependencies: + raise ValueError(f"unknown field path {path!r}") + + def _topological_order(self) -> tuple[str, ...]: + """Kahn's algorithm. Ties broken alphabetically so the order is deterministic. + + Determinism matters beyond tidiness: resolution order is observable through which error + surfaces first when several derivations would fail, and a run-to-run reshuffle would make + that irreproducible. + """ + remaining = {node: len(deps) for node, deps in self._dependencies.items()} + ready = deque(sorted(node for node, count in remaining.items() if count == 0)) + order: list[str] = [] + while ready: + node = ready.popleft() + order.append(node) + newly_ready = [] + for dependent in self._dependents[node]: + remaining[dependent] -= 1 + if remaining[dependent] == 0: + newly_ready.append(dependent) + for dependent in sorted(newly_ready): + ready.append(dependent) + if len(order) != len(self._dependencies): + cyclic = sorted(node for node, count in remaining.items() if count > 0) + raise CyclicDependencyError( + f"derived_from contains a cycle involving {cyclic}. No resolution order exists; " + f"fix the metadata rather than breaking the cycle at an arbitrary edge." + ) + return tuple(order) + + +def schema_derived_fields(model_cls: type[RunConfig] = RunConfig) -> tuple[str, ...]: + """Paths the schema marks ``DERIVED``, in topological order. + + Deliberately derived from the PROVENANCE, not from "has dependencies": the two must agree, and + ``studio/tests/unit/test_resolve_registry.py`` asserts that they do. If they ever diverge, that + is a schema bug -- a field with inputs but not marked derived would never be recomputed. + """ + catalogue = field_catalogue(model_cls) + graph = DependencyGraph.from_schema(model_cls) + return tuple( + path for path in graph.nodes if catalogue[path]["provenance"] == Provenance.DERIVED.value + ) + + +__all__ = ["CyclicDependencyError", "DependencyGraph", "schema_derived_fields"] diff --git a/studio/resolve/registry.py b/studio/resolve/registry.py new file mode 100644 index 0000000..055181b --- /dev/null +++ b/studio/resolve/registry.py @@ -0,0 +1,123 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Which function computes which derived field. + +The schema says a field is derived and what from; ``studio/science`` says how. This module is the +one place those two are bound together, and the binding is checked rather than trusted: a test +asserts that every ``DERIVED`` field in the schema has an entry here, and that each entry's declared +inputs are **exactly** the field's ``derived_from``. + +That check is the whole point of the module. Without it, a field could declare inputs the derivation +ignores -- so editing one of them would mark things stale and recompute to the same number -- or a +derivation could read a value the graph does not know about, so editing THAT one would silently +leave a stale result behind. Both are the kind of wrong that looks right. + +Derivations take a mapping of ``{dotted path: value}`` and index it by full path. Positional +arguments would be shorter and would eventually pass ``pressure`` where ``temperature`` belongs. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from typing import Any, Final + +from studio.science import ( + SO2_MOLAR_MASS_G_PER_MOL, + initial_mixing_ratio_pptv, + plume_volume_cm3, +) + + +@dataclass(frozen=True) +class Derivation: + """How to compute one derived field. + + Attributes: + inputs: Dotted paths this derivation reads. Must equal the field's ``derived_from``. + fn: Takes ``{path: value}`` for exactly ``inputs`` and returns the value. + summary: One line for the UI, explaining what the recomputation did. + """ + + inputs: tuple[str, ...] + fn: Callable[[Mapping[str, Any]], Any] + summary: str + + def compute(self, values: Mapping[str, Any]) -> Any: + """Run the derivation, checking that it was handed exactly what it declared. + + Raises: + KeyError: If an input is missing. It means the resolver and this table disagree, which + is a bug in one of them and never something to paper over with a default. + """ + missing = [path for path in self.inputs if path not in values] + if missing: + raise KeyError(f"derivation is missing declared inputs {missing}") + return self.fn({path: values[path] for path in self.inputs}) + + +def _plume_volume(values: Mapping[str, Any]) -> float: + return plume_volume_cm3( + length_m=values["injection.plume_length_m"], + width_m=values["injection.plume_width_m"], + height_m=values["injection.plume_height_m"], + ) + + +def _so2_initial_pptv(values: Mapping[str, Any]) -> float: + """Note this reads the DERIVED ``plume_volume_cm3``, not the three dimensions. + + That is what makes it a chained derivation and the reason resolution runs in topological order: + if the volume were recomputed after this, this would use the previous one and be quietly stale. + """ + return initial_mixing_ratio_pptv( + mass_kg=values["injection.so2_mass_kg"], + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=values["injection.plume_volume_cm3"], + pressure_mbar=values["site.pressure_mbar"], + temperature_k=values["site.temperature_k"], + ) + + +#: Derived field path -> how to compute it. Completeness against the schema is enforced by test. +DERIVATIONS: Final[dict[str, Derivation]] = { + "injection.plume_volume_cm3": Derivation( + inputs=( + "injection.plume_length_m", + "injection.plume_width_m", + "injection.plume_height_m", + ), + fn=_plume_volume, + summary="V0 = length x width x height", + ), + "injection.so2_initial_pptv": Derivation( + inputs=( + "injection.so2_mass_kg", + "injection.plume_volume_cm3", + "site.temperature_k", + "site.pressure_mbar", + ), + fn=_so2_initial_pptv, + summary="injected mass -> number density -> mixing ratio at this site's air density", + ), +} + + +def derivation_for(path: str) -> Derivation: + """The derivation for ``path``. + + Raises: + NotImplementedError: If the schema marks a field derived and nothing here computes it. The + field would otherwise stay ``None`` all the way to the model seam, where it would fail + far from its cause -- or worse, be defaulted (ADR-005). + """ + try: + return DERIVATIONS[path] + except KeyError: + raise NotImplementedError( + f"no derivation registered for {path!r}. The schema marks it DERIVED, so something " + f"must compute it: add it here, with the function itself in studio/science." + ) from None + + +__all__ = ["DERIVATIONS", "Derivation", "derivation_for"] diff --git a/studio/resolve/resolver.py b/studio/resolve/resolver.py new file mode 100644 index 0000000..a25aa3b --- /dev/null +++ b/studio/resolve/resolver.py @@ -0,0 +1,275 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Resolution and override semantics. + +This is the mechanism behind "go back and edit stage 1 without losing your stage 6 choices". It is +a data-model property, not a UI trick: the rules below hold for the CLI and the API equally, and +nothing above this layer needs to reimplement them. + +Three states a derived field can be in: + +* **auto** -- nobody has touched it, so it is recomputed silently when anything upstream changes. +* **user_override** -- somebody typed a value. It is NEVER overwritten by a recomputation. +* **user_override + stale** -- an override whose inputs have since changed, so the value no longer + follows from the rest of the config. + +Staleness is defined against a fingerprint: when an override is set, the upstream values at that +moment are recorded with it. A field is stale when those recorded values differ from the current +ones. That makes staleness a property of the config alone -- no edit history, no ordering +assumptions -- and it is what lets the UI show *the old value, the newly-derived value, and what +changed between them* rather than a bare warning. + +The user then chooses, and both choices are explicit: + +* :func:`accept_derived` -- drop the override, go back to auto. +* :func:`keep_override` -- keep the value and re-anchor its fingerprint to the current inputs. The + field stops being stale because the user has said, knowingly, that their value still applies. + +**A config with stale fields is never quietly persisted.** :meth:`ResolvedConfig.require_consistent` +raises, and the stale list travels with the object so that anything which does persist one has to +carry the list too. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + +from studio.resolve.graph import DependencyGraph, schema_derived_fields +from studio.resolve.registry import derivation_for +from studio.schema import RunConfig + + +class InconsistentConfigError(ValueError): + """A config with stale overrides was asked to behave as if it were consistent. + + Raised rather than resolved automatically, because both resolutions -- discard the user's value + or ignore the changed input -- are decisions only the user can make. + """ + + +class ChangedInput(BaseModel): + """One input whose value moved since an override was anchored.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + path: str + was: Any + now: Any + + +class OverrideRecord(BaseModel): + """A user-supplied value for a derived field, with the inputs it was anchored against.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + value: Any + #: Upstream values when the override was set or last re-anchored. Compared exactly: these are + #: floats copied from the same config, so a tolerance would only hide a real change. + inputs: dict[str, Any] = Field(default_factory=dict) + + +class StaleField(BaseModel): + """An override that no longer follows from the config, and everything needed to decide. + + Carries the newly-derived value as well as the current one, because "this is stale" without + "here is what it would be" leaves the user to recompute it by hand -- which is how a wrong value + gets kept. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + path: str + current_value: Any + derived_value: Any + changed_inputs: tuple[ChangedInput, ...] + summary: str + + +class ResolvedConfig(BaseModel): + """A ``RunConfig`` with its derived fields filled in, plus how they got there.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + config: RunConfig + #: Derived path -> the user's value and its anchor. Absent means auto. + overrides: dict[str, OverrideRecord] = Field(default_factory=dict) + #: Overrides whose inputs have moved. Empty means internally consistent. + stale: tuple[StaleField, ...] = () + + @property + def is_consistent(self) -> bool: + """True when every derived field either was recomputed or is an anchored override.""" + return not self.stale + + @property + def stale_fields(self) -> tuple[str, ...]: + """Just the paths, for a log line or an error message.""" + return tuple(entry.path for entry in self.stale) + + def require_consistent(self) -> None: + """Raise unless the config is internally consistent. + + Called before anything that treats the config as a description of a run -- submission, + hashing for identity, writing it beside results. A stale config still HAS a hash, and that + is exactly the trap: it would be a stable identity for a set of numbers that do not follow + from each other. + """ + if self.stale: + details = "; ".join( + f"{entry.path} = {entry.current_value!r} but now derives to {entry.derived_value!r}" + for entry in self.stale + ) + raise InconsistentConfigError( + f"config has {len(self.stale)} stale override(s): {details}. Resolve each with " + f"accept_derived() or keep_override() before using this config." + ) + + def value_at(self, path: str) -> Any: + """The current value at a dotted path.""" + return _get(self.config.model_dump(mode="python"), path) + + +def resolve( + config: RunConfig, + overrides: Mapping[str, OverrideRecord] | None = None, + *, + graph: DependencyGraph | None = None, +) -> ResolvedConfig: + """Compute every derived field, honouring overrides, and report what has gone stale. + + Derived fields are visited in topological order, so a derivation that reads another derived + field (``so2_initial_pptv`` reads ``plume_volume_cm3``) sees the recomputed value rather than + the previous one. + """ + graph = graph or DependencyGraph.from_schema() + held = dict(overrides or {}) + payload = config.model_dump(mode="python") + stale: list[StaleField] = [] + + for path in schema_derived_fields(): + derivation = derivation_for(path) + inputs = {name: _get(payload, name) for name in derivation.inputs} + derived_value = derivation.compute(inputs) + record = held.get(path) + if record is None: + _set(payload, path, derived_value) + continue + _set(payload, path, record.value) + changed = tuple( + ChangedInput(path=name, was=record.inputs[name], now=value) + for name, value in inputs.items() + if name in record.inputs and record.inputs[name] != value + ) + if changed or set(record.inputs) != set(inputs): + stale.append( + StaleField( + path=path, + current_value=record.value, + derived_value=derived_value, + changed_inputs=changed, + summary=derivation.summary, + ) + ) + return ResolvedConfig( + config=RunConfig.model_validate(payload), overrides=held, stale=tuple(stale) + ) + + +def apply_change(resolved: ResolvedConfig, path: str, value: Any) -> ResolvedConfig: + """Set ``path`` to ``value`` and recompute the downstream closure. + + A change to a DERIVED field is an override -- that is what a user typing into a computed box + means -- so it is routed to :func:`set_override` rather than being silently recomputed away on + the next edit. + """ + graph = DependencyGraph.from_schema() + if path in set(schema_derived_fields()): + return set_override(resolved, path, value, graph=graph) + graph.dependents_of(path) # validates the path against the schema, and raises if unknown + payload = resolved.config.model_dump(mode="python") + _set(payload, path, value) + return resolve(RunConfig.model_validate(payload), resolved.overrides, graph=graph) + + +def set_override( + resolved: ResolvedConfig, path: str, value: Any, *, graph: DependencyGraph | None = None +) -> ResolvedConfig: + """Pin ``path`` to a user-supplied ``value``, anchored to the config's current inputs. + + Anchoring at the moment of the override is what makes it non-stale now and detectably stale + later. + """ + graph = graph or DependencyGraph.from_schema() + derived = set(schema_derived_fields()) + if path not in derived: + raise ValueError( + f"{path!r} is not a derived field, so it cannot be overridden -- set it directly with " + f"apply_change(). Derived fields: {sorted(derived)}" + ) + payload = resolved.config.model_dump(mode="python") + inputs = {name: _get(payload, name) for name in derivation_for(path).inputs} + overrides = dict(resolved.overrides) + overrides[path] = OverrideRecord(value=value, inputs=inputs) + return resolve(resolved.config, overrides, graph=graph) + + +def accept_derived(resolved: ResolvedConfig, path: str) -> ResolvedConfig: + """Drop the override at ``path``; the field goes back to auto and is recomputed.""" + if path not in resolved.overrides: + raise ValueError(f"{path!r} is not overridden, so there is nothing to accept") + overrides = {key: record for key, record in resolved.overrides.items() if key != path} + return resolve(resolved.config, overrides) + + +def keep_override(resolved: ResolvedConfig, path: str) -> ResolvedConfig: + """Keep the user's value at ``path`` and re-anchor it to the current inputs. + + The field stops being stale because the user has confirmed it still applies -- knowingly, which + is the difference between this and never having flagged it. + """ + record = resolved.overrides.get(path) + if record is None: + raise ValueError(f"{path!r} is not overridden, so there is nothing to keep") + payload = resolved.config.model_dump(mode="python") + inputs = {name: _get(payload, name) for name in derivation_for(path).inputs} + overrides = dict(resolved.overrides) + overrides[path] = OverrideRecord(value=record.value, inputs=inputs) + return resolve(resolved.config, overrides) + + +def downstream_of(paths: Iterable[str]) -> tuple[str, ...]: + """Fields that must be recomputed when ``paths`` change. Convenience over the schema graph.""" + return DependencyGraph.from_schema().downstream_of(paths) + + +def _get(payload: Mapping[str, Any], path: str) -> Any: + cursor: Any = payload + for part in path.split("."): + cursor = cursor[part] + return cursor + + +def _set(payload: dict[str, Any], path: str, value: Any) -> None: + parts = path.split(".") + cursor: dict[str, Any] = payload + for part in parts[:-1]: + cursor = cursor[part] + cursor[parts[-1]] = value + + +__all__ = [ + "ChangedInput", + "InconsistentConfigError", + "OverrideRecord", + "ResolvedConfig", + "StaleField", + "accept_derived", + "apply_change", + "downstream_of", + "keep_override", + "resolve", + "set_override", +] diff --git a/studio/tests/unit/test_import_boundaries.py b/studio/tests/unit/test_import_boundaries.py index 2df793e..20765c6 100644 --- a/studio/tests/unit/test_import_boundaries.py +++ b/studio/tests/unit/test_import_boundaries.py @@ -2,11 +2,11 @@ # SPDX-License-Identifier: Apache-2.0 """The package boundaries from ADR-001, enforced rather than documented. -``studio.schema`` and ``studio.science`` must be usable from a bare Python session: no ``coupled``, -no JAX, no database, no web framework. This is not tidiness. Constructing a ``CoupledScenario`` -imports JAX transitively -- ``coupled_scenario.__post_init__`` validates ``background_dist`` against -``coupled.tomas_bridge``, which sets ``jax_enable_x64`` at import -- and an API that validates a -form on every keystroke cannot pay a JAX import. +``studio.schema``, ``studio.science`` and ``studio.resolve`` must be usable from a bare Python +session: no ``coupled``, no JAX, no database, no web framework. This is not tidiness. Constructing +a ``CoupledScenario`` imports JAX transitively -- ``coupled_scenario.__post_init__`` validates +``background_dist`` against ``coupled.tomas_bridge``, which sets ``jax_enable_x64`` at import -- +and an API that validates a form on every keystroke cannot pay a JAX import. Two complementary checks: @@ -35,7 +35,8 @@ STUDIO_ROOT = REPO_ROOT / "studio" #: Packages that must stay importable without the model. See ``studio/__init__.py``. -CLEAN_PACKAGES = ["studio.schema", "studio.science"] +#: ``studio.resolve`` is here because the API resolves a config on every keystroke. +CLEAN_PACKAGES = ["studio.schema", "studio.science", "studio.resolve"] #: The single package permitted to import ``coupled`` (ADR-001). Adding to this set requires #: amending ADR-001 in the same change. @@ -102,7 +103,7 @@ def test_clean_packages_do_not_import_the_model(package: str) -> None: leaked = _forbidden_imports_after(package) assert leaked == [], ( f"{package} imported {leaked}, breaking the boundary in ADR-001. " - f"studio.schema and studio.science must be usable from a bare Python session; " + f"{CLEAN_PACKAGES} must be usable from a bare Python session; " f"only {sorted(MODEL_SEAM_PACKAGES)} may import `coupled`." ) diff --git a/studio/tests/unit/test_resolve_graph.py b/studio/tests/unit/test_resolve_graph.py new file mode 100644 index 0000000..977742c --- /dev/null +++ b/studio/tests/unit/test_resolve_graph.py @@ -0,0 +1,131 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The dependency graph, on synthetic graphs and on the real schema. + +Most of these run on hand-built graphs rather than on ``RunConfig``. That is deliberate: the +properties under test (transitivity, ordering, cycle detection) are properties of the algorithm, and +the schema currently has exactly one chain of length two. A graph engine tested only against the +shape it happens to be used with is a graph engine that breaks the first time the schema grows. +""" + +from __future__ import annotations + +import pytest + +from studio.resolve import CyclicDependencyError, DependencyGraph, schema_derived_fields + +#: a -> b -> d, a -> c -> d, and an isolated node. +DIAMOND = {"a": (), "b": ("a",), "c": ("a",), "d": ("b", "c"), "lonely": ()} + + +@pytest.mark.tier_a +def test_dependents_are_one_hop() -> None: + graph = DependencyGraph(DIAMOND) + assert set(graph.dependents_of("a")) == {"b", "c"} + assert graph.dependents_of("d") == () + assert graph.dependencies_of("d") == ("b", "c") + assert graph.dependencies_of("a") == () + + +@pytest.mark.tier_a +def test_downstream_is_transitive_and_excludes_the_seed() -> None: + """The recompute set. Transitive, because a one-hop answer leaves the far end silently stale.""" + graph = DependencyGraph(DIAMOND) + assert graph.downstream_of(["a"]) == ("b", "c", "d") + assert graph.downstream_of(["b"]) == ("d",) + assert graph.downstream_of(["d"]) == () + assert graph.downstream_of(["lonely"]) == () + + +@pytest.mark.tier_a +def test_downstream_of_several_seeds_is_the_union_without_duplicates() -> None: + """Editing two fields at once must not recompute the shared descendant twice.""" + graph = DependencyGraph(DIAMOND) + assert graph.downstream_of(["b", "c"]) == ("d",) + + +@pytest.mark.tier_a +def test_downstream_is_in_topological_order() -> None: + """Ordering is the contract: a dependent must never be computed before its dependency.""" + graph = DependencyGraph({"x": (), "mid": ("x",), "far": ("mid",)}) + assert graph.downstream_of(["x"]) == ("mid", "far") + + +@pytest.mark.tier_a +def test_topological_order_is_deterministic() -> None: + """Ties broken alphabetically, so two runs of the same engine order identically.""" + first = DependencyGraph(DIAMOND).nodes + second = DependencyGraph(dict(reversed(list(DIAMOND.items())))).nodes + assert first == second == ("a", "lonely", "b", "c", "d") + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + "graph", + [ + {"a": ("a",)}, # self-edge + {"a": ("b",), "b": ("a",)}, # two-cycle + {"a": ("c",), "b": ("a",), "c": ("b",)}, # three-cycle + {"ok": (), "a": ("b",), "b": ("a",)}, # cycle alongside a healthy node + ], +) +def test_cycles_raise_rather_than_hang(graph: dict[str, tuple[str, ...]]) -> None: + """No fixed-point iteration, no arbitrary edge-breaking: both would make the result depend on + where the engine started.""" + with pytest.raises(CyclicDependencyError, match="cycle"): + DependencyGraph(graph) + + +@pytest.mark.tier_a +def test_a_dependency_on_an_unknown_node_raises() -> None: + """A ``derived_from`` naming a field that does not exist is a typo that drops an edge.""" + with pytest.raises(ValueError, match="not a node in the graph"): + DependencyGraph({"a": ("ghost",)}) + + +@pytest.mark.tier_a +def test_querying_an_unknown_path_raises() -> None: + graph = DependencyGraph(DIAMOND) + with pytest.raises(ValueError, match="unknown field path"): + graph.dependents_of("nope") + with pytest.raises(ValueError, match="unknown field path"): + graph.downstream_of(["nope"]) + + +@pytest.mark.tier_a +def test_the_real_schema_graph_is_acyclic_and_complete() -> None: + """Building it is the assertion: a cycle or a dangling ``derived_from`` raises here.""" + graph = DependencyGraph.from_schema() + from studio.schema import field_catalogue + + assert set(graph.nodes) == set(field_catalogue()) + + +@pytest.mark.tier_a +def test_the_schemas_derived_chain() -> None: + """``so2_initial_pptv`` depends on ``plume_volume_cm3``, which is itself derived. + + Pinned because it is the case that makes topological order matter rather than be decoration: a + resolver that recomputed in declaration order could use last round's volume. + """ + graph = DependencyGraph.from_schema() + assert schema_derived_fields() == ( + "injection.plume_volume_cm3", + "injection.so2_initial_pptv", + ) + assert "injection.plume_volume_cm3" in graph.dependencies_of("injection.so2_initial_pptv") + assert graph.downstream_of(["injection.plume_length_m"]) == ( + "injection.plume_volume_cm3", + "injection.so2_initial_pptv", + ) + assert graph.downstream_of(["site.temperature_k"]) == ("injection.so2_initial_pptv",) + + +@pytest.mark.tier_a +def test_primary_fields_have_no_dependencies() -> None: + """Everything the user types is a root; only DERIVED fields have inputs.""" + graph = DependencyGraph.from_schema() + derived = set(schema_derived_fields()) + for path in graph.nodes: + if path not in derived: + assert graph.dependencies_of(path) == (), f"{path} is primary but has dependencies" diff --git a/studio/tests/unit/test_resolve_registry.py b/studio/tests/unit/test_resolve_registry.py new file mode 100644 index 0000000..b3c4919 --- /dev/null +++ b/studio/tests/unit/test_resolve_registry.py @@ -0,0 +1,106 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The registry must agree with the schema, exactly. + +Two failure modes this guards, both of which produce a config that looks resolved and is not: + +* a field declares an input the derivation ignores -- editing it marks things stale and recomputes + to the same number, so the UI reports a change that did not happen; +* a derivation reads a value the schema does not list -- editing THAT one recomputes nothing, and + the stale result reaches the model. +""" + +from __future__ import annotations + +import pytest + +from studio.resolve import DERIVATIONS, derivation_for, schema_derived_fields +from studio.schema import field_catalogue + + +@pytest.mark.tier_a +def test_every_derived_field_has_a_derivation() -> None: + """A DERIVED field with nothing to compute it stays None all the way to the model seam.""" + missing = [path for path in schema_derived_fields() if path not in DERIVATIONS] + assert ( + missing == [] + ), f"schema fields marked DERIVED with no entry in studio/resolve/registry.py: {missing}" + + +@pytest.mark.tier_a +def test_no_derivation_exists_for_a_field_the_schema_does_not_derive() -> None: + """The reverse direction: a stale registry entry would never run and would rot unnoticed.""" + extra = [path for path in DERIVATIONS if path not in set(schema_derived_fields())] + assert ( + extra == [] + ), f"registered derivations for fields the schema does not mark DERIVED: {extra}" + + +@pytest.mark.tier_a +def test_declared_inputs_match_derived_from_exactly() -> None: + """Order-insensitive, but membership must be identical -- this is the DAG's correctness.""" + catalogue = field_catalogue() + for path in schema_derived_fields(): + schema_inputs = set(catalogue[path]["derived_from"]) + registry_inputs = set(derivation_for(path).inputs) + assert registry_inputs == schema_inputs, ( + f"{path}: schema says it derives from {sorted(schema_inputs)}, the registry reads " + f"{sorted(registry_inputs)}. The graph and the computation must agree or the recompute " + f"set is wrong in one direction or the other." + ) + + +@pytest.mark.tier_a +def test_an_unregistered_field_raises_rather_than_returning_none() -> None: + with pytest.raises(NotImplementedError, match="no derivation registered"): + derivation_for("site.temperature_k") + + +@pytest.mark.tier_a +def test_a_derivation_refuses_inputs_it_did_not_declare() -> None: + """A missing input means resolver and registry disagree; never default it away.""" + derivation = derivation_for("injection.plume_volume_cm3") + with pytest.raises(KeyError, match="missing declared inputs"): + derivation.compute({"injection.plume_length_m": 15000.0}) + + +@pytest.mark.tier_a +def test_derivations_produce_the_same_values_as_studio_science_directly() -> None: + """The registry is a binding, not a second implementation. Tolerance: exact.""" + from studio.science import ( + SO2_MOLAR_MASS_G_PER_MOL, + initial_mixing_ratio_pptv, + plume_volume_cm3, + ) + + volume = derivation_for("injection.plume_volume_cm3").compute( + { + "injection.plume_length_m": 15000.0, + "injection.plume_width_m": 10.0, + "injection.plume_height_m": 10.0, + } + ) + assert volume == plume_volume_cm3(15000.0, 10.0, 10.0) + + pptv = derivation_for("injection.so2_initial_pptv").compute( + { + "injection.so2_mass_kg": 1000.0, + "injection.plume_volume_cm3": volume, + "site.temperature_k": 210.0, + "site.pressure_mbar": 55.0, + } + ) + assert pptv == initial_mixing_ratio_pptv( + mass_kg=1000.0, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=volume, + pressure_mbar=55.0, + temperature_k=210.0, + ) + + +@pytest.mark.tier_a +def test_every_derivation_carries_a_summary() -> None: + """The UI has to say what it recomputed and why; an empty string is not an explanation.""" + for path in schema_derived_fields(): + assert derivation_for(path).summary.strip(), f"{path} has no summary" diff --git a/studio/tests/unit/test_resolve_resolver.py b/studio/tests/unit/test_resolve_resolver.py new file mode 100644 index 0000000..654daae --- /dev/null +++ b/studio/tests/unit/test_resolve_resolver.py @@ -0,0 +1,257 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Resolution and override semantics -- the "edit stage 1 without losing stage 6" guarantee. + +The load-bearing test is ``test_an_edit_changes_exactly_the_downstream_closure``: it captures every +field before and after an edit and asserts the set that moved is EXACTLY the edited field plus its +closure. Both failure directions matter and both are silent -- recomputing too little leaves a stale +number that reaches the model, recomputing too much quietly discards something the user set. +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from studio.resolve import ( + InconsistentConfigError, + accept_derived, + apply_change, + keep_override, + resolve, + set_override, +) +from studio.schema import RunConfig, field_catalogue + +SO2_PPTV = "injection.so2_initial_pptv" +VOLUME = "injection.plume_volume_cm3" + + +def _flat(resolved: Any) -> dict[str, Any]: + """Every leaf value, by path, for before/after comparison.""" + return {path: resolved.value_at(path) for path in field_catalogue()} + + +@pytest.mark.tier_a +def test_a_fresh_config_resolves_its_derived_fields() -> None: + """The schema ships them unset (0.2 declares, 0.3 resolves); this is where they get values.""" + assert RunConfig().injection.plume_volume_cm3 is None + resolved = resolve(RunConfig()) + assert resolved.config.injection.plume_volume_cm3 == 1.5e12 + assert resolved.config.injection.so2_initial_pptv == pytest.approx( + 3.309115922996412e9, rel=1e-15 + ) + assert resolved.is_consistent + assert resolved.overrides == {} + + +@pytest.mark.tier_a +def test_chained_derivations_resolve_in_order() -> None: + """``so2_initial_pptv`` must see the NEW volume, not the previous one. + + Halving the track length halves V0 and therefore doubles nothing -- it halves the concentration. + A resolver running in declaration order rather than topological order would return the old + value here, which is why this is asserted numerically rather than structurally. + """ + resolved = apply_change(resolve(RunConfig()), "injection.plume_length_m", 30000.0) + assert resolved.config.injection.plume_volume_cm3 == 3.0e12 + assert resolved.config.injection.so2_initial_pptv == pytest.approx( + 1.654557961498206e9, rel=1e-15 + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("path", "value", "expected_changed"), + [ + ("injection.plume_length_m", 30000.0, {VOLUME, SO2_PPTV}), + ("injection.plume_width_m", 20.0, {VOLUME, SO2_PPTV}), + ("site.temperature_k", 213.0, {SO2_PPTV}), + ("site.pressure_mbar", 120.0, {SO2_PPTV}), + ("injection.so2_mass_kg", 2000.0, {SO2_PPTV}), + ("microphysics.n_bins", 40, set()), + ("chemistry.so2_ho2_rate", 1e-16, set()), + ("switches.heating_to_t", True, set()), + ], +) +def test_an_edit_changes_exactly_the_downstream_closure( + path: str, value: Any, expected_changed: set[str] +) -> None: + """Not "at least" and not "at most". Exactly. + + Under-recomputing leaves a stale number that reaches the model; over-recomputing silently + discards a value the user chose. The last three cases matter as much as the first five: editing + a field with no dependents must move nothing else at all. + """ + before_resolved = resolve(RunConfig()) + before = _flat(before_resolved) + after = _flat(apply_change(before_resolved, path, value)) + changed = {key for key in before if before[key] != after[key]} + assert changed == expected_changed | {path} + + +@pytest.mark.tier_a +def test_an_override_is_never_overwritten_by_a_recomputation() -> None: + """The stage-6 choice survives the stage-1 edit. This is the whole point of the task.""" + overridden = set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9) + assert overridden.config.injection.so2_initial_pptv == 5.0e9 + assert overridden.is_consistent, "an override anchored to the current inputs is not stale" + + after_edit = apply_change(overridden, "site.temperature_k", 213.0) + assert after_edit.config.injection.so2_initial_pptv == 5.0e9 + + +@pytest.mark.tier_a +def test_a_moved_input_marks_the_override_stale_with_both_values() -> None: + """ "Stale" without "here is what it would be" leaves the user to recompute by hand.""" + overridden = set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9) + after_edit = apply_change(overridden, "site.temperature_k", 213.0) + + assert after_edit.stale_fields == (SO2_PPTV,) + (entry,) = after_edit.stale + assert entry.current_value == 5.0e9 + assert entry.derived_value == pytest.approx(3.3563890076106462e9, rel=1e-15) + assert entry.summary + assert [(c.path, c.was, c.now) for c in entry.changed_inputs] == [ + ("site.temperature_k", 210.0, 213.0) + ] + + +@pytest.mark.tier_a +def test_an_unrelated_edit_does_not_make_an_override_stale() -> None: + """Only a change to one of ITS inputs counts. Flagging on every edit would train users to + dismiss the flag.""" + overridden = set_override(resolve(RunConfig()), VOLUME, 3.0e12) + after = apply_change(overridden, "site.temperature_k", 213.0) + assert after.is_consistent + assert after.config.injection.plume_volume_cm3 == 3.0e12 + + +@pytest.mark.tier_a +def test_a_downstream_auto_field_uses_the_overridden_value() -> None: + """An override is the value in force, so anything computed from it must use it.""" + overridden = set_override(resolve(RunConfig()), VOLUME, 3.0e12) + assert overridden.config.injection.so2_initial_pptv == pytest.approx( + 1.654557961498206e9, rel=1e-15 + ), "so2_initial_pptv must be computed from the overridden V0, not the geometric one" + + +@pytest.mark.tier_a +def test_accept_derived_drops_the_override_and_recomputes() -> None: + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + accepted = accept_derived(stale, SO2_PPTV) + assert accepted.is_consistent + assert accepted.overrides == {} + assert accepted.config.injection.so2_initial_pptv == pytest.approx( + 3.3563890076106462e9, rel=1e-15 + ) + + +@pytest.mark.tier_a +def test_keep_override_re_anchors_and_clears_staleness() -> None: + """The user has said, knowingly, that their value still applies -- that is the difference + between this and never having flagged it.""" + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + kept = keep_override(stale, SO2_PPTV) + assert kept.is_consistent + assert kept.config.injection.so2_initial_pptv == 5.0e9 + assert kept.overrides[SO2_PPTV].inputs["site.temperature_k"] == 213.0 + + # ...and it goes stale again on the NEXT change, rather than being permanently silenced + assert apply_change(kept, "site.temperature_k", 220.0).stale_fields == (SO2_PPTV,) + + +@pytest.mark.tier_a +def test_a_stale_config_refuses_to_pass_as_consistent() -> None: + """A stale config still has a hash, and that is the trap: a stable identity for numbers that do + not follow from each other.""" + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + with pytest.raises(InconsistentConfigError, match="stale override"): + stale.require_consistent() + assert resolve(RunConfig()).require_consistent() is None + + +@pytest.mark.tier_a +def test_the_stale_list_travels_with_the_config() -> None: + """Serialising must carry the stale list, or a persisted config loses the fact that it is + inconsistent -- exactly what the plan forbids.""" + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + restored = type(stale).model_validate_json(stale.model_dump_json()) + assert restored.stale_fields == (SO2_PPTV,) + assert restored.overrides[SO2_PPTV].value == 5.0e9 + with pytest.raises(InconsistentConfigError): + restored.require_consistent() + + +@pytest.mark.tier_a +def test_editing_a_derived_field_directly_is_an_override() -> None: + """A user typing into a computed box means "I want this value", not "recompute me away".""" + edited = apply_change(resolve(RunConfig()), VOLUME, 2.0e12) + assert edited.overrides[VOLUME].value == 2.0e12 + assert edited.config.injection.plume_volume_cm3 == 2.0e12 + + +@pytest.mark.tier_a +def test_overriding_a_primary_field_is_refused() -> None: + """Primary fields have no derivation to be stale against; the concept does not apply.""" + with pytest.raises(ValueError, match="not a derived field"): + set_override(resolve(RunConfig()), "site.temperature_k", 999.0) + + +@pytest.mark.tier_a +def test_settling_a_field_that_is_not_overridden_is_refused() -> None: + resolved = resolve(RunConfig()) + with pytest.raises(ValueError, match="nothing to accept"): + accept_derived(resolved, SO2_PPTV) + with pytest.raises(ValueError, match="nothing to keep"): + keep_override(resolved, SO2_PPTV) + + +@pytest.mark.tier_a +def test_an_unknown_path_raises() -> None: + with pytest.raises(ValueError, match="unknown field"): + apply_change(resolve(RunConfig()), "site.temprature_k", 210.0) + + +@pytest.mark.tier_a +def test_an_invalid_value_is_rejected_by_the_schema_during_resolution() -> None: + """Resolution does not bypass validation: bounds still apply to an edited value.""" + with pytest.raises(ValueError, match="condensation_alpha"): + apply_change(resolve(RunConfig()), "microphysics.condensation_alpha", 1.5) + + +@pytest.mark.tier_a +def test_a_degenerate_input_is_caught_by_the_schema_before_the_derivation_runs() -> None: + """A zero plume dimension is rejected at validation, not deep in the arithmetic. + + Both layers refuse it -- ``plume_volume_cm3`` raises on a non-positive dimension too (see + ``test_science_plume.py``) -- but the schema's ``gt=0`` fires first, which is the better place: + the error names the field the user typed in rather than a function they have never heard of. + The derivation's own check remains as the guard for any caller that does not come through the + schema. + """ + with pytest.raises(ValueError, match="plume_length_m"): + apply_change(resolve(RunConfig()), "injection.plume_length_m", 0.0) + + +@pytest.mark.tier_a +def test_resolution_does_not_import_the_model() -> None: + """The API resolves on every keystroke; a JAX import on that path would be unaffordable. + + The static and runtime boundary checks live in test_import_boundaries.py; this one asserts the + same thing about the actual call path, which is what the cost is attached to. + """ + import sys + + resolve(RunConfig()) + assert "coupled" not in sys.modules + assert "jax" not in sys.modules From 8c12422721debf9c8f5bba7bfe485df9c0e9bce7 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 19:04:42 -0700 Subject: [PATCH 05/18] studio: the model seam and RunSummary (#69) Task 0.4 (#68). studio/modelio: scenario.py converts RunConfig to CoupledScenario, and summary.py reduces a state.npz to a versioned RunSummary. 23 new Tier-A tests, 171 total. The equivalence test passes: to_scenario(resolve(RunConfig())) is field-for-field identical to run_ensemble.build_scenario() for the golden case, compared as dataclasses.asdict with exact equality on every field including the floats. The schema is a faithful superset of what the ensemble ran, proven before any run rather than discovered from a diverging result. The derived SO2 initial concentration matches to the last bit, which is the evidence that consolidating that derivation in 0.5 changed nothing. to_scenario is deliberately not re-exported from the package. Measured: importing studio.modelio.scenario costs ~0.13 s and pulls in no JAX; the first CALL costs ~1.05 s, because CoupledScenario.__post_init__ imports tomas_bridge to validate background_dist (coupled_scenario.py:196). A comparison view reading a RunSummary should not pay for a model it is not using, and the expensive import should be visible at the import site. RunSummary encodes the traps rather than documenting them: every series declares whether it is on a wet or dry basis, species are indexed by name from the npz's own species list, and the time axis is the stored t. The synthetic test archive orders species so SO2 sits at index 2 -- nothing like the 32/34/35 an existing script hard-codes -- so a positional reduction would report ozone as SO2, plausibly and silently. Termination is an argument, never inferred: the npz records what the state did, not why the loop stopped. The conservation check refuses to report a number when one would mislead. With dilution on the box is an open system, so a large residual would be measuring the dilution and a small one would mean something was wrong; the check returns not_applicable with the reason and the start/end values, and a closed box gets a real residual. Also fixed: a sum() over a generator starting at integer 0 that mypy caught; mypy reporting errors from the model's own untyped source once modelio imported it (follow_imports = "silent" on the model modules -- Studio's use is still checked); and a test of mine that inspected sys.modules in-process, which is the exact mistake test_import_boundaries.py warns about and now runs in a fresh interpreter. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 59 +++- docs/studio/plan/PHASE_0.md | 2 +- pyproject.toml | 16 +- studio/modelio/__init__.py | 59 +++- studio/modelio/scenario.py | 128 +++++++ studio/modelio/summary.py | 320 ++++++++++++++++++ studio/tests/unit/test_import_boundaries.py | 7 +- studio/tests/unit/test_modelio_equivalence.py | 175 ++++++++++ studio/tests/unit/test_modelio_summary.py | 236 +++++++++++++ studio/tests/unit/test_resolve_resolver.py | 31 +- 10 files changed, 1010 insertions(+), 23 deletions(-) create mode 100644 studio/modelio/scenario.py create mode 100644 studio/modelio/summary.py create mode 100644 studio/tests/unit/test_modelio_equivalence.py create mode 100644 studio/tests/unit/test_modelio_summary.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 12df06c..7cbf3f2 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -17,7 +17,7 @@ with full provenance, and the golden tests pass. | 0.1 Repo, CI, docs skeleton | **done** | | 0.2 `studio/schema` v0 — **review gate** | **done** (#62, reviewed) | | 0.3 Dependency-graph engine + override semantics | **done** (#66) | -| 0.4 `studio/modelio` seam + `RunSummary` | not started | +| 0.4 `studio/modelio` seam + `RunSummary` | **done** (#68) | | 0.5 `studio/science` derivations | **done** (#64) | | 0.6 `studio/runner` + job lifecycle | not started | | 0.7 Golden-file harness (two tiers) | not started | @@ -29,6 +29,63 @@ derivations to resolve rather than fixtures. --- +### 2026-08-13 — Task 0.4: the model seam and `RunSummary` (issue #68) + +`studio/modelio/`: `scenario.py` (the seam) and `summary.py` (the reduction). 23 new Tier-A tests +(171 total). + +**The equivalence test passes.** `to_scenario(resolve(RunConfig()))` is **field-for-field identical** +to `run_ensemble.build_scenario()` for `30N_20km__sabr220__D2med__a1p0__nuc1__cg1`, compared as +`dataclasses.asdict` with exact equality on every field including the floats. The schema is a +faithful superset of what the ensemble ran, and that is now proven rather than intended — before any +run, instead of via a diverging result days later. The derived `SO2` initial concentration matches +to the last bit, which is the evidence that consolidating that derivation in 0.5 changed nothing. + +**Measured, because the cost is not where anyone would guess.** Importing +`studio.modelio.scenario` takes ~0.13 s and pulls in **no JAX at all**. The first `to_scenario()` +*call* takes ~1.05 s, because `CoupledScenario.__post_init__` imports `coupled.tomas_bridge` to +validate `background_dist` (`coupled_scenario.py:196`) and *that* is what loads JAX; later calls are +free. So `to_scenario` is deliberately **not** re-exported from `studio/modelio/__init__.py` — +a comparison view reading a `RunSummary` should not pay for a model it is not using, and the +expensive import should be visible at the import site. Task 0.8 tracks making the model's import +lazy. + +**`RunSummary`** — versioned, and self-describing about the three traps: + +- **Every series declares its basis.** `SA`/`radius_cm` are WET, `dp_mid_um`/`dNdlogDp`/`total_n` + are DRY, gas mixing ratios are not-applicable. It is a required field, so a plot axis cannot be + labelled by guesswork. +- **Species are indexed by name** from the npz's own `species` list. The synthetic test archive + deliberately orders species so SO2 sits at index 2 — nothing like the 32/34/35 an existing script + hard-codes — so a positional reduction would report ozone as SO2, plausibly and silently. +- **The time axis is the stored `t`**, never `i × DT`. +- **Termination is an argument, never inferred.** The npz records what the state did, not why the + loop stopped; a run that hit a wall-clock limit and one that finished look identical in it. + Archived runs are `UNKNOWN` and flagged `NO_PROVENANCE_RECORD` (ADR-006). + +**The conservation check refuses to report a number when one would mislead.** In-box sulfur is not +expected to be conserved with dilution on — the box is an open system, so a large "residual" would +be measuring the dilution and a small one would mean something was wrong. When `V(t)/V0 > 1` the +check returns `not_applicable` with the reason and the start/end values, so the decay is visible +without being dressed up as a budget error. A closed box gets a real residual. + +**Found and fixed while writing it** + +- A `sum()` over a generator starting at integer `0`, which mypy caught: the sulfur total was + `ndarray | Literal[0]` and would have been unindexable had the species list ever been empty. +- Once `studio/modelio` imported `coupled`, `mypy` began reporting errors from the **model's own** + source (its untyped `yaml`, `scipy`, and flat `aerosol` imports). Fixed with + `follow_imports = "silent"` on the model modules — Studio's use of them is still checked; the + model is not ours to annotate. +- Two of my own test constants were wrong: an "irregular" time axis whose last point was exactly + `4 × 600 s` (so it proved nothing about nominal grids), and a closed-box sulfur budget that did + not close. Both were caught by the tests failing, which is the system working. +- I wrote a test checking `sys.modules` in-process for the resolver — the exact mistake + `test_import_boundaries.py`'s own docstring warns about, and it failed as soon as another module + imported the seam. It now runs in a fresh interpreter, where it is meaningful. + +--- + ### 2026-08-13 — Task 0.3: dependency graph and override semantics (issue #66) **New package: `studio/resolve/`** — `graph.py` (the DAG), `registry.py` (which function computes diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index 30222e3..249a3c7 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -68,7 +68,7 @@ construction. It is a data-model problem, not a UI problem. --- -## 0.4 — `studio/modelio` seam + `RunSummary` +## 0.4 — `studio/modelio` seam + `RunSummary` · *done* (issue #68) - `to_scenario(RunConfig) -> CoupledScenario` — the single conversion point, and the **only** package permitted to import `coupled`. diff --git a/pyproject.toml b/pyproject.toml index df9b25f..64aae9b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -121,5 +121,19 @@ check_untyped_defs = true [[tool.mypy.overrides]] # The model is untyped and is not ours to annotate; studio/modelio is the only importer (ADR-001). -module = ["coupled.*", "config", "tomas_jax.*", "tuvx_photolysis.*"] +# `follow_imports = "silent"` matters as much as ignore_missing_imports: once studio/modelio imports +# `coupled`, mypy would otherwise report errors from the MODEL's source (its own untyped deps -- +# yaml, scipy, the flat `aerosol` module), which are not Studio's to fix and would make the check +# unusable. Studio's own use of those imports is still checked. +module = [ + "coupled.*", + "config", + "aerosol", + "background_aerosol_distribution", + "tomas_jax.*", + "tuvx_photolysis.*", + "yaml", + "scipy.*", +] ignore_missing_imports = true +follow_imports = "silent" diff --git a/studio/modelio/__init__.py b/studio/modelio/__init__.py index 8f5a26e..f5675b3 100644 --- a/studio/modelio/__init__.py +++ b/studio/modelio/__init__.py @@ -5,31 +5,66 @@ THE ONLY PACKAGE PERMITTED TO IMPORT ``coupled`` (ADR-001). If Studio is ever extracted to its own repository, this is the seam to sever. -Contents (task 0.4, not yet implemented): +* ``scenario.py`` -- ``to_scenario(RunConfig) -> CoupledScenario``, the single conversion point and + therefore the single place unit conversion or name translation happens (ADR-003). +* ``summary.py`` -- ``RunSummary``, the versioned reduction of a run. Comparison plots and figures + read this; they never read the raw npz (ADR-004). It does not import ``coupled`` -- it only reads + arrays -- but reading the model's output format is this package's job. -* ``to_scenario(RunConfig) -> CoupledScenario`` -- the single conversion point, and therefore the - single place unit conversion happens (ADR-003). -* ``RunSummary`` -- the versioned, queryable reduction of a run: scalar time series, final size - distribution, integrated diagnostics, termination reason, flags, conservation residuals. - Comparison plots and figures read this; they never read the raw npz (ADR-004). - -The equivalence test is the point of this package: ``to_scenario`` applied to a ``RunConfig`` -describing case ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` must produce a ``CoupledScenario`` +The equivalence test is the point of ``scenario.py``: ``to_scenario`` applied to a ``RunConfig`` +describing case ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` produces a ``CoupledScenario`` FIELD-FOR-FIELD IDENTICAL to ``run_ensemble.build_scenario()`` for that case. That proves the schema -is a faithful superset before anything is run. +is a faithful superset before anything is run, which is cheaper and sharper than discovering it from +a diverging result days later. Traps this package exists to contain (see docs/studio/CAVEATS.md), each an assertion here: * ``state.npz`` carries BOTH dry and wet quantities: ``dp_mid_um`` / ``dNdlogDp`` are dry; - ``SA`` / ``radius_cm`` are wet. Every RunSummary array declares its basis. + ``SA`` / ``radius_cm`` are wet. Every RunSummary series declares its basis. * Never reconstruct the time axis as ``i * DT``. Outer intervals snap to the terminator, so the mean step is ~592 s against a nominal 600 s -- ~1.4% drift, about 0.5 days by day 36. Use stored ``t``. * Any resampling is UNIFORM in time. A coarsening grid aliases morning particle-number spikes by up to 8x. * Index gas species by NAME using the npz's own ``species`` list, never by position. (An existing analysis script hardcodes ``_SO2, _SO3, _H2SO4 = 32, 34, 35``; that is the failure to avoid.) + +**``to_scenario`` is deliberately NOT re-exported here.** Import it as +``from studio.modelio.scenario import to_scenario``. Re-exporting it would make every comparison +view that reads a ``RunSummary`` pay for the model it is not using. + +Measured, because the cost is not where one would guess: importing ``studio.modelio.scenario`` +takes ~0.13 s and pulls in no JAX at all. The first ``to_scenario()`` CALL takes ~1.05 s, because +``CoupledScenario.__post_init__`` imports ``coupled.tomas_bridge`` to validate ``background_dist`` +(``coupled_scenario.py:196``) and that is what loads JAX. Subsequent calls are free. So the expense +belongs to constructing a scenario, not to importing this package -- which is precisely why an API +that validates a form on every keystroke must stay on the schema side of this seam. Task 0.8 tracks +making that import lazy in the model. """ from __future__ import annotations -__all__: list[str] = [] +from studio.modelio.summary import ( + SECONDS_PER_DAY, + SUMMARY_SCHEMA_VERSION, + Basis, + ConservationCheck, + RunSummary, + Series, + SizeDistribution, + SummaryFlag, + TerminationReason, + summarise_state_npz, +) + +__all__ = [ + "SECONDS_PER_DAY", + "SUMMARY_SCHEMA_VERSION", + "Basis", + "ConservationCheck", + "RunSummary", + "Series", + "SizeDistribution", + "SummaryFlag", + "TerminationReason", + "summarise_state_npz", +] diff --git a/studio/modelio/scenario.py b/studio/modelio/scenario.py new file mode 100644 index 0000000..97b1516 --- /dev/null +++ b/studio/modelio/scenario.py @@ -0,0 +1,128 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunConfig`` -> ``CoupledScenario``. The single conversion point. + +This module is the seam. It is the ONLY place in Studio that imports ``coupled`` (ADR-001), and +therefore the only place where units are converted or names are translated (ADR-003). Everything +above it works in the schema's vocabulary; everything below is the model's. + +The mapping is deliberately dull. Canonical units are the model's native units precisely so this +function does not have to do arithmetic -- almost every field is passed through unchanged, and the +handful of places where something *does* happen are the interesting ones: + +* ``DilutionRegime.CONSTANT`` -> ``""``. The model spells "constant rate, no V(t) curve" as an empty + string, which cannot be a usable dropdown key. The only enum value that is not the model's own. +* ``background.so2_pptv`` is merged into ``dilution_background``, because the model has one field + for "what the plume relaxes toward" and the schema separates the background's SO2 from any other + species overrides. +* ``injection.so2_initial_pptv`` becomes the ``SO2`` entry of ``concentrations``. It is a DERIVED + field, so a config that has not been through ``studio.resolve`` will have ``None`` there -- which + raises here rather than reaching the model as a missing species. + +Fields the schema does not carry are left at the model's own defaults, exactly as +``run_ensemble.build_scenario`` leaves them: ``SA``, ``Yn2o5``, ``opt``, the ``micro_*`` step +controls, the aerosol placement, and ``output_dir``. Adding them to the schema before anything needs +them would be modelling the model rather than the runs. +""" + +from __future__ import annotations + +from typing import Any + +from coupled.coupled_scenario import CoupledScenario, Switches +from studio.resolve import ResolvedConfig +from studio.schema import DilutionRegime, RunConfig + +#: Schema regime -> the model's ``dilution_regime`` string. Only CONSTANT differs; the rest are +#: identical by construction, and the test asserts that rather than trusting it. +_REGIME_TO_MODEL: dict[DilutionRegime, str] = { + DilutionRegime.CONSTANT: "", + DilutionRegime.D1: "D1", + DilutionRegime.D2: "D2", + DilutionRegime.D3: "D3", + DilutionRegime.D5: "D5", + DilutionRegime.BURST: "burst", +} + + +def to_scenario(config: RunConfig | ResolvedConfig) -> CoupledScenario: + """Build the model's input object from a resolved Studio config. + + Args: + config: A ``RunConfig`` whose derived fields have been resolved, or the ``ResolvedConfig`` + that resolved them. Passing the latter is preferred: it carries the stale list, and this + function refuses to convert an inconsistent config. + + Raises: + InconsistentConfigError: If a ``ResolvedConfig`` has stale overrides. Converting one would + hand the model a set of numbers that do not follow from each other, and the result would + look like any other run. + ValueError: If a derived field is still unresolved. The alternative -- defaulting it -- is + exactly the silent fallback ADR-005 forbids. + """ + if isinstance(config, ResolvedConfig): + config.require_consistent() + run = config.config + else: + run = config + + so2_pptv = run.injection.so2_initial_pptv + if so2_pptv is None: + raise ValueError( + "injection.so2_initial_pptv is unresolved. It is a DERIVED field: run the config " + "through studio.resolve.resolve() before converting it, rather than letting the model " + "start with no SO2." + ) + + concentrations: dict[str, float] = {**run.background.gas_pptv, "SO2": so2_pptv} + dilution_background: dict[str, float] = { + "SO2": run.background.so2_pptv, + **run.dilution.background_overrides_pptv, + } + + return CoupledScenario( + T=run.site.temperature_k, + P=run.site.pressure_mbar, + WTR=run.site.h2o_ppmv, + latitude=run.site.latitude_deg, + longitude=run.site.longitude_deg, + day_of_year=run.schedule.day_of_year, + start_utc_hour=run.schedule.start_utc_hour, + days=run.schedule.duration_days, + DT=run.numerics.output_dt_s, + dt_couple=run.numerics.couple_dt_s, + photolysis=run.chemistry.photolysis.value, + tomas_nbins=run.microphysics.n_bins, + background_dist=run.background.aerosol.value, + dilution_regime=_REGIME_TO_MODEL[run.dilution.regime], + dilution_rate=run.dilution.rate_per_s, + dilution_zero_species=tuple(run.dilution.zero_species), + dilution_background=dilution_background, + ion_pair_rate=run.microphysics.ion_pair_rate, + so2_ho2_rate=run.chemistry.so2_ho2_rate, + condensation_alpha=run.microphysics.condensation_alpha, + nucleation_rate_scale=run.microphysics.nucleation_rate_scale, + coag_kernel_scale=run.microphysics.coag_kernel_scale, + switches=Switches( + sulfur=run.switches.sulfur, + nucleation=run.switches.nucleation, + condensation=run.switches.condensation, + coagulation=run.switches.coagulation, + aerosol_to_j=run.switches.aerosol_to_j, + heating_to_t=run.switches.heating_to_t, + dilution=run.switches.dilution, + ), + concentrations=concentrations, + ) + + +def scenario_as_dict(scenario: CoupledScenario) -> dict[str, Any]: + """``CoupledScenario`` -> plain dict, for the equivalence test and for provenance records. + + Thin wrapper over the model's own ``to_dict`` so callers need not import ``coupled`` to compare + or serialise a scenario -- which is the entire point of this package existing. + """ + return scenario.to_dict() + + +__all__ = ["scenario_as_dict", "to_scenario"] diff --git a/studio/modelio/summary.py b/studio/modelio/summary.py new file mode 100644 index 0000000..a21ef69 --- /dev/null +++ b/studio/modelio/summary.py @@ -0,0 +1,320 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSummary`` -- the versioned, queryable reduction of a run. + +Comparison views and figures read this; they never read the raw ``state.npz`` (ADR-004). The npz is +3.6 MB per case, carries 36 gas species and an (n_times, n_bins) size distribution, and knows +nothing about which of its arrays are wet and which are dry. A summary that fixes those three +things is what makes 810 runs comparable without opening 810 archives. + +**Every array declares its basis.** In ``state.npz``, ``dp_mid_um`` and ``dNdlogDp`` are DRY +diameters while ``SA`` and ``radius_cm`` are WET -- the same file, no labelling, and the difference +is a factor of a few in radius at stratospheric humidity. Here it is a required field on every +series, so a plot axis cannot be labelled by guesswork. + +Two more traps encoded rather than documented: + +* **Species are indexed by NAME** from the npz's own ``species`` list. An existing analysis script + hard-codes ``_SO2, _SO3, _H2SO4 = 32, 34, 35``; if the mechanism ever gains a species, that script + silently plots the wrong one. +* **The time axis comes from the stored ``t``**, never from ``i * DT``. Outer steps snap to the + terminator, so the mean step is ~592 s against a nominal 600 s -- about half a day of drift by day + 36. + +This module does not import ``coupled``; it only reads arrays. It lives here because reading the +model's output format is the model seam's job, not because it needs the model. +""" + +from __future__ import annotations + +from enum import StrEnum +from pathlib import Path +from typing import Any + +import numpy as np +from pydantic import BaseModel, ConfigDict, Field + +#: Bumped whenever the reduction changes shape or meaning. Stored in every summary so a comparison +#: view can refuse to plot two runs reduced under different rules rather than plotting them anyway. +SUMMARY_SCHEMA_VERSION = "0.1.0" + +#: Seconds per day, for the time axis. Named rather than inline (studio/CLAUDE.md). +SECONDS_PER_DAY = 86400.0 + + +class Basis(StrEnum): + """Whether a quantity is on a dry or an ambient (wet) basis. Required on every series.""" + + #: Particle without its water. ``dp_mid_um`` and ``dNdlogDp`` in ``state.npz``. + DRY = "dry" + #: Includes condensed water at ambient conditions. ``SA`` and ``radius_cm`` in ``state.npz``. + WET = "wet" + #: Not a particle-size quantity, so the distinction does not apply (gas mixing ratios, T, V/V0). + NOT_APPLICABLE = "not_applicable" + + +class TerminationReason(StrEnum): + """Why a run stopped. Never inferred from the data.""" + + COMPLETED = "completed" + FAILED = "failed" + CANCELLED = "cancelled" + TERMINATED_ON_LIMIT = "terminated_on_limit" + #: No record exists. The archived ensemble predates provenance capture (ADR-006), so a summary + #: built from one of its npz files is UNKNOWN rather than assumed to have completed. + UNKNOWN = "unknown" + + +class SummaryFlag(StrEnum): + """Machine-readable caveats that must travel with the numbers.""" + + #: Reduced from an archived npz with no provenance record: no config hash, no model version. + NO_PROVENANCE_RECORD = "no_provenance_record" + #: Dilution was active, so the box is an open system and sulfur is not expected to be conserved. + OPEN_SYSTEM_DILUTION = "open_system_dilution" + #: The run stopped on a limit rather than converging. Never presented as a converged result. + STOPPED_ON_LIMIT = "stopped_on_limit" + + +class Series(BaseModel): + """One scalar time series, with everything needed to plot it honestly.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + values: tuple[float, ...] + unit: str + basis: Basis + description: str + + +class SizeDistribution(BaseModel): + """The final size spectrum. Dry diameters, and it says so.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + diameter_um: tuple[float, ...] + dn_dlogdp_cm3: tuple[float, ...] + number_cm3: tuple[float, ...] + basis: Basis = Basis.DRY + #: Total number concentration, i.e. sum of the per-bin counts -- not of dN/dlogDp. + total_number_cm3: float + + +class ConservationCheck(BaseModel): + """A budget check, or an explicit statement that it does not apply. + + The honest part is ``status``. In-box sulfur is **not** expected to be conserved when dilution + is on: the box is an open system and entrainment removes plume sulfur while adding background + sulfur. Reporting a large "residual" for a diluting run would be reporting the dilution, and + reporting a small one would mean something had gone wrong. So the check is computed only when + the run is closed, and otherwise says why not -- rather than producing a number that invites a + reader to draw a conclusion from it. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + status: str + #: ``(S_end - S_start) / S_start`` for a closed box; ``None`` when not applicable. + relative_residual: float | None = None + initial_value: float | None = None + final_value: float | None = None + unit: str = "molec cm^-3" + reason: str = "" + + +class RunSummary(BaseModel): + """The reduction of one run. Versioned, and self-describing about basis and provenance.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + schema_version: str = SUMMARY_SCHEMA_VERSION + #: Identity of the config that produced this run (ADR-006). ``None`` for archived runs, which + #: have no provenance record -- flagged rather than left ambiguous. + config_hash: str | None = None + label: str | None = None + termination: TerminationReason = TerminationReason.UNKNOWN + flags: tuple[SummaryFlag, ...] = () + time_days: tuple[float, ...] = () + series: dict[str, Series] = Field(default_factory=dict) + final_size_distribution: SizeDistribution | None = None + sulfur_conservation: ConservationCheck | None = None + + def write(self, path: Path) -> Path: + """Write as JSON next to the run's ``state.npz``.""" + path.write_text(self.model_dump_json(indent=2), encoding="utf-8") + return path + + @classmethod + def read(cls, path: Path) -> RunSummary: + """Read one back, validating it against this version of the schema.""" + return cls.model_validate_json(path.read_text(encoding="utf-8")) + + +#: Gas species reduced to mixing-ratio series, by NAME. Extending this list is the supported way to +#: add a series; indexing by position is how the existing analysis scripts got it wrong. +_GAS_SERIES: tuple[tuple[str, str], ...] = ( + ("SO2", "sulfur dioxide, the injected species"), + ("SO3", "sulfur trioxide, the intermediate"), + ("H2SO4", "gas-phase sulfuric acid, the condensable"), + ("OH", "hydroxyl radical, the oxidant that starts the chain"), + ("HO2", "hydroperoxyl radical"), + ("O3", "ozone"), +) + +#: Aerosol/environment series carried straight through, with the basis each one is actually on. +_DIRECT_SERIES: tuple[tuple[str, str, Basis, str], ...] = ( + ("SA", "um^2 cm^-3", Basis.WET, "aerosol surface area density (wet)"), + ("radius_cm", "cm", Basis.WET, "effective particle radius (wet)"), + ("h2so4wp", "1", Basis.NOT_APPLICABLE, "H2SO4 weight fraction of the aerosol"), + ("particulate_S", "molec cm^-3", Basis.NOT_APPLICABLE, "sulfur held in the particle phase"), + ("T", "K", Basis.NOT_APPLICABLE, "box temperature"), + ("total_n", "cm^-3", Basis.DRY, "total particle number concentration"), + ("V_ratio", "1", Basis.NOT_APPLICABLE, "plume volume expansion V(t)/V0"), +) + +#: Species whose number density IS its sulfur content -- each carries exactly one S atom. Used for +#: the closed-box budget check. ``particulate_S`` is added separately; it is already a sulfur count. +_SULFUR_GAS_SPECIES = ("SO2", "SO3", "H2SO4") + + +def summarise_state_npz( + path: Path, + *, + label: str | None = None, + config_hash: str | None = None, + termination: TerminationReason = TerminationReason.UNKNOWN, +) -> RunSummary: + """Reduce a ``state.npz`` to a :class:`RunSummary`. + + ``termination`` is an argument rather than something inferred from the arrays: the npz records + what happened to the state, not why the loop stopped. A run that hit a wall-clock limit and one + that finished look identical here, and guessing would be the difference between "converged" and + "cut short" (ADR-005). + """ + with np.load(path, allow_pickle=True) as archive: + data = {key: archive[key] for key in archive.files} + + missing = {"t", "x", "species", "M"} - set(data) + if missing: + raise ValueError(f"{path} is missing required arrays {sorted(missing)}") + + # From the STORED t, never i * DT: outer steps snap to the terminator (~592 s vs 600 s nominal). + time_s = np.asarray(data["t"], dtype=np.float64) + air_number_density = float(data["M"]) + species = [str(name) for name in data["species"]] + state = np.asarray(data["x"], dtype=np.float64) + + series: dict[str, Series] = {} + for name, description in _GAS_SERIES: + if name not in species: + continue # a mechanism without this species is not an error, just fewer series + column = state[:, species.index(name)] # BY NAME. Never by position. + series[name] = Series( + values=tuple(column / air_number_density * 1.0e12), + unit="pptv", + basis=Basis.NOT_APPLICABLE, + description=description, + ) + for key, unit, basis, description in _DIRECT_SERIES: + if key in data: + series[key] = Series( + values=tuple(np.asarray(data[key], dtype=np.float64)), + unit=unit, + basis=basis, + description=description, + ) + + flags: list[SummaryFlag] = [] + if config_hash is None: + flags.append(SummaryFlag.NO_PROVENANCE_RECORD) + if termination is TerminationReason.TERMINATED_ON_LIMIT: + flags.append(SummaryFlag.STOPPED_ON_LIMIT) + + diluting = "V_ratio" in data and float(np.max(np.asarray(data["V_ratio"]))) > 1.0 + if diluting: + flags.append(SummaryFlag.OPEN_SYSTEM_DILUTION) + + return RunSummary( + config_hash=config_hash, + label=label, + termination=termination, + flags=tuple(flags), + time_days=tuple(time_s / SECONDS_PER_DAY), + series=series, + final_size_distribution=_final_size_distribution(data), + sulfur_conservation=_sulfur_budget(data, species, state, diluting=diluting), + ) + + +def _final_size_distribution(data: dict[str, Any]) -> SizeDistribution | None: + """The last spectrum. ``dp_mid_um`` and ``dNdlogDp`` are DRY (see the module docstring).""" + if not {"dp_mid_um", "dNdlogDp", "n_cm3"} <= set(data): + return None + diameters = np.asarray(data["dp_mid_um"], dtype=np.float64) + dn_dlogdp = np.asarray(data["dNdlogDp"], dtype=np.float64)[-1] + counts = np.asarray(data["n_cm3"], dtype=np.float64)[-1] + return SizeDistribution( + diameter_um=tuple(diameters), + dn_dlogdp_cm3=tuple(dn_dlogdp), + number_cm3=tuple(counts), + total_number_cm3=float(counts.sum()), + ) + + +def _sulfur_budget( + data: dict[str, Any], species: list[str], state: np.ndarray, *, diluting: bool +) -> ConservationCheck: + """In-box sulfur at the start and end, and a residual only when the box is closed.""" + present = [name for name in _SULFUR_GAS_SPECIES if name in species] + if not present: + return ConservationCheck( + status="not_applicable", + reason="the mechanism carries none of SO2, SO3, H2SO4", + ) + total = np.zeros(state.shape[0], dtype=np.float64) + for name in present: + total = total + state[:, species.index(name)] + if "particulate_S" in data: + total = total + np.asarray(data["particulate_S"], dtype=np.float64) + initial, final = float(total[0]), float(total[-1]) + + if diluting: + return ConservationCheck( + status="not_applicable", + initial_value=initial, + final_value=final, + reason=( + "dilution was active (V(t)/V0 > 1), so the box is an open system: entrainment " + "removes plume sulfur and adds background sulfur. A residual here would be a " + "measure of the dilution, not of conservation. Start and end values are reported " + "so the decay is visible." + ), + ) + if initial == 0.0: + return ConservationCheck( + status="not_applicable", + initial_value=initial, + final_value=final, + reason="no sulfur at t = 0, so a relative residual is undefined", + ) + return ConservationCheck( + status="computed", + relative_residual=(final - initial) / initial, + initial_value=initial, + final_value=final, + reason="closed box (no dilution): total sulfur should be conserved", + ) + + +__all__ = [ + "SECONDS_PER_DAY", + "SUMMARY_SCHEMA_VERSION", + "Basis", + "ConservationCheck", + "RunSummary", + "Series", + "SizeDistribution", + "SummaryFlag", + "TerminationReason", + "summarise_state_npz", +] diff --git a/studio/tests/unit/test_import_boundaries.py b/studio/tests/unit/test_import_boundaries.py index 20765c6..8135835 100644 --- a/studio/tests/unit/test_import_boundaries.py +++ b/studio/tests/unit/test_import_boundaries.py @@ -115,7 +115,12 @@ def test_only_the_model_seam_imports_coupled() -> None: Catches imports hidden inside function bodies, which the runtime check above cannot see. """ offenders = _modules_importing_coupled() - stray = {module for module in offenders if module not in MODEL_SEAM_PACKAGES} + stray = { + module + for module in offenders + # a submodule of the seam (studio.modelio.scenario) is the seam; a sibling package is not + if not any(module == seam or module.startswith(f"{seam}.") for seam in MODEL_SEAM_PACKAGES) + } assert stray == set(), ( f"{sorted(stray)} import `coupled`, but ADR-001 names {sorted(MODEL_SEAM_PACKAGES)} as the " f"only model seam. Move the call behind studio.modelio, or amend ADR-001 in this change." diff --git a/studio/tests/unit/test_modelio_equivalence.py b/studio/tests/unit/test_modelio_equivalence.py new file mode 100644 index 0000000..da7f432 --- /dev/null +++ b/studio/tests/unit/test_modelio_equivalence.py @@ -0,0 +1,175 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The equivalence test: the schema is a faithful superset of what the ensemble actually ran. + +``to_scenario`` applied to the golden ``RunConfig`` must produce a ``CoupledScenario`` +**field-for-field identical** to ``run_ensemble.build_scenario()`` for case +``30N_20km__sabr220__D2med__a1p0__nuc1__cg1``. Tolerance: **exact**, on every field, including the +floats -- this is not a physics comparison, it is a claim that two code paths build the same object, +and "close" would mean one of them is doing arithmetic the other is not. + +Proving it here is much cheaper and sharper than discovering it from a diverging result: a mismatch +names the field, now, instead of showing up as a 3 % difference in particle number after a 4-minute +run and a day of bisection. + +These tests need the model, so they need the submodules checked out. CI does not check them out +(see ``.github/workflows/studio-ci.yml``), so this skips there and runs locally -- the same +asymmetry as the ``air_number_density`` mirror check, and worth the same caution. +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path +from typing import Any + +import pytest + +from studio.resolve import resolve, set_override +from studio.schema import BackgroundAerosol, DilutionRegime, PhotolysisMode, RunConfig + +#: The axis values for the golden case, exactly as ``run_ensemble.py:61-76`` spells them. +GOLDEN_AXES: dict[str, Any] = { + "lat_alt": ("30N_20km", 30.0, 210.0, 55.0, 6.9104), + "background": ("sabr220", "sabr_220", 20.0), + "dilution": ("D2med", "D2"), + "sticking": ("a1p0", 1.0), + "nucleation": ("nuc1", 1.0), + "coag": ("cg1", 1.0), +} + + +@pytest.fixture(scope="module") +def build_scenario(repo_root: Path) -> Any: + """``run_ensemble.build_scenario``, or skip if the model is not checked out.""" + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip( + "model submodules not checked out (`git submodule update --init`); the equivalence " + "test compares against run_ensemble.build_scenario, which needs them" + ) + from coupled.paper_ensemble.run_ensemble import build_scenario as builder + + return builder + + +@pytest.fixture(scope="module") +def studio_scenario() -> Any: + from studio.modelio.scenario import to_scenario + + return to_scenario(resolve(RunConfig())) + + +@pytest.mark.tier_a +def test_the_golden_case_is_the_schemas_default(build_scenario: Any) -> None: + """``RunConfig()`` with no arguments IS the golden case; nothing has to be set up to get it.""" + reference = build_scenario(GOLDEN_AXES) + config = RunConfig() + assert config.site.temperature_k == reference.T + assert config.site.pressure_mbar == reference.P + assert config.background.aerosol is BackgroundAerosol.SABR_220 + assert config.dilution.regime is DilutionRegime.D2 + assert config.chemistry.photolysis is PhotolysisMode.TUVX + + +@pytest.mark.tier_a +def test_to_scenario_is_field_for_field_identical( + build_scenario: Any, studio_scenario: Any +) -> None: + """The whole point of the task. Every field, exact equality, no exceptions.""" + reference = dataclasses.asdict(build_scenario(GOLDEN_AXES)) + produced = dataclasses.asdict(studio_scenario) + + assert set(produced) == set(reference), "the two scenarios have different field sets" + differing = { + key: (reference[key], produced[key]) for key in reference if produced[key] != reference[key] + } + assert differing == {}, ( + "to_scenario diverges from run_ensemble.build_scenario for the golden case " + f"(reference, produced): {differing}. The schema is meant to be a faithful superset; " + f"either the mapping is wrong or the schema default is." + ) + + +@pytest.mark.tier_a +def test_the_initial_so2_matches_to_the_last_bit(build_scenario: Any, studio_scenario: Any) -> None: + """Called out separately because it is the one value Studio DERIVES rather than passes through. + + Everything else is a copy; this one goes mass -> number density -> mixing ratio through + ``studio.science`` while the ensemble does the same arithmetic inline. Exact equality is the + evidence that consolidating that derivation changed nothing. + """ + reference = build_scenario(GOLDEN_AXES) + assert studio_scenario.concentrations["SO2"] == reference.concentrations["SO2"] + assert studio_scenario.concentrations == reference.concentrations + + +@pytest.mark.tier_a +def test_the_constant_regime_maps_to_the_models_empty_string(build_scenario: Any) -> None: + """The one enum value that is not the model's own string (``enums.py``). + + Asserted against a real ``CoupledScenario`` rather than against the mapping table, because the + model validates ``dilution_regime`` in ``__post_init__`` -- if ``""`` ever stopped being the + spelling, this fails here rather than in a run. + """ + from studio.modelio.scenario import to_scenario + + config = RunConfig.model_validate( + {**RunConfig().model_dump(), "dilution": {"regime": "constant"}} + ) + assert to_scenario(resolve(config)).dilution_regime == "" + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("regime", "expected"), + [("D1", "D1"), ("D2", "D2"), ("D3", "D3"), ("D5", "D5"), ("burst", "burst")], +) +def test_every_other_regime_passes_through_unchanged( + build_scenario: Any, regime: str, expected: str +) -> None: + """The claim that the enum values ARE the model's strings, checked rather than trusted.""" + from studio.modelio.scenario import to_scenario + + config = RunConfig.model_validate({**RunConfig().model_dump(), "dilution": {"regime": regime}}) + assert to_scenario(resolve(config)).dilution_regime == expected + + +@pytest.mark.tier_a +def test_an_unresolved_config_is_refused(build_scenario: Any) -> None: + """A config that never went through the resolver has ``so2_initial_pptv = None``. + + Handing that to the model would start the plume with no SO2 -- a run that completes and means + nothing. It raises instead (ADR-005). + """ + from studio.modelio.scenario import to_scenario + + with pytest.raises(ValueError, match="so2_initial_pptv is unresolved"): + to_scenario(RunConfig()) + + +@pytest.mark.tier_a +def test_a_stale_config_is_refused(build_scenario: Any) -> None: + """Converting a stale config would hand the model numbers that do not follow from each other.""" + from studio.modelio.scenario import to_scenario + from studio.resolve import InconsistentConfigError, apply_change + + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError): + to_scenario(stale) + + +@pytest.mark.tier_a +def test_a_resolved_override_reaches_the_model(build_scenario: Any) -> None: + """The other half: an override the user has settled must actually arrive.""" + from studio.modelio.scenario import to_scenario + from studio.resolve import keep_override + + overridden = set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9) + assert ( + to_scenario(keep_override(overridden, "injection.so2_initial_pptv")).concentrations["SO2"] + == 5.0e9 + ) diff --git a/studio/tests/unit/test_modelio_summary.py b/studio/tests/unit/test_modelio_summary.py new file mode 100644 index 0000000..fa18c24 --- /dev/null +++ b/studio/tests/unit/test_modelio_summary.py @@ -0,0 +1,236 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSummary``, against a synthetic archive and against a real one. + +The synthetic cases run everywhere, including CI, and are where the traps are tested deliberately: +shuffled species order, a non-uniform time axis, a closed box vs a diluting one. The real-archive +case runs only where the 810-run ensemble exists and checks that the reduction survives contact with +an actual 3.6 MB file. + +Note ``summary.py`` does not import ``coupled``: it reads arrays. So all of this runs without the +model, which is why these tests are not skipped in CI while the equivalence tests are. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest + +from studio.modelio import ( + SUMMARY_SCHEMA_VERSION, + Basis, + RunSummary, + SummaryFlag, + TerminationReason, + summarise_state_npz, +) + +GOLDEN_CASE = "30N_20km__sabr220__D2med__a1p0__nuc1__cg1" + +#: The mechanism's species list, in a deliberately awkward order: SO2/SO3/H2SO4 are NOT at the +#: positions the existing analysis scripts hard-code (32, 34, 35). +SPECIES = ("H2SO4", "OH", "SO2", "O3", "HO2", "SO3") + + +def _write_npz( + path: Path, + *, + n_times: int = 5, + n_bins: int = 4, + diluting: bool = True, + time_s: np.ndarray | None = None, +) -> Path: + """A synthetic ``state.npz`` with the real file's key set and shapes.""" + rng = np.random.default_rng(20260813) # seeded: any stochastic component records its seed + time_s = np.arange(n_times, dtype=np.float64) * 592.0 if time_s is None else time_s + state = np.zeros((n_times, len(SPECIES)), dtype=np.float64) + state[:, SPECIES.index("SO2")] = np.linspace(6.0e15, 1.0e15, n_times) + state[:, SPECIES.index("SO3")] = np.linspace(0.0, 1.0e10, n_times) + state[:, SPECIES.index("H2SO4")] = np.linspace(0.0, 5.0e13, n_times) + state[:, SPECIES.index("OH")] = rng.uniform(1e5, 1e7, n_times) + state[:, SPECIES.index("HO2")] = rng.uniform(1e6, 1e8, n_times) + state[:, SPECIES.index("O3")] = np.full(n_times, 1.18e12) + edges = np.geomspace(1.7e-3, 17.5, n_bins + 1) + counts = rng.uniform(1.0, 100.0, (n_times, n_bins)) + np.savez( + path, + t=time_s, + x=state, + species=np.array(SPECIES), + M=np.float64(1.8956916099773243e18), + SA=np.linspace(2.0, 40.0, n_times), + radius_cm=np.full(n_times, 1.2e-5), + h2so4wp=np.full(n_times, 0.72), + # closes the closed-box budget exactly: SO2 1.0e15 + SO3 1.0e10 + H2SO4 5.0e13 + this + # == the initial 6.0e15 + particulate_S=np.linspace(0.0, 6.0e15 - 1.0e15 - 1.0e10 - 5.0e13, n_times), + T=np.full(n_times, 210.0), + n_cm3=counts, + Dp_m=np.tile(np.sqrt(edges[:-1] * edges[1:]) * 1e-6, (n_times, 1)), + dp_mid_um=np.sqrt(edges[:-1] * edges[1:]), + dNdlogDp=counts / np.log10(edges[1:] / edges[:-1]), + V_ratio=np.linspace(1.0, 1500.0, n_times) if diluting else np.ones(n_times), + total_n=counts.sum(axis=1), + ) + return path + + +@pytest.fixture +def synthetic_npz(tmp_path: Path) -> Path: + return _write_npz(tmp_path / "state.npz") + + +@pytest.mark.tier_a +def test_species_are_indexed_by_name_not_position(synthetic_npz: Path) -> None: + """The trap, tested head-on. + + ``SPECIES`` puts SO2 at index 2 and H2SO4 at index 0 -- nothing like the 32/34/35 an existing + analysis script hard-codes. A summary that indexed by position would report ozone as SO2 here, + and would report something plausible rather than crashing. + """ + summary = summarise_state_npz(synthetic_npz) + m_air = 1.8956916099773243e18 + assert summary.series["SO2"].values[0] == pytest.approx(6.0e15 / m_air * 1e12, rel=1e-12) + assert summary.series["O3"].values[0] == pytest.approx(1.18e12 / m_air * 1e12, rel=1e-12) + assert summary.series["SO2"].values[-1] < summary.series["SO2"].values[0] + + +@pytest.mark.tier_a +def test_the_time_axis_comes_from_the_stored_t(tmp_path: Path) -> None: + """Never ``i * DT``. Outer steps snap to the terminator, so the mean step is ~592 s, not 600. + + Over a 36-day run that is about half a day of drift -- enough to put a diurnal feature on the + wrong side of local noon. + """ + irregular = np.array([0.0, 592.0, 1184.0, 1776.0, 2368.0]) # 4 x 592, not 4 x 600 + summary = summarise_state_npz(_write_npz(tmp_path / "state.npz", time_s=irregular)) + np.testing.assert_allclose(summary.time_days, irregular / 86400.0, rtol=0.0, atol=0.0) + nominal = np.arange(5) * 600.0 / 86400.0 + assert summary.time_days[-1] != pytest.approx(nominal[-1]), "a nominal grid would differ here" + + +@pytest.mark.tier_a +def test_every_series_declares_its_basis(synthetic_npz: Path) -> None: + """Wet vs dry is not optional metadata; it is a factor of a few in radius at 55 hPa.""" + summary = summarise_state_npz(synthetic_npz) + assert summary.series["SA"].basis is Basis.WET + assert summary.series["radius_cm"].basis is Basis.WET + assert summary.series["total_n"].basis is Basis.DRY + assert summary.series["SO2"].basis is Basis.NOT_APPLICABLE + assert summary.final_size_distribution is not None + assert summary.final_size_distribution.basis is Basis.DRY + for name, series in summary.series.items(): + assert series.unit, f"{name} has no unit" + assert series.description, f"{name} has no description" + + +@pytest.mark.tier_a +def test_the_final_size_distribution_is_the_last_step(synthetic_npz: Path) -> None: + with np.load(synthetic_npz) as archive: + expected = archive["n_cm3"][-1] + distribution = summarise_state_npz(synthetic_npz).final_size_distribution + assert distribution is not None + np.testing.assert_allclose(distribution.number_cm3, expected, rtol=0.0) + assert distribution.total_number_cm3 == pytest.approx(float(expected.sum()), rel=1e-15) + assert len(distribution.diameter_um) == len(distribution.dn_dlogdp_cm3) == len(expected) + + +@pytest.mark.tier_a +def test_a_diluting_run_reports_no_conservation_residual(synthetic_npz: Path) -> None: + """The box is an open system, so a residual would measure the dilution, not conservation. + + Reporting a number here would invite a reader to conclude something from it. The start and end + values are still reported, so the decay is visible without being dressed up as a budget error. + """ + check = summarise_state_npz(synthetic_npz).sulfur_conservation + assert check is not None + assert check.status == "not_applicable" + assert check.relative_residual is None + assert "open system" in check.reason + assert check.initial_value is not None and check.final_value is not None + assert SummaryFlag.OPEN_SYSTEM_DILUTION in summarise_state_npz(synthetic_npz).flags + + +@pytest.mark.tier_a +def test_a_closed_box_gets_a_real_residual(tmp_path: Path) -> None: + """With V(t)/V0 == 1 throughout, sulfur should be conserved and the residual means something. + + The synthetic archive is built so gas + particulate sulfur closes exactly: 6.0e15 at t = 0, and + SO2 1.0e15 + SO3 1.0e10 + H2SO4 5.0e13 + particulate 4.94999e15 at the end. Tolerance 1e-12 + relative -- float64 summation noise on five terms, not a physical tolerance, because the + quantity being checked is arithmetic rather than physics. + """ + check = summarise_state_npz( + _write_npz(tmp_path / "state.npz", diluting=False) + ).sulfur_conservation + assert check is not None + assert check.status == "computed" + assert check.relative_residual == pytest.approx(0.0, abs=1e-12) + assert ( + SummaryFlag.OPEN_SYSTEM_DILUTION + not in summarise_state_npz(_write_npz(tmp_path / "closed.npz", diluting=False)).flags + ) + + +@pytest.mark.tier_a +def test_termination_is_recorded_never_inferred(synthetic_npz: Path) -> None: + """The npz says what the state did, not why the loop stopped. Guessing would be the difference + between "converged" and "cut short".""" + assert summarise_state_npz(synthetic_npz).termination is TerminationReason.UNKNOWN + stopped = summarise_state_npz(synthetic_npz, termination=TerminationReason.TERMINATED_ON_LIMIT) + assert SummaryFlag.STOPPED_ON_LIMIT in stopped.flags + + +@pytest.mark.tier_a +def test_a_summary_without_provenance_says_so(synthetic_npz: Path) -> None: + """The archived ensemble has no config hash (ADR-006). That is a flag, not a blank field.""" + assert SummaryFlag.NO_PROVENANCE_RECORD in summarise_state_npz(synthetic_npz).flags + with_hash = summarise_state_npz(synthetic_npz, config_hash="abc123", label=GOLDEN_CASE) + assert SummaryFlag.NO_PROVENANCE_RECORD not in with_hash.flags + assert with_hash.config_hash == "abc123" + assert with_hash.label == GOLDEN_CASE + + +@pytest.mark.tier_a +def test_a_summary_round_trips_through_json(synthetic_npz: Path, tmp_path: Path) -> None: + """It is written next to state.npz and read back by comparison views; both directions matter.""" + original = summarise_state_npz(synthetic_npz, label="case", config_hash="abc123") + path = original.write(tmp_path / "summary.json") + restored = RunSummary.read(path) + assert restored == original + assert restored.schema_version == SUMMARY_SCHEMA_VERSION + + +@pytest.mark.tier_a +def test_a_truncated_archive_raises(tmp_path: Path) -> None: + """Missing arrays are a corrupted run, not a run with fewer series.""" + path = tmp_path / "state.npz" + np.savez(path, t=np.zeros(3), species=np.array(SPECIES)) + with pytest.raises(ValueError, match="missing required arrays"): + summarise_state_npz(path) + + +@pytest.mark.tier_a +def test_summarising_a_real_archived_run(paper_ensemble_runs: Path) -> None: + """The golden case, straight from the 810-run ensemble. Skips where the archive is absent.""" + path = paper_ensemble_runs / GOLDEN_CASE / "state.npz" + if not path.is_file(): + pytest.skip(f"{path} not present") + summary = summarise_state_npz(path, label=GOLDEN_CASE) + + assert len(summary.time_days) == 1461 + assert summary.time_days[0] == 0.0 + assert summary.time_days[-1] == pytest.approx(10.0, rel=1e-12), "a 10-day run" + assert summary.series["SO2"].values[-1] < summary.series["SO2"].values[0], "SO2 is consumed" + assert max(summary.series["H2SO4"].values) > 0.0, "H2SO4 is produced" + assert summary.final_size_distribution is not None + assert len(summary.final_size_distribution.diameter_um) == 80 + assert summary.termination is TerminationReason.UNKNOWN + assert set(summary.flags) >= { + SummaryFlag.NO_PROVENANCE_RECORD, + SummaryFlag.OPEN_SYSTEM_DILUTION, + } + assert summary.sulfur_conservation is not None + assert summary.sulfur_conservation.status == "not_applicable" diff --git a/studio/tests/unit/test_resolve_resolver.py b/studio/tests/unit/test_resolve_resolver.py index 654daae..f0cb080 100644 --- a/studio/tests/unit/test_resolve_resolver.py +++ b/studio/tests/unit/test_resolve_resolver.py @@ -10,6 +10,10 @@ from __future__ import annotations +import subprocess +import sys +import textwrap +from pathlib import Path from typing import Any import pytest @@ -247,11 +251,24 @@ def test_a_degenerate_input_is_caught_by_the_schema_before_the_derivation_runs() def test_resolution_does_not_import_the_model() -> None: """The API resolves on every keystroke; a JAX import on that path would be unaffordable. - The static and runtime boundary checks live in test_import_boundaries.py; this one asserts the - same thing about the actual call path, which is what the cost is attached to. + Run in a FRESH interpreter, not in-process. ``test_import_boundaries.py`` covers the same ground + for imports; this one covers the CALL, because a lazy import inside ``resolve()`` would slip + past an import-time check. In-process it would prove nothing either way: another test module in + this session imports ``studio.modelio.scenario``, which is allowed to reach the model, and that + alone would put ``coupled`` in ``sys.modules``. """ - import sys - - resolve(RunConfig()) - assert "coupled" not in sys.modules - assert "jax" not in sys.modules + probe = textwrap.dedent(""" + import sys + from studio.resolve import resolve + from studio.schema import RunConfig + resolve(RunConfig()) + print(sorted({"coupled", "jax", "jaxlib"} & {m.split(".")[0] for m in sys.modules})) + """) + proc = subprocess.run( + [sys.executable, "-c", probe], + capture_output=True, + text=True, + cwd=str(Path(__file__).resolve().parents[3]), + ) + assert proc.returncode == 0, proc.stderr + assert proc.stdout.strip() == "[]", f"resolving pulled in {proc.stdout.strip()}" From bd284b2c407dd602cf6bd8e35b230bfc070ce6b1 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 19:39:46 -0700 Subject: [PATCH 06/18] studio: the runner, the job lifecycle and the first end-to-end run (#72) * studio: the runner, the job lifecycle and the first end-to-end run Task 0.6 (#72). studio/runner (base.py, local.py), studio/modelio/execute.py and studio/cli/run.py. 20 new Tier-A tests, 191 total. The first end-to-end run works: `python -m studio.cli.run input.json outdir` runs the real model and writes state.npz plus summary.json. Verified on a 1-day/40-bin case in 21 s -- SO2 3.309e9 -> 1.72e6 pptv, peak H2SO4 15.05 pptv, peak number 3.07e6 cm^-3, npz key set identical to the canonical one at run_ensemble.py:150-156, termination "completed" with a real config_hash. A Studio-created run therefore has the provenance the archived ensemble lacks. The lifecycle is data and its transitions are enforced. Every transition is timestamped and kept, because "it failed" is not debuggable and "QUEUED 14:02:11, RUNNING 14:02:11, FAILED 14:06:48 exit 1" is. An illegal transition raises: a job that appears to move backwards means the runner lost track of a process, and accepting it silently would turn the record from a log into a story. TERMINATED_ON_LIMIT is kept distinct from FAILED all the way through -- one means the model could not produce a result, the other that it was still going when we stopped it, and the second leaves partial output that can look complete. A failed run is debuggable without re-running it: the resolved input is written at submit rather than at completion, so a job that dies immediately still has it; stdout and stderr are captured in full, stdout being the log stream because run_coupled prints rather than logs; the exit code is recorded. Exit codes distinguish "never started" (2, bad input, rejected before the model is imported so it costs milliseconds) from "the model raised" (1). entry_module is a parameter rather than a test hook -- the runner launches a module by name, nothing branches on the value, the default is the real path, and the real path is exercised by the exit-code test. Slurm and cloud-batch raise instead of falling back to local execution, because a job running somewhere other than where it was sent is worse than an error. execute.py reuses coupled.dilution.volume_ratio and studio.science's size-distribution reduction rather than inlining a fifth copy, and enforces max_sim_time through a *args stop-condition that works either side of task 0.8's widening of that callback. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * studio: the stop-condition shape the model will actually accept PR #73 widens run_coupled's stop_condition to a diagnostics dict and dispatches on the callback's DECLARED arity: two positional parameters is accepted as legacy, one is the new dict shape, and *args raises TypeError because it matches both. Raising on *args is right -- guessing which shape a variadic callback wanted would be a silent wrong answer -- but it means the "works with either" spelling I had reached for is the one thing that does not work. So the callback is now two-positional, which the model accepts both before and after that change, and three tests pin the arity so the coupling is visible rather than latent. Without them the break would have been silent: nothing today sets max_sim_time, so the path is unexercised until someone does. Migrating to the dict shape is worth doing once #73 is in studio/dev, since that is what makes termination criteria on SO2 or particle number possible. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 (1M context) --- .gitignore | 3 + docs/studio/PROGRESS.md | 53 ++++- docs/studio/plan/PHASE_0.md | 2 +- studio/cli/run.py | 70 ++++++ studio/modelio/execute.py | 149 ++++++++++++ studio/runner/__init__.py | 30 ++- studio/runner/base.py | 292 +++++++++++++++++++++++ studio/runner/local.py | 252 ++++++++++++++++++++ studio/tests/fixtures/__init__.py | 3 + studio/tests/fixtures/fake_run.py | 56 +++++ studio/tests/unit/test_runner.py | 370 ++++++++++++++++++++++++++++++ 11 files changed, 1277 insertions(+), 3 deletions(-) create mode 100644 studio/cli/run.py create mode 100644 studio/modelio/execute.py create mode 100644 studio/runner/base.py create mode 100644 studio/runner/local.py create mode 100644 studio/tests/fixtures/__init__.py create mode 100644 studio/tests/fixtures/fake_run.py create mode 100644 studio/tests/unit/test_runner.py diff --git a/.gitignore b/.gitignore index 048d368..c3f9add 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,6 @@ gas_phase_chemistry/ # -- SQLite database and run artefacts, both regenerable and both machine-specific .env var/ + +# Subagent worktrees: separate checkouts, never part of this repo +.claude/worktrees/ diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 7cbf3f2..43bde2d 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -19,7 +19,7 @@ with full provenance, and the golden tests pass. | 0.3 Dependency-graph engine + override semantics | **done** (#66) | | 0.4 `studio/modelio` seam + `RunSummary` | **done** (#68) | | 0.5 `studio/science` derivations | **done** (#64) | -| 0.6 `studio/runner` + job lifecycle | not started | +| 0.6 `studio/runner` + job lifecycle | **done** (#72) | | 0.7 Golden-file harness (two tiers) | not started | | 0.8 Four contained fixes in `coupled/` | not started | | 0.9 Vertical slice: CLI + API + minimal UI | not started | @@ -29,6 +29,57 @@ derivations to resolve rather than fixtures. --- +### 2026-08-13 — Task 0.6: the runner and the job lifecycle (issue #72) + +`studio/runner/` (`base.py`, `local.py`), `studio/modelio/execute.py`, `studio/cli/run.py`. 20 new +Tier-A tests (191 total). + +**The first end-to-end run.** `python -m studio.cli.run ` runs the real model +and writes `state.npz` plus `summary.json`. Verified on a 1-day, 40-bin case: **21 s wall**, +SO₂ 3.309e9 → 1.72e6 pptv, peak H₂SO₄ 15.05 pptv, peak number 3.07e6 cm⁻³, the npz key set identical +to the canonical one from `run_ensemble.py:150-156`, and `termination = completed` with a real +`config_hash`. That last part matters: a Studio-created run has provenance, which is exactly what the +archived ensemble lacks (ADR-006). + +**Lifecycle as data, transitions enforced.** `DRAFT → QUEUED → RUNNING → (SUCCEEDED | FAILED | +CANCELLED | TERMINATED_ON_LIMIT)`, every transition timestamped and kept — "it failed" is not +debuggable, "QUEUED 14:02:11, RUNNING 14:02:11, FAILED 14:06:48 exit 1" is. Illegal transitions +raise: a job that appears to move backwards means the runner lost a process, and accepting it +silently would turn the record from a log into a story. + +**`TERMINATED_ON_LIMIT` is not `FAILED`.** One means the model could not produce a result; the other +means it was still going when we stopped it, and its partial output can look complete. Kept distinct +all the way through, and the detail string says so. + +**A failed run is debuggable without re-running it.** The resolved input is written at *submit*, not +at completion, so a job that dies immediately still has its input; stdout and stderr are captured in +full (`run_coupled` prints rather than logs, so stdout IS the log stream); the exit code is recorded. +That set is chosen for the case that actually hurts: a four-minute run that fails intermittently. + +**Decisions** + +- **`entry_module` is a parameter, not a test hook.** The runner launches a module by name; tests + point it at a fixture module so the lifecycle can be exercised in milliseconds instead of four + minutes. Nothing in the runner branches on the value, the default is the real entry point, and the + real one is exercised separately by the exit-code test. +- **Exit codes mean something specific**: 0 ran, 2 the input was bad and nothing started, 1 the model + raised. The runner needs to tell "never started" from "broke", and `studio.cli.run` validates the + input *before* importing the model so a bad input fails in milliseconds rather than after a JAX + load. +- **Slurm and cloud-batch raise** (ADR-008) rather than falling back to local execution. A job + running somewhere other than where it was sent is worse than an error. +- **`max_sim_time` is enforced through a `*args` stop-condition**, which works either side of task + 0.8's widening of that callback rather than depending on which has landed. +- `studio/modelio/execute.py` reuses `coupled.dilution.volume_ratio` and `studio.science`'s + size-distribution reduction rather than inlining a fifth copy — which is what 0.5 was for. + +**Test-design note.** The runner tests launch **real subprocesses**; a mocked `Popen` would test the +mock. The fixture module's behaviour arrives by environment variable, set before `submit()`, because +a directive file written *after* submission races the subprocess start — the standard way process +tests become flaky. + +--- + ### 2026-08-13 — Task 0.4: the model seam and `RunSummary` (issue #68) `studio/modelio/`: `scenario.py` (the seam) and `summary.py` (the reduction). 23 new Tier-A tests diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index 249a3c7..d13087c 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -112,7 +112,7 @@ Avogadro mismatch at the gas/TOMAS seam. Studio inherits it and does not silentl --- -## 0.6 — `studio/runner` + job lifecycle +## 0.6 — `studio/runner` + job lifecycle · *done* (issue #72) `JobRunner` Protocol; `LocalSubprocessRunner` launching `python -m studio.cli.run` with the thread-pinning environment from `launch_parallel.py:26-30`. Lifecycle diff --git a/studio/cli/run.py b/studio/cli/run.py new file mode 100644 index 0000000..33b9f32 --- /dev/null +++ b/studio/cli/run.py @@ -0,0 +1,70 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``python -m studio.cli.run `` -- the subprocess entry point. + +Deliberately thin. ``run_coupled`` is a library call with no ``__main__`` of its own (BLOCKING-3), +so something has to be the process that ``LocalSubprocessRunner`` launches, and this is it. All the +work is in ``studio.modelio.execute``; everything here is argument handling and exit codes. + +Exit codes are the runner's only structured signal, so they mean something specific: + +* ``0`` -- the model ran and the outputs were written. +* ``2`` -- the input could not be read or validated. Nothing was run. +* ``1`` -- the model raised. The traceback is on stderr, which the runner captures in full so the + failure can be diagnosed **without re-running it**. + +Nothing is caught and turned into a plausible result. A run that fails must look failed. +""" + +from __future__ import annotations + +import argparse +import sys +import traceback +from pathlib import Path + +from studio.resolve import ResolvedConfig + +#: Exit code for a bad or unreadable input, distinguished from a model failure so the runner can +#: tell "we never started" from "it broke". +EXIT_BAD_INPUT = 2 +EXIT_MODEL_FAILURE = 1 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="python -m studio.cli.run", + description="Run one resolved Plume Studio configuration and write its outputs.", + ) + parser.add_argument( + "input", type=Path, help="resolved config JSON (a serialised ResolvedConfig)" + ) + parser.add_argument( + "outdir", type=Path, help="directory to write state.npz and summary.json into" + ) + args = parser.parse_args(argv) + + try: + config = ResolvedConfig.model_validate_json(args.input.read_text(encoding="utf-8")) + config.require_consistent() + except Exception as exc: + print(f"[studio] cannot run {args.input}: {type(exc).__name__}: {exc}", file=sys.stderr) + return EXIT_BAD_INPUT + + # Imported here, not at module scope: it pulls in the model and therefore JAX, and a bad input + # should fail in milliseconds rather than after a second of imports. + from studio.modelio.execute import run_and_write + + try: + written = run_and_write(config, args.outdir) + except Exception: + traceback.print_exc() + return EXIT_MODEL_FAILURE + + for name, path in written.items(): + print(f"[studio] wrote {name}: {path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/modelio/execute.py b/studio/modelio/execute.py new file mode 100644 index 0000000..db1070b --- /dev/null +++ b/studio/modelio/execute.py @@ -0,0 +1,149 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Running the model and writing what it produced. + +Lives at the seam because it calls ``run_coupled``. The runner (``studio/runner``) knows about +processes, limits and lifecycle; this module knows about the model. Neither imports the other's +concerns. + +Two things about the model shape the design here, both from BLOCKING-3: + +* ``run_coupled`` is a **library call with no ``__main__``**. It returns arrays in memory and writes + nothing. So the caller writes the npz, and ``studio/cli/run.py`` exists to be the process. +* It **prints** its diagnostics rather than logging them, so stdout is the run's log stream and the + runner captures it as one. + +The npz key set mirrors ``run_ensemble.py:150-156``, which is the canonical one every existing +figure script reads. The size-distribution reduction, though, goes through ``studio.science`` rather +than being inlined a fifth time -- that consolidation is exactly what task 0.5 was for. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np + +from studio.modelio.scenario import to_scenario +from studio.modelio.summary import RunSummary, TerminationReason, summarise_state_npz +from studio.resolve import ResolvedConfig +from studio.science import bin_midpoints_um, dn_dlogdp + +#: Tolerance for "the run reached the end of its requested duration", in seconds. One output step is +#: 600 s nominal, so half a step is comfortably inside the noise of a completed run and far outside +#: an early stop. +_COMPLETION_TOLERANCE_S = 300.0 + + +def run_and_write(config: ResolvedConfig, out_dir: Path) -> dict[str, Path]: + """Run the model and write ``state.npz`` and ``summary.json`` into ``out_dir``. + + Returns the paths written. Raises whatever the model raises: a failed run is a failed run, and + the runner records the traceback from stderr rather than this function inventing a result. + """ + from coupled.driver import run_coupled + from coupled.tomas_bridge import _bad + + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + scenario = to_scenario(config) + + requested_s = float(config.config.schedule.duration_days) * 86400.0 + stop_condition = _max_sim_time_stop(config) + + times, states, aerosol, final_state, size_dist, photolysis = run_coupled( + scenario, + return_aerosol=True, + return_state=True, + return_size_dist=True, + return_photolysis=True, + stop_condition=stop_condition, + ) + + from config import IDX, air_number_density # the model's flat modules, via the seam + + edges_um = _bad._xk_to_dp_um(np.asarray(final_state.xk)) + state_path = out_dir / "state.npz" + np.savez( + state_path, + t=times, + x=states, + species=list(IDX), + M=air_number_density(scenario.P, scenario.T), + SA=aerosol["SA"], + radius_cm=aerosol["radius_cm"], + h2so4wp=aerosol["h2so4wp"], + particulate_S=aerosol["particulate_S"], + T=aerosol["T"], + n_cm3=size_dist["n_cm3"], + Dp_m=size_dist["Dp_m"], + dp_mid_um=bin_midpoints_um(edges_um), + dNdlogDp=dn_dlogdp(size_dist["n_cm3"], edges_um), + V_ratio=_volume_ratio(times, scenario), + total_n=size_dist["n_cm3"].sum(axis=1), + J_tmid=photolysis["t_mid"], + J=photolysis["J"], + J_equations=photolysis["equations"], + ) + + termination = ( + TerminationReason.COMPLETED + if float(times[-1]) >= requested_s - _COMPLETION_TOLERANCE_S + else TerminationReason.TERMINATED_ON_LIMIT + ) + summary = summarise_state_npz( + state_path, + label=config.config.config_hash()[:12], + config_hash=config.config.config_hash(), + termination=termination, + ) + summary_path = summary.write(out_dir / "summary.json") + return {"state": state_path, "summary": summary_path} + + +def _max_sim_time_stop(config: ResolvedConfig) -> Any: + """A ``stop_condition`` enforcing ``termination.max_sim_time_days``, or ``None``. + + **The two-argument shape is deliberate and load-bearing.** Task 0.8 (PR #73) widens the model's + callback to take a diagnostics dict and dispatches on the callback's DECLARED ARITY: + + =========================== ========================================== + ``def stop(t1, wet_SA)`` accepted as legacy, with a DeprecationWarning + ``def stop(diag)`` accepted as the new dict shape + ``def stop(*args)`` **TypeError** -- it matches both, so it is ambiguous + =========================== ========================================== + + Raising on ``*args`` is the right call by the model: guessing which shape a variadic callback + wanted would be a silent wrong answer. But it means the obvious "works with either" spelling is + the one thing that does not, so this stays two-positional -- which works against the model both + before and after that change. + + Migrating to the dict shape is worth doing once #73 is in ``studio/dev``: it is what makes + termination criteria on SO2 or particle number possible, which is the reason the callback was + widened at all. Until then this only needs the simulated time, which is the first argument in + both shapes. + """ + limit_days = config.config.termination.max_sim_time_days + if limit_days is None: + return None + limit_s = float(limit_days) * 86400.0 + + def stop(t1_seconds: float, wet_surface_area: float) -> bool: + return float(t1_seconds) >= limit_s + + return stop + + +def _volume_ratio(times: np.ndarray, scenario: Any) -> np.ndarray: + """V(t)/V0 for the run, from the model's own tested implementation. + + Reused rather than reimplemented: ``coupled.dilution.volume_ratio`` is tested, and a second copy + in ``studio/science`` is exactly what task 0.5 removed elsewhere. + """ + from coupled import dilution + + return dilution.volume_ratio(times, scenario.dilution_regime) + + +__all__ = ["RunSummary", "run_and_write"] diff --git a/studio/runner/__init__.py b/studio/runner/__init__.py index 5e9dff5..a8612e9 100644 --- a/studio/runner/__init__.py +++ b/studio/runner/__init__.py @@ -31,4 +31,32 @@ from __future__ import annotations -__all__: list[str] = [] +from studio.runner.base import ( + TERMINAL_STATES, + CloudBatchRunner, + InvalidTransitionError, + JobRecord, + JobRegistry, + JobRunner, + JobState, + NotImplementedRunner, + SlurmRunner, + Transition, +) +from studio.runner.local import DEFAULT_MAX_WORKERS, THREAD_PINNING, LocalSubprocessRunner + +__all__ = [ + "DEFAULT_MAX_WORKERS", + "TERMINAL_STATES", + "THREAD_PINNING", + "CloudBatchRunner", + "InvalidTransitionError", + "JobRecord", + "JobRegistry", + "JobRunner", + "JobState", + "LocalSubprocessRunner", + "NotImplementedRunner", + "SlurmRunner", + "Transition", +] diff --git a/studio/runner/base.py b/studio/runner/base.py new file mode 100644 index 0000000..5750e01 --- /dev/null +++ b/studio/runner/base.py @@ -0,0 +1,292 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The job lifecycle and the ``JobRunner`` interface. + +**Nothing above this interface may assume an execution backend** (ADR-008). Today there is one +implementation, ``LocalSubprocessRunner``; Slurm and cloud-batch adapters do not exist and raise +rather than degrade. That constraint is why the lifecycle lives here as data rather than inside the +local runner: a scheduler-backed runner would report the same states and the same transitions. + +Every transition is timestamped, and the record keeps all of them rather than just the current +state. "It failed" is not debuggable; "QUEUED at 14:02:11, RUNNING at 14:02:11, FAILED at 14:06:48 +with exit code 1" is. The same reasoning drives what a finished job keeps on disk: the resolved +input, the captured stdout, the captured stderr and the exit code, so **a failed run can be +diagnosed without re-running it** -- which matters when re-running costs four minutes and the +failure is intermittent. + +``run_coupled`` prints diagnostics to stdout rather than logging them (``[gas] dt0 stall...``, +``[coupled] NOTE:...``, ``[stop]...``), so stdout IS the run's log stream and is captured as such. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from datetime import UTC, datetime +from enum import StrEnum +from pathlib import Path +from typing import Protocol, runtime_checkable + +from pydantic import BaseModel, ConfigDict, Field + + +class JobState(StrEnum): + """Where a job is. The terminal states are distinguished on purpose. + + ``FAILED`` and ``TERMINATED_ON_LIMIT`` are not the same thing and must never be collapsed: the + first means the model could not produce a result, the second means it was still going when we + stopped it. A run stopped on a limit has partial output that may look complete, so it is flagged + and **never presented as converged**. + """ + + DRAFT = "draft" + QUEUED = "queued" + RUNNING = "running" + SUCCEEDED = "succeeded" + FAILED = "failed" + CANCELLED = "cancelled" + TERMINATED_ON_LIMIT = "terminated_on_limit" + + +#: States from which no further transition is possible. +TERMINAL_STATES = frozenset( + { + JobState.SUCCEEDED, + JobState.FAILED, + JobState.CANCELLED, + JobState.TERMINATED_ON_LIMIT, + } +) + +#: The only transitions the lifecycle allows. Enforced rather than documented: a job that went +#: RUNNING -> QUEUED, or that reported SUCCEEDED twice, means the runner lost track of a process, +#: and silently accepting it would make the record a story rather than a log. +_ALLOWED: dict[JobState, frozenset[JobState]] = { + JobState.DRAFT: frozenset({JobState.QUEUED, JobState.CANCELLED}), + JobState.QUEUED: frozenset({JobState.RUNNING, JobState.CANCELLED, JobState.FAILED}), + JobState.RUNNING: frozenset( + { + JobState.SUCCEEDED, + JobState.FAILED, + JobState.CANCELLED, + JobState.TERMINATED_ON_LIMIT, + } + ), + JobState.SUCCEEDED: frozenset(), + JobState.FAILED: frozenset(), + JobState.CANCELLED: frozenset(), + JobState.TERMINATED_ON_LIMIT: frozenset(), +} + + +class InvalidTransitionError(ValueError): + """An illegal lifecycle transition. Raised, never tolerated.""" + + +def _now() -> datetime: + """Timezone-aware UTC. Naive timestamps compare wrongly across a DST boundary.""" + return datetime.now(UTC) + + +class Transition(BaseModel): + """One state change, with when and why.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + state: JobState + at: datetime + detail: str = "" + + +class JobRecord(BaseModel): + """Everything known about one job. Immutable; a transition produces a new record. + + Immutable because this is the audit trail. A record that can be edited in place is a record that + can quietly disagree with what happened. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + job_id: str + #: Identity of the config being run (ADR-006). Links the job to its inputs and its cache entry. + config_hash: str + label: str = "" + state: JobState = JobState.DRAFT + transitions: tuple[Transition, ...] = () + #: Where the resolved input, the logs and the output live. Present from submission, so a job + #: that dies early still says where to look. + work_dir: Path | None = None + input_path: Path | None = None + stdout_path: Path | None = None + stderr_path: Path | None = None + exit_code: int | None = None + #: Set when the job ends for any reason; a short human-facing explanation. + detail: str = "" + + @property + def is_terminal(self) -> bool: + return self.state in TERMINAL_STATES + + def _first_time(self, state: JobState) -> datetime | None: + """When the job FIRST entered ``state``, or ``None`` if it never did.""" + for transition in self.transitions: + if transition.state is state: + return transition.at + return None + + @property + def submitted_at(self) -> datetime | None: + return self._first_time(JobState.QUEUED) + + @property + def started_at(self) -> datetime | None: + return self._first_time(JobState.RUNNING) + + @property + def ended_at(self) -> datetime | None: + for transition in reversed(self.transitions): + if transition.state in TERMINAL_STATES: + return transition.at + return None + + @property + def duration_s(self) -> float | None: + """Wall-clock from RUNNING to the terminal state, or ``None`` if it has not run yet.""" + started, ended = self.started_at, self.ended_at + if started is None or ended is None: + return None + return (ended - started).total_seconds() + + def transition_to(self, state: JobState, detail: str = "", **updates: object) -> JobRecord: + """Return a new record in ``state``. + + Raises: + InvalidTransitionError: If the lifecycle does not allow it. + """ + if state not in _ALLOWED[self.state]: + allowed = sorted(s.value for s in _ALLOWED[self.state]) + raise InvalidTransitionError( + f"job {self.job_id} cannot go {self.state.value} -> {state.value}; " + f"allowed from {self.state.value}: {allowed or ['(terminal)']}" + ) + return self.model_copy( + update={ + "state": state, + "transitions": ( + *self.transitions, + Transition(state=state, at=_now(), detail=detail), + ), + "detail": detail or self.detail, + **updates, + } + ) + + +@runtime_checkable +class JobRunner(Protocol): + """How Studio executes runs. The only assumption anything above may make. + + Deliberately small. ``submit`` takes a resolved config and returns a record; everything else + operates on a job id. A Slurm implementation would satisfy this without any caller changing. + """ + + def submit(self, config: object, *, label: str = "") -> JobRecord: + """Queue a run. Returns immediately with a record in QUEUED.""" + ... + + def poll(self, job_id: str) -> JobRecord: + """Current record, advancing the state if the process has finished or exceeded its limit.""" + ... + + def cancel(self, job_id: str) -> JobRecord: + """Stop a queued or running job. Terminal jobs are returned unchanged.""" + ... + + def artifacts(self, job_id: str) -> Sequence[Path]: + """Files this job produced, if any.""" + ... + + +class NotImplementedRunner: + """Base for backends that do not exist yet (ADR-008). + + Present so that "Slurm is not supported" is a class you can point at rather than a gap in a + dispatch table. Every method raises with the same message; nothing degrades to local execution, + because a job silently running somewhere other than where it was sent is worse than an error. + """ + + backend_name = "unimplemented" + + def _raise(self) -> None: + raise NotImplementedError( + f"the {self.backend_name} execution backend is not implemented. ADR-008 records local " + f"subprocesses as the only supported backend; nothing falls back to local execution, " + f"because a job running somewhere other than where it was sent is worse than an error." + ) + + def submit(self, config: object, *, label: str = "") -> JobRecord: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + def poll(self, job_id: str) -> JobRecord: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + def cancel(self, job_id: str) -> JobRecord: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + def artifacts(self, job_id: str) -> Sequence[Path]: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + +class SlurmRunner(NotImplementedRunner): + """Not implemented (ADR-008, BLOCKING-1).""" + + backend_name = "Slurm" + + +class CloudBatchRunner(NotImplementedRunner): + """Not implemented (ADR-008, BLOCKING-1).""" + + backend_name = "cloud-batch" + + +class JobRegistry(BaseModel): + """In-memory job records, keyed by id. + + Phase 0 keeps this in the process that owns the worker pool (ADR-007). ADR-004 puts job state in + the database so a handle survives an API restart; this class is the seam that will move there, + and it is deliberately dumb so that swap is a swap rather than a rewrite. + """ + + model_config = ConfigDict(extra="forbid") + + records: dict[str, JobRecord] = Field(default_factory=dict) + + def put(self, record: JobRecord) -> JobRecord: + self.records[record.job_id] = record + return record + + def get(self, job_id: str) -> JobRecord: + try: + return self.records[job_id] + except KeyError: + raise KeyError(f"unknown job id {job_id!r}") from None + + def __contains__(self, job_id: object) -> bool: + return job_id in self.records + + +__all__ = [ + "TERMINAL_STATES", + "CloudBatchRunner", + "InvalidTransitionError", + "JobRecord", + "JobRegistry", + "JobRunner", + "JobState", + "NotImplementedRunner", + "SlurmRunner", + "Transition", +] diff --git a/studio/runner/local.py b/studio/runner/local.py new file mode 100644 index 0000000..d81943d --- /dev/null +++ b/studio/runner/local.py @@ -0,0 +1,252 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``LocalSubprocessRunner`` -- the only execution backend that exists (ADR-008). + +Each run is one thread-pinned subprocess. The pinning is not incidental: setting +``OMP/OPENBLAS/MKL/VECLIB/NUMEXPR_NUM_THREADS=1`` and disabling XLA's multithreaded Eigen is what +gives ~N times the throughput for N workers on this workload -- **not** ``vmap``. It is copied from +``coupled/paper_ensemble/launch_parallel.py:26-30``, which is the configuration the 810-run ensemble +was actually produced with. + +Why a subprocess at all: ``run_coupled`` is a library call with no ``__main__`` of its own, it +returns arrays in memory and writes nothing, and it prints its diagnostics. So something has to be +the process, that something is ``python -m studio.cli.run``, and its stdout is the run's log. + +What is on disk when a job ends, whatever the outcome: + +* ``input.json`` -- the RESOLVED config that was actually run +* ``stdout.log`` / ``stderr.log`` -- captured in full +* the exit code and every state transition, in the record + +That set is chosen so a failure can be diagnosed **without re-running it**, which matters when a +re-run costs minutes and the failure is intermittent. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import uuid +from collections.abc import Sequence +from concurrent.futures import Future, ThreadPoolExecutor +from pathlib import Path +from typing import Any, Final + +from studio.resolve import ResolvedConfig +from studio.runner.base import JobRecord, JobRegistry, JobState + +#: Single-thread pinning, copied from ``launch_parallel.py:26-30``. This is what makes N concurrent +#: runs ~N times the throughput; without it they fight over cores and each one gets slower. +THREAD_PINNING: Final[dict[str, str]] = { + "OMP_NUM_THREADS": "1", + "OPENBLAS_NUM_THREADS": "1", + "MKL_NUM_THREADS": "1", + "VECLIB_MAXIMUM_THREADS": "1", + "NUMEXPR_NUM_THREADS": "1", + "XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false", +} + +#: Default concurrent subprocesses (ASSUMPTION-4). Each run is single-threaded by the pinning above, +#: so this is a core count rather than a guess about memory. Existing practice is 10. +DEFAULT_MAX_WORKERS = 4 + +#: Grace period between SIGTERM and SIGKILL when stopping a run, in seconds. Long enough for Python +#: to unwind and flush the log; short enough that a wedged process does not hold a worker. +_TERMINATE_GRACE_S = 5.0 + + +class LocalSubprocessRunner: + """Run jobs as local subprocesses from a bounded pool. + + Args: + work_root: Directory under which each job gets ``//``. + max_workers: Concurrent subprocesses (ASSUMPTION-4). + entry_module: The module launched with ``-m``. Overridable so the LIFECYCLE can be tested + without a four-minute model run -- the default is the real path, and nothing in this + class branches on the value. It is a parameter, not a test hook. + python_executable: Interpreter for the subprocess; defaults to the current one, so a job + inherits the environment that submitted it rather than whatever is first on PATH. + """ + + def __init__( + self, + work_root: Path, + *, + max_workers: int = DEFAULT_MAX_WORKERS, + entry_module: str = "studio.cli.run", + python_executable: str | None = None, + ) -> None: + if max_workers < 1: + raise ValueError(f"max_workers must be >= 1, got {max_workers}") + self.work_root = Path(work_root) + self.work_root.mkdir(parents=True, exist_ok=True) + self.max_workers = max_workers + self.entry_module = entry_module + self.python_executable = python_executable or sys.executable + self.registry = JobRegistry() + self._pool = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="studio-run") + self._processes: dict[str, subprocess.Popen[bytes]] = {} + self._futures: dict[str, Future[None]] = {} + + # -- submission ------------------------------------------------------------------------ + + def submit(self, config: ResolvedConfig, *, label: str = "") -> JobRecord: + """Write the resolved input, queue the run, return immediately. + + Raises: + InconsistentConfigError: If the config has stale overrides. A run started from one would + produce results that do not follow from their own inputs, and nothing downstream + could tell. + """ + config.require_consistent() + job_id = uuid.uuid4().hex[:12] + work_dir = self.work_root / job_id + work_dir.mkdir(parents=True, exist_ok=False) + + input_path = work_dir / "input.json" + input_path.write_text(config.model_dump_json(indent=2), encoding="utf-8") + + record = JobRecord( + job_id=job_id, + config_hash=config.config.config_hash(), + label=label, + work_dir=work_dir, + input_path=input_path, + stdout_path=work_dir / "stdout.log", + stderr_path=work_dir / "stderr.log", + ).transition_to(JobState.QUEUED, detail=f"queued for {self.entry_module}") + self.registry.put(record) + + max_wall_time_s = config.config.termination.max_wall_time_s + self._futures[job_id] = self._pool.submit(self._execute, job_id, max_wall_time_s) + return record + + # -- execution ------------------------------------------------------------------------- + + def _execute(self, job_id: str, max_wall_time_s: float) -> None: + """Run one job to completion. Runs on a pool thread; never raises into the pool.""" + record = self.registry.get(job_id) + if record.state is JobState.CANCELLED: + return # cancelled while queued + work_dir = record.work_dir + assert work_dir is not None and record.stdout_path and record.stderr_path + + command = [ + self.python_executable, + "-m", + self.entry_module, + str(record.input_path), + str(work_dir), + ] + env = {**os.environ, **THREAD_PINNING} + try: + with ( + record.stdout_path.open("wb") as stdout, + record.stderr_path.open("wb") as stderr, + ): + process = subprocess.Popen(command, stdout=stdout, stderr=stderr, env=env) + self._processes[job_id] = process + self.registry.put( + self.registry.get(job_id).transition_to( + JobState.RUNNING, detail=" ".join(command) + ) + ) + try: + exit_code = process.wait(timeout=max_wall_time_s) + except subprocess.TimeoutExpired: + self._stop(process) + self._finish( + job_id, + JobState.TERMINATED_ON_LIMIT, + detail=( + f"exceeded max_wall_time_s = {max_wall_time_s}; partial output is " + f"NOT a converged result" + ), + exit_code=process.returncode, + ) + return + except Exception as exc: + self._finish(job_id, JobState.FAILED, detail=f"{type(exc).__name__}: {exc}") + return + finally: + self._processes.pop(job_id, None) + + current = self.registry.get(job_id) + if current.state is JobState.CANCELLED: + return + if exit_code == 0: + self._finish(job_id, JobState.SUCCEEDED, detail="completed", exit_code=exit_code) + else: + self._finish( + job_id, + JobState.FAILED, + detail=( + f"exit code {exit_code}; see {record.stderr_path.name} in {work_dir} -- the " + f"resolved input and both log streams are kept so this needs no re-run" + ), + exit_code=exit_code, + ) + + def _finish( + self, job_id: str, state: JobState, *, detail: str, exit_code: int | None = None + ) -> None: + record = self.registry.get(job_id) + if record.is_terminal: + return + self.registry.put(record.transition_to(state, detail=detail, exit_code=exit_code)) + + @staticmethod + def _stop(process: subprocess.Popen[Any]) -> None: + """SIGTERM, then SIGKILL if it will not go. Python gets a chance to flush its log first.""" + process.terminate() + try: + process.wait(timeout=_TERMINATE_GRACE_S) + except subprocess.TimeoutExpired: + process.kill() + process.wait() + + # -- inspection ------------------------------------------------------------------------ + + def poll(self, job_id: str) -> JobRecord: + """The current record. State is advanced by the worker thread, not by polling. + + Polling is therefore free and side-effect-free, which matters because the API pushes + progress over SSE (spec 7.3) and would otherwise poll this on a timer per connected client. + """ + return self.registry.get(job_id) + + def cancel(self, job_id: str) -> JobRecord: + """Stop a queued or running job. A terminal job is returned unchanged, not an error.""" + record = self.registry.get(job_id) + if record.is_terminal: + return record + process = self._processes.get(job_id) + if process is not None: + self._stop(process) + cancelled = record.transition_to(JobState.CANCELLED, detail="cancelled by request") + return self.registry.put(cancelled) + + def artifacts(self, job_id: str) -> Sequence[Path]: + """Files this job produced, sorted. Includes the logs and the resolved input on failure.""" + record = self.registry.get(job_id) + if record.work_dir is None or not record.work_dir.is_dir(): + return () + return tuple(sorted(p for p in record.work_dir.iterdir() if p.is_file())) + + def wait(self, job_id: str, timeout: float | None = None) -> JobRecord: + """Block until the job reaches a terminal state. For the CLI and for tests, not the API.""" + future = self._futures.get(job_id) + if future is not None: + future.result(timeout=timeout) + return self.registry.get(job_id) + + def shutdown(self, *, cancel_running: bool = False) -> None: + """Stop accepting work; optionally stop what is already running.""" + if cancel_running: + for job_id in list(self._processes): + self.cancel(job_id) + self._pool.shutdown(wait=True) + + +__all__ = ["DEFAULT_MAX_WORKERS", "THREAD_PINNING", "LocalSubprocessRunner"] diff --git a/studio/tests/fixtures/__init__.py b/studio/tests/fixtures/__init__.py new file mode 100644 index 0000000..d8c6981 --- /dev/null +++ b/studio/tests/fixtures/__init__.py @@ -0,0 +1,3 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Fixture modules launched as real subprocesses by the runner tests.""" diff --git a/studio/tests/fixtures/fake_run.py b/studio/tests/fixtures/fake_run.py new file mode 100644 index 0000000..6d4595a --- /dev/null +++ b/studio/tests/fixtures/fake_run.py @@ -0,0 +1,56 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""A stand-in for ``studio.cli.run`` with the same argv shape and no model. + +``LocalSubprocessRunner`` takes ``entry_module`` as a parameter so its lifecycle -- queueing, +timeouts, cancellation, log capture, exit codes -- can be exercised in milliseconds instead of the +three to five minutes a real 10-day case costs. Nothing in the runner branches on the value; the +default is the real entry point, and the real one is exercised separately by +``test_a_bad_input_file_exits_distinctly_from_a_model_failure``. + +Behaviour comes from the ``STUDIO_FAKE_RUN`` environment variable, as JSON. An environment variable +rather than a file in the work directory because the subprocess can start before a file written +after ``submit()`` lands -- a race that would make these tests flaky in exactly the way process +tests usually are. + +Modes: ``ok`` (print and exit 0), ``fail`` (print to stderr and exit 1), ``sleep`` (sleep, to be +killed by the wall-clock limit or by cancellation), ``dump_env`` (print the thread-pinning variables +this process actually received). +""" + +from __future__ import annotations + +import json +import os +import sys +import time + +#: The pinning the runner is supposed to apply. Imported rather than restated so that the test +#: asserting it cannot pass against a stale copy of the list. +from studio.runner.local import THREAD_PINNING + + +def main(argv: list[str] | None = None) -> int: + argv = sys.argv[1:] if argv is None else argv + if len(argv) != 2: + print(f"usage: fake_run ; got {argv}", file=sys.stderr) + return 2 + + directive = json.loads(os.environ.get("STUDIO_FAKE_RUN", '{"mode": "ok"}')) + mode = directive.get("mode", "ok") + + if mode == "dump_env": + print(json.dumps({key: os.environ.get(key) for key in THREAD_PINNING})) + return 0 + if mode == "sleep": + time.sleep(float(directive.get("seconds", 30))) + return 0 + if mode == "fail": + print(directive.get("message", "failed"), file=sys.stderr) + return 1 + print(directive.get("message", "[studio] fake run complete")) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/tests/unit/test_runner.py b/studio/tests/unit/test_runner.py new file mode 100644 index 0000000..5f56b04 --- /dev/null +++ b/studio/tests/unit/test_runner.py @@ -0,0 +1,370 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The job lifecycle and the local subprocess runner. + +These launch **real subprocesses** -- the runner's whole job is process management, and a mocked +``Popen`` would test the mock. What they do not launch is the model: ``entry_module`` points at a +small fixture module that exits, fails, or sleeps on command. That is a parameter of the runner +rather than a test hook: nothing in ``LocalSubprocessRunner`` branches on its value, and the default +is the real entry point. + +The tests that matter most are the ones about **what survives a failure**. A run that dies at minute +three of four must leave enough behind to diagnose it without paying those three minutes again. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import pytest + +from studio.resolve import ResolvedConfig, apply_change, resolve, set_override +from studio.runner import ( + THREAD_PINNING, + InvalidTransitionError, + JobRecord, + JobState, + LocalSubprocessRunner, + SlurmRunner, +) +from studio.schema import RunConfig + +#: A stand-in for ``studio.cli.run``: same argv shape, controllable outcome, no model. +FIXTURE_MODULE = "studio.tests.fixtures.fake_run" + + +@pytest.fixture +def resolved() -> ResolvedConfig: + return resolve(RunConfig()) + + +@pytest.fixture +def runner(tmp_path: Path) -> LocalSubprocessRunner: + made = LocalSubprocessRunner(tmp_path / "jobs", max_workers=2, entry_module=FIXTURE_MODULE) + yield made + made.shutdown(cancel_running=True) + + +class TestLifecycle: + """Transitions are data, and illegal ones raise.""" + + @pytest.mark.tier_a + def test_the_happy_path_records_every_step(self) -> None: + record = ( + JobRecord(job_id="j", config_hash="h") + .transition_to(JobState.QUEUED) + .transition_to(JobState.RUNNING) + .transition_to(JobState.SUCCEEDED) + ) + assert [t.state for t in record.transitions] == [ + JobState.QUEUED, + JobState.RUNNING, + JobState.SUCCEEDED, + ] + assert record.is_terminal + assert record.submitted_at and record.started_at and record.ended_at + assert record.duration_s is not None and record.duration_s >= 0.0 + + @pytest.mark.tier_a + @pytest.mark.parametrize( + ("from_state", "to_state"), + [ + (JobState.DRAFT, JobState.RUNNING), # never ran the queue + (JobState.RUNNING, JobState.QUEUED), # backwards + (JobState.SUCCEEDED, JobState.RUNNING), # terminal + (JobState.FAILED, JobState.SUCCEEDED), # rewriting history + ], + ) + def test_illegal_transitions_raise(self, from_state: JobState, to_state: JobState) -> None: + """A job that appears to move backwards means the runner lost track of a process. + + Accepting it silently would turn the record from a log into a story. + """ + record = JobRecord(job_id="j", config_hash="h").model_copy(update={"state": from_state}) + with pytest.raises(InvalidTransitionError): + record.transition_to(to_state) + + @pytest.mark.tier_a + def test_a_record_is_immutable(self) -> None: + """It is an audit trail; one that can be edited in place can disagree with what happened.""" + record = JobRecord(job_id="j", config_hash="h") + with pytest.raises(ValueError, match="frozen"): + record.state = JobState.RUNNING # type: ignore[misc] + assert record.transition_to(JobState.QUEUED) is not record + + @pytest.mark.tier_a + def test_terminated_on_limit_is_not_failed(self) -> None: + """Two different things: "could not produce a result" vs "we stopped it mid-flight". + + Collapsing them would let a partial run be read as a converged one. + """ + assert JobState.TERMINATED_ON_LIMIT != JobState.FAILED + record = ( + JobRecord(job_id="j", config_hash="h") + .transition_to(JobState.QUEUED) + .transition_to(JobState.RUNNING) + .transition_to(JobState.TERMINATED_ON_LIMIT, detail="exceeded max_wall_time_s") + ) + assert record.is_terminal + assert "max_wall_time" in record.detail + + +class TestLocalSubprocessRunner: + @pytest.mark.tier_a + def test_a_successful_run_reaches_succeeded( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + record = runner.submit(resolved, label="ok") + assert record.state is JobState.QUEUED + final = runner.wait(record.job_id, timeout=30) + assert final.state is JobState.SUCCEEDED + assert final.exit_code == 0 + assert final.config_hash == resolved.config.config_hash() + + @pytest.mark.tier_a + def test_the_resolved_input_is_written_before_the_run( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Written at submit, not completion, so a job that dies at once still has its input.""" + record = runner.submit(resolved) + assert record.input_path is not None and record.input_path.is_file() + restored = ResolvedConfig.model_validate_json(record.input_path.read_text()) + assert restored.config.config_hash() == resolved.config.config_hash() + runner.wait(record.job_id, timeout=30) + + @pytest.mark.tier_a + def test_a_failed_run_keeps_everything_needed_to_diagnose_it( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The point of the whole design: no re-run required. + + A four-minute case that fails intermittently must not have to be reproduced to be + understood, so the exit code, both log streams and the exact input are all on disk. + """ + _directive(monkeypatch, mode="fail", message="synthetic failure") + record = runner.submit(resolved, label="boom") + final = runner.wait(record.job_id, timeout=30) + + assert final.state is JobState.FAILED + assert final.exit_code == 1 + assert final.stderr_path is not None + assert "synthetic failure" in final.stderr_path.read_text() + assert final.input_path is not None and final.input_path.is_file() + names = {path.name for path in runner.artifacts(final.job_id)} + assert {"input.json", "stdout.log", "stderr.log"} <= names + + @pytest.mark.tier_a + def test_stdout_is_captured_as_the_log_stream( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """``run_coupled`` prints rather than logs, so stdout IS the run log.""" + _directive(monkeypatch, mode="ok", message="[coupled] NOTE: something happened") + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + assert final.stdout_path is not None + assert "[coupled] NOTE: something happened" in final.stdout_path.read_text() + + @pytest.mark.tier_a + def test_exceeding_the_wall_clock_limit_is_terminated_not_failed( + self, runner: LocalSubprocessRunner, monkeypatch: pytest.MonkeyPatch + ) -> None: + """And the record says so, so nothing downstream reads the partial output as converged.""" + config = resolve( + RunConfig.model_validate( + {**RunConfig().model_dump(), "termination": {"max_wall_time_s": 1.0}} + ) + ) + _directive(monkeypatch, mode="sleep", seconds=30) + record = runner.submit(config, label="slow") + final = runner.wait(record.job_id, timeout=60) + + assert final.state is JobState.TERMINATED_ON_LIMIT + assert "max_wall_time_s" in final.detail + assert "NOT a converged result" in final.detail + + @pytest.mark.tier_a + def test_cancelling_a_running_job_stops_it( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + _directive(monkeypatch, mode="sleep", seconds=30) + record = runner.submit(resolved) + _wait_for_state(runner, record.job_id, JobState.RUNNING) + cancelled = runner.cancel(record.job_id) + assert cancelled.state is JobState.CANCELLED + assert runner.poll(record.job_id).state is JobState.CANCELLED + + @pytest.mark.tier_a + def test_cancelling_a_finished_job_is_not_an_error( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Cancelling something that already finished is a race, not a mistake by the caller.""" + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + assert runner.cancel(final.job_id).state is final.state + + @pytest.mark.tier_a + def test_a_stale_config_is_refused_at_submission(self, runner: LocalSubprocessRunner) -> None: + """Nothing downstream could tell that the numbers did not follow from each other.""" + from studio.resolve import InconsistentConfigError + + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError): + runner.submit(stale) + + @pytest.mark.tier_a + def test_polling_an_unknown_job_raises(self, runner: LocalSubprocessRunner) -> None: + with pytest.raises(KeyError, match="unknown job id"): + runner.poll("nope") + + @pytest.mark.tier_a + def test_jobs_get_separate_work_directories( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Two runs of the SAME config must not share a directory and overwrite each other.""" + first = runner.submit(resolved) + second = runner.submit(resolved) + assert first.work_dir != second.work_dir + assert first.config_hash == second.config_hash, "same config, same identity" + for job in (first, second): + runner.wait(job.job_id, timeout=30) + + @pytest.mark.tier_a + def test_the_subprocess_gets_the_thread_pinning_environment( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """This pinning -- not vmap -- is what gives ~N times the throughput for N workers. + + Asserted by having the fixture module dump its own environment, because a runner that + *intends* to pin threads and does not would only show up as everything being slow. + """ + _directive(monkeypatch, mode="dump_env") + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + assert final.stdout_path is not None + reported = json.loads(final.stdout_path.read_text()) + assert {key: reported.get(key) for key in THREAD_PINNING} == THREAD_PINNING + + @pytest.mark.tier_a + def test_a_bad_input_file_exits_distinctly_from_a_model_failure( + self, tmp_path: Path, resolved: ResolvedConfig + ) -> None: + """Exit code 2 means "never started"; 1 means "the model raised". The runner needs both. + + Runs the REAL entry point (``studio.cli.run``), because this is its contract, and a bad + input is rejected before any model import -- so it costs milliseconds, not a JAX load. + """ + import subprocess + + bad = tmp_path / "bad.json" + bad.write_text('{"config": {"site": {"temperature_k": -5}}}', encoding="utf-8") + proc = subprocess.run( + [sys.executable, "-m", "studio.cli.run", str(bad), str(tmp_path / "out")], + capture_output=True, + text=True, + cwd=str(Path(__file__).resolve().parents[3]), + ) + assert proc.returncode == 2, proc.stderr + assert "cannot run" in proc.stderr + + +class TestUnimplementedBackends: + @pytest.mark.tier_a + def test_slurm_raises_rather_than_falling_back_to_local(self) -> None: + """A job running somewhere other than where it was sent is worse than an error (ADR-008).""" + with pytest.raises(NotImplementedError, match="Slurm"): + SlurmRunner().submit(None) + with pytest.raises(NotImplementedError, match="not implemented"): + SlurmRunner().poll("x") + + +def _directive(monkeypatch: pytest.MonkeyPatch, **directive: object) -> None: + """Tell the fixture module what to do, BEFORE the job is submitted. + + Via the environment rather than a file in the work directory: the subprocess can start before a + file written after ``submit()`` lands, which is exactly how process tests become flaky. + """ + monkeypatch.setenv("STUDIO_FAKE_RUN", json.dumps(directive)) + + +def _wait_for_state( + runner: LocalSubprocessRunner, job_id: str, state: JobState, timeout: float = 30.0 +) -> None: + import time + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if runner.poll(job_id).state is state: + return + time.sleep(0.02) + raise AssertionError(f"job {job_id} never reached {state}") + + +class TestMaxSimTimeStopCondition: + """The ``stop_condition`` this layer hands the model, and the shape it must have. + + Pure-function tests: no model run, but they pin the coupling that PR #73 exposed. The model + dispatches on the callback's DECLARED ARITY and raises ``TypeError`` on ``*args`` -- correctly, + since a variadic callback matches both shapes and guessing would be a silent wrong answer. That + makes the arity part of this function's contract rather than an implementation detail, so it is + asserted here where a change is cheap to notice. + """ + + @pytest.mark.tier_a + def test_no_limit_means_no_stop_condition(self) -> None: + """``None`` is not a callback that never fires; it is no callback at all.""" + from studio.modelio.execute import _max_sim_time_stop + + assert _max_sim_time_stop(resolve(RunConfig())) is None + + @pytest.mark.tier_a + def test_the_callback_takes_exactly_two_positional_arguments(self) -> None: + """Not ``*args``: PR #73 rejects a variadic callback as ambiguous. + + If this assertion ever fails because the shape moved to the diagnostics dict, that is the + intended migration -- update it deliberately, in the commit that does the migration. + """ + import inspect + + from studio.modelio.execute import _max_sim_time_stop + + stop = _max_sim_time_stop(_with_sim_limit(2.0)) + assert stop is not None + parameters = list(inspect.signature(stop).parameters.values()) + assert len(parameters) == 2 + assert all(p.kind is inspect.Parameter.POSITIONAL_OR_KEYWORD for p in parameters) + + @pytest.mark.tier_a + def test_it_fires_exactly_at_the_limit(self) -> None: + """Boundary included: at the limit the run has reached its cap, not almost reached it.""" + from studio.modelio.execute import _max_sim_time_stop + + stop = _max_sim_time_stop(_with_sim_limit(2.0)) + assert stop is not None + two_days_s = 2.0 * 86400.0 + assert stop(two_days_s - 1.0, 12.0) is False + assert stop(two_days_s, 12.0) is True + assert stop(two_days_s + 1.0, 12.0) is True + + +def _with_sim_limit(days: float) -> ResolvedConfig: + payload = RunConfig().model_dump() + payload["termination"]["max_sim_time_days"] = days + return resolve(RunConfig.model_validate(payload)) From d7bbbced8eaa62729420d0727d39aa0278e601fd Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 19:54:18 -0700 Subject: [PATCH 07/18] studio: refuse the temperature feedback (schema 0.2.0) (#75) Decision by Ali: longwave radiation is not in the radiative calculation, so there must be no temperature feedback in the model. The reasoning matters for how it is encoded. The model's heating term is shortwave-only (AD-5.4), so enabling it does not make the box thermodynamics more complete -- it makes them one-sided, and the ~+1.2 K / 10 d warm drift it produces is an artefact of the missing cooling rather than a physical response. A default of False would leave that a switch someone can flip; Literal[False] makes True fail validation, which is the same treatment dilution.background_evolves already had for the same class of reason. Two tests cover it: the direct one, and the sweep-axis path, which is the one that would slip past a guard implemented in the UI. SCHEMA_VERSION goes 0.1.0 -> 0.2.0 and the pinned config hash moves with it. The VALUE of heating_to_t did not change -- it was already False -- but schema_version is part of the hashed payload, and that is precisely what makes "old configs are never silently reinterpreted under new semantics" true rather than stated. A config written before this commit no longer hashes to a 0.2.0 identity, which is intended. Recorded where a reader of results would look: CAVEATS.md gains a top-level entry saying every run is isothermal at the configured temperature and must not be read as containing a plume-warming signal, and SCIENCE-4 is marked partly answered -- buoyant rise, sedimentation and the isobaric assumption remain open. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/CAVEATS.md | 9 +++++++ docs/studio/OPEN_QUESTIONS.md | 14 ++++++++--- docs/studio/PROGRESS.md | 29 ++++++++++++++++++++++ studio/schema/config.py | 27 +++++++++++++------- studio/tests/unit/test_config_hash.py | 7 ++++-- studio/tests/unit/test_resolve_resolver.py | 2 +- studio/tests/unit/test_schema_export.py | 28 ++++++++++++++++++++- 7 files changed, 100 insertions(+), 16 deletions(-) diff --git a/docs/studio/CAVEATS.md b/docs/studio/CAVEATS.md index b933ee6..79aab8e 100644 --- a/docs/studio/CAVEATS.md +++ b/docs/studio/CAVEATS.md @@ -8,6 +8,15 @@ ones that arise from *configuring and comparing* runs. --- +## The box has no temperature feedback, by decision + +Studio refuses `switches.heating_to_t` (schema 0.2.0). The model can enable a radiative heating term, +but that term is **shortwave-only** — there is no longwave cooling in the radiative calculation +(AD-5.4) — so switching it on produces a one-sided ~+1.2 K / 10 d warm drift that is an artefact of +the missing cooling, not a physical response. Every run is therefore **isothermal at the configured +temperature**, and a result must not be read as containing a plume-warming signal. Decision by Ali, +2026-08-13; revisit when longwave cooling lands (SCIENCE-4, issue #56). + ## Top-level caveats — shown on every results view ### The definition of t = 0 is unresolved, and it dominates particle number diff --git a/docs/studio/OPEN_QUESTIONS.md b/docs/studio/OPEN_QUESTIONS.md index 55db52b..e1d59a4 100644 --- a/docs/studio/OPEN_QUESTIONS.md +++ b/docs/studio/OPEN_QUESTIONS.md @@ -169,7 +169,7 @@ material is **not** represented. --- -### SCIENCE-4 — Box thermodynamics · **OPEN** · blocks Phase 5 · [#56](https://github.com/reflective-org/SANDBOX/issues/56) +### SCIENCE-4 — Box thermodynamics · **PARTLY ANSWERED** (no temperature feedback) · blocks Phase 5 · [#56](https://github.com/reflective-org/SANDBOX/issues/56) *Is the box isobaric? isothermal? does it rise buoyantly? do particles sediment out?* Absent from the original brief. Current behaviour, from the code: @@ -179,8 +179,16 @@ Absent from the original brief. Current behaviour, from the code: AD-5.4), producing a one-sided ≈ +1.2 K / 10 d warm drift. Every science script leaves it off. - **No buoyant rise.** No sedimentation. -Each must become a schema field with a documented default, and the SW-only asymmetry must warn in -the UI when the switch is enabled rather than silently producing a drifting temperature. +**Partly answered (Ali, 2026-08-13): no temperature feedback.** Longwave radiation is not in the +radiative calculation, so the heating term cannot represent the box's energy balance — enabling it +does not make the thermodynamics more complete, it makes them one-sided, and the ~+1.2 K / 10 d +drift is an artefact of the missing cooling rather than a result. `switches.heating_to_t` is +therefore `Literal[False]` in the schema from version 0.2.0: `True` fails validation rather than +being defaulted off, so it cannot be enabled by a form, a YAML file or a sweep axis without the +schema changing first. Revisit when longwave cooling lands. + +The rest of SCIENCE-4 stands: buoyant rise, sedimentation and the isobaric assumption are still +undecided, and each must become a schema field with a documented default. --- diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 43bde2d..50ebf99 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -29,6 +29,35 @@ derivations to resolve rather than fixtures. --- +### 2026-08-13 — Schema 0.2.0: the temperature feedback is refused, not defaulted off + +Ali's decision, and the reason is worth stating precisely: **longwave radiation is not in the +radiative calculation**, so the model's heating term cannot represent the box's energy balance. +Enabling it does not make the thermodynamics more complete — it makes them *one-sided*, and the +resulting ~+1.2 K / 10 d warm drift is an artefact of the missing cooling rather than a physical +response. + +`switches.heating_to_t` is therefore `Literal[False]`, the same treatment `dilution.background_evolves` +already had: `True` **fails validation** rather than being defaulted off, so it cannot be enabled by a +form, a YAML file, or a sweep axis without the schema changing first. Two tests cover it — the direct +one and the axis path, which is the one that would slip past a UI-level guard. + +**`SCHEMA_VERSION` 0.1.0 → 0.2.0, and the pinned hash moved with it** (…46cbe3 → …373ab4). Note the +*value* of `heating_to_t` did not change — it was already `False` — but `schema_version` is part of +the hashed payload, which is exactly what makes "old configs are never silently reinterpreted under +new semantics" true rather than merely stated. A config written yesterday no longer hashes to a +0.2.0 identity, which is the intended behaviour. + +Recorded where a reader would actually look: `CAVEATS.md` gains a top-level entry (every run is +isothermal at the configured temperature, and a result must not be read as containing a +plume-warming signal), and SCIENCE-4 is marked **partly answered** — buoyant rise, sedimentation and +the isobaric assumption remain open. + +Two tests that set `heating_to_t=True` as an innocuous example were updated to use +`switches.aerosol_to_j` instead. They were not weakened; the value they used simply became illegal. + +--- + ### 2026-08-13 — Task 0.6: the runner and the job lifecycle (issue #72) `studio/runner/` (`base.py`, `local.py`), `studio/modelio/execute.py`, `studio/cli/run.py`. 20 new diff --git a/studio/schema/config.py b/studio/schema/config.py index 957f0ce..ffb7c0e 100644 --- a/studio/schema/config.py +++ b/studio/schema/config.py @@ -43,7 +43,7 @@ #: The schema's own version. Bumped on any change to field names, semantics or defaults, because #: those change the config hash and therefore run identity (ADR-006). Old configs are never silently #: reinterpreted under new semantics. -SCHEMA_VERSION = "0.1.0" +SCHEMA_VERSION = "0.2.0" #: Stratospheric background gas composition [pptv] used by the 810-run ensemble #: (``run_ensemble.py:56-57``). Module-level so the default is one object with one source, and so a @@ -111,8 +111,9 @@ class Site(SchemaModel): gt=0.0, label="Temperature", description=( - "Box temperature. Isobaric and isothermal unless the heating switch is on; see the " - "caveat on switches.heating_to_t and SCIENCE-4 (issue #56)." + "Box temperature. The box is isobaric and ISOTHERMAL: the temperature feedback is " + "refused while the radiative calculation has no longwave component, so this value " + "holds for the whole run. See switches.heating_to_t and SCIENCE-4 (issue #56)." ), provenance=Provenance.PAPER_ENSEMBLE, source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Site: 210 K, 55 hPa)", @@ -601,17 +602,25 @@ class ProcessSwitches(SchemaModel): "validation status)." ), ) - heating_to_t: bool = SciField( + heating_to_t: Literal[False] = SciField( default=False, unit=Unit.DIMENSIONLESS, label="Radiative heating -> T", - description="Let radiative heating change the box temperature.", + description=( + "Let radiative heating change the box temperature. FALSE IS THE ONLY ACCEPTED VALUE: " + "the radiative calculation has no longwave component, so there is no temperature " + "feedback to enable. True fails validation rather than being quietly ignored." + ), provenance=Provenance.PAPER_ENSEMBLE, source="coupled/paper_ensemble/run_ensemble.py:106 (heating_to_t=False)", caveat=( - "The heating term is SHORTWAVE-ONLY -- no longwave cooling (AD-5.4) -- so switching it " - "on gives a one-sided ~+1.2 K / 10 d warm drift, not an energy balance. Every science " - "script leaves it off. The UI must warn on enable rather than silently drifting." + "Decision (Ali, 2026-08-13): no temperature feedback while longwave radiation is " + "absent from the radiative calculation. The model's heating term is SHORTWAVE-ONLY " + "(AD-5.4), so enabling it does not make the box thermodynamics more complete -- it " + "makes them one-sided, giving a ~+1.2 K / 10 d warm drift that is an artefact of the " + "missing cooling rather than a physical result. The MODEL still defaults this on and " + "every science script turns it off; Studio refuses it outright. Revisit when longwave " + "cooling lands (SCIENCE-4, issue #56)." ), ) dilution: bool = SciField( @@ -681,7 +690,7 @@ class RunConfig(SchemaModel): physics, and two runs whose only difference is a name are the same computation. """ - schema_version: Literal["0.1.0"] = SciField( + schema_version: Literal["0.2.0"] = SciField( default=SCHEMA_VERSION, unit=Unit.DIMENSIONLESS, label="Schema version", diff --git a/studio/tests/unit/test_config_hash.py b/studio/tests/unit/test_config_hash.py index 9a3bb57..1d14f2b 100644 --- a/studio/tests/unit/test_config_hash.py +++ b/studio/tests/unit/test_config_hash.py @@ -27,8 +27,11 @@ from studio.schema.hashing import CANONICAL_FORM_VERSION #: SHA-256 of the canonical JSON of ``RunConfig()`` -- the paper ensemble's golden case, which is -#: also the schema's default configuration. Tied to SCHEMA_VERSION 0.1.0 and canonical form 1. -GOLDEN_DEFAULT_HASH = "629fc801779ca43a4a7ae43d74c43f3221b213d78e36c1dfce1e94f17d46cbe3" +#: also the schema's default configuration. Tied to SCHEMA_VERSION 0.2.0 and canonical form 1. +#: Moved from ...46cbe3 when 0.2.0 refused the temperature feedback: the VALUE of heating_to_t did +#: not change (False either way), but schema_version is part of the hashed payload, which is what +#: makes "old configs are never silently reinterpreted under new semantics" true rather than stated. +GOLDEN_DEFAULT_HASH = "e9d207b91d74d45076433cdd25b0f3b59f365dbefd4f8c1b47afe1fdc2373ab4" @pytest.mark.tier_a diff --git a/studio/tests/unit/test_resolve_resolver.py b/studio/tests/unit/test_resolve_resolver.py index f0cb080..5346d4b 100644 --- a/studio/tests/unit/test_resolve_resolver.py +++ b/studio/tests/unit/test_resolve_resolver.py @@ -76,7 +76,7 @@ def test_chained_derivations_resolve_in_order() -> None: ("injection.so2_mass_kg", 2000.0, {SO2_PPTV}), ("microphysics.n_bins", 40, set()), ("chemistry.so2_ho2_rate", 1e-16, set()), - ("switches.heating_to_t", True, set()), + ("switches.aerosol_to_j", True, set()), # heating_to_t cannot be True (schema 0.2.0) ], ) def test_an_edit_changes_exactly_the_downstream_closure( diff --git a/studio/tests/unit/test_schema_export.py b/studio/tests/unit/test_schema_export.py index fa6fa68..a79f20d 100644 --- a/studio/tests/unit/test_schema_export.py +++ b/studio/tests/unit/test_schema_export.py @@ -105,7 +105,7 @@ def test_round_trip_survives_a_non_default_config() -> None: "microphysics": {"n_bins": 160, "condensation_alpha": 0.5, "ion_pair_rate": 0.0}, "chemistry": {"photolysis": "sza", "so2_ho2_rate": 1e-16}, "numerics": {"output_dt_s": 1200.0, "couple_dt_s": 300.0}, - "switches": {"heating_to_t": True}, + "switches": {"aerosol_to_j": True}, # heating_to_t is refused (schema 0.2.0) "termination": {"max_wall_time_s": 60.0, "max_sim_time_days": 5.0}, } ) @@ -141,6 +141,32 @@ def test_background_evolves_accepts_only_false() -> None: RunConfig.model_validate(payload) +@pytest.mark.tier_a +def test_the_temperature_feedback_cannot_be_enabled() -> None: + """Decision (Ali, 2026-08-13): no temperature feedback while longwave radiation is missing. + + The model's heating term is shortwave-only, so enabling it does not make the thermodynamics + more complete -- it makes them one-sided, and the resulting ~+1.2 K / 10 d drift is an artefact + of the absent cooling. Refused outright rather than defaulted off, so it cannot be turned on by + a form, a YAML file or a sweep axis without the schema changing first. + """ + payload = RunConfig().model_dump() + payload["switches"]["heating_to_t"] = True + with pytest.raises(ValueError, match="heating_to_t"): + RunConfig.model_validate(payload) + assert RunConfig().switches.heating_to_t is False + + +@pytest.mark.tier_a +def test_a_sweep_cannot_enable_the_temperature_feedback_either() -> None: + """The axis path is the one that would slip past a UI-level guard.""" + from studio.schema import Axis, RunSet + + runset = RunSet(axes=(Axis.over("heating", "switches.heating_to_t", {"on": True}),)) + with pytest.raises(ValueError, match="heating_to_t"): + runset.expand() + + @pytest.mark.tier_a def test_bin_count_is_restricted_to_the_grids_the_model_has() -> None: """40/80/160 are the only TOMAS grids; 100 must fail here, not inside tomas_bridge.""" From a479dc8ff0eddb7ee67a8247ec2b8bef003a5d82 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Thu, 13 Aug 2026 19:55:40 -0700 Subject: [PATCH 08/18] studio: measure the archived-ensemble reproduction tolerance (#74) * test(studio): measure the archived-ensemble reproduction tolerance (#70) ADR-009's first golden-harness task is a measurement, not an assertion. Two archived cases re-run at today's submodule SHAs and compared per quantity against the archived state.npz: the golden case (index 121, 30N_20km__sabr220__D2med__a1p0__nuc1__cg1) and a contrast on two axes (index 67, 30N_20km__sabr330__burst__a1p0__nuc1__cg1 -- burst dilution, loaded background). Result: reproduction is CLOSE BUT NOT BIT-FOR-BIT. Every headline quantity agrees to <= 2.1e-12 and the worst deviation anywhere is 3.4e-12, but only ~31% of gas state-vector elements and ~1% of aerosol samples are exactly equal -- an atol=0 golden test would have failed on arrival. Two controls fix the interpretation: running the same case twice today is bit-identical across all 18 stored arrays (so the residual is toolchain drift, not run-to-run noise), and the worst deviations scatter across days 1.3-9.8 rather than accumulating. Also recorded: unguarded relative error over the raw gas state vector peaks at 4.2e+04, entirely on night-time O1D/O at O(1e-35) molec/cm3 oscillating about zero. Golden tests must floor by series magnitude. No assertions written in this pass, by design. Tolerances proposed for the Tier-B assertions, each with its rationale, in REFERENCE_TOLERANCES.md. ASSUMPTION-2 marked settled. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * docs(studio): record the measured non-uniform time grid in the tolerance file The archived t spans 0-864000 s in 1461 samples with 41 distinct step sizes (0.56-600 s, mean 591.78 s), confirming the CAVEATS warning against reconstructing time as i x DT rather than taking it on trust. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 --- docs/studio/ASSUMPTIONS.md | 15 +- docs/studio/PROGRESS.md | 33 ++- studio/tests/golden/REFERENCE_TOLERANCES.md | 241 ++++++++++++++++++++ studio/tests/golden/measure_deviation.py | 208 +++++++++++++++++ 4 files changed, 494 insertions(+), 3 deletions(-) create mode 100644 studio/tests/golden/REFERENCE_TOLERANCES.md create mode 100644 studio/tests/golden/measure_deviation.py diff --git a/docs/studio/ASSUMPTIONS.md b/docs/studio/ASSUMPTIONS.md index d7f6540..31a0229 100644 --- a/docs/studio/ASSUMPTIONS.md +++ b/docs/studio/ASSUMPTIONS.md @@ -36,9 +36,17 @@ plus a conversion layer becomes the better trade. ## ASSUMPTION-2 — The archived `state.npz` files are the golden reference, at a tolerance yet to be measured -**Made:** 2026-08-13 · **Affects:** `studio/tests/golden/` · +**Made:** 2026-08-13 · **Settled:** 2026-08-13 (#70) · **Affects:** `studio/tests/golden/` · **Recorded in:** [ADR-009](adr/ADR-009-golden-file-strategy.md) +> **Settled.** The measurement exists: +> [`studio/tests/golden/REFERENCE_TOLERANCES.md`](../../studio/tests/golden/REFERENCE_TOLERANCES.md). +> Reproduction is **close but not bit-for-bit** — every headline quantity within 2.1e-12, worst +> deviation anywhere 3.4e-12, but only ~31 % of gas state-vector elements bit-identical. The archive +> is usable as a golden reference at ~1e-12 (endpoints) / ~1e-10 (series and per-bin size +> distribution); exact equality is not. The paragraphs below stand as the reasoning that made the +> measurement necessary; the *consequence* below still holds and cannot be retrofitted. + Golden fixtures are derived from the existing `coupled/paper_ensemble/runs*/` outputs. Whether re-running those cases **today** reproduces them bit-for-bit is *unverified*: the submodule commits at which they were produced were never recorded (there is no provenance record for the existing @@ -52,7 +60,10 @@ the measurement together with the SHAs it was taken at. **Consequence.** Golden fixtures record the SHA at which the reference was *measured*, not the SHA at which the data was originally produced. This is an honest limitation and cannot be retrofitted. -**What would settle it.** The measurement itself, in `studio/tests/golden/REFERENCE_TOLERANCES.md`. +**What settled it.** The measurement itself, in `studio/tests/golden/REFERENCE_TOLERANCES.md` +(2026-08-13, issue #70). It also produced a result nobody had asked for: the model is bit-for-bit +deterministic run-to-run *today*, so the residual is drift between the archive's toolchain and this +one — which is what makes a 1e-12 tolerance defensible rather than arbitrary. --- diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 50ebf99..4bfafc7 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -20,7 +20,7 @@ with full provenance, and the golden tests pass. | 0.4 `studio/modelio` seam + `RunSummary` | **done** (#68) | | 0.5 `studio/science` derivations | **done** (#64) | | 0.6 `studio/runner` + job lifecycle | **done** (#72) | -| 0.7 Golden-file harness (two tiers) | not started | +| 0.7 Golden-file harness (two tiers) | **in progress** — reference deviation measured (#70); assertions not yet written | | 0.8 Four contained fixes in `coupled/` | not started | | 0.9 Vertical slice: CLI + API + minimal UI | not started | @@ -107,6 +107,37 @@ mock. The fixture module's behaviour arrives by environment variable, set before a directive file written *after* submission races the subprocess start — the standard way process tests become flaky. +### 2026-08-13 — Task 0.7 (first half): the reproduction tolerance, measured (issue #70) + +ASSUMPTION-2 is settled. Two archived cases re-run at today's SHAs and compared per quantity against +the archived `state.npz`: the golden case `30N_20km__sabr220__D2med__a1p0__nuc1__cg1` (index 121) and +a deliberate contrast, `30N_20km__sabr330__burst__a1p0__nuc1__cg1` (index 67) — `burst` dilution and +the loaded background, the regime where a regime-dependent residual would show. Full record with the +SHAs, the environment and the per-quantity table: +[`studio/tests/golden/REFERENCE_TOLERANCES.md`](../../studio/tests/golden/REFERENCE_TOLERANCES.md). + +**Reproduction is close but not bit-for-bit.** Every headline quantity agrees to **≤ 2.1e-12**, the +worst deviation anywhere in either run is **3.4e-12**, and the time axis, `V_ratio`, `T` and the dry +bin edges are bit-identical. But only ~31 % of gas state-vector elements and ~1 % of aerosol samples +reproduce exactly, so `atol=0` would have failed on arrival — exactly the outcome ADR-009 was written +to catch. + +Two controls make the reading firm rather than hopeful: running the same case twice **today** is +bit-identical across all 18 stored arrays (so the residual is environment drift, not run-to-run +noise), and the worst deviations are scattered across days 1.3–9.8 rather than accumulating (the +signature of round-off, not of a diverging integration). The `bd289e9` day-12 solver change is +consistent with being invisible here: a 10-day run never reaches t = 2²⁰ s, so the retry branch is +never taken. + +**The trap worth knowing before writing the assertions:** unguarded relative error over the raw gas +state vector peaks at **4.2e+04**, entirely on night-time `O1D`/`O` at O(1e-35) molec cm⁻³ — values +that oscillate about zero, including negative, on a species whose peak is ~3 molec cm⁻³. Golden tests +must floor by series magnitude or they will fail by four orders of magnitude over an absolute +difference of 1e-34. + +No assertions were written in this pass, by design. Wall clock: ~4.6 min per 10-day / 80-bin case, +matching BLOCKING-4. + --- ### 2026-08-13 — Task 0.4: the model seam and `RunSummary` (issue #68) diff --git a/studio/tests/golden/REFERENCE_TOLERANCES.md b/studio/tests/golden/REFERENCE_TOLERANCES.md new file mode 100644 index 0000000..ca5fb26 --- /dev/null +++ b/studio/tests/golden/REFERENCE_TOLERANCES.md @@ -0,0 +1,241 @@ +# Reference tolerances — measured, not assumed + +**Measured:** 2026-08-13 · **Task:** Phase 0 / 0.7 (issue #70) · **Decides:** +[ASSUMPTION-2](../../../docs/studio/ASSUMPTIONS.md#assumption-2--the-archived-statenpz-files-are-the-golden-reference-at-a-tolerance-yet-to-be-measured) +· **Per:** [ADR-009](../../../docs/studio/adr/ADR-009-golden-file-strategy.md) + +This file records **what the deviation is**, not what anyone hoped it would be. No assertions were +written in this pass; Tier-B tolerances are chosen from these numbers and must cite this file. + +--- + +## Verdict + +**Reproduction is close, but it is not bit-for-bit.** + +Every physically meaningful quantity in both cases agrees with the archive to **≤ 3.4e-12**, and every +headline endpoint (final SO₂, peak/final H₂SO₄, peak/final particle number, peak/final wet surface +area, particulate sulfur, final size distribution) to **≤ 2.1e-12**. That is float64 round-off +territory — 1–4 decimal digits above machine epsilon after 1461 coupled intervals — not a physics +change. The archived ensemble **can** serve as a golden reference. + +It cannot serve as a *bit-exact* one: only ~31 % of the gas state-vector elements and ~1 % of the +aerosol samples reproduce exactly. An `array_equal` / `atol=0` golden test would fail on arrival. + +Two supporting measurements make the interpretation firm rather than hopeful: + +1. **The model is bit-for-bit deterministic today.** Case 121 was run twice in this environment; all + 18 stored arrays compared `array_equal == True`. So the ~1e-12 residual is *drift between the + archive's environment and today's*, not run-to-run noise. Re-running a case at fixed SHAs is + reproducible to the last bit, which is what makes a tight tolerance defensible. +2. **The deviation does not grow with time or with regime.** The worst per-quantity deviations land + at day 1.3, 3.2, 5.7, 8.4, 8.8, 9.3, 9.8 — scattered, not accumulating — and the `burst` / + `sabr330` case is if anything *quieter* than the `D2med` / `sabr220` one. That is the signature of + round-off, not of a diverging integration. + +--- + +## Provenance of this measurement + +| | | +| --- | --- | +| SANDBOX commit | `8c12422721debf9c8f5bba7bfe485df9c0e9bce7` (branch `feat/70-golden-tolerances`, off `studio/dev`) | +| `stratchem-jax` | `19fec0fafc35a5cae2a184ae5b1849236e2ca315` (`heads/main`) | +| `tomas-jax` | `39535ea021f9fe189dcfece5f3eb1167538d4503` (`remotes/origin/feat/marianna-dilution`) | +| `tuvx-jax` | `06f6777a73703fa607d3c87556f436c129f60cdf` (`heads/main`) | +| Platform | macOS 26.2, arm64, CPU only (`jax.devices() == [CpuDevice(id=0)]`) | +| Python / JAX / jaxlib | 3.12.12 / 0.11.0 / 0.11.0 | +| diffrax / numpy / scipy | 0.7.2 / 2.5.2 / 1.18.0 | +| Environment | `studio/requirements.lock`, `uv pip install -e . --no-deps` | +| Thread pinning | `OMP/OPENBLAS/MKL/VECLIB/NUMEXPR_NUM_THREADS=1`, `XLA_FLAGS=--xla_cpu_multi_thread_eigen=false` (`coupled/paper_ensemble/launch_parallel.py:26-30`) | + +Note the submodule SHAs above are **where the reference was measured**, not where the archive was +produced — the latter was never recorded (ADR-006), and this gap cannot be retrofitted. + +### What the archive is, and what changed since + +The two archived `state.npz` files carry mtimes of **2026-07-04 03:53** (case 121) and +**2026-07-04 06:07** (case 67). Changes landing after that date and reachable from today's SHAs: + +- `coupled/driver.py` — `bd289e9` (2026-07-08), the day-12.139 fix: SciPy-BDF fallback removed, + fail-fast probe budget, sticky `first_step=1e-2` retry. Its commit message claims the change + "lives on the failure branch only, so all existing 10-day ensemble results are unchanged". + **This measurement is consistent with that claim** — a 10-day run never reaches t = 2²⁰ s + (day 12.14), so the retry branch is never taken, and the observed deviation is round-off, not a + solver-path change. +- `stratchem-jax` `19fec0f` and `tuvx-jax` `06f6777` (both 2026-07-15) — standalone packaging only. +- `tomas-jax` `39535ea` (2026-07-04) — wires `coag_kernel_scale` into `coag_euler_step`. This one + may straddle the archive run itself. **Both cases here use `cg1` (scale = 1.0)**, for which the + wiring is a no-op, so it cannot be the source of the residual. A `cg0p5` or `cg2` case is *not* + covered by this measurement and should be checked separately before being used as a golden case. + +The residual therefore most plausibly comes from the toolchain (JAX/XLA/LLVM codegen: fused +multiply-add and reduction-order choices differ across versions), not from repository code. + +--- + +## Cases run + +| # | index | case_id | why | +| --- | --- | --- | --- | +| 1 | 121 | `30N_20km__sabr220__D2med__a1p0__nuc1__cg1` | the designated golden case: mid dilution, clean `sabr220` background | +| 2 | 67 | `30N_20km__sabr330__burst__a1p0__nuc1__cg1` | **contrast on two axes** — `burst` dilution (the fastest early transient in the ensemble, the regime ADR-009 flags for 8× aliasing) and the loaded `sabr330` background instead of the clean one. If the residual were regime-dependent, this is where it would show. | + +Command (from the SANDBOX root, thread-pinned as above): + +```bash +python -m coupled.paper_ensemble.run_ensemble one 121 +python -m coupled.paper_ensemble.run_ensemble one 67 +``` + +Both produced `steps = 1461`, a time axis **bit-identical** to the archive (`t` and `V_ratio` both +100 % equal), an identical species list and identical dry diameter bins — so nothing below is +confounded by a shifted grid. + +The CAVEATS note on the time axis is confirmed rather than assumed: the archived `t` spans +0 → 864000 s in 1461 samples with **41 distinct step sizes** ranging 0.56 s to 600 s and a **mean of +591.78 s**, not the nominal 600. Reconstructing time as `i × DT` would misplace day 10 by ~0.5 % +here and far more on longer runs. Every "at day" below comes from the stored `t`. + +### Wall clock + +| run | seconds | conditions | +| --- | --- | --- | +| case 121 | 287.3 | two cases concurrently, 1 thread each | +| case 67 | 275.2 | two cases concurrently, 1 thread each | +| case 121, repeat | 272.3 | alone | + +~4.6 min per 10-day / 80-bin case. Consistent with ADR-009's 3–5 min budget; Tier B of 4–6 cases is +~20–30 min serial, less in parallel. + +--- + +## Measured deviations + +Relative deviation is `|fresh − archived| / |archived|`, evaluated only where the archived series +exceeds **1e-6 × its own peak** (see "The one trap"). "max over t" is over all 1461 samples; +"at day" locates it using the **stored `t`**, never `i × DT`. + +Produced by: + +```bash +python studio/tests/golden/measure_deviation.py \ + --archive /Users/ali/Documents/GitHub/gas-phase-chemistry/SANDBOX/coupled/paper_ensemble/runs +``` + +### Case 121 — `30N_20km__sabr220__D2med__a1p0__nuc1__cg1` + +| quantity | endpoint / extremum | max over t | at day | +| --- | --- | --- | --- | +| SO₂, final [pptv] | 2.01e-14 | 4.54e-14 | 7.24 | +| H₂SO₄, peak [pptv] | 2.97e-14 | 3.38e-12 | 1.34 | +| H₂SO₄, final [pptv] | 1.19e-14 | 3.38e-12 | 1.34 | +| total N, peak [cm⁻³] | 1.80e-15 | 3.85e-13 | 1.34 | +| total N, final [cm⁻³] | 1.00e-14 | 3.85e-13 | 1.34 | +| wet SA, peak [µm² cm⁻³] | 1.73e-15 | 2.73e-14 | 8.40 | +| wet SA, final [µm² cm⁻³] | 9.01e-15 | 2.73e-14 | 8.40 | +| particulate S, peak | 5.52e-15 | 2.25e-14 | 8.80 | +| particulate S, final | 1.74e-14 | 2.25e-14 | 8.80 | +| wet radius, final [cm] | 1.05e-14 | 1.76e-14 | 1.34 | +| H₂SO₄ wt %, final | 2.16e-16 | 6.49e-16 | 8.55 | +| **final size dist** `n_cm3`, per bin | 2.04e-12 | — | 10.00 | +| **final size dist** `dNdlogDp`, per bin | 2.04e-12 | — | 10.00 | +| photolysis `J`, all 23 reactions × all steps | 1.09e-13 | — | — | + +Final size distribution: worst bin at **Dp_dry = 0.4611 µm**, 50 of 80 bins above the floor, +L2-relative deviation of the whole final profile **1.76e-13**. Integrated `sum(n_cm3)` final agrees +to 1.00e-14. + +Worst gas species (floored): `Cl2` 1.14e-11 @ day 0.354, then `H2SO4` 3.38e-12 @ day 1.340, +`Cl` 1.85e-12, `NO2` 1.67e-12, `NO` 9.20e-13. + +Bit-identical fraction: `t` 100 %, `V_ratio` 100 %, `T` 100 %, `J` 58.5 %, `h2so4wp` 48.7 %, +`x` 31.7 %, `radius_cm` 1.9 %, `SA` 1.6 %, `particulate_S` 1.4 %, `total_n` 1.1 %, +`dNdlogDp` 0.85 %, `n_cm3` 0.84 %. + +### Case 67 — `30N_20km__sabr330__burst__a1p0__nuc1__cg1` + +| quantity | endpoint / extremum | max over t | at day | +| --- | --- | --- | --- | +| SO₂, final [pptv] | 2.01e-14 | 3.16e-14 | 3.24 | +| H₂SO₄, peak [pptv] | 0.00e+00 | 4.44e-13 | 5.74 | +| H₂SO₄, final [pptv] | 2.74e-14 | 4.44e-13 | 5.74 | +| total N, peak [cm⁻³] | 7.43e-16 | 1.05e-13 | 9.30 | +| total N, final [cm⁻³] | 9.35e-15 | 1.05e-13 | 9.30 | +| wet SA, peak [µm² cm⁻³] | 3.39e-15 | 2.44e-14 | 9.44 | +| wet SA, final [µm² cm⁻³] | 2.15e-14 | 2.44e-14 | 9.44 | +| particulate S, peak | 5.02e-16 | 2.17e-14 | 9.84 | +| particulate S, final | 2.07e-14 | 2.17e-14 | 9.84 | +| wet radius, final [cm] | 1.92e-15 | 1.36e-14 | 8.22 | +| H₂SO₄ wt %, final | 2.16e-16 | 8.68e-16 | 1.36 | +| **final size dist** `n_cm3`, per bin | 1.91e-12 | — | 10.00 | +| **final size dist** `dNdlogDp`, per bin | 1.91e-12 | — | 10.00 | +| photolysis `J`, all 23 reactions × all steps | 1.09e-13 | — | — | + +Final size distribution: worst bin at **Dp_dry = 0.1027 µm**, 54 of 80 bins above the floor, +L2-relative deviation **1.53e-13**. Integrated `sum(n_cm3)` final agrees to 9.35e-15. + +Worst gas species (floored): `Cl2` 5.56e-12 @ day 0.347, then `Cl` 7.00e-13, `NO` 6.11e-13, +`H2SO4` 4.44e-13, `ClO` 1.79e-13. + +Bit-identical fraction: `x` 31.5 %, `particulate_S` 2.5 %, `SA` 0.9 %, `total_n` 0.75 %, +`n_cm3` 0.73 %. + +### Determinism control (same environment, two runs of case 121) + +All 18 stored arrays `array_equal == True`. Wall clock 287.3 s vs 272.3 s. The gas/microphysics path +carries no seed and no nondeterministic reduction on this platform. + +--- + +## The one trap: unguarded relative error explodes on near-zero species + +Without the 1e-6 floor, the worst relative deviation across the gas state vector is **4.24e+04** +(case 121) / **3.81e+03** (case 67). Both are `O1D`: + +``` +case 121, day 2.167: archived -4.850e-39 fresh 2.057e-34 rel 4.242e+04 +case 67, day 0.104: archived -3.698e-36 fresh -1.411e-32 rel 3.814e+03 +``` + +`O1D` peaks at ~3.1 molec cm⁻³ and collapses to O(1e-35) — *and to small negative values* — at night. +`O` behaves the same way (`min_nonzero` 2.2e-299). These are solver residuals oscillating about zero, +not physics: the absolute difference is ~1e-34 molec cm⁻³ against a species peak of 3 molec cm⁻³. + +**Consequence for the assertions written next:** a golden test that computes relative error over the +raw `x` array without a magnitude floor will fail by four orders of magnitude for a difference of +1e-34 molec cm⁻³. Either floor at a fraction of the series peak (as here) or assert per species on an +absolute floor. Do not "fix" this by widening a global tolerance to 1e5 — that would make the test +meaningless for every real species. + +--- + +## Recommended tolerances for the Tier-B assertions (not yet written) + +Proposed, each with the rationale ADR-009 requires. All are ~50–100× the measured deviation, which +leaves headroom for a toolchain bump without admitting a physics change (the smallest physically +interesting change in any of these quantities is ≫ 1e-6 relative). + +| assertion target | proposed `rtol` | rationale | +| --- | --- | --- | +| headline endpoints — final SO₂, peak/final H₂SO₄, peak/final N, peak/final wet SA, particulate S | `1e-12` | measured ≤ 2.9e-14 across both cases; 1e-12 is ~35× headroom and still ~6 orders below any meaningful physics change | +| any stored series, max over time | `1e-10` | measured worst 3.4e-12 (`H2SO4`, case 121, day 1.34); 1e-10 covers the `Cl2` trace-species worst case of 1.1e-11 with ~10× margin | +| final size distribution, per bin, bins above 1e-6 × peak | `1e-10` | measured 2.0e-12 worst bin; per-bin conditioning is worse than integrated N, so it gets its own (looser) number | +| photolysis `J` | `1e-11` | measured 1.09e-13, identical in both cases — TUV-x is the most reproducible part of the pipeline | +| time axis `t`, `V_ratio`, `T`, dry bin edges `dp_mid_um` | **exact** | measured bit-identical in both cases; these are analytic, not integrated, so any drift is a real bug | +| near-zero species (`O1D`, `O`, and any series below 1e-6 × peak) | **excluded** | see "The one trap"; assert an absolute floor instead if coverage is wanted | + +These numbers describe **this environment**. A JAX/jaxlib bump is the most likely thing to move them, +and the correct response is to re-run this measurement and update this file — not to widen a +tolerance in a test file. + +## Not covered by this measurement + +- Only `cg1` was measured; `cg0p5` / `cg2` cases may be affected by `tomas-jax` `39535ea` landing + around the archive date. Verify before adopting one as a golden case. +- Only `30N_20km` and only 10-day / 80-bin runs. The 60-day runs cross the day-12.14 boundary where + the sticky-retry branch *is* taken, and are expected to differ from any pre-`bd289e9` archive. +- Only the two dilution regimes above; `D1low`, `D3high`, `D5vhigh` and the `cesm` background are + unmeasured. +- Only this machine, single-threaded. Deviations on CI's Linux runners are unmeasured, and Tier B + does not run there. diff --git a/studio/tests/golden/measure_deviation.py b/studio/tests/golden/measure_deviation.py new file mode 100644 index 0000000..b6b1f84 --- /dev/null +++ b/studio/tests/golden/measure_deviation.py @@ -0,0 +1,208 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Measure the per-quantity deviation between a freshly-run case and the archived ``state.npz``. + +**This is a measurement tool, not a test.** It contains no assertions and no tolerances of its own; +it prints what the deviation *is*. The numbers it produced are recorded in +[`REFERENCE_TOLERANCES.md`](REFERENCE_TOLERANCES.md) together with the SHAs they were taken at +(ADR-009: measure before asserting). Tier-B assertions, when written, cite that file. + +Usage (from the SANDBOX root, with the studio venv): + + python -m coupled.paper_ensemble.run_ensemble one 121 # produce the fresh run + python studio/tests/golden/measure_deviation.py 30N_20km__sabr220__D2med__a1p0__nuc1__cg1 \\ + --archive [--fresh ] + +``--archive`` must point at a **read-only** copy of the archived ensemble: those outputs are not +regenerable at their original provenance (ADR-006) and this script never writes to that tree. + +Relative-error floor +-------------------- +Relative error is evaluated only where the archived series exceeds ``FLOOR_FRAC`` times its own +peak. Without a floor the metric is dominated by night-time O(1e-35) O1D/O values that oscillate +about zero — solver noise on a species whose peak is ~3 molec/cm3. See REFERENCE_TOLERANCES.md, +"The one trap". +""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +import numpy as np + +#: Samples below this fraction of the archived series peak are excluded from the relative metric. +FLOOR_FRAC = 1e-6 + + +def _rel_series(fresh: np.ndarray, arch: np.ndarray, t: np.ndarray) -> tuple[float, float]: + """Max relative deviation over the time axis, and the day at which it occurs.""" + peak = np.nanmax(np.abs(arch)) + mask = np.abs(arch) > FLOOR_FRAC * peak + rel = np.full(arch.shape, np.nan) + rel[mask] = np.abs(fresh[mask] - arch[mask]) / np.abs(arch[mask]) + i = int(np.nanargmax(rel)) + return float(np.nanmax(rel)), float(t[i] / 86400.0) + + +def _rel_scalar(fresh: float, arch: float) -> float: + if arch == 0.0: + return 0.0 if fresh == 0.0 else float("nan") + return abs(fresh - arch) / abs(arch) + + +def compare(arch_npz: Path, fresh_npz: Path) -> None: + a = np.load(arch_npz, allow_pickle=True) + f = np.load(fresh_npz, allow_pickle=True) + + if a["t"].shape != f["t"].shape: + raise ValueError( + f"time-grid length differs: archived {a['t'].shape} vs fresh {f['t'].shape}" + ) + t = a["t"] + print(f"time axis identical: {np.array_equal(a['t'], f['t'])}") + print(f"species list identical: {list(a['species']) == list(f['species'])}") + + print("\n-- bit-for-bit per stored array --") + for k in a.files: + A, F = a[k], f[k] + eq = A.shape == F.shape and np.array_equal(A, F) + if eq or A.dtype.kind in "US": + print(f" {k}: equal={eq}") + else: + frac = 100.0 * (A == F).sum() / A.size + print( + f" {k}: equal=False bit-identical {frac:.2f}% " + f"max|abs diff|={np.nanmax(np.abs(F.astype(float) - A.astype(float))):.3e}" + ) + + M = float(a["M"]) + ix = {s: i for i, s in enumerate(list(a["species"]))} + so2_a, so2_f = a["x"][:, ix["SO2"]] / M * 1e12, f["x"][:, ix["SO2"]] / M * 1e12 + h_a, h_f = a["x"][:, ix["H2SO4"]] / M * 1e12, f["x"][:, ix["H2SO4"]] / M * 1e12 + + rows: list[tuple[str, float, float, float]] = [] + + def add(name: str, fv: float, av: float, series: tuple[np.ndarray, np.ndarray]) -> None: + r, day = _rel_series(series[0], series[1], t) + rows.append((name, _rel_scalar(fv, av), r, day)) + + add("SO2, final [pptv]", float(so2_f[-1]), float(so2_a[-1]), (so2_f, so2_a)) + add("H2SO4, peak [pptv]", float(h_f.max()), float(h_a.max()), (h_f, h_a)) + add("H2SO4, final [pptv]", float(h_f[-1]), float(h_a[-1]), (h_f, h_a)) + add( + "total N, peak [cm-3]", + float(f["total_n"].max()), + float(a["total_n"].max()), + (f["total_n"], a["total_n"]), + ) + add( + "total N, final [cm-3]", + float(f["total_n"][-1]), + float(a["total_n"][-1]), + (f["total_n"], a["total_n"]), + ) + add( + "wet SA, peak [um2 cm-3]", + float(np.nanmax(f["SA"])), + float(np.nanmax(a["SA"])), + (f["SA"], a["SA"]), + ) + add("wet SA, final [um2 cm-3]", float(f["SA"][-1]), float(a["SA"][-1]), (f["SA"], a["SA"])) + add( + "particulate S, peak", + float(np.nanmax(f["particulate_S"])), + float(np.nanmax(a["particulate_S"])), + (f["particulate_S"], a["particulate_S"]), + ) + add( + "particulate S, final", + float(f["particulate_S"][-1]), + float(a["particulate_S"][-1]), + (f["particulate_S"], a["particulate_S"]), + ) + add( + "wet radius, final [cm]", + float(f["radius_cm"][-1]), + float(a["radius_cm"][-1]), + (f["radius_cm"], a["radius_cm"]), + ) + add( + "H2SO4 wt%, final", + float(f["h2so4wp"][-1]), + float(a["h2so4wp"][-1]), + (f["h2so4wp"], a["h2so4wp"]), + ) + + print("\n-- per-quantity relative deviation --") + print(f"{'quantity':30s} {'endpoint':>10s} {'max over t':>12s} {'at day':>8s}") + for nm, sc, se, day in rows: + print(f"{nm:30s} {sc:10.2e} {se:12.2e} {day:8.2f}") + + print("\n-- final size distribution (dry bins) --") + for key in ("n_cm3", "dNdlogDp"): + av, fv = a[key][-1], f[key][-1] + mask = np.abs(av) > FLOOR_FRAC * np.abs(av).max() + rel = np.abs(fv[mask] - av[mask]) / np.abs(av[mask]) + j = int(np.argmax(rel)) + bins = np.where(mask)[0] + print( + f" {key}: max per-bin rel dev {rel.max():.2e} at Dp_dry=" + f"{a['dp_mid_um'][bins[j]]:.4g} um ({mask.sum()}/{len(av)} bins above floor); " + f"L2 rel {np.linalg.norm(fv - av) / np.linalg.norm(av):.2e}" + ) + + print("\n-- photolysis J --") + Ja, Jf = a["J"], f["J"] + mask = np.abs(Ja) > FLOOR_FRAC * np.abs(Ja).max() + rel = np.abs(Jf[mask] - Ja[mask]) / np.abs(Ja[mask]) + print(f" max rel dev {rel.max():.2e} over {mask.sum()} entries above floor") + + print("\n-- gas species, worst 5 (floored) --") + worst = [] + for s, j in ix.items(): + av, fv = a["x"][:, j], f["x"][:, j] + if np.abs(av).max() == 0.0: + continue + r, day = _rel_series(fv, av, t) + worst.append((r, s, day, float(np.abs(av).max()))) + worst.sort(reverse=True) + for r, s, day, pk in worst[:5]: + print(f" {s:8s} {r:10.2e} at day {day:6.3f} (series peak {pk:.3e} molec/cm3)") + + print("\n-- unguarded worst over the whole gas state vector (shows why the floor exists) --") + xa, xf = a["x"], f["x"] + nz = np.abs(xa) > 0 + ru = np.zeros_like(xa) + ru[nz] = np.abs(xf[nz] - xa[nz]) / np.abs(xa[nz]) + i, j = np.unravel_index(int(ru.argmax()), ru.shape) + sp = list(a["species"])[j] + print( + f" {ru.max():.3e} in {sp} at day {t[i] / 86400.0:.3f}: " + f"archived {xa[i, j]:.3e} vs fresh {xf[i, j]:.3e} " + f"(series peak {np.abs(xa[:, j]).max():.3e} molec/cm3)" + ) + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("case_id") + p.add_argument( + "--archive", + required=True, + type=Path, + help="read-only root holding /state.npz for the archived ensemble", + ) + p.add_argument( + "--fresh", + type=Path, + default=Path(__file__).resolve().parents[3] / "coupled/paper_ensemble/runs", + help="root holding the freshly-produced /state.npz", + ) + args = p.parse_args() + print(f"=== {args.case_id} ===") + compare(args.archive / args.case_id / "state.npz", args.fresh / args.case_id / "state.npz") + + +if __name__ == "__main__": + main() From 3e07874669162daab65e6976b09868ec0c1edbaa Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 10:07:16 -0700 Subject: [PATCH 09/18] =?UTF-8?q?studio:=20correct=20the=20dp=5Fmid=5Fum?= =?UTF-8?q?=20tolerance=20row=20=E2=80=94=20not=20exact=20for=20a=20Studio?= =?UTF-8?q?=20run=20(#76)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit studio: correct the dp_mid_um tolerance row -- not exact for a Studio run #74 proposed asserting dp_mid_um exact. That holds only when the fresh run comes from run_ensemble. A run produced by Studio differs in 44 of 80 bins by up to 8.1e-16, because task 0.5 deliberately adopted sqrt(a*b) where run_ensemble writes 10**(0.5*(log10 a + log10 b)) -- the two spellings 0.5 proved algebraically identical and measured a few ULP apart. Asserting exact equality would therefore pass against the old pipeline and fail against every run Studio itself produces, presenting as a physics regression over a rounding difference. Corrected to 1e-15; t, V_ratio and T stay exact. dNdlogDp inherits the difference at 3.2e-13, inside its own 1e-10, so no other row moved. Found by re-running the golden case through studio.cli.run as an independent check of the measurement. Every other number reproduced to the digit; both runs are now tabulated in the record, which also makes the point that a tolerance measured through one pipeline is not automatically a tolerance for another. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 19 +++++++++++++++++ studio/tests/golden/REFERENCE_TOLERANCES.md | 23 ++++++++++++++++++++- 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 4bfafc7..c85a9e1 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -29,6 +29,25 @@ derivations to resolve rather than fixtures. --- +### 2026-08-14 — Correction: `dp_mid_um` is not bit-identical for a Studio-produced run + +The tolerance record merged in #74 proposed asserting `dp_mid_um` **exact**. That holds only when the +fresh run comes from `run_ensemble`. A run produced by Studio differs in **44 of 80 bins by up to +8.1e-16**, because task 0.5 deliberately adopted `sqrt(a*b)` where `run_ensemble` writes +`10**(0.5*(log10 a + log10 b))` — the two spellings 0.5 proved algebraically identical. + +Asserting exact equality there would have passed against the old pipeline and failed against every +run Studio itself produces, presenting as a physics regression over a rounding difference. Corrected +to `1e-15`; `t`, `V_ratio` and `T` stay exact. `dNdlogDp` inherits the difference at 3.2e-13, inside +its own `1e-10`, so no other row moved. + +Found by re-running the golden case through `studio.cli.run` as an independent check — every other +number reproduced to the digit, and both runs are now tabulated in the record. Narrow lesson worth +keeping: **a reproduction tolerance measured through one pipeline is not automatically a tolerance +for another**, even when the two are meant to agree. + +--- + ### 2026-08-13 — Schema 0.2.0: the temperature feedback is refused, not defaulted off Ali's decision, and the reason is worth stating precisely: **longwave radiation is not in the diff --git a/studio/tests/golden/REFERENCE_TOLERANCES.md b/studio/tests/golden/REFERENCE_TOLERANCES.md index ca5fb26..61fb553 100644 --- a/studio/tests/golden/REFERENCE_TOLERANCES.md +++ b/studio/tests/golden/REFERENCE_TOLERANCES.md @@ -222,13 +222,34 @@ interesting change in any of these quantities is ≫ 1e-6 relative). | any stored series, max over time | `1e-10` | measured worst 3.4e-12 (`H2SO4`, case 121, day 1.34); 1e-10 covers the `Cl2` trace-species worst case of 1.1e-11 with ~10× margin | | final size distribution, per bin, bins above 1e-6 × peak | `1e-10` | measured 2.0e-12 worst bin; per-bin conditioning is worse than integrated N, so it gets its own (looser) number | | photolysis `J` | `1e-11` | measured 1.09e-13, identical in both cases — TUV-x is the most reproducible part of the pipeline | -| time axis `t`, `V_ratio`, `T`, dry bin edges `dp_mid_um` | **exact** | measured bit-identical in both cases; these are analytic, not integrated, so any drift is a real bug | +| time axis `t`, `V_ratio`, `T` | **exact** | measured bit-identical in both cases; these are analytic, not integrated, so any drift is a real bug | +| dry bin edges `dp_mid_um` | `1e-15` | **not exact — corrected 2026-08-14.** Bit-identical only when the fresh run is produced by `run_ensemble`. A run produced by Studio differs in 44 of 80 bins by up to **8.1e-16**, because task 0.5 adopted `sqrt(a*b)` where `run_ensemble` writes `10**(0.5*(log10 a + log10 b))` — algebraically identical, differently rounded. Asserting `exact` here would pass against the old pipeline and fail against every Studio run, looking like a physics regression over a spelling difference. | | near-zero species (`O1D`, `O`, and any series below 1e-6 × peak) | **excluded** | see "The one trap"; assert an absolute floor instead if coverage is wanted | These numbers describe **this environment**. A JAX/jaxlib bump is the most likely thing to move them, and the correct response is to re-run this measurement and update this file — not to widen a tolerance in a test file. +## Independently reproduced (2026-08-14) + +The golden case was re-run a second time by a different route — through `python -m studio.cli.run` +(the task-0.6 entry point) rather than `run_ensemble` — and compared against the archive again: + +| quantity | first measurement | independent re-run | +| --- | --- | --- | +| SO₂ final | 2.01e-14 | 2.01e-14 | +| H₂SO₄ final / peak | 1.19e-14 / 2.97e-14 | 1.19e-14 / 2.98e-14 | +| total N final / peak | 1.00e-14 / 1.80e-15 | 1.00e-14 / 1.80e-15 | +| wet SA final / peak | 9.01e-15 / 1.73e-15 | 9.01e-15 / 1.73e-15 | +| particulate S final / peak | 1.74e-14 / 5.52e-15 | 1.74e-14 / 5.52e-15 | +| final size dist, worst bin | 2.04e-12 | 2.04e-12 | +| gas elements exactly equal | ~31 % | 31.7 % | + +`t` and `V_ratio` bit-identical, as first measured. **`dp_mid_um` was not**, which is what produced +the correction in the table above — and it could only surface via the Studio pipeline, which did not +exist on the branch where the first measurement was taken. `dNdlogDp` inherits that difference at +3.2e-13, comfortably inside its own `1e-10`, so only the `dp_mid_um` row needed changing. + ## Not covered by this measurement - Only `cg1` was measured; `cg0p5` / `cg2` cases may be affected by `tomas-jax` `39535ea` landing From d2156532bee084e009a82e8fff61e71f415ad3ac Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 10:27:41 -0700 Subject: [PATCH 10/18] studio: re-land the heating/buoyancy scope decision dropped by #75's squash (#77) studio: heating and buoyancy are out of scope, not pending Ali's follow-up decision: ignore buoyancy and anything heating-related, because this model cannot answer those questions -- simulating the feedback of heating needs a different model. That is a scope boundary rather than an undecided item, and the two are recorded differently on purpose. Longwave radiation is absent from the radiative calculation, so the heating term is shortwave-only and enabling it makes the thermodynamics one-sided rather than more complete. Buoyant rise follows from a heating rate this model cannot compute, so a rise velocity would be a free parameter dressed as physics. Answering either needs a model with longwave radiation and plume dynamics. The interface consequence: no buoyancy or heating-rate fields are added to the schema at all. A field for a capability the model does not have would advertise it, and the absence is the honest interface (ADR-005). SCIENCE-4 is answered for heating and buoyancy, and `numerics.box_thermodynamics.*` moves in the capability register from "see SCIENCE-4" to "not exposed, by decision". Sedimentation is deliberately left open. It is a particle-loss process, not a thermodynamic response; "we decided not to model heating" is not an argument about gravitational settling, and folding it into this decision would have quietly closed a question nobody answered. CAVEATS.md now says every run is isobaric and isothermal at the configured temperature, and that a result must not be read as containing a plume-warming signal, a lofting signal, or an altitude change. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/CAVEATS.md | 27 ++++++++++++++++++-------- docs/studio/OPEN_QUESTIONS.md | 36 ++++++++++++++++++++++++----------- docs/studio/PROGRESS.md | 23 +++++++++++++++++----- studio/schema/config.py | 15 ++++++++------- 4 files changed, 70 insertions(+), 31 deletions(-) diff --git a/docs/studio/CAVEATS.md b/docs/studio/CAVEATS.md index 79aab8e..2b9a28d 100644 --- a/docs/studio/CAVEATS.md +++ b/docs/studio/CAVEATS.md @@ -8,14 +8,25 @@ ones that arise from *configuring and comparing* runs. --- -## The box has no temperature feedback, by decision - -Studio refuses `switches.heating_to_t` (schema 0.2.0). The model can enable a radiative heating term, -but that term is **shortwave-only** — there is no longwave cooling in the radiative calculation -(AD-5.4) — so switching it on produces a one-sided ~+1.2 K / 10 d warm drift that is an artefact of -the missing cooling, not a physical response. Every run is therefore **isothermal at the configured -temperature**, and a result must not be read as containing a plume-warming signal. Decision by Ali, -2026-08-13; revisit when longwave cooling lands (SCIENCE-4, issue #56). +## The box does not heat and does not rise — and this model cannot answer whether it should + +**Every run is isobaric and isothermal at the configured temperature.** There is no radiative +heating response and no buoyant rise. A result must not be read as containing a plume-warming +signal, a lofting signal, or an altitude change. + +This is a **scope boundary, not a pending feature** (Ali, 2026-08-13). The model's heating term is +shortwave-only — longwave cooling is absent from the radiative calculation (AD-5.4) — so enabling it +would not make the thermodynamics more complete, it would make them one-sided, producing a +~+1.2 K / 10 d drift that is an artefact of the missing cooling. Buoyant rise follows from a heating +rate this model cannot compute, so a rise velocity would be a free parameter dressed as physics. + +**Answering either question requires a different model**, with longwave radiation and plume +dynamics. Studio therefore refuses `switches.heating_to_t` outright (schema 0.2.0, `True` fails +validation) and exposes **no** buoyancy or heating-rate fields at all — a knob for a capability the +model does not have would advertise it. + +Sedimentation is a separate question and remains genuinely open (SCIENCE-4, issue #56); it is a +particle-loss process, not a thermodynamic response, and this decision says nothing about it. ## Top-level caveats — shown on every results view diff --git a/docs/studio/OPEN_QUESTIONS.md b/docs/studio/OPEN_QUESTIONS.md index e1d59a4..7b66aff 100644 --- a/docs/studio/OPEN_QUESTIONS.md +++ b/docs/studio/OPEN_QUESTIONS.md @@ -169,7 +169,7 @@ material is **not** represented. --- -### SCIENCE-4 — Box thermodynamics · **PARTLY ANSWERED** (no temperature feedback) · blocks Phase 5 · [#56](https://github.com/reflective-org/SANDBOX/issues/56) +### SCIENCE-4 — Box thermodynamics · **ANSWERED for heating and buoyancy** (2026-08-13); sedimentation open · [#56](https://github.com/reflective-org/SANDBOX/issues/56) *Is the box isobaric? isothermal? does it rise buoyantly? do particles sediment out?* Absent from the original brief. Current behaviour, from the code: @@ -179,16 +179,30 @@ Absent from the original brief. Current behaviour, from the code: AD-5.4), producing a one-sided ≈ +1.2 K / 10 d warm drift. Every science script leaves it off. - **No buoyant rise.** No sedimentation. -**Partly answered (Ali, 2026-08-13): no temperature feedback.** Longwave radiation is not in the -radiative calculation, so the heating term cannot represent the box's energy balance — enabling it -does not make the thermodynamics more complete, it makes them one-sided, and the ~+1.2 K / 10 d -drift is an artefact of the missing cooling rather than a result. `switches.heating_to_t` is -therefore `Literal[False]` in the schema from version 0.2.0: `True` fails validation rather than -being defaulted off, so it cannot be enabled by a form, a YAML file or a sweep axis without the -schema changing first. Revisit when longwave cooling lands. +**Answered (Ali, 2026-08-13): heating and buoyancy are out of scope for this model.** -The rest of SCIENCE-4 stands: buoyant rise, sedimentation and the isobaric assumption are still -undecided, and each must become a schema field with a documented default. +Not "undecided" — **out of scope**, which is a different status and is why this row is closed rather +than left open. Longwave radiation is not in the radiative calculation, so the heating term cannot +represent the box's energy balance: enabling it does not make the thermodynamics more complete, it +makes them one-sided, and the ~+1.2 K / 10 d drift is an artefact of the missing cooling rather than +a result. Buoyant rise follows the same logic — a parcel rises in response to a heating rate this +model cannot compute, so a rise velocity here would be a free parameter dressed as physics. + +**Answering either question needs a different model**, one with longwave radiation and plume +dynamics. It is not a gap to be filled in by a later Studio phase, and Studio must not present a +knob implying otherwise: + +- `switches.heating_to_t` is `Literal[False]` from schema 0.2.0 — `True` fails validation rather + than being defaulted off, so it cannot be enabled by a form, a YAML file or a sweep axis. +- **No buoyancy or heating-rate fields are added to the schema at all.** A field for a capability + the model does not have would advertise it; the absence is the honest interface (ADR-005). +- Every run is therefore **isobaric and isothermal at the configured temperature**, and results + carry that as a top-level caveat rather than a footnote. + +**Still open: sedimentation.** It is untouched by this decision — a particle-loss process, not a +thermodynamic response — and the model does not have it. It is deliberately left in this register +rather than swept in with the rest, because "we decided not to model heating" is not an argument +about gravitational settling. --- @@ -245,7 +259,7 @@ issue at that point rather than sitting in a backlog now. | `chemistry.rate_overrides[]` (general) | Only `so2_ho2_rate` is a knob (`coupled_scenario.py:130`). Arbitrary per-reaction overrides do not exist. | | `chemistry.photolysis.tuvx_settings.{o3_column, albedo, aod}` | Not exposed. Only mode + lat/lon/doy/hour reach TUV-x (`model_bridge.py:46`). | | `numerics.bin_scheme.{d_min, d_max, mass_doubling}` | Fixed by the TOMAS grid; only `tomas_nbins ∈ {40, 80, 160}` is selectable (`tomas_bridge.py:146`). Ratio = `2**(40/nbins)`; the top boundary is pinned. | -| `numerics.box_thermodynamics.*` | See SCIENCE-4. | +| `numerics.box_thermodynamics.*` | **Not exposed, by decision.** Heating and buoyancy are out of scope (SCIENCE-4): the model cannot compute them and a field would imply it can. Isobaric + isothermal is the only behaviour. | | `dilution.entrainment.{entrains_background_gases, entrains_background_aerosol}` | Entrainment is unconditional when `switches.dilution` is on. Separate flags are new code. | | `dilution.background_evolves` | See SCIENCE-5. Only `false` is accepted. | | `background.aerosol` custom lognormal modes | Six named modes + tabulated `redcircles` only — but `_seed_lognormal` (`tomas_bridge.py:110`) already accepts arbitrary `(N, Dg, σg)` tuples, so this is a small, worthwhile early addition. | diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index c85a9e1..89773a3 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -48,7 +48,7 @@ for another**, even when the two are meant to agree. --- -### 2026-08-13 — Schema 0.2.0: the temperature feedback is refused, not defaulted off +### 2026-08-13 — Schema 0.2.0: heating and buoyancy are out of scope, not pending Ali's decision, and the reason is worth stating precisely: **longwave radiation is not in the radiative calculation**, so the model's heating term cannot represent the box's energy balance. @@ -67,10 +67,23 @@ the hashed payload, which is exactly what makes "old configs are never silently new semantics" true rather than merely stated. A config written yesterday no longer hashes to a 0.2.0 identity, which is the intended behaviour. -Recorded where a reader would actually look: `CAVEATS.md` gains a top-level entry (every run is -isothermal at the configured temperature, and a result must not be read as containing a -plume-warming signal), and SCIENCE-4 is marked **partly answered** — buoyant rise, sedimentation and -the isobaric assumption remain open. +**Buoyancy is closed for the same reason** (Ali, same day): a parcel rises in response to a heating +rate this model cannot compute, so a rise velocity here would be a free parameter dressed as physics. +Answering either question needs a **different model**, with longwave radiation and plume dynamics — +so this is a scope boundary, not a gap for a later phase to fill. Consequently **no buoyancy or +heating-rate fields are added to the schema at all**: a field for a capability the model lacks would +advertise it, and the absence is the honest interface. + +SCIENCE-4 is therefore **answered for heating and buoyancy**. `numerics.box_thermodynamics.*` moves +in the capability register from "see SCIENCE-4" to "not exposed, by decision". + +**Sedimentation stays open, deliberately.** It is a particle-loss process, not a thermodynamic +response; "we decided not to model heating" is not an argument about gravitational settling, and +sweeping it into this decision would have quietly closed a question nobody answered. + +`CAVEATS.md` says it where a reader of results would look: every run is isobaric and isothermal at +the configured temperature, and a result must not be read as containing a plume-warming signal, a +lofting signal, or an altitude change. Two tests that set `heating_to_t=True` as an innocuous example were updated to use `switches.aerosol_to_j` instead. They were not weakened; the value they used simply became illegal. diff --git a/studio/schema/config.py b/studio/schema/config.py index ffb7c0e..7cdefb4 100644 --- a/studio/schema/config.py +++ b/studio/schema/config.py @@ -614,13 +614,14 @@ class ProcessSwitches(SchemaModel): provenance=Provenance.PAPER_ENSEMBLE, source="coupled/paper_ensemble/run_ensemble.py:106 (heating_to_t=False)", caveat=( - "Decision (Ali, 2026-08-13): no temperature feedback while longwave radiation is " - "absent from the radiative calculation. The model's heating term is SHORTWAVE-ONLY " - "(AD-5.4), so enabling it does not make the box thermodynamics more complete -- it " - "makes them one-sided, giving a ~+1.2 K / 10 d warm drift that is an artefact of the " - "missing cooling rather than a physical result. The MODEL still defaults this on and " - "every science script turns it off; Studio refuses it outright. Revisit when longwave " - "cooling lands (SCIENCE-4, issue #56)." + "Decision (Ali, 2026-08-13): heating and buoyancy are OUT OF SCOPE for this model, " + "not pending features. Longwave radiation is absent from the radiative calculation " + "(AD-5.4), so the heating term is shortwave-only: enabling it does not make the box " + "thermodynamics more complete, it makes them one-sided, giving a ~+1.2 K / 10 d drift " + "that is an artefact of the missing cooling. Answering whether the box should heat, or " + "rise, needs a DIFFERENT model with longwave radiation and plume dynamics. The MODEL " + "still defaults this on and every science script turns it off; Studio refuses it. " + "See SCIENCE-4 (issue #56)." ), ) dilution: bool = SciField( From 22f442897e5a5e2b7c5f92f148cfbfa2dac128b8 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 10:36:10 -0700 Subject: [PATCH 11/18] chore(studio): merge main into studio/dev; stop condition takes the diagnostics dict (#78) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(coupled): four contained fixes — JAX-free scenario validation, diagnostics-dict stop_condition, user lognormal backgrounds, drop output_dir (#73) * refactor(coupled): hoist BACKGROUND_MODES into a JAX-free module CoupledScenario.__post_init__ validated background_dist via a function-body `from coupled.tomas_bridge import BACKGROUND_MODES`. tomas_bridge imports jax (and sets jax_enable_x64) at module scope, so merely CONSTRUCTING a scenario cost ~1.0 s and left jax resident. An API that validates a form on every keystroke cannot pay that. The mode tables move verbatim to coupled/backgrounds.py -- pure data, no numpy/jax/scipy, no coupled sibling that imports them. tomas_bridge re-exports BACKGROUND_MODES / AMBIENT_BACKGROUNDS so paper scripts and docs referring to tomas_bridge.BACKGROUND_MODES keep working; it remains where the tables are USED. coupled_scenario imports them at module scope instead. Measured with a fresh interpreter, repo root as cwd: python -c "import time,sys; t0=time.perf_counter(); from coupled.coupled_scenario import CoupledScenario; t1=time.perf_counter(); CoupledScenario(); t2=time.perf_counter(); print(t1-t0, t2-t1, 'jax' in sys.modules)" before: import 0.007 s, first construct 0.99-1.19 s, jax loaded True after: import 0.010 s, first construct 0.000 s, jax loaded False The proof is a subprocess test: an in-process `"jax" not in sys.modules` assertion is vacuous, because the rest of the suite has already imported the driver by the time any single test runs. The import in coupled_scenario is absolute (`from coupled.backgrounds import`) rather than relative because that module is also imported FLAT as `coupled_scenario`, with coupled/ on sys.path, by coupled/conftest.py. Co-Authored-By: Claude Opus 5 * feat(coupled): stop_condition receives a diagnostics dict The early-stop callback was called with only (t1_seconds, wet_SA), so a termination criterion could not reference SO2 or particle number (docs/studio/OPEN_QUESTIONS.md, termination.criteria[]). It now receives one dict, evaluated on the end-of-interval state: t, interval, T, SA, radius_cm, h2so4wp, particulate_S, N_total, gas `gas` is {species_name: molec/cm^3} for all 34 species, keyed by NAME so a state-vector reordering cannot silently misread it. The aerosol entries are the WET quantities the heterogeneous chemistry sees and are NaN -- never 0 -- when TOMAS is inactive, so a `SA < x` criterion cannot read "no aerosol model" as "the plume has relaxed". BOTH shapes are supported, dispatched on the callback's DECLARED arity (inspect.signature) once at run_coupled entry rather than by attempting a call and catching TypeError: one positional parameter -> the dict form, two -> the legacy (t1, wet_SA) form with a DeprecationWarning. Zero, three-plus, or a bare *args raises TypeError, because *args is compatible with both shapes and guessing would pass a dict where a float was expected. Resolving at entry means a mis-shaped callback fails immediately instead of minutes into a run. The legacy shape is kept because coupled/paper_ensemble/README.md documents it and out-of-tree run scripts use it. The two in-repo callers (run_60day.py, run_bgstop.py) are migrated to the dict form; the criterion is unchanged (same diag["SA"] / diag["t"] values as the arguments they were passed before). Co-Authored-By: Claude Opus 5 * feat(coupled): accept user-defined lognormal background modes _seed_lognormal already took arbitrary (N, Dg, sigma_g) tuples, but background_dist was restricted to the six named mode sets plus "redcircles". It now also accepts a mode list, seeded through the identical code path -- a test asserts a user list equal to a named set's own modes yields a BIT-IDENTICAL initial state. Validation lives in coupled/backgrounds.normalize_modes (still JAX-free, so scenario validation stays cheap): N > 0, Dg > 0, sigma_g > 1, exactly three numeric entries per mode, at least one mode. sigma_g == 1 is rejected specifically because log10(sigma_g) = 0 divides by zero inside the dN/dlogDp evaluation; every one of these would otherwise seed a silently degenerate background rather than raise. Lists (as YAML/JSON return them) canonicalize to a tuple of float 3-tuples, so a scenario round-trips. New field `background_modes_basis` ("stp" | "ambient"), REQUIRED for a mode list and REJECTED for a name. It is not defaulted: a bare mode list carries no number basis, the named sets carry theirs via AMBIENT_BACKGROUNDS, and the STP->ambient factor is ~0.09 at 68 mbar / 210 K -- so silently picking one is an order-of-magnitude error in the background number concentration, not a rounding difference. A test pins that factor to the two branches. Co-Authored-By: Claude Opus 5 * refactor(coupled)!: remove the dead CoupledScenario.output_dir field The driver never read it. `run_coupled` returns arrays and writes nothing -- every caller (paper_ensemble/run_*.py, run_dilution_d1_clean.py) computes its own path and calls np.savez itself. REMOVED rather than honoured, deliberately. Honouring it would give the driver a filesystem side effect it does not have today, and would create two competing sources of truth for where a run's results live: the field, and the path the caller passes to np.savez. Studio's runner (task 0.6) owns run directories and must be the only such source. A config field that nothing reads is worse than no field, because it looks authoritative. Loading an archived config that still carries the key raises with an explanation naming the removal and what to do instead, rather than the generic "Unknown CoupledScenario keys" -- that config was valid when it was written. The archived paper runs are unaffected: they store only state.npz, no config. coupled/scenarios/coupled_default.yaml drops the key. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 * chore(studio): merge main into studio/dev; stop condition takes the diagnostics dict studio/dev now has #73's four model fixes. Studio's suite (196 Tier-A tests, equivalence tests included) passes against the changed model unmodified -- output_dir is gone and nothing missed it, which is the evidence that removing it was safe rather than tidy. Conflict resolutions, stated because the reasoning matters more than the diff: test_import_boundaries.py takes studio/dev's structure (three clean packages including studio.resolve, and the seam check that allows submodules of studio.modelio) with main's wording, since the __post_init__ pattern it describes is now past tense -- #73 fixed it. studio/__init__.py merged cleanly and the merge exposed a stale docstring of mine, which still claimed two clean packages after 0.3 added a third; fixed here, where it became visible. The stop condition moves to the diagnostics dict. #73 deprecated the two-argument form, so leaving it would have Studio emitting a DeprecationWarning on a normal path. Only diag["t"] is read today, but the dict also carries SA, N_total and every gas species by name, which is what makes the spec's SO2- or number-based termination criteria possible. max_sim_time enforcement is now proven end to end rather than only unit-tested: a 1-day request with max_sim_time_days = 0.5, run through studio.cli.run with -W error::DeprecationWarning, stops at t = 0.500 d and is labelled TERMINATED_ON_LIMIT with the stopped_on_limit flag. That path had never run before -- nothing set max_sim_time -- and it also confirms 0.6's termination inference, so partial output cannot be read as converged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 --- coupled/backgrounds.py | 102 +++++++++++ coupled/coupled_scenario.py | 64 +++++-- coupled/driver.py | 109 +++++++++++- coupled/paper_ensemble/FIGURES.md | 2 +- coupled/paper_ensemble/README.md | 19 +- coupled/paper_ensemble/run_60day.py | 3 +- coupled/paper_ensemble/run_bgstop.py | 3 +- coupled/scenarios/coupled_default.yaml | 4 +- coupled/tests/test_coupled_scenario.py | 106 ++++++++++++ coupled/tests/test_stop_condition.py | 182 ++++++++++++++++++++ coupled/tests/test_tomas_bridge.py | 46 +++++ coupled/tomas_bridge.py | 58 +++---- docs/studio/OPEN_QUESTIONS.md | 6 +- docs/studio/PROGRESS.md | 46 ++++- studio/__init__.py | 8 +- studio/modelio/execute.py | 29 ++-- studio/tests/unit/test_import_boundaries.py | 13 +- studio/tests/unit/test_runner.py | 19 +- 18 files changed, 724 insertions(+), 95 deletions(-) create mode 100644 coupled/backgrounds.py create mode 100644 coupled/tests/test_stop_condition.py diff --git a/coupled/backgrounds.py b/coupled/backgrounds.py new file mode 100644 index 0000000..96d8e7e --- /dev/null +++ b/coupled/backgrounds.py @@ -0,0 +1,102 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Background aerosol size distributions -- the *data*, with NO model imports. + +Split out of ``tomas_bridge`` so that validating a scenario does not cost a JAX import. +``CoupledScenario.__post_init__`` has to check ``background_dist``, and importing ``tomas_bridge`` +to do it pulled in ``jax`` (which also sets ``jax_enable_x64``) -- ~1 s on the first +``CoupledScenario(...)`` in a process. A form-validating API cannot pay that per keystroke. + +**This module must stay free of numpy/jax/scipy and of any ``coupled`` sibling that imports them.** +It is pure data + pure-Python validation; ``tomas_bridge`` re-exports the tables (so existing +``tomas_bridge.BACKGROUND_MODES`` references keep working) and does the actual seeding. + +A scenario's ``background_dist`` is either + +* a NAME -- :data:`TABULATED_BACKGROUND` or a key of :data:`BACKGROUND_MODES`. Each named mode set + carries its own number basis: it is ambient iff its name is in :data:`AMBIENT_BACKGROUNDS`. +* a USER-SUPPLIED list of lognormal modes ``[(N, Dg, sigma_g), ...]``, validated by + :func:`normalize_modes`. These carry no basis of their own, so the scenario must state it + (``background_modes_basis``, one of :data:`MODE_BASES`). That is required, not defaulted: at + 68 mbar / 210 K the STP->ambient factor is ~0.09, so picking the wrong basis is an + order-of-magnitude error in N, not a rounding difference. +""" + +from __future__ import annotations + +#: The tabulated (non-lognormal) background: Marianna's digitized "red circles" distribution, +#: loaded by ``background_aerosol_distribution.get_initial_state`` in ``tomas_bridge``. +TABULATED_BACKGROUND = "redcircles" + +#: Allowed ``CoupledScenario.background_modes_basis`` values for a user-supplied mode list. +#: "stp" -- N is a standard-temperature-and-pressure number density, converted to ambient on +#: seeding (as the digitized SABR/CESM sets are); "ambient" -- N is already at the box T, P. +MODE_BASES = ("stp", "ambient") + +# --- background aerosol size distributions as (multi-)lognormal modes, DIAMETER basis. Each entry is +# a list of (N [cm^-3, STP], Dg [um], sigma_g). DIGITIZED (approximate) from the SABR / CESM plots the +# user provided; see coupled/analyses/paper_ensemble/DECISIONS.md for the source figures and the +# overlay-verification. "redcircles" (Marianna, tabulated loader) stays the default and is NOT here. --- +# N chosen so each mode's PEAK dN/dlogDp = N/(sqrt(2pi)*log10(sigma_g)) matches the value read off the +# source plot (the most reliable digitized feature): SABR 330->~1000, 220->~95; CESM Aitken->~50, +# Accumulation->~12, Coarse->~0.5 cm^-3 STP. +BACKGROUND_MODES = { + "sabr_330": [(810.0, 0.045, 2.1)], # young air (high N2O), peak ~1000 + "sabr_310": [(205.0, 0.060, 1.8)], # mid air (310-320 ppbv), peak ~320 + "sabr_220": [(49.0, 0.12, 1.6)], # aged air (low N2O), peak ~95 + "cesm_g6": [(22.0, 0.040, 1.5), (5.3, 0.20, 1.5), (0.18, 0.90, 1.4)], # CESM G6 SAI (r->D x2) + # AER 2D geoengineered stratosphere (Pierce et al. fig. 2, gray curve: 5 Mt-S/yr, 95 nm case). + # Dg = 0.30 um (mode radius 0.15 um) and sigma_g = 1.7 fitted to the curve; N = 120 cm^-3 per + # user spec (paper caption quotes 50 cm^-3). Values are AMBIENT -> no STP conversion on seeding. + "aer_geo": [(120.0, 0.30, 1.7)], + # CESM G6 with the source plot read as AMBIENT (user-confirmed): same modes as cesm_g6 but + # seeded without the STP->ambient factor. cesm_g6 is kept unchanged so the original 810-run + # ensemble stays reproducible. + "cesm_g6_amb": [(22.0, 0.040, 1.5), (5.3, 0.20, 1.5), (0.18, 0.90, 1.4)], +} +# mode sets specified at AMBIENT conditions (seeding skips the STP->ambient factor) +AMBIENT_BACKGROUNDS = {"aer_geo", "cesm_g6_amb"} + + +def normalize_modes(modes) -> tuple[tuple[float, float, float], ...]: + """Validate a user-supplied lognormal mode list; return it as a tuple of ``(N, Dg, sigma_g)``. + + Units follow :data:`BACKGROUND_MODES`: ``N`` [cm^-3], ``Dg`` [um] on a DIAMETER basis, + ``sigma_g`` dimensionless. + + ``N`` and ``Dg`` must be > 0 and ``sigma_g`` > 1. A non-positive N or Dg is not a distribution + at all; ``sigma_g <= 1`` collapses the lognormal to a delta function, and at exactly 1 + ``log10(sigma_g) == 0`` divides by zero inside the dN/dlogDp evaluation. Seeding any of those + gives a silently degenerate background rather than an error, which is the whole point of + validating here. + + Returned as a tuple of tuples so a mode list round-trips to a stable canonical form through + YAML/JSON (which hand back lists). + """ + if isinstance(modes, (str, bytes)) or not isinstance(modes, (list, tuple)): + raise ValueError( + f"background_dist must be a name ({TABULATED_BACKGROUND!r} or one of " + f"{sorted(BACKGROUND_MODES)}) or a list of (N, Dg, sigma_g) lognormal modes, " + f"got {type(modes).__name__}") + if len(modes) == 0: + raise ValueError("background_dist mode list is empty; give at least one " + "(N [cm^-3], Dg [um], sigma_g) mode") + out: list[tuple[float, float, float]] = [] + for i, mode in enumerate(modes): + if isinstance(mode, (str, bytes)) or not isinstance(mode, (list, tuple)) or len(mode) != 3: + raise ValueError(f"background_dist mode {i} must be a 3-tuple " + f"(N [cm^-3], Dg [um], sigma_g), got {mode!r}") + try: + N, Dg, sigma_g = (float(v) for v in mode) + except (TypeError, ValueError) as exc: + raise ValueError(f"background_dist mode {i} has non-numeric entries: {mode!r}") from exc + if not N > 0.0: + raise ValueError(f"background_dist mode {i}: N must be > 0 cm^-3, got {N}") + if not Dg > 0.0: + raise ValueError(f"background_dist mode {i}: Dg must be > 0 um, got {Dg}") + if not sigma_g > 1.0: + raise ValueError(f"background_dist mode {i}: sigma_g must be > 1 (sigma_g <= 1 is a " + f"degenerate lognormal; == 1 divides by log10(sigma_g) = 0), " + f"got {sigma_g}") + out.append((N, Dg, sigma_g)) + return tuple(out) diff --git a/coupled/coupled_scenario.py b/coupled/coupled_scenario.py index 96e26f8..5f0ce74 100644 --- a/coupled/coupled_scenario.py +++ b/coupled/coupled_scenario.py @@ -7,6 +7,9 @@ photolysis mode, the initial gas composition, and per-process **switches**. Later phases (TOMAS microphysics, aerosol->photolysis radiation, radiative heating, dilution) read their switch here. +It describes the PHYSICS, not the bookkeeping: there is no output path, because ``run_coupled`` +returns arrays and writes nothing -- the caller owns where results land. + Phase 2 scaffolding: only the gas chemistry + photolysis are wired. Switches for not-yet-implemented processes must stay OFF (enabling one raises, so a config can never silently claim a capability the model doesn't have yet -- the same "no silent assumptions" guard as the photolysis-mode validation). @@ -18,6 +21,13 @@ import os from dataclasses import asdict, dataclass, field +# Deliberately the JAX-free table module, NOT ``coupled.tomas_bridge``: this dataclass is constructed +# by form validation and must stay cheap to import and to build (see coupled/backgrounds.py). +# Absolute (not relative) because this module is also imported FLAT as ``coupled_scenario`` with +# coupled/ on sys.path -- see coupled/conftest.py -- where a relative import has no package to resolve. +from coupled.backgrounds import (BACKGROUND_MODES, MODE_BASES, TABULATED_BACKGROUND, + normalize_modes) + #: Photolysis drivers understood by the model (validated -> no silent mis-gate of the sulfur chain). PHOTOLYSIS_MODES = ("reference", "sza", "tuvx") @@ -26,6 +36,14 @@ _IMPLEMENTED_SWITCHES = frozenset( {"sulfur", "nucleation", "condensation", "coagulation", "aerosol_to_j", "heating_to_t", "dilution"}) +#: Fields that used to exist. Loading an archived config that still carries one must say what +#: happened, not just "unknown key" -- the config was valid when it was written. +_REMOVED_FIELDS = { + "output_dir": ("removed -- the driver never read it. ``run_coupled`` returns arrays and writes " + "nothing; the CALLER chooses where to save the .npz. Drop the key and pass the " + "path to whatever writes the output."), +} + @dataclass class Switches: @@ -119,10 +137,17 @@ class CoupledScenario: # both span dry Dp 1.7 nm - 17.5 um. Everything downstream (initial state, Mie table, optics) # follows the state's own xk grid. tomas_nbins: int = 40 - # Background aerosol size distribution seeded into the initial TomasState. "redcircles" (Marianna, - # default) uses the tabulated loader; "sabr_330"/"sabr_220"/"cesm_g6" seed a (multi-)lognormal - # from tomas_bridge.BACKGROUND_MODES (digitized from SABR/CESM plots -- see paper_ensemble docs). - background_dist: str = "redcircles" + # Background aerosol size distribution seeded into the initial TomasState. Either a NAME -- + # "redcircles" (Marianna, default) uses the tabulated loader, "sabr_330"/"sabr_220"/"cesm_g6"/... + # seed a (multi-)lognormal from coupled.backgrounds.BACKGROUND_MODES (digitized from SABR/CESM + # plots -- see paper_ensemble) -- or a USER-SUPPLIED list of lognormal modes + # [(N [cm^-3], Dg [um], sigma_g), ...] on a DIAMETER basis, normalized to a tuple of tuples. + background_dist: str | tuple = TABULATED_BACKGROUND + # Number basis of a USER-SUPPLIED background_dist mode list: "stp" or "ambient". REQUIRED for a + # mode list and REJECTED for a name (the named sets carry their own basis via + # backgrounds.AMBIENT_BACKGROUNDS). Not defaulted: the STP->ambient factor is ~0.09 at + # 68 mbar/210 K, so a wrong basis is an order-of-magnitude error in N (see backgrounds.py). + background_modes_basis: str = "" # Rate constant [cm^3/molec/s] for SO2 + HO2 -> SO3 + OH (JPL 19-5 I34). JPL gives only an UPPER # LIMIT (~1e-18) and recommends NO products, so this is a deliberate sensitivity knob: 0.0 # eliminates the channel; 1e-18/1e-17/1e-16 scan the plausible range. Only active in the sulfur @@ -142,9 +167,9 @@ class CoupledScenario: aerosol_thickness_km: float = 1.0 # plume vertical extent (km), anchored on the box altitude aerosol_band_km: tuple | None = None # optional ABSOLUTE (lo, hi) km override; None -> anchored - # --- switches & output --- + # --- switches --- + # (No output path here: ``run_coupled`` returns arrays and writes nothing -- see _REMOVED_FIELDS.) switches: Switches = field(default_factory=Switches) - output_dir: str = "coupled_output" # --- initial gas composition (pptv); species omitted start at 0 --- concentrations: dict = field(default_factory=dict) @@ -193,10 +218,26 @@ def __post_init__(self): raise ValueError(f"condensation_alpha must be in (0, 1], got {self.condensation_alpha}") if self.coag_kernel_scale < 0.0: # now wired (AD-7.2): free multiplier on the coag kernel raise ValueError(f"coag_kernel_scale must be >= 0, got {self.coag_kernel_scale}") - from coupled.tomas_bridge import BACKGROUND_MODES - if str(self.background_dist) not in ("redcircles", *BACKGROUND_MODES): - raise ValueError(f"background_dist must be 'redcircles' or one of " - f"{sorted(BACKGROUND_MODES)}, got {self.background_dist!r}") + if isinstance(self.background_dist, str): + if self.background_dist not in (TABULATED_BACKGROUND, *BACKGROUND_MODES): + raise ValueError(f"background_dist must be {TABULATED_BACKGROUND!r}, one of " + f"{sorted(BACKGROUND_MODES)}, or a list of (N, Dg, sigma_g) " + f"lognormal modes, got {self.background_dist!r}") + if self.background_modes_basis: + raise ValueError( + f"background_modes_basis={self.background_modes_basis!r} applies only to a " + f"user-supplied background_dist mode list; the named distribution " + f"{self.background_dist!r} carries its own basis (backgrounds." + f"AMBIENT_BACKGROUNDS). Leave it empty.") + else: # user-supplied lognormal modes + self.background_dist = normalize_modes(self.background_dist) + if self.background_modes_basis not in MODE_BASES: + raise ValueError( + f"a user-supplied background_dist mode list needs an explicit " + f"background_modes_basis, one of {list(MODE_BASES)}; got " + f"{self.background_modes_basis!r}. It is not defaulted because the STP->ambient " + f"factor is ~0.09 at 68 mbar/210 K -- guessing it is an order-of-magnitude " + f"error in the background number concentration.") if self.dt_couple > self.DT: raise ValueError(f"dt_couple ({self.dt_couple}) must be <= output step DT ({self.DT})") # dt_couple drives sub-stepping within an output interval, so DT must be a whole multiple of it @@ -210,6 +251,9 @@ def from_dict(cls, d: dict) -> "CoupledScenario": known = set(cls.__dataclass_fields__) unknown = set(d) - known if unknown: + removed = sorted(unknown & set(_REMOVED_FIELDS)) + if removed: # name what happened rather than "unknown key" on a config that once worked + raise ValueError("; ".join(f"{k}: {_REMOVED_FIELDS[k]}" for k in removed)) raise ValueError(f"Unknown CoupledScenario keys: {sorted(unknown)}") return cls(**d) diff --git a/coupled/driver.py b/coupled/driver.py index cb11ac5..3992340 100644 --- a/coupled/driver.py +++ b/coupled/driver.py @@ -38,7 +38,9 @@ from __future__ import annotations +import inspect import os +import warnings import numpy as np @@ -213,6 +215,56 @@ def _envelope_grid(t0, t1, n_pts: int = 301): return np.linspace(t0, t1, n_pts) +def _adapt_stop_condition(stop_condition): + """Normalize a ``run_coupled`` stop condition to the current ``f(diagnostics: dict) -> bool``. + + TWO call shapes are accepted, dispatched on the callback's DECLARED arity -- resolved once, here, + at ``run_coupled`` entry, so a mis-shaped callback raises immediately instead of at the end of + the first outer interval (minutes into a run): + + * ``f(diag)`` -- CURRENT. One dict argument; see ``run_coupled``'s docstring for its keys. + A criterion can reference SO2 (``diag["gas"]["SO2"]``) or particle number (``diag["N_total"]``), + which the two-argument form could not express. + * ``f(t1, SA)`` -- LEGACY: ``(t1 [s], wet SA [um^2/cm^3])``. Kept working because out-of-tree + run scripts and ``coupled/paper_ensemble/README.md`` document it. Emits a DeprecationWarning. + + Anything else -- zero or three-plus positional parameters, or a bare ``*args``, which is + compatible with BOTH shapes -- raises ``TypeError``. Passing a dict to a callback that expected + ``t1`` would compare a dict against a float or, worse, succeed silently; the caller must declare + which shape it means. + """ + if stop_condition is None: + return None + if not callable(stop_condition): + raise TypeError(f"stop_condition must be callable, got {type(stop_condition).__name__}") + try: + sig = inspect.signature(stop_condition) + except (TypeError, ValueError) as exc: # builtins / C callables have no signature + raise TypeError( + "stop_condition's signature could not be inspected, so its call shape cannot be " + "determined; wrap it in a plain `def stop(diag): ...` taking one dict argument") from exc + positional = [p for p in sig.parameters.values() + if p.kind in (p.POSITIONAL_ONLY, p.POSITIONAL_OR_KEYWORD)] + var_positional = any(p.kind is p.VAR_POSITIONAL for p in sig.parameters.values()) + if not var_positional and len(positional) == 1: + return stop_condition + if not var_positional and len(positional) == 2: + warnings.warn( + "The two-argument stop_condition f(t1, wet_SA) is deprecated; take a single " + "diagnostics dict instead -- f(diag) with diag['t'] and diag['SA'], plus SO2 " + "(diag['gas']['SO2']) and particle number (diag['N_total']).", + DeprecationWarning, stacklevel=3) + + def _legacy(diag, _fn=stop_condition): + return _fn(diag["t"], diag["SA"]) + return _legacy + raise TypeError( + f"stop_condition takes {'*args' if var_positional else len(positional)} positional " + f"argument(s); run_coupled supports exactly two shapes: f(diagnostics_dict) (current) or " + f"f(t1_seconds, wet_SA) (legacy). Declare one explicitly -- *args is compatible with both, " + f"so it cannot be dispatched without guessing.") + + def run_coupled(scenario, return_aerosol=False, return_state=False, return_size_dist=False, return_photolysis=False, stop_condition=None): """Integrate a CoupledScenario with operator splitting. @@ -226,7 +278,29 @@ def run_coupled(scenario, return_aerosol=False, return_state=False, return_size_ ``{"t_mid" [n_int], "J" [n_int, n_photo] (s^-1), "equations" [n_photo]}`` (the ``photo_override`` rows of ``reactions.photolysis_coeffs``, i.e. absolute TUV-x J or the j45*j_scale fallback). Extra outputs are appended in that order. + + ``stop_condition`` is an optional ``f(diagnostics) -> bool`` evaluated once per outer interval on + the END-of-interval state; returning True ends the run there (that interval is still recorded). + ``diagnostics`` is a dict: + + ================== ========================================================================= + ``t`` seconds since run start (use this, never ``interval * DT`` -- the outer + grid snaps to the terminator, so the mean step is ~592 s, not 600 s) + ``interval`` 1-based index of the completed outer interval + ``T`` box temperature [K] (evolves when ``heating_to_t`` is on) + ``SA`` **wet** aerosol surface area [um^2/cm^3]; NaN when TOMAS is inactive + ``radius_cm`` **wet** effective radius [cm]; NaN when TOMAS is inactive + ``h2so4wp`` aerosol H2SO4 weight percent; NaN when TOMAS is inactive + ``particulate_S`` aerosol sulfur [molec/cm^3, H2SO4-equivalent]; NaN when TOMAS is inactive + ``N_total`` total particle number [#/cm^3 at ambient T,P]; NaN when TOMAS is inactive + ``gas`` ``{species_name: concentration [molec/cm^3]}``, all 34 gas species + ================== ========================================================================= + + The NaNs are deliberate: a criterion such as ``diag["SA"] < x`` must not read "TOMAS is off" as + "the plume has relaxed". The legacy two-argument shape ``f(t1_seconds, wet_SA)`` still works and + is dispatched explicitly by arity -- see ``_adapt_stop_condition``. """ + _stop = _adapt_stop_condition(stop_condition) # fail fast on a mis-shaped callback cfg = to_model_config(scenario) y0 = initial_state(scenario) sulfur = float(scenario.switches.sulfur) # single gate source (NumPy match: build_env(sulfur_chain=)) @@ -314,6 +388,31 @@ def _sizedist_record(): Dpk, _ = _wet_diameters_m(tstate) return np.asarray(tstate.Nk) / float(tstate.boxvol), np.asarray(Dpk) + _species = tuple(IDX) # state-vector order; the gas dict is keyed by NAME, never by position + + def _stop_diagnostics(t_now, ivl, y, aero): + """State a termination criterion may reference, at the END of an outer interval. + + Aerosol entries are the WET quantities the heterogeneous chemistry sees (``dp_mid_um`` in the + saved size distribution is dry -- see docs/studio/CAVEATS.md), and are NaN, never 0, when + TOMAS is inactive: a ``< threshold`` criterion must not read "no aerosol model" as + "converged". Gas concentrations are the model's native molec/cm^3. + """ + sa, radius_cm, h2so4wp, particulate_S, T_box = aero + n_total = (float(np.sum(np.asarray(tstate.Nk))) / float(tstate.boxvol) + if tomas_active else float("nan")) + return { + "t": float(t_now), # s since run start + "interval": int(ivl), # 1-based index of the completed outer interval + "T": float(T_box), # box temperature, K + "SA": float(sa), # WET aerosol surface area, um^2/cm^3 + "radius_cm": float(radius_cm), # WET effective radius, cm + "h2so4wp": float(h2so4wp), # aerosol H2SO4 weight percent + "particulate_S": float(particulate_S), # molec/cm^3, as H2SO4-equivalent (AD-3.8) + "N_total": float(n_total), # particle number, #/cm^3 at ambient T,P + "gas": dict(zip(_species, (float(v) for v in np.asarray(y)))), # molec/cm^3, by name + } + # --- adaptive inner (micro) step for the gas<->TOMAS handoff (two-level integration, approach B) -- eps = float(scenario.micro_eps) floor = float(scenario.micro_floor_s) @@ -438,15 +537,15 @@ def _sizedist_record(): t_list.append(t1) x_list.append(np.asarray(yc)) - aero_list.append(_aero_record()) + _aero = _aero_record() + aero_list.append(_aero) if nk_list is not None: nk, dp = _sizedist_record() nk_list.append(nk) dp_list.append(dp) - # optional early stop (e.g. plume relaxed to background): called once per outer - # interval with (t1 [s], wet SA [um^2/cm^3] or nan when TOMAS inactive) - if stop_condition is not None and stop_condition( - float(t1), float(het["SA"]) if het is not None else float("nan")): + # optional early stop (e.g. plume relaxed to background): called once per outer interval + # with the end-of-interval diagnostics dict (built only when a stop condition is present) + if _stop is not None and _stop(_stop_diagnostics(t1, _ivl, yc, _aero)): print(f"[stop] condition met at t={t1/86400.0:.3f} d -- ending run early", flush=True) break diff --git a/coupled/paper_ensemble/FIGURES.md b/coupled/paper_ensemble/FIGURES.md index 62bc031..b167d5e 100644 --- a/coupled/paper_ensemble/FIGURES.md +++ b/coupled/paper_ensemble/FIGURES.md @@ -53,7 +53,7 @@ Every run dir has `manifest.csv`, per-case `state.npz` (full time series: 36 gas SciPy-BDF fallback is removed (see DECISIONS addendum 2026-07-08 and `debug_day12_isolation.py`). - aer_geo with N = 50 cm^-3 (the Pierce caption value) — one number in - `tomas_bridge.BACKGROUND_MODES`, then rerun `runs_geo` (resumable runner). + `coupled/backgrounds.py` `BACKGROUND_MODES`, then rerun `runs_geo` (resumable runner). - Start-time / box-size sweeps for other sites, regimes, or backgrounds (all runners parameterized). diff --git a/coupled/paper_ensemble/README.md b/coupled/paper_ensemble/README.md index 3aac780..6894d6e 100644 --- a/coupled/paper_ensemble/README.md +++ b/coupled/paper_ensemble/README.md @@ -134,16 +134,25 @@ other figure module: ## 4. Extending the model runs -- **New background aerosol**: add a mode to `BACKGROUND_MODES` in `coupled/tomas_bridge.py` +- **New background aerosol**: add a mode to `BACKGROUND_MODES` in `coupled/backgrounds.py` (number, median diameter, σg; add it to `AMBIENT_BACKGROUNDS` if the numbers are ambient - rather than STP) and reference it via `CoupledScenario(background_dist=...)`. + rather than STP) and reference it via `CoupledScenario(background_dist=...)`. Re-exported as + `tomas_bridge.BACKGROUND_MODES`, which is where it is consumed. For a one-off, pass the modes + inline instead of naming them: + `CoupledScenario(background_dist=[(50.0, 0.10, 1.6)], background_modes_basis="stp")` — + `(N [cm⁻³], Dg [µm] diameter basis, σg)`, N > 0, Dg > 0, σg > 1, and the basis + (`"stp"`/`"ambient"`) is **required**, since the STP→ambient factor is ~0.09 at 68 mbar/210 K. - **Scenario knobs** (`coupled/coupled_scenario.py`): `nucleation_rate_scale` and `coag_kernel_scale` are pure multipliers; `condensation_alpha` is the absolute accommodation coefficient; plus `ion_pair_rate`, `so2_ho2_rate`, `dilution_regime`, `dilution_background`, `tomas_nbins`, `start_utc_hour`, `days`. -- **Early stopping**: `run_coupled(..., stop_condition=f)` with `f(t1, wet_SA) -> bool` is - checked every coupling interval (see `run_60day.py` for the "within 10% of background SA - for 24 h" criterion). +- **Early stopping**: `run_coupled(..., stop_condition=f)` with `f(diag) -> bool` is checked + every coupling interval on the end-of-interval state (see `run_60day.py` for the "within + 10% of background SA for 24 h" criterion). `diag` carries `t` [s], `interval`, `T`, the wet + aerosol quantities `SA`/`radius_cm`/`h2so4wp`/`particulate_S`, `N_total` [#/cm³] and + `gas` — all 34 species by name in molec/cm³, so SO₂- or number-based criteria are + expressible. Aerosol entries are NaN (never 0) when TOMAS is inactive. The old two-argument + `f(t1, wet_SA)` still works, dispatched by arity, but is deprecated. - **Long runs**: >10-day integrations are routine (~30–40 min per 60-day run). The old day-12.14 stall was a float64 first-step pathology, fixed in `coupled/driver.py` (DECISIONS 2026-07-08); do not reintroduce a tiny `first_step`. diff --git a/coupled/paper_ensemble/run_60day.py b/coupled/paper_ensemble/run_60day.py index 2cbd83d..78dcbb5 100644 --- a/coupled/paper_ensemble/run_60day.py +++ b/coupled/paper_ensemble/run_60day.py @@ -66,7 +66,8 @@ def make_stop(sc): sa_bg = float(het_inputs(initial_tomas_state(sc))["SA"]) state = {"run": 0} - def stop(t1, sa): + def stop(diag): # run_coupled's end-of-interval diagnostics dict + sa, t1 = diag["SA"], diag["t"] # SA is the WET surface area; nan when TOMAS is inactive if not np.isfinite(sa): return False state["run"] = state["run"] + 1 if sa <= 1.10 * sa_bg else 0 diff --git a/coupled/paper_ensemble/run_bgstop.py b/coupled/paper_ensemble/run_bgstop.py index c15c71e..30abe7d 100644 --- a/coupled/paper_ensemble/run_bgstop.py +++ b/coupled/paper_ensemble/run_bgstop.py @@ -57,7 +57,8 @@ def make_stop(sc): sa_bg = float(het_inputs(initial_tomas_state(sc))["SA"]) state = {"run": 0} - def stop(t1, sa): + def stop(diag): # run_coupled's end-of-interval diagnostics dict + sa, t1 = diag["SA"], diag["t"] # SA is the WET surface area; nan when TOMAS is inactive if not np.isfinite(sa): return False state["run"] = state["run"] + 1 if sa <= 1.10 * sa_bg else 0 diff --git a/coupled/scenarios/coupled_default.yaml b/coupled/scenarios/coupled_default.yaml index e2eac99..1906a83 100644 --- a/coupled/scenarios/coupled_default.yaml +++ b/coupled/scenarios/coupled_default.yaml @@ -38,8 +38,8 @@ switches: heating_to_t: false # Phase 5 (radiative heating -> temperature) dilution: false # Phase 6 (dilution + background entrainment) -# --- output --- -output_dir: coupled_output +# NOTE: no output path here. run_coupled() returns arrays and writes nothing; the caller decides +# where to save the .npz (see coupled/paper_ensemble/run_*.py). # --- initial gas composition (pptv); species omitted start at 0 --- concentrations: diff --git a/coupled/tests/test_coupled_scenario.py b/coupled/tests/test_coupled_scenario.py index d1f0371..2612bd3 100644 --- a/coupled/tests/test_coupled_scenario.py +++ b/coupled/tests/test_coupled_scenario.py @@ -3,12 +3,16 @@ """CoupledScenario: the single coupled-model input, its switches, and validation.""" import os +import subprocess +import sys +import textwrap import pytest from coupled_scenario import CoupledScenario, Switches _DEFAULT_YAML = os.path.join(os.path.dirname(__file__), "..", "scenarios", "coupled_default.yaml") +_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) def test_defaults(): @@ -64,6 +68,14 @@ def test_unknown_key_rejected(): CoupledScenario.from_dict({"not_a_field": 1}) +def test_output_dir_is_gone_and_says_why(): + # Removed, not honoured: run_coupled returns arrays and writes nothing, so the caller owns the + # output path. An archived config carrying the key gets an explanation, not "unknown key". + assert "output_dir" not in CoupledScenario.__dataclass_fields__ + with pytest.raises(ValueError, match="the CALLER chooses where to save"): + CoupledScenario.from_dict({"output_dir": "coupled_output"}) + + def test_unknown_switch_key_raises_friendly_valueerror(): # a typo'd switch name gives a clear ValueError, not a cryptic TypeError with pytest.raises(ValueError): @@ -87,3 +99,97 @@ def test_DT_must_be_multiple_of_dt_couple(): with pytest.raises(ValueError): CoupledScenario(DT=600.0, dt_couple=250.0) # 600 / 250 is not an integer CoupledScenario(DT=600.0, dt_couple=200.0) # 600 / 200 = 3 -> ok + + +def test_bad_background_dist_raises(): + with pytest.raises(ValueError): + CoupledScenario(background_dist="sabr_999") # not a known name + + +# --- user-supplied lognormal background modes ------------------------------------------------ +_USER_MODES = [(50.0, 0.10, 1.6), (2.0, 0.5, 1.4)] + + +def test_user_modes_are_accepted_and_canonicalized(): + sc = CoupledScenario(background_dist=_USER_MODES, background_modes_basis="stp") + # lists (as YAML/JSON hand them back) canonicalize to a tuple of float 3-tuples + assert sc.background_dist == ((50.0, 0.10, 1.6), (2.0, 0.5, 1.4)) + assert all(isinstance(v, float) for mode in sc.background_dist for v in mode) + + +def test_user_modes_round_trip_through_yaml(tmp_path): + sc = CoupledScenario(background_dist=_USER_MODES, background_modes_basis="ambient") + p = tmp_path / "s.yaml" + sc.save(str(p)) + assert CoupledScenario.load(str(p)).to_dict() == sc.to_dict() + + +def test_user_modes_require_an_explicit_basis(): + # The STP->ambient factor is ~0.09 at 68 mbar/210 K, so the basis is an order-of-magnitude + # decision and must never be defaulted. + with pytest.raises(ValueError, match="background_modes_basis"): + CoupledScenario(background_dist=_USER_MODES) + with pytest.raises(ValueError, match="background_modes_basis"): + CoupledScenario(background_dist=_USER_MODES, background_modes_basis="STP") # not a member + + +def test_named_background_rejects_a_basis(): + # the named sets carry their own basis (backgrounds.AMBIENT_BACKGROUNDS); accepting a + # contradicting one here would let a config claim a conversion that never happens + with pytest.raises(ValueError, match="background_modes_basis"): + CoupledScenario(background_dist="sabr_220", background_modes_basis="ambient") + + +@pytest.mark.parametrize("modes, why", [ + ([], "empty"), + ([(0.0, 0.1, 1.6)], "N == 0"), + ([(-1.0, 0.1, 1.6)], "N < 0"), + ([(50.0, 0.0, 1.6)], "Dg == 0"), + ([(50.0, -0.1, 1.6)], "Dg < 0"), + ([(50.0, 0.1, 1.0)], "sigma_g == 1 -> log10(sigma_g) = 0, a divide by zero"), + ([(50.0, 0.1, 0.8)], "sigma_g < 1"), + ([(50.0, 0.1)], "not a 3-tuple"), + ([(50.0, 0.1, 1.6, 2.0)], "too long"), + ([50.0], "not a tuple at all"), + ([(50.0, 0.1, "wide")], "non-numeric"), + (42, "not a sequence"), +]) +def test_degenerate_user_modes_raise(modes, why): + with pytest.raises(ValueError): + CoupledScenario(background_dist=modes, background_modes_basis="stp"), why + + +# --------------------------------------------------------------------------------------------- +# Constructing a scenario must not import JAX. +# +# ``__post_init__`` validates ``background_dist``; it used to do that via +# ``from coupled.tomas_bridge import BACKGROUND_MODES``, which imports jax and sets +# jax_enable_x64 -- ~1 s on the first CoupledScenario() in a process. The tables now live in the +# JAX-free ``coupled.backgrounds``. +# +# This MUST run in a fresh interpreter. An in-process ``"jax" not in sys.modules`` assertion is +# vacuous here: the rest of this suite imports the driver, so jax is already resident by the time +# any single test runs, and the assertion would pass (or fail) for reasons unrelated to this module. +# --------------------------------------------------------------------------------------------- +_NO_JAX_PROBE = textwrap.dedent(""" + import sys + from coupled.coupled_scenario import CoupledScenario + CoupledScenario() # default: the tabulated background + CoupledScenario(background_dist="sabr_220") # a named lognormal mode set + try: + CoupledScenario(background_dist="not_a_background") + except ValueError: + pass + else: + raise AssertionError("an unknown background_dist must still be rejected") + leaked = sorted({n.split(".")[0] for n in sys.modules} & {"jax", "jaxlib"}) + assert not leaked, f"constructing a CoupledScenario imported {leaked}" + """) + + +def test_constructing_a_scenario_does_not_import_jax(): + proc = subprocess.run([sys.executable, "-c", _NO_JAX_PROBE], + capture_output=True, text=True, cwd=_REPO_ROOT) + # cwd = repo root, where `coupled` (and therefore jax) IS importable: the check is only + # meaningful if the expensive import is available and simply never made. + assert proc.returncode == 0, f"probe failed:\n{proc.stdout}\n{proc.stderr}" diff --git a/coupled/tests/test_stop_condition.py b/coupled/tests/test_stop_condition.py new file mode 100644 index 0000000..b59f9b5 --- /dev/null +++ b/coupled/tests/test_stop_condition.py @@ -0,0 +1,182 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``run_coupled(stop_condition=...)``: the diagnostics dict, and the legacy two-argument shape. + +The callback used to receive only ``(t1_seconds, wet_SA)``, so a criterion could not reference SO2 +or particle number. It now receives one dict; the old shape is still accepted and dispatched by +declared arity, never by trying a call and catching the TypeError. +""" + +import warnings + +import numpy as np +import pytest + +import coupled.driver as cd +from coupled import CoupledScenario +from coupled.coupled_scenario import Switches +from config import IDX + +_COMP = {"O2": 2.1e11, "O3": 1.18e6, "CH4": 1.6e6, "SO2": 9.5e5, "ClO": 10.0, + "ClONO2": 127.0, "HCl": 777.0, "N2O5": 20.0, "NO": 450.0, "NO2": 450.0, + "OH": 0.5, "HO2": 3.0} + +# Gas-only: this file tests the callback plumbing, not microphysics. TOMAS-inactive is also the case +# where the aerosol entries must be NaN, which is half of what is asserted below. +_GAS_ONLY = dict(sulfur=True, nucleation=False, condensation=False, coagulation=False, + aerosol_to_j=False, heating_to_t=False, dilution=False) + + +def _scn(**kw): + base = dict(P=68.0, T=210.0, WTR=5.0, latitude=0.0, longitude=0.0, day_of_year=80, + start_utc_hour=0.0, days=1, DT=14400.0, dt_couple=14400.0, photolysis="sza", + concentrations=_COMP, switches=Switches(**_GAS_ONLY)) + base.update(kw) + return CoupledScenario(**base) + + +# -------------------------------------------------------------------------------------------- +# Arity dispatch, in isolation from a run. +# -------------------------------------------------------------------------------------------- +def test_one_argument_callback_passes_through_unwrapped(): + def stop(diag): + return False + assert cd._adapt_stop_condition(stop) is stop + + +def test_none_stays_none(): + assert cd._adapt_stop_condition(None) is None + + +def test_two_argument_callback_is_adapted_and_deprecated(): + seen = [] + + def legacy(t1, sa): + seen.append((t1, sa)) + return t1 > 100.0 + + with pytest.warns(DeprecationWarning, match="two-argument stop_condition"): + adapted = cd._adapt_stop_condition(legacy) + assert adapted is not legacy + # the adapter must map exactly diag["t"] -> t1 and diag["SA"] -> sa, in that order + assert adapted({"t": 500.0, "SA": 2.5, "gas": {}}) is True + assert seen == [(500.0, 2.5)] + + +@pytest.mark.parametrize("fn", [ + lambda: False, # 0 positional -> no shape + lambda a, b, c: False, # 3 positional -> no shape + lambda *args: False, # matches BOTH shapes -> ambiguous, must not be guessed +]) +def test_undispatchable_arities_raise(fn): + with pytest.raises(TypeError, match="stop_condition"): + cd._adapt_stop_condition(fn) + + +def test_non_callable_raises(): + with pytest.raises(TypeError, match="callable"): + cd._adapt_stop_condition(3.0) + + +def test_dispatch_counts_all_positional_parameters_not_only_required_ones(): + # `def stop(t1, sa=nan)` declares two positionals and is therefore read as the LEGACY shape, + # even though it is callable with one argument. Dispatch is on the declared signature, so it + # does not depend on defaults -- and a one-argument callback must not be written that way. + def stop(t1, sa=float("nan")): + return False + with pytest.warns(DeprecationWarning): + adapted = cd._adapt_stop_condition(stop) + assert adapted is not stop + + +# -------------------------------------------------------------------------------------------- +# End to end, through a real (gas-only) run. +# -------------------------------------------------------------------------------------------- +def test_diagnostics_dict_contents_and_early_stop(): + seen = [] + + def stop(diag): + seen.append(diag) + # An SO2-based criterion -- not expressible with the old (t1, wet_SA) signature at all. + # SO2 is a pure sink here (the chain has no source), so it is non-increasing. + return diag["gas"]["SO2"] <= seen[0]["gas"]["SO2"] and diag["interval"] >= 2 + + sc = _scn(days=1) + t, x = cd.run_coupled(sc, stop_condition=stop) + + assert len(seen) == 2 # stopped at the end of the 2nd outer interval + assert t.shape[0] == 3 # t=0 plus the two completed intervals + assert x.shape[0] == t.shape[0] + + first, last = seen[0], seen[-1] + assert set(first) == {"t", "interval", "T", "SA", "radius_cm", "h2so4wp", + "particulate_S", "N_total", "gas"} + assert first["interval"] == 1 and last["interval"] == 2 + # `t` is the driver's own clock, matching the returned time vector exactly -- the outer grid + # snaps to the terminator, so interval * DT would drift from it. + assert first["t"] == pytest.approx(t[1]) and last["t"] == pytest.approx(t[2]) + assert last["t"] > first["t"] + assert first["T"] == pytest.approx(sc.T) # heating_to_t off -> box T unchanged + # TOMAS inactive -> aerosol entries are NaN, never 0: a `SA < x` criterion must not read + # "no aerosol model" as "the plume has relaxed". + for key in ("SA", "radius_cm", "h2so4wp", "particulate_S", "N_total"): + assert np.isnan(first[key]), key + # gas is keyed by species NAME and matches the recorded state row exactly (same units, + # molec/cm^3); indexing by name is what protects against a state-vector reordering. + assert set(last["gas"]) == set(IDX) + for name, idx in IDX.items(): + assert last["gas"][name] == pytest.approx(float(x[2, idx]), rel=0.0, abs=0.0) + + +def test_never_stopping_condition_runs_to_completion(): + calls = [] + sc = _scn(days=1) + t, _x = cd.run_coupled(sc, stop_condition=lambda diag: bool(calls.append(diag["t"]))) + assert t[-1] == 86400.0 # ran the full day + assert calls == list(t[1:]) # called once per completed outer interval + + +def test_legacy_two_argument_shape_stops_at_the_same_point_as_the_dict_form(): + """The old f(t1, wet_SA) contract, unchanged: identical stop point, identical time vector.""" + def legacy(t1, sa): + assert np.isnan(sa) # gas-only run: wet SA is NaN, exactly as before + return t1 >= 20000.0 + + def modern(diag): + assert np.isnan(diag["SA"]) + return diag["t"] >= 20000.0 + + sc = _scn(days=1) + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + t_legacy, _x = cd.run_coupled(sc, stop_condition=legacy) + t_modern, _x2 = cd.run_coupled(sc, stop_condition=modern) + assert np.array_equal(t_legacy, t_modern) + assert t_legacy[-1] < 86400.0 # and it really did stop early + + +def test_mis_shaped_callback_raises_before_any_integration(monkeypatch): + # a bad callback must be rejected at entry, not after the first (minutes-long) interval + def boom(*_a, **_kw): + raise AssertionError("run_coupled started integrating despite a bad stop_condition") + monkeypatch.setattr(cd, "to_model_config", boom) + with pytest.raises(TypeError, match="stop_condition"): + cd.run_coupled(_scn(), stop_condition=lambda a, b, c: True) + + +@pytest.mark.slow +def test_tomas_active_run_reports_finite_aerosol_and_number(): + """With TOMAS on, the aerosol entries are real numbers -- including N_total, which is the + quantity the old two-argument callback could not see at all.""" + sc = _scn(days=1, DT=3600.0, dt_couple=3600.0, + concentrations={**_COMP, "SO2": 1.0e4}, + switches=Switches(**{**_GAS_ONLY, "nucleation": True, "condensation": True, + "coagulation": True})) + seen = [] + cd.run_coupled(sc, stop_condition=lambda diag: bool(seen.append(diag)) or diag["interval"] >= 2) + diag = seen[-1] + for key in ("SA", "radius_cm", "h2so4wp", "particulate_S", "N_total"): + assert np.isfinite(diag[key]) and diag[key] > 0.0, key + # N_total is a number density at ambient T,P: the stratospheric background is O(1-100) cm^-3, + # so anything outside a very generous 1e-3..1e6 band means a units or per-cell/per-cm^3 mix-up. + assert 1.0e-3 < diag["N_total"] < 1.0e6 diff --git a/coupled/tests/test_tomas_bridge.py b/coupled/tests/test_tomas_bridge.py index 7aac645..6492dbc 100644 --- a/coupled/tests/test_tomas_bridge.py +++ b/coupled/tests/test_tomas_bridge.py @@ -3,6 +3,7 @@ """Phase 3.1: the TOMAS bridge -- initial state from the Marianna distribution + the SO2-off step.""" import numpy as np +import pytest from coupled import CoupledScenario from coupled import tomas_bridge as tb @@ -97,6 +98,51 @@ def test_80bin_initial_state_matches_40bin_totals(): assert abs(h80["radius_cm"] / h40["radius_cm"] - 1.0) < 0.05 +def test_user_modes_reproduce_the_equivalent_named_background_exactly(): + """A user-supplied mode list is seeded by the same code path as a named set. + + Passing the named set's own modes with its own basis must give a BIT-IDENTICAL initial state: + the only difference between the two branches is where the tuples came from, so anything less + than exact equality would mean the user path applies a different conversion. + """ + import jax.numpy as jnp + from coupled.backgrounds import BACKGROUND_MODES, AMBIENT_BACKGROUNDS + for name in ("sabr_220", "cesm_g6", "aer_geo"): + basis = "ambient" if name in AMBIENT_BACKGROUNDS else "stp" + named = tb.initial_tomas_state(_scenario(background_dist=name)) + user = tb.initial_tomas_state(_scenario(background_dist=BACKGROUND_MODES[name], + background_modes_basis=basis)) + assert jnp.array_equal(named.Nk, user.Nk), name + assert jnp.array_equal(named.Mk, user.Mk), name + + +def test_user_modes_basis_changes_the_seeded_number(): + """stp vs ambient is not cosmetic: at 68 mbar / 210 K the conversion is ~0.09, so a scenario + that omitted the basis and got a default would be wrong by an order of magnitude.""" + import jax.numpy as jnp + modes = [(50.0, 0.10, 1.6)] + n_stp = float(jnp.sum(tb.initial_tomas_state( + _scenario(background_dist=modes, background_modes_basis="stp")).Nk)) + n_amb = float(jnp.sum(tb.initial_tomas_state( + _scenario(background_dist=modes, background_modes_basis="ambient")).Nk)) + from background_aerosol_distribution import stp_to_ambient_factor + f = stp_to_ambient_factor(210.0, 6800.0) + assert f < 0.2 # ~0.09 at 68 mbar / 210 K + assert n_stp == pytest.approx(n_amb * f, rel=1e-12) + + +def test_user_modes_multi_mode_number_is_physical(): + # two well-separated modes: total seeded number ~= sum of the mode N's (x the STP->ambient + # factor), to a few percent -- the grid spans 1.7 nm - 17.5 um, so almost nothing falls outside. + import jax.numpy as jnp + from background_aerosol_distribution import stp_to_ambient_factor + modes = [(50.0, 0.05, 1.6), (3.0, 0.4, 1.5)] + st = tb.initial_tomas_state(_scenario(background_dist=modes, background_modes_basis="stp")) + expected = (50.0 + 3.0) * stp_to_ambient_factor(210.0, 6800.0) * tb.BOXVOL_CM3 + assert float(jnp.sum(st.Nk)) == pytest.approx(expected, rel=0.02) + assert float(jnp.sum(st.Mk[:, tb.SRTSO4])) > 0.0 + + def test_ion_pair_rate_scales_nucleation(): # fion > 0 must strengthen nucleation (Dunne ion-induced channels) for the same H2SO4. import jax.numpy as jnp diff --git a/coupled/tomas_bridge.py b/coupled/tomas_bridge.py index a42d819..ce01d9d 100644 --- a/coupled/tomas_bridge.py +++ b/coupled/tomas_bridge.py @@ -19,6 +19,12 @@ import os import sys +# The background mode tables live in coupled.backgrounds -- a JAX-free module, so that scenario +# validation can reach them without importing this one. Re-exported here because the paper scripts and +# docs refer to ``tomas_bridge.BACKGROUND_MODES``; this module remains where they are USED. +from .backgrounds import (BACKGROUND_MODES, AMBIENT_BACKGROUNDS, # noqa: F401 (re-export) + MODE_BASES, TABULATED_BACKGROUND, normalize_modes) + # gas model on path first (for the water-activity calc used to set RH) -- model_bridge does the insert. from . import model_bridge # noqa: F401 (side effect: puts gas_phase_chemistry on sys.path) from aerosol import h2so4wp_at # noqa: E402 (gas model: water activity a_W from T,P,H2O) @@ -82,31 +88,6 @@ def _grid_for(nbins): return tcfg.make_grid(nbins, tcfg.XK0, 2.0 ** (40.0 / nbins)) -# --- background aerosol size distributions as (multi-)lognormal modes, DIAMETER basis. Each entry is -# a list of (N [cm^-3, STP], Dg [um], sigma_g). DIGITIZED (approximate) from the SABR / CESM plots the -# user provided; see coupled/analyses/paper_ensemble/DECISIONS.md for the source figures and the -# overlay-verification. "redcircles" (Marianna, tabulated loader) stays the default and is NOT here. --- -# N chosen so each mode's PEAK dN/dlogDp = N/(sqrt(2pi)*log10(sigma_g)) matches the value read off the -# source plot (the most reliable digitized feature): SABR 330->~1000, 220->~95; CESM Aitken->~50, -# Accumulation->~12, Coarse->~0.5 cm^-3 STP. -BACKGROUND_MODES = { - "sabr_330": [(810.0, 0.045, 2.1)], # young air (high N2O), peak ~1000 - "sabr_310": [(205.0, 0.060, 1.8)], # mid air (310-320 ppbv), peak ~320 - "sabr_220": [(49.0, 0.12, 1.6)], # aged air (low N2O), peak ~95 - "cesm_g6": [(22.0, 0.040, 1.5), (5.3, 0.20, 1.5), (0.18, 0.90, 1.4)], # CESM G6 SAI (r->D x2) - # AER 2D geoengineered stratosphere (Pierce et al. fig. 2, gray curve: 5 Mt-S/yr, 95 nm case). - # Dg = 0.30 um (mode radius 0.15 um) and sigma_g = 1.7 fitted to the curve; N = 120 cm^-3 per - # user spec (paper caption quotes 50 cm^-3). Values are AMBIENT -> no STP conversion on seeding. - "aer_geo": [(120.0, 0.30, 1.7)], - # CESM G6 with the source plot read as AMBIENT (user-confirmed): same modes as cesm_g6 but - # seeded without the STP->ambient factor. cesm_g6 is kept unchanged so the original 810-run - # ensemble stays reproducible. - "cesm_g6_amb": [(22.0, 0.040, 1.5), (5.3, 0.20, 1.5), (0.18, 0.90, 1.4)], -} -# mode sets specified at AMBIENT conditions (seeding skips the STP->ambient factor) -AMBIENT_BACKGROUNDS = {"aer_geo", "cesm_g6_amb"} - - def _seed_lognormal(xk_np, boxvol, modes, temp, pres, ambient=False): """Nk [#/cell], Mk [kg/cell] from (multi-)lognormal modes -- mirrors _bad.map_to_grid: integrate dN/dlog10Dp over each bin [cm^-3 STP], x STP->ambient x boxvol, Mk = Nk x geometric-mean bin mass @@ -134,7 +115,12 @@ def dNdlogDp(x): def initial_tomas_state(scenario) -> TomasState: - """Build the initial ``TomasState`` from the scenario using the Marianna 'redcircles' distribution. + """Build the initial ``TomasState`` from the scenario's background aerosol distribution. + + ``scenario.background_dist`` is either a NAME -- ``"redcircles"`` (Marianna, the tabulated + loader) or a key of ``BACKGROUND_MODES`` -- or a USER-SUPPLIED list of ``(N, Dg, sigma_g)`` + lognormal modes, in which case ``scenario.background_modes_basis`` ("stp" / "ambient") states + the number basis, since a bare mode list carries none (see coupled/backgrounds.py). Gc is all-zero: gaseous H2SO4 is handed in by the driver each outer step (the gas model owns it). Number/mass are per grid cell (``boxvol=BOXVOL_CM3``); T in K, P in Pa (scenario.P is mbar). @@ -147,12 +133,22 @@ def initial_tomas_state(scenario) -> TomasState: raise ValueError(f"tomas_nbins must be 40, 80, or 160, got {nbins}") pres_pa = scenario.P * 100.0 # mbar -> Pa (TOMAS uses Pa) xk = _grid_for(nbins) - bg = str(getattr(scenario, "background_dist", "redcircles")) - if bg in BACKGROUND_MODES: + bg = getattr(scenario, "background_dist", TABULATED_BACKGROUND) + if not isinstance(bg, str): + # user-supplied lognormal modes; the basis is explicit (CoupledScenario enforces it, and it + # is re-checked here because initial_tomas_state accepts any scenario-shaped object) + modes = normalize_modes(bg) + basis = str(getattr(scenario, "background_modes_basis", "")) + if basis not in MODE_BASES: + raise ValueError(f"a user-supplied background_dist mode list needs " + f"background_modes_basis in {list(MODE_BASES)}, got {basis!r}") + Nk_np, Mk_np = _seed_lognormal(np.asarray(xk), BOXVOL_CM3, modes, + scenario.T, pres_pa, ambient=basis == "ambient") + elif bg in BACKGROUND_MODES: # seed a (multi-)lognormal background (SABR / CESM) on our grid Nk_np, Mk_np = _seed_lognormal(np.asarray(xk), BOXVOL_CM3, BACKGROUND_MODES[bg], scenario.T, pres_pa, ambient=bg in AMBIENT_BACKGROUNDS) - elif bg == "redcircles": + elif bg == TABULATED_BACKGROUND: if nbins in (40, 80): # validated paths: get_initial_state special-cases 80 (sqrt2); 40 uses ratio 2.0. Nk_np, Mk_np = _bad.get_initial_state( @@ -165,8 +161,8 @@ def initial_tomas_state(scenario) -> TomasState: f = _bad.stp_to_ambient_factor(scenario.T, pres_pa) # STP dN/dlogDp -> ambient Nk_np, Mk_np = Nk_np * f, Mk_np * f else: - raise ValueError(f"unknown background_dist {bg!r}; valid: 'redcircles' + " - f"{sorted(BACKGROUND_MODES)}") + raise ValueError(f"unknown background_dist {bg!r}; valid: {TABULATED_BACKGROUND!r}, " + f"{sorted(BACKGROUND_MODES)}, or a list of (N, Dg, sigma_g) modes") Nk = jnp.asarray(Nk_np, dtype=jnp.float64) Mk = jnp.asarray(Mk_np, dtype=jnp.float64) Gc = jnp.zeros(tcfg.N_GAS_SPECIES, dtype=jnp.float64) diff --git a/docs/studio/OPEN_QUESTIONS.md b/docs/studio/OPEN_QUESTIONS.md index 7b66aff..840c270 100644 --- a/docs/studio/OPEN_QUESTIONS.md +++ b/docs/studio/OPEN_QUESTIONS.md @@ -161,9 +161,9 @@ Required per background distribution: diameter basis (dry vs ambient, and at wha ambient), composition and mixing state (internal vs external, sulfate mass fraction), and whether meteoric material is represented at all. -**Partially constrained by the code already:** `coupled/tomas_bridge.py:92-105` defines six named -lognormal backgrounds as sulfate-only (`Mk[:, SRTSO4]`, `_seed_lognormal:131-132`), and -`AMBIENT_BACKGROUNDS` (`:107`) marks which mode sets are specified at ambient vs STP — so the +**Partially constrained by the code already:** `coupled/backgrounds.py` defines six named +lognormal backgrounds, seeded as sulfate-only (`Mk[:, SRTSO4]` in `tomas_bridge._seed_lognormal`), +and `AMBIENT_BACKGROUNDS` marks which mode sets are specified at ambient vs STP — so the dry/ambient distinction exists but is per-dataset and implicit rather than a declared field. Meteoric material is **not** represented. diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 89773a3..dce1bd9 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -29,6 +29,42 @@ derivations to resolve rather than fixtures. --- +### 2026-08-14 — `main` merged into `studio/dev`; the stop condition moves to the diagnostics dict + +`studio/dev` now has #73's four model fixes. Studio's whole suite (196 Tier-A tests, equivalence +tests included) passes against the changed model unmodified — `output_dir` is gone and nothing missed +it, which is the evidence that removing it was safe rather than merely tidy. + +**Conflicts resolved, one per file, deliberately:** + +- `studio/tests/unit/test_import_boundaries.py` — took `studio/dev`'s *structure* (three clean + packages including `studio.resolve`, and the seam check that allows submodules of `studio.modelio`) + with `main`'s *wording* (the `__post_init__` pattern it describes is now past tense, because #73 + fixed it). Asserted both survived rather than eyeballing the merge. +- `studio/__init__.py` — merged cleanly, and the merge exposed a stale docstring of mine: it still + claimed two clean packages after 0.3 added a third. Fixed here, since this is where it became + visible. + +**The stop condition now takes the diagnostics dict.** #73 deprecated the two-argument form, so +leaving it would have had Studio emit a `DeprecationWarning` on a normal path. `diag["t"]` is all it +reads today, but the dict also carries `SA`, `N_total` and every gas species by name — which is what +makes the spec's SO2- or number-based `termination.criteria[]` possible at all. + +**`max_sim_time` enforcement is now proven end to end, not just unit-tested.** A 1-day request with +`max_sim_time_days = 0.5`, run through `studio.cli.run` with `-W error::DeprecationWarning`: + +``` +[stop] condition met at t=0.500 d -- ending run early +termination: terminated_on_limit flags: [stopped_on_limit, open_system_dilution] +t_end: 0.5 d (requested 1.0) 74 steps +``` + +That path had never actually run before — nothing set `max_sim_time`, so it was the one part of 0.6 +covered only by unit tests. It also confirms 0.6's termination inference: the run is labelled +`TERMINATED_ON_LIMIT` and flagged, so its partial output cannot be read as converged. + +--- + ### 2026-08-14 — Correction: `dp_mid_um` is not bit-identical for a Studio-produced run The tolerance record merged in #74 proposed asserting `dp_mid_um` **exact**. That holds only when the @@ -506,10 +542,14 @@ Scaffolding only; no runnable app code yet. **Still open:** BLOCKING-2 (tenancy/auth), SCIENCE-1 through SCIENCE-5. **Findings worth flagging beyond the docs** — each is a tracked issue, not a TODO comment: -- Constructing a `CoupledScenario` imports JAX, via `from coupled.tomas_bridge import +- ~~Constructing a `CoupledScenario` imports JAX, via `from coupled.tomas_bridge import BACKGROUND_MODES` at `coupled_scenario.py:196`. An API validating a form per keystroke cannot pay - that. → task 0.8. -- `CoupledScenario.output_dir` is dead — the driver never reads it. → task 0.8. + that.~~ → fixed in task 0.8: the tables moved to the JAX-free `coupled/backgrounds.py`; the first + `CoupledScenario()` in a process went from ~1.0 s to ~0 s, and no longer loads JAX at all. +- ~~`CoupledScenario.output_dir` is dead — the driver never reads it.~~ → task 0.8 **removed** it + (rather than honouring it): `run_coupled` returns arrays and writes nothing, so the caller owns + the output path; honouring the field would have given the driver a filesystem side effect and a + second, competing source of truth for where a run's results live. - `make_paper_candidate_plots.py:70` hardcodes species indices `_SO2, _SO3, _H2SO4 = 32, 34, 35` while the npz carries its own `species` list. Latent breakage; Studio indexes by name. - The figure caches have no version stamp and no input-hash key, and are invalidated by manual diff --git a/studio/__init__.py b/studio/__init__.py index c80f683..197d7c1 100644 --- a/studio/__init__.py +++ b/studio/__init__.py @@ -8,10 +8,10 @@ Package boundaries -- enforced by ``studio/tests/unit/test_import_boundaries.py``: -* ``studio.schema`` and ``studio.science`` must import NOTHING from the API, the database, the web - layer, or ``coupled``. Importing ``coupled`` pulls in JAX and TOMAS (``coupled_scenario.py`` - validates ``background_dist`` against ``coupled.tomas_bridge``), which an API validating a form on - every keystroke cannot afford. Both packages must be usable from a bare Python session. +* ``studio.schema``, ``studio.science`` and ``studio.resolve`` must import NOTHING from the API, the + database, the web layer, or ``coupled``. Importing the model (``coupled.driver`` / + ``coupled.tomas_bridge``) pulls in JAX and TOMAS, which an API resolving a config on every + keystroke cannot afford. All three must be usable from a bare Python session. * ``studio.modelio`` is the ONLY package permitted to import ``coupled``. It is the seam that would be severed if Studio were ever extracted to its own repository. diff --git a/studio/modelio/execute.py b/studio/modelio/execute.py index db1070b..750eb0b 100644 --- a/studio/modelio/execute.py +++ b/studio/modelio/execute.py @@ -20,6 +20,7 @@ from __future__ import annotations +from collections.abc import Mapping from pathlib import Path from typing import Any @@ -105,32 +106,32 @@ def run_and_write(config: ResolvedConfig, out_dir: Path) -> dict[str, Path]: def _max_sim_time_stop(config: ResolvedConfig) -> Any: """A ``stop_condition`` enforcing ``termination.max_sim_time_days``, or ``None``. - **The two-argument shape is deliberate and load-bearing.** Task 0.8 (PR #73) widens the model's - callback to take a diagnostics dict and dispatches on the callback's DECLARED ARITY: + **Single-parameter, taking the model's diagnostics dict.** The model dispatches on the + callback's DECLARED ARITY (``coupled/driver.py``, ``_adapt_stop_condition``): =========================== ========================================== - ``def stop(t1, wet_SA)`` accepted as legacy, with a DeprecationWarning - ``def stop(diag)`` accepted as the new dict shape + ``def stop(diag)`` the current shape + ``def stop(t1, wet_SA)`` legacy, accepted with a ``DeprecationWarning`` ``def stop(*args)`` **TypeError** -- it matches both, so it is ambiguous =========================== ========================================== - Raising on ``*args`` is the right call by the model: guessing which shape a variadic callback - wanted would be a silent wrong answer. But it means the obvious "works with either" spelling is - the one thing that does not, so this stays two-positional -- which works against the model both - before and after that change. + Raising on ``*args`` is right: guessing which shape a variadic callback wanted would be a + silent wrong answer. It does mean the obvious "works with either" spelling is the one that does + not, so the arity here is part of the contract rather than an implementation detail -- see the + tests that pin it. - Migrating to the dict shape is worth doing once #73 is in ``studio/dev``: it is what makes - termination criteria on SO2 or particle number possible, which is the reason the callback was - widened at all. Until then this only needs the simulated time, which is the first argument in - both shapes. + Only ``diag["t"]`` is read today. The dict also carries ``SA``, ``N_total`` and every gas + species by name (``diag["gas"]["SO2"]``), which is what makes SO2- or number-based termination + criteria possible when the schema grows them -- the spec's ``termination.criteria[]`` needed + exactly this. """ limit_days = config.config.termination.max_sim_time_days if limit_days is None: return None limit_s = float(limit_days) * 86400.0 - def stop(t1_seconds: float, wet_surface_area: float) -> bool: - return float(t1_seconds) >= limit_s + def stop(diag: Mapping[str, Any]) -> bool: + return float(diag["t"]) >= limit_s return stop diff --git a/studio/tests/unit/test_import_boundaries.py b/studio/tests/unit/test_import_boundaries.py index 8135835..f2f06d0 100644 --- a/studio/tests/unit/test_import_boundaries.py +++ b/studio/tests/unit/test_import_boundaries.py @@ -3,17 +3,18 @@ """The package boundaries from ADR-001, enforced rather than documented. ``studio.schema``, ``studio.science`` and ``studio.resolve`` must be usable from a bare Python -session: no ``coupled``, no JAX, no database, no web framework. This is not tidiness. Constructing -a ``CoupledScenario`` imports JAX transitively -- ``coupled_scenario.__post_init__`` validates -``background_dist`` against ``coupled.tomas_bridge``, which sets ``jax_enable_x64`` at import -- -and an API that validates a form on every keystroke cannot pay a JAX import. +session: no ``coupled``, no JAX, no database, no web framework. This is not tidiness. +``coupled.tomas_bridge`` and ``coupled.driver`` import JAX (and set ``jax_enable_x64``) at module +scope, and an API that validates a form on every keystroke cannot pay a JAX import. Two complementary checks: * a RUNTIME one, importing each clean package in a fresh interpreter and inspecting ``sys.modules``; * a STATIC one, scanning the source tree, which catches an import added inside a function body where - the runtime check would not reach it. ``coupled_scenario.py:196`` is exactly that pattern, so it - is not a hypothetical. + the runtime check would not reach it. ``CoupledScenario.__post_init__`` used to hold exactly that + pattern -- a function-body ``from coupled.tomas_bridge import BACKGROUND_MODES`` that made merely + *constructing* a scenario cost ~1 s (fixed in task 0.8 by ``coupled/backgrounds.py``) -- so it is + not a hypothetical. The runtime check must run in a fresh interpreter. Inspecting ``sys.modules`` in-process would be meaningless: by then pytest and its plugins have imported plenty, and another test module importing diff --git a/studio/tests/unit/test_runner.py b/studio/tests/unit/test_runner.py index 5f56b04..2b32885 100644 --- a/studio/tests/unit/test_runner.py +++ b/studio/tests/unit/test_runner.py @@ -335,11 +335,12 @@ def test_no_limit_means_no_stop_condition(self) -> None: assert _max_sim_time_stop(resolve(RunConfig())) is None @pytest.mark.tier_a - def test_the_callback_takes_exactly_two_positional_arguments(self) -> None: - """Not ``*args``: PR #73 rejects a variadic callback as ambiguous. + def test_the_callback_takes_exactly_one_parameter(self) -> None: + """The diagnostics-dict shape. Not ``*args``, which the model rejects as ambiguous. - If this assertion ever fails because the shape moved to the diagnostics dict, that is the - intended migration -- update it deliberately, in the commit that does the migration. + Arity is part of the contract, not an implementation detail: two parameters still work but + emit a ``DeprecationWarning``, and a variadic callback raises ``TypeError``. Asserted here + because it is cheap to notice and expensive to discover from a run. """ import inspect @@ -348,8 +349,8 @@ def test_the_callback_takes_exactly_two_positional_arguments(self) -> None: stop = _max_sim_time_stop(_with_sim_limit(2.0)) assert stop is not None parameters = list(inspect.signature(stop).parameters.values()) - assert len(parameters) == 2 - assert all(p.kind is inspect.Parameter.POSITIONAL_OR_KEYWORD for p in parameters) + assert len(parameters) == 1 + assert parameters[0].kind is inspect.Parameter.POSITIONAL_OR_KEYWORD @pytest.mark.tier_a def test_it_fires_exactly_at_the_limit(self) -> None: @@ -359,9 +360,9 @@ def test_it_fires_exactly_at_the_limit(self) -> None: stop = _max_sim_time_stop(_with_sim_limit(2.0)) assert stop is not None two_days_s = 2.0 * 86400.0 - assert stop(two_days_s - 1.0, 12.0) is False - assert stop(two_days_s, 12.0) is True - assert stop(two_days_s + 1.0, 12.0) is True + assert stop({"t": two_days_s - 1.0, "SA": 12.0}) is False + assert stop({"t": two_days_s, "SA": 12.0}) is True + assert stop({"t": two_days_s + 1.0, "SA": 12.0}) is True def _with_sim_limit(days: float) -> ResolvedConfig: From 994666c711031a2edf32af78543cb3750e37844a Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 16:11:06 -0700 Subject: [PATCH 12/18] studio: the two-tier golden harness (task 0.7) (#79) * studio: the two-tier golden harness (task 0.7) The assertions, on the tolerances #70 measured and #76 corrected. studio/tests/golden: tolerances.py, paper_cases.py, make_fixture.py, one test module per tier. 17 new Tier-A tests (214 total) plus 3 Tier-B tests. Tier A is a real 1-day/40-bin run in 19 s against a committed 40 kB fixture -- the cheapest run that still exercises gas chemistry, TUV-x photolysis, all three microphysics processes and dilution. The fixture is a UNIFORM-stride reduction (every 4th sample plus the last) because a coarsening grid aliases the morning number spike by up to 8x, and a fixture built on one would encode the aliasing and assert it forever. It catches drift in Studio's own pipeline, which is a different claim from reproducing the archive. Tier B reproduces six curated 10-day cases from the archive in ~28 min, nightly or manual: D1/D2/D3/burst against both backgrounds, all cg1 because REFERENCE_TOLERANCES.md records that cg0p5/cg2 may straddle the tomas-jax commit that wired coag_kernel_scale through. The tolerances live in one module, each citing the measurement it came from, and the failure messages say re-measure rather than widen -- a tolerance widened to make a test pass is a test that no longer tests anything. The near-zero floor lives in the comparison rather than in each test, because unguarded relative error reaches 4.24e+04 on night-time O1D at 1e-35 molec/cm3. Photolysis is compared per reaction rather than summed, since a compensating pair of errors across two reactions survives a total; J was added to the fixture for it. Tier B reports every deviation before failing rather than being parametrised per case, because after 28 minutes the whole table beats the first failure and shows whether a deviation is systematic. Both tiers carry a physical floor -- SO2 consumed, H2SO4 produced, particles formed, plume expanded -- because a tolerance test cannot tell that a run did nothing. Honest limitation: the plan calls Tier A "CI, seconds", but CI does not check out the private submodules, so the model cannot run there and this module skips in CI. Fixing that needs a deploy key and is its own change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * studio: apply the endpoint and series tolerances to the right things Tier B's first real run against the archive failed with three exceedances -- H2SO4 in D2med at 3.377e-12, SO3 and OH in D3high at ~6e-12, all against 1e-12. It was not a reproduction failure: 3.377e-12 is essentially the 3.38e-12 the measurement itself recorded for H2SO4 max-over-time. The harness was applying the ENDPOINT tolerance to whole-SERIES comparisons, and those are two different rows of REFERENCE_TOLERANCES.md -- 1e-12 for final and peak values, 1e-10 for a series maximum. No tolerance was widened; that is what this harness's own failure messages forbid. The two numbers the record already specifies now apply to the two things they describe, through a named assert_headline_matches so the call sites read like the record's rows and the conflation is hard to repeat. Tier A carried the same mistake -- invisible there, because it compares against its own fixture where the deviation is ~0 -- and is fixed too. Tier B now reports every deviation rather than only the exceedances: 27 minutes of compute should produce a measurement, not just a verdict. The D3high series maxima (~6e-12, comfortably inside 1e-10) are data the original two-case measurement did not have. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 62 +++++- docs/studio/plan/PHASE_0.md | 2 +- studio/tests/golden/__init__.py | 0 .../golden/fixtures/tier_a_short_run.npz | Bin 0 -> 41265 bytes studio/tests/golden/make_fixture.py | 158 ++++++++++++++ studio/tests/golden/paper_cases.py | 133 ++++++++++++ studio/tests/golden/test_tier_a_short_run.py | 183 ++++++++++++++++ studio/tests/golden/test_tier_b_archive.py | 199 ++++++++++++++++++ studio/tests/golden/tolerances.py | 173 +++++++++++++++ 9 files changed, 908 insertions(+), 2 deletions(-) create mode 100644 studio/tests/golden/__init__.py create mode 100644 studio/tests/golden/fixtures/tier_a_short_run.npz create mode 100644 studio/tests/golden/make_fixture.py create mode 100644 studio/tests/golden/paper_cases.py create mode 100644 studio/tests/golden/test_tier_a_short_run.py create mode 100644 studio/tests/golden/test_tier_b_archive.py create mode 100644 studio/tests/golden/tolerances.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index dce1bd9..352cfc8 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -20,7 +20,7 @@ with full provenance, and the golden tests pass. | 0.4 `studio/modelio` seam + `RunSummary` | **done** (#68) | | 0.5 `studio/science` derivations | **done** (#64) | | 0.6 `studio/runner` + job lifecycle | **done** (#72) | -| 0.7 Golden-file harness (two tiers) | **in progress** — reference deviation measured (#70); assertions not yet written | +| 0.7 Golden-file harness (two tiers) | **done** (#70 measured, #79 asserted) | | 0.8 Four contained fixes in `coupled/` | not started | | 0.9 Vertical slice: CLI + API + minimal UI | not started | @@ -29,6 +29,66 @@ derivations to resolve rather than fixtures. --- +### 2026-08-14 — Task 0.7 (second half): the two-tier golden harness + +The assertions, built on the tolerances #70 measured and #76 corrected. `studio/tests/golden/`: +`tolerances.py`, `paper_cases.py`, `make_fixture.py`, and one test module per tier. 17 new Tier-A +tests (214 total) plus 3 Tier-B tests. + +**Tier B's first real run found a bug in the harness, and it was mine.** Three exceedances — +`D2med/H2SO4 3.377e-12`, `D3high/SO3 5.995e-12`, `D3high/OH 5.535e-12` — all against `1e-12`. Not a +reproduction failure: `3.377e-12` is essentially the **3.38e-12 the measurement itself recorded** for +H₂SO₄ max-over-time. The harness applied the *endpoint* tolerance to whole-*series* comparisons, which +are two different rows of the record (`1e-12` for final/peak, `1e-10` for a series maximum). + +No tolerance was widened — that is what this harness's own failure messages forbid. The two numbers +the record already specifies are now applied to the two things they describe, via a named +`assert_headline_matches` so the call sites read like the record's rows. Tier A had the same +conflation, invisible there because it compares against its own fixture where the deviation is ~0. + +Tier B also now reports **every** deviation rather than only the exceedances: 27 minutes of compute +should produce a measurement, not a verdict. The D3high series maxima (~6e-12, inside `1e-10`) are +new data the original two-case measurement did not have. + +**Tier A — 19 s, a real run against a committed fixture.** 1 day, 40 bins: the cheapest run that +still exercises gas chemistry, TUV-x photolysis, all three microphysics processes and dilution. The +fixture is a **uniform-stride** reduction (every 4th sample plus the last — 38 of 147, 40 kB) because +a coarsening grid aliases the morning number spike by up to 8×, and a fixture built on one would +encode the aliasing and then assert it forever. It catches drift in *Studio's own* pipeline, which is +a different claim from reproducing the archive. + +**Tier B — six curated 10-day cases against the archive**, ~28 min, nightly/manual. D1/D2/D3/burst × +sabr220/sabr330, all `cg1` deliberately: `REFERENCE_TOLERANCES.md` records that cg0p5/cg2 may +straddle the tomas-jax commit that wired `coag_kernel_scale` through, so adopting one needs its own +measurement first. + +**Decisions** + +- **The tolerances live in one module, each citing its measurement**, and the failure messages say + *re-measure, do not widen*. A tolerance widened to make a test pass is a test that no longer tests + anything; putting the provenance at the point of failure is the cheapest defence against that. +- **The near-zero floor is in the comparison, not in each test.** Unguarded relative error reaches + 4.24e+04 on night-time `O1D` at 1e-35 molec cm⁻³; comparing only samples above 1e-6 × a series' own + peak is what makes the comparison mean anything, and `O1D`/`O` are excluded outright. +- **Photolysis is compared per reaction, not summed.** A compensating pair of errors across two + reactions survives a total. J was added to the fixture for this — at 1.09e-13 measured it is the + most reproducible part of the pipeline, so drift there is signal rather than noise. +- **Tier B reports every deviation before failing**, and is not parametrised per case: after 28 + minutes of compute, the whole table is worth much more than the first failure, and it shows whether + a deviation is systematic or specific to one regime. +- **Both tiers carry a physical floor** — SO₂ consumed, H₂SO₄ produced, particles formed, plume + expanded. A tolerance-based test cannot tell that a run did nothing at all. +- `paper_cases.py` maps a case ID back to a `RunConfig` by parsing the ensemble's own token + convention. It lives under `studio/tests/` rather than in `studio/`: Tier B needs the axes as test + data, which is not the same as needing a preset library (task 0.2 deferred that deliberately). + +**Honest limitation, and it undercuts the plan's wording.** The plan calls Tier A "CI, seconds", but +CI does not check out the private submodules, so the model cannot run there — in CI this module +**skips**, and Tier A there remains the pure schema/units/DAG/hash/expansion tests. Fixing it means +giving CI a deploy key, which is its own change. Recorded rather than papered over. + +--- + ### 2026-08-14 — `main` merged into `studio/dev`; the stop condition moves to the diagnostics dict `studio/dev` now has #73's four model fixes. Studio's whole suite (196 Tier-A tests, equivalence diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index d13087c..a6b8b01 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -125,7 +125,7 @@ input file. `run_coupled` prints rather than logs, so stdout is captured as the --- -## 0.7 — Golden-file harness · *do this early* +## 0.7 — Golden-file harness · *done* (measurement #70/#74, harness #79) **First task is a measurement, not an assertion.** Re-run two archived cases at today's submodule SHAs and record the observed per-quantity deviation in diff --git a/studio/tests/golden/__init__.py b/studio/tests/golden/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/studio/tests/golden/fixtures/tier_a_short_run.npz b/studio/tests/golden/fixtures/tier_a_short_run.npz new file mode 100644 index 0000000000000000000000000000000000000000..e61f4330ec935f2fe80c7f59a6d77d73313e7f39 GIT binary patch literal 41265 zcmb@sV~nQFxBuI=ZTEEdv~5k>wx?~|p5{Gmd)l^b+veT2d1jxzpX9%v5-=!!64nYqB0s`~*BL&Ge)ROu7pMe5`0b*_H;9_8DXJTn=>dav0;0X$X z0#XC|cj@1e{ofNv)<`Wps1nHVjSAGLArlUQz2emZK1P2bK0-X`rjSzPIDb~##=~X& z4_pMO;OFqH*H1{l@#5_#)+~h%(_S8JbKGMWYZC=Y_bW-t0|t(?8|1SUh}Y$Czr@py z;=G*W_jN*v#>AG$FO4QmR2j|Q?@KkYh!`uP_?S(#QAlko!=SMa(L_@YfwUWj0i@!o zhGA44JBU}$Jyc!SF$RkND?Vy4p5gy9K05;!OIuTC12b0}n}6c_^*?cm|IeIpj(v)c zGCHC-q*$39ubrv;$F4BfF2ORdsvtKz*+5k}F%>(-Nj2zk>2fir?RKP(o3C_LAYAyvMxc^X|oGa3dW7DLS_+E0Wbu|CK}_?ZGi z&l!0QKnWOMV(ApWpqY)Fyvu;OBZJ(_z&SJn!hhwUz2%1S|IC5MKRFPh{qNu#3IBVp zT=C>mo&AEoW>%FXMU_hAmq9C}a~5chr=#RmdZZwGQ(rnK)2T~TTnag-%%e-Hi=iBg z_odaTTeSDcuJ)U*?)viZ1Lk;Mw%%p8-laQFbLZeYs_RKGEUFHQ8Ug}6+(YySAXe~s!V9E(qYZ?n+y zHHU!YbH{y82QM%sIqR49t!a{TM^%NLuP`POqKdeAj*a%C=u2I1%W+h_M0&fX%g|ItXqmzw}h z7U?~oD5}Q@0H5f4y$DU^iR$D#D+@$>+Py7B4tzXksbGRnMSJ{(0?~}nD`E;{Di2Yh z+9k@Gf3xey>zQWMvQL%beYXn#L$gAST`t6j)K+SAkPaKq z!80F@1&5(FjdTnk1cwf`DYb_i9?4smw#9Ty5ai55b=QQ=z~t=m#WM-rVHoV^Q~^9| zWc?w^#s)HcfuRy#FX0_@Ev|gN?%1OeiCf0e5g$BOjKp1ctNd^R=iK{Li>pn3SxbnB z)&f?9-ue!o{pN?+2$wXYR8qTBTWZ9-dD%}nR9pEU7~07NS-^nD9(X~0Bgc};&#`g> zECra9WA-5v}?aW-=!Bq8hys{9pd>n?M#uU zniD7FPJ?6GHCn#R2XP+wK}|Nd2hG#qTH&dPmZ&@kw}e%Di1IU->(E=jhzyu}ZkcRS z3gXew6B4ly@tQ1mk_-!K|wWwwq9q3Ni!wB4QtK!8jF6rI6H(dyC8&H4#Ch-1+ zwdi}4&)P^ziOKk8Lzh}3_L3KI#BA--JKAYz{$bV!pSC{>M?q|URC2FCkz){{|8nUl zAW0E3M5efpg|S_@$W8J#Om+K`!k&{g#%+}NbzR&32O~$~R%x4|ui^CWg6L^BJC)I* zm^a2TxRBCKf=tZn^EmFbCb%Gh#<11cwd1Z zH{7!vO-IfZ2IgkB2ipSJPx|OI-(IL07hIeYe4<^62Bx!!MMw1y_Pu4QM1ra85i8lj z2%OItXgH}tO6G&^AdY6XBJZReIl?G&iZ^HqvME2*9Mce9G9u`KQ2LR)-3%M`*!S<0CnDC&$cJX$Dr5zOh&0hUK`K3;eQ3()pt1kSyjlquRK`I>^K zDRFw|_2q?(T8oIYu1%LRzRixrtU3ys?!t?c1h>3PLRdZ&M6yXdyOM~#JXY>A(~=tTV4});?j2Fz zqIAT0<2rPl`+=43^XIdylaIk&+SF_v;d?kB9jBN&?w|n8`n>+MpLSZT(cA_vl%FTt z#Q^}VIe<}4FG%?kVu^()ME>fELvvr4e~;t zy7I&;2bRES3t6BlHYa5nhC7tUqC581y8P*iRDd*AgS!&0<+~sin|G&h;SW`70TVU^ zE3vQZRjgXsVX0e3%P;B9oCD|IC(m@1;M{x(+n7^5hoFr*eETl_uc)y;Cf$j$h_{{Z zOPsS`Tqcv#A;=-gC<2WFQ^yLG!XF#erZva#_>VVxpCnrQ=3Uv#thsHk3**ZkBOoO0;OZVUE?%JHIp7s*Kdicg$TNKO19Oc( zs>?7~=;$}&orX_8Jy$r9fz!y#>GI2Ik32s{(n|8HXoE|N?}V-t^@zQmqk`9lv| zuQkR2|6s`Xv;t>-K13TA0a!qEscK))d)?rWg%)@?QrUpNv58X2uTmN=gW2>Fmtnwx8Y&X>7E)E$c}`dBTdPd7rp#|;&KJ+_LCXFc%6A(Zdc zp%dB54?fzVttIF9fKg-Ki@dJL?a=fmTk5@|INi<}Blz;h)D8L0vvbMj${g)^()Jql zsTWoE4$Kf!r)N}6yaqpn`4X{<)lG1`Rsu_Z40ySZLL0aCFX!v@a|@FMy(x|miKCD) z^iL~E?4&rVqaa&KeBkQ%o80RJqgd-lE*|SFY#(BFfv13;zMKaWsfZdHJ$3uUaihe@ zxa~MSu?1We++i?mV9vN@^sJO|1*2j5F8PQqP*>R`y{(%IPp_LWB_p)sauy_s-QrTI z>kVP4lJ3IEZLDY2?YMsJ@NemelxW)7>-vAfjY(8?S958DPsZu&$BMH;TtJL!=+m2^ z6k984)-K!j3vAF~X8RV~R|5+|oT>Z?V1@(+Ce$)5DTJmAY4@LUoP;R)VcW7hr^XTQwIGN!Xm!!E*p>>{^Z3YIj28p z5|{3ms+trTJ%Ej53gF+5DyJPX<$^D>Y(pEie;H~xpD=wbDo*V;T04)5flH%alzy>c6g z0}|d13JOFw2SMs1K^Vz_O345ea0A8e`oUUZNF81Z;~nvIcYHV6#m=X&2U3cutjtWH z-DS=bSy%Dm$crg<6Rq(yZ#3F1WKjZvT7Om%Pem7Gx@~G`T0r*%%|4u{Vu)Si)={4g zJdWr0wC|KGY~B3It^qgow%+lfthjDhH9U~ww~lIck#jRmA<(zJ+LV}C$Jv3yz&A|t z<3VU#bAoInH~bD)ua0qa7mEezgzY%;*(crC(;WAn`_Hqwi1q@`^M3IJ@0Wpg!6)p8 zn2C_9L*dl#N{Lw)jqDVnxFvO6f)GINFvO8T-8{iD_8$8xd_AWJG~Y2hNfUmf!-mYXE93G2z2s>HhEF0_{=* z5JoKQKM6Wm7#t>izY>$;a1I~h*FYZ z!H`i7l%=3ZunK`Wgo<$+cAZ!3J_xzj6#;)&+}-1iQ0tjZD8kz)%>cE3ak?v;PMF%W z4Fi5YI@+4=^Hhk3Ax^d(v$6mAsVXf18b9UMQ{gq+ks0%gLoADhSD7;s$wUe(^z0mB zc;n^y6udb%zpc=?%Lg8<4Vz=7I|VnrOoC(8fztN<1u*a_m!wB}_9Ei=xn5QUX;erx z_jB9DvsYK6Gw|05Y6;WtM=zXxup2NZl{{XrS=`OpE3F?$J@F+xnq2qN7z3@@7yNS3`~v5_Mp+m&MA%aw>i@Ygejl@e?x4{C_k>5D1OI8=bf z26BgeuhPn?B0!6~J)mPw57sC{pQ5bcOQ)z9e1rbz!L_G3w<}yc@D-@Cdjvy|-IW6a z$rAW>@QeFb7mfBzWFD$pq#GPQUqwK&oQ|mqdlQH${%|@#Qm~3g?7=<2<8mBDokZTvx6GDXeXYa)H-a3GBmR)R5ZRBDZ5j)b1LP%jUBt##OULY zC(1R9K*^HEb1k%wM7v)Si&Bh85FRF#-1ljsws z{4?a}9*OOTE1!7rj3OxCcs6Z1Cr5DSyH=!H&Y793w?Ns~MgO!jooZklLVp4*z|u!@ z@IL3yc)~2Vh;ZjkIIW7@h*;XG>g=ZIm<``%_ z$~#6NIq;0NS024nR`#_Ke zh(MxKEt{|yvhbM60~BWud*-svrdwy~8_UwqvHZ}v9&pY8*sXuNR|_{7ZujMESzC3B z;6AWf38(D1lg?fsCPv!z^bQ3ErpCML(Xm_VB{CMmVz2sOygpfGtFNAerGqACycc-Q zKEOZ9*f}B@?r>?_RY`~{zY-1e*sAmjP;e|&MZtjWJ%;+)430>I-IfgZinnA{&Xsgj zqb*##U$7JJ2PU-Rb)VK--r87aJZ=m0zl&t?)N`FL_7j#iUa2tCzUpHRRm(ffc&Ii{ z6vHa5tb{83iOE0LzS-Akx(ES^W6kS}YzYazFn<=3a$9!d=qg5-29o4^=#ZFhh54dA zDO|pkybcdPiCm?O<_b-eq-{EIb3r+UY?(akp@p9w!l?6#n3t@pxv0x_U?bKcivVe$ zTfWx2o`u*@)9{*Y*3+}#-uYq~3k6U~OABiS=({iw)uqf^SvZ4!@nDLy*ihQoYaOsj zzr={f)>Xc|cX->;oc3)CSVC$csr7SJh`ozpM+=v?QL5UVW^8$)T4S?qdD= zi7WQq@iDZQk2m+C*Ew5=g%O8^cUX=OAd)AI(S88$7N|H2_``8$j{o@bW4+pUejflR zR!q|T3KgQ$LqmAA`AIt3F;jGSZDeT6^LsYy3}WLdI>qOQ*9_Kp-*hkTqajd--q;WD zE7m~%2~kTv6gJ%8k!mesM>QLprB^F##g*+W^BX?Mbn``fgB&FcuC56kP)|snGiX%Z z*Mj|doAva=_-jML-lY^e{G=v?_pM825G&2P+B5z%95w37H-($a-%_MI&i()CLtt@(8K#%MBtzu0i>}yLD7dYS8V!$b8e0P!$SpO!_0Si2v?3g--s@61ARPkA}>5NQ^bYe?CP$T zEyIvMn%8HK;pNefX?XOl3oil^Xz1dR$>2H8Md&5&G#%e^K6X!(&0ol)Ks7h(>nZK8 zhWhm;uNs=eAOofYWVuw9L2(fNm}^0{D`Jjkz$OJZk!ae|Zxhc1HFP1i>DkK%L+=-< z2Z@q}@D=m5*Z%>#&?e1*<=X~ft#Wrwdr}3RL=RlzBoH|kbH%k=1V#@`O1Yfa=M0XSibQJ85^Nq65YZZ9&VJ`IuW z4q7E91rbr2tkm(CU*>~$26ppLqC)W>^QI$t6Kx-d%~uCNk`|#1`PMkYFo65Qui6Sj za`HsKm{+KHUe>T)Wl!{>nws@cXzDByC@ zr#=Yh4kl&c5$Yho&z3T$4NQs+R~7jTA7wk&O$^x=iUN}0oWtf@AL8!5_X5opsU_Uc z(BU0#u5F^FE^lY{)DEkz#OHFfP;K6d7l;+ncILCMi+X>uu+T=#@U(_sL*u`FMw&1# z38)*v_>(c{LDR8REz_r3jq{Li8pS#Nh_c0>t`>h)`mjU6hdq6=fO|rSKN>&ZSn(+y z`ZuOFGpyAh6Q?*Qr9J1o@_Y9jNar|$h!-|akS$5x-n2E>S%#^8ZEikkY79fXdzYL| z=tcg&0+BI&H-5(qxPN}gA-Jszi(IQy)S&Mj}t z9GbgD^$@o&p0O?C5lFxHngr5{th77{4k3SJ!lbaCe4^ca-JZdW79Sa9t{q#`UVq4* z8|0n|F)}|$b4JjFOkA!~9J%IN5s`8hlb z46*xcajO~DY;jg?6pL~wd!5#;)Gu@kD?gG-nEx8+P`F@~Ml+n(>sB2mp~JgErMP%Q zF;nlmZCcMED41>Ry!`}t;Z`P3t*;Jd6TCt)N+y2^@ngL6O11%i15EiOWP)m1u$=c$ zpzNGCo>ksRN#+RZ(4IiTYqEyl4cD`zc<8G6f{oKk4$2VwSNZJ=tQSp8K>;_krg^>8 zlMv=049(E#x7Lzxe!B|KJF%MbyC;Sf1MQ{aw!PzsMyU62iDx&7i6>`wj765PFY(0qhm zq@2ll3wY4eIFm^9#YRFmUQ^E(xpnOOh)mwAhjtecG?VAh`t{b(lA9!a8-dEPQ{M$} z_|AtY`pK-?LTSVC16L!zY1~1$(+6^Mi=#P%)^lG#Fj~&d#vi{i`)dG2BKYk9u;|+n zkLk@tq};jcJmfJsgRViC*mwdFOmFLr{AW)Iw^c?9#UvUxq!So^kJ~1Kve!d>rcHB!M)d) z_{0_C`K-flrrEUA_Fb6KHk_Bv(HngroR0SV-42fpx;w`H(Tt)Yaz3O;b`PHI^)6+( zV{5is7=CHgg}IBp?1u<}pGL>I)+4s6U*wc5m!713%VKjSIA%gKiOx@_?*-+w*xUn> z$XvS-0VcvUxmg)chQ*f9@Z+O7q(A07+VrNEeA)*8L}z{N&R@^nz1;{9U~PZY%>^IS zE1vM{vL7ML>Lmt>v=tkbzYY}iu*FZC8ExadLc^HjR%Wl$w-a%?5hfN$kHSH4LJEL2@JiK|~g==v2z>lb1C zc|GGj?6L-LrrYK%{#JNBme6P|aCi4CdXJow(WVj1i2Qr$3DojYGJbjr^A^4#$c`lY ztDe06fi07%Tcq(m^?-GB_Ba@*4BD zk$=kHx{%tE-HlSgG)%D%zy)huL9gceg$S^%>h>bSbh2gU;7^HQi|6H+pBYDweB41a zZtYM+(VH68u)&qY6kSm}o8Ns;^GyPy$mv?EOpoH32vFfAW1FT$bgRfoE(l%^px5Xi z7tn$G`guK`KAjeb&`kez?Ps;DJpQ~8d2<|2QrYLS$tX?ObdCpnCD}=ZlM|7M?YhsmYa)5XULajbetQvx z2~Dwx^AT6_@Up|^Nj20EEfUbzJzr=&cDKfrk`L7ti0^dF(X&Ou@B%6?-O+LeU7}`u z)DW|mMNKR?8Vaag+cZbPMOHSk0$d1MoYkO91Qk6GwEZ;f<}Z!01PIj#`S zEb`En(E=io2X}NTsQWQL!823hNo||i-rxCtn17#xTFcK4r=ODE)p6TdI(2_Z=GfZs zf3|23IboICW5Oy!PTBtLvvuHic|`LVV4iKuBAejka5#E{?@}(X@=S<~9S@i|$+S<* znN?`Zo#Euv$J|ceJosr^7cK9#yk#1-)&e*|x%q&P6`VGY`jNuLne=Ql7vkA9QL9P) z-RCQ-Nf(mPbQgY*lV9u${D14+IsV~I|7g8XfAww^;eW*5e+2Kq|LH3De^A^1{|MfH z1$Tq;ke^kn#kTW6$F-sJ8j^i=A$P%Qk>PZcFl&t$0=~pYwdJ~NZr#<>d6EOtotnFw z*g0oGNP^7fDXsQ&y&0q4*vo?q5i#SUqZt1*q_w3S70$u$l90C-2msAH~KIfM(GU)6hyUlSh zN~Agks>YX&?~cEf+V8Vk%NK;VeH$0Y#tV_cfhX-{*)*<-EBmB-wYdwV8%F^JjT9LBltADB~I}w<4-!e^QC0;ObTaFt%+-V`9SvPTx zVw!UQyHSBLHB$N?aEO1wIT@K)x;h&e+y0wh=zoqM!%*>hwH(0Dlq!FL{zSWZJ2t{} z)heT1=fX>mU;Y%*zL6D)K8Yr2#ca58mFW6r4~utxnx@nhROh_u>t17;X6xsZaamZw zem1=uvj6!F(5q|B&h>B2 z$M`t|&OnD{y^gJh@w81Rr&X|Ce8A-QkDW<3&*^r~#6i3Ibw+){wwq`9_Rnfi>^r*0 zHTt#g{jcF@a5r3Njfr1(JSN3|pW(CuH&A&Cr)AWt+6n4a;OxxSmtFhhO$)wO!H4Bn zU%2iy{d~NJVprXjXV&iHv^*>dSKBxc9?Mc(;#e!mE&MgBlse?Sph|&)F*R zXc@ulW3hHYB(E6z2F6vb^Dbwyyr+1=uge$vbOXIQ6`Hga9+CIDe>Ympy_jbI!w~*2 zLkkvXdp386e-l*u4-!P*E=A^Sg5Xl+eYyfIuReI+Jvc(mbT`|4#0Bc?WisfrOuQ(yFBAgA?SNxRH7%lG;f7k(>t;D+UFIzLs|bG4d^>a2-U z=^5p4jyZET7N*K-7G~L8`yo%2Ch&*Oi}v8a)%a`Z(#eP_cxUikd^jYFsB^T;@0$5NXHI+bP7J5W zw;mVMNk395{7@=>e05lE(0_5HMw#LLhvR>>ajE{Bo&0~YjSIzL#0jS`q5$-NpRHeT zjm`f!@n7YwZs26(VrlV`)-Jrl-Av4NQ4-}kq(d1IFIAV*?LUQs>J4+=nfhBZ}jZC@0{BHM+S+;NL?+C5Wc*8T}?O>;U7qlO@2`}t}w_oW_ zzqmGWUu^W+9$sa@R0CM>-xbk@xy2b`8mV-$2^>QN_$k($qy=Uw*x%cXkYpk~- zDff;Q%2)$=R0ZL`>VAIdhX2-pJIzK0^8EXRmHplb{(s_gv3D`DF|hkLzTp2HzN$~x zdA5pWC*Fw!JM9~zZwdP$f-(cyP4&F;Wjt}?{xPdJLlg6yQ&KEkQ<>=rx`x-&Mu$LW z_l$Bs-t)GHiT9H=VkCAnXP+G1*P|W`e)Dz^&gz24gz?bBwx!s@*6>a6qcz^Ht~E0^ zU}1rALLksKyk0&wJR0Dmmw$9xa_IRKzt4{?G5bp!_d}~H7SgA<%FfK$IdkX;PdVxA z@|jc8JtiPMV?PRW&z~hhkB}IPgXzP{)lK!pzvR@hr?<9`NiSD>tXE0u!O#FTBdI{u zWWNc29FcIFU)x}!pyYDo%U3>zyZVo@rszxzLSr|QnXlkxe-IImgCrf6;`i>o8MfUR z?VZvnBxkHE&*t|v&+sW;N3|q{X)aBx5IzqT-?6~LRR(?yjv+Rrq-myV zq9Y^R)ytt_@gv`8r64nM!eP;$h}RfP3-HQ)G55jpL{+;uO7zt-LiPHNJEc4bD2IYMe#n%+%K}W=LW0zTKFp~f}AhgckSZpNZ(eR zp;^gw(9Z?COAcJ+nT-O^GOxJ4as=VJd&;vT$YNQR&9AQbb6#jvR>YO1y;X2X%cBa^ zEpEiOI(1Qh@vQ?EYBmoVa5_S|YEu!O9mio}uxfIhQKdxru}cN9(Gd&E31+2bfh7UC z^&9hL%rw_4bblY@&R7izNPWjHuHd`AXy|FZvcM)<-;)-*G=c#RWH2` zac1D!&wf5RAsttm!A*7Rfk?nEiM{h_S_>EwtNz@(bxhZLtyx zsdAtm>$BNC|G9vT6@YhRi#P#APARG?G|v}O$aV=iB4w`mO!8A`X@aKbNXBlkjS<4l zCsyWLnh`>D;d<6rnJaj0b@v#1N=NAuQaGhhrV2SbR~}0w;k;E;*d^DHO_+!}zv#p! zC}KJFS-$+Pf;Og06x(I<$^PCH?R{48`*yrJ{B3&hz7}bdS~}(7C(RCs3$WGWsJ1LvfSxeSD@FtzzRjufJ~tu_{B0 z^$#!>l!~Qbm1)9$kFGJjc^C}Gv3s?0lMntjXei-@O1$UVWb!wnHMC}(2PJopE|dkk z=#|H9GJK)Lk!=5wt4fE1fT-hF0f3qs@SH=XTZG&zU_O;^#} zw)PBS=mxf#5p7COTPv^o$HtS%+K+DsCpQUbSjJgTG#oWs$B!Luv^!F#J zSS$8#8Os-HI9}5>Tf^1HC3R(y${8}=%Caaxr{47-0pIQzEr>fdmC2Q$Gic*_4RnfT zj=eH-ci#|&Fm-8==Nw^IS7nm}u zQb49=llwYQ#dFOfcJwx&i5*_ZFDA+dO==3&F&X@d(NZ;_G*|3h0)xf@E8SGFpimd= zR{RqRV;>8mzz;AB3&N(`(=141t;rkCuV|3ZD8h5%5T1a3Qaz_+AzYXapRb4jGqP)| zx<{3oM~FKCIe%HThaOH!LD5BF44mN<%#IlnDB{%-HNteS8Owe8@PjRwhV|asRS^a7 zN>797Qc($ja=K>WBOwQA$mf{%8(XQQwY`&pw=o>r@X>_Q;_sk8kf!UehKr~hc1%O3 z2`IoX80$a$A`x`z(~d*@j;N6i{!I zy=3?e2tXm`G>CX}eBnAcKfbpZcNYwwanR>XLMe8%hKtjpb5o<3C4Y!8g2po7wfkeU zfFa{M!}Hp|1k5HO3EbF^8uHu+^>WEjO8Vnvv{y8zd*^&l{ zs0*?e5!8(kZr(fMwV&qwMovOzndo}s2F+BYz41Y<4<)WC64Hu*)*EjgQ_^d#;1F?` za_LeD;3AZqy{D<`~{~ezTmqRe0 z!akGKY!u`+LNJPzY1W`pfAY0$JEVL+sw{)spivvpS_$^`!ZIu@#A*>AbAgOY!bN@gA+sD$5XTLeaJ$)v8 zGpXB$uV>K0L-R0fxnG+~ShP5@@rOTc zXYo&{uJ)m|u{U!fzqq$ac}F8SG{`PPMH7o4QKhPnk-yJqnGU&lwP8QT$he!9d%=4` z+B(i2M)?gjUz^60OyFFZ^q9{g1>nKh#cXUM2S5Rm16d>@A|!L~8kCdU_yceF&d+6( z;CoTue%PPVf^=_uv>XpGD6%_m^d5 zMj&O<6}RcMY=2M$ug~XqJ*GRa^k4JLL_@&M=cma#ns=A7JRM;n2*UWvAI)e!J^nfb z5g%Y7LRB?;oU3RwFfz;3`hqCnfvqZo_~THy&mAXZdAXdat24?x!&wNA-!0B)`oH(k zoVe&!x??+-YU$Xx;eqS$b&U%oI~$UpEGH@+fQ6$z@b734R((5Wwalfw8zcNcN2g%0k;PKuwE zN5ppN8)G)}10r4Hvz2vC96YJ*U4M@cn}007Q`94=1ecju?W$m~gUc$k%i5ShD&97x zLUR;zNQx7A{`ZXY;S{(gJ%Q}5dj*es;!w9y3@8O%mkfW6Lo1z)fDIVYXqCTjGcTy; zk456@^!vUgcPq~G0eGY9m68pP7uuEk&%EE#_~aB;l;3fDGeG}97caFAz4X{mEiw-w zD9?1HPu|REAw-|V>(Q&Qg2+#BtcCk`1S|>58juRJ?C7(0{5ewu{b9>lb+qCVXlYt4 zIL~36+jSD+zw#5QCPP6&UV3TqI-Wy&p zTj~3TJFmh(qRh`D|8Wt3)A@h{3pywHJT7)d_D_8!+3flprj#^JLC(e6CT<)!c%vBQ zF>{li?;AxN5&>#rZ$y4T5=2KLKBh|st6;uTckrZcK#%CIPt;Bxv!qPG-FeFbbjEy2 zdCT{1;SapX_Ty6MKf+6=YHd}0Qwm}xS3o!rxo9}{a(J*Rhmo+`$M~SBF}^J}Vdj~W z%E~TDLl9MqqW$bfPf*^IgW!-x3KAz>_G~*=8L2~n-O!NM7CDwJ&7c6j|C+%2)YpV15dMUbSXE zZ)>ZO>D_Q4i|YVMzx^&v zl${3txi67@aZEh|MDE!O!!#e`!t@4Ikjpg8v>rc1lGI0|5hHts$#Kacc&=uaV>5v6K1ZUk9AsOWi&l* zaQ7IWHdTV~4Qd>NL!%pd0qVVWZo80Cv2X2$ks%n{`m;j^g;Kp2nRpDgM&4$^Pj96) ztemy98~b1?7M|?hkX55G(Q(MZ2P2RuNw_^UOC8H zEF*D8j?;8$yrNs!>L^PL&5vMa!O39S3**Ff~Y4ZCFW6x7S&{V zC)o{ayvO6_XJ584RSO!c_VdmL4vL=ISAj{;3;zk|p&GoQr&s&%#Swu+L`8$+2N#?y z?(~-Y)-j0KV6AuvxoHo_*158QIoZxjE0=WJIjruMm1<<>JaiLnvBE~N80czsY7oG4 zz|bi@K`f#Hn|`v{r>qw%;@kH^$RrBuTt@{gd;IUS1^se5MoHM|afP&*{GS*e6jX`l zc6g*jj{b5DZJzOAx_p|kJmFbUiEQ=U{HfKCNvC1_6O)N7akE6_t|%v@2^X%qhE3ka zcG#>)DU}RlvBg6#5`5UxtQ7r~^`1&0mN~tspLblU^edr*a0Y|le2Vf;uKdqtrzmU6 z*j`;<44j{dTN``(W(5sDGxSrO7!fEuQk5dv%7Xk6wp#$D+d(orft~wjUjeasH;hHR zg#jlyvr!u2ggf19d$8#tNY_YQc4-T2 zuTU;Q`aha6EtHXt%005Rwh5gExG*yc<>nDxW0f%UxvjPeM4%2UUa`2y?E>JqeoRcu51 zSg=XpB^^fwpDC)lAjXft9#xl9FCKXuwaPyzyH${jNpliC9E zD(=h0mf&(PQ1!(!Qy^A?r%`nYZoxk2=SnS7K{e^Dzw$OIp^i_z;ey|#KrqwNz{J!U zwMlw0d>uK^*friX0ymzTSmMm5xgcxtlQDM;57K0wqzebIU^--7$)xZ zO^3*rU#Q+hYdtFt+?X*2j}*x!#hF3wq69t_7UAz$&*g3%7tsd~S93BV7EJpclXX*? zo$?3xeIpZ05d=cb<5Igj|A-Anx!ZNHAb4XqYCGR!jsB(&|@yw033#+jb3}x03GunoWu$|`&za&mQ z4n^dpZX{_Bk%@gLrsqdSG<4RVnsW=_ZfrNRw`-*JxE5!vi6X<1)k#3%nKYvst`f#s=%kJO+;x12 z)RUoHyF2Hg=>b0Ae3W&$Y*g=>bS2=8iF75|=NEy8+p$xhmSO=tQ*vbQ!mPhkS^k{+ zu_6o==D^vFyQNb9cr%#6 zhwF-;W^Yi;v2%IG&)6y z$G7*4b-pY!%KXX;EG5`9c;;)%YN^&V+N1vkHx%I&MC(omlT=cyCrNtP-T4sa+bSC@i& zzDvm78_q*s3E7QA*B1`}9^9`3wwrrKyUVgAs;9xSK4#Z3Dn3f^{8{5nO)z+d?w{r! zAnEUAZqmnok9j6$yd+gDsk8Q>>cGA5&Bx`oAx*d0@!cv}R$2~%04mxL`8^##0mnUQ zdMXoF-Tal;Z>ifwh4gmj#OsVjGG;WxXk4_h2dd$447xc`sy6Ngy@O215Pn^<0oE9< zdwiJdL3imFf>jHQBD#5}Zv9u5Eu89|;sU%7)>c{>7;@hv+8zaioZY+hM3#rSm%FD=$Yl!uH_^LF9oX=@)%k zB&GFF!Z}8;d_;`mG zn=+K-WmC6rqoo~bj+W^KM^l|N;5doy-%|6k;h+4xXnQRZ@Z9b1#U#RaXc~&pj9ctl zU@_IpSgtG?gn8|9JgI_dJbV~6mlCW|B3BOTKIe`DwK^E4pEA9661E)#>kUDOEULQH z>+nO!8wj*~l?ja~u1nRJ{bkpoV~Hx9KV2#yt-cVv*Q4h^3``27++Mt| zvp+;CclKCwhdrdnFezb?uFz_?cu2qpoqaXNDgE=K^!Xddoo7q%D%+OZ|dvWIZs?QjFwg`YeH%Go;eRvLvq^Hh8?yR;j`NvX;0=oe)!KU8H9VQ+k)3s z<5c2uGxqqX${lXDJ5n#zyFYMDDx{7OTfE-rQmIb8XaU{xAO+IJxKHSLjouze$UeIS zj{SBx9=?}D-zFk=q5Q50=8?)n$a&7s^PaakOdwZlp?cxyTJQ2D8E+j=@t|EsHqu*{+S@+8~_yaNTCODAG4ScU+Ha*Q{L8|B`r^20G zB^{MXcBCIs+T77vK0x@h{B+7tBPT(pA`;c^d=1mOfG6C`&e!LtBukHZ$2x1Q>J#(A ztm=V zu`_kTycJ?D(=QyJFCDbD8r=^A>i+L-iJnr@069q7K>H}7C^)^JKn~{uxx*T2s^E1R zw7q}a3XR^dA^ViAlS@Bjhq0m2`t|3WYu6R zS{!&qj1-n!#69`H0AE0$ztND=1q0t}eHU|Yg50PDopob9e14F;z-?9o`73|^2~KE$ zenXs#J?U*QYodyNa<3VlL4^p<7?#2q582j)r=Q_NlL}fd0VQldY3r7k>;tn#qN}65 zOMpeA!Q{v367UOL{LT1h1~|W5kcn6N30`exI{G)%1KZTK?ryFOK(w~L8yge={Hm8; z5?nL^IwFD$MPj$XMdzWj&+bwJ=kXP~?u7wVE3<9jeqSv5*&tS;vP~ELhQu(`j$S}} zJD;6OkhLNPbq}`Fl{1kSb50k`JJOM%U>enPgLuThpvBr#7~8)(#qa}m?it>c(fNS+ zg-p3!eu(+0`qcU)07+b}6Yr2mBs;FKFL1H1{Va~hSAlYedk-$pJXLO2{RIn$DaE>1 zCg7Qrxe1fCK6nR_&i`@H3P;1_boj*@pasLHuJ(tuaBxuGQ!=&;-oF2Md;eJ#jAVX9 zSLpQvaLgZQf8CDJ@mQ?SlsU8|6vZe6uu-=2*fSE18GO{Y+5J#-zo-bs4Zw z-);KkvkZzi^tT5hXTiPd)XHxc20&xLu6oq#?_j#L?N7nEOu(FQXJo)N5M0%{+3={} z1SlOgd%mQS0q%IE*AxU)U{Jj3p_=6&N@#a|Q`sU8y(zu^^18}X)Jn!amgX@NsyN=l zbWOSyX=%*S`x2Lll>X z#-wya2z^}ga_kWKw!Fcx9z8B^}p!io>v%a4x zD62x{C&y$!viWi*Nj?=|G;TXN^J@_8`=|cWbw3V0!xNSUyPl#fRkMM`6-=n#5rgv= zvsPr$(SKcRDie|Y6BGD_E(6JzCl{{xgWW@m>5pjCQgdGlNlE^^z4H4Cn8*7?|+uhoi?Bp#e~uYvst(?Z{>I*H~*kByL;QSmbP z4^eLqE2@LqJSNO6FY4geIdx0Y^98WPb%31bRR|;rGF|vsqy(A$)OcA|&OjG0YFf72 zzrd|mMUVFy=Rnpi;d|xyK~UKC<1WES10cSBT~Ka53uIS~T%msa0ern&s^Tj40$dco zLPV2<0O{s&o>VnzFl@DOrc7@L#clI`33B-lb)O|5ScH10cxu9S{Uv6Uoi*H#Ijj{C zGf*0=Vah^sg9&Mq1Tqj4<&RDDn|LI*=!oM2CmtC{8jA76=I-&1iv2p~w&~a@v$vCm ztgy!3yhkO1TM4;!&N+<%7q8`pp4T~oyB`KGQ@AZd?v7lEKBiHaF&UZj(W@IiB6jxm z*8Twpq|NMe@7F-NxjPG#FG^s(4=XF*zkDdX`(bE&_bZ&`B6}p@n*~q)B$eJQFNAX! zn0UJkdKQg$%*wh z$YcDs%YArl8B zp^5@qx~~)oi`2kEWKNO$#}NARwY{R@pa0Mz_708s6g_lj;N_saEi>BPDE#76X)BWB zkrB|3%R=rfNZhxU$UweMbQRtCi%0OwUHhc;cw~-=XdwWbyMI*aHzY7W&pXWiFEBqT zA;J*aC|=w}nN1md6csL}NtODl>?UM8x?f{wHwn8e$^_j!x}ltRH_MCg26(4oOg-sh zHB?CPN-p9rff_Ia*UpmSjg4f|8A#w7J~SUY|GL+O(m(!7LlQhg%>psEG{>YbZ!xzyA^M|%?KFhxtt8<( zwl`V``&U|P330L7GtN2}=V8@upwLtCZur$^ijnGV9ekR&M0MO!0?&_#a4SURLnLIu zk;(ZhOrdvF{=k?H{UU@eM)9OV8wB1w6-|bI<(^lYl#-w#`5zlkg>+afXe=E-RRsCw zzdMY6DTCJ*3OD2vi=mb+N4r9MGVB+k?i6WofyBcBb~3@@ux{Q|)V3z{hfz7Ngx~2A@u=jhNnGDeeN^H{CH8sFjHZg+u>G^rio`x`U=K9SLY8jU z#D5maKvsU7p|$G6Ba98(bZwYh2;D?i2>>FbbKBK3xO)$)a+Sb&YJEDKqqZxJ zDeXFZR-g1%=XV#}vB(*Vwk?I(7VYM*{Iejg?eE@&z!Z1|XE<)9mIQSzwtZ#3Cc^Tx zq4>C{1SlN+L$P$}Klmh6KR+oq4i0#RoxEX*gRW+~xzC!DAsg$*8CaSJXTR8+ib??LjK!Y@1=;QSydPPYoi6=H;NZR+yE|#v$l8yZQHx$^s}Bo{dmI zMnL@j!`~y9nt)P6m0VFv4v2YaPZ+}-0%DjrJDCx4V8y8M!(t48p=u{-oeQ+!=arN9 zTc3wf^WeZr#g2INowo7Eb_0F%KVxxyb8;3G*Z5zM5Yy5P}M z2vxq$>^+si!>y{Xhw)ba(Bk5-S@Jt&*f*6|d%fTYa4pzIv-vE5vOR=LGh+m}gdS)U zyEg&5gmHx$kPC7Yj~qsPLx5eDpTm;BISBPV5kEW+fz{&|J8h3?foP~w%Sq`ldjH3E zT<1nSni7)E`sJNI+P8AG?I{NfS|;~&^BQRzGUf7f-{(md()IJ#TJ{<2{jGhaT@sB) zMqb4FeZst!OSdi8F|XIL$NT|UkC;wxJi?kk&il34vIdSxTCq^+%nis(QAN7Ln!v(NFU*u_c z8Xmd{kSKqs4T8x>JSR5W>d;BrLhVsK39MFkdXkp400c)r-#J+t0i}}jSwv+`Kz7=@ zzrZUO9B^8FF02m$`#l6rSq0{RH80}qz5oOvL0$T)Hnbq2>qc!?`!MQr;mNiTNdn5R zlJ&&ygFZ^nagvfEz=Hmc*Q(v2XhXg^=m(l$`=hzA!61Dl9pRlQn@@MgBO43(KI|uH zNbX$Bb!M!`ep3tQv08WCSfG;*}4-RHvK_C*a}xXg<8q zoe=nQLUMzRL?7l$NiYAVA%a_9r0GgG7C?L4CSi}jD4=^>tNh<;6R;6bb7rf~1;4%f zzEo_50L?BFC5C=;5R>9*^5r%JF1ocNDj%`=3ktH=o*YJN7mb&d6@8gT0<2 zfiP2*6=jnSgOj$1L~dmgY?f-c^wsh!j4jjTQTNP+we*#4bt-tcsY`wVBH6xuEz&mq?8^6j-F* zx&Z!~gUUK9X0-}>4jn44MWImJE9OZL%3Up`ix zRx*PR)|f2e3Lx z0`Ebl4jJjR3%ZL;VT)4vvsf($SR_Z2+_UBaWg>kt^LV}BOJ8Sbx9AU$P7n&W5&6_3vL8NP|kOX#tfgpP|pD_K?~07clnx6Dwa= zX1H=*t9yiN3E&^n_TBz(6jb(ICH;7>8Qdk3uw>@Q124ZNhemUU0{Wues!L=RAj$9j zRX$A!hH0A7iYi)=D`CPrcQ}lSd}Aaf6G=c1?7jtQXX~Rn)Z^p75Ek?@!{opKLmN^- z?^4UVnu)NF$Ml-Jrz5*1xBmLl;1T)1Q9n2^Khf|Ti^Z5%u@P5|J=S3Bqni6z#g>=& zfHWa^qBpPd@$dn#^!ez+(Vq-1}u zCcmm4^%NE#AbDwmmQa;y$M7QJ2!EKjk2p)cgSv?hWIncDP}UiqQu=>@bW%l$1qQKD zWGPo2iqQt1SgSW>EFs!$(Q@fcuxfn!01LNDn(Wo$#v(7yS@)scy0 z=wxgpXr&{CAN0o^F*ojV_6I|lQ`AW3iznMsh2b5TunR+@qYP1(!?o z+HTN}psYXsDmus}pfi!EPE)Bqniem45-rDqO0{-ARK3uKSf`CIr(|U!w~HPnmI|jM zb+%(uS1~`+2*QwX%&DeX!&d_90AY;5@dkDtj`9ugotzABP%jUqxL0fi=uR}w zSlj>s-|5?l9$yRxYm=1YOyNJlbAvPR_Y+d+M1XeJd4NAJM^3Om-ZWnFk> z>90kijyYVMAo)>k;{YdgL`Qc0-$K&!qndoBzA$amZzw-50%mLcCoXW91gYzr7|$8u z;cagt)u^r%IL??`=M@zWdFbyuuAA7yrpBCmeti6Jt0O1b(r^{@Tf)~Gd}H8)<7wSsK>b(m1e~R6c+8h ze^jlH79KuIi^cpheg*e6V}9aOFZsS3XCf_VZ}Zv&9!U}&c8tf|_OHaUUBG%yj`e#z z=9RkE$M*rNOYmEM3#?@-3Z20!o+!Vk^uwYqS@fw&biPjDHdI~k6d?_GG#9svh} zrB#WaJHn!uuGPd`qVQ-WrT?w`I*_n7C2}$x1MSO~m3)6UgDWS+E0v)(r*~C6!DBKJ$&vX~83-%JwAhFt_gyqH6arr>Jyq#Wbu-G_=2Mv3_NP z1&^`*A5cpJ1T6pn00;m803iUp!Sjdz|NsC0|Ns9C02BaZPGoFvXGCx=Zg6=401yBG zFaQ7m00003E&u=k0001;*Y`hH@BhGYduC*|{m69o|65wUmhz2PHr zkzYi`QBmX^zlfu!kDrg-&Hp@o>>d91T+`0o*8zX->ul%cfd3bjQ#{9iURvxN|2_Wy z=Z|KD@;S$VW06Wy7uBO_5*oyqu*5xl&K~)b=jB@Q`UArA%(5-_JteB=RM?}qZGc`A zY%ELjPC~7&6|M4}8$fB8R*Kkv(}A=C#jNt%7zpV5{z-z{3#8}ye9m%82CYBm@4e`& z1uh96nVG;asK|eMO;vjx$bVRgx#LF)Q=T{|bZRic!5kjfCvqHci8}5UM>&c$U zH)DrbPtDZiNk&N5`pJJkfEe0E96eweK}0Ybi?5bJ1zVCv`X1lEO{facs~(i?@(fu!WHI;T=C zXq*`+*PR;zq|yv=I;YnG@rE=DjSVUEF*)udLYd%htD(j_+^<5a4|2A0!3fgjH(B=V z&{E*eD@GnhxRyEmxiN$o?wOUr0K6fGk@u>CQ6N%?<}PsAey#m z;zhxO$l3DLf0ivjkV`C^nyF1c5bxzX1UtNSRWoNUZMb@Jyhj;Z|z;yfh z^VmD1ruhxkQ&=yMsQDwe2_t3tKQ$(&K_UA}=9U3rC}83<)Nn5!_!`M{CO1+7D~m@| z+M)euqj~K!c2`;e1GbgR*ddsoByu)>;Q^v-FS|<3CxIld+YM?dHDDwz(BC$42%Kp2 zO22ln2Aub4=eIRTpBIaBeA46IO9nTSZXqZ|&oT=rv;Xxw=_$B__C^xpQgq)pZVBCvMlcf-LjvfbnU z@A+Auz+EtglCa5oR8=>>B#V0lCvRkSK{A)beUz zi)Jwmphu5^swx1)44)Ys>fM2RC|r2ulmwPcST6i0Sp)bT2eLKvhCnqpNAk7lH4xuj ztK%n63fE4M)nigjki%W7fC%^7`_#{<_=5|UZKeJm3}S~`4?Vwao@0cgb{n&MNyM=F zSgy`fbOnq#dZZrMN}`%kTu0M&S&ocAIb zP?NifeUxiQJ>txk{iY*PUPM$%HCz)No4c{%{ptYGVSDh#wPOUS=5S|del~)XX`Q^e z6+MEuxhn0v$E*7i8b{;JnO*Yt!CPI>eBT~#jH9{a&on+v!ap->Y>v7Kw+<@1jsm>4O%MxK^X zD$WE`{?IJO%5p%mpN`fw!(8yiX*$Re%nqY1=){?18R6H06NM6K#L)b~utkL23UJov z=?`=t0Jr&04Bu&~1V6nMZs!}sgUGT5aeZxP5Kwfgb;C{xP)u>~XsnQdB8MnxpxBPG z9}P^${)$9rPTK4YM`)rVe#YC}uMdzaZ0C`y+$i!`_{_b(1PQ8gv$0|Iqa#b6cqGeU!2w;7gvVyK#fZTp|9L$TivhpZ+;W_~kR4Za@#%r!{ z`+hNAGRc^!bi9Ebnmk<6^q4eB=UgN&8I7f+iv=$8>g8gs-c&2$Ac37ipcdpH1U?_IfeZMqVawFIB( zvX2KvS*{|Qw$8vlwEuIGuM(g&NqgL|M+WpmNAor{+fj>!gt*m>NOV#y{L_OZO_ZjN z#3HQl0P)14UJ&D6l;0}a`EVDtK<30=I|3pidnzFvZx%x}(QUkyOqLAS@v6S9oVbQp z_*0PREIm19{NGwS;(ii?zd8)_3r}MF7tejOC?Us0)1T*$m(Iaeos`GH+Xb*+y}XfH zSO@A`-D+;SKLG?KE~L}cSc7j$qU~3dQ~*-arM>yU1=O&w33r4gfNlDZ44)IL0g;lw zub9Um(7IH(nI5zTTBr?+1h+`w#Nm?JJR1|tBl?i4Bh3Nt`-zJ9&U3+Hn!J$Am+a8+ zT|oZ9B}RDDku+khiWr7g`zU#BlAM1%l4*-SGf ^O0c1K+M&xW9^4`>`0?J^ z8T3Kk0FpaOV0eQfUxSDooUR!dNYZUbubaO(I`J`}m{Z+H3Y$8m8 z2-7g5WyGXz?~~mzWx@`hUvCwUrNDOUzO_;q{DCRU3i;mWDj|PfpPpugEwmPT9`fan*gDnEA>aN(8Uvu!~akY6A$`&=&pG<AIbWod-%l zOf8y@nv5K<2CRDs8MdQ#REnz_hmmN9)T!qk8Jg$}k;h8Rn*&5yX~ic4_d9#{N7e%F zw<&T%_XFOOQ`*Td@WRw6(lETls;p6dc)jOK0^RUhct%iIr7K__{(SFGCo*B+4H4Jk zMMiAP3?}{ON{gi^$(%W9y91RtgB?wx+o2Pk|9S1sP&gIBao5M{B)m-SU2?1(4GgnV z7La)We4a3qGvqmeqX&Upz4&>0nqwxslTrm1)EmP-P5c1LtoaoIifiD;DA(b;4iXqJ zy}9y^k_jFuNU#h_a==hU4+*xvT=3WLUN_yh?9g1=Z*R(s5$<)x)7i8W!xWo}gbPnr zz*b(JevhOI3>3+*58jUlg{0@{HSaqE?l%%{%aKZei~cb;;|X%Gpm6b& zsbxEA>@l+MNgjn-Y2BVm%+*Bu&ZE04We3Q^)CjsH+@mh^ko^)qSEU7|S%m)xNQNev zTms&hdQBrcyly+O(EzUlVeh0YUT>p_bz4_EjHmly2idV9Rv++qAe3Get7Sa1iCmTf63+2GJdm-SLDU<=HSOuJ;-gtMWtNbNmtLfBs-HPvXNJ zYI|_owbUqdD+WyQeL7jY`3}@n8A%yf4*;wv)zy=H6|{~;&8L`?z!`(^7Ha|{)Ty54 zcNF7*Q)j=|asoq zAQ$3swF*$0inSHi!~qrV0#Kwp|EFg+;lOr2_%cs#|bMI$yU9i1QtxF;H4z`c@f@`nL5Z4$%1^u6_i0&^80MCqv?6)tLfm^;}nhWNApuE({cjn*^c=D@F{l7QFP=344b}*3<7R$eK_{YTo zNkn41W>mRh`-=|uYQO;xM4WW&e=tIk3yCx>4kWNwGI62t)hejTpd<=%{sFKlr6t&1 z1%^TewaWz(!1PN{rlINr94fl16%{W6p#!Mt8Vj}<2v#oZt`t{Q8RzeioL9M8D z_RJ-8xYfx<_V)qebK)||=`sQm5}!g%^@M<&_977^^Clo3BcIYFaL2*~Tju}m-Il0L zfgRk>vhU3NBJOt`m5!Fyb;RaM!ZZT^=wn5Hu4&7eK}=4=If$D6Jf_pX@t8zW8Y>&y zh-TA~##m~}_q&KCFvrPy#tR4h*kEpg;z}_oMr}hY5t`5e*&;2N{BAkGb_z_r%#su? zr2bq}`BMs393yK3%ez5n8@stf;362b5ggj>A%X`ls!n%YV1zM#0=ng79PpCIOZkWQ zxZz)(_FG6X2fTZhMeUUW6P%4ow(yuDfr8uWe@?5d0qHC+O&7{RFnv;_1l+0yAKyRS z`Banu!oA!66J2rv*EAcJxcV=Glrxis^28M2Q>30uLP!T%z^P08nI#$>Su1$|G364< zD5yZu%yozm=a}SNCLKdK`hJ+`SK^LpBJpJ51mst)!LBntk8ftL%t+vKNG0FM!;Q~j z@oeFve%$Xqhfit#DQC=}mwEc^WkbxI^_%WXZVdCWf43Z-FNe7njeYuOD~7T6OTN5u zTL^pG)=`wobsDQuXk-vYgfYdyc74j;Q&^tRlF{23YV5`b{j(#(6OdG~K!cDR1Cu?) zWAs^2u%P^}Lj1-&5X(`TlSnuQ2R1B$4L7T+fyG{+8;k&VEhMsBG1 zd060<7$*!qZok`7!~`RH*JrqVNujn$wb|+RH8AXShe_n?AejAiq@9g@05>_fNIvT( zg5>0pr`7LVf$iAXPxeS<;MjQKgnJ|faLJPVPJE^lmG-nqVJMD9TcquH(oSii_Ur1# zv_^->?5YQQyTcelo+I2z^#`9b?@#H^vj|A`Sq+_RF9MR2=9YZ}_Z#$tZY;Q;?yrQ{ z9NbToyWM$G#{+xyGVl9Sb|Xw*;BMf`IEL|YrQZ0kE{7?xB~KFsMT=$z|0PHcnBM71u30!wcwuQM^4f?QYQ50tOQ z!sC_Wy@gQ;NZ6mPa|&7jcQX9@Kad^*9k(EhfP3`N!AEAz@H0CM7=D#v#mEEqP>D;I zKXAfaa~79V(kJ0Rt!$p_++?tfxrx2aaUEd1HW`WOL!gEHNtF1t`YagsABIWvThWE^OzUCNQP{>DAqQrzZ5UXkEz<%CEBX6 zVKliCe=7M2a zd8@QvPeL9#*R!lcWYD;4YQK|t15Cag^yZ2l2BvkiNp}Tn0mI@;L+73(pmLho!Y0Zc z6h7klD}50G%F(Qe(;bw+qpkdaGrJ3&YgiMMCw+!eE9d0U>0d^j{gyelejFmS2@DhO zFO4IQo}4MEavek5&C&BtlLW+y+K;$2jDTqN^qjeg`_*3!z1N2O-Sl&86~g_f%<0CS zOnYF+asQ2cRuhbZ{H9{`kt$YBI}y^(bsl5SGSA6Adg#f|`5w(oIqG9Q8JG`N#h@EkOeYxRW2dw?Tu zO*;({6l4>hT)2Lh3Jm7@KX`Dn8x2qAFnd||3>{nT488pKGU{K<_A!+E2#FYX8$}z& zk!X`AG;*b5h&}(7<#(bn9Dat z*AaDT>jxC@NQpVv?ZsbEDPsu-UX21Xhpb9@y*g05>^DLgnGE{6>|QD7+yt{<%if;l z0f2%^*tbcV8Z3QzNTR{tgRVNMwbiS{qLiMrOnfQYs0_U+tJw7;L?f?W)=6msL7(JO zKU^O}3@MGfVg$wz-S$TtQ_l&AenR);|GtBR_zG`G;C{F1c9s2czh^^iSB6WyvGYn_ z6?%e9FeaY3bGuV&#XsT!;z0Yq#Vz2YL zi^kos%{u#)=Ij$FDHcpK`QkLJU3v7y_7)oyo%rx~)sYt}T?(X&^5B6w=e^s#x>(?! z2=BfIM@snkm3#Ao>n2G5G4j*?3jsI?(tcG9`~-+^1kyA5y#O*C1t;ceJi(I^b_HP? z2%4fE7Dmib1Ej%FI4+_GeLZZ!%$6F9s?eYRsEz8N7vE<*O(-}*3h9SW2$oGCbBjMz z^JK@7Wi6r~H{lr4N$>r~B!hrde@-a7gZu5wyGAtQ=T^bohJyk3JGzkO8}Zi*+ihQ` zSm!gwPVpqmuT4X&@Nny|o0tr?+!-lb_Ff1JI5s!vHRizV_p^SAbTeSp&i5jcnJBQ< zl-bw>Ng~W(=#ze3{4OlXv~(Z4^bhiY@6y!`#25#4nSyEwDVFk7|E`$PKKx)9;*YTZ zgxMA}(lV@N@Fv&yO0_sE7(SI|c5j>$9ybYnR!ZW9%e@mGy61RdE~9r)Xf!KKDnG&Y z(S!;P5gP4Isc!+FBD-I5^J8GkEi~f>Un--bWK{rBXzN;?d^B2PU;)K8?Vi!zJuje_A zbV0HUKP|@)8jolbo*ewUqq*6FJ2hGcd@hC393dl*l0Jrz|hUP2Kqw?#$w#a zFJGbsaj6ZivPQkA{k#v&eZ4r8Lb+zx`PLQG>2qPmxa2Y7*RTz_A|{a#GO8xO+;L=+ z{MnNPpD_f%Y@|(I6A+AUtc?%%OC!a&v+=q6M)i1`5%=@oFwNgPe-opV4|s1tcMWs7 z8CzWQ8o?5R4iW3VbJ&SAUoz6@Q<&C80zXmCNsL_}@mM{H63d9w2>B^Ngt4iAsyX{- z2ksS`=^^+3!p!3A7p(a!(63!o%b;`}4i}wE`W^KTI*2a9dEWzwg>taSW^F*3r+!bE zSq9;y?a=rTBpa?rXYWHq?1; zWM9O{>uJ{RUl+rULeyfk*ts!f?kg&uB@9@2=(xr+Rx<3gmci4o2~B^TpsviD zg8H7~zVj-TQ2fLGi+?AaV25gO?pZ@2*zxny;78e0P`>qwT&4CUB#rf5-*<@ zxPUEI9k=i{3S;hCeyN+wY?#VReT^&EX|VsQc5|KMiLiNfc<;*5HneB@a-U6P9cETH zZGN?0gw+0lv7+q1AVngTGjaVa_&$@W8f5GL-(&Bk#hp6BfbkKZf#Q$;6zYFCioL?p=1USr#ewyw&Gkn)yFiYfjTMJhlNToin(RLWfWa0;GwaMC+AoQHG8R92%_>u}9Na)OcNFRXpIvj6q`0=!eW z;5`JCMlNRA=tubzY9fa*8f%VZ@_x$z3-Kli?H@w)y^~CS!kgA@rirb z6eK%q{5{-!3?|T}uXUx5Le(>;>>jKBgs(=~f3uYTfy3J8CB~H3;cx7T8ClM62wOWg zt7H11QO9c*iP2n0)7c{=muCU%tdbt5-V%UWrWEU%1D;Y)bq-FAqDVgL1l1=-sw;jL#IAT#z z9!CPQ~wSM&+)zCdNgUT%cFp4SwiC#8cr;8{cXRR z7b&KrDF5YQ_$Ksy;ooH|v;dec!qVnXT!tVS z6|Yu@b~)T)6U&qtb%oV3PuHNX5cH|}JyE2`1p_|C95J`j!U@QSl%>7w*c zAlG57C{Eb~Zh11bi*98Ak0uvwv**4bMpH44@0=P~ZeqMUXGsst6>lWBo#;oc(#^z_ zyW>$eInlMpWP0dKJW;foFcBJ2Y;5t$dkPu#V_yssnn3Px18JI@V+ec6l7tuT7ASqX zVgdKsFTI?+U`{|7^bt=zyx&*LETUd%V7pedhlRwaF}-tWeGID;E4%jR43Tg?&=t#W&h^v4DQzbgU+E9_j-2j ze&$KMPDpA_z8L{oZwfs4>n@D_l4Fc086v?dzSkU>QcpqYy=TNi@7v&I{y^J;Q_aw1 zn~p`xxCsUyf0(j5{}t-&wtjf6+yc|4es#;sErnSfpMW`Uyh(-_bk1Y68Mz zTh!;eGl30n-N|$+Kk&`?esXQF8kjT@FaOg<56+Hczn9YNM<<_uQ|&yANA>-gG%9p( zKaQ&~O`Zsq^mO15cbGzWcuz9uE{-Fww^^qCeOM$oO<#oKi z82c}FyjBLQ;e{I;uzAp9^USp$@K(HF;;{Dzc!~3QdGJ&TEI!tmdTCh-Ntns=M@lMS zSzGU;@R$#j`$RL~hP~)f$@eC}lv@7%A5j)? zrCqXW67&NrJG{?r64ikHD=WFQ8G2v_Pp@Kz{pcgX>@YgY1XTM$Z{hGyl;)9&D8Oh zrM2GhPg6~P+uWp^{fV{4g z8_A9%aAmY<>!<7tWEx(79ThtZ1vT|*sinrC*E^C*zS~VOaY*NGQ$;+C)7?0jysrQm zg?09MO}OEEPNqxUI*hO^w~M7X<^bqy4Efs#&wxhrru4zNCV)~8_qDKQfwphmn$?Pa zfM3As0$siuXqLXNS-D3KjQ4w|fOS9Wf5JN&FeIRbhur3lH}udgOEHtfi$v(PaQjPK z+EWPKT33Qi_Baw3nUv=;LO?n!A6BLOZ{Bv+n56^t0Eq0X{BY)ldHW9a{Y@P?cgIgipVQ zkZG|@!yrN}{O^pQ0I^(P~~nd+v88VXH^e^((w&Aje*UsIeyLa=#g z;5Uw>Q`~m5Dk30VT%u(cakuL%UQD1_ATd;hLb|vY z1UGwbQGKF=OQc~&`Bd`oPpVh#;X_l{BeRi5n&S&I8K!kpA|$|3Ec>B60~DoCI5!R&#}M=1ST`henDGi+-wvu`x(hm3Dd1)4=nKrZJA zm*tpgcqTU7%ukj8E7`ji z48XvEF(uBUADtFRlv?IWK=XL{#J+gzq2yWdfB8@%RP3xYa1x$Ere96%MQDs8ep4hA z#JBN%>k+R%i978?#7SA;9phj9N{KgGWZ$pfgn)dxEH1L)FAs=6Rv7ko1_O)jxj)*z zLm)DH^}%s7BP^ul++Gbp;1d>+Ybg|VFqkT9Z%*qj45{>d7u_8POI#%q^Omz9wO-ET zDgPq49e&ulBv%15D%PI(?N-4zTRbz)U%$Zc!?U@4Cwie!gU*z{I{~`g4O7`snS#7Q z#sZFBqmWGG=aC&{3yfA*F-yOc4fV!ynLo*xz-!&Rw82Td(D9@(7FNLs2~l0W-((NL zs=HVz`@I?9%cBq_A=M0Ih+KT{t7n0)wD)9Ioc+K_s?CdTJ!;@#$lr6c=NQ0gov_9f z-+pweLOSq}F9F?E(p%sR&_nasuF9SSL}-tdPLn^=6ykG6hFg(!9OHs$20-Y9jR>>Cyl_y zz&}Tyw_Bj#!|Y!VuDphQU+-owF&RNVH*2~G1%60=G(I=h#RxwxeA*Geeh5ry)H!34 zW&p8he0YLtGZ6ky^ijBe7P#CjCcEV42j1|w^(Bm|0in68A0sa?fTtbYJZ`t}_rJnh zt2BZM=$T^3K6FbDZJp93D#87>Tc4K5{+mP=>1~}tX2y`!z`JLbaWD0CqB$A7Nw->$ zKjB^pDz*lGcugOQ#A)DtPcm5Qr}7s4_N?MQi{UTSF8$(;yN(YyNhd=;MbioxX9d3{ zy`qMNf4Cy#x|JdI+fVN~p4&mw!Mazl`W}?(+PaNhkAsbD@r#d=GGXYm73iQ)0ADH? z1Ta4?gUBMCk2u{&t(|vNQ_b4;LvNvX=?DVS0t%t`CP)`)5|A#vgx;kI2q+z-O79>b zMS2sZsx*-fN~i(p{YxI@ya&$t>wDjP*9a-@ z#0vvu2C6&q0wMt|X8PKu%3)8P9WCw~N^|OY8SCZp17P}+O z?^Mj^k-mHj0W0LHt8aQU-W6pPo-X=Huu$^`=_+l95b@2%+ri!M|%6sdaVE!7oaMq*x)hd<*B z?43%>czt>f^9!p#@CYRxq45BqVQpDyT<|%Q9@gwDS-$B{lZ!bjB(Q_WFiFrQerDhD zz$tIYd?7khuQcJ@e3Ua~^g}CMG==oRVSPh|;iHZkEfXj6d7844M#N68xeUL62R#?{0>I$a zhcdfkVcFi!E<;CSf|Jr+r#!6n*p{mLQ8f5M(wm?PhH{%irk3PxS0j-->~5BE&V`#q zu;l;s9h`iK&uScw+*Kv2eDh8iyQjUiFeRUK;)8t2#7^j!?Lry`Q(n$|ZBKdvQLR>@(m5rhH&&+SOZ1Qi*{ts9|JeWj68q0tJ)+8Ai z_zhF-Fa(>gVWEPk&-X;Y*C7KG_FGaht!93>VeUq!0AR!aVSK%0;`4fj? zftGym*zH8XDWWer9~logS8f=sNcu@cSfr;;8qma6!hRpmnS;BM6uL%c zOp*7?kB?p6_V>(`OBi3@fVPOhBs1Yxi|7HjKGXTAt}^#|bsdd+WB0_>hOzd%pT&KP zXq^|EE&k2)0538Vvex6cDmzKdn+i|CyicTLCCjmg*8_P^KCuZ`kUd~%Q?6X^4AL^8 zZaUq1dK;xICd;;;8l z3`0vCEpc<@RgNg7-m}xw$n$8@vs<1Hb)c9lq;utwB5?PQ(1{?gq9 za9V$E6tYxWf~{#Sq^fjVM?6p{|NXYW$2U$c_tu)SaZoUoQoiQZoU==dnqNi|VTg@r z*?)*W4(a6&aeL4UUXY>&*&fVuRMr9Z)rW*jZK*&ou5G>dDl((7^_Gsa_9#07R;A(N z58BsQ#Nn#VglTxE!;$IiObn9hbl%oDI?$9u5mqf`kB4M5W}g)I2I+EN>u5!I3UqPt zPm}_>7>aGC67gA6=hEJa?R6A-N9;XNq)keIe#TM{eV%P)mZf}~etI`rAI)MQXl%4) z1$+u3$$Umht6uoLFAX*fjLKzKO4n71pdrM%E5-0ZGvV$K?qi6~vWS{;@bi8}y25H8 z+2IAXY^yfsLwWdJ?$EgQ$3IV1xKuzPG*GNh_VlP6;R7D0Vm>eezObD$-N)vQ=C4b< zPw`G(U)XfbG!cB4c&IdYVT30A7(}m=iidgNKiur%2$+AYVW4vGMi?$O;pcNAGZ)u&{wBg#)wW# z=s3eI2krjn9r^u;ofm}JHI!cX&OfBtxSX8S?v_NYZt;zad5De86La#TB`;WW=v>tU zVJd)~x@H+y!S?~&)fT^AR{$laQz?$3ER}1on(Ay}9EDaYE)cER{ z?#FE$=tNAQ(0nZBFZe{QpzWZc7uuzt< z4?`O}M_I^=-$xeoMdVjW>TQgaNV!Vj8z&8`0uSW`MyazqG>Y* z!sFW8@|P#krs86460%eu^w5hx;8JeEXgUzR|6TPR%%XsY1f(3yCRXk_(syD3k!s?r z(B*NEMb9W%u|$*>X|aFve0I3TIQOif-ELpr1j)yHih*c}^Be?+GM-%of}zLHJ#5LO ze&vA+U7+{|n*=K<%C=amsCeqV<>o2k`sNX$q?ir-6C;XE0~&U2YnGv6G|d49DI=2Z zw^SYOf1La}css`8eY26BD1RJO0y1exD61AfKFZbeN+_PVNjeB?Ju6Ez%m5_Cr3`Ox zQ~(&)mph4I5XX4NZ~ay@=wd&NK9^1w-plnHtSv~TuPDiBPQ=5rJ`WvrYN975cob&Z zV$6_4yV?R>fzOolb}y>_cZ;|Y1N7us6`Gpb7);Y(s2 zZ8j$pwHOT@2!Am6pc=ouuC3%avlzXCs^TzuXmR1Bz4S+2k{)8t`*_RrCAz?~H#N%+ za#h^iHtek?Xtq#k*{=`rtQ9j(6B}=#J=ywFxKC~@M=BrL)}2MXAZ4NP>15ll;TwIg zR}zQoEGKH^hZpD*N~DnyW&^E#V<;Cm_d|OmII_Q#YS2tkVVW8g=yNx6IW_vIb=wD$ zYTY@RCSe|WOM^GnXqiRv`qn|NP3I0z!o9^?hpYZ*o;K_ozqcs6E+!L;7Nx~rkh_63 z=Bu~UA~QG>_(>$(w)mFlUYJN4ae3+XVs7AXH5#2s#D$VjX?e2ME_;K*U8~4bwMCcuB zXonCe)s4EQ{rFL&ias~3la4}CwXhayC>C#|>MQQIAY5WMt0!!6CXvbFB|iYIlU{^N zQ>H@PoO3>_eDq|i++#R#dV1R@O&2?^=JTITNgJ+FU0Pm!&AZ8{@Z3TJejr*)+paS8(QMd zt}9~p;;Aai$xpp0y1^>R!{Ekx!!9770l06peo|`1BfD}slBy~ArQ&C$U6@`%l(wCZ z=%Sy70Z)SwVfvSHYaCr*LwEgv5RTZSXCSPa8VFo1;Bf6#Y|oW?hZr-MQ(RgE^f>Cm zNHBc!e!dajo6I{3D+1Gj-XpeNGf9yhEORCfVML`ZLjM>%PiqxOhfY>-N=Km?sLkz2 z=gA6ro2N$^#TxJ4uH+@~ESVK87VdYUYnJ2m)~U&29+Q+z@5$xrf)9Gq#QUYU`0|=& zX&Fn9eJ&B#=bb;NH8>;tTIBlND#F17B4G|+@Cb zOdq;%qbg|m88J#G^CIa^oMeaNt?x!939HdwQ9O3ZqE@40(rVS)f)-j#AmiN3+m81` z0zQMcqSTd8O84z0*Iqhd+q{e05@eAQ60Bok?#4ZnwPd!hv(#v@7qOsV#fN?HYhZ&~ zcQ7~rv4z-2RjYTCBT(P%v)7VNCMFSpYKT4(j{wNa69#}lY*^*1) z(&F&p{#|k@skxcgakOaIUQ+DY%db&u3VFWbMG-EH+|Ny1&T<{~WbLPm^P`binobS~ zQ_YF6HEbYV8*P`o_wU+pr};0Km( zy9D15KmA!g&)c2km6wH{pTW?n49DnFpUrxUfp@=(t3$7Y?0qa(^dMZ}w<|NH;fDq}g;jr&_6 z@b@?`j`hv=W@^y=Z0}(R2IK6#Ro|T=!)LXYFI39HIeDo-d0NTwh^js49nr3Z-A|_q z){VP!)ke&sSnz?&gafU-Z1OmReQRs_ikBxjGiO|PXvI4A!^BWDq+uwLLYO<)M1!dlGH^T#Rc`V-UeIJq_ z0~KwEwiIJwnRXpX*3XzXZbEsVP&&!ZfWHnA|FVx%GVHiC$_Q67Spc(SF+I%E4?$Xqf$7`N-Y~ z>}!6Tw50K&z(&M`iFVxg+`*&!)5QH{0vz~Zy=Kh&`e$JyAG?(u;L_;f*q^ESBT>kz znnWoUS|Y}i)aIJJVEFIS9sI+0#{<#SgZgW7Bbb@PMzT&}qwx|<4TU)^$|i2o^vQ#< zk~-e_Htz|BwVs<%>-S<{FC)&&$BhvS1_n-{?>3DlDi~>xDZsf;DX>ek_GfC| zyD_Ckmpsqu!d^b#{oy}cn{&AtaC|ih)V;jHFSJ9X<933G!j{&w&7))}wKo~n9_K}sGt#BAOumqB5SQ8c5%wo#UhM} zgI_>+FZWZEOZVTiIV@J({jrsirYyioO8&sjdU`DNTZMC?-)?@f;kuX4eXI51V4;Yx z*S>w1xmmbOHam3fQ4GyeRs*}+qZ0QXEuC0*E*RZvU?Wh&OScL72vqMXtHtzv=ow}i zi8C?qB=u+yk6G~qXJNWb9wv`=;(GDYmq%o8)X|sM*Gf!ueArLKl74;^YhXt$NIT$N zjAQ#*t-_(HFGR-ph=DY34to~dtT+f;g-8*)ZEs;*cSej{3azf9gmeK^uUvo^f8(O^ z_xS&9ilm9)*5Q(&@cde)O9Y?03*Yo1tQWw|?2Nx`-g$93QB3EJkgXb}VvcFRte-A}$ckUOdfN4BsqmUu4@@3F;Y^?ctT} zahx2+h?=W6Mx_VOi4E`HP~|6dsC9vWVJ)2M zuMMRVx(tMXOay@i5bK7Gb{rkULb^L)THO{?+8d!Ok#g^*h3pZl2+{<%C`Fxp%P8Os zS)@n7Ef9gtuWO3=Srp$=hY}$0jmq-2Hh1?iDCw^E55c3BnNdmnR^t_U_jtsA)`q&3 zfGYWW)`>p9tZ_F9Hor%=7=v5g)ZxbMZCd&Y^FIeZR_$|eVhOjZ=wqJzcFk!%x|F?( zwa7OV{)?qrRMMN?Ob^;tY)b220j-q1)wM`VPiPf-05N?~LR+}&rPLQcAtr}-#H!F$ zU%M6@ByYPY!Ss>jWCfI@BPH3(|LQgzMNIeeij&bckPl=J#@cTtnPB17VDzpmL)jS&G<*HxnS!L|+AI zb0RHIqCb)S=QHr1@cy+&&7Yb58BK1g*-=G4Dj=>$-a*&Z^N2GJ#?WW>7OuvYZgzIa zd+nxLwvl3K5OE!{Yz)Nl>J^A$jW`4SZ;}rUJG^yt0Du(dO^F>U0O0xGBuH>N5{-^5 z1@o#1#4!*>j}h<}qkgkyNM?8FUk&~>4KXv|I>Gy<6p4yPa-fmL;a!uC-3I(SIQpg# zNi@DHllYp@iWTsmeB+yPBuE#@%Kgt^^0EQ`V0CXwE)m#UB-8dkt6GbY{2K)KrVt5S zMbcFNQ|QWp_~HHszj{-MBz^t^@&o|@IM>>E5AlQki~D&~hTQh$pDkSwGT=3tkv`zs zCNMXZNWcH8dy`yKW`+a)i~E05k95+2l{ngaOulzy`;WL{H22$478^|I1Z0e`aGO$jm)A#)5e-(2tC zv}uTd{(D-wDMb#^$gvkW4ih52;_4NMBK`&$k!u6Ch6=_%w;UN_EO$T$0Ls&mm;XO= C+;@lo literal 0 HcmV?d00001 diff --git a/studio/tests/golden/make_fixture.py b/studio/tests/golden/make_fixture.py new file mode 100644 index 0000000..2c1a6f9 --- /dev/null +++ b/studio/tests/golden/make_fixture.py @@ -0,0 +1,158 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Build the Tier-A golden fixture: a short run, reduced on a UNIFORM stride. + + python -m studio.tests.golden.make_fixture # writes tier_a_short_run.npz + python -m studio.tests.golden.make_fixture --check # reports drift, writes nothing + +**Uniform stride, not an adaptive or thinned grid.** A coarsening grid aliases the morning +particle-number spike by up to 8x (ADR-009, and ``CAVEATS.md``), so a fixture built on one would +encode the aliasing and then assert it forever. Every ``STRIDE``-th stored sample, and the last one +so the endpoint is always present. + +The fixture is committed because it is small (tens of kB) and because Tier A must not depend on the +3.6 MB archive, which is gitignored and absent on a fresh clone. Regenerating it is a deliberate act +with a recorded reason -- ``--check`` exists so drift can be measured without overwriting the +reference by accident. +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path +from typing import Any + +import numpy as np + +from studio.resolve import resolve +from studio.schema import RunConfig +from studio.tests.golden.tolerances import ( + RTOL_HEADLINE, + RTOL_SERIES, + worst_relative_deviation, +) + +#: Where the fixture lives. Next to the tests that read it. +FIXTURE_PATH = Path(__file__).parent / "fixtures" / "tier_a_short_run.npz" + +#: 1 day, 40 bins: the cheapest run that still exercises the whole pipeline -- gas chemistry, TUV-x +#: photolysis, all three microphysics processes and dilution. ~21 s on an M-series CPU. +TIER_A_DAYS = 1 +TIER_A_BINS = 40 + +#: Keep every 4th sample. 1 day at 600 s nominal is ~147 stored samples, so this is 38 -- enough +#: for a series comparison to be meaningful, small enough to commit. +STRIDE = 4 + +#: Arrays reduced along time. Everything a Tier-A assertion compares. +_TIME_SERIES_KEYS = ( + "t", + "x", + "SA", + "radius_cm", + "h2so4wp", + "particulate_S", + "T", + "V_ratio", + "total_n", + "n_cm3", + "dNdlogDp", +) + +#: Arrays that are not time series and are stored whole. +_WHOLE_KEYS = ("species", "M", "dp_mid_um", "J_equations") + +#: Photolysis is on the INTERVAL axis (n_times - 1), not the sample axis, so it strides separately. +#: Included because ``REFERENCE_TOLERANCES.md`` measured it as the most reproducible part of the +#: pipeline (1.09e-13), which makes any drift there a strong signal rather than noise. +_INTERVAL_SERIES_KEYS = ("J_tmid", "J") + + +def tier_a_config() -> Any: + """The Tier-A case: the golden case's defaults, shortened and coarsened.""" + payload = RunConfig().model_dump() + payload["schedule"]["duration_days"] = TIER_A_DAYS + payload["microphysics"]["n_bins"] = TIER_A_BINS + return resolve(RunConfig.model_validate(payload)) + + +def run_tier_a_case(out_dir: Path) -> Path: + """Run the case and return the path to its ``state.npz``.""" + from studio.modelio.execute import run_and_write + + return run_and_write(tier_a_config(), out_dir)["state"] + + +def reduce_state(npz_path: Path) -> dict[str, np.ndarray]: + """Uniform-stride reduction of a ``state.npz``, keeping the final sample.""" + with np.load(npz_path, allow_pickle=True) as archive: + data = {key: archive[key] for key in archive.files} + n_times = len(data["t"]) + keep = sorted(set(range(0, n_times, STRIDE)) | {n_times - 1}) + reduced: dict[str, np.ndarray] = {"kept_indices": np.asarray(keep), "n_times_full": n_times} + for key in _TIME_SERIES_KEYS: + reduced[key] = np.asarray(data[key])[keep] + n_intervals = len(data["J_tmid"]) + keep_intervals = sorted(set(range(0, n_intervals, STRIDE)) | {n_intervals - 1}) + reduced["kept_intervals"] = np.asarray(keep_intervals) + for key in _INTERVAL_SERIES_KEYS: + reduced[key] = np.asarray(data[key])[keep_intervals] + for key in _WHOLE_KEYS: + reduced[key] = np.asarray(data[key]) + return reduced + + +def write_fixture(out_dir: Path) -> Path: + """Run the case, reduce it, write the fixture.""" + state = run_tier_a_case(out_dir) + reduced = reduce_state(state) + FIXTURE_PATH.parent.mkdir(parents=True, exist_ok=True) + np.savez_compressed(FIXTURE_PATH, **reduced) + size_kb = FIXTURE_PATH.stat().st_size / 1024 + print( + f"wrote {FIXTURE_PATH} ({size_kb:.0f} kB): {len(reduced['kept_indices'])} of " + f"{reduced['n_times_full']} samples, stride {STRIDE}" + ) + return FIXTURE_PATH + + +def check_against_fixture(out_dir: Path) -> int: + """Re-run and report drift against the committed fixture. Writes nothing.""" + if not FIXTURE_PATH.is_file(): + print(f"no fixture at {FIXTURE_PATH}; run without --check first", file=sys.stderr) + return 2 + fresh = reduce_state(run_tier_a_case(out_dir)) + with np.load(FIXTURE_PATH, allow_pickle=True) as archive: + reference = {key: archive[key] for key in archive.files} + species = [str(name) for name in reference["species"]] + print(f"{'quantity':22s} {'worst rel':>12s} tolerance") + for name, rtol in (("SO2", RTOL_HEADLINE), ("H2SO4", RTOL_HEADLINE)): + index = species.index(name) + worst = worst_relative_deviation(fresh["x"][:, index], reference["x"][:, index]) + print(f"{name:22s} {worst:12.3e} {rtol:.0e}") + for key in ("total_n", "SA", "particulate_S", "dNdlogDp"): + worst = worst_relative_deviation(fresh[key], reference[key]) + print(f"{key:22s} {worst:12.3e} {RTOL_SERIES:.0e}") + return 0 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--check", action="store_true", help="re-run and report drift; do not overwrite the fixture" + ) + parser.add_argument( + "--work-dir", type=Path, default=None, help="where to run (default: a tempdir)" + ) + args = parser.parse_args() + + import tempfile + + with tempfile.TemporaryDirectory(prefix="studio-golden-") as tmp: + out_dir = args.work_dir or Path(tmp) + return check_against_fixture(out_dir) if args.check else (write_fixture(out_dir) and 0) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/tests/golden/paper_cases.py b/studio/tests/golden/paper_cases.py new file mode 100644 index 0000000..5a7d57f --- /dev/null +++ b/studio/tests/golden/paper_cases.py @@ -0,0 +1,133 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Case ID -> ``RunConfig``, for the Tier-B archive comparison. + +The 810-run ensemble names each case by its axis levels joined with ``__`` -- +``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` -- and that name IS the parameter set +(``run_ensemble.py:84``). This module reverses it, so a Tier-B test can say "reproduce this archived +directory" and get the config that produced it. + +Lives under ``studio/tests/`` rather than in ``studio/``: task 0.2 deliberately kept the paper +ensemble's axes out of the package, on the grounds that they would earn a home there when something +needed them. Tier B needs them *as test data*, which is not the same as needing a preset library, so +they stay here until production asks. + +Token tables are copied from ``run_ensemble.py:61-76`` and checked against it by +``test_tier_b_archive.py`` -- a token that stops matching is a case that would silently be run with +the wrong parameters and compared against the right archive. +""" + +from __future__ import annotations + +from typing import Final + +from studio.resolve import ResolvedConfig, resolve +from studio.schema import BackgroundAerosol, DilutionRegime, RunConfig + +#: site token -> (latitude, T [K], p [mbar], H2O [ppmv]). ``run_ensemble.py:61-65``. +SITES: Final[dict[str, tuple[float, float, float, float]]] = { + "30N_20km": (30.0, 210.0, 55.0, 6.9104), + "60N_15km": (60.0, 210.0, 120.0, 3.1673), + "30N_20km_213K": (30.0, 213.0, 55.0, 10.1834), +} + +#: background token -> (aerosol distribution, background SO2 [pptv]). ``run_ensemble.py:67-71``. +BACKGROUNDS: Final[dict[str, tuple[BackgroundAerosol, float]]] = { + "sabr330": (BackgroundAerosol.SABR_330, 20.0), + "sabr220": (BackgroundAerosol.SABR_220, 20.0), + "cesm": (BackgroundAerosol.CESM_G6, 100.0), +} + +#: dilution token -> regime. ``run_ensemble.py:72-73``. +DILUTIONS: Final[dict[str, DilutionRegime]] = { + "D1low": DilutionRegime.D1, + "D2med": DilutionRegime.D2, + "D3high": DilutionRegime.D3, + "burst": DilutionRegime.BURST, + "D5vhigh": DilutionRegime.D5, +} + +#: condensation alpha, nucleation scale, coagulation scale. ``run_ensemble.py:74-76``. +STICKING: Final[dict[str, float]] = {"a0p5": 0.5, "a1p0": 1.0} +NUCLEATION: Final[dict[str, float]] = {"nuc0p01": 0.01, "nuc1": 1.0, "nuc100": 100.0} +COAGULATION: Final[dict[str, float]] = {"cg0p5": 0.5, "cg1": 1.0, "cg2": 2.0} + +#: The ensemble's fixed values, which are the schema's defaults: 80 bins, day 172, 00:00, 10 days, +#: ion pair rate 30, SO2+HO2 1e-18, aerosol->J and heating off. Asserted, not assumed, by +#: ``test_tier_b_archive.py``. +ENSEMBLE_BINS: Final = 80 +ENSEMBLE_DAYS: Final = 10 + + +def config_for_case(case_id: str) -> ResolvedConfig: + """The resolved config that produced the archived directory ``case_id``. + + Raises: + ValueError: On an unknown token or the wrong number of them. A mistyped case would otherwise + be run with default parameters and compared against a real archive, which fails in a way + that looks like a physics regression. + """ + tokens = case_id.split("__") + if len(tokens) != 6: + raise ValueError( + f"case id {case_id!r} has {len(tokens)} tokens, expected 6: " + f"site__background__dilution__sticking__nucleation__coag" + ) + site, background, dilution, sticking, nucleation, coagulation = tokens + for token, table, what in ( + (site, SITES, "site"), + (background, BACKGROUNDS, "background"), + (dilution, DILUTIONS, "dilution"), + (sticking, STICKING, "sticking"), + (nucleation, NUCLEATION, "nucleation"), + (coagulation, COAGULATION, "coagulation"), + ): + if token not in table: + raise ValueError(f"unknown {what} token {token!r}; known: {sorted(table)}") + + latitude, temperature, pressure, h2o = SITES[site] + aerosol, background_so2 = BACKGROUNDS[background] + payload = RunConfig().model_dump() + payload["site"].update( + latitude_deg=latitude, + temperature_k=temperature, + pressure_mbar=pressure, + h2o_ppmv=h2o, + ) + payload["background"].update(aerosol=aerosol.value, so2_pptv=background_so2) + payload["dilution"]["regime"] = DILUTIONS[dilution].value + payload["microphysics"].update( + condensation_alpha=STICKING[sticking], + nucleation_rate_scale=NUCLEATION[nucleation], + coag_kernel_scale=COAGULATION[coagulation], + n_bins=ENSEMBLE_BINS, + ) + payload["schedule"]["duration_days"] = ENSEMBLE_DAYS + return resolve(RunConfig.model_validate(payload)) + + +#: The curated Tier-B set: D1/D2/D3/burst against the clean and loaded backgrounds (ADR-009 asks for +#: 4-6). All are ``cg1`` deliberately -- ``REFERENCE_TOLERANCES.md`` records that cg0p5/cg2 may +#: straddle the tomas-jax commit that wired ``coag_kernel_scale`` through, so adopting one as a +#: golden case needs its own measurement first. +TIER_B_CASES: Final[tuple[str, ...]] = ( + "30N_20km__sabr220__D1low__a1p0__nuc1__cg1", + "30N_20km__sabr220__D2med__a1p0__nuc1__cg1", + "30N_20km__sabr220__D3high__a1p0__nuc1__cg1", + "30N_20km__sabr220__burst__a1p0__nuc1__cg1", + "30N_20km__sabr330__D2med__a1p0__nuc1__cg1", + "30N_20km__sabr330__burst__a1p0__nuc1__cg1", +) + +__all__ = [ + "BACKGROUNDS", + "COAGULATION", + "DILUTIONS", + "ENSEMBLE_BINS", + "ENSEMBLE_DAYS", + "NUCLEATION", + "SITES", + "STICKING", + "TIER_B_CASES", + "config_for_case", +] diff --git a/studio/tests/golden/test_tier_a_short_run.py b/studio/tests/golden/test_tier_a_short_run.py new file mode 100644 index 0000000..d58f413 --- /dev/null +++ b/studio/tests/golden/test_tier_a_short_run.py @@ -0,0 +1,183 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Tier A: one short real run against a committed fixture. + +This is the tier that catches **drift in Studio's own pipeline** -- a changed derivation, a +reordered operator split, a dependency bump that moves the solver. It compares against a fixture +this project generated (``make_fixture.py``), not against the 2026-07 archive; reproducing the +archive is Tier B's job and costs ~4.6 min per case. + +**Honest limitation, stated because it undercuts the plan's intent:** the plan calls Tier A "CI, +seconds", but CI does not check out the private submodules, so the model cannot run there. In CI +this module **skips**, and Tier A there is the pure schema/units/DAG/hash/expansion tests; locally +it runs. Fixing that means giving CI a deploy key for the submodules, which is its own change -- +tracked rather than quietly ignored. + +Cost when it does run: ~19 s for the run plus the comparison. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np +import pytest + +from studio.tests.golden.make_fixture import ( + FIXTURE_PATH, + STRIDE, + TIER_A_BINS, + TIER_A_DAYS, + reduce_state, + run_tier_a_case, + tier_a_config, +) +from studio.tests.golden.tolerances import ( + EXACT_ARRAYS, + RTOL_BIN_EDGES, + RTOL_PHOTOLYSIS, + RTOL_SERIES, + RTOL_SIZE_DISTRIBUTION, + assert_exact, + assert_headline_matches, + assert_series_matches, +) + +#: Series compared sample-by-sample at the series tolerance. +_SERIES_KEYS = ("SA", "radius_cm", "h2so4wp", "particulate_S", "total_n") + +#: Gas species compared at the headline tolerance. The sulfur chain plus its oxidant: what a result +#: is actually read for. Near-zero species are excluded by the floor in ``tolerances.py``. +_GAS_KEYS = ("SO2", "SO3", "H2SO4", "OH") + + +@pytest.fixture(scope="module") +def reference() -> dict[str, Any]: + """The committed fixture, or skip if it has not been generated.""" + if not FIXTURE_PATH.is_file(): + pytest.skip( + f"no Tier-A fixture at {FIXTURE_PATH}; generate it with " + f"`python -m studio.tests.golden.make_fixture`" + ) + with np.load(FIXTURE_PATH, allow_pickle=True) as archive: + return {key: archive[key] for key in archive.files} + + +@pytest.fixture(scope="module") +def fresh(repo_root: Path, tmp_path_factory: pytest.TempPathFactory) -> dict[str, Any]: + """Re-run the Tier-A case now and reduce it the same way. ~19 s.""" + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip( + "model submodules not checked out (`git submodule update --init`); the Tier-A golden " + "run needs them. CI does not check them out, so this skips there -- see the module " + "docstring." + ) + out_dir = tmp_path_factory.mktemp("tier_a_golden") + return reduce_state(run_tier_a_case(out_dir)) + + +@pytest.mark.tier_a +def test_the_case_is_what_the_fixture_was_built_from(reference: dict[str, Any]) -> None: + """Guard against comparing a re-run of one case against a fixture built from another. + + Cheap, and it fails clearly. Without it, changing ``TIER_A_DAYS`` would produce a shape mismatch + deep inside a series comparison instead of saying "the fixture is stale". + """ + config = tier_a_config().config + assert config.schedule.duration_days == TIER_A_DAYS + assert config.microphysics.n_bins == TIER_A_BINS + assert len(reference["dp_mid_um"]) == TIER_A_BINS + + # the reduction rule, restated: every STRIDE-th sample plus the last one + n_full = int(reference["n_times_full"]) + expected = sorted(set(range(0, n_full, STRIDE)) | {n_full - 1}) + assert list(reference["kept_indices"]) == expected + + +@pytest.mark.tier_a +def test_the_time_axis_is_bit_identical(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """``t`` is built from the terminator schedule, not integrated. Any drift is a real bug.""" + assert_exact("t", fresh["t"], reference["t"]) + + +@pytest.mark.tier_a +@pytest.mark.parametrize("key", EXACT_ARRAYS) +def test_analytic_arrays_are_bit_identical( + key: str, fresh: dict[str, Any], reference: dict[str, Any] +) -> None: + """``t``, ``V_ratio`` and ``T``: analytic, so exact equality is the right assertion.""" + assert_exact(key, fresh[key], reference[key]) + + +@pytest.mark.tier_a +@pytest.mark.parametrize("species", _GAS_KEYS) +def test_gas_species_reproduce( + species: str, fresh: dict[str, Any], reference: dict[str, Any] +) -> None: + names = [str(name) for name in reference["species"]] + index = names.index(species) # BY NAME, never by position + # Endpoints at the headline tolerance, the series at its own looser one -- two different rows of + # REFERENCE_TOLERANCES.md. Conflating them is what broke Tier B's first real run. + assert_headline_matches(species, fresh["x"][:, index], reference["x"][:, index]) + assert_series_matches( + f"{species} (series)", fresh["x"][:, index], reference["x"][:, index], RTOL_SERIES + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize("key", _SERIES_KEYS) +def test_aerosol_series_reproduce( + key: str, fresh: dict[str, Any], reference: dict[str, Any] +) -> None: + assert_series_matches(key, fresh[key], reference[key], RTOL_SERIES) + + +@pytest.mark.tier_a +def test_the_size_distribution_reproduces(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """Per bin, over the whole stored series -- not just the final spectrum.""" + assert_series_matches( + "dNdlogDp", fresh["dNdlogDp"], reference["dNdlogDp"], RTOL_SIZE_DISTRIBUTION + ) + assert_series_matches("n_cm3", fresh["n_cm3"], reference["n_cm3"], RTOL_SIZE_DISTRIBUTION) + + +@pytest.mark.tier_a +def test_photolysis_reproduces(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """J is the most reproducible part of the pipeline (measured 1.09e-13), so drift here is signal. + + Compared per reaction, not summed: a compensating pair of errors across two reactions would + survive a total and is exactly the kind of thing this tier exists to catch. + """ + assert_exact("J_tmid", fresh["J_tmid"], reference["J_tmid"]) + equations = [str(name) for name in reference["J_equations"]] + for index, equation in enumerate(equations): + assert_series_matches( + f"J[{equation}]", fresh["J"][:, index], reference["J"][:, index], RTOL_PHOTOLYSIS + ) + + +@pytest.mark.tier_a +def test_the_dry_bin_edges_reproduce(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """Not exact: Studio and the archive spell the geometric mean differently (see tolerances.py). + + Within Studio's own pipeline they should agree bit-for-bit, so this passing at 1e-15 rather than + exactly would itself be information -- the tolerance is the one measured against the archive. + """ + assert_series_matches("dp_mid_um", fresh["dp_mid_um"], reference["dp_mid_um"], RTOL_BIN_EDGES) + + +@pytest.mark.tier_a +def test_the_run_is_still_physically_recognisable(fresh: dict[str, Any]) -> None: + """A sanity floor under the tolerances: a comparison can only be meaningful if the run happened. + + All four assertions would hold for any correct run of this case, and none of them would hold for + a run that silently did nothing -- which is the failure a tolerance-based test cannot see. + """ + names = [str(name) for name in fresh["species"]] + so2 = fresh["x"][:, names.index("SO2")] + h2so4 = fresh["x"][:, names.index("H2SO4")] + assert so2[-1] < so2[0], "SO2 must be consumed" + assert h2so4.max() > 0.0, "H2SO4 must be produced" + assert fresh["total_n"].max() > 0.0, "particles must form" + assert fresh["V_ratio"][-1] > 1.0, "the plume must expand" diff --git a/studio/tests/golden/test_tier_b_archive.py b/studio/tests/golden/test_tier_b_archive.py new file mode 100644 index 0000000..045e5f0 --- /dev/null +++ b/studio/tests/golden/test_tier_b_archive.py @@ -0,0 +1,199 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Tier B: reproduce the archived 810-run ensemble, at the measured tolerance. + +**Nightly or manual, never in CI.** Six 10-day cases at ~4.6 min each is ~28 minutes, and the +archive it compares against is gitignored, so a fresh clone does not have it. Run it explicitly: + + pytest studio/tests/golden -m tier_b + +What it asserts is **reproduction of results produced in July 2026 by a different toolchain**, +which is a different claim from Tier A's "Studio has not drifted since its own fixture". +Bit-for-bit is already known to be false -- ~31 % of gas state elements differ -- so every tolerance +here comes from ``REFERENCE_TOLERANCES.md``, measured before it was asserted. + +**A failure here is not automatically a regression.** A JAX or diffrax bump moves these numbers; the +correct response is to re-run ``measure_deviation.py``, update the record with the new SHAs, and +decide whether the new deviation is acceptable -- not to widen a constant until the test passes. The +assertion messages say so at the point of failure, where the temptation is. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np +import pytest + +from studio.tests.golden.paper_cases import ( + ENSEMBLE_BINS, + ENSEMBLE_DAYS, + TIER_B_CASES, + config_for_case, +) +from studio.tests.golden.tolerances import ( + EXACT_ARRAYS, + RTOL_HEADLINE, + RTOL_SERIES, + RTOL_SIZE_DISTRIBUTION, + assert_exact, + endpoint_deviations, + worst_relative_deviation, +) + +#: Gas species compared at the headline tolerance, by name. +_GAS_KEYS = ("SO2", "SO3", "H2SO4", "OH") + +#: Aerosol series compared at the series tolerance. +_SERIES_KEYS = ("SA", "radius_cm", "h2so4wp", "particulate_S", "total_n") + + +@pytest.fixture(scope="module") +def archive_root(repo_root: Path) -> Path: + """The archived ensemble, or skip. Read-only: this is irreplaceable reference data.""" + runs = repo_root / "coupled" / "paper_ensemble" / "runs" + if not runs.is_dir(): + pytest.skip( + f"archived ensemble not present at {runs} (gitignored and regenerable); Tier B " + f"reproduces it and cannot run without it" + ) + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + return runs + + +@pytest.fixture(scope="module") +def reproduced( + archive_root: Path, tmp_path_factory: pytest.TempPathFactory +) -> dict[str, dict[str, Any]]: + """Re-run every curated case once, and pair each with its archived counterpart. + + Module-scoped because each case costs ~4.6 min: running them once and sharing the arrays across + assertions is the difference between 28 minutes and several hours. + """ + from studio.modelio.execute import run_and_write + + out_root = tmp_path_factory.mktemp("tier_b_golden") + paired: dict[str, dict[str, Any]] = {} + for case_id in TIER_B_CASES: + archived_path = archive_root / case_id / "state.npz" + if not archived_path.is_file(): + continue # a case absent from this machine's archive is a skip, not a failure + fresh_path = run_and_write(config_for_case(case_id), out_root / case_id)["state"] + with ( + np.load(fresh_path, allow_pickle=True) as fresh, + np.load(archived_path, allow_pickle=True) as archived, + ): + paired[case_id] = { + "fresh": {key: fresh[key] for key in fresh.files}, + "archived": {key: archived[key] for key in archived.files}, + } + if not paired: + pytest.skip(f"none of {list(TIER_B_CASES)} is present in {archive_root}") + return paired + + +@pytest.mark.tier_b +def test_the_curated_set_covers_the_dilution_regimes_and_both_backgrounds() -> None: + """ADR-009 asks for 4-6 cases across D1/D2/D3/burst x sabr220/sabr330. Assert the set, cheaply. + + Pure — it needs neither the model nor the archive, so a mis-curated set is caught in + milliseconds rather than 28 minutes in. + """ + assert 4 <= len(TIER_B_CASES) <= 6 + regimes = {config_for_case(case).config.dilution.regime.value for case in TIER_B_CASES} + backgrounds = {config_for_case(case).config.background.aerosol.value for case in TIER_B_CASES} + assert {"D1", "D2", "D3", "burst"} <= regimes + assert {"sabr_220", "sabr_330"} <= backgrounds + for case in TIER_B_CASES: + config = config_for_case(case).config + assert config.microphysics.n_bins == ENSEMBLE_BINS + assert config.schedule.duration_days == ENSEMBLE_DAYS + assert config.microphysics.coag_kernel_scale == 1.0, ( + "the curated set stays on cg1 until the coag_kernel_scale coverage gap recorded in " + "REFERENCE_TOLERANCES.md is measured" + ) + + +@pytest.mark.tier_b +def test_every_case_reproduces(reproduced: dict[str, dict[str, Any]]) -> None: + """One test over all cases, reporting every deviation before failing. + + Deliberately not parametrised per case: after 28 minutes of compute, "SO2 failed in case 3" is + much less useful than the whole table. A per-case failure would also hide whether the deviation + is systematic or specific to one regime, which is the first thing to want to know. + """ + failures: list[str] = [] + #: every deviation, not only the ones that exceed: 28 minutes of compute should produce a + #: measurement, not just a verdict. Printed below so a passing run still reports numbers. + observed: dict[str, tuple[float, float]] = {} + for case_id, pair in sorted(reproduced.items()): + fresh, archived = pair["fresh"], pair["archived"] + species = [str(name) for name in archived["species"]] + + for key in EXACT_ARRAYS: + try: + assert_exact(f"{case_id}/{key}", fresh[key], archived[key]) + except AssertionError as exc: + failures.append(str(exc)) + + # Two tolerances, from two different rows of the record: the endpoints a result is read + # for (1e-12) and the series they come from (1e-10). Applying the endpoint number to a + # whole series is the mistake this harness made on its first real run. + for name in _GAS_KEYS: + index = species.index(name) # BY NAME + series_fresh, series_archived = fresh["x"][:, index], archived["x"][:, index] + for label, deviation in endpoint_deviations(series_fresh, series_archived).items(): + observed[f"{case_id}/{name} ({label})"] = (deviation, RTOL_HEADLINE) + if deviation > RTOL_HEADLINE: + failures.append( + f"{case_id}/{name} {label}: {deviation:.3e} > {RTOL_HEADLINE:.0e}" + ) + worst = worst_relative_deviation(series_fresh, series_archived) + observed[f"{case_id}/{name} (series)"] = (worst, RTOL_SERIES) + if worst > RTOL_SERIES: + failures.append(f"{case_id}/{name} series: {worst:.3e} > {RTOL_SERIES:.0e}") + + for key in _SERIES_KEYS: + for label, deviation in endpoint_deviations(fresh[key], archived[key]).items(): + observed[f"{case_id}/{key} ({label})"] = (deviation, RTOL_HEADLINE) + if deviation > RTOL_HEADLINE: + failures.append( + f"{case_id}/{key} {label}: {deviation:.3e} > {RTOL_HEADLINE:.0e}" + ) + worst = worst_relative_deviation(fresh[key], archived[key]) + observed[f"{case_id}/{key} (series)"] = (worst, RTOL_SERIES) + if worst > RTOL_SERIES: + failures.append(f"{case_id}/{key} series: {worst:.3e} > {RTOL_SERIES:.0e}") + + worst = worst_relative_deviation(fresh["dNdlogDp"], archived["dNdlogDp"]) + observed[f"{case_id}/dNdlogDp (per bin)"] = (worst, RTOL_SIZE_DISTRIBUTION) + if worst > RTOL_SIZE_DISTRIBUTION: + failures.append(f"{case_id}/dNdlogDp: {worst:.3e} > {RTOL_SIZE_DISTRIBUTION:.0e}") + + print(f"\n{'quantity':52s} {'worst rel':>11s} tolerance") + for label, (deviation, tolerance) in sorted(observed.items()): + print(f"{label:52s} {deviation:11.3e} {tolerance:.0e}") + + assert not failures, ( + "Tier-B reproduction deviates beyond the MEASURED tolerances:\n " + + "\n ".join(failures) + + "\n\nThese tolerances were measured, not chosen (see REFERENCE_TOLERANCES.md in this " + "directory). A toolchain bump moves them: re-run measure_deviation.py, update that record " + "with the new SHAs, and decide whether the new deviation is acceptable. Do not widen the " + "constants to make this pass." + ) + + +@pytest.mark.tier_b +def test_the_runs_are_physically_recognisable(reproduced: dict[str, dict[str, Any]]) -> None: + """The floor under the tolerances: a tolerance test cannot tell that a run did nothing.""" + for case_id, pair in sorted(reproduced.items()): + fresh = pair["fresh"] + species = [str(name) for name in fresh["species"]] + so2 = fresh["x"][:, species.index("SO2")] + assert so2[-1] < so2[0], f"{case_id}: SO2 must be consumed" + assert fresh["x"][:, species.index("H2SO4")].max() > 0.0, f"{case_id}: H2SO4 must form" + assert fresh["total_n"].max() > 0.0, f"{case_id}: particles must form" + assert fresh["V_ratio"][-1] > 1.0, f"{case_id}: the plume must expand" diff --git a/studio/tests/golden/tolerances.py b/studio/tests/golden/tolerances.py new file mode 100644 index 0000000..e70f260 --- /dev/null +++ b/studio/tests/golden/tolerances.py @@ -0,0 +1,173 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The measured tolerances, as code, and the comparison that applies them. + +**Every number here was measured before it was asserted** and is traceable to +``REFERENCE_TOLERANCES.md`` in this directory, which records the SHAs, the environment and the +per-quantity deviations it came from (task 0.7, issue #70). None of them is a value somebody chose +because a test failed. + +The rule that follows: **if an assertion here starts failing, re-run the measurement and update the +record — do not widen the number.** A tolerance widened to make a test pass is a test that no longer +tests anything, and this file exists to make that a visible edit rather than a quiet one. + +Two subtleties the measurement turned up, both encoded below rather than left to whoever writes the +next assertion: + +* **Near-zero series must be floored.** Unguarded relative error reaches 4.24e+04 on night-time + ``O1D``/``O`` at O(1e-35) molec cm^-3 oscillating about zero -- the archived value is literally + negative on a species that peaks around 3 molec cm^-3. Comparing only samples above + ``1e-6 x the series' own peak`` is what makes the comparison mean anything. +* **``dp_mid_um`` is not exact.** The archive writes ``10**(0.5*(log10 a + log10 b))``; Studio + writes ``sqrt(a*b)`` (task 0.5, deliberately). Algebraically identical, ~8e-16 apart in float64. +""" + +from __future__ import annotations + +from typing import Final + +import numpy as np +import numpy.typing as npt + +#: Final and peak values of the quantities a result is actually read for. Measured <= 2.9e-14 across +#: both reference cases; 1e-12 is ~35x headroom and still ~6 orders below any meaningful physics +#: change, so a real regression cannot hide under it. +RTOL_HEADLINE: Final = 1e-12 + +#: Any stored series, compared sample by sample. Measured worst 3.4e-12 (H2SO4, day 1.34); 1e-10 +#: covers the worst trace species (Cl2, 1.1e-11) with ~10x margin. +RTOL_SERIES: Final = 1e-10 + +#: Per-bin size distribution. Per-bin conditioning is worse than integrated number, so it gets its +#: own looser number; measured worst bin 2.0e-12. +RTOL_SIZE_DISTRIBUTION: Final = 1e-10 + +#: Photolysis J. Measured 1.09e-13 and identical in both reference cases -- TUV-x is the most +#: reproducible part of the pipeline. +RTOL_PHOTOLYSIS: Final = 1e-11 + +#: Dry bin mid-points. NOT exact: see the module docstring. A few ULP, measured 8.1e-16. +RTOL_BIN_EDGES: Final = 1e-15 + +#: Arrays measured bit-identical, because they are analytic rather than integrated. Any difference +#: at all is a real bug, so these are compared exactly and deliberately have no tolerance. +EXACT_ARRAYS: Final = ("t", "V_ratio", "T") + +#: A sample is compared only if it exceeds this fraction of its own series' peak. Below it, relative +#: error is meaningless (see the module docstring); an absolute-floor assertion would be the way to +#: cover those, and is not attempted here rather than being faked. +NEAR_ZERO_FRACTION_OF_PEAK: Final = 1e-6 + +#: Species excluded outright: they spend most of the run at O(1e-35) and oscillate about zero, so +#: even the floor above leaves too few comparable samples to mean anything. +EXCLUDED_SPECIES: Final = ("O1D", "O") + +FloatArray = npt.NDArray[np.float64] + + +def worst_relative_deviation( + fresh: npt.ArrayLike, reference: npt.ArrayLike, *, floor_by_peak: bool = True +) -> float: + """Worst relative deviation between two series, ignoring samples below the near-zero floor. + + Returns 0.0 when nothing is comparable, rather than NaN: a series entirely below its own floor + carries no information either way, and propagating NaN into an assertion would fail for the + wrong reason. + """ + a = np.asarray(reference, dtype=np.float64) + b = np.asarray(fresh, dtype=np.float64) + if a.shape != b.shape: + raise ValueError(f"shape mismatch: reference {a.shape} vs fresh {b.shape}") + magnitude = np.abs(a) + if floor_by_peak: + peak = float(np.nanmax(magnitude)) if magnitude.size else 0.0 + comparable = magnitude > peak * NEAR_ZERO_FRACTION_OF_PEAK + else: + comparable = magnitude > 0.0 + comparable &= np.isfinite(a) & np.isfinite(b) + if not np.any(comparable): + return 0.0 + return float(np.max(np.abs(b[comparable] - a[comparable]) / magnitude[comparable])) + + +def assert_series_matches( + name: str, fresh: npt.ArrayLike, reference: npt.ArrayLike, rtol: float +) -> None: + """Compare a series, or raise with the measured number and where to look. + + The message names the tolerance's provenance on purpose: the first instinct on a failure here is + to widen the number, and the right response is to re-measure. + """ + worst = worst_relative_deviation(fresh, reference) + if worst > rtol: + raise AssertionError( + f"{name}: worst relative deviation {worst:.3e} exceeds {rtol:.0e}.\n" + f"This tolerance was MEASURED (see REFERENCE_TOLERANCES.md in this directory), not " + f"chosen. Re-run the measurement and update that record with the new SHAs -- do not " + f"widen this number to make the test pass." + ) + + +def endpoint_deviations(fresh: npt.ArrayLike, reference: npt.ArrayLike) -> dict[str, float]: + """Relative deviation of the FINAL and PEAK values of a series. + + Separate from :func:`worst_relative_deviation` because the record gives these two a tighter + tolerance than the series they come from -- ``1e-12`` against ``1e-10``. Conflating them is an + easy mistake with a misleading symptom: it looks like a reproduction failure when it is a test + reading the wrong row. + """ + a = np.asarray(reference, dtype=np.float64) + b = np.asarray(fresh, dtype=np.float64) + if a.shape != b.shape: + raise ValueError(f"shape mismatch: reference {a.shape} vs fresh {b.shape}") + out: dict[str, float] = {} + for label, reference_value, fresh_value in ( + ("final", float(a[-1]), float(b[-1])), + ("peak", float(np.nanmax(a)), float(np.nanmax(b))), + ): + if reference_value == 0.0: + continue # a zero endpoint has no relative deviation; the series check still covers it + out[label] = abs(fresh_value - reference_value) / abs(reference_value) + return out + + +def assert_headline_matches(name: str, fresh: npt.ArrayLike, reference: npt.ArrayLike) -> None: + """Final and peak at ``RTOL_HEADLINE``. What a result is actually read for.""" + for label, deviation in endpoint_deviations(fresh, reference).items(): + if deviation > RTOL_HEADLINE: + raise AssertionError( + f"{name} ({label}): relative deviation {deviation:.3e} exceeds " + f"{RTOL_HEADLINE:.0e}. This is the ENDPOINT tolerance; the series that produced it " + f"has its own, looser one ({RTOL_SERIES:.0e}). Both were measured -- see " + f"REFERENCE_TOLERANCES.md in this directory." + ) + + +def assert_exact(name: str, fresh: npt.ArrayLike, reference: npt.ArrayLike) -> None: + """Bit-for-bit, for the analytic arrays. Any difference is a real bug.""" + a = np.asarray(reference) + b = np.asarray(fresh) + if not np.array_equal(a, b): + differing = int(np.sum(a != b)) if a.shape == b.shape else -1 + raise AssertionError( + f"{name} is expected to be bit-identical (analytic, not integrated) but " + f"{differing} element(s) differ. Worst relative deviation " + f"{worst_relative_deviation(b, a, floor_by_peak=False):.3e}." + ) + + +__all__ = [ + "EXACT_ARRAYS", + "EXCLUDED_SPECIES", + "NEAR_ZERO_FRACTION_OF_PEAK", + "RTOL_BIN_EDGES", + "RTOL_HEADLINE", + "RTOL_PHOTOLYSIS", + "RTOL_SERIES", + "RTOL_SIZE_DISTRIBUTION", + "assert_exact", + "assert_headline_matches", + "assert_series_matches", + "endpoint_deviations", + "worst_relative_deviation", +] From 1a864fe1f3f2e67cb55ea2dc5b1465577f34c2d0 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 16:22:19 -0700 Subject: [PATCH 13/18] studio: provenance records, written before the run (task 0.9a) (#81) * studio: provenance records, written before the run (task 0.9a) 0.9 is not one PR -- the exit criteria need FastAPI, SQLAlchemy/Alembic, a CLI, React/Vite with SSE and a figure -- so it is split, and this is the piece nothing implemented and the exit criteria depend on: a stored result with full provenance (ADR-006). A record pins config_hash, studio.__version__, the SANDBOX SHA, all three submodule SHAs, whether each checkout was dirty and which paths, and the resolved post-derivation parameter set -- what the model actually received rather than what the user typed -- plus any override with the value in force. datasets is present and empty rather than omitted so its emptiness is never ambiguous. This is the one place Studio shells out to git and it is strict: not-a-checkout, git missing, a repo with no commits, or a missing submodule all raise, because an empty SHA in a provenance record is worse than no record -- it looks like an answer. Cleanliness comes from status --porcelain rather than diff --quiet, so an untracked file counts; an untracked module that a run imported is exactly what makes a SHA a lie. The runner writes it at submit, before the process starts, so a run that dies in minute three of four still says what produced it. The test asserts against the record submit() returns rather than after wait(), since checking afterwards would pass even if it were written at completion. Verified end to end. Three times here a test failed and the code was right, each time because my expectation of git was wrong: a nested repo is not a registered submodule (the parent sees it as untracked and reads dirty, so the fixture now uses git submodule add); a dirty submodule flags both it and the parent, which is git being helpful because an edited submodule then cannot hide behind a clean-looking SANDBOX; and local-path submodules need protocol.file.allow=always. The tests build real git repositories rather than mocking subprocess -- the module is a thin shell over git's behaviour, so a mocked git would test the mock. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * studio: the runner records a checkout it is given, so its tests run in CI CI caught what local tests could not: making provenance mandatory at submit means the runner needs a pinnable checkout, and CI checks out no submodules, so every runner submit test failed there while passing locally. LocalSubprocessRunner now takes repo_root -- which checkout to record -- defaulting to the one the code came from. This does not weaken the guarantee: a run still cannot start unless the checkout it names can be pinned. And "which checkout produced this?" is a question a runner genuinely has to answer; a worker executing code from elsewhere would answer it differently. The synthetic-checkout builder moves from the provenance tests into conftest, since the runner tests need it too. Both suites now run in CI rather than skipping: 40 tests across the two, no submodules required. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 51 +++++- studio/modelio/provenance.py | 219 ++++++++++++++++++++++++++ studio/runner/base.py | 3 + studio/runner/local.py | 18 +++ studio/tests/conftest.py | 62 ++++++++ studio/tests/unit/test_provenance.py | 224 +++++++++++++++++++++++++++ studio/tests/unit/test_runner.py | 55 ++++++- 7 files changed, 629 insertions(+), 3 deletions(-) create mode 100644 studio/modelio/provenance.py create mode 100644 studio/tests/unit/test_provenance.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 352cfc8..ea4f92f 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -22,13 +22,62 @@ with full provenance, and the golden tests pass. | 0.6 `studio/runner` + job lifecycle | **done** (#72) | | 0.7 Golden-file harness (two tiers) | **done** (#70 measured, #79 asserted) | | 0.8 Four contained fixes in `coupled/` | not started | -| 0.9 Vertical slice: CLI + API + minimal UI | not started | +| 0.9 Vertical slice: CLI + API + minimal UI | **in progress** — 0.9a provenance (#80); 0.9b–e to come | Task order note: 0.5 was taken **before 0.3**, so the dependency-graph engine has real derivations to resolve rather than fixtures. --- +### 2026-08-14 — Task 0.9a: provenance records (issue #80) + +**0.9 is not one PR.** The exit criteria need FastAPI + SQLAlchemy/Alembic on SQLite, a CLI, +React/Vite with SSE, and a figure from `RunSummary`. Split into 0.9a (this), 0.9b persistence, 0.9c +CLI, 0.9d API, 0.9e UI + figure. This is the piece nothing implemented and the exit criteria depend +on: *"produces a stored result **with full provenance**"*. + +`studio/modelio/provenance.py`, written by the runner **at submit time**. 17 new Tier-A tests +(213 total). + +**What a record pins**: `config_hash`, `studio.__version__`, the SANDBOX SHA, **all three submodule +SHAs**, whether each checkout was dirty (with the offending paths), and the **resolved, +post-derivation** parameter set — what the model actually received, not what the user typed. Plus any +override with the value in force. `datasets` is present and empty rather than omitted, so its +emptiness is never ambiguous. + +**This is the one place Studio shells out to git, and it is strict about it.** Not-a-checkout, +git-not-installed, a repo with no commits, or a missing submodule all **raise**: an empty SHA in a +provenance record is worse than no record, because it looks like an answer (ADR-005). Cleanliness +comes from `status --porcelain`, not `diff --quiet`, so an **untracked** file counts — an untracked +module that a run imported is exactly what makes a SHA a lie. + +**Written before execution, and proven so.** The test asserts against the record `submit()` returns, +not after `wait()` — checking afterwards would pass even if it were written at completion. Verified +end to end: at submit the work dir holds `input.json` + `provenance.json`; on completion, six +artifacts including `state.npz` and `summary.json`. + +**Three times in this task the tests failed and the code was right.** Each was my expectation of git +being wrong, and each is documented where it will be re-read: + +1. A *nested* repo is not a *registered* submodule — the parent reports the nested one as untracked + and so reads dirty. The fixture was lying about the shape of a real checkout; it now uses + `git submodule add`. +2. A dirty submodule flags **both** it and the parent, because the parent's recorded pointer no + longer matches the working tree. That is git being helpful: an edited submodule cannot hide behind + a clean-looking SANDBOX. +3. `protocol.file.allow=always` is needed for local-path submodules (CVE-2022-39253). + +The tests build **real git repositories** rather than mocking `subprocess`: the module is a thin +shell over git's behaviour, so a mocked git would test the mock. ~1 s, worth it. + +**A fourth thing CI caught that local tests could not.** Making provenance mandatory at submit means +the runner now needs a pinnable checkout — and CI checks out no submodules, so every runner submit +test failed there while passing locally. The fix is a `repo_root` parameter on the runner (which +checkout to record), pointed at the shared synthetic-checkout fixture in the tests. It does **not** +weaken the guarantee: a run still cannot start unless the checkout it names can be pinned, and the +production default is the real one. "Which checkout produced this?" is a question a runner genuinely +has to answer — a worker executing code from elsewhere would answer it differently. + ### 2026-08-14 — Task 0.7 (second half): the two-tier golden harness The assertions, built on the tolerances #70 measured and #76 corrected. `studio/tests/golden/`: diff --git a/studio/modelio/provenance.py b/studio/modelio/provenance.py new file mode 100644 index 0000000..2959e95 --- /dev/null +++ b/studio/modelio/provenance.py @@ -0,0 +1,219 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""What produced a result: config identity, app version, model version (ADR-006). + +The requirement is that **any figure traces to an exact configuration, model version and input +dataset checksum**. What existed before this module was a hand-composed case ID -- good design for a +fixed factorial, and genuinely useful, but it captures the *axes* rather than the resolved +configuration: everything held fixed across the 810-run ensemble (``ion_pair_rate=30``, +``day_of_year=172``, the whole background composition) is invisible in it, so two ensembles +differing only in a "fixed" value collide. + +This is **the one place Studio shells out to git**, and it is deliberately strict about it: + +* "Not a git checkout" **raises**. An empty SHA in a provenance record is worse than no record at + all, because it looks like an answer (ADR-005). +* **A dirty working tree is recorded and flags the run.** Uncommitted changes mean the SHA does not + describe the code that ran, and that is exactly the case where someone later wants to know. +* Submodule SHAs are read individually. The SANDBOX SHA alone does not pin the model: the three + submodules are what contain it (ADR-001), and a submodule pointer that has moved without a commit + here is invisible in the parent SHA. + +The record is written **before execution begins** -- a run that dies in minute three still has its +provenance -- and is never mutated. An edited config is a new config and a new run (ADR-004). + +Note what this module does NOT do: it does not import ``coupled``. Model *identity* is a question +about the checkout, not about the model's API, so recording it costs nothing. +""" + +from __future__ import annotations + +import subprocess +from datetime import UTC, datetime +from pathlib import Path +from typing import Any, Final + +from pydantic import BaseModel, ConfigDict, Field + +import studio +from studio.resolve import ResolvedConfig + +#: The submodules that, together with the SANDBOX SHA, pin the model exactly (ADR-001). +MODEL_SUBMODULES: Final[tuple[str, ...]] = ("tuvx-jax", "stratchem-jax", "tomas-jax") + +#: Bumped if the record's shape changes, so a stored record is never read under new semantics. +PROVENANCE_SCHEMA_VERSION: Final = "0.1.0" + + +class NotAGitCheckoutError(RuntimeError): + """The repository root is not a git checkout, so the model cannot be pinned. + + Raised rather than recording an empty SHA: a provenance record that cannot say what ran is not a + provenance record, and one that says ``""`` looks like it can. + """ + + +class GitCheckout(BaseModel): + """The state of one checkout: its commit, and whether it was clean.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + path: str + commit: str + #: True when ``git status --porcelain`` was non-empty. The commit then does NOT describe the + #: code that ran, which is precisely when someone will want to know. + dirty: bool + #: The porcelain output, truncated. Enough to see WHAT was uncommitted without storing a diff. + dirty_files: tuple[str, ...] = () + + +class ProvenanceRecord(BaseModel): + """Everything needed to say what produced a result. Written before the run, never mutated.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + schema_version: str = PROVENANCE_SCHEMA_VERSION + recorded_at: datetime + #: Identity of the resolved config (ADR-006). Doubles as the cache key. + config_hash: str + #: The orchestration layer's version. The model has none -- it is pinned by SHA alone. + studio_version: str + #: The SANDBOX checkout: the coupling layer plus the paper pipeline. + sandbox: GitCheckout + #: The three submodules that contain the model itself. + submodules: dict[str, GitCheckout] + #: Input datasets consulted, by identifier -> checksum. Empty in Phase 0: the schema carries no + #: dataset inputs yet. Present and empty rather than omitted, so its absence is never ambiguous. + datasets: dict[str, str] = Field(default_factory=dict) + #: The RESOLVED, post-derivation parameter set -- what the model actually received, not what the + #: user typed. This is the field that makes the record self-contained. + resolved_config: dict[str, Any] + #: Any derived field the user overrode, with the value that was in force. + overrides: dict[str, Any] = Field(default_factory=dict) + + @property + def is_reproducible(self) -> bool: + """True when every checkout was clean, so the SHAs fully describe the code that ran.""" + return not self.sandbox.dirty and not any(sub.dirty for sub in self.submodules.values()) + + @property + def dirty_checkouts(self) -> tuple[str, ...]: + """Names of checkouts with uncommitted changes. Empty when the run is reproducible.""" + names = ["SANDBOX"] if self.sandbox.dirty else [] + names.extend(name for name, sub in sorted(self.submodules.items()) if sub.dirty) + return tuple(names) + + def write(self, path: Path) -> Path: + """Write as JSON. Called before the run starts.""" + path.write_text(self.model_dump_json(indent=2), encoding="utf-8") + return path + + @classmethod + def read(cls, path: Path) -> ProvenanceRecord: + return cls.model_validate_json(path.read_text(encoding="utf-8")) + + +def _git(repo_root: Path, *args: str) -> str: + """Run git in ``repo_root`` and return stdout, or raise. + + Raises: + NotAGitCheckoutError: If git fails for any reason -- not installed, not a checkout, a broken + submodule. All of them mean the same thing here: the code that ran cannot be identified. + """ + try: + proc = subprocess.run( + ["git", *args], + cwd=str(repo_root), + capture_output=True, + text=True, + check=False, + ) + except FileNotFoundError as exc: # git itself is missing + raise NotAGitCheckoutError( + f"git is not available, so the model cannot be pinned for a run in {repo_root}. " + f"Provenance is required (ADR-006); it is not optional metadata." + ) from exc + if proc.returncode != 0: + raise NotAGitCheckoutError( + f"`git {' '.join(args)}` failed in {repo_root}: {proc.stderr.strip() or 'no output'}. " + f"A provenance record with no commit is not a provenance record, so this raises rather " + f"than recording an empty SHA." + ) + return proc.stdout.strip() + + +def describe_checkout(path: Path, *, dirty_file_limit: int = 20) -> GitCheckout: + """The commit and cleanliness of the checkout at ``path``. + + ``git status --porcelain`` rather than ``diff --quiet`` because it also reports untracked + files, and an untracked module that a run imported is exactly the kind of thing that makes a + SHA a lie. + """ + commit = _git(path, "rev-parse", "HEAD") + status = _git(path, "status", "--porcelain") + lines = tuple(line.strip() for line in status.splitlines() if line.strip()) + return GitCheckout( + path=str(path), + commit=commit, + dirty=bool(lines), + dirty_files=lines[:dirty_file_limit], + ) + + +def repository_root() -> Path: + """The SANDBOX root, derived from this file's location rather than the working directory. + + Deliberately not ``Path.cwd()``: a run submitted by an API process started anywhere at all must + still record the checkout that the code came from. + """ + return Path(studio.__file__).resolve().parent.parent + + +def record_for( + config: ResolvedConfig, + *, + repo_root: Path | None = None, + datasets: dict[str, str] | None = None, +) -> ProvenanceRecord: + """Build the record for a config, at submit time. + + Raises: + InconsistentConfigError: If the config has stale overrides. Recording provenance for a + config whose numbers do not follow from each other would give an inconsistent run a + respectable-looking pedigree. + NotAGitCheckoutError: If any checkout cannot be identified. + """ + config.require_consistent() + root = repo_root or repository_root() + submodules = {} + for name in MODEL_SUBMODULES: + path = root / name + if not (path / ".git").exists(): + raise NotAGitCheckoutError( + f"submodule {name} is not checked out at {path} (`git submodule update --init`). " + f"The SANDBOX SHA alone does not pin the model: the three submodules are what " + f"contain it (ADR-001)." + ) + submodules[name] = describe_checkout(path) + return ProvenanceRecord( + recorded_at=datetime.now(UTC), + config_hash=config.config.config_hash(), + studio_version=studio.__version__, + sandbox=describe_checkout(root), + submodules=submodules, + datasets=dict(datasets or {}), + resolved_config=config.config.model_dump(mode="json"), + overrides={path: record.value for path, record in sorted(config.overrides.items())}, + ) + + +__all__ = [ + "MODEL_SUBMODULES", + "PROVENANCE_SCHEMA_VERSION", + "GitCheckout", + "NotAGitCheckoutError", + "ProvenanceRecord", + "describe_checkout", + "record_for", + "repository_root", +] diff --git a/studio/runner/base.py b/studio/runner/base.py index 5750e01..d7350a8 100644 --- a/studio/runner/base.py +++ b/studio/runner/base.py @@ -116,6 +116,9 @@ class JobRecord(BaseModel): #: that dies early still says where to look. work_dir: Path | None = None input_path: Path | None = None + #: The provenance record (ADR-006), written at submit time -- before execution -- so a run that + #: dies in minute three of four still says exactly what produced it. + provenance_path: Path | None = None stdout_path: Path | None = None stderr_path: Path | None = None exit_code: int | None = None diff --git a/studio/runner/local.py b/studio/runner/local.py index d81943d..2349557 100644 --- a/studio/runner/local.py +++ b/studio/runner/local.py @@ -15,6 +15,8 @@ What is on disk when a job ends, whatever the outcome: * ``input.json`` -- the RESOLVED config that was actually run +* ``provenance.json`` -- what produced it: config hash, app version, SANDBOX and submodule SHAs, + and whether any checkout was dirty (ADR-006). Written BEFORE the process starts. * ``stdout.log`` / ``stderr.log`` -- captured in full * the exit code and every state transition, in the record @@ -33,6 +35,7 @@ from pathlib import Path from typing import Any, Final +from studio.modelio.provenance import record_for from studio.resolve import ResolvedConfig from studio.runner.base import JobRecord, JobRegistry, JobState @@ -65,6 +68,12 @@ class LocalSubprocessRunner: entry_module: The module launched with ``-m``. Overridable so the LIFECYCLE can be tested without a four-minute model run -- the default is the real path, and nothing in this class branches on the value. It is a parameter, not a test hook. + repo_root: Which checkout to record in each run's provenance. Defaults to the one this code + came from, which is what a local runner should record. It is a parameter because + "which checkout produced this?" is a real question a runner has to answer -- a worker + executing code from elsewhere would answer it differently -- and because CI has no + submodules, so the tests point it at a synthetic checkout. **It does not weaken the + guarantee**: a run still cannot start unless the checkout it names can be pinned. python_executable: Interpreter for the subprocess; defaults to the current one, so a job inherits the environment that submitted it rather than whatever is first on PATH. """ @@ -76,6 +85,7 @@ def __init__( max_workers: int = DEFAULT_MAX_WORKERS, entry_module: str = "studio.cli.run", python_executable: str | None = None, + repo_root: Path | None = None, ) -> None: if max_workers < 1: raise ValueError(f"max_workers must be >= 1, got {max_workers}") @@ -84,6 +94,7 @@ def __init__( self.max_workers = max_workers self.entry_module = entry_module self.python_executable = python_executable or sys.executable + self.repo_root = repo_root self.registry = JobRegistry() self._pool = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="studio-run") self._processes: dict[str, subprocess.Popen[bytes]] = {} @@ -107,12 +118,19 @@ def submit(self, config: ResolvedConfig, *, label: str = "") -> JobRecord: input_path = work_dir / "input.json" input_path.write_text(config.model_dump_json(indent=2), encoding="utf-8") + # Provenance BEFORE execution (ADR-006). Deliberately not in a try/except: if the model + # cannot be pinned, the run must not start. A result whose origin is unknown is worth less + # than no result, because it looks like the others. + provenance = record_for(config, repo_root=self.repo_root) + provenance_path = provenance.write(work_dir / "provenance.json") + record = JobRecord( job_id=job_id, config_hash=config.config.config_hash(), label=label, work_dir=work_dir, input_path=input_path, + provenance_path=provenance_path, stdout_path=work_dir / "stdout.log", stderr_path=work_dir / "stderr.log", ).transition_to(JobState.QUEUED, detail=f"queued for {self.entry_module}") diff --git a/studio/tests/conftest.py b/studio/tests/conftest.py index 0e7ecca..68a6ce8 100644 --- a/studio/tests/conftest.py +++ b/studio/tests/conftest.py @@ -13,6 +13,7 @@ from __future__ import annotations +import subprocess from pathlib import Path import pytest @@ -44,3 +45,64 @@ def paper_ensemble_runs(repo_root: Path) -> Path: f"docs/studio/adr/ADR-009-golden-file-strategy.md" ) return runs + + +def _git(path: Path, *args: str) -> None: + """Run git in ``path`` with an identity, so commits work on a machine with no global config.""" + subprocess.run( + [ + "git", + "-c", + "user.email=test@example.invalid", + "-c", + "user.name=Test", + "-c", + "commit.gpgsign=false", + *args, + ], + cwd=str(path), + check=True, + capture_output=True, + ) + + +def _make_repo(path: Path, filename: str = "file.txt") -> Path: + """A real git repo with one commit.""" + path.mkdir(parents=True, exist_ok=True) + _git(path, "init", "--quiet") + (path / filename).write_text("content\n", encoding="utf-8") + _git(path, "add", filename) + _git(path, "commit", "--quiet", "-m", "initial") + return path + + +@pytest.fixture +def fake_sandbox(tmp_path: Path) -> Path: + """A SANDBOX-shaped tree: a root repo with the three model submodules REGISTERED as such. + + Shared because both the provenance tests and the runner tests need a pinnable checkout, and CI + checks out no submodules -- so pointing at a synthetic one is what lets those tests run + everywhere rather than skipping in CI. + + Registered rather than merely nested: a nested repo the parent does not know about shows up in + ``git status --porcelain`` as untracked, so the parent reads dirty. + ``protocol.file.allow=always`` is required for local-path submodules (CVE-2022-39253); safe + here, since the "remote" is a directory the test just created. + """ + from studio.modelio.provenance import MODEL_SUBMODULES + + root = _make_repo(tmp_path / "SANDBOX") + for name in MODEL_SUBMODULES: + origin = _make_repo(tmp_path / "origins" / name) + _git( + root, + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "--quiet", + str(origin), + name, + ) + _git(root, "commit", "--quiet", "-m", "add submodules") + return root diff --git a/studio/tests/unit/test_provenance.py b/studio/tests/unit/test_provenance.py new file mode 100644 index 0000000..8582eef --- /dev/null +++ b/studio/tests/unit/test_provenance.py @@ -0,0 +1,224 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Provenance records (ADR-006). + +These build **real git repositories** in a temp directory rather than mocking ``subprocess``. +The module is a thin shell around git's behaviour, so a mocked git would test the mock: whether +``status --porcelain`` reports an untracked file, whether a missing ``.git`` fails as expected, +whether ``rev-parse`` in a fresh repo with no commits errors — those are the questions, and only +git answers them. + +Cost: ~1 s for a handful of ``git init`` calls. Cheap enough for Tier A, and it means the dirty-tree +and not-a-checkout paths are genuinely exercised rather than asserted about a stub. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from studio.modelio.provenance import ( + MODEL_SUBMODULES, + PROVENANCE_SCHEMA_VERSION, + NotAGitCheckoutError, + ProvenanceRecord, + describe_checkout, + record_for, + repository_root, +) +from studio.resolve import ResolvedConfig, apply_change, resolve, set_override +from studio.schema import RunConfig + + +@pytest.fixture +def resolved() -> ResolvedConfig: + return resolve(RunConfig()) + + +@pytest.mark.tier_a +def test_the_record_carries_every_field_adr_006_requires( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """config hash, app version, SANDBOX SHA, three submodule SHAs, datasets, resolved config.""" + record = record_for(resolved, repo_root=fake_sandbox) + + assert record.schema_version == PROVENANCE_SCHEMA_VERSION + assert record.config_hash == resolved.config.config_hash() + assert record.studio_version # studio.__version__, whatever it currently is + assert len(record.sandbox.commit) == 40 + assert set(record.submodules) == set(MODEL_SUBMODULES) + assert all(len(sub.commit) == 40 for sub in record.submodules.values()) + assert record.datasets == {}, "present and empty, so its absence is never ambiguous" + assert record.recorded_at.tzinfo is not None, "a naive timestamp compares wrongly across zones" + + +@pytest.mark.tier_a +def test_the_recorded_config_is_the_resolved_one( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """ "What the model actually received", not what the user typed. + + The derived fields are the test: a record of the user's inputs would have ``None`` here, and + would not let anyone reconstruct the run. + """ + record = record_for(resolved, repo_root=fake_sandbox) + injection = record.resolved_config["injection"] + assert injection["plume_volume_cm3"] == 1.5e12 + assert injection["so2_initial_pptv"] == pytest.approx(3.309115922996412e9, rel=1e-15) + assert record.resolved_config["schema_version"] == resolved.config.schema_version + + +@pytest.mark.tier_a +def test_an_override_is_recorded_with_the_value_in_force(fake_sandbox: Path) -> None: + """A user-supplied derived value must be visible as an override, not silently indistinguishable + from a computed one.""" + from studio.resolve import keep_override + + overridden = keep_override( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "injection.so2_initial_pptv", + ) + record = record_for(overridden, repo_root=fake_sandbox) + assert record.overrides == {"injection.so2_initial_pptv": 5.0e9} + assert record.resolved_config["injection"]["so2_initial_pptv"] == 5.0e9 + + +@pytest.mark.tier_a +def test_a_clean_tree_is_reproducible(fake_sandbox: Path, resolved: ResolvedConfig) -> None: + record = record_for(resolved, repo_root=fake_sandbox) + assert record.is_reproducible + assert record.dirty_checkouts == () + + +@pytest.mark.tier_a +def test_a_dirty_sandbox_is_recorded_and_flags_the_run( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """The SHA no longer describes the code that ran -- exactly when someone wants to know.""" + (fake_sandbox / "file.txt").write_text("modified\n", encoding="utf-8") + record = record_for(resolved, repo_root=fake_sandbox) + + assert record.sandbox.dirty + assert record.sandbox.dirty_files, "the record must say WHAT was uncommitted" + assert not record.is_reproducible + assert record.dirty_checkouts == ("SANDBOX",) + + +@pytest.mark.tier_a +def test_an_untracked_file_counts_as_dirty(fake_sandbox: Path, resolved: ResolvedConfig) -> None: + """``status --porcelain`` rather than ``diff --quiet``, on purpose. + + An untracked module that a run imported is exactly the kind of thing that makes a SHA a lie, and + ``git diff`` would not see it. + """ + (fake_sandbox / "scratch_module.py").write_text("x = 1\n", encoding="utf-8") + assert record_for(resolved, repo_root=fake_sandbox).sandbox.dirty + + +@pytest.mark.tier_a +def test_a_dirty_submodule_flags_the_run_and_the_parent( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """The model lives in the submodules; a dirty one means the model that ran is not any commit. + + **Both** checkouts are flagged, and that is git being helpful rather than the record being + imprecise: a registered submodule with a dirty working tree also shows up in the PARENT's + ``status --porcelain`` as modified, because the parent's recorded submodule pointer no longer + describes what is on disk. So an edited submodule cannot hide behind a clean-looking SANDBOX. + """ + (fake_sandbox / "stratchem-jax" / "file.txt").write_text("edited\n", encoding="utf-8") + record = record_for(resolved, repo_root=fake_sandbox) + assert record.submodules["stratchem-jax"].dirty + assert not record.is_reproducible + assert record.dirty_checkouts == ("SANDBOX", "stratchem-jax") + assert not record.submodules["tuvx-jax"].dirty, "only the edited submodule is dirty" + + +@pytest.mark.tier_a +def test_not_a_git_checkout_raises(tmp_path: Path, resolved: ResolvedConfig) -> None: + """An empty SHA looks like an answer, so this refuses to produce one (ADR-005).""" + plain = tmp_path / "not-a-repo" + plain.mkdir() + for name in MODEL_SUBMODULES: + (plain / name).mkdir() + with pytest.raises(NotAGitCheckoutError): + record_for(resolved, repo_root=plain) + + +@pytest.mark.tier_a +def test_a_missing_submodule_raises(tmp_path: Path, resolved: ResolvedConfig) -> None: + """The SANDBOX SHA alone does not pin the model (ADR-001), so a missing submodule is fatal.""" + from studio.tests.conftest import _make_repo + + root = _make_repo(tmp_path / "partial") + _make_repo(root / "tuvx-jax") # the other two are absent + with pytest.raises(NotAGitCheckoutError, match="submodule"): + record_for(resolved, repo_root=root) + + +@pytest.mark.tier_a +def test_a_repo_with_no_commits_raises(tmp_path: Path, resolved: ResolvedConfig) -> None: + """``rev-parse HEAD`` has nothing to report, which is a failure rather than an empty string.""" + from studio.tests.conftest import _git + + root = tmp_path / "empty" + root.mkdir() + _git(root, "init", "--quiet") + with pytest.raises(NotAGitCheckoutError): + describe_checkout(root) + + +@pytest.mark.tier_a +def test_a_stale_config_is_refused(fake_sandbox: Path) -> None: + """Recording provenance for an inconsistent config would give it a respectable pedigree.""" + from studio.resolve import InconsistentConfigError + + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError): + record_for(stale, repo_root=fake_sandbox) + + +@pytest.mark.tier_a +def test_the_record_is_immutable(fake_sandbox: Path, resolved: ResolvedConfig) -> None: + """Written once, before the run, never mutated (ADR-006).""" + record = record_for(resolved, repo_root=fake_sandbox) + with pytest.raises(ValueError, match="frozen"): + record.config_hash = "tampered" # type: ignore[misc] + + +@pytest.mark.tier_a +def test_the_record_round_trips_through_json( + fake_sandbox: Path, resolved: ResolvedConfig, tmp_path: Path +) -> None: + original = record_for(resolved, repo_root=fake_sandbox) + restored = ProvenanceRecord.read(original.write(tmp_path / "provenance.json")) + assert restored == original + assert restored.is_reproducible == original.is_reproducible + + +@pytest.mark.tier_a +def test_the_repository_root_is_derived_from_the_package_not_the_cwd() -> None: + """An API process started anywhere must still record the checkout the code came from.""" + root = repository_root() + assert (root / "studio").is_dir() + assert (root / "coupled").is_dir() + + +@pytest.mark.tier_a +def test_this_checkout_can_be_pinned(resolved: ResolvedConfig) -> None: + """The real repository, not a synthetic one: submodules present, SHAs readable. + + Skips where the submodules are absent, which is the same condition every other model-touching + test skips on -- including in CI. + """ + root = repository_root() + if not all((root / name / ".git").exists() for name in MODEL_SUBMODULES): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + record = record_for(resolved, repo_root=root) + assert len(record.sandbox.commit) == 40 + assert set(record.submodules) == set(MODEL_SUBMODULES) diff --git a/studio/tests/unit/test_runner.py b/studio/tests/unit/test_runner.py index 2b32885..b46ad9d 100644 --- a/studio/tests/unit/test_runner.py +++ b/studio/tests/unit/test_runner.py @@ -41,8 +41,19 @@ def resolved() -> ResolvedConfig: @pytest.fixture -def runner(tmp_path: Path) -> LocalSubprocessRunner: - made = LocalSubprocessRunner(tmp_path / "jobs", max_workers=2, entry_module=FIXTURE_MODULE) +def runner(tmp_path: Path, fake_sandbox: Path) -> LocalSubprocessRunner: + """A runner over the fixture entry point, recording provenance for a synthetic checkout. + + ``repo_root=fake_sandbox`` because submitting now writes a provenance record, which requires a + pinnable checkout -- and CI checks out no submodules. Pointing at a synthetic checkout is what + lets these tests run in CI rather than skipping; the production default is the real one. + """ + made = LocalSubprocessRunner( + tmp_path / "jobs", + max_workers=2, + entry_module=FIXTURE_MODULE, + repo_root=fake_sandbox, + ) yield made made.shutdown(cancel_running=True) @@ -369,3 +380,43 @@ def _with_sim_limit(days: float) -> ResolvedConfig: payload = RunConfig().model_dump() payload["termination"]["max_sim_time_days"] = days return resolve(RunConfig.model_validate(payload)) + + +class TestProvenanceIsWrittenAtSubmit: + """A run's provenance must exist before it can fail (ADR-006).""" + + @pytest.mark.tier_a + def test_provenance_exists_the_moment_submit_returns( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Before the process finishes -- so a run that dies in minute three still has it. + + Asserted against the record returned by ``submit()``, not after ``wait()``: the point is the + ordering, and checking afterwards would pass even if it were written at completion. + """ + from studio.modelio.provenance import ProvenanceRecord + + record = runner.submit(resolved, label="prov") + assert record.provenance_path is not None + assert record.provenance_path.is_file(), "written at submit, not at completion" + + provenance = ProvenanceRecord.read(record.provenance_path) + assert provenance.config_hash == resolved.config.config_hash() == record.config_hash + assert set(provenance.submodules) # the model is pinned, not just the app + runner.wait(record.job_id, timeout=30) + + @pytest.mark.tier_a + def test_a_failed_run_still_has_its_provenance( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The case that matters: four minutes in, exit 1, and someone asks what produced it.""" + _directive(monkeypatch, mode="fail", message="synthetic failure") + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + + assert final.state is JobState.FAILED + assert final.provenance_path is not None and final.provenance_path.is_file() + assert "provenance.json" in {p.name for p in runner.artifacts(final.job_id)} From 0e06f55dd3e84db9527f9dba14d7f460c7954d7c Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 16:24:03 -0700 Subject: [PATCH 14/18] studio: record what the archive comparison assumes, and correct two claims (#82) Ali confirmed that coupled/paper_ensemble/runs/ is a valid reproduction reference. That is now written into REFERENCE_TOLERANCES.md rather than left implicit in the harness, because it is an assumption the data cannot support on its own: those files carry no provenance record, which is the gap ADR-006 closes going forward and cannot close retroactively. Every other archive directory is excluded and the record now says why. runs_60day initialises from a spun-up control run; runs_bgstop*, runs_boxsize, runs_geo, runs_no_sai, runs_special and runs_start_time* were produced with different vintages and configurations. Rebuilding one of those from the axis tables would compare two different computations, where a pass is luck and a failure means nothing. All six curated cases are now measured rather than two: every endpoint within 4.1e-14 against 1e-12, every series within 1.6e-11 against 1e-10, t/V_ratio/T bit-identical throughout. The wider data corrected two claims. The record read as though deviation were tied to the early nucleation burst, because the two-case measurement found the worst H2SO4 deviation at day 1.34; across six cases the worst days are 5.83, 1.34, 9.27, 2.15, 3.03 and 5.74, with no common feature -- it is round-off scattered through the run rather than accumulation. And the size-distribution outlier is about sparsity rather than timing: the four sabr220 cases share an identical 1.63e-11 at one early cell while the sabr330 cases peak late and elsewhere, but every one of them is a bin holding 1.7-4.2 particles per cm3. Adds measure_all_cases.py and plot_fidelity.py as tools rather than tests. Re-measuring must never "fail" -- it reports, and a human decides whether the numbers are acceptable. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 37 +++ studio/tests/golden/REFERENCE_TOLERANCES.md | 61 ++++ studio/tests/golden/measure_all_cases.py | 98 ++++++ studio/tests/golden/plot_fidelity.py | 348 ++++++++++++++++++++ 4 files changed, 544 insertions(+) create mode 100644 studio/tests/golden/measure_all_cases.py create mode 100644 studio/tests/golden/plot_fidelity.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index ea4f92f..37cd98e 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -29,6 +29,43 @@ derivations to resolve rather than fixtures. --- +### 2026-08-14 — The archive's status as a reference, settled and written down + +Ali, 2026-08-14: **`coupled/paper_ensemble/runs/` is a valid reproduction reference.** Recorded in +`REFERENCE_TOLERANCES.md` rather than left as an implicit property of the harness, because it is an +assumption the data cannot support on its own — those files carry no provenance record, which is +exactly the gap ADR-006 closes going forward and cannot close retroactively. + +**Every other archive directory is excluded, and now says why**: `runs_60day` (initialises from a +spun-up control run), `runs_bgstop*`, `runs_boxsize`, `runs_geo`, `runs_no_sai`, `runs_special`, +`runs_start_time*`. They were produced differently — different initialisation, different vintages, +different configurations — so rebuilding one from the axis tables would compare two different +computations, where a pass is luck and a failure means nothing. The exclusion is structural: +`paper_cases.py` only maps the factorial's case IDs. + +**All six curated cases now measured** (the record previously had two): every endpoint ≤ 4.1e-14 +against `1e-12`, every series ≤ 1.6e-11 against `1e-10`, `t`/`V_ratio`/`T` bit-identical in all six, +256–293 s per case. + +**Two claims corrected by the wider data.** + +1. *Timing.* This record read as though deviation were tied to the early nucleation burst — the + two-case measurement had found the worst H₂SO₄ deviation at day 1.34. Across six cases the worst + days are 5.83, 1.34, 9.27, 2.15, 3.03, 5.74: **no common feature**. It is a flat ~1e-14 baseline + with occasional spikes — round-off scattered through the run, not accumulation. +2. *The size-distribution outlier.* The four `sabr220` cases share an identical 1.63e-11 at the same + cell (t = 0.042 d, bin 5, 4.218 counts); the two `sabr330` cases peak late and elsewhere (day 7.34 + bin 6; day 8.41 bin 11). The common thread is **sparsity, not timing** — every one is a bin + holding 1.7–4.2 particles cm⁻³. It is inherited from `n_cm3`, not from the `dlogdp` normalisation, + whose divisors agree to 6.1e-15. + +**Added**: `measure_all_cases.py` (re-measure all six, incrementally, asserting nothing) and +`plot_fidelity.py` (three figures: headroom against tolerance, deviation against time, and the +near-zero floor). Both are tools rather than tests — re-measuring must never "fail", it reports, and +a human decides. Figures are regenerable and not committed. + +--- + ### 2026-08-14 — Task 0.9a: provenance records (issue #80) **0.9 is not one PR.** The exit criteria need FastAPI + SQLAlchemy/Alembic on SQLite, a CLI, diff --git a/studio/tests/golden/REFERENCE_TOLERANCES.md b/studio/tests/golden/REFERENCE_TOLERANCES.md index 61fb553..c040e01 100644 --- a/studio/tests/golden/REFERENCE_TOLERANCES.md +++ b/studio/tests/golden/REFERENCE_TOLERANCES.md @@ -250,6 +250,67 @@ the correction in the table above — and it could only surface via the Studio p exist on the branch where the first measurement was taken. `dNdlogDp` inherits that difference at 3.2e-13, comfortably inside its own `1e-10`, so only the `dp_mid_um` row needed changing. +## What this comparison assumes about the archive — settled 2026-08-14 + +The archived files carry **no provenance record**, so the code that produced them is not knowable +from the data (that gap is exactly what ADR-006 closes going forward and explicitly cannot close +retroactively). This comparison therefore rests on an assumption, and it is written down rather than +left implicit: + +> **`coupled/paper_ensemble/runs/` is the uniform 810-run factorial produced by `run_ensemble.py`'s +> axis tables, and is a valid reproduction reference.** Confirmed by Ali, 2026-08-14. + +**Every other archive directory is excluded**, because they were produced differently — some +initialise from a spun-up control run (`runs_60day/` uses `frank_control_ic.json`), some predate that +practice, some carry different configurations: + +`runs_60day` · `runs_bgstop` · `runs_bgstop_ctrl` · `runs_boxsize` · `runs_geo` · `runs_no_sai` · +`runs_special` · `runs_start_time` · `runs_start_time_geo` + +Reproducing one of those from the axis tables would compare two different computations, where a pass +is luck and a failure means nothing. `paper_cases.py` only maps the factorial's case IDs, so this +exclusion is structural rather than a convention someone has to remember. + +Supporting evidence, for the record: all 810 `state.npz` in `runs/` were written on **2026-07-04** +within one 02:18–06:53 window, and six cases spanning four dilution regimes and both backgrounds +reproduce to ≤ 4.1e-14 on every endpoint. Mtimes are not provenance, but a heterogeneous set would +not behave that way. + +--- + +## Six-case measurement, 2026-08-14 + +The original measurement covered two cases. All six curated Tier-B cases have since been re-run and +compared (the four dilution regimes on `sabr220`, plus `D2med` and `burst` on `sabr330`): + +| case | worst endpoint | worst series | `t`/`V_ratio`/`T` | +| --- | --- | --- | --- | +| D1low · sabr220 | 4.1e-14 | 1.6e-11 | bit-identical | +| D2med · sabr220 | 3.0e-14 | 1.6e-11 | bit-identical | +| D3high · sabr220 | 1.7e-14 | 1.6e-11 | bit-identical | +| burst · sabr220 | 3.5e-14 | 1.6e-11 | bit-identical | +| D2med · sabr330 | 2.4e-14 | 1.5e-11 | bit-identical | +| burst · sabr330 | 3.7e-14 | 7.4e-12 | bit-identical | + +Every endpoint is ≤ 4.1e-14 against a `1e-12` tolerance; every series ≤ 1.6e-11 against `1e-10`. +Wall clock 256–293 s per case. + +**Correction to the timing claim.** The two-case measurement noted the worst H₂SO₄ deviation at day +1.34 and this file previously read as though deviation were tied to the early nucleation burst. +Across six cases the worst-deviation days are **5.83, 1.34, 9.27, 2.15, 3.03, 5.74** — no common +feature. The deviation is a flat ~1e-14 baseline with occasional spikes; it is float round-off +scattered through the run, **not** accumulation and not burst-timed. + +**Where the worst per-bin size-distribution deviation sits.** All four `sabr220` cases give an +*identical* 1.63e-11, at the same cell — t = 0.042 d, bin 5 (Dp 3.2 nm), 4.218 counts — while the two +`sabr330` cases peak late and elsewhere (t = 7.34 d bin 6; t = 8.41 d bin 11). The common thread is +not timing but **sparsity**: every one of them is a bin holding 1.7–4.2 particles cm⁻³. It is +inherited from `n_cm3`, not from the `dlogdp` normalisation — those divisors agree to 6.1e-15. + +Regenerate the figures behind these numbers with `plot_fidelity.py` in this directory. + +--- + ## Not covered by this measurement - Only `cg1` was measured; `cg0p5` / `cg2` cases may be affected by `tomas-jax` `39535ea` landing diff --git a/studio/tests/golden/measure_all_cases.py b/studio/tests/golden/measure_all_cases.py new file mode 100644 index 0000000..8cfc93e --- /dev/null +++ b/studio/tests/golden/measure_all_cases.py @@ -0,0 +1,98 @@ +"""Re-run the curated Tier-B cases and dump every deviation to JSON, incrementally. + + python -m studio.tests.golden.measure_all_cases + +This is the tool behind the numbers in ``REFERENCE_TOLERANCES.md``. It is not a test: it asserts +nothing, so re-measuring never "fails" -- it reports, and a human decides whether the new numbers +are acceptable. + +**Incremental on purpose.** Six 10-day cases is ~28 minutes and an interrupted run used to leave +nothing behind; each case now writes its results the moment it finishes, and an already-present run +is reused rather than repeated. Outputs live in the work directory you name, not a temp dir, so they +survive to be re-analysed. +""" + +import json +import sys +import time +from pathlib import Path + +import numpy as np + +from studio.modelio.execute import run_and_write +from studio.tests.golden.paper_cases import TIER_B_CASES, config_for_case +from studio.tests.golden.tolerances import ( + endpoint_deviations, + worst_relative_deviation, +) + +ARCHIVE = Path( + "/Users/ali/Documents/GitHub/gas-phase-chemistry/SANDBOX/coupled/paper_ensemble/runs" +) +OUT = Path(sys.argv[1]) +OUT.mkdir(parents=True, exist_ok=True) +RESULTS = OUT / "deviations.json" +GAS = ("SO2", "SO3", "H2SO4", "OH") +SERIES = ("SA", "radius_cm", "h2so4wp", "particulate_S", "total_n") + +results = json.loads(RESULTS.read_text()) if RESULTS.is_file() else {} +for case in TIER_B_CASES: + if case in results: + print(f"[skip] {case} already measured", flush=True) + continue + state = OUT / case / "state.npz" + t0 = time.perf_counter() + if not state.is_file(): + print(f"[run ] {case} ...", flush=True) + state = run_and_write(config_for_case(case), OUT / case)["state"] + wall = time.perf_counter() - t0 + + with ( + np.load(state, allow_pickle=True) as f, + np.load(ARCHIVE / case / "state.npz", allow_pickle=True) as a, + ): + fresh = {k: f[k] for k in f.files} + arch = {k: a[k] for k in a.files} + species = [str(s) for s in arch["species"]] + entry = {"wall_s": wall, "endpoints": {}, "series": {}, "time_days": arch["t"].tolist()} + + for name in GAS: + i = species.index(name) + entry["endpoints"][name] = endpoint_deviations(fresh["x"][:, i], arch["x"][:, i]) + entry["series"][name] = worst_relative_deviation(fresh["x"][:, i], arch["x"][:, i]) + # deviation vs time, floored the same way the harness floors it + ref = np.abs(arch["x"][:, i]) + peak = float(np.nanmax(ref)) + ok = ref > peak * 1e-6 + dev = np.where( + ok, np.abs(fresh["x"][:, i] - arch["x"][:, i]) / np.where(ok, ref, 1), np.nan + ) + entry.setdefault("dev_vs_time", {})[name] = dev.tolist() + for key in SERIES: + entry["endpoints"][key] = endpoint_deviations(fresh[key], arch[key]) + entry["series"][key] = worst_relative_deviation(fresh[key], arch[key]) + entry["series"]["dNdlogDp"] = worst_relative_deviation(fresh["dNdlogDp"], arch["dNdlogDp"]) + entry["exact"] = {k: bool(np.array_equal(fresh[k], arch[k])) for k in ("t", "V_ratio", "T")} + # the near-zero trap, as data: unfloored relative error against series magnitude + unfloored = {} + for name in ("O1D", "O", "SO2", "H2SO4"): + if name in species: + i = species.index(name) + ref = np.abs(arch["x"][:, i]) + peak = float(np.nanmax(ref)) + nz = ref > 0 + rel = np.abs(fresh["x"][:, i] - arch["x"][:, i])[nz] / ref[nz] + unfloored[name] = { + "peak": peak, + "worst_unfloored": float(np.max(rel)) if rel.size else 0.0, + "worst_floored": worst_relative_deviation(fresh["x"][:, i], arch["x"][:, i]), + } + entry["near_zero"] = unfloored + + results[case] = entry + RESULTS.write_text(json.dumps(results, indent=1)) + print( + f"[done] {case} {wall:6.1f}s worst series {max(entry['series'].values()):.2e}", flush=True + ) + +print(f"\n{len(results)}/{len(TIER_B_CASES)} cases measured -> {RESULTS}") diff --git a/studio/tests/golden/plot_fidelity.py b/studio/tests/golden/plot_fidelity.py new file mode 100644 index 0000000..1e235e9 --- /dev/null +++ b/studio/tests/golden/plot_fidelity.py @@ -0,0 +1,348 @@ +"""Plots of reproduction fidelity: how closely today's code reproduces the 2026-07 archive. + +Not a performance benchmark. Each figure answers one question the tolerance table cannot: + +1. headroom -- how far every quantity sits below the tolerance that was MEASURED for it +2. timing -- WHEN the deviation peaks (the answer is the nucleation burst, not the endpoint) +3. the floor -- why near-zero species are excluded, in one glance + +Palette: the dataviz reference categorical order, validated (worst adjacent CVD dE 9.1 protan, +normal-vision 22.9). Two hues fall below 3:1 against the surface, which obliges visible labels +rather than colour alone -- so every series is direct-labelled. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +SURFACE = "#fcfcfb" +INK = "#0b0b0b" +INK_2 = "#52514e" +GRID = "#e1e0d9" +# validated categorical order, slots 1-4 +SERIES = {"SO2": "#2a78d6", "SO3": "#eb6834", "H2SO4": "#1baf7a", "OH": "#eda100"} +STATUS_BAD = "#e34948" +SEQ = "#2a78d6" + +plt.rcParams.update( + { + "font.family": "Helvetica", + "font.size": 9, + "axes.spines.top": False, + "axes.spines.right": False, + "axes.edgecolor": INK_2, + "axes.labelcolor": INK, + "text.color": INK, + "xtick.color": INK_2, + "ytick.color": INK_2, + "axes.grid": True, + "grid.color": GRID, + "grid.linewidth": 0.6, + "axes.axisbelow": True, + "figure.facecolor": SURFACE, + "axes.facecolor": SURFACE, + "savefig.facecolor": SURFACE, + } +) + +RTOL_HEADLINE, RTOL_SERIES, RTOL_SIZE = 1e-12, 1e-10, 1e-10 +SHORT = { + "30N_20km__sabr220__D1low__a1p0__nuc1__cg1": "D1 low", + "30N_20km__sabr220__D2med__a1p0__nuc1__cg1": "D2 med", + "30N_20km__sabr220__D3high__a1p0__nuc1__cg1": "D3 high", + "30N_20km__sabr220__burst__a1p0__nuc1__cg1": "burst", + "30N_20km__sabr330__D2med__a1p0__nuc1__cg1": "D2 med · sabr330", + "30N_20km__sabr330__burst__a1p0__nuc1__cg1": "burst · sabr330", +} + + +def _fig_headroom(data: dict, out: Path) -> Path: + """Worst deviation per quantity across all cases, against the tolerance measured for it. + + Dot plot on a log axis: the job is magnitude against a threshold, so one hue, and the + threshold is a line rather than a second colour. + """ + endpoint_rows, series_rows = [], [] + for entry in data.values(): + for name, devs in entry["endpoints"].items(): + for label, value in devs.items(): + endpoint_rows.append((f"{name} {label}", value)) + for name, value in entry["series"].items(): + series_rows.append((f"{name}", value)) + + def collapse(rows): + worst: dict[str, float] = {} + for name, value in rows: + worst[name] = max(worst.get(name, 0.0), value) + return sorted(worst.items(), key=lambda kv: kv[1]) + + endpoints, series = collapse(endpoint_rows), collapse(series_rows) + fig, axes = plt.subplots( + 1, 2, figsize=(11.6, 5.9), gridspec_kw={"width_ratios": [1, 1], "wspace": 0.40} + ) + for ax, rows, tol, title, sub in ( + ( + axes[0], + endpoints, + RTOL_HEADLINE, + "Endpoints — final and peak values", + "what a result is read for", + ), + ( + axes[1], + series, + RTOL_SERIES, + "Series — worst over the whole run", + "every stored sample, floored at 1e-6 x peak", + ), + ): + labels = [name for name, _ in rows] + values = np.array([max(v, 1e-17) for _, v in rows]) + y = np.arange(len(rows)) + ax.hlines(y, 1e-17, values, color=SEQ, linewidth=2, alpha=0.35) + ax.scatter(values, y, s=44, color=SEQ, zorder=3, edgecolor=SURFACE, linewidth=1.2) + ax.axvline(tol, color=STATUS_BAD, linewidth=1.6, linestyle="--", zorder=2) + ax.text( + tol * 0.85, + len(rows) - 0.05, + f"measured tolerance {tol:.0e} ", + color=STATUS_BAD, + fontsize=8.5, + va="bottom", + ha="right", + ) + for yi, value in zip(y, values, strict=True): + ax.text(value * 1.35, yi, f"{value:.1e}", va="center", fontsize=7.6, color=INK_2) + ax.set_yticks(y, labels, fontsize=8.5) + ax.set_xscale("log") + ax.set_xlim(1e-17, tol * 60) + ax.set_ylim(-0.8, len(rows) + 0.35) + ax.set_xlabel("worst relative deviation vs the archive") + ax.set_title(title, fontsize=10.5, loc="left", pad=14) + ax.text(0, 1.015, sub, transform=ax.transAxes, fontsize=8.4, color=INK_2) + fig.suptitle( + "Reproduction fidelity: every quantity sits orders of magnitude inside its tolerance", + fontsize=12, + x=0.005, + ha="left", + y=0.995, + ) + fig.text( + 0.012, + 0.015, + f"{len(data)} cases from coupled/paper_ensemble/runs/ (all 810 files written 2026-07-04), " + f"re-run today, configs rebuilt from run_ensemble.py's axis tables.\n" + f"CAVEAT: those files carry no provenance record, so the code that produced them is not " + f"known — only assumed to be the axis tables.\n" + f"Other archive directories (runs_60day, runs_bgstop*, runs_geo, …) were produced " + f"differently and are excluded from this comparison.\n" + f"Bit-for-bit reproduction is false (~31% of gas state elements differ). Tolerances were " + f"measured before they were asserted.", + fontsize=7.8, + color=INK_2, + linespacing=1.7, + ) + # explicit margins rather than tight_layout + bbox_inches="tight": the two fight over the + # multi-line caption and the axis labels lose + fig.subplots_adjust(left=0.135, right=0.985, top=0.855, bottom=0.26, wspace=0.42) + fig.savefig(out, dpi=200) + plt.close(fig) + return out + + +def _fig_timing(data: dict, out: Path) -> Path: + """Deviation against simulated time, one panel per case. Small multiples, 4 species.""" + cases = list(data) + ncols = 3 + nrows = int(np.ceil(len(cases) / ncols)) + fig, axes = plt.subplots( + nrows, ncols, figsize=(11.5, 3.1 * nrows), sharex=True, sharey=True, squeeze=False + ) + for index, case in enumerate(cases): + ax = axes[index // ncols][index % ncols] + entry = data[case] + days = np.array(entry["time_days"]) / 86400.0 + for name, colour in SERIES.items(): + dev = np.array(entry["dev_vs_time"][name], dtype=float) + ax.plot(days, dev, color=colour, linewidth=1.5, label=name, solid_capstyle="round") + worst_name = max( + entry["series"], key=lambda k: entry["series"].get(k, 0) if k in SERIES else 0 + ) + dev = np.array(entry["dev_vs_time"][worst_name], dtype=float) + peak_index = int(np.nanargmax(dev)) + ax.scatter( + [days[peak_index]], + [dev[peak_index]], + s=40, + color=SERIES[worst_name], + edgecolor=SURFACE, + linewidth=1.4, + zorder=4, + ) + late = days[peak_index] > 0.65 * float(days[-1]) + ax.annotate( + f"{worst_name} {dev[peak_index]:.1e}\nday {days[peak_index]:.2f}", + (days[peak_index], dev[peak_index]), + textcoords="offset points", + xytext=(-8 if late else 8, 6), + ha="right" if late else "left", + fontsize=7.8, + color=INK_2, + ) + ax.axhline(RTOL_SERIES, color=STATUS_BAD, linewidth=1.3, linestyle="--") + ax.set_yscale("log") + ax.set_ylim(1e-18, 1e-8) + ax.set_title(SHORT.get(case, case), fontsize=10, loc="left") + if index % ncols == 0: + ax.set_ylabel("relative deviation") + if index // ncols == nrows - 1: + ax.set_xlabel("simulated time (days)") + for spare in range(len(cases), nrows * ncols): + axes[spare // ncols][spare % ncols].axis("off") + handles = [ + plt.Line2D([], [], color=colour, linewidth=2, label=name) for name, colour in SERIES.items() + ] + handles.append( + plt.Line2D( + [], [], color=STATUS_BAD, linewidth=1.3, linestyle="--", label="series tolerance 1e-10" + ) + ) + fig.legend( + handles=handles, + loc="upper right", + ncols=5, + frameon=False, + fontsize=9, + bbox_to_anchor=(0.995, 1.0), + ) + + def _peak_day(entry: dict) -> float: + dev = np.array(entry["dev_vs_time"]["H2SO4"], dtype=float) + return float(np.array(entry["time_days"])[int(np.nanargmax(dev))] / 86400.0) + + peaks = ", ".join(f"{_peak_day(entry):.1f}" for entry in data.values()) + fig.suptitle( + "Deviation is scattered round-off, not accumulation — it does not grow with time", + fontsize=12, + x=0.005, + ha="left", + y=1.005, + ) + fig.text( + 0.005, + 0.975, + f"A baseline near 1e-14 for the whole run with occasional spikes to ~1e-12, three orders " + f"below the series tolerance. Worst-deviation days across the six cases: {peaks} — no " + f"common feature, so this is float round-off rather than a diverging integration.", + fontsize=8.4, + color=INK_2, + ) + fig.tight_layout(rect=(0, 0, 1, 0.935)) + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + return out + + +def _fig_floor(data: dict, out: Path) -> Path: + """Why near-zero species are excluded: the same comparison, floored and unfloored.""" + names, unfloored, floored, peaks = [], [], [], [] + for name in ("O1D", "O", "SO2", "H2SO4"): + values = [entry["near_zero"][name] for entry in data.values() if name in entry["near_zero"]] + if not values: + continue + names.append(name) + unfloored.append(max(v["worst_unfloored"] for v in values)) + floored.append(max(v["worst_floored"] for v in values)) + peaks.append(max(v["peak"] for v in values)) + + y = np.arange(len(names)) + fig, ax = plt.subplots(figsize=(9.6, 3.5)) + ax.hlines(y, np.maximum(floored, 1e-17), unfloored, color=GRID, linewidth=3, zorder=1) + ax.scatter( + np.maximum(floored, 1e-17), + y, + s=52, + color=SEQ, + zorder=3, + edgecolor=SURFACE, + linewidth=1.2, + label="floored — what the harness compares", + ) + ax.scatter( + unfloored, + y, + s=52, + color=STATUS_BAD, + zorder=3, + edgecolor=SURFACE, + linewidth=1.2, + label="unfloored — every sample, including noise about zero", + ) + for yi, (low, high, peak) in enumerate(zip(floored, unfloored, peaks, strict=True)): + ax.text( + max(low, 1e-17) * 0.5, + yi, + f"{max(low, 0):.0e}", + va="center", + ha="right", + fontsize=8, + color=SEQ, + ) + ax.text(high * 2.0, yi, f"{high:.0e}", va="center", fontsize=8, color=STATUS_BAD) + ax.text(1e-16, yi + 0.32, f"peak {peak:.1e} molec cm$^{{-3}}$", fontsize=7.4, color=INK_2) + ax.axvline(RTOL_SERIES, color=INK_2, linewidth=1.2, linestyle=":") + ax.text(RTOL_SERIES, len(names) - 0.45, " series tolerance 1e-10", fontsize=8, color=INK_2) + ax.set_yticks(y, names, fontsize=10) + ax.set_xscale("log") + ax.set_xlim(1e-17, 1e7) + ax.set_ylim(-0.6, len(names) - 0.25) + ax.set_xlabel("worst relative deviation") + ax.legend(loc="upper right", frameon=False, fontsize=8.6, bbox_to_anchor=(1.0, 0.92)) + ax.set_title( + "The near-zero trap: why O1D and O are excluded rather than merely toleranced", + fontsize=11.5, + loc="left", + pad=12, + ) + fig.text( + 0.005, + -0.06, + "O1D spends the night at ~1e-35 molec cm$^{-3}$ oscillating about zero — the archived " + "value is literally negative. Unfloored, its relative error reaches 1e+4 over an absolute " + "difference of 1e-34.", + fontsize=8.2, + color=INK_2, + wrap=True, + ) + fig.tight_layout() + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + return out + + +def main() -> int: + bench = Path(sys.argv[1]) + data = json.loads((bench / "deviations.json").read_text()) + out_dir = bench / "figures" + out_dir.mkdir(exist_ok=True) + written = [ + _fig_headroom(data, out_dir / "fidelity_headroom.png"), + _fig_timing(data, out_dir / "fidelity_timing.png"), + _fig_floor(data, out_dir / "fidelity_near_zero_floor.png"), + ] + for path in written: + print(f"wrote {path} ({path.stat().st_size / 1024:.0f} kB)") + print(f"\n{len(data)} cases plotted") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 550b15f0a42b920967b3143fa8d8d5e8f155e9e5 Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 16:58:30 -0700 Subject: [PATCH 15/18] =?UTF-8?q?studio:=20persistence=20=E2=80=94=20model?= =?UTF-8?q?s,=20artefact=20store,=20and=20Alembic=20from=20the=20first=20m?= =?UTF-8?q?igration=20(#84)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit studio: persistence -- models, artefact store, and Alembic from the first migration Task 0.9b (#83). studio/store: nine tables, an artefact store behind an interface, a repository with no update path, and migrations that are exercised by every test. Immutability is structural rather than conventional. run_config is keyed by the config's own hash and ensure_config is get-or-create; a test asserts the repository exposes no update, delete or overwrite helper at all. An edited config is a different row and a different run, linked by derived_from_run_id. The database stores pointers, never arrays. Artefacts go behind an ArtifactStore protocol -- a directory today, MinIO or S3 later without touching callers -- and the row keeps a relative path, size, content type and a SHA-256 computed on write. That checksum makes "still the file that was written" checkable, since silent corruption and a tidied directory look identical from the database otherwise. A missing artefact raises rather than being recorded as an absence. Two portability decisions, both because the backends would otherwise disagree silently. UtcDateTime, a TypeDecorator, because DateTime(timezone=True) returns an aware datetime on Postgres and a NAIVE one on SQLite -- caught by a test asserting tzinfo is not None, which failed on the first run; naive input now raises rather than being assigned a guessed zone. And foreign_keys=ON for SQLite, without which its foreign keys are ignored entirely and the constraints in models.py would be documentation on the Phase-0 backend while being enforced in production. The drift test runs Alembic's compare_metadata against a migrated database and requires an empty diff, so a column added without a migration fails in seconds rather than on the first real deployment. Nothing uses Base.metadata.create_all, including the tests: every test upgrades through the migrations, so they are exercised continuously rather than for the first time on someone's database. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 53 ++- studio/store/__init__.py | 85 +++++ studio/store/alembic.ini | 47 +++ studio/store/artifacts.py | 119 +++++++ studio/store/engine.py | 98 ++++++ studio/store/migrate.py | 50 +++ studio/store/migrations/env.py | 63 ++++ studio/store/migrations/script.py.mako | 24 ++ ...427673e9a31_initial_run_metadata_schema.py | 215 ++++++++++++ studio/store/models.py | 293 ++++++++++++++++ studio/store/repository.py | 259 ++++++++++++++ studio/tests/unit/test_store.py | 323 ++++++++++++++++++ 12 files changed, 1628 insertions(+), 1 deletion(-) create mode 100644 studio/store/__init__.py create mode 100644 studio/store/alembic.ini create mode 100644 studio/store/artifacts.py create mode 100644 studio/store/engine.py create mode 100644 studio/store/migrate.py create mode 100644 studio/store/migrations/env.py create mode 100644 studio/store/migrations/script.py.mako create mode 100644 studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py create mode 100644 studio/store/models.py create mode 100644 studio/store/repository.py create mode 100644 studio/tests/unit/test_store.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 37cd98e..d146a36 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -22,13 +22,64 @@ with full provenance, and the golden tests pass. | 0.6 `studio/runner` + job lifecycle | **done** (#72) | | 0.7 Golden-file harness (two tiers) | **done** (#70 measured, #79 asserted) | | 0.8 Four contained fixes in `coupled/` | not started | -| 0.9 Vertical slice: CLI + API + minimal UI | **in progress** — 0.9a provenance (#80); 0.9b–e to come | +| 0.9 Vertical slice: CLI + API + minimal UI | **in progress** — 0.9a provenance (#80), 0.9b persistence (#83); 0.9c–e to come | Task order note: 0.5 was taken **before 0.3**, so the dependency-graph engine has real derivations to resolve rather than fixtures. --- +### 2026-08-14 — Task 0.9b: persistence (issue #83) + +`studio/store/`: models, engine, artefact store, repository, and **Alembic from the first +migration**. 15 new Tier-A tests (246 total). + +**Nine tables**: `run_set`, `run`, `run_config`, `job`, `job_transition`, `result_artifact`, +`dataset_version`, `run_dataset`, `run_summary`. `dataset_version` is empty in Phase 0 and exists +anyway — "which ERA5 product was this run built on?" is a question Phase 1 must be able to ask about +runs made before it existed. + +**Immutability is structural, not conventional.** `run_config` is keyed by the config's own hash and +`ensure_config` is get-or-create; there is no update path, and a test asserts the repository exposes +no `update`/`delete`/`overwrite` helper at all. An edited config is a different row and a different +run, linked by `derived_from_run_id`. + +**The database stores pointers, never arrays.** Artefacts go to `LocalDirectoryStore` behind an +`ArtifactStore` protocol — a directory today, MinIO or S3 later without touching callers — and the +row keeps a *relative* path, size, content type and **SHA-256 computed on write**. That checksum is +what makes "still the file that was written" checkable: silent corruption and a helpfully tidied +directory look identical from the database otherwise. A missing artefact **raises** rather than +being recorded as an absence. + +**Two portability decisions, both because SQLite and Postgres would otherwise disagree silently:** + +1. **`UtcDateTime`, a `TypeDecorator`.** `DateTime(timezone=True)` is not enough — Postgres returns + an *aware* datetime and **SQLite returns a naive one**, so the same comparison is right on one + backend and wrong on the other. Caught by a test asserting `tzinfo is not None`, which failed on + the first run. Naive input now *raises*: a caller who does not know their own timezone cannot be + handed one by guessing. +2. **`foreign_keys=ON` for SQLite.** Without it SQLite ignores foreign keys entirely, so the + constraints in `models.py` would be documentation on the Phase-0 backend and enforced in + production. A test inserts a job for a nonexistent run and requires an `IntegrityError`. + +**The drift test.** `test_the_models_and_the_migration_agree` runs Alembic's `compare_metadata` +against a migrated database and requires an empty diff. Without it, a column added to `models.py` +without a migration works everywhere the schema was built from the models and fails on the first +real deployment. Related: **nothing uses `Base.metadata.create_all`, including the tests** — every +test upgrades through the migrations, so the migrations are exercised continuously rather than for +the first time on someone's database. + +Four headline scalars (`final_so2_pptv`, `peak_h2so4_pptv`, `peak_number_cm3`, +`final_surface_area`) are promoted out of the summary JSON into columns, so "every run where peak +number exceeded X" is a query rather than 810 deserialisations. They are read from the summary +rather than recomputed, so column and JSON cannot disagree. + +`runs_for_config` deliberately returns *runs* rather than a cached-result verdict: an identical hash +is necessary but not sufficient, because the caller must also compare the model version in each +run's provenance (ADR-006). + +--- + ### 2026-08-14 — The archive's status as a reference, settled and written down Ali, 2026-08-14: **`coupled/paper_ensemble/runs/` is a valid reproduction reference.** Recorded in diff --git a/studio/store/__init__.py b/studio/store/__init__.py new file mode 100644 index 0000000..faf93a3 --- /dev/null +++ b/studio/store/__init__.py @@ -0,0 +1,85 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Persistence for run metadata (ADR-004, ADR-007). + +Three stores, matched to three shapes. This package is the relational one: run sets, runs, configs, +jobs and pointers to output. **Arrays live on disk behind ``artifacts.ArtifactStore``**, never in +the database, and climatology is a separate Phase-1 concern. + +* ``models.py`` -- the tables. No ``AUTOINCREMENT``, no reliance on SQLite type affinity, + timezone-aware timestamps: moving to Postgres must be a connection-string change. +* ``engine.py`` -- engine and short-transaction session scope; SQLite gets ``foreign_keys=ON`` and + WAL, because otherwise its constraints would be documentation while Postgres enforced them. +* ``artifacts.py`` -- the storage interface. A directory today; MinIO or S3 later without touching + callers. Checksums are computed on write so "still the file that was written" is checkable. +* ``repository.py`` -- the operations, and no update path for a config: immutability is structural. +* ``migrations/`` -- Alembic from the first migration, so the second one is routine. + +This package must not import ``coupled``. +""" + +from __future__ import annotations + +from studio.store.artifacts import ArtifactStore, LocalDirectoryStore, StoredArtifact, sha256_of +from studio.store.engine import ( + DATABASE_URL_ENV, + create_db_engine, + database_url, + session_factory, + session_scope, +) +from studio.store.migrate import current_revision, upgrade_to_head +from studio.store.models import ( + Base, + DatasetVersionRow, + JobRow, + JobTransitionRow, + ResultArtifactRow, + RunConfigRow, + RunRow, + RunSetRow, + RunSummaryRow, +) +from studio.store.repository import ( + artifact_for, + create_run, + create_run_set, + ensure_config, + record_artifact, + record_job, + record_summary, + record_transition, + runs_for_config, +) + +__all__ = [ + "DATABASE_URL_ENV", + "ArtifactStore", + "Base", + "DatasetVersionRow", + "JobRow", + "JobTransitionRow", + "LocalDirectoryStore", + "ResultArtifactRow", + "RunConfigRow", + "RunRow", + "RunSetRow", + "RunSummaryRow", + "StoredArtifact", + "artifact_for", + "create_db_engine", + "create_run", + "create_run_set", + "current_revision", + "database_url", + "ensure_config", + "record_artifact", + "record_job", + "record_summary", + "record_transition", + "runs_for_config", + "session_factory", + "session_scope", + "sha256_of", + "upgrade_to_head", +] diff --git a/studio/store/alembic.ini b/studio/store/alembic.ini new file mode 100644 index 0000000..baa9848 --- /dev/null +++ b/studio/store/alembic.ini @@ -0,0 +1,47 @@ +# Alembic configuration for Plume Studio's run-metadata database. +# +# alembic -c studio/store/alembic.ini upgrade head +# alembic -c studio/store/alembic.ini revision --autogenerate -m "what changed" +# +# The URL is deliberately NOT set here: it comes from STUDIO_DATABASE_URL (or the local SQLite +# default) via studio.store.engine, so one configuration works for a developer machine, CI, and a +# Postgres deployment without editing a file that is committed. +[alembic] +script_location = studio/store/migrations +prepend_sys_path = . +# explicit, or alembic warns and falls back to splitting on spaces, commas and colons +path_separator = os +file_template = %%(rev)s_%%(slug)s + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARNING +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARNING +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s diff --git a/studio/store/artifacts.py b/studio/store/artifacts.py new file mode 100644 index 0000000..c1bf447 --- /dev/null +++ b/studio/store/artifacts.py @@ -0,0 +1,119 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Artefact storage, behind an interface (ADR-004). + +"Object storage" in Phase 0 is a directory on disk. MinIO or S3 substitutes without touching +callers, which is why every path a caller sees is **relative to the store root** -- an absolute path +baked into the database would be a deployment detail that outlives the deployment. + +The store computes a SHA-256 as it copies. That is what makes "this artefact is still the file that +was written" a checkable claim rather than an assumption, and it costs one pass over a file that is +being read anyway. +""" + +from __future__ import annotations + +import hashlib +import shutil +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol, runtime_checkable + +#: Read in 1 MiB blocks: large enough that the syscall overhead disappears, small enough that a +#: 3.6 MB npz never sits in memory twice. +_CHUNK = 1024 * 1024 + + +@dataclass(frozen=True) +class StoredArtifact: + """What the database records about one stored file. Never the bytes.""" + + key: str + size_bytes: int + sha256: str + content_type: str + + +@runtime_checkable +class ArtifactStore(Protocol): + """Where run outputs live. Nothing above this may assume a filesystem.""" + + def put(self, run_id: str, kind: str, source: Path) -> StoredArtifact: + """Copy ``source`` into the store under ``run_id``, returning what to record.""" + ... + + def open_path(self, key: str) -> Path: + """Resolve a stored key to a readable path.""" + ... + + def exists(self, key: str) -> bool: ... + + +def sha256_of(path: Path) -> str: + """Streaming SHA-256 of a file.""" + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(_CHUNK): + digest.update(chunk) + return digest.hexdigest() + + +#: Extension -> content type, for the artefacts Studio actually writes. Deliberately small: a +#: general mimetype lookup would guess, and a wrong content type on an npz is worse than none. +_CONTENT_TYPES = { + ".npz": "application/x-npz", + ".json": "application/json", + ".log": "text/plain", + ".png": "image/png", +} + + +class LocalDirectoryStore: + """The Phase-0 implementation: a directory tree, one subdirectory per run.""" + + def __init__(self, root: Path) -> None: + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + + def put(self, run_id: str, kind: str, source: Path) -> StoredArtifact: + """Copy in, hash, and return the record. + + Raises: + FileNotFoundError: If the source is missing. A run that was supposed to produce an + artefact and did not is a failure to surface, not a row to omit. + """ + source = Path(source) + if not source.is_file(): + raise FileNotFoundError( + f"artefact {kind!r} for run {run_id} is missing at {source}; a run that did not " + f"produce it should fail rather than be recorded without it" + ) + destination = self.root / run_id / source.name + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, destination) + return StoredArtifact( + key=str(destination.relative_to(self.root)), + size_bytes=destination.stat().st_size, + sha256=sha256_of(destination), + content_type=_CONTENT_TYPES.get(source.suffix, "application/octet-stream"), + ) + + def open_path(self, key: str) -> Path: + path = self.root / key + if not path.is_file(): + raise FileNotFoundError(f"artefact {key!r} is recorded but missing from {self.root}") + return path + + def exists(self, key: str) -> bool: + return (self.root / key).is_file() + + def verify(self, key: str, expected_sha256: str) -> bool: + """Whether the stored bytes still hash to what was recorded. + + The reason the checksum is stored at all: silent corruption and a helpfully "tidied" + directory look identical from the database. + """ + return self.exists(key) and sha256_of(self.open_path(key)) == expected_sha256 + + +__all__ = ["ArtifactStore", "LocalDirectoryStore", "StoredArtifact", "sha256_of"] diff --git a/studio/store/engine.py b/studio/store/engine.py new file mode 100644 index 0000000..43a3683 --- /dev/null +++ b/studio/store/engine.py @@ -0,0 +1,98 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Engine and session factory. + +SQLite in Phase 0; a Postgres URL substitutes without code changes, which is the whole point of +putting Alembic in from the first migration (ADR-007). + +Two SQLite-specific pragmas are set **for SQLite only**, and neither is a workaround leaking into +the models: + +* ``foreign_keys=ON`` -- SQLite ignores foreign keys unless asked, so without this the constraints + in ``models.py`` would be documentation on that backend and enforced on Postgres. Silent + divergence between environments is worse than either behaviour. +* ``journal_mode=WAL`` -- readers stop blocking the writer. ADR-007 flags SQLite's single-writer + model as a real constraint, and the API reads job state far more often than it writes. +""" + +from __future__ import annotations + +import os +from collections.abc import Iterator +from contextlib import contextmanager +from pathlib import Path +from typing import Any + +from sqlalchemy import Engine, create_engine, event +from sqlalchemy.orm import Session, sessionmaker + +#: Read at import of a session factory, never at module import, so a test can set it first. +DATABASE_URL_ENV = "STUDIO_DATABASE_URL" + +#: Where a local SQLite database lives when nothing says otherwise. Matches studio/.env.example. +DEFAULT_SQLITE_PATH = Path("var/studio/studio.db") + + +def database_url(url: str | None = None) -> str: + """Resolve the database URL: explicit argument, then environment, then the local default.""" + if url: + return url + from_env = os.environ.get(DATABASE_URL_ENV) + if from_env: + return from_env + DEFAULT_SQLITE_PATH.parent.mkdir(parents=True, exist_ok=True) + return f"sqlite:///{DEFAULT_SQLITE_PATH}" + + +def create_db_engine(url: str | None = None, *, echo: bool = False) -> Engine: + """An engine with SQLite's footguns disarmed.""" + resolved = database_url(url) + engine = create_engine(resolved, echo=echo, future=True) + if resolved.startswith("sqlite"): + + @event.listens_for(engine, "connect") + def _sqlite_pragmas(dbapi_connection: Any, _record: Any) -> None: + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.execute("PRAGMA journal_mode=WAL") + cursor.close() + + return engine + + +def session_factory(engine: Engine) -> sessionmaker[Session]: + """Sessions that do not expire objects on commit. + + ``expire_on_commit=False`` because callers read attributes off returned rows after the + transaction closes; the alternative is a lazy reload per attribute, which on SQLite means + re-acquiring the very lock ADR-007 warns about. + """ + return sessionmaker(bind=engine, expire_on_commit=False, future=True) + + +@contextmanager +def session_scope(factory: sessionmaker[Session]) -> Iterator[Session]: + """A transaction that commits on success and rolls back on any exception. + + Short by design: SQLite has one writer, so a session held open across a model run would block + every other job's state update. + """ + session = factory() + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + finally: + session.close() + + +__all__ = [ + "DATABASE_URL_ENV", + "DEFAULT_SQLITE_PATH", + "create_db_engine", + "database_url", + "session_factory", + "session_scope", +] diff --git a/studio/store/migrate.py b/studio/store/migrate.py new file mode 100644 index 0000000..d3c89f2 --- /dev/null +++ b/studio/store/migrate.py @@ -0,0 +1,50 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Running migrations from code, so nothing depends on the ``alembic`` CLI being on PATH. + +``Base.metadata.create_all`` is deliberately **not** used anywhere, including in tests. It would +produce a schema that no migration ever created, so the migrations would be exercised for the first +time on someone's real database -- which is the failure mode Alembic-from-the-first-migration exists +to prevent (ADR-007). Tests upgrade to head like everything else. +""" + +from __future__ import annotations + +from pathlib import Path + +from alembic import command +from alembic.config import Config +from alembic.migration import MigrationContext +from sqlalchemy import Engine + +#: The committed configuration; the URL is supplied at call time, never from the file. +ALEMBIC_INI = Path(__file__).resolve().parent / "alembic.ini" +MIGRATIONS_DIR = Path(__file__).resolve().parent / "migrations" + + +def alembic_config(url: str) -> Config: + """An Alembic config pointed at this package's migrations and the given URL.""" + config = Config(str(ALEMBIC_INI)) + config.set_main_option("script_location", str(MIGRATIONS_DIR)) + config.set_main_option("sqlalchemy.url", url) + return config + + +def upgrade_to_head(url: str) -> None: + """Bring a database up to the latest revision. Idempotent.""" + command.upgrade(alembic_config(url), "head") + + +def current_revision(engine: Engine) -> str | None: + """The revision a database is at, or ``None`` if it has never been migrated.""" + with engine.connect() as connection: + return MigrationContext.configure(connection).get_current_revision() + + +__all__ = [ + "ALEMBIC_INI", + "MIGRATIONS_DIR", + "alembic_config", + "current_revision", + "upgrade_to_head", +] diff --git a/studio/store/migrations/env.py b/studio/store/migrations/env.py new file mode 100644 index 0000000..f0a5fbe --- /dev/null +++ b/studio/store/migrations/env.py @@ -0,0 +1,63 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Alembic environment. + +The URL comes from ``studio.store.engine.database_url`` rather than ``alembic.ini`` so that one +committed configuration serves a developer machine, CI and a Postgres deployment -- the difference +is ``STUDIO_DATABASE_URL``, not an edited file. + +``render_as_batch=True`` is set for SQLite only. SQLite cannot ALTER most things in place, so +Alembic rebuilds the table; without it, the first migration that drops a column would fail on the +Phase-0 backend and pass on Postgres. That divergence is exactly what ADR-007 says to avoid. +""" + +from __future__ import annotations + +from logging.config import fileConfig + +from alembic import context + +from studio.store.engine import create_db_engine +from studio.store.models import Base + +config = context.config +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +target_metadata = Base.metadata + + +def run_migrations_offline() -> None: + """Emit SQL without a connection, for review or for a DBA to apply.""" + url = config.get_main_option("sqlalchemy.url") or None + from studio.store.engine import database_url + + context.configure( + url=database_url(url), + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + compare_type=True, + ) + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + engine = create_db_engine(config.get_main_option("sqlalchemy.url") or None) + with engine.connect() as connection: + context.configure( + connection=connection, + target_metadata=target_metadata, + compare_type=True, + render_as_batch=connection.dialect.name == "sqlite", + ) + with context.begin_transaction(): + context.run_migrations() + engine.dispose() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/studio/store/migrations/script.py.mako b/studio/store/migrations/script.py.mako new file mode 100644 index 0000000..ee40d2c --- /dev/null +++ b/studio/store/migrations/script.py.mako @@ -0,0 +1,24 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Created: ${create_date} +""" +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} +branch_labels = ${repr(branch_labels)} +depends_on = ${repr(depends_on)} + + +def upgrade() -> None: + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + ${downgrades if downgrades else "pass"} diff --git a/studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py b/studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py new file mode 100644 index 0000000..f4481b5 --- /dev/null +++ b/studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py @@ -0,0 +1,215 @@ +"""initial run metadata schema + +Revision ID: 1427673e9a31 +Revises: +Created: 2026-08-14 16:27:10.579709 +""" + +from __future__ import annotations + +import sqlalchemy as sa +from alembic import op + +revision = "1427673e9a31" +down_revision = None +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.create_table( + "dataset_version", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("identifier", sa.String(length=255), nullable=False), + sa.Column("version", sa.String(length=64), nullable=False), + sa.Column("sha256", sa.String(length=64), nullable=False), + sa.Column("source", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint("identifier", "version", name="uq_dataset_identifier_version"), + ) + with op.batch_alter_table("dataset_version", schema=None) as batch_op: + batch_op.create_index( + batch_op.f("ix_dataset_version_identifier"), ["identifier"], unique=False + ) + + op.create_table( + "run_config", + sa.Column("config_hash", sa.String(length=64), nullable=False), + sa.Column("schema_version", sa.String(length=32), nullable=False), + sa.Column("resolved_config", sa.JSON(), nullable=False), + sa.Column("overrides", sa.JSON(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint("config_hash"), + ) + op.create_table( + "run_set", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("label", sa.String(length=255), nullable=False), + sa.Column("axes", sa.JSON(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("owner", sa.String(length=255), nullable=True), + sa.PrimaryKeyConstraint("id"), + ) + op.create_table( + "run", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("run_set_id", sa.String(length=64), nullable=False), + sa.Column("config_hash", sa.String(length=64), nullable=False), + sa.Column("label", sa.String(length=255), nullable=False), + sa.Column("derived_from_run_id", sa.String(length=64), nullable=True), + sa.Column("provenance", sa.JSON(), nullable=True), + sa.Column("reproducible", sa.Integer(), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["config_hash"], + ["run_config.config_hash"], + ), + sa.ForeignKeyConstraint( + ["derived_from_run_id"], + ["run.id"], + ), + sa.ForeignKeyConstraint( + ["run_set_id"], + ["run_set.id"], + ), + sa.PrimaryKeyConstraint("id"), + ) + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_run_config_hash"), ["config_hash"], unique=False) + batch_op.create_index( + "ix_run_config_hash_created", ["config_hash", "created_at"], unique=False + ) + batch_op.create_index(batch_op.f("ix_run_run_set_id"), ["run_set_id"], unique=False) + + op.create_table( + "job", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("state", sa.String(length=32), nullable=False), + sa.Column("backend", sa.String(length=64), nullable=False), + sa.Column("work_dir", sa.Text(), nullable=True), + sa.Column("exit_code", sa.Integer(), nullable=True), + sa.Column("detail", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("id"), + ) + with op.batch_alter_table("job", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_job_run_id"), ["run_id"], unique=False) + batch_op.create_index(batch_op.f("ix_job_state"), ["state"], unique=False) + + op.create_table( + "result_artifact", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("kind", sa.String(length=32), nullable=False), + sa.Column("path", sa.Text(), nullable=False), + sa.Column("size_bytes", sa.Integer(), nullable=False), + sa.Column("sha256", sa.String(length=64), nullable=False), + sa.Column("content_type", sa.String(length=64), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint("run_id", "kind", name="uq_artifact_run_kind"), + ) + with op.batch_alter_table("result_artifact", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_result_artifact_run_id"), ["run_id"], unique=False) + + op.create_table( + "run_dataset", + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("dataset_id", sa.String(length=64), nullable=False), + sa.ForeignKeyConstraint( + ["dataset_id"], + ["dataset_version.id"], + ), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("run_id", "dataset_id"), + ) + op.create_table( + "run_summary", + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("schema_version", sa.String(length=32), nullable=False), + sa.Column("termination", sa.String(length=32), nullable=False), + sa.Column("flags", sa.JSON(), nullable=False), + sa.Column("final_so2_pptv", sa.Float(), nullable=True), + sa.Column("peak_h2so4_pptv", sa.Float(), nullable=True), + sa.Column("peak_number_cm3", sa.Float(), nullable=True), + sa.Column("final_surface_area", sa.Float(), nullable=True), + sa.Column("summary", sa.JSON(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("run_id"), + ) + with op.batch_alter_table("run_summary", schema=None) as batch_op: + batch_op.create_index( + batch_op.f("ix_run_summary_termination"), ["termination"], unique=False + ) + + op.create_table( + "job_transition", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("job_id", sa.String(length=64), nullable=False), + sa.Column("state", sa.String(length=32), nullable=False), + sa.Column("at", sa.DateTime(timezone=True), nullable=False), + sa.Column("detail", sa.Text(), nullable=False), + sa.ForeignKeyConstraint( + ["job_id"], + ["job.id"], + ), + sa.PrimaryKeyConstraint("id"), + ) + with op.batch_alter_table("job_transition", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_job_transition_job_id"), ["job_id"], unique=False) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("job_transition", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_job_transition_job_id")) + + op.drop_table("job_transition") + with op.batch_alter_table("run_summary", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_run_summary_termination")) + + op.drop_table("run_summary") + op.drop_table("run_dataset") + with op.batch_alter_table("result_artifact", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_result_artifact_run_id")) + + op.drop_table("result_artifact") + with op.batch_alter_table("job", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_job_state")) + batch_op.drop_index(batch_op.f("ix_job_run_id")) + + op.drop_table("job") + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_run_run_set_id")) + batch_op.drop_index("ix_run_config_hash_created") + batch_op.drop_index(batch_op.f("ix_run_config_hash")) + + op.drop_table("run") + op.drop_table("run_set") + op.drop_table("run_config") + with op.batch_alter_table("dataset_version", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_dataset_version_identifier")) + + op.drop_table("dataset_version") + # ### end Alembic commands ### diff --git a/studio/store/models.py b/studio/store/models.py new file mode 100644 index 0000000..e900195 --- /dev/null +++ b/studio/store/models.py @@ -0,0 +1,293 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Relational models for run metadata (ADR-004). + +**The database stores pointers, never arrays.** Raw output is object storage keyed by run id -- a +directory on disk today, MinIO or S3 later without touching callers -- and what lives here is the +path, size, checksum and content type. A 3.6 MB npz per run times 810 runs is not a database's job. + +Two rules from ADR-004 are structural rather than conventional: + +* **Configs are immutable once submitted.** ``run_config`` is insert-only, keyed by ``config_hash``: + the same config submitted twice is the same row, and an edited config is a *different* row and a + *different* run. There is deliberately no update path -- see ``repository.py``, which offers + get-or-create and nothing else. +* **Results are never overwritten.** An artefact row is written once per run. + +Portability is a requirement, not an aspiration (ADR-007): moving to Postgres must be a +connection-string change. So: + +* **No ``AUTOINCREMENT``.** Primary keys are explicit strings -- content hashes where a natural one + exists, UUID hex otherwise. This also makes a run's identity meaningful rather than positional, + and lets a row be written before the database has ever seen it. +* **No reliance on SQLite's type affinity.** Every column has a real type, and timestamps go + through :class:`UtcDateTime` -- because ``DateTime(timezone=True)`` alone returns *naive* + datetimes on SQLite and aware ones on Postgres, which is a comparison bug that only appears in + production. +* **JSON columns via SQLAlchemy's portable ``JSON``**, which is native on both backends. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import ( + DateTime, + Float, + ForeignKey, + Index, + Integer, + String, + Text, + UniqueConstraint, +) +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship +from sqlalchemy.types import JSON, TypeDecorator + + +class UtcDateTime(TypeDecorator[datetime]): + """A timestamp that is timezone-aware on **every** backend. + + ``DateTime(timezone=True)`` is not enough. Postgres stores the offset and returns an aware + datetime; **SQLite stores a string and hands back a naive one**, so the same code compares + correctly on one backend and silently wrongly on the other -- the exact cross-backend divergence + ADR-007 says to avoid, and the reason this class exists rather than a convention that everyone + remembers to call ``.replace(tzinfo=UTC)``. + + Naive input **raises**: a caller who does not know their own timezone cannot be given one by + guessing (ADR-005). + """ + + impl = DateTime(timezone=True) + cache_ok = True + + def process_bind_param(self, value: datetime | None, dialect: Any) -> datetime | None: + if value is None: + return None + if value.tzinfo is None: + raise ValueError( + f"naive datetime {value!r} cannot be stored: its timezone is unknown, and assuming " + f"one would be wrong on every machine but the one that wrote it. Use " + f"datetime.now(UTC)." + ) + return value.astimezone(UTC) + + def process_result_value(self, value: datetime | None, dialect: Any) -> datetime | None: + if value is None: + return None + return value if value.tzinfo is not None else value.replace(tzinfo=UTC) + + +class Base(DeclarativeBase): + """Declarative base. Alembic's autogenerate compares against this metadata.""" + + +#: Identifier column width: a SHA-256 hex digest is 64, a UUID hex 32. Fixed rather than unbounded +#: so Postgres gets a sensible column and the intent is visible. +_ID = String(64) + + +class RunSetRow(Base): + """A sweep. A single run is the N = 1 case of one (see ``studio.schema.runset``).""" + + __tablename__ = "run_set" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + label: Mapped[str] = mapped_column(String(255), default="") + #: The axes, as submitted. Stored whole because a RunSet is small and re-expanding it must give + #: exactly the runs that were created, not today's interpretation of the axes. + axes: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + #: Nullable owner, present from the first migration so adopting tenancy later is a backfill + #: rather than a migration of every query (BLOCKING-2 / ASSUMPTION-3). + owner: Mapped[str | None] = mapped_column(String(255), nullable=True) + + runs: Mapped[list[RunRow]] = relationship( + back_populates="run_set", cascade="all, delete-orphan" + ) + + +class RunConfigRow(Base): + """A resolved configuration, keyed by its own hash. **Insert-only.** + + The primary key IS the identity (ADR-006): the same configuration submitted from the CLI and + from the API is one row, which is what makes the hash usable as a cache key. Two runs of the + same config share this row and differ only in their run id. + """ + + __tablename__ = "run_config" + + config_hash: Mapped[str] = mapped_column(_ID, primary_key=True) + schema_version: Mapped[str] = mapped_column(String(32)) + #: The resolved, post-derivation parameter set -- what the model actually received. + resolved_config: Mapped[dict[str, Any]] = mapped_column(JSON) + #: Derived fields the user overrode, with the value in force. + overrides: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + +class RunRow(Base): + """One simulation: its config, its lineage, and where its outputs live.""" + + __tablename__ = "run" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + run_set_id: Mapped[str] = mapped_column(_ID, ForeignKey("run_set.id"), index=True) + config_hash: Mapped[str] = mapped_column(_ID, ForeignKey("run_config.config_hash"), index=True) + #: The ensemble-style case label (``30N_20km__sabr220__…``). A label, never an identity -- it + #: captures the axes rather than the resolved configuration, so two sweeps differing only in a + #: "fixed" value would collide on it (ADR-006). + label: Mapped[str] = mapped_column(String(255), default="") + #: Lineage for an edited config: this run supersedes that one (ADR-004). Never a mutation. + derived_from_run_id: Mapped[str | None] = mapped_column( + _ID, ForeignKey("run.id"), nullable=True + ) + #: The provenance record (ADR-006) as written at submit time, stored verbatim. + provenance: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True) + #: False when any checkout was dirty: the SHAs do not describe the code that ran. + reproducible: Mapped[bool | None] = mapped_column(Integer, nullable=True) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + run_set: Mapped[RunSetRow] = relationship(back_populates="runs") + jobs: Mapped[list[JobRow]] = relationship(back_populates="run", cascade="all, delete-orphan") + artifacts: Mapped[list[ResultArtifactRow]] = relationship( + back_populates="run", cascade="all, delete-orphan" + ) + + __table_args__ = (Index("ix_run_config_hash_created", "config_hash", "created_at"),) + + +class JobRow(Base): + """Execution of a run. State lives here, not in memory, so a handle survives an API restart.""" + + __tablename__ = "job" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), index=True) + #: Current lifecycle state (``studio.runner.JobState``). Stored as its string value so the + #: database is readable without the enum, and a new state is a data change not a schema one. + state: Mapped[str] = mapped_column(String(32), index=True) + backend: Mapped[str] = mapped_column(String(64), default="local-subprocess") + work_dir: Mapped[str | None] = mapped_column(Text, nullable=True) + exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True) + detail: Mapped[str] = mapped_column(Text, default="") + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + updated_at: Mapped[datetime] = mapped_column(UtcDateTime) + + run: Mapped[RunRow] = relationship(back_populates="jobs") + transitions: Mapped[list[JobTransitionRow]] = relationship( + back_populates="job", cascade="all, delete-orphan", order_by="JobTransitionRow.at" + ) + + +class JobTransitionRow(Base): + """One timestamped state change. The audit trail, kept rather than collapsed to a status. + + "It failed" is not debuggable; "QUEUED 14:02:11, RUNNING 14:02:11, FAILED 14:06:48 exit 1" is. + """ + + __tablename__ = "job_transition" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + job_id: Mapped[str] = mapped_column(_ID, ForeignKey("job.id"), index=True) + state: Mapped[str] = mapped_column(String(32)) + at: Mapped[datetime] = mapped_column(UtcDateTime) + detail: Mapped[str] = mapped_column(Text, default="") + + job: Mapped[JobRow] = relationship(back_populates="transitions") + + +class ResultArtifactRow(Base): + """A pointer to one output file. **The bytes are not here.** + + Path, size, checksum and content type -- enough to find it, verify it, and notice when it has + gone missing. ``sha256`` is what makes "the file is still the one that was written" checkable + rather than assumed. + """ + + __tablename__ = "result_artifact" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), index=True) + #: ``state``, ``summary``, ``provenance``, ``stdout``, ``stderr``, ``input``. + kind: Mapped[str] = mapped_column(String(32)) + #: Relative to the artifact store's root, never absolute: the root moves between deployments. + path: Mapped[str] = mapped_column(Text) + size_bytes: Mapped[int] = mapped_column(Integer) + sha256: Mapped[str] = mapped_column(String(64)) + content_type: Mapped[str] = mapped_column(String(64), default="application/octet-stream") + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + run: Mapped[RunRow] = relationship(back_populates="artifacts") + + __table_args__ = (UniqueConstraint("run_id", "kind", name="uq_artifact_run_kind"),) + + +class DatasetVersionRow(Base): + """An input dataset consulted by a run, with its checksum (ADR-006). + + Empty in Phase 0 -- the schema carries no dataset inputs yet -- but present from the first + migration, because "which ERA5 product was this run built on?" is a question Phase 1 must be + able to answer about runs made before it existed. + """ + + __tablename__ = "dataset_version" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + identifier: Mapped[str] = mapped_column(String(255), index=True) + version: Mapped[str] = mapped_column(String(64)) + sha256: Mapped[str] = mapped_column(String(64)) + source: Mapped[str] = mapped_column(Text, default="") + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + __table_args__ = ( + UniqueConstraint("identifier", "version", name="uq_dataset_identifier_version"), + ) + + +class RunDatasetRow(Base): + """Which datasets a run consulted. Many-to-many, so a dataset is recorded once.""" + + __tablename__ = "run_dataset" + + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), primary_key=True) + dataset_id: Mapped[str] = mapped_column(_ID, ForeignKey("dataset_version.id"), primary_key=True) + + +class RunSummaryRow(Base): + """The versioned reduction (ADR-004), stored so comparison views never open the raw npz. + + Kept as JSON rather than shredded into columns: it is read whole, its shape is versioned, and + normalising it would make ``schema_version`` a migration problem instead of a field. + """ + + __tablename__ = "run_summary" + + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), primary_key=True) + schema_version: Mapped[str] = mapped_column(String(32)) + termination: Mapped[str] = mapped_column(String(32), index=True) + flags: Mapped[list[str]] = mapped_column(JSON, default=list) + #: Headline scalars promoted out of the JSON for querying and sorting without deserialising: + #: "show me every run where peak number exceeded X" must not require reading 810 blobs. + final_so2_pptv: Mapped[float | None] = mapped_column(Float, nullable=True) + peak_h2so4_pptv: Mapped[float | None] = mapped_column(Float, nullable=True) + peak_number_cm3: Mapped[float | None] = mapped_column(Float, nullable=True) + final_surface_area: Mapped[float | None] = mapped_column(Float, nullable=True) + summary: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + +__all__ = [ + "Base", + "DatasetVersionRow", + "JobRow", + "JobTransitionRow", + "ResultArtifactRow", + "RunConfigRow", + "RunDatasetRow", + "RunRow", + "RunSetRow", + "RunSummaryRow", + "UtcDateTime", +] diff --git a/studio/store/repository.py b/studio/store/repository.py new file mode 100644 index 0000000..778a8d7 --- /dev/null +++ b/studio/store/repository.py @@ -0,0 +1,259 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The operations the CLI, the API and the runner need -- and deliberately no others. + +**There is no update path for a config.** ADR-004 says a submitted config is immutable and an edit +produces a new config and a new run; a repository that offered ``update_config`` would make that a +convention people remember rather than a property of the system. :func:`ensure_config` is +get-or-create, keyed by the config's own hash, and is the only way a config enters the database. + +Everything here takes a ``Session`` rather than opening its own. Transaction boundaries belong to +the caller, because SQLite has a single writer (ADR-007) and only the caller knows what else belongs +in the same short transaction. +""" + +from __future__ import annotations + +import uuid +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from studio.resolve import ResolvedConfig +from studio.store.artifacts import ArtifactStore +from studio.store.models import ( + JobRow, + JobTransitionRow, + ResultArtifactRow, + RunConfigRow, + RunRow, + RunSetRow, + RunSummaryRow, +) + + +def _now() -> datetime: + return datetime.now(UTC) + + +def _new_id() -> str: + return uuid.uuid4().hex + + +def ensure_config(session: Session, config: ResolvedConfig) -> RunConfigRow: + """Get-or-create the config row. **The only way a config enters the database.** + + Idempotent by construction: the primary key is the config's own hash, so submitting the same + configuration from the CLI and from the API converges on one row rather than creating a second + identity for the same computation. + + Raises: + InconsistentConfigError: If the config has stale overrides. Persisting one would give an + inconsistent set of numbers a permanent identity. + """ + config.require_consistent() + config_hash = config.config.config_hash() + existing = session.get(RunConfigRow, config_hash) + if existing is not None: + return existing + row = RunConfigRow( + config_hash=config_hash, + schema_version=config.config.schema_version, + resolved_config=config.config.model_dump(mode="json"), + overrides={path: record.value for path, record in sorted(config.overrides.items())}, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def create_run_set( + session: Session, + *, + label: str = "", + axes: dict[str, Any] | None = None, + owner: str | None = None, +) -> RunSetRow: + """Create a sweep. A single run is the N = 1 case of one, with no axes.""" + row = RunSetRow(id=_new_id(), label=label, axes=axes or {}, owner=owner, created_at=_now()) + session.add(row) + session.flush() + return row + + +def create_run( + session: Session, + *, + run_set: RunSetRow, + config: ResolvedConfig, + label: str = "", + provenance: dict[str, Any] | None = None, + derived_from_run_id: str | None = None, +) -> RunRow: + """Create a run, ensuring its config exists first. + + ``provenance`` is the record written at submit time (ADR-006), stored verbatim. ``reproducible`` + is derived from it here rather than recomputed later, because it is a property of the moment the + run started, not of the checkout as it stands now. + """ + config_row = ensure_config(session, config) + reproducible: bool | None = None + if provenance is not None: + dirty = bool(provenance.get("sandbox", {}).get("dirty")) or any( + bool(sub.get("dirty")) for sub in provenance.get("submodules", {}).values() + ) + reproducible = not dirty + row = RunRow( + id=_new_id(), + run_set_id=run_set.id, + config_hash=config_row.config_hash, + label=label, + derived_from_run_id=derived_from_run_id, + provenance=provenance, + reproducible=reproducible, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def record_job( + session: Session, + *, + run: RunRow, + state: str, + backend: str = "local-subprocess", + work_dir: Path | str | None = None, + detail: str = "", +) -> JobRow: + """Create a job in its first state, with the matching transition.""" + now = _now() + job = JobRow( + id=_new_id(), + run_id=run.id, + state=state, + backend=backend, + work_dir=str(work_dir) if work_dir is not None else None, + detail=detail, + created_at=now, + updated_at=now, + ) + session.add(job) + session.add(JobTransitionRow(id=_new_id(), job_id=job.id, state=state, at=now, detail=detail)) + session.flush() + return job + + +def record_transition( + session: Session, *, job: JobRow, state: str, detail: str = "", exit_code: int | None = None +) -> JobRow: + """Advance a job and append the transition. + + Appends rather than replaces: the current state is a convenience column, and the transition list + is the record. A job whose state went backwards is visible here instead of being overwritten. + """ + now = _now() + job.state = state + job.detail = detail or job.detail + job.updated_at = now + if exit_code is not None: + job.exit_code = exit_code + session.add(JobTransitionRow(id=_new_id(), job_id=job.id, state=state, at=now, detail=detail)) + session.flush() + return job + + +def record_artifact( + session: Session, *, run: RunRow, kind: str, source: Path, store: ArtifactStore +) -> ResultArtifactRow: + """Store a file and record the pointer. The bytes never enter the database. + + Raises: + FileNotFoundError: If the file is missing -- surfaced rather than recorded as an absent row. + """ + stored = store.put(run.id, kind, Path(source)) + row = ResultArtifactRow( + id=_new_id(), + run_id=run.id, + kind=kind, + path=stored.key, + size_bytes=stored.size_bytes, + sha256=stored.sha256, + content_type=stored.content_type, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def record_summary(session: Session, *, run: RunRow, summary: dict[str, Any]) -> RunSummaryRow: + """Store the RunSummary, promoting four headline scalars into columns. + + Promoted so "every run whose peak number exceeded X" is a query rather than 810 + deserialisations. The scalars are read from the summary rather than recomputed, so the column + and the JSON can never disagree. + """ + series = summary.get("series", {}) + + def last(name: str) -> float | None: + values = series.get(name, {}).get("values") or [] + return float(values[-1]) if values else None + + def peak(name: str) -> float | None: + values = series.get(name, {}).get("values") or [] + return float(max(values)) if values else None + + row = RunSummaryRow( + run_id=run.id, + schema_version=str(summary.get("schema_version", "")), + termination=str(summary.get("termination", "unknown")), + flags=list(summary.get("flags", [])), + final_so2_pptv=last("SO2"), + peak_h2so4_pptv=peak("H2SO4"), + peak_number_cm3=peak("total_n"), + final_surface_area=last("SA"), + summary=summary, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def runs_for_config(session: Session, config_hash: str) -> list[RunRow]: + """Every run of one configuration, newest first. The cache lookup (ADR-006). + + Identical hash is necessary but **not sufficient** to reuse a result: the caller must also + compare the model version in each run's provenance, which is why this returns the runs rather + than a verdict. + """ + statement = ( + select(RunRow).where(RunRow.config_hash == config_hash).order_by(RunRow.created_at.desc()) + ) + return list(session.scalars(statement)) + + +def artifact_for(session: Session, run_id: str, kind: str) -> ResultArtifactRow | None: + statement = select(ResultArtifactRow).where( + ResultArtifactRow.run_id == run_id, ResultArtifactRow.kind == kind + ) + return session.scalars(statement).one_or_none() + + +__all__ = [ + "artifact_for", + "create_run", + "create_run_set", + "ensure_config", + "record_artifact", + "record_job", + "record_summary", + "record_transition", + "runs_for_config", +] diff --git a/studio/tests/unit/test_store.py b/studio/tests/unit/test_store.py new file mode 100644 index 0000000..e53193c --- /dev/null +++ b/studio/tests/unit/test_store.py @@ -0,0 +1,323 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Persistence: the schema, the immutability rules, and the model/migration agreement. + +Every test upgrades a real SQLite database **through the migrations**, never via +``Base.metadata.create_all``. Creating tables straight from the models would test a schema no +migration ever produced, leaving the migrations to be exercised for the first time on someone's real +database -- which is the failure Alembic-from-the-first-migration exists to prevent. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from sqlalchemy import inspect + +from studio.resolve import apply_change, resolve, set_override +from studio.schema import RunConfig +from studio.store import ( + LocalDirectoryStore, + artifact_for, + create_db_engine, + create_run, + create_run_set, + current_revision, + ensure_config, + record_artifact, + record_job, + record_summary, + record_transition, + runs_for_config, + session_factory, + session_scope, + sha256_of, + upgrade_to_head, +) + +EXPECTED_TABLES = { + "run_set", + "run", + "run_config", + "job", + "job_transition", + "result_artifact", + "dataset_version", + "run_dataset", + "run_summary", +} + + +@pytest.fixture +def database(tmp_path: Path): + """A migrated SQLite database and its session factory.""" + url = f"sqlite:///{tmp_path / 'studio.db'}" + upgrade_to_head(url) + engine = create_db_engine(url) + yield engine, session_factory(engine), url + engine.dispose() + + +@pytest.fixture +def store(tmp_path: Path) -> LocalDirectoryStore: + return LocalDirectoryStore(tmp_path / "artifacts") + + +@pytest.mark.tier_a +def test_migrating_creates_every_table(database) -> None: + engine, _, _ = database + assert EXPECTED_TABLES <= set(inspect(engine).get_table_names()) + assert current_revision(engine) is not None + + +@pytest.mark.tier_a +def test_upgrading_twice_is_idempotent(database) -> None: + """Startup must be able to call this unconditionally.""" + engine, _, url = database + before = current_revision(engine) + upgrade_to_head(url) + assert current_revision(engine) == before + + +@pytest.mark.tier_a +def test_the_models_and_the_migration_agree(database) -> None: + """The drift test: autogenerate against a migrated database must find nothing to do. + + Without it, a column added to ``models.py`` without a migration works on every developer machine + (where the table was created from the models by some other path) and fails on the first real + deployment. Here it fails immediately, in seconds. + """ + from alembic.autogenerate import compare_metadata + from alembic.migration import MigrationContext + + engine, _, _ = database + from studio.store.models import Base + + with engine.connect() as connection: + context = MigrationContext.configure(connection, opts={"compare_type": True}) + diff = compare_metadata(context, Base.metadata) + assert diff == [], ( + f"models.py and the migrations have drifted: {diff}. Generate a migration " + f"(`alembic -c studio/store/alembic.ini revision --autogenerate`) rather than editing the " + f"models alone." + ) + + +@pytest.mark.tier_a +def test_the_same_config_is_one_row(database) -> None: + """Identity is the hash (ADR-006): CLI and API submissions converge rather than duplicating.""" + _, factory, _ = database + config = resolve(RunConfig()) + with session_scope(factory) as session: + first = ensure_config(session, config) + second = ensure_config(session, resolve(RunConfig())) + assert first.config_hash == second.config_hash + assert first is second + + +@pytest.mark.tier_a +def test_an_edited_config_is_a_different_row(database) -> None: + """Configs are immutable: an edit produces a new config, never a mutation (ADR-004).""" + _, factory, _ = database + original = resolve(RunConfig()) + edited = apply_change(original, "site.temperature_k", 213.0) + with session_scope(factory) as session: + first = ensure_config(session, original) + second = ensure_config(session, edited) + assert first.config_hash != second.config_hash + assert first.resolved_config["site"]["temperature_k"] == 210.0 + assert second.resolved_config["site"]["temperature_k"] == 213.0 + + +@pytest.mark.tier_a +def test_the_repository_offers_no_way_to_update_a_config() -> None: + """Immutability is structural, not a convention someone has to remember.""" + import studio.store.repository as repository + + forbidden = [ + name + for name in dir(repository) + if any(word in name.lower() for word in ("update", "delete", "overwrite")) + and not name.startswith("_") + ] + assert forbidden == [], f"repository exposes mutation helpers: {forbidden}" + + +@pytest.mark.tier_a +def test_a_stale_config_is_refused(database) -> None: + """Persisting one would give an inconsistent set of numbers a permanent identity.""" + from studio.resolve import InconsistentConfigError + + _, factory, _ = database + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError), session_scope(factory) as session: + ensure_config(session, stale) + + +@pytest.mark.tier_a +def test_a_run_records_its_provenance_and_reproducibility(database) -> None: + """``reproducible`` describes the moment the run started, so it is stored, not recomputed.""" + _, factory, _ = database + provenance = { + "sandbox": {"commit": "a" * 40, "dirty": True}, + "submodules": {"tuvx-jax": {"commit": "b" * 40, "dirty": False}}, + } + with session_scope(factory) as session: + run_set = create_run_set(session, label="sweep") + run = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + label="case", + provenance=provenance, + ) + assert run.reproducible is False + assert run.provenance["sandbox"]["commit"] == "a" * 40 + + clean = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + provenance={"sandbox": {"dirty": False}, "submodules": {}}, + ) + assert clean.reproducible is True + + +@pytest.mark.tier_a +def test_job_transitions_are_appended_not_replaced(database) -> None: + """The transition list is the record; the state column is a convenience.""" + _, factory, _ = database + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + job = record_job(session, run=run, state="queued", work_dir="/tmp/x") + record_transition(session, job=job, state="running", detail="launched") + record_transition(session, job=job, state="failed", detail="exit 1", exit_code=1) + + assert job.state == "failed" + assert job.exit_code == 1 + assert [t.state for t in job.transitions] == ["queued", "running", "failed"] + assert all(t.at.tzinfo is not None for t in job.transitions), "timestamps must be aware" + + +@pytest.mark.tier_a +def test_an_artifact_stores_a_pointer_and_a_checksum(database, store, tmp_path: Path) -> None: + """The bytes stay on disk; the database gets a path, a size and a hash.""" + _, factory, _ = database + source = tmp_path / "state.npz" + source.write_bytes(b"not really an npz, but bytes are bytes") + + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + row = record_artifact(session, run=run, kind="state", source=source, store=store) + run_id = run.id + + assert row.sha256 == sha256_of(source) + assert row.size_bytes == source.stat().st_size + assert not Path(row.path).is_absolute(), "paths are relative to the store root" + assert store.verify(row.path, row.sha256) + with session_scope(factory) as session: + assert artifact_for(session, run_id, "state").sha256 == row.sha256 + + +@pytest.mark.tier_a +def test_a_missing_artifact_raises_rather_than_recording_an_absence(database, store) -> None: + """A run that was supposed to produce an artefact and did not is a failure to surface.""" + _, factory, _ = database + with pytest.raises(FileNotFoundError), session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + record_artifact( + session, run=run, kind="state", source=Path("/nonexistent.npz"), store=store + ) + + +@pytest.mark.tier_a +def test_corruption_is_detectable(database, store, tmp_path: Path) -> None: + """Why the checksum is stored: silent corruption and a tidied directory look the same.""" + _, factory, _ = database + source = tmp_path / "summary.json" + source.write_text('{"schema_version": "0.1.0"}', encoding="utf-8") + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + row = record_artifact(session, run=run, kind="summary", source=source, store=store) + stored = store.open_path(row.path) + key, digest = row.path, row.sha256 + + assert store.verify(key, digest) + stored.write_text('{"schema_version": "tampered"}', encoding="utf-8") + assert not store.verify(key, digest) + + +@pytest.mark.tier_a +def test_a_summary_promotes_headline_scalars_for_querying(database) -> None: + """So "every run where peak number exceeded X" is a query, not 810 deserialisations.""" + _, factory, _ = database + summary = { + "schema_version": "0.1.0", + "termination": "completed", + "flags": ["open_system_dilution"], + "series": { + "SO2": {"values": [3.3e9, 1.7e6]}, + "H2SO4": {"values": [0.0, 15.05, 2.1]}, + "total_n": {"values": [0.0, 3.07e6, 1.2e6]}, + "SA": {"values": [2.0, 41.3]}, + }, + } + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + row = record_summary(session, run=run, summary=summary) + + assert row.final_so2_pptv == 1.7e6 + assert row.peak_h2so4_pptv == 15.05 + assert row.peak_number_cm3 == 3.07e6 + assert row.final_surface_area == 41.3 + assert row.termination == "completed" + assert row.summary["series"]["SO2"]["values"][-1] == row.final_so2_pptv, "column and JSON agree" + + +@pytest.mark.tier_a +def test_runs_for_a_config_are_the_cache_lookup(database) -> None: + """Identical hash is necessary but not sufficient to reuse a result, so this returns runs. + + The caller still has to compare the model version in each run's provenance -- which is why this + is not ``cached_result_for()`` returning a verdict. + """ + _, factory, _ = database + config = resolve(RunConfig()) + with session_scope(factory) as session: + run_set = create_run_set(session) + create_run(session, run_set=run_set, config=config, label="first") + create_run(session, run_set=run_set, config=config, label="second") + found = runs_for_config(session, config.config.config_hash()) + assert {run.label for run in found} == {"first", "second"} + assert len({run.id for run in found}) == 2, "same config, two distinct runs" + + +@pytest.mark.tier_a +def test_foreign_keys_are_enforced_on_sqlite(database) -> None: + """Without the pragma, SQLite ignores them and Postgres does not -- silent divergence.""" + from sqlalchemy.exc import IntegrityError + + from studio.store.models import JobRow + + _, factory, _ = database + with pytest.raises(IntegrityError), session_scope(factory) as session: + session.add( + JobRow( + id="j", + run_id="does-not-exist", + state="queued", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ) + ) From 6b113bb253f3d730d31050c0fa5eabb8d6bd6d3c Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Fri, 14 Aug 2026 17:34:44 -0700 Subject: [PATCH 16/18] =?UTF-8?q?studio:=20the=20CLI=20=E2=80=94=20run,=20?= =?UTF-8?q?sweep,=20status=20(task=200.9c)=20(#86)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit studio: the CLI -- run, sweep, status (task 0.9c) Half the exit criteria: a run can be submitted from the CLI, produces a stored result with full provenance, and is readable afterwards from a different process. Verified end to end at 19 s for a 1-day/40-bin case -- resolve, provenance, persist, submit, wait, six artefacts and the summary recorded, exit succeeded -- and then `status` from a separate process after the first had exited, printing the full transition trail, every artefact, and "reproducible NO -- a checkout was dirty", which is the honest answer for a tree with uncommitted work. The CLI is not a wrapper over the API (ADR-002): it goes through the same schema, resolver, store and runner, so a sweep launched from a terminal and one launched from the web produce identical rows and identical provenance. --plan mirrors run_ensemble.py's plan verb, because deciding to spend 810 x 4.6 minutes should take a second command; it prints what would run, with each case's hash, and creates no row. Two things fixed by looking at real output rather than at tests. The persisted trail was thinner than the runner's -- the first run recorded queued -> succeeded and dropped running -- which defeats the point of persisting a trail at all, since that hides how long a job waited for a worker and whether a failed one ever started; it now copies every transition and a test asserts the sequence. And session.get() returns None, which I was passing straight into the repository where it would have failed frames later as an AttributeError about None; mypy caught it, and it now raises naming the row and the key. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 45 ++++- pyproject.toml | 8 + studio/cli/main.py | 331 ++++++++++++++++++++++++++++++++++ studio/tests/unit/test_cli.py | 203 +++++++++++++++++++++ 4 files changed, 586 insertions(+), 1 deletion(-) create mode 100644 studio/cli/main.py create mode 100644 studio/tests/unit/test_cli.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index d146a36..0c22c01 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -22,13 +22,56 @@ with full provenance, and the golden tests pass. | 0.6 `studio/runner` + job lifecycle | **done** (#72) | | 0.7 Golden-file harness (two tiers) | **done** (#70 measured, #79 asserted) | | 0.8 Four contained fixes in `coupled/` | not started | -| 0.9 Vertical slice: CLI + API + minimal UI | **in progress** — 0.9a provenance (#80), 0.9b persistence (#83); 0.9c–e to come | +| 0.9 Vertical slice: CLI + API + minimal UI | **in progress** — 0.9a provenance (#80), 0.9b persistence (#83), 0.9c CLI (#85); 0.9d–e to come | Task order note: 0.5 was taken **before 0.3**, so the dependency-graph engine has real derivations to resolve rather than fixtures. --- +### 2026-08-15 — Task 0.9c: the CLI (issue #85) + +**Half the exit criteria now works**: a run can be submitted from the CLI, produces a stored result +with full provenance, and is readable afterwards from a different process. 10 new Tier-A tests +(256 total). + +``` +plume-studio run config.yaml --out runs/ # 19 s for 1 day / 40 bins, end to end +plume-studio sweep sweep.yaml --plan # expand axes, print N, submit NOTHING +plume-studio status # from any process, after the CLI has exited +``` + +Verified end to end: `run` resolved, recorded provenance, persisted, submitted, waited, recorded six +artefacts and the summary, and exited `succeeded`. `status` — in a **separate process, after the +first had exited** — printed the full transition trail, every artefact with its size, and +`reproducible NO — a checkout was dirty`, which is the honest answer for a tree with uncommitted +work. + +**The CLI is not a wrapper over the API** (ADR-002). It goes through the same schema, resolver, +store and runner, so a sweep launched from a terminal and one launched from the web produce +identical rows and identical provenance. Scripted ensembles must not require the browser, and the +existing workflow is entirely scripted. + +**`--plan` mirrors `run_ensemble.py`'s `plan` verb** because deciding to spend 810 × 4.6 minutes +should take a second command. It prints what would run, with each case's hash, and creates no row. + +**Two things fixed after looking at real output rather than at tests:** + +1. **The persisted trail was thinner than the runner's.** The first end-to-end run recorded + `queued → succeeded`, dropping `running`. Persisting transitions is pointless if it drops one: + `queued → succeeded` hides how long a job waited for a worker, and `queued → failed` hides whether + it ever started. It now copies every transition the runner saw, and a test asserts the exact + sequence. +2. **`session.get()` returns `None`,** and I was passing it straight into the repository, where it + would have failed several frames later as an `AttributeError` about `None`. mypy caught it; it now + raises naming the row and the key, since it means the database changed under a run in flight. + +Exit codes keep 0.6's meaning: **2** for "never started" (a bad file, an invalid config, an unknown +run), **1** for a run that did not succeed. `--dry-run` and `--plan` are the cheap paths, so most of +the tests need neither the model nor a database. + +--- + ### 2026-08-14 — Task 0.9b: persistence (issue #83) `studio/store/`: models, engine, artefact store, repository, and **Alembic from the first diff --git a/pyproject.toml b/pyproject.toml index 64aae9b..9f6658f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -59,6 +59,11 @@ studio-dev = [ # the models themselves live in the tuvx-jax / stratchem-jax / tomas-jax git submodules # (put on sys.path by coupled/model_bridge.py and coupled/tomas_bridge.py); this project # installs the coupling layer + shared dependencies. +[project.scripts] +# The scripted path is first-class (ADR-002): a sweep launched from a terminal goes through the same +# schema, resolver, store and runner as one launched from the web. +plume-studio = "studio.cli.main:app" + [tool.setuptools.packages.find] where = ["."] include = ["coupled*", "studio*"] @@ -101,6 +106,9 @@ ignore = ["ANN401"] [tool.ruff.lint.per-file-ignores] "studio/tests/**" = ["ANN"] +# B008 flags function calls in argument defaults, which is exactly how typer declares options. +# Scoped to the CLI rather than disabled globally, where it catches real mutable-default bugs. +"studio/cli/main.py" = ["B008"] [tool.black] line-length = 100 diff --git a/studio/cli/main.py b/studio/cli/main.py new file mode 100644 index 0000000..796addc --- /dev/null +++ b/studio/cli/main.py @@ -0,0 +1,331 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``plume-studio`` -- the scripted path to everything the UI can do. + +**Scripted ensembles must not require the browser** (ADR-002). This is not a convenience wrapper +over the API: it goes through the same schema, resolver, store and runner, so a sweep launched from +a terminal and one launched from the web produce identical rows and identical provenance. + +Verbs stay recognisable to anyone who has used the existing runners, which share a +``plan | one | run `` shape (``run_ensemble.py:174``): + + plume-studio run config.yaml --out runs/ # one run, end to end + plume-studio sweep sweep.yaml --plan # expand axes, print N, submit NOTHING + plume-studio sweep sweep.yaml --out runs/ # expand and submit + plume-studio status # persisted state, from any process + +``--plan`` exists because deciding to spend 810 x 4.6 minutes should take a second command. It is +the same reflex the existing runners encode, and it prints what would run without creating a row. + +``status`` reads from the database rather than from memory, which is the point: job state outlives +the process that submitted it (ADR-007), so a crashed CLI does not orphan a running job. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +import typer +import yaml + +from studio.modelio.provenance import record_for +from studio.resolve import ResolvedConfig, resolve +from studio.schema import RunConfig, RunSet +from studio.store import ( + LocalDirectoryStore, + create_db_engine, + create_run, + create_run_set, + database_url, + record_artifact, + record_job, + record_summary, + record_transition, + session_factory, + session_scope, + upgrade_to_head, +) + +app = typer.Typer( + add_completion=False, + no_args_is_help=True, + help="Configure, run and inspect coupled SAI plume box-model simulations.", +) + +#: Artefacts recorded for every run, by kind -> filename in the job's work directory. The set is +#: fixed rather than "whatever the directory contains", so a missing one is an error rather than a +#: silently shorter list. +_ARTIFACTS = { + "input": "input.json", + "provenance": "provenance.json", + "state": "state.npz", + "summary": "summary.json", + "stdout": "stdout.log", + "stderr": "stderr.log", +} + + +def _load_mapping(path: Path) -> dict[str, Any]: + """Read a YAML or JSON document, or exit with a message naming the file. + + YAML by suffix, not by sniffing: a file called ``.json`` that happens to parse as YAML is still + a mistake worth reporting. + """ + if not path.is_file(): + typer.echo(f"[studio] no such file: {path}", err=True) + raise typer.Exit(code=2) + text = path.read_text(encoding="utf-8") + try: + data = json.loads(text) if path.suffix.lower() == ".json" else yaml.safe_load(text) + except (json.JSONDecodeError, yaml.YAMLError) as exc: + typer.echo( + f"[studio] {path} is not valid {path.suffix.lstrip('.') or 'YAML'}: {exc}", err=True + ) + raise typer.Exit(code=2) from exc + if not isinstance(data, dict): + typer.echo(f"[studio] {path} must contain a mapping, got {type(data).__name__}", err=True) + raise typer.Exit(code=2) + return data + + +def _resolved_config(path: Path) -> ResolvedConfig: + """Load a RunConfig document and resolve its derived fields.""" + try: + return resolve(RunConfig.model_validate(_load_mapping(path))) + except ValueError as exc: + typer.echo(f"[studio] {path} is not a valid configuration:\n{exc}", err=True) + raise typer.Exit(code=2) from exc + + +def _run_set(path: Path) -> RunSet: + try: + return RunSet.model_validate(_load_mapping(path)) + except ValueError as exc: + typer.echo(f"[studio] {path} is not a valid sweep:\n{exc}", err=True) + raise typer.Exit(code=2) from exc + + +def _prepare(database: str | None, out: Path) -> tuple[Any, LocalDirectoryStore]: + """Migrate the database and open the artefact store. Idempotent: startup can always call it.""" + url = database_url(database) + upgrade_to_head(url) + return session_factory(create_db_engine(url)), LocalDirectoryStore(out / "artifacts") + + +def _require(row: Any, what: str, key: str) -> Any: + """Return ``row``, or raise saying what went missing. + + ``Session.get`` returns ``None`` for a row that is not there, and passing that on would fail + several frames later with an ``AttributeError`` about ``None``. Here it means the database + changed underneath a run that is mid-flight -- rare, and worth naming precisely when it happens. + """ + if row is None: + raise RuntimeError( + f"{what} {key!r} vanished from the database while its run was in progress; " + f"the run may have been deleted concurrently" + ) + return row + + +def _submit_and_record( + factory: Any, + store: LocalDirectoryStore, + runner: Any, + *, + config: ResolvedConfig, + label: str, + run_set_row: Any, + wait: bool, +) -> tuple[str, str]: + """Persist a run, submit it, and record what came back. Returns ``(run_id, job_state)``.""" + provenance = record_for(config) + with session_scope(factory) as session: + run = create_run( + session, + run_set=run_set_row, + config=config, + label=label, + provenance=provenance.model_dump(mode="json"), + ) + run_id = run.id + + record = runner.submit(config, label=label) + with session_scope(factory) as session: + from studio.store.models import RunRow + + run_row = _require(session.get(RunRow, run_id), "run", run_id) + job = record_job( + session, + run=run_row, + state=record.state.value, + work_dir=record.work_dir, + detail=record.detail, + ) + job_id = job.id + + if not wait: + return run_id, record.state.value + + final = runner.wait(record.job_id) + with session_scope(factory) as session: + from studio.store.models import JobRow, RunRow + + job_row = _require(session.get(JobRow, job_id), "job", job_id) + # EVERY transition the runner saw, not just the final state. A thinner trail than the + # runner's defeats the point of persisting one: "queued -> succeeded" hides how long it + # waited for a worker, and "queued -> failed" hides whether it ever started. + for transition in final.transitions[1:]: + record_transition( + session, + job=job_row, + state=transition.state.value, + detail=transition.detail, + exit_code=final.exit_code if transition.state is final.state else None, + ) + run_row = _require(session.get(RunRow, run_id), "run", run_id) + for kind, filename in _ARTIFACTS.items(): + source = Path(final.work_dir or ".") / filename + if source.is_file(): + record_artifact(session, run=run_row, kind=kind, source=source, store=store) + summary_path = Path(final.work_dir or ".") / "summary.json" + if summary_path.is_file(): + record_summary( + session, run=run_row, summary=json.loads(summary_path.read_text(encoding="utf-8")) + ) + return run_id, final.state.value + + +@app.command() +def run( + config: Path = typer.Argument(..., help="RunConfig as YAML or JSON"), + out: Path = typer.Option(Path("var/studio/runs"), "--out", help="where jobs and artefacts go"), + label: str = typer.Option( + "", "--label", help="human-readable name; identity is still the hash" + ), + wait: bool = typer.Option(True, "--wait/--no-wait", help="block until the run finishes"), + database: str | None = typer.Option(None, "--database", help="override STUDIO_DATABASE_URL"), + dry_run: bool = typer.Option( + False, "--dry-run", help="resolve and print the identity; touch nothing" + ), +) -> None: + """Run one configuration, end to end.""" + resolved = _resolved_config(config) + typer.echo(f"[studio] config hash {resolved.config.config_hash()}") + typer.echo(f"[studio] initial SO2 {resolved.config.injection.so2_initial_pptv:.6e} pptv") + if dry_run: + typer.echo("[studio] --dry-run: nothing written, nothing submitted") + return + + from studio.runner import LocalSubprocessRunner + + factory, store = _prepare(database, out) + runner = LocalSubprocessRunner(out / "jobs") + try: + with session_scope(factory) as session: + run_set_row = create_run_set(session, label=label or config.stem) + run_id, state = _submit_and_record( + factory, store, runner, config=resolved, label=label, run_set_row=run_set_row, wait=wait + ) + finally: + runner.shutdown() + typer.echo(f"[studio] run {run_id} -> {state}") + if state != "succeeded" and wait: + raise typer.Exit(code=1) + + +@app.command() +def sweep( + sweep_file: Path = typer.Argument(..., help="RunSet as YAML or JSON: base config plus axes"), + out: Path = typer.Option(Path("var/studio/runs"), "--out", help="where jobs and artefacts go"), + plan: bool = typer.Option( + False, "--plan", help="print what would run and submit NOTHING (mirrors run_ensemble plan)" + ), + database: str | None = typer.Option(None, "--database", help="override STUDIO_DATABASE_URL"), +) -> None: + """Expand a sweep into runs, and submit them unless asked to plan.""" + run_set = _run_set(sweep_file) + typer.echo(f"[studio] {run_set.size()} run(s) from {len(run_set.axes)} axis/axes") + expanded = run_set.expand() + if plan: + for index, item in enumerate(expanded): + typer.echo(f" [{index}] {item.label or '(unlabelled)'} {item.config_hash[:12]}") + typer.echo("[studio] --plan: nothing submitted") + return + + from studio.runner import LocalSubprocessRunner + + factory, store = _prepare(database, out) + runner = LocalSubprocessRunner(out / "jobs") + failures = 0 + try: + with session_scope(factory) as session: + run_set_row = create_run_set( + session, + label=sweep_file.stem, + axes=run_set.model_dump(mode="json")["axes"], + ) + for item in expanded: + run_id, state = _submit_and_record( + factory, + store, + runner, + config=resolve(item.config), + label=item.label, + run_set_row=run_set_row, + wait=True, + ) + typer.echo(f"[studio] {item.label or run_id} -> {state}") + failures += state != "succeeded" + finally: + runner.shutdown() + if failures: + typer.echo(f"[studio] {failures} run(s) did not succeed", err=True) + raise typer.Exit(code=1) + + +@app.command() +def status( + run_id: str = typer.Argument(..., help="run id, as printed by `run` or `sweep`"), + database: str | None = typer.Option(None, "--database", help="override STUDIO_DATABASE_URL"), +) -> None: + """Show a run's persisted state. + + Reads the database, not memory: job state outlives the process that submitted it, so this works + from a different shell, after a restart, or while the run is still going. + """ + from studio.store.models import RunRow + + factory = session_factory(create_db_engine(database_url(database))) + with session_scope(factory) as session: + run = session.get(RunRow, run_id) + if run is None: + typer.echo(f"[studio] no run {run_id!r} in this database", err=True) + raise typer.Exit(code=2) + typer.echo(f"run {run.id}") + typer.echo(f"label {run.label or '(none)'}") + typer.echo(f"config {run.config_hash}") + typer.echo(f"created {run.created_at.isoformat()}") + reproducible = {True: "yes", False: "NO — a checkout was dirty", None: "unknown"} + typer.echo(f"reproducible {reproducible[run.reproducible]}") + for job in run.jobs: + typer.echo(f"job {job.id} {job.state} exit={job.exit_code}") + for transition in job.transitions: + typer.echo( + f" {transition.at.isoformat()} {transition.state} {transition.detail}" + ) + for artifact in run.artifacts: + typer.echo( + f"artifact {artifact.kind:10s} {artifact.size_bytes:>10,d} B {artifact.path}" + ) + + +def main() -> int: + app() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/studio/tests/unit/test_cli.py b/studio/tests/unit/test_cli.py new file mode 100644 index 0000000..5d71281 --- /dev/null +++ b/studio/tests/unit/test_cli.py @@ -0,0 +1,203 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The CLI: the scripted path to everything the UI can do (ADR-002). + +Most of these need neither the model nor a long run -- ``--plan`` and ``--dry-run`` exist precisely +so that deciding to spend compute is a separate act from spending it, and that makes them cheap to +test. The one end-to-end test runs the real model for one simulated day and skips without the +submodules, like every other model-touching test. + +No test hooks: the CLI is exercised through its real arguments, and the seams it uses +(``--database``, ``--out``) are ones a real deployment uses too. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import yaml +from typer.testing import CliRunner + +from studio.cli.main import app +from studio.schema import RunConfig + +runner = CliRunner() + + +@pytest.fixture +def config_file(tmp_path: Path) -> Path: + path = tmp_path / "config.yaml" + path.write_text( + yaml.safe_dump({"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}), + encoding="utf-8", + ) + return path + + +@pytest.fixture +def sweep_file(tmp_path: Path) -> Path: + path = tmp_path / "sweep.yaml" + path.write_text( + yaml.safe_dump( + { + "base": {"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}, + "axes": [ + { + "name": "nucleation", + "kind": "grid", + "points": [ + { + "label": "nuc1", + "assignments": {"microphysics.nucleation_rate_scale": 1.0}, + }, + { + "label": "nuc100", + "assignments": {"microphysics.nucleation_rate_scale": 100.0}, + }, + ], + } + ], + } + ), + encoding="utf-8", + ) + return path + + +@pytest.mark.tier_a +def test_a_dry_run_prints_the_identity_and_touches_nothing( + config_file: Path, tmp_path: Path +) -> None: + """Identity before compute: the hash is knowable without running anything.""" + result = runner.invoke( + app, ["run", str(config_file), "--dry-run", "--out", str(tmp_path / "o")] + ) + assert result.exit_code == 0, result.output + assert "config hash" in result.output + assert "nothing written, nothing submitted" in result.output + assert not (tmp_path / "o").exists(), "--dry-run must not create the output tree" + + +@pytest.mark.tier_a +def test_the_dry_run_hash_is_the_schema_hash(config_file: Path) -> None: + """The CLI must not have its own idea of identity.""" + from studio.resolve import resolve + + expected = resolve( + RunConfig.model_validate({"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}) + ).config.config_hash() + result = runner.invoke(app, ["run", str(config_file), "--dry-run"]) + assert expected in result.output + + +@pytest.mark.tier_a +def test_plan_lists_the_runs_and_submits_nothing(sweep_file: Path, tmp_path: Path) -> None: + """Mirrors ``run_ensemble.py``'s ``plan`` verb: deciding to spend compute is its own command.""" + result = runner.invoke(app, ["sweep", str(sweep_file), "--plan", "--out", str(tmp_path / "o")]) + assert result.exit_code == 0, result.output + assert "2 run(s) from 1 axis/axes" in result.output + assert "nuc1" in result.output and "nuc100" in result.output + assert "nothing submitted" in result.output + assert not (tmp_path / "o").exists() + + +@pytest.mark.tier_a +def test_planned_runs_have_distinct_identities(sweep_file: Path) -> None: + """A sweep whose points collided would silently run the same case twice.""" + result = runner.invoke(app, ["sweep", str(sweep_file), "--plan"]) + hashes = [line.split()[-1] for line in result.output.splitlines() if line.startswith(" [")] + assert len(hashes) == len(set(hashes)) == 2 + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("contents", "message"), + [ + ("not: [valid", "not valid"), + ("- a\n- b\n", "must contain a mapping"), + ("site:\n temperature_k: -5\n", "not a valid configuration"), + ], +) +def test_a_bad_config_file_exits_two_with_a_message( + tmp_path: Path, contents: str, message: str +) -> None: + """Exit 2 is "we never started", distinct from a model failure (exit 1).""" + path = tmp_path / "bad.yaml" + path.write_text(contents, encoding="utf-8") + result = runner.invoke(app, ["run", str(path), "--dry-run"]) + assert result.exit_code == 2 + assert message in result.output + + +@pytest.mark.tier_a +def test_a_missing_file_names_itself(tmp_path: Path) -> None: + result = runner.invoke(app, ["run", str(tmp_path / "absent.yaml"), "--dry-run"]) + assert result.exit_code == 2 + assert "no such file" in result.output + + +@pytest.mark.tier_a +def test_status_of_an_unknown_run_exits_two(tmp_path: Path) -> None: + """Distinguishes "no such run" from "a run with nothing to report".""" + from studio.store import upgrade_to_head + + url = f"sqlite:///{tmp_path / 'studio.db'}" + upgrade_to_head(url) + result = runner.invoke(app, ["status", "nope", "--database", url]) + assert result.exit_code == 2 + assert "no run" in result.output + + +@pytest.mark.tier_a +def test_a_run_is_persisted_and_readable_from_a_new_process( + config_file: Path, tmp_path: Path, repo_root: Path +) -> None: + """The vertical slice's CLI half, end to end, with the real model. ~20 s. + + ``status`` runs against a session opened after the run finished -- the point being that job + state lives in the database rather than in the submitting process's memory (ADR-007), so a + crashed or exited CLI leaves a readable record rather than an orphan. + """ + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + + url = f"sqlite:///{tmp_path / 'studio.db'}" + out = tmp_path / "runs" + result = runner.invoke( + app, ["run", str(config_file), "--out", str(out), "--database", url, "--label", "slice"] + ) + assert result.exit_code == 0, result.output + assert "succeeded" in result.output + + from sqlalchemy import select + + from studio.store import create_db_engine, session_factory, session_scope + from studio.store.models import RunRow + + factory = session_factory(create_db_engine(url)) + with session_scope(factory) as session: + run = session.scalars(select(RunRow)).one() + run_id = run.id + assert run.label == "slice" + assert run.provenance is not None, "provenance is recorded at submit (ADR-006)" + assert set(run.provenance["submodules"]), "the model is pinned, not just the app" + kinds = {artifact.kind for artifact in run.artifacts} + assert {"input", "provenance", "state", "summary", "stdout"} <= kinds + (job,) = run.jobs + assert job.state == "succeeded" + assert [t.state for t in job.transitions] == [ + "queued", + "running", + "succeeded", + ], "the persisted trail must be as complete as the runner's, or persisting it is pointless" + + status = runner.invoke(app, ["status", run_id, "--database", url]) + assert status.exit_code == 0, status.output + assert run_id in status.output + assert "succeeded" in status.output + assert "state" in status.output and "summary" in status.output + + summary_row = json.loads((out / "artifacts" / run_id / "summary.json").read_text()) + assert summary_row["config_hash"] == run.config_hash, "the summary carries the same identity" From 3f1a10143e6a26baa29d45cce4c327496027de6e Mon Sep 17 00:00:00 2001 From: Ali Akherati <26212821+aliakherati@users.noreply.github.com> Date: Mon, 17 Aug 2026 14:31:28 -0700 Subject: [PATCH 17/18] studio: the API, the page, and the service both front ends share (0.9d, 0.9e) (#87) Phase 0's exit criteria are met: a run can be submitted from the CLI and from the web UI, produces a stored result with full provenance, and the golden tests pass. studio/service.py was extracted the moment there were two callers. A second copy of "resolve, record provenance, persist, submit, follow, store artefacts" would drift within a week and the drift would be invisible, since both paths would keep working and only their database rows would disagree. Verified against a real uvicorn server rather than only a test client: POST returns 202, the SSE stream reports running immediately and succeeded at t+20 s, and the run lists with reproducible false and termination completed. Three bugs came from looking at output rather than at green tests. The database never showed running, because transitions were written only after a job finished -- a trail, but useless as progress -- so finalise now follows the runner and persists each transition as it happens. reproducible came back as 0 rather than false, an Integer column where a Boolean belonged; the drift test caught it the moment the model changed and the second migration took one command. And SSE looked broken under TestClient, which serialises requests: the test now runs a real server in a thread and changes state while the stream is open, the only shape that catches a stream reporting nothing until the end. Deliberate divergence from ADR-007, recorded rather than silent: the page is one self-contained HTML file rather than React + Vite. A build toolchain for a single form is machinery ahead of need, the same reasoning that kept Redis and Docker out of Phase 0, and the repository already has precedent in coupled/viz/*.html. /api/schema exists so the form can be generated when the UI outgrows one form. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) --- docs/studio/PROGRESS.md | 58 +++- pyproject.toml | 1 + studio/api/__init__.py | 4 +- studio/api/app.py | 277 ++++++++++++++++ studio/api/static/index.html | 309 ++++++++++++++++++ studio/cli/main.py | 124 +------ studio/requirements.lock | 28 +- studio/service.py | 219 +++++++++++++ .../c0236fb572d2_reproducible_is_a_boolean.py | 36 ++ studio/store/models.py | 3 +- studio/tests/unit/test_api.py | 252 ++++++++++++++ 11 files changed, 1189 insertions(+), 122 deletions(-) create mode 100644 studio/api/app.py create mode 100644 studio/api/static/index.html create mode 100644 studio/service.py create mode 100644 studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py create mode 100644 studio/tests/unit/test_api.py diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 0c22c01..029205f 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -22,13 +22,69 @@ with full provenance, and the golden tests pass. | 0.6 `studio/runner` + job lifecycle | **done** (#72) | | 0.7 Golden-file harness (two tiers) | **done** (#70 measured, #79 asserted) | | 0.8 Four contained fixes in `coupled/` | not started | -| 0.9 Vertical slice: CLI + API + minimal UI | **in progress** — 0.9a provenance (#80), 0.9b persistence (#83), 0.9c CLI (#85); 0.9d–e to come | +| 0.9 Vertical slice: CLI + API + minimal UI | **done** — 0.9a provenance (#80), 0.9b persistence (#83), 0.9c CLI (#85), 0.9d API + 0.9e UI (#87) | Task order note: 0.5 was taken **before 0.3**, so the dependency-graph engine has real derivations to resolve rather than fixtures. --- +### 2026-08-15 — Tasks 0.9d and 0.9e: the API, the page, and the shared service + +**Phase 0's exit criteria are met.** A run can be submitted from the CLI *and* from the web UI, +produces a stored result with full provenance, and the golden tests pass. 13 new Tier-A tests +(269 total). + +**One flow, two front ends.** `studio/service.py` was extracted the moment there were two callers: +a second copy of "resolve, record provenance, persist, submit, follow, store artefacts" would drift +within a week, and the drift would be invisible — both paths would keep working and only their rows +would disagree. ADR-002's "identical rows from either path" is a property of there being one +function, not of two being written carefully. + +**Verified against a real server**, not only a test client: + +``` +POST /api/runs -> 202 {"state":"queued"} +SSE t+ 0.0s running + t+20.1s succeeded +GET /api/runs -> reproducible: false · termination: completed + SO2 3.309e9 -> 1.720e6 pptv · peak N 3.07e6 cm^-3 +``` + +**Three bugs found by looking at output rather than at green tests:** + +1. **The database never showed `running`.** Transitions were written only after a job finished, so + the stream would sit at `queued` for four minutes and then jump to `succeeded` — a trail, but + useless as progress. `finalise` now follows the runner and persists each transition as it + happens, which is also why the stream can read the *database* and still be live. +2. **`reproducible` came back as `0`, not `false`** — an `Integer` column where a `Boolean` + belonged. The **drift test caught it** the moment the model changed, and the second migration + took one command. That is the return on putting Alembic in at 0.9b rather than later. +3. **SSE looked broken under `TestClient`**, reporting only the terminal state. It was the test + client serialising requests, not the code. The SSE test now runs a real uvicorn server in a + thread and changes state while the stream is open — the only shape that can catch a stream which + reports nothing until the end. + +**Deliberate divergence from ADR-007: no React + Vite.** The page is one self-contained HTML file +served by FastAPI. A build toolchain for a single form is machinery ahead of need — the same +reasoning that kept Redis and Docker out of Phase 0 — and the repository already has precedent in +`coupled/viz/*.html`. Recorded here rather than taken silently; React earns its place when the UI +outgrows one form, and `/api/schema` already exists so the form can be generated rather than +hand-written when it does. + +**The figure is deterministic from `RunSummary`** (ADR-004), drawn as inline SVG from the stored +summary — never from the raw npz. The same summary always draws the same figure, so a lost figure is +never a lost result. + +Smaller points: `POST /api/runs` returns **202**, because the run is accepted rather than finished, +and finalising happens on a worker thread — a four-minute `await` would stall every other request +including the stream reporting on that very run. Invalid configs are **422**, including +`heating_to_t: true`, which the schema refuses because the model cannot represent the physics +(SCIENCE-4); that refusal reaches the browser rather than crashing the server. `httpx2` joins +`studio-dev` as starlette's test-client dependency. + +--- + ### 2026-08-15 — Task 0.9c: the CLI (issue #85) **Half the exit criteria now works**: a run can be submitted from the CLI, produces a stored result diff --git a/pyproject.toml b/pyproject.toml index 9f6658f..76e6642 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -51,6 +51,7 @@ studio = [ # studio linting/typing; separate from `dev` so the model's suite is unaffected studio-dev = [ "ruff>=0.5", + "httpx2>=0.1", # starlette's TestClient dependency; plain httpx is deprecated for it "mypy>=1.10", "black>=24.3", "hypothesis>=6.100", # property tests for unit round-trips across the valid T-p range diff --git a/studio/api/__init__.py b/studio/api/__init__.py index bbcf6be..9889688 100644 --- a/studio/api/__init__.py +++ b/studio/api/__init__.py @@ -35,4 +35,6 @@ from __future__ import annotations -__all__: list[str] = [] +from studio.api.app import create_app + +__all__ = ["create_app"] diff --git a/studio/api/app.py b/studio/api/app.py new file mode 100644 index 0000000..54109d3 --- /dev/null +++ b/studio/api/app.py @@ -0,0 +1,277 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The FastAPI application: the web half of the Phase-0 slice. + +Every endpoint here is a thin shell over code the CLI already uses -- the schema, the resolver, +``studio.service``, the store. The API adds HTTP and progress streaming and nothing else, which is +what makes "identical rows from either path" (ADR-002) true rather than aspirational. + +**Progress is pushed, not polled** (spec 7.3). ``/api/events/runs/{id}`` is a Server-Sent Events +stream; it works without a broker because this process owns the worker pool (ADR-007). Job state is +read from the **database** rather than the pool, so the stream is correct for a run this process did +not submit and survives a restart of the one that did. + +**Auth is delegated to a reverse proxy** and there is no user model (ASSUMPTION-3, BLOCKING-2). This +is also the first component in the repository that serves anything, so it is the first place the +private-repo / public-page boundary can be enforced by code: it binds to localhost by default, and +nothing here writes to ``gh-pages`` or any public location. +""" + +from __future__ import annotations + +import asyncio +import json +import os +from collections.abc import AsyncIterator +from pathlib import Path +from typing import Any + +from fastapi import FastAPI, HTTPException +from fastapi.responses import FileResponse, HTMLResponse +from pydantic import BaseModel, Field +from sqlalchemy import select +from sse_starlette.sse import EventSourceResponse + +from studio.resolve import InconsistentConfigError, ResolvedConfig, resolve +from studio.runner import LocalSubprocessRunner +from studio.schema import RunConfig, run_config_json_schema +from studio.service import create_pending_run, finalise, prepare, submit +from studio.store import create_run_set, session_scope +from studio.store.models import JobRow, RunRow, RunSummaryRow + +#: Where runs, artefacts and the database live. Overridable for tests and for a deployment that +#: keeps its data somewhere other than the working directory. +STUDIO_HOME_ENV = "STUDIO_HOME" + +#: How often the SSE stream re-reads job state. 0.5 s is well under human reaction time and far +#: above the cost of one indexed SELECT; the alternative -- pushing from the worker thread -- would +#: only work for runs this process submitted. +_POLL_INTERVAL_S = 0.5 + +_STATIC = Path(__file__).resolve().parent / "static" + + +class SubmitRequest(BaseModel): + """A configuration to run. Partial: anything omitted takes the schema's default.""" + + config: dict[str, Any] = Field(default_factory=dict) + label: str = "" + + +class ResolveRequest(BaseModel): + config: dict[str, Any] = Field(default_factory=dict) + + +def create_app(home: Path | None = None, database: str | None = None) -> FastAPI: + """Build the application. + + A factory rather than a module-level singleton so tests get an isolated home and database + without monkeypatching, and so a deployment can run two instances against different data. + """ + root = Path(home or os.environ.get(STUDIO_HOME_ENV, "var/studio")) + factory, store = prepare(database, root / "runs") + runner = LocalSubprocessRunner(root / "runs" / "jobs") + + app = FastAPI( + title="Plume Studio", + version="0.1.0", + summary="Configure, run and compare coupled SAI plume box-model simulations.", + ) + app.state.factory = factory + app.state.store = store + app.state.runner = runner + + @app.get("/", response_class=HTMLResponse, include_in_schema=False) + def index() -> HTMLResponse: + return HTMLResponse((_STATIC / "index.html").read_text(encoding="utf-8")) + + @app.get("/api/schema") + def get_schema() -> dict[str, Any]: + """The JSON Schema the UI generates its form from (ADR-002). + + Nothing in the UI may invent a field that does not exist here, which is why the form is + driven by this rather than by a hand-written list that would drift. + """ + return run_config_json_schema() + + @app.post("/api/config/resolve") + def resolve_config(request: ResolveRequest) -> dict[str, Any]: + """Apply the derivations and return the resolved config, its identity, and any stale fields. + + Cheap by construction: no model import, no JAX (see the import-boundary tests), so the UI + can call this on every edit. + """ + try: + resolved = resolve(RunConfig.model_validate(request.config)) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + return { + "config": resolved.config.model_dump(mode="json"), + "config_hash": resolved.config.config_hash(), + "stale_fields": list(resolved.stale_fields), + "derived": { + "plume_volume_cm3": resolved.config.injection.plume_volume_cm3, + "so2_initial_pptv": resolved.config.injection.so2_initial_pptv, + }, + } + + @app.post("/api/runs", status_code=202) + async def create_run_endpoint(request: SubmitRequest) -> dict[str, Any]: + """Submit a run and return immediately with its identity. + + 202, not 200: the run has been accepted and is not finished. The client follows + ``/api/events/runs/{id}`` for progress rather than holding a request open for four minutes. + """ + try: + resolved: ResolvedConfig = resolve(RunConfig.model_validate(request.config)) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + try: + with session_scope(factory) as session: + run_set = create_run_set(session, label=request.label or "web") + run_id = create_pending_run( + factory, run_set=run_set, config=resolved, label=request.label + ) + job_id, runner_job_id = submit( + factory, runner, run_id=run_id, config=resolved, label=request.label + ) + except InconsistentConfigError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + + # Finalising blocks on the subprocess, so it goes to a worker thread rather than the event + # loop: a four-minute await here would stall every other request, including the SSE stream + # reporting on this very run. + asyncio.get_running_loop().run_in_executor( + None, + lambda: finalise( + factory, + store, + runner, + run_id=run_id, + job_id=job_id, + runner_job_id=runner_job_id, + ), + ) + return { + "run_id": run_id, + "config_hash": resolved.config.config_hash(), + "state": "queued", + } + + @app.get("/api/runs") + def list_runs(limit: int = 50) -> list[dict[str, Any]]: + """Recent runs, newest first.""" + with session_scope(factory) as session: + rows = session.scalars( + select(RunRow).order_by(RunRow.created_at.desc()).limit(limit) + ).all() + return [_run_brief(session, row) for row in rows] + + @app.get("/api/runs/{run_id}") + def get_run(run_id: str) -> dict[str, Any]: + with session_scope(factory) as session: + run = session.get(RunRow, run_id) + if run is None: + raise HTTPException(status_code=404, detail=f"no run {run_id!r}") + detail = _run_brief(session, run) + detail["provenance"] = run.provenance + detail["artifacts"] = [ + { + "kind": artifact.kind, + "size_bytes": artifact.size_bytes, + "sha256": artifact.sha256, + } + for artifact in run.artifacts + ] + detail["transitions"] = [ + {"state": t.state, "at": t.at.isoformat(), "detail": t.detail} + for job in run.jobs + for t in job.transitions + ] + return detail + + @app.get("/api/runs/{run_id}/summary") + def get_summary(run_id: str) -> dict[str, Any]: + """The RunSummary. Comparison views and figures read this, never the raw npz (ADR-004).""" + with session_scope(factory) as session: + row = session.get(RunSummaryRow, run_id) + if row is None: + raise HTTPException(status_code=404, detail=f"run {run_id!r} has no summary yet") + return dict(row.summary) + + @app.get("/api/runs/{run_id}/artifacts/{kind}") + def get_artifact(run_id: str, kind: str) -> FileResponse: + """Serve a stored artefact from the store, by kind.""" + from studio.store import artifact_for + + with session_scope(factory) as session: + row = artifact_for(session, run_id, kind) + if row is None: + raise HTTPException(status_code=404, detail=f"run {run_id!r} has no {kind!r}") + path, media_type, filename = row.path, row.content_type, Path(row.path).name + return FileResponse(store.open_path(path), media_type=media_type, filename=filename) + + @app.get("/api/events/runs/{run_id}") + async def stream_run(run_id: str) -> EventSourceResponse: + """Progress over SSE, read from the database rather than the worker pool. + + From the database on purpose: the stream is then correct for a run submitted by the CLI or + by a previous instance of this process, and does not depend on the job being in *this* + pool's memory. + """ + + async def events() -> AsyncIterator[dict[str, str]]: + last: str | None = None + while True: + with session_scope(factory) as session: + run = session.get(RunRow, run_id) + if run is None: + yield {"event": "error", "data": json.dumps({"detail": "unknown run"})} + return + state = run.jobs[-1].state if run.jobs else "pending" + payload = _run_brief(session, run) + if state != last: + yield {"event": "state", "data": json.dumps(payload)} + last = state + if state in {"succeeded", "failed", "cancelled", "terminated_on_limit"}: + return + await asyncio.sleep(_POLL_INTERVAL_S) + + return EventSourceResponse(events()) + + return app + + +def _run_brief(session: Any, run: RunRow) -> dict[str, Any]: + """The shape every endpoint returns for a run. One function, so they cannot disagree.""" + summary = session.get(RunSummaryRow, run.id) + job: JobRow | None = run.jobs[-1] if run.jobs else None + return { + "run_id": run.id, + "label": run.label, + "config_hash": run.config_hash, + "created_at": run.created_at.isoformat(), + "reproducible": run.reproducible, + "state": job.state if job else "pending", + "exit_code": job.exit_code if job else None, + "detail": job.detail if job else "", + "termination": summary.termination if summary else None, + "headline": ( + { + "final_so2_pptv": summary.final_so2_pptv, + "peak_h2so4_pptv": summary.peak_h2so4_pptv, + "peak_number_cm3": summary.peak_number_cm3, + "final_surface_area": summary.final_surface_area, + "flags": summary.flags, + } + if summary + else None + ), + } + + +#: Module-level app for ``uvicorn studio.api.app:app``. Built from the environment, so a deployment +#: needs no code; tests call ``create_app`` with their own home instead. +app = create_app() + +__all__ = ["STUDIO_HOME_ENV", "app", "create_app"] diff --git a/studio/api/static/index.html b/studio/api/static/index.html new file mode 100644 index 0000000..6c5b6d6 --- /dev/null +++ b/studio/api/static/index.html @@ -0,0 +1,309 @@ + + + + + +Plume Studio + + + + +
+

Plume Studio

+

Configure, run and inspect coupled SAI plume box-model simulations. + Phase 0 slice — a subset of the schema, the whole pipeline.

+
+ +
+
+

Configure

+
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+ + +
+ +
+
+
Plume volume V₀
—
+
Initial SO₂
—
+
+

—

+

Derived by the server, not the form — the same resolver the CLI uses. + Identity is the hash; two runs differing only in label are one computation.

+
+

+
+ +
+

Runs

+

No runs yet.

+
+
+
+ + + + diff --git a/studio/cli/main.py b/studio/cli/main.py index 796addc..8673f31 100644 --- a/studio/cli/main.py +++ b/studio/cli/main.py @@ -31,22 +31,15 @@ import typer import yaml -from studio.modelio.provenance import record_for from studio.resolve import ResolvedConfig, resolve from studio.schema import RunConfig, RunSet +from studio.service import prepare, submit_and_record from studio.store import ( - LocalDirectoryStore, create_db_engine, - create_run, create_run_set, database_url, - record_artifact, - record_job, - record_summary, - record_transition, session_factory, session_scope, - upgrade_to_head, ) app = typer.Typer( @@ -55,18 +48,6 @@ help="Configure, run and inspect coupled SAI plume box-model simulations.", ) -#: Artefacts recorded for every run, by kind -> filename in the job's work directory. The set is -#: fixed rather than "whatever the directory contains", so a missing one is an error rather than a -#: silently shorter list. -_ARTIFACTS = { - "input": "input.json", - "provenance": "provenance.json", - "state": "state.npz", - "summary": "summary.json", - "stdout": "stdout.log", - "stderr": "stderr.log", -} - def _load_mapping(path: Path) -> dict[str, Any]: """Read a YAML or JSON document, or exit with a message naming the file. @@ -108,96 +89,6 @@ def _run_set(path: Path) -> RunSet: raise typer.Exit(code=2) from exc -def _prepare(database: str | None, out: Path) -> tuple[Any, LocalDirectoryStore]: - """Migrate the database and open the artefact store. Idempotent: startup can always call it.""" - url = database_url(database) - upgrade_to_head(url) - return session_factory(create_db_engine(url)), LocalDirectoryStore(out / "artifacts") - - -def _require(row: Any, what: str, key: str) -> Any: - """Return ``row``, or raise saying what went missing. - - ``Session.get`` returns ``None`` for a row that is not there, and passing that on would fail - several frames later with an ``AttributeError`` about ``None``. Here it means the database - changed underneath a run that is mid-flight -- rare, and worth naming precisely when it happens. - """ - if row is None: - raise RuntimeError( - f"{what} {key!r} vanished from the database while its run was in progress; " - f"the run may have been deleted concurrently" - ) - return row - - -def _submit_and_record( - factory: Any, - store: LocalDirectoryStore, - runner: Any, - *, - config: ResolvedConfig, - label: str, - run_set_row: Any, - wait: bool, -) -> tuple[str, str]: - """Persist a run, submit it, and record what came back. Returns ``(run_id, job_state)``.""" - provenance = record_for(config) - with session_scope(factory) as session: - run = create_run( - session, - run_set=run_set_row, - config=config, - label=label, - provenance=provenance.model_dump(mode="json"), - ) - run_id = run.id - - record = runner.submit(config, label=label) - with session_scope(factory) as session: - from studio.store.models import RunRow - - run_row = _require(session.get(RunRow, run_id), "run", run_id) - job = record_job( - session, - run=run_row, - state=record.state.value, - work_dir=record.work_dir, - detail=record.detail, - ) - job_id = job.id - - if not wait: - return run_id, record.state.value - - final = runner.wait(record.job_id) - with session_scope(factory) as session: - from studio.store.models import JobRow, RunRow - - job_row = _require(session.get(JobRow, job_id), "job", job_id) - # EVERY transition the runner saw, not just the final state. A thinner trail than the - # runner's defeats the point of persisting one: "queued -> succeeded" hides how long it - # waited for a worker, and "queued -> failed" hides whether it ever started. - for transition in final.transitions[1:]: - record_transition( - session, - job=job_row, - state=transition.state.value, - detail=transition.detail, - exit_code=final.exit_code if transition.state is final.state else None, - ) - run_row = _require(session.get(RunRow, run_id), "run", run_id) - for kind, filename in _ARTIFACTS.items(): - source = Path(final.work_dir or ".") / filename - if source.is_file(): - record_artifact(session, run=run_row, kind=kind, source=source, store=store) - summary_path = Path(final.work_dir or ".") / "summary.json" - if summary_path.is_file(): - record_summary( - session, run=run_row, summary=json.loads(summary_path.read_text(encoding="utf-8")) - ) - return run_id, final.state.value - - @app.command() def run( config: Path = typer.Argument(..., help="RunConfig as YAML or JSON"), @@ -221,13 +112,13 @@ def run( from studio.runner import LocalSubprocessRunner - factory, store = _prepare(database, out) + factory, store = prepare(database, out) runner = LocalSubprocessRunner(out / "jobs") try: with session_scope(factory) as session: run_set_row = create_run_set(session, label=label or config.stem) - run_id, state = _submit_and_record( - factory, store, runner, config=resolved, label=label, run_set_row=run_set_row, wait=wait + run_id, state = submit_and_record( + factory, store, runner, config=resolved, run_set=run_set_row, label=label, wait=wait ) finally: runner.shutdown() @@ -257,7 +148,7 @@ def sweep( from studio.runner import LocalSubprocessRunner - factory, store = _prepare(database, out) + factory, store = prepare(database, out) runner = LocalSubprocessRunner(out / "jobs") failures = 0 try: @@ -268,14 +159,13 @@ def sweep( axes=run_set.model_dump(mode="json")["axes"], ) for item in expanded: - run_id, state = _submit_and_record( + run_id, state = submit_and_record( factory, store, runner, config=resolve(item.config), + run_set=run_set_row, label=item.label, - run_set_row=run_set_row, - wait=True, ) typer.echo(f"[studio] {item.label or run_id} -> {state}") failures += state != "succeeded" diff --git a/studio/requirements.lock b/studio/requirements.lock index 0ca74f1..fcc7d44 100644 --- a/studio/requirements.lock +++ b/studio/requirements.lock @@ -22,6 +22,7 @@ anyio==4.14.2 \ --hash=sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494 \ --hash=sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f # via + # httpx2 # sse-starlette # starlette # watchfiles @@ -415,7 +416,13 @@ greenlet==3.5.5 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or h11==0.16.0 \ --hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \ --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86 - # via uvicorn + # via + # httpcore2 + # uvicorn +httpcore2==2.10.0 ; sys_platform != 'emscripten' \ + --hash=sha256:13c0cc3d1919d4f28457f60cd2c2abe04113a8af184ccf1142811beba936f9dc \ + --hash=sha256:7df06cfb34070cae4f7c89be69dc1095eca138e9704ceffb98d25c1912ab6f01 + # via httpx2 httptools==0.8.0 \ --hash=sha256:0770728beb05094c809b98e814edff5fef69d26ad7d21185f2f6d5884a0ba683 \ --hash=sha256:0ea897f0c729581ebf72131a438a7932d9b14efef72d75ada966700cac3caaeb \ @@ -468,6 +475,14 @@ httptools==0.8.0 \ --hash=sha256:f256d6ce930c52ca1cb2a960b7da03548c454e7d28b06059ad41bfe789036ce0 \ --hash=sha256:fe2a4c95aeba2209434e7b31172da572846cae8ca0bf1e7013e61b99fbbf5e72 # via uvicorn +httpx2==2.10.0 \ + --hash=sha256:5e3194a432701e1cc6f69a8b1b2fa199ef907013fede8d9a09a2c5b7b8141a18 \ + --hash=sha256:8741d7329fe2c7885fc9ceb61c8217acfb87a85f75723714b89ebf7ad7196338 + # via sandbox-coupled (pyproject.toml) +httpx2-jsfetch==1.0 ; sys_platform == 'emscripten' \ + --hash=sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60 \ + --hash=sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32 + # via httpx2 hypothesis==6.165.5 \ --hash=sha256:05e7e8288b2f5fbb34a30b45c9df72a5ce9da0d5ce90705c76b28dda75aac984 \ --hash=sha256:0a5004c3fe761b642ca556abf4551bc9a94d190320bcf7ce118cf8b792eaf71c \ @@ -535,7 +550,9 @@ hypothesis==6.165.5 \ idna==3.18 \ --hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \ --hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848 - # via anyio + # via + # anyio + # httpx2 iniconfig==2.3.0 \ --hash=sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730 \ --hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12 @@ -1753,6 +1770,12 @@ starlette==1.6.0 \ # via # fastapi # sse-starlette +truststore==0.10.4 ; sys_platform != 'emscripten' \ + --hash=sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301 \ + --hash=sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981 + # via + # httpcore2 + # httpx2 typer==0.27.1 \ --hash=sha256:53150287edd11baeb4e4722c8e394fcdf8181c0ae89485cba8d25c778d5edd56 \ --hash=sha256:a79bef8469a79c45498e7b814ecf8d603cc7644e9acbd9e19cac0334240b18df @@ -1768,6 +1791,7 @@ typing-extensions==4.16.0 \ # fastapi # flexcache # flexparser + # httpx2 # lineax # mypy # optimistix diff --git a/studio/service.py b/studio/service.py new file mode 100644 index 0000000..58da77b --- /dev/null +++ b/studio/service.py @@ -0,0 +1,219 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Submitting a run: the one flow the CLI and the API both use. + +Extracted the moment there were two callers. A second copy of "resolve, record provenance, persist, +submit, record every transition, store the artefacts" would drift within a week, and the drift would +be invisible -- both paths would keep working, and only their database rows would disagree. ADR-002 +requires that a sweep launched from a terminal and one launched from the web produce identical rows +and identical provenance; that is a property of there being *one function*, not of two being written +carefully. + +The two callers differ in what they do *around* this -- the CLI blocks and prints, the API returns a +handle and streams progress -- and neither difference reaches in here. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any + +from sqlalchemy.orm import sessionmaker + +from studio.modelio.provenance import record_for +from studio.resolve import ResolvedConfig +from studio.runner import LocalSubprocessRunner +from studio.store import ( + LocalDirectoryStore, + create_db_engine, + create_run, + database_url, + record_artifact, + record_job, + record_summary, + record_transition, + session_factory, + session_scope, + upgrade_to_head, +) +from studio.store.models import JobRow, RunRow, RunSetRow + +#: How often a running job's state is copied into the database. Short against a 4-minute run, long +#: against a single SELECT. +_FOLLOW_INTERVAL_S = 0.2 + +#: Artefacts recorded for every run, by kind -> filename in the job's work directory. A fixed set +#: rather than "whatever the directory contains", so a missing one is a visible gap rather than a +#: silently shorter list. +ARTIFACTS: dict[str, str] = { + "input": "input.json", + "provenance": "provenance.json", + "state": "state.npz", + "summary": "summary.json", + "stdout": "stdout.log", + "stderr": "stderr.log", +} + + +def prepare(database: str | None, out: Path) -> tuple[sessionmaker[Any], LocalDirectoryStore]: + """Migrate the database and open the artefact store. + + Idempotent, so both entry points can call it unconditionally at startup: a fresh clone must not + need a manual ``alembic upgrade`` before the first run. + """ + url = database_url(database) + upgrade_to_head(url) + return session_factory(create_db_engine(url)), LocalDirectoryStore(Path(out) / "artifacts") + + +def _require(row: Any, what: str, key: str) -> Any: + """Return ``row``, or raise saying what went missing. + + ``Session.get`` returns ``None`` for a row that is not there, and passing that on would fail + several frames later as an ``AttributeError`` about ``None``. Here it means the database changed + underneath a run that is mid-flight -- rare, and worth naming precisely when it happens. + """ + if row is None: + raise RuntimeError( + f"{what} {key!r} vanished from the database while its run was in progress; " + f"the run may have been deleted concurrently" + ) + return row + + +def create_pending_run( + factory: sessionmaker[Any], + *, + run_set: RunSetRow, + config: ResolvedConfig, + label: str = "", +) -> str: + """Record the run and its provenance **before** anything is submitted. + + Provenance first, deliberately (ADR-006): if the model cannot be pinned, the run must not start, + and a run that dies in its first second still says exactly what produced it. + """ + provenance = record_for(config) + with session_scope(factory) as session: + run = create_run( + session, + run_set=run_set, + config=config, + label=label, + provenance=provenance.model_dump(mode="json"), + ) + return str(run.id) + + +def submit( + factory: sessionmaker[Any], + runner: LocalSubprocessRunner, + *, + run_id: str, + config: ResolvedConfig, + label: str = "", +) -> tuple[str, str]: + """Submit a recorded run. Returns ``(job_id, runner_job_id)``. + + Returns both because they are different things: the database's job id is durable and survives a + restart, while the runner's is a handle into this process's pool. + """ + record = runner.submit(config, label=label) + with session_scope(factory) as session: + run_row = _require(session.get(RunRow, run_id), "run", run_id) + job = record_job( + session, + run=run_row, + state=record.state.value, + work_dir=record.work_dir, + detail=record.detail, + ) + return str(job.id), record.job_id + + +def finalise( + factory: sessionmaker[Any], + store: LocalDirectoryStore, + runner: LocalSubprocessRunner, + *, + run_id: str, + job_id: str, + runner_job_id: str, +) -> str: + """Follow a job to completion, persisting each transition **as it happens**, then store its + artefacts. Returns the final state. + + Polling the runner rather than only waiting, because the database is what the API streams from + (a stream reading the worker pool would be blind to runs submitted by the CLI or by a previous + process). Writing only at the end would leave a job showing ``queued`` for its whole four + minutes and then jumping to ``succeeded`` -- technically a trail, useless as progress. + + ``_FOLLOW_INTERVAL_S`` is short relative to a run and long relative to a SELECT; SQLite has one + writer, and each write here is a single short transaction. + """ + seen = 1 # the queued transition is already persisted by submit() + while True: + record = runner.poll(runner_job_id) + new_transitions = record.transitions[seen:] + if new_transitions: + with session_scope(factory) as session: + job_row = _require(session.get(JobRow, job_id), "job", job_id) + for transition in new_transitions: + record_transition( + session, + job=job_row, + state=transition.state.value, + detail=transition.detail, + exit_code=record.exit_code if transition.state is record.state else None, + ) + seen = len(record.transitions) + if record.is_terminal: + break + time.sleep(_FOLLOW_INTERVAL_S) + + final = runner.wait(runner_job_id) + with session_scope(factory) as session: + run_row = _require(session.get(RunRow, run_id), "run", run_id) + work_dir = Path(final.work_dir or ".") + for kind, filename in ARTIFACTS.items(): + source = work_dir / filename + if source.is_file(): + record_artifact(session, run=run_row, kind=kind, source=source, store=store) + summary_path = work_dir / "summary.json" + if summary_path.is_file(): + record_summary( + session, run=run_row, summary=json.loads(summary_path.read_text(encoding="utf-8")) + ) + return str(final.state.value) + + +def submit_and_record( + factory: sessionmaker[Any], + store: LocalDirectoryStore, + runner: LocalSubprocessRunner, + *, + config: ResolvedConfig, + run_set: RunSetRow, + label: str = "", + wait: bool = True, +) -> tuple[str, str]: + """The whole flow, for a caller that just wants a run to happen. Returns ``(run_id, state)``.""" + run_id = create_pending_run(factory, run_set=run_set, config=config, label=label) + job_id, runner_job_id = submit(factory, runner, run_id=run_id, config=config, label=label) + if not wait: + return run_id, "queued" + return run_id, finalise( + factory, store, runner, run_id=run_id, job_id=job_id, runner_job_id=runner_job_id + ) + + +__all__ = [ + "ARTIFACTS", + "create_pending_run", + "finalise", + "prepare", + "submit", + "submit_and_record", +] diff --git a/studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py b/studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py new file mode 100644 index 0000000..fff0e77 --- /dev/null +++ b/studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py @@ -0,0 +1,36 @@ +"""reproducible is a boolean + +Revision ID: c0236fb572d2 +Revises: 1427673e9a31 +Created: 2026-08-14 17:16:21.643335 +""" + +from __future__ import annotations + +import sqlalchemy as sa +from alembic import op + +revision = "c0236fb572d2" +down_revision = "1427673e9a31" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.alter_column( + "reproducible", existing_type=sa.INTEGER(), type_=sa.Boolean(), existing_nullable=True + ) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.alter_column( + "reproducible", existing_type=sa.Boolean(), type_=sa.INTEGER(), existing_nullable=True + ) + + # ### end Alembic commands ### diff --git a/studio/store/models.py b/studio/store/models.py index e900195..c92495f 100644 --- a/studio/store/models.py +++ b/studio/store/models.py @@ -33,6 +33,7 @@ from typing import Any from sqlalchemy import ( + Boolean, DateTime, Float, ForeignKey, @@ -146,7 +147,7 @@ class RunRow(Base): #: The provenance record (ADR-006) as written at submit time, stored verbatim. provenance: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True) #: False when any checkout was dirty: the SHAs do not describe the code that ran. - reproducible: Mapped[bool | None] = mapped_column(Integer, nullable=True) + reproducible: Mapped[bool | None] = mapped_column(Boolean, nullable=True) created_at: Mapped[datetime] = mapped_column(UtcDateTime) run_set: Mapped[RunSetRow] = relationship(back_populates="runs") diff --git a/studio/tests/unit/test_api.py b/studio/tests/unit/test_api.py new file mode 100644 index 0000000..08d4eff --- /dev/null +++ b/studio/tests/unit/test_api.py @@ -0,0 +1,252 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The API, and the page it serves. + +Most of this needs neither the model nor a run: the endpoints that matter for the form -- schema, +resolve, list, fetch -- are cheap by design, because the UI calls ``/api/config/resolve`` on every +keystroke and a JAX import on that path would be unaffordable. + +**The SSE test runs a real uvicorn server in a thread.** ``TestClient`` serialises requests, so a +stream opened against it does not observe a state change made concurrently -- when I first checked +progress under ``TestClient`` it reported only the terminal state, which looked exactly like a +broken stream and was not one. A claim as central as "progress is pushed" (spec 7.3) has to be +tested against something that can actually push. +""" + +from __future__ import annotations + +import json +import socket +import threading +import time +from pathlib import Path +from typing import Any + +import pytest +from fastapi.testclient import TestClient + +from studio.api.app import create_app +from studio.resolve import resolve +from studio.schema import RunConfig +from studio.store import create_run, create_run_set, record_job, record_transition, session_scope + + +@pytest.fixture +def api(tmp_path: Path) -> Any: + """An app with its own home and database, built by the factory rather than monkeypatched.""" + app = create_app(home=tmp_path / "home", database=f"sqlite:///{tmp_path / 'studio.db'}") + with TestClient(app) as client: + yield client, app + + +@pytest.mark.tier_a +def test_the_page_is_served(api: Any) -> None: + client, _ = api + response = client.get("/") + assert response.status_code == 200 + assert "Plume Studio" in response.text + assert "/api/config/resolve" in response.text, "the page must talk to the real resolver" + + +@pytest.mark.tier_a +def test_the_schema_endpoint_is_the_schema(api: Any) -> None: + """Nothing in the UI may invent a field that does not exist here (ADR-002).""" + from studio.schema import SCHEMA_VERSION + + client, _ = api + schema = client.get("/api/schema").json() + assert schema["x-studio-schema-version"] == SCHEMA_VERSION + assert "Site" in schema["$defs"] + assert schema["$defs"]["Site"]["properties"]["temperature_k"]["x-studio"]["unit"] == "K" + + +@pytest.mark.tier_a +def test_resolve_returns_the_derived_values_and_the_identity(api: Any) -> None: + """The form shows what the server derived, never what the browser guessed.""" + client, _ = api + response = client.post( + "/api/config/resolve", json={"config": {"schedule": {"duration_days": 1}}} + ) + assert response.status_code == 200 + body = response.json() + expected = resolve(RunConfig.model_validate({"schedule": {"duration_days": 1}})) + assert body["config_hash"] == expected.config.config_hash() + assert body["derived"]["plume_volume_cm3"] == 1.5e12 + assert body["derived"]["so2_initial_pptv"] == pytest.approx(3.309115922996412e9, rel=1e-15) + assert body["stale_fields"] == [] + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + "config", + [ + {"site": {"temperature_k": -5}}, + {"microphysics": {"n_bins": 100}}, + {"switches": {"heating_to_t": True}}, + {"site": {"temprature_k": 210}}, + ], +) +def test_an_invalid_config_is_422_not_a_500(api: Any, config: dict[str, Any]) -> None: + """Validation failures are the schema working, so they are reported as client errors. + + The heating case is the interesting one: it is refused because the model cannot represent the + physics (SCIENCE-4), and that refusal has to reach the browser rather than crashing the server. + """ + client, _ = api + assert client.post("/api/config/resolve", json={"config": config}).status_code == 422 + assert client.post("/api/runs", json={"config": config}).status_code == 422 + + +@pytest.mark.tier_a +def test_unknown_things_are_404(api: Any) -> None: + client, _ = api + assert client.get("/api/runs/nope").status_code == 404 + assert client.get("/api/runs/nope/summary").status_code == 404 + assert client.get("/api/runs/nope/artifacts/state").status_code == 404 + + +@pytest.mark.tier_a +def test_runs_start_empty_and_list_what_exists(api: Any, tmp_path: Path) -> None: + client, app = api + assert client.get("/api/runs").json() == [] + + with session_scope(app.state.factory) as session: + run_set = create_run_set(session, label="direct") + run = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + label="written directly", + provenance={"sandbox": {"dirty": False}, "submodules": {}}, + ) + record_job(session, run=run, state="queued") + + listed = client.get("/api/runs").json() + assert len(listed) == 1 + assert listed[0]["label"] == "written directly" + assert listed[0]["state"] == "queued" + assert listed[0]["reproducible"] is True, "a JSON boolean, not 0/1" + + +@pytest.mark.tier_a +def test_the_run_detail_carries_provenance_and_transitions(api: Any) -> None: + """What the page needs to say honestly what produced a result (ADR-006).""" + client, app = api + with session_scope(app.state.factory) as session: + run_set = create_run_set(session) + run = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + provenance={"sandbox": {"commit": "a" * 40, "dirty": True}, "submodules": {}}, + ) + job = record_job(session, run=run, state="queued") + record_transition(session, job=job, state="running", detail="launched") + run_id = run.id + + detail = client.get(f"/api/runs/{run_id}").json() + assert detail["reproducible"] is False + assert detail["provenance"]["sandbox"]["commit"] == "a" * 40 + assert [t["state"] for t in detail["transitions"]] == ["queued", "running"] + + +def _free_port() -> int: + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return int(sock.getsockname()[1]) + + +@pytest.mark.tier_a +def test_progress_is_pushed_as_it_happens(tmp_path: Path) -> None: + """A real server, a real SSE client, and a state change made while the stream is open. + + This is the test that would have caught a stream reporting only the terminal state. It uses no + model: the stream reads job state from the DATABASE, so writing a transition from the test is + exactly what a running job does -- and is also why the stream works for runs submitted by the + CLI or by a previous process. + """ + import httpx2 as httpx + import uvicorn + + database = f"sqlite:///{tmp_path / 'studio.db'}" + app = create_app(home=tmp_path / "home", database=database) + with session_scope(app.state.factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig()), label="streamed") + job = record_job(session, run=run, state="queued") + run_id, job_id = run.id, job.id + + port = _free_port() + server = uvicorn.Server( + uvicorn.Config(app, host="127.0.0.1", port=port, log_level="error", lifespan="off") + ) + thread = threading.Thread(target=server.run, daemon=True) + thread.start() + deadline = time.monotonic() + 15 + while not server.started and time.monotonic() < deadline: + time.sleep(0.05) + assert server.started, "uvicorn did not start" + + def advance() -> None: + """Move the job on while the stream is open, as a running job would.""" + from studio.store.models import JobRow + + for state in ("running", "succeeded"): + time.sleep(0.6) + with session_scope(app.state.factory) as session: + record_transition(session, job=session.get(JobRow, job_id), state=state) + + try: + threading.Thread(target=advance, daemon=True).start() + seen: list[str] = [] + with httpx.Client(timeout=20.0) as client: + with client.stream( + "GET", f"http://127.0.0.1:{port}/api/events/runs/{run_id}" + ) as stream: + for line in stream.iter_lines(): + if line.startswith("data:"): + seen.append(json.loads(line[5:])["state"]) + if seen and seen[-1] == "succeeded": + break + finally: + server.should_exit = True + thread.join(timeout=10) + + assert seen[0] == "queued", "the stream must report the state it finds, not only changes" + assert "running" in seen, "an intermediate state must arrive while the run is in flight" + assert seen[-1] == "succeeded" + + +@pytest.mark.tier_a +def test_the_stream_reports_an_unknown_run_rather_than_hanging(api: Any) -> None: + """A client asking about a run that does not exist gets an error event and a closed stream.""" + client, _ = api + with client.stream("GET", "/api/events/runs/nope") as stream: + events = [line for line in stream.iter_lines() if line.startswith(("event:", "data:"))] + assert any("error" in line for line in events) + + +@pytest.mark.tier_a +def test_a_submitted_run_appears_immediately_with_202(api: Any, repo_root: Path) -> None: + """Submission returns a handle, not a finished run: 202 means accepted, not complete. + + Needs the model, because provenance pins the submodules before anything starts (ADR-006). + """ + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + + client, _ = api + response = client.post( + "/api/runs", + json={ + "config": {"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}, + "label": "accepted", + }, + ) + assert response.status_code == 202 + body = response.json() + assert body["state"] == "queued" + assert client.get(f"/api/runs/{body['run_id']}").json()["label"] == "accepted" + assert ( + client.get(f"/api/runs/{body['run_id']}").json()["config_hash"] == body["config_hash"] + ), "the identity in the response is the one that was stored" From 5f55a53db1f8fa3a7ab596ee397ff3d3e181906f Mon Sep 17 00:00:00 2001 From: Ali Akherati Date: Mon, 17 Aug 2026 14:38:05 -0700 Subject: [PATCH 18/18] docs(studio): Phase 0 complete Nine tasks, 269 Tier-A tests. Every exit criterion met, with the narrow ones named rather than rounded up: no React (one self-contained page instead), CI cannot see the submodules so three model-facing checks run only locally, and PROGRESS.md conflicts on almost every parallel PR. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --- docs/studio/PROGRESS.md | 54 +++++++++++++++++++++++++++++++++++-- docs/studio/plan/PHASE_0.md | 2 +- 2 files changed, 53 insertions(+), 3 deletions(-) diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index 029205f..48828e7 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -7,10 +7,11 @@ Phase plan: [`plan/PHASE_0.md`](plan/PHASE_0.md). Decisions: [`adr/`](adr/). Ope --- -## Phase 0 — Skeleton and vertical slice · **in progress** +## Phase 0 — Skeleton and vertical slice · **COMPLETE** (2026-08-17) Exit criteria: a run can be submitted from the CLI **and** from the web UI, produces a stored result -with full provenance, and the golden tests pass. +with full provenance, and the golden tests pass. **All three met** — see the phase-completion entry +below for what was verified and what was deliberately left undone. | Task | Status | |---|---| @@ -29,6 +30,55 @@ derivations to resolve rather than fixtures. --- +### 2026-08-17 — Phase 0 complete: `studio/dev` merged to `main` + +Nine tasks, 269 Tier-A tests, 43 typed source files. Every exit criterion met, and the ones that were +met *narrowly* are named below rather than rounded up. + +**What works end to end.** A configuration goes from a YAML file or a browser form through the +schema, the resolver, the model seam and the runner into a real coupled run, and comes back as a +`state.npz`, a versioned `RunSummary`, six recorded artefacts and an immutable provenance record +naming the SANDBOX commit and all three submodule SHAs. Both front ends share one submit function, +so their rows cannot diverge. + +**The merge into `main` had exactly the conflict predicted in the #73 review**: the two `studio/` +files that PR edited on `main`, which `studio/dev` had since moved past. Resolved as recorded then — +`studio/dev`'s structure (three clean packages, the seam-submodule check) with `main`'s corrected +past-tense wording (the `__post_init__` pattern it describes was fixed by #73). Both sides asserted +present rather than eyeballed. + +**Verified on the merged result, not before it**: 269 Studio Tier-A tests, the model's own 132 tests, +`ruff`, `black`, `mypy --strict`. + +**What is deliberately not done, and why** + +- **No React + Vite** (ADR-007 named it). One self-contained HTML page instead: a build toolchain for + a single form is machinery ahead of need. `/api/schema` exists so the form can be *generated* when + the UI outgrows one form. +- **CI cannot see the submodules**, so three checks skip there and run only locally: Tier A's real + 1-day run, the 0.4 equivalence test, and the `air_number_density` mirror check. A deploy key would + fix it; until then CI verifies the pure layers and a developer machine verifies the model-facing + ones. This is the weakest point in the setup and is worth saying plainly. +- **`PROGRESS.md` conflicts on almost every parallel PR** — six times in this phase, once losing a + commit to a squash. One file per entry would end it. +- **BLOCKING-2** (tenancy) and **SCIENCE-1, -2, -3, -5, -6** remain open; SCIENCE-4 is answered for + heating and buoyancy and open for sedimentation. + +**The five findings from this phase that changed the code rather than the docs** + +1. The plan's claim that the repository held two different dN/dlogDp conventions was wrong — they are + the same expression, 7e-16 apart. +2. Bit-for-bit reproduction of the archive is **false** (~31 % of gas elements differ), so ADR-009's + "measure before asserting" was load-bearing rather than cautious. +3. `dp_mid_um` is not bit-identical for a *Studio*-produced run, because 0.5 deliberately changed the + spelling — a tolerance measured through one pipeline is not a tolerance for another. +4. The first Tier-B run failed on a tolerance the harness had misread (endpoint vs series), not on a + regression. +5. `DateTime(timezone=True)` returns naive datetimes on SQLite and aware ones on Postgres; the same + comparison would have been right in production and wrong in development. + +--- + ### 2026-08-15 — Tasks 0.9d and 0.9e: the API, the page, and the shared service **Phase 0's exit criteria are met.** A run can be submitted from the CLI *and* from the web UI, diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index a6b8b01..3d7deb9 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -1,4 +1,4 @@ -# Phase 0 — Skeleton and vertical slice +# Phase 0 — Skeleton and vertical slice · **COMPLETE** (2026-08-17) **Goal:** one hardcoded end-to-end path, real from top to bottom, however narrow.