diff --git a/.github/workflows/studio-ci.yml b/.github/workflows/studio-ci.yml index 41eb718..7c92197 100644 --- a/.github/workflows/studio-ci.yml +++ b/.github/workflows/studio-ci.yml @@ -13,7 +13,11 @@ name: studio-ci on: push: - branches: ["main"] + # `studio/dev` is the integration branch every Studio task PRs into; main sees Studio in + # reviewed batches. Running on pushes to both means a merge into either is verified, not just + # the PR that preceded it -- a PR is tested against its own head, and a stale one can go green + # and still break the branch it lands on. + branches: ["main", "studio/dev"] paths: - "studio/**" - "docs/studio/**" @@ -79,10 +83,10 @@ jobs: - name: black run: .venv/bin/black --check studio/ - # --strict on the two packages that must stay pure; the pyproject baseline covers the rest. + # --strict on the three packages that must stay pure; the pyproject baseline covers the rest. - name: mypy run: | - .venv/bin/mypy --strict studio/schema studio/science + .venv/bin/mypy --strict studio/schema studio/science studio/resolve .venv/bin/mypy # Tier B (full-case golden reproduction against the archived ensemble) is NOT run here: the diff --git a/.gitignore b/.gitignore index 048d368..c3f9add 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,6 @@ gas_phase_chemistry/ # -- SQLite database and run artefacts, both regenerable and both machine-specific .env var/ + +# Subagent worktrees: separate checkouts, never part of this repo +.claude/worktrees/ diff --git a/docs/studio/ASSUMPTIONS.md b/docs/studio/ASSUMPTIONS.md index d7f6540..31a0229 100644 --- a/docs/studio/ASSUMPTIONS.md +++ b/docs/studio/ASSUMPTIONS.md @@ -36,9 +36,17 @@ plus a conversion layer becomes the better trade. ## ASSUMPTION-2 — The archived `state.npz` files are the golden reference, at a tolerance yet to be measured -**Made:** 2026-08-13 · **Affects:** `studio/tests/golden/` · +**Made:** 2026-08-13 · **Settled:** 2026-08-13 (#70) · **Affects:** `studio/tests/golden/` · **Recorded in:** [ADR-009](adr/ADR-009-golden-file-strategy.md) +> **Settled.** The measurement exists: +> [`studio/tests/golden/REFERENCE_TOLERANCES.md`](../../studio/tests/golden/REFERENCE_TOLERANCES.md). +> Reproduction is **close but not bit-for-bit** — every headline quantity within 2.1e-12, worst +> deviation anywhere 3.4e-12, but only ~31 % of gas state-vector elements bit-identical. The archive +> is usable as a golden reference at ~1e-12 (endpoints) / ~1e-10 (series and per-bin size +> distribution); exact equality is not. The paragraphs below stand as the reasoning that made the +> measurement necessary; the *consequence* below still holds and cannot be retrofitted. + Golden fixtures are derived from the existing `coupled/paper_ensemble/runs*/` outputs. Whether re-running those cases **today** reproduces them bit-for-bit is *unverified*: the submodule commits at which they were produced were never recorded (there is no provenance record for the existing @@ -52,7 +60,10 @@ the measurement together with the SHAs it was taken at. **Consequence.** Golden fixtures record the SHA at which the reference was *measured*, not the SHA at which the data was originally produced. This is an honest limitation and cannot be retrofitted. -**What would settle it.** The measurement itself, in `studio/tests/golden/REFERENCE_TOLERANCES.md`. +**What settled it.** The measurement itself, in `studio/tests/golden/REFERENCE_TOLERANCES.md` +(2026-08-13, issue #70). It also produced a result nobody had asked for: the model is bit-for-bit +deterministic run-to-run *today*, so the residual is drift between the archive's toolchain and this +one — which is what makes a 1e-12 tolerance defensible rather than arbitrary. --- diff --git a/docs/studio/CAVEATS.md b/docs/studio/CAVEATS.md index b933ee6..2b9a28d 100644 --- a/docs/studio/CAVEATS.md +++ b/docs/studio/CAVEATS.md @@ -8,6 +8,26 @@ ones that arise from *configuring and comparing* runs. --- +## The box does not heat and does not rise — and this model cannot answer whether it should + +**Every run is isobaric and isothermal at the configured temperature.** There is no radiative +heating response and no buoyant rise. A result must not be read as containing a plume-warming +signal, a lofting signal, or an altitude change. + +This is a **scope boundary, not a pending feature** (Ali, 2026-08-13). The model's heating term is +shortwave-only — longwave cooling is absent from the radiative calculation (AD-5.4) — so enabling it +would not make the thermodynamics more complete, it would make them one-sided, producing a +~+1.2 K / 10 d drift that is an artefact of the missing cooling. Buoyant rise follows from a heating +rate this model cannot compute, so a rise velocity would be a free parameter dressed as physics. + +**Answering either question requires a different model**, with longwave radiation and plume +dynamics. Studio therefore refuses `switches.heating_to_t` outright (schema 0.2.0, `True` fails +validation) and exposes **no** buoyancy or heating-rate fields at all — a knob for a capability the +model does not have would advertise it. + +Sedimentation is a separate question and remains genuinely open (SCIENCE-4, issue #56); it is a +particle-loss process, not a thermodynamic response, and this decision says nothing about it. + ## Top-level caveats — shown on every results view ### The definition of t = 0 is unresolved, and it dominates particle number diff --git a/docs/studio/GLOSSARY.md b/docs/studio/GLOSSARY.md index 1e1311a..d00dcd4 100644 --- a/docs/studio/GLOSSARY.md +++ b/docs/studio/GLOSSARY.md @@ -15,9 +15,17 @@ uses, plus the repository-specific names that are otherwise unguessable. **RunSet** — the primary user-facing object: a base `RunConfig` plus zero or more **axes**. A single run is a RunSet with zero axes, so there is no separate code path for N = 1. -**Axis** — a schema path marked as varying, either `{path, values: [...]}` or -`{path, range: {start, stop, n, spacing}}`. Expanded by **GRID** (Cartesian product), **ZIP** -(paired), or **LIST** (explicit configs). +**Axis** — one dimension of a sweep: a name, a kind, and **points**. Each point is a short `label` +plus the field `assignments` it stands for. Kinds: **GRID** (crossed with the other GRID/LIST axes), +**ZIP** (advanced in lockstep with the other ZIP axes, the group then crossed with the rest), and +**LIST** (crossed, but each point sets *several* fields at once — a covarying group, e.g. the paper +ensemble's site axis, where latitude, T, p and H₂O move together). Expansion order is +`itertools.product`: the last axis varies fastest, which is what reproduces the existing ensemble's +case order. Assignments name **leaf** paths only; a whole group has no unit, provenance or DAG node. + +**Axis point label** — the short token that becomes part of the run label, e.g. `sabr220`, `a1p0`. +Joined by `__` across axes to give the **case ID**, which is how the existing ensemble names its +directories. **Derived field** — a value computed from other fields, declared via `derived_from` metadata. Each carries a state: **auto** (recomputed silently when an upstream field changes) or **user_override** diff --git a/docs/studio/OPEN_QUESTIONS.md b/docs/studio/OPEN_QUESTIONS.md index 2a79b72..840c270 100644 --- a/docs/studio/OPEN_QUESTIONS.md +++ b/docs/studio/OPEN_QUESTIONS.md @@ -169,7 +169,7 @@ material is **not** represented. --- -### SCIENCE-4 — Box thermodynamics · **OPEN** · blocks Phase 5 · [#56](https://github.com/reflective-org/SANDBOX/issues/56) +### SCIENCE-4 — Box thermodynamics · **ANSWERED for heating and buoyancy** (2026-08-13); sedimentation open · [#56](https://github.com/reflective-org/SANDBOX/issues/56) *Is the box isobaric? isothermal? does it rise buoyantly? do particles sediment out?* Absent from the original brief. Current behaviour, from the code: @@ -179,8 +179,30 @@ Absent from the original brief. Current behaviour, from the code: AD-5.4), producing a one-sided ≈ +1.2 K / 10 d warm drift. Every science script leaves it off. - **No buoyant rise.** No sedimentation. -Each must become a schema field with a documented default, and the SW-only asymmetry must warn in -the UI when the switch is enabled rather than silently producing a drifting temperature. +**Answered (Ali, 2026-08-13): heating and buoyancy are out of scope for this model.** + +Not "undecided" — **out of scope**, which is a different status and is why this row is closed rather +than left open. Longwave radiation is not in the radiative calculation, so the heating term cannot +represent the box's energy balance: enabling it does not make the thermodynamics more complete, it +makes them one-sided, and the ~+1.2 K / 10 d drift is an artefact of the missing cooling rather than +a result. Buoyant rise follows the same logic — a parcel rises in response to a heating rate this +model cannot compute, so a rise velocity here would be a free parameter dressed as physics. + +**Answering either question needs a different model**, one with longwave radiation and plume +dynamics. It is not a gap to be filled in by a later Studio phase, and Studio must not present a +knob implying otherwise: + +- `switches.heating_to_t` is `Literal[False]` from schema 0.2.0 — `True` fails validation rather + than being defaulted off, so it cannot be enabled by a form, a YAML file or a sweep axis. +- **No buoyancy or heating-rate fields are added to the schema at all.** A field for a capability + the model does not have would advertise it; the absence is the honest interface (ADR-005). +- Every run is therefore **isobaric and isothermal at the configured temperature**, and results + carry that as a top-level caveat rather than a footnote. + +**Still open: sedimentation.** It is untouched by this decision — a particle-loss process, not a +thermodynamic response — and the model does not have it. It is deliberately left in this register +rather than swept in with the rest, because "we decided not to model heating" is not an argument +about gravitational settling. --- @@ -204,6 +226,27 @@ required. Until then `background_evolves` is a schema field whose only accepted --- +### SCIENCE-6 — GCR ion-pair production rate has no derivation · **OPEN** · Phase 0/4 · [#63](https://github.com/reflective-org/SANDBOX/issues/63) +*What is the ion-pair production rate as a function of altitude, latitude and solar-cycle phase?* + +Raised by task 0.5. The two values available in the repository are an **uncited constant** and a +value that **switches off a physical process**: the paper ensemble uses a bare `30.0` cm⁻³ s⁻¹ +(`run_ensemble.py:102`, described in `TABLE_microphysics_parameters.md` as "galactic cosmic rays at +~20 km"), and the model defaults to `0.0`, which disables ion-induced nucleation entirely +(`coupled/coupled_scenario.py:117`). + +It feeds the ion-induced channels of Dunne et al. (2016) nucleation — the most sensitive part of this +system. GCR ionisation varies by roughly a factor of two over the solar cycle and strongly with +latitude and altitude, so one number is wrong nearly everywhere except where it was read off. + +`studio/science/gcr.py` therefore raises `NotImplementedError` rather than interpolating an uncited +number, and exposes `PAPER_ENSEMBLE_ION_PAIR_RATE = 30.0` as a constant with its provenance attached. + +**Answered when** either a citable parameterisation is agreed and implemented with its reference, or +the decision is recorded that the fixed value stands, with its sensitivity quantified. + +--- + ## Register of capabilities the spec assumes but the model does not have Not open questions — settled facts, listed here because the spec's stage descriptions imply @@ -216,7 +259,7 @@ issue at that point rather than sitting in a backlog now. | `chemistry.rate_overrides[]` (general) | Only `so2_ho2_rate` is a knob (`coupled_scenario.py:130`). Arbitrary per-reaction overrides do not exist. | | `chemistry.photolysis.tuvx_settings.{o3_column, albedo, aod}` | Not exposed. Only mode + lat/lon/doy/hour reach TUV-x (`model_bridge.py:46`). | | `numerics.bin_scheme.{d_min, d_max, mass_doubling}` | Fixed by the TOMAS grid; only `tomas_nbins ∈ {40, 80, 160}` is selectable (`tomas_bridge.py:146`). Ratio = `2**(40/nbins)`; the top boundary is pinned. | -| `numerics.box_thermodynamics.*` | See SCIENCE-4. | +| `numerics.box_thermodynamics.*` | **Not exposed, by decision.** Heating and buoyancy are out of scope (SCIENCE-4): the model cannot compute them and a field would imply it can. Isobaric + isothermal is the only behaviour. | | `dilution.entrainment.{entrains_background_gases, entrains_background_aerosol}` | Entrainment is unconditional when `switches.dilution` is on. Separate flags are new code. | | `dilution.background_evolves` | See SCIENCE-5. Only `false` is accepted. | | `background.aerosol` custom lognormal modes | Six named modes + tabulated `redcircles` only — but `_seed_lognormal` (`tomas_bridge.py:110`) already accepts arbitrary `(N, Dg, σg)` tuples, so this is a small, worthwhile early addition. | diff --git a/docs/studio/PROGRESS.md b/docs/studio/PROGRESS.md index d86cad0..48828e7 100644 --- a/docs/studio/PROGRESS.md +++ b/docs/studio/PROGRESS.md @@ -7,24 +7,805 @@ Phase plan: [`plan/PHASE_0.md`](plan/PHASE_0.md). Decisions: [`adr/`](adr/). Ope --- -## Phase 0 — Skeleton and vertical slice · **in progress** +## Phase 0 — Skeleton and vertical slice · **COMPLETE** (2026-08-17) Exit criteria: a run can be submitted from the CLI **and** from the web UI, produces a stored result -with full provenance, and the golden tests pass. +with full provenance, and the golden tests pass. **All three met** — see the phase-completion entry +below for what was verified and what was deliberately left undone. | Task | Status | |---|---| | 0.1 Repo, CI, docs skeleton | **done** | -| 0.2 `studio/schema` v0 — **review gate** | not started | -| 0.3 Dependency-graph engine + override semantics | not started | -| 0.4 `studio/modelio` seam + `RunSummary` | not started | -| 0.5 `studio/science` derivations | not started | -| 0.6 `studio/runner` + job lifecycle | not started | -| 0.7 Golden-file harness (two tiers) | not started | +| 0.2 `studio/schema` v0 — **review gate** | **done** (#62, reviewed) | +| 0.3 Dependency-graph engine + override semantics | **done** (#66) | +| 0.4 `studio/modelio` seam + `RunSummary` | **done** (#68) | +| 0.5 `studio/science` derivations | **done** (#64) | +| 0.6 `studio/runner` + job lifecycle | **done** (#72) | +| 0.7 Golden-file harness (two tiers) | **done** (#70 measured, #79 asserted) | | 0.8 Four contained fixes in `coupled/` | not started | -| 0.9 Vertical slice: CLI + API + minimal UI | not started | +| 0.9 Vertical slice: CLI + API + minimal UI | **done** — 0.9a provenance (#80), 0.9b persistence (#83), 0.9c CLI (#85), 0.9d API + 0.9e UI (#87) | -Nothing is built on top of `studio/schema` until 0.2 is reviewed and merged. +Task order note: 0.5 was taken **before 0.3**, so the dependency-graph engine has real +derivations to resolve rather than fixtures. + +--- + +### 2026-08-17 — Phase 0 complete: `studio/dev` merged to `main` + +Nine tasks, 269 Tier-A tests, 43 typed source files. Every exit criterion met, and the ones that were +met *narrowly* are named below rather than rounded up. + +**What works end to end.** A configuration goes from a YAML file or a browser form through the +schema, the resolver, the model seam and the runner into a real coupled run, and comes back as a +`state.npz`, a versioned `RunSummary`, six recorded artefacts and an immutable provenance record +naming the SANDBOX commit and all three submodule SHAs. Both front ends share one submit function, +so their rows cannot diverge. + +**The merge into `main` had exactly the conflict predicted in the #73 review**: the two `studio/` +files that PR edited on `main`, which `studio/dev` had since moved past. Resolved as recorded then — +`studio/dev`'s structure (three clean packages, the seam-submodule check) with `main`'s corrected +past-tense wording (the `__post_init__` pattern it describes was fixed by #73). Both sides asserted +present rather than eyeballed. + +**Verified on the merged result, not before it**: 269 Studio Tier-A tests, the model's own 132 tests, +`ruff`, `black`, `mypy --strict`. + +**What is deliberately not done, and why** + +- **No React + Vite** (ADR-007 named it). One self-contained HTML page instead: a build toolchain for + a single form is machinery ahead of need. `/api/schema` exists so the form can be *generated* when + the UI outgrows one form. +- **CI cannot see the submodules**, so three checks skip there and run only locally: Tier A's real + 1-day run, the 0.4 equivalence test, and the `air_number_density` mirror check. A deploy key would + fix it; until then CI verifies the pure layers and a developer machine verifies the model-facing + ones. This is the weakest point in the setup and is worth saying plainly. +- **`PROGRESS.md` conflicts on almost every parallel PR** — six times in this phase, once losing a + commit to a squash. One file per entry would end it. +- **BLOCKING-2** (tenancy) and **SCIENCE-1, -2, -3, -5, -6** remain open; SCIENCE-4 is answered for + heating and buoyancy and open for sedimentation. + +**The five findings from this phase that changed the code rather than the docs** + +1. The plan's claim that the repository held two different dN/dlogDp conventions was wrong — they are + the same expression, 7e-16 apart. +2. Bit-for-bit reproduction of the archive is **false** (~31 % of gas elements differ), so ADR-009's + "measure before asserting" was load-bearing rather than cautious. +3. `dp_mid_um` is not bit-identical for a *Studio*-produced run, because 0.5 deliberately changed the + spelling — a tolerance measured through one pipeline is not a tolerance for another. +4. The first Tier-B run failed on a tolerance the harness had misread (endpoint vs series), not on a + regression. +5. `DateTime(timezone=True)` returns naive datetimes on SQLite and aware ones on Postgres; the same + comparison would have been right in production and wrong in development. + +--- + +### 2026-08-15 — Tasks 0.9d and 0.9e: the API, the page, and the shared service + +**Phase 0's exit criteria are met.** A run can be submitted from the CLI *and* from the web UI, +produces a stored result with full provenance, and the golden tests pass. 13 new Tier-A tests +(269 total). + +**One flow, two front ends.** `studio/service.py` was extracted the moment there were two callers: +a second copy of "resolve, record provenance, persist, submit, follow, store artefacts" would drift +within a week, and the drift would be invisible — both paths would keep working and only their rows +would disagree. ADR-002's "identical rows from either path" is a property of there being one +function, not of two being written carefully. + +**Verified against a real server**, not only a test client: + +``` +POST /api/runs -> 202 {"state":"queued"} +SSE t+ 0.0s running + t+20.1s succeeded +GET /api/runs -> reproducible: false · termination: completed + SO2 3.309e9 -> 1.720e6 pptv · peak N 3.07e6 cm^-3 +``` + +**Three bugs found by looking at output rather than at green tests:** + +1. **The database never showed `running`.** Transitions were written only after a job finished, so + the stream would sit at `queued` for four minutes and then jump to `succeeded` — a trail, but + useless as progress. `finalise` now follows the runner and persists each transition as it + happens, which is also why the stream can read the *database* and still be live. +2. **`reproducible` came back as `0`, not `false`** — an `Integer` column where a `Boolean` + belonged. The **drift test caught it** the moment the model changed, and the second migration + took one command. That is the return on putting Alembic in at 0.9b rather than later. +3. **SSE looked broken under `TestClient`**, reporting only the terminal state. It was the test + client serialising requests, not the code. The SSE test now runs a real uvicorn server in a + thread and changes state while the stream is open — the only shape that can catch a stream which + reports nothing until the end. + +**Deliberate divergence from ADR-007: no React + Vite.** The page is one self-contained HTML file +served by FastAPI. A build toolchain for a single form is machinery ahead of need — the same +reasoning that kept Redis and Docker out of Phase 0 — and the repository already has precedent in +`coupled/viz/*.html`. Recorded here rather than taken silently; React earns its place when the UI +outgrows one form, and `/api/schema` already exists so the form can be generated rather than +hand-written when it does. + +**The figure is deterministic from `RunSummary`** (ADR-004), drawn as inline SVG from the stored +summary — never from the raw npz. The same summary always draws the same figure, so a lost figure is +never a lost result. + +Smaller points: `POST /api/runs` returns **202**, because the run is accepted rather than finished, +and finalising happens on a worker thread — a four-minute `await` would stall every other request +including the stream reporting on that very run. Invalid configs are **422**, including +`heating_to_t: true`, which the schema refuses because the model cannot represent the physics +(SCIENCE-4); that refusal reaches the browser rather than crashing the server. `httpx2` joins +`studio-dev` as starlette's test-client dependency. + +--- + +### 2026-08-15 — Task 0.9c: the CLI (issue #85) + +**Half the exit criteria now works**: a run can be submitted from the CLI, produces a stored result +with full provenance, and is readable afterwards from a different process. 10 new Tier-A tests +(256 total). + +``` +plume-studio run config.yaml --out runs/ # 19 s for 1 day / 40 bins, end to end +plume-studio sweep sweep.yaml --plan # expand axes, print N, submit NOTHING +plume-studio status # from any process, after the CLI has exited +``` + +Verified end to end: `run` resolved, recorded provenance, persisted, submitted, waited, recorded six +artefacts and the summary, and exited `succeeded`. `status` — in a **separate process, after the +first had exited** — printed the full transition trail, every artefact with its size, and +`reproducible NO — a checkout was dirty`, which is the honest answer for a tree with uncommitted +work. + +**The CLI is not a wrapper over the API** (ADR-002). It goes through the same schema, resolver, +store and runner, so a sweep launched from a terminal and one launched from the web produce +identical rows and identical provenance. Scripted ensembles must not require the browser, and the +existing workflow is entirely scripted. + +**`--plan` mirrors `run_ensemble.py`'s `plan` verb** because deciding to spend 810 × 4.6 minutes +should take a second command. It prints what would run, with each case's hash, and creates no row. + +**Two things fixed after looking at real output rather than at tests:** + +1. **The persisted trail was thinner than the runner's.** The first end-to-end run recorded + `queued → succeeded`, dropping `running`. Persisting transitions is pointless if it drops one: + `queued → succeeded` hides how long a job waited for a worker, and `queued → failed` hides whether + it ever started. It now copies every transition the runner saw, and a test asserts the exact + sequence. +2. **`session.get()` returns `None`,** and I was passing it straight into the repository, where it + would have failed several frames later as an `AttributeError` about `None`. mypy caught it; it now + raises naming the row and the key, since it means the database changed under a run in flight. + +Exit codes keep 0.6's meaning: **2** for "never started" (a bad file, an invalid config, an unknown +run), **1** for a run that did not succeed. `--dry-run` and `--plan` are the cheap paths, so most of +the tests need neither the model nor a database. + +--- + +### 2026-08-14 — Task 0.9b: persistence (issue #83) + +`studio/store/`: models, engine, artefact store, repository, and **Alembic from the first +migration**. 15 new Tier-A tests (246 total). + +**Nine tables**: `run_set`, `run`, `run_config`, `job`, `job_transition`, `result_artifact`, +`dataset_version`, `run_dataset`, `run_summary`. `dataset_version` is empty in Phase 0 and exists +anyway — "which ERA5 product was this run built on?" is a question Phase 1 must be able to ask about +runs made before it existed. + +**Immutability is structural, not conventional.** `run_config` is keyed by the config's own hash and +`ensure_config` is get-or-create; there is no update path, and a test asserts the repository exposes +no `update`/`delete`/`overwrite` helper at all. An edited config is a different row and a different +run, linked by `derived_from_run_id`. + +**The database stores pointers, never arrays.** Artefacts go to `LocalDirectoryStore` behind an +`ArtifactStore` protocol — a directory today, MinIO or S3 later without touching callers — and the +row keeps a *relative* path, size, content type and **SHA-256 computed on write**. That checksum is +what makes "still the file that was written" checkable: silent corruption and a helpfully tidied +directory look identical from the database otherwise. A missing artefact **raises** rather than +being recorded as an absence. + +**Two portability decisions, both because SQLite and Postgres would otherwise disagree silently:** + +1. **`UtcDateTime`, a `TypeDecorator`.** `DateTime(timezone=True)` is not enough — Postgres returns + an *aware* datetime and **SQLite returns a naive one**, so the same comparison is right on one + backend and wrong on the other. Caught by a test asserting `tzinfo is not None`, which failed on + the first run. Naive input now *raises*: a caller who does not know their own timezone cannot be + handed one by guessing. +2. **`foreign_keys=ON` for SQLite.** Without it SQLite ignores foreign keys entirely, so the + constraints in `models.py` would be documentation on the Phase-0 backend and enforced in + production. A test inserts a job for a nonexistent run and requires an `IntegrityError`. + +**The drift test.** `test_the_models_and_the_migration_agree` runs Alembic's `compare_metadata` +against a migrated database and requires an empty diff. Without it, a column added to `models.py` +without a migration works everywhere the schema was built from the models and fails on the first +real deployment. Related: **nothing uses `Base.metadata.create_all`, including the tests** — every +test upgrades through the migrations, so the migrations are exercised continuously rather than for +the first time on someone's database. + +Four headline scalars (`final_so2_pptv`, `peak_h2so4_pptv`, `peak_number_cm3`, +`final_surface_area`) are promoted out of the summary JSON into columns, so "every run where peak +number exceeded X" is a query rather than 810 deserialisations. They are read from the summary +rather than recomputed, so column and JSON cannot disagree. + +`runs_for_config` deliberately returns *runs* rather than a cached-result verdict: an identical hash +is necessary but not sufficient, because the caller must also compare the model version in each +run's provenance (ADR-006). + +--- + +### 2026-08-14 — The archive's status as a reference, settled and written down + +Ali, 2026-08-14: **`coupled/paper_ensemble/runs/` is a valid reproduction reference.** Recorded in +`REFERENCE_TOLERANCES.md` rather than left as an implicit property of the harness, because it is an +assumption the data cannot support on its own — those files carry no provenance record, which is +exactly the gap ADR-006 closes going forward and cannot close retroactively. + +**Every other archive directory is excluded, and now says why**: `runs_60day` (initialises from a +spun-up control run), `runs_bgstop*`, `runs_boxsize`, `runs_geo`, `runs_no_sai`, `runs_special`, +`runs_start_time*`. They were produced differently — different initialisation, different vintages, +different configurations — so rebuilding one from the axis tables would compare two different +computations, where a pass is luck and a failure means nothing. The exclusion is structural: +`paper_cases.py` only maps the factorial's case IDs. + +**All six curated cases now measured** (the record previously had two): every endpoint ≤ 4.1e-14 +against `1e-12`, every series ≤ 1.6e-11 against `1e-10`, `t`/`V_ratio`/`T` bit-identical in all six, +256–293 s per case. + +**Two claims corrected by the wider data.** + +1. *Timing.* This record read as though deviation were tied to the early nucleation burst — the + two-case measurement had found the worst H₂SO₄ deviation at day 1.34. Across six cases the worst + days are 5.83, 1.34, 9.27, 2.15, 3.03, 5.74: **no common feature**. It is a flat ~1e-14 baseline + with occasional spikes — round-off scattered through the run, not accumulation. +2. *The size-distribution outlier.* The four `sabr220` cases share an identical 1.63e-11 at the same + cell (t = 0.042 d, bin 5, 4.218 counts); the two `sabr330` cases peak late and elsewhere (day 7.34 + bin 6; day 8.41 bin 11). The common thread is **sparsity, not timing** — every one is a bin + holding 1.7–4.2 particles cm⁻³. It is inherited from `n_cm3`, not from the `dlogdp` normalisation, + whose divisors agree to 6.1e-15. + +**Added**: `measure_all_cases.py` (re-measure all six, incrementally, asserting nothing) and +`plot_fidelity.py` (three figures: headroom against tolerance, deviation against time, and the +near-zero floor). Both are tools rather than tests — re-measuring must never "fail", it reports, and +a human decides. Figures are regenerable and not committed. + +--- + +### 2026-08-14 — Task 0.9a: provenance records (issue #80) + +**0.9 is not one PR.** The exit criteria need FastAPI + SQLAlchemy/Alembic on SQLite, a CLI, +React/Vite with SSE, and a figure from `RunSummary`. Split into 0.9a (this), 0.9b persistence, 0.9c +CLI, 0.9d API, 0.9e UI + figure. This is the piece nothing implemented and the exit criteria depend +on: *"produces a stored result **with full provenance**"*. + +`studio/modelio/provenance.py`, written by the runner **at submit time**. 17 new Tier-A tests +(213 total). + +**What a record pins**: `config_hash`, `studio.__version__`, the SANDBOX SHA, **all three submodule +SHAs**, whether each checkout was dirty (with the offending paths), and the **resolved, +post-derivation** parameter set — what the model actually received, not what the user typed. Plus any +override with the value in force. `datasets` is present and empty rather than omitted, so its +emptiness is never ambiguous. + +**This is the one place Studio shells out to git, and it is strict about it.** Not-a-checkout, +git-not-installed, a repo with no commits, or a missing submodule all **raise**: an empty SHA in a +provenance record is worse than no record, because it looks like an answer (ADR-005). Cleanliness +comes from `status --porcelain`, not `diff --quiet`, so an **untracked** file counts — an untracked +module that a run imported is exactly what makes a SHA a lie. + +**Written before execution, and proven so.** The test asserts against the record `submit()` returns, +not after `wait()` — checking afterwards would pass even if it were written at completion. Verified +end to end: at submit the work dir holds `input.json` + `provenance.json`; on completion, six +artifacts including `state.npz` and `summary.json`. + +**Three times in this task the tests failed and the code was right.** Each was my expectation of git +being wrong, and each is documented where it will be re-read: + +1. A *nested* repo is not a *registered* submodule — the parent reports the nested one as untracked + and so reads dirty. The fixture was lying about the shape of a real checkout; it now uses + `git submodule add`. +2. A dirty submodule flags **both** it and the parent, because the parent's recorded pointer no + longer matches the working tree. That is git being helpful: an edited submodule cannot hide behind + a clean-looking SANDBOX. +3. `protocol.file.allow=always` is needed for local-path submodules (CVE-2022-39253). + +The tests build **real git repositories** rather than mocking `subprocess`: the module is a thin +shell over git's behaviour, so a mocked git would test the mock. ~1 s, worth it. + +**A fourth thing CI caught that local tests could not.** Making provenance mandatory at submit means +the runner now needs a pinnable checkout — and CI checks out no submodules, so every runner submit +test failed there while passing locally. The fix is a `repo_root` parameter on the runner (which +checkout to record), pointed at the shared synthetic-checkout fixture in the tests. It does **not** +weaken the guarantee: a run still cannot start unless the checkout it names can be pinned, and the +production default is the real one. "Which checkout produced this?" is a question a runner genuinely +has to answer — a worker executing code from elsewhere would answer it differently. + +### 2026-08-14 — Task 0.7 (second half): the two-tier golden harness + +The assertions, built on the tolerances #70 measured and #76 corrected. `studio/tests/golden/`: +`tolerances.py`, `paper_cases.py`, `make_fixture.py`, and one test module per tier. 17 new Tier-A +tests (214 total) plus 3 Tier-B tests. + +**Tier B's first real run found a bug in the harness, and it was mine.** Three exceedances — +`D2med/H2SO4 3.377e-12`, `D3high/SO3 5.995e-12`, `D3high/OH 5.535e-12` — all against `1e-12`. Not a +reproduction failure: `3.377e-12` is essentially the **3.38e-12 the measurement itself recorded** for +H₂SO₄ max-over-time. The harness applied the *endpoint* tolerance to whole-*series* comparisons, which +are two different rows of the record (`1e-12` for final/peak, `1e-10` for a series maximum). + +No tolerance was widened — that is what this harness's own failure messages forbid. The two numbers +the record already specifies are now applied to the two things they describe, via a named +`assert_headline_matches` so the call sites read like the record's rows. Tier A had the same +conflation, invisible there because it compares against its own fixture where the deviation is ~0. + +Tier B also now reports **every** deviation rather than only the exceedances: 27 minutes of compute +should produce a measurement, not a verdict. The D3high series maxima (~6e-12, inside `1e-10`) are +new data the original two-case measurement did not have. + +**Tier A — 19 s, a real run against a committed fixture.** 1 day, 40 bins: the cheapest run that +still exercises gas chemistry, TUV-x photolysis, all three microphysics processes and dilution. The +fixture is a **uniform-stride** reduction (every 4th sample plus the last — 38 of 147, 40 kB) because +a coarsening grid aliases the morning number spike by up to 8×, and a fixture built on one would +encode the aliasing and then assert it forever. It catches drift in *Studio's own* pipeline, which is +a different claim from reproducing the archive. + +**Tier B — six curated 10-day cases against the archive**, ~28 min, nightly/manual. D1/D2/D3/burst × +sabr220/sabr330, all `cg1` deliberately: `REFERENCE_TOLERANCES.md` records that cg0p5/cg2 may +straddle the tomas-jax commit that wired `coag_kernel_scale` through, so adopting one needs its own +measurement first. + +**Decisions** + +- **The tolerances live in one module, each citing its measurement**, and the failure messages say + *re-measure, do not widen*. A tolerance widened to make a test pass is a test that no longer tests + anything; putting the provenance at the point of failure is the cheapest defence against that. +- **The near-zero floor is in the comparison, not in each test.** Unguarded relative error reaches + 4.24e+04 on night-time `O1D` at 1e-35 molec cm⁻³; comparing only samples above 1e-6 × a series' own + peak is what makes the comparison mean anything, and `O1D`/`O` are excluded outright. +- **Photolysis is compared per reaction, not summed.** A compensating pair of errors across two + reactions survives a total. J was added to the fixture for this — at 1.09e-13 measured it is the + most reproducible part of the pipeline, so drift there is signal rather than noise. +- **Tier B reports every deviation before failing**, and is not parametrised per case: after 28 + minutes of compute, the whole table is worth much more than the first failure, and it shows whether + a deviation is systematic or specific to one regime. +- **Both tiers carry a physical floor** — SO₂ consumed, H₂SO₄ produced, particles formed, plume + expanded. A tolerance-based test cannot tell that a run did nothing at all. +- `paper_cases.py` maps a case ID back to a `RunConfig` by parsing the ensemble's own token + convention. It lives under `studio/tests/` rather than in `studio/`: Tier B needs the axes as test + data, which is not the same as needing a preset library (task 0.2 deferred that deliberately). + +**Honest limitation, and it undercuts the plan's wording.** The plan calls Tier A "CI, seconds", but +CI does not check out the private submodules, so the model cannot run there — in CI this module +**skips**, and Tier A there remains the pure schema/units/DAG/hash/expansion tests. Fixing it means +giving CI a deploy key, which is its own change. Recorded rather than papered over. + +--- + +### 2026-08-14 — `main` merged into `studio/dev`; the stop condition moves to the diagnostics dict + +`studio/dev` now has #73's four model fixes. Studio's whole suite (196 Tier-A tests, equivalence +tests included) passes against the changed model unmodified — `output_dir` is gone and nothing missed +it, which is the evidence that removing it was safe rather than merely tidy. + +**Conflicts resolved, one per file, deliberately:** + +- `studio/tests/unit/test_import_boundaries.py` — took `studio/dev`'s *structure* (three clean + packages including `studio.resolve`, and the seam check that allows submodules of `studio.modelio`) + with `main`'s *wording* (the `__post_init__` pattern it describes is now past tense, because #73 + fixed it). Asserted both survived rather than eyeballing the merge. +- `studio/__init__.py` — merged cleanly, and the merge exposed a stale docstring of mine: it still + claimed two clean packages after 0.3 added a third. Fixed here, since this is where it became + visible. + +**The stop condition now takes the diagnostics dict.** #73 deprecated the two-argument form, so +leaving it would have had Studio emit a `DeprecationWarning` on a normal path. `diag["t"]` is all it +reads today, but the dict also carries `SA`, `N_total` and every gas species by name — which is what +makes the spec's SO2- or number-based `termination.criteria[]` possible at all. + +**`max_sim_time` enforcement is now proven end to end, not just unit-tested.** A 1-day request with +`max_sim_time_days = 0.5`, run through `studio.cli.run` with `-W error::DeprecationWarning`: + +``` +[stop] condition met at t=0.500 d -- ending run early +termination: terminated_on_limit flags: [stopped_on_limit, open_system_dilution] +t_end: 0.5 d (requested 1.0) 74 steps +``` + +That path had never actually run before — nothing set `max_sim_time`, so it was the one part of 0.6 +covered only by unit tests. It also confirms 0.6's termination inference: the run is labelled +`TERMINATED_ON_LIMIT` and flagged, so its partial output cannot be read as converged. + +--- + +### 2026-08-14 — Correction: `dp_mid_um` is not bit-identical for a Studio-produced run + +The tolerance record merged in #74 proposed asserting `dp_mid_um` **exact**. That holds only when the +fresh run comes from `run_ensemble`. A run produced by Studio differs in **44 of 80 bins by up to +8.1e-16**, because task 0.5 deliberately adopted `sqrt(a*b)` where `run_ensemble` writes +`10**(0.5*(log10 a + log10 b))` — the two spellings 0.5 proved algebraically identical. + +Asserting exact equality there would have passed against the old pipeline and failed against every +run Studio itself produces, presenting as a physics regression over a rounding difference. Corrected +to `1e-15`; `t`, `V_ratio` and `T` stay exact. `dNdlogDp` inherits the difference at 3.2e-13, inside +its own `1e-10`, so no other row moved. + +Found by re-running the golden case through `studio.cli.run` as an independent check — every other +number reproduced to the digit, and both runs are now tabulated in the record. Narrow lesson worth +keeping: **a reproduction tolerance measured through one pipeline is not automatically a tolerance +for another**, even when the two are meant to agree. + +--- + +### 2026-08-13 — Schema 0.2.0: heating and buoyancy are out of scope, not pending + +Ali's decision, and the reason is worth stating precisely: **longwave radiation is not in the +radiative calculation**, so the model's heating term cannot represent the box's energy balance. +Enabling it does not make the thermodynamics more complete — it makes them *one-sided*, and the +resulting ~+1.2 K / 10 d warm drift is an artefact of the missing cooling rather than a physical +response. + +`switches.heating_to_t` is therefore `Literal[False]`, the same treatment `dilution.background_evolves` +already had: `True` **fails validation** rather than being defaulted off, so it cannot be enabled by a +form, a YAML file, or a sweep axis without the schema changing first. Two tests cover it — the direct +one and the axis path, which is the one that would slip past a UI-level guard. + +**`SCHEMA_VERSION` 0.1.0 → 0.2.0, and the pinned hash moved with it** (…46cbe3 → …373ab4). Note the +*value* of `heating_to_t` did not change — it was already `False` — but `schema_version` is part of +the hashed payload, which is exactly what makes "old configs are never silently reinterpreted under +new semantics" true rather than merely stated. A config written yesterday no longer hashes to a +0.2.0 identity, which is the intended behaviour. + +**Buoyancy is closed for the same reason** (Ali, same day): a parcel rises in response to a heating +rate this model cannot compute, so a rise velocity here would be a free parameter dressed as physics. +Answering either question needs a **different model**, with longwave radiation and plume dynamics — +so this is a scope boundary, not a gap for a later phase to fill. Consequently **no buoyancy or +heating-rate fields are added to the schema at all**: a field for a capability the model lacks would +advertise it, and the absence is the honest interface. + +SCIENCE-4 is therefore **answered for heating and buoyancy**. `numerics.box_thermodynamics.*` moves +in the capability register from "see SCIENCE-4" to "not exposed, by decision". + +**Sedimentation stays open, deliberately.** It is a particle-loss process, not a thermodynamic +response; "we decided not to model heating" is not an argument about gravitational settling, and +sweeping it into this decision would have quietly closed a question nobody answered. + +`CAVEATS.md` says it where a reader of results would look: every run is isobaric and isothermal at +the configured temperature, and a result must not be read as containing a plume-warming signal, a +lofting signal, or an altitude change. + +Two tests that set `heating_to_t=True` as an innocuous example were updated to use +`switches.aerosol_to_j` instead. They were not weakened; the value they used simply became illegal. + +--- + +### 2026-08-13 — Task 0.6: the runner and the job lifecycle (issue #72) + +`studio/runner/` (`base.py`, `local.py`), `studio/modelio/execute.py`, `studio/cli/run.py`. 20 new +Tier-A tests (191 total). + +**The first end-to-end run.** `python -m studio.cli.run ` runs the real model +and writes `state.npz` plus `summary.json`. Verified on a 1-day, 40-bin case: **21 s wall**, +SO₂ 3.309e9 → 1.72e6 pptv, peak H₂SO₄ 15.05 pptv, peak number 3.07e6 cm⁻³, the npz key set identical +to the canonical one from `run_ensemble.py:150-156`, and `termination = completed` with a real +`config_hash`. That last part matters: a Studio-created run has provenance, which is exactly what the +archived ensemble lacks (ADR-006). + +**Lifecycle as data, transitions enforced.** `DRAFT → QUEUED → RUNNING → (SUCCEEDED | FAILED | +CANCELLED | TERMINATED_ON_LIMIT)`, every transition timestamped and kept — "it failed" is not +debuggable, "QUEUED 14:02:11, RUNNING 14:02:11, FAILED 14:06:48 exit 1" is. Illegal transitions +raise: a job that appears to move backwards means the runner lost a process, and accepting it +silently would turn the record from a log into a story. + +**`TERMINATED_ON_LIMIT` is not `FAILED`.** One means the model could not produce a result; the other +means it was still going when we stopped it, and its partial output can look complete. Kept distinct +all the way through, and the detail string says so. + +**A failed run is debuggable without re-running it.** The resolved input is written at *submit*, not +at completion, so a job that dies immediately still has its input; stdout and stderr are captured in +full (`run_coupled` prints rather than logs, so stdout IS the log stream); the exit code is recorded. +That set is chosen for the case that actually hurts: a four-minute run that fails intermittently. + +**Decisions** + +- **`entry_module` is a parameter, not a test hook.** The runner launches a module by name; tests + point it at a fixture module so the lifecycle can be exercised in milliseconds instead of four + minutes. Nothing in the runner branches on the value, the default is the real entry point, and the + real one is exercised separately by the exit-code test. +- **Exit codes mean something specific**: 0 ran, 2 the input was bad and nothing started, 1 the model + raised. The runner needs to tell "never started" from "broke", and `studio.cli.run` validates the + input *before* importing the model so a bad input fails in milliseconds rather than after a JAX + load. +- **Slurm and cloud-batch raise** (ADR-008) rather than falling back to local execution. A job + running somewhere other than where it was sent is worse than an error. +- **`max_sim_time` is enforced through a `*args` stop-condition**, which works either side of task + 0.8's widening of that callback rather than depending on which has landed. +- `studio/modelio/execute.py` reuses `coupled.dilution.volume_ratio` and `studio.science`'s + size-distribution reduction rather than inlining a fifth copy — which is what 0.5 was for. + +**Test-design note.** The runner tests launch **real subprocesses**; a mocked `Popen` would test the +mock. The fixture module's behaviour arrives by environment variable, set before `submit()`, because +a directive file written *after* submission races the subprocess start — the standard way process +tests become flaky. + +### 2026-08-13 — Task 0.7 (first half): the reproduction tolerance, measured (issue #70) + +ASSUMPTION-2 is settled. Two archived cases re-run at today's SHAs and compared per quantity against +the archived `state.npz`: the golden case `30N_20km__sabr220__D2med__a1p0__nuc1__cg1` (index 121) and +a deliberate contrast, `30N_20km__sabr330__burst__a1p0__nuc1__cg1` (index 67) — `burst` dilution and +the loaded background, the regime where a regime-dependent residual would show. Full record with the +SHAs, the environment and the per-quantity table: +[`studio/tests/golden/REFERENCE_TOLERANCES.md`](../../studio/tests/golden/REFERENCE_TOLERANCES.md). + +**Reproduction is close but not bit-for-bit.** Every headline quantity agrees to **≤ 2.1e-12**, the +worst deviation anywhere in either run is **3.4e-12**, and the time axis, `V_ratio`, `T` and the dry +bin edges are bit-identical. But only ~31 % of gas state-vector elements and ~1 % of aerosol samples +reproduce exactly, so `atol=0` would have failed on arrival — exactly the outcome ADR-009 was written +to catch. + +Two controls make the reading firm rather than hopeful: running the same case twice **today** is +bit-identical across all 18 stored arrays (so the residual is environment drift, not run-to-run +noise), and the worst deviations are scattered across days 1.3–9.8 rather than accumulating (the +signature of round-off, not of a diverging integration). The `bd289e9` day-12 solver change is +consistent with being invisible here: a 10-day run never reaches t = 2²⁰ s, so the retry branch is +never taken. + +**The trap worth knowing before writing the assertions:** unguarded relative error over the raw gas +state vector peaks at **4.2e+04**, entirely on night-time `O1D`/`O` at O(1e-35) molec cm⁻³ — values +that oscillate about zero, including negative, on a species whose peak is ~3 molec cm⁻³. Golden tests +must floor by series magnitude or they will fail by four orders of magnitude over an absolute +difference of 1e-34. + +No assertions were written in this pass, by design. Wall clock: ~4.6 min per 10-day / 80-bin case, +matching BLOCKING-4. + +--- + +### 2026-08-13 — Task 0.4: the model seam and `RunSummary` (issue #68) + +`studio/modelio/`: `scenario.py` (the seam) and `summary.py` (the reduction). 23 new Tier-A tests +(171 total). + +**The equivalence test passes.** `to_scenario(resolve(RunConfig()))` is **field-for-field identical** +to `run_ensemble.build_scenario()` for `30N_20km__sabr220__D2med__a1p0__nuc1__cg1`, compared as +`dataclasses.asdict` with exact equality on every field including the floats. The schema is a +faithful superset of what the ensemble ran, and that is now proven rather than intended — before any +run, instead of via a diverging result days later. The derived `SO2` initial concentration matches +to the last bit, which is the evidence that consolidating that derivation in 0.5 changed nothing. + +**Measured, because the cost is not where anyone would guess.** Importing +`studio.modelio.scenario` takes ~0.13 s and pulls in **no JAX at all**. The first `to_scenario()` +*call* takes ~1.05 s, because `CoupledScenario.__post_init__` imports `coupled.tomas_bridge` to +validate `background_dist` (`coupled_scenario.py:196`) and *that* is what loads JAX; later calls are +free. So `to_scenario` is deliberately **not** re-exported from `studio/modelio/__init__.py` — +a comparison view reading a `RunSummary` should not pay for a model it is not using, and the +expensive import should be visible at the import site. Task 0.8 tracks making the model's import +lazy. + +**`RunSummary`** — versioned, and self-describing about the three traps: + +- **Every series declares its basis.** `SA`/`radius_cm` are WET, `dp_mid_um`/`dNdlogDp`/`total_n` + are DRY, gas mixing ratios are not-applicable. It is a required field, so a plot axis cannot be + labelled by guesswork. +- **Species are indexed by name** from the npz's own `species` list. The synthetic test archive + deliberately orders species so SO2 sits at index 2 — nothing like the 32/34/35 an existing script + hard-codes — so a positional reduction would report ozone as SO2, plausibly and silently. +- **The time axis is the stored `t`**, never `i × DT`. +- **Termination is an argument, never inferred.** The npz records what the state did, not why the + loop stopped; a run that hit a wall-clock limit and one that finished look identical in it. + Archived runs are `UNKNOWN` and flagged `NO_PROVENANCE_RECORD` (ADR-006). + +**The conservation check refuses to report a number when one would mislead.** In-box sulfur is not +expected to be conserved with dilution on — the box is an open system, so a large "residual" would +be measuring the dilution and a small one would mean something was wrong. When `V(t)/V0 > 1` the +check returns `not_applicable` with the reason and the start/end values, so the decay is visible +without being dressed up as a budget error. A closed box gets a real residual. + +**Found and fixed while writing it** + +- A `sum()` over a generator starting at integer `0`, which mypy caught: the sulfur total was + `ndarray | Literal[0]` and would have been unindexable had the species list ever been empty. +- Once `studio/modelio` imported `coupled`, `mypy` began reporting errors from the **model's own** + source (its untyped `yaml`, `scipy`, and flat `aerosol` imports). Fixed with + `follow_imports = "silent"` on the model modules — Studio's use of them is still checked; the + model is not ours to annotate. +- Two of my own test constants were wrong: an "irregular" time axis whose last point was exactly + `4 × 600 s` (so it proved nothing about nominal grids), and a closed-box sulfur budget that did + not close. Both were caught by the tests failing, which is the system working. +- I wrote a test checking `sys.modules` in-process for the resolver — the exact mistake + `test_import_boundaries.py`'s own docstring warns about, and it failed as soon as another module + imported the seam. It now runs in a fresh interpreter, where it is meaningful. + +--- + +### 2026-08-13 — Task 0.3: dependency graph and override semantics (issue #66) + +**New package: `studio/resolve/`** — `graph.py` (the DAG), `registry.py` (which function computes +which field), `resolver.py` (resolution, overrides, staleness). 47 new Tier-A tests (148 total). + +**Why a fourth pure package rather than a module in `studio/schema`.** `studio/schema` is data and +stays free of computation; `studio/science` is computation and stays free of the config model. +Resolution is the composition of the two. Naming it keeps that layering visible — and keeps schema +and science usable, and testable, without it. It joins schema and science in the import-boundary +test and under `mypy --strict`, because the API resolves a config on every keystroke and a JAX +import on that path would be unaffordable. + +**The semantics** + +- **auto** → recomputed silently whenever anything upstream changes. +- **user_override** → never overwritten by a recomputation. +- **user_override + stale** → an override whose inputs have moved since it was set. + +Staleness is defined against a **fingerprint**: setting an override records the upstream values at +that moment. Stale means those recorded values differ from the current ones. That makes staleness a +property of the config alone — no edit history, no ordering assumptions — and it is what lets the UI +show the old value, the newly-derived value, *and what changed between them* rather than a bare +warning. Two explicit ways out, both the user's call: `accept_derived` (drop the override) or +`keep_override` (keep the value, re-anchor the fingerprint; it goes stale again on the next change +rather than being permanently silenced). + +`ResolvedConfig.require_consistent()` raises on any stale field, and the stale list survives +serialisation — a persisted config cannot lose the fact that it is inconsistent. The trap it guards: +a stale config still *has* a hash, which would be a stable identity for numbers that do not follow +from each other. + +**The load-bearing test** is `test_an_edit_changes_exactly_the_downstream_closure`: capture every +field before and after an edit, assert the set that moved is **exactly** the edited field plus its +closure. Both directions are silent failures — recomputing too little leaves a stale number that +reaches the model, recomputing too much discards something the user chose. It runs over eight edits +including three fields with no dependents, where the expected change set is the edited field alone. + +**Decisions** + +- **Editing a derived field IS an override.** A user typing into a computed box means "I want this + value", not "recompute me away on the next edit". +- **The registry is checked against the schema, not trusted.** Every `DERIVED` field must have a + derivation, no derivation may exist for a field the schema does not derive, and each derivation's + declared inputs must equal the field's `derived_from` exactly. Without that, a field could declare + an input its derivation ignores (the UI reports a change that did not happen) or read one the + graph does not know about (the stale result reaches the model). +- **Cycles raise at graph construction.** Not fixed-point iteration, not breaking an arbitrary edge: + both would produce numbers that depend on where the engine started. Tested on synthetic graphs, + since the real schema has no cycle to exhibit. +- **Topological order, ties broken alphabetically** — deterministic because resolution order is + observable through which error surfaces first. +- The graph tests run mostly on **hand-built graphs**: the schema has exactly one chain of length + two today, and an engine tested only against the shape it currently meets breaks the first time + the schema grows. + +**Corrected while writing the tests:** a test asserted that a zero plume dimension would fail inside +the derivation. It fails earlier, at the schema's `gt=0` bound — which is the better layer, because +the error names the field the user typed rather than a function they have never heard of. The +derivation's own check stays as the guard for callers that do not come through the schema. + +--- + +### 2026-08-13 — Task 0.5: `studio/science` derivations (issue #64) + +Taken **before 0.3** at Ali's direction, so the dependency-graph engine has real derivations to +resolve rather than fixtures. + +**Added** — `studio/science/`: `constants.py`, `air.py`, `plume.py`, `size_distribution.py`, +`gcr.py`. 40 new Tier-A tests (101 total). + +**What the consolidation actually found.** The plan described this task from memory, and two of its +claims did not survive contact with the code. Both are corrected in `plan/PHASE_0.md`: + +- **Six copies of the V₀ / initial-concentration derivation, not five** — the missed one is + `make_rf_runs.py:44` — and `run_dilution_d1_clean.py` is at `coupled/`, not + `coupled/paper_ensemble/`. The divergence is real and it matters: `run_ensemble.py` uses a **15 km** + track, the D1 flagship a **30 km** one. Same injected mass, half the concentration. Which is right + depends on SCIENCE-2 (#54). Also verified: `run_60day.py:37`'s hard-coded `6.273063291666667e15` + is **bit-identical** to what `run_ensemble.py:46` computes, so it is a frozen copy, not a variant. +- **The "two different mid-point expressions" are one expression.** + `10**(0.5*(log a + log b))` and `sqrt(a*b)` are algebraically identical, as are + `log b - log a` and `log(b/a)`. Measured difference on an 80-bin grid: ≤ 7e-16 (mid-point) and + ≤ 5e-15 (dlog10Dp) relative — a few ULP of float64. Consolidating is still worth doing; believing + there were two conventions was not. `test_the_repositorys_two_spellings_are_the_same_quantity` + measures it rather than asserting it, because that belief would otherwise get worked around. + +**Decisions** + +- **`air_number_density` is a MIRROR, not a fork.** `studio.science` may not import the model + (ADR-001), so this one relation is duplicated — and `test_science_air.py` runs the model's own + implementation in a subprocess and asserts **exact** agreement at the four T–p corners the runs + use. That is what makes a duplicate acceptable. Note the asymmetry: CI does not check out the + private submodules, so this check *skips* in CI and only really runs on a developer machine. +- **`gcr.py` computes nothing.** `ion_pair_production_rate` raises `NotImplementedError` naming + SCIENCE-6; `PAPER_ENSEMBLE_ION_PAIR_RATE = 30.0` is available as a constant with its provenance. + A test asserts it refuses **even at ~20 km / 30°N**, where the uncited 30.0 came from — returning + the known value at the known point and raising elsewhere is the most tempting version of this + mistake, because it looks like a working function with gaps. +- **Two constants are deliberately the model's rounded values**, recorded as such in `constants.py`: + SO₂ at 64.0 g/mol (true 64.066, a 0.10 % difference) and H₂SO₄ at 98.0 (true 98.079). Studio + inherits them so Phase 0 reproduces the golden runs; a silent correction would shift every derived + initial concentration and make a Studio bug indistinguishable from a model change. The ~0.036 % + Avogadro seam at the gas/TOMAS boundary is likewise recorded (`AVOGADRO_GAS_MODEL`) and not used. +- **Reused, not rewritten**: `coupled.dilution.volume_ratio` / `kdil_from_regime`, + `coupled.aerosol_props`, `coupled.units`. They are already tested in the model, and Studio reaches + them through `studio/modelio` rather than keeping a second copy. + +**Toolchain** — Studio's Python floor is now stated as **3.12** in all three tools. numpy's bundled +type stubs use 3.12-only `type` statements, so `mypy --strict` could not check `studio/science` +against 3.11 at all; the lockfile and CI were already 3.12. No dependency changed, so the lockfile +is untouched. + +--- + +### 2026-08-13 — Task 0.2: `studio/schema` v0 · **merged** (#62) + +The review gate. Nothing is built on top of this until it is reviewed and merged. + +**Added** — `studio/schema/`: `units.py` (closed canonical registry), `fields.py` (`SciField`, +`Provenance`), `enums.py`, `config.py` (`RunConfig` and its ten groups), `runset.py` (`RunSet`, +`Axis`, expansion), `hashing.py` (canonical JSON + SHA-256), `export.py` (JSON Schema + flat field +catalogue). 41 leaf fields, every one carrying unit, description, range and provenance. Four test +modules, 61 tests, all Tier A. + +**The load-bearing decisions** + +- **Provenance is required and its rules are enforced at import time.** `MODEL_DEFAULT` and + `PAPER_ENSEMBLE` must give a `source`; `LITERATURE` must give a `cite`; `DERIVED` must give + `derived_from` and must *not* give a value. A field whose default has no recorded origin cannot be + declared — which is the one failure mode `studio/CLAUDE.md` is most emphatic about, made + structural rather than aspirational. +- **Defaults are the paper ensemble's, not the model's**, where they differ (ASSUMPTION-5). The + visible case is `ion_pair_rate`: the model defaults to 0.0, which disables ion-induced nucleation + entirely, while the ensemble uses 30 cm⁻³ s⁻¹. Both are recorded, with the divergence stated on + the field. +- **`RunConfig()` with no arguments is the golden case.** That is not a convenience: it is the + form's opening state and the base of every RunSet. +- **Identity contains only what changes the result.** No `label`, `notes` or `output_dir` field — + two runs differing only in a name are the same computation. Labels live on `ExpandedRun`. +- **The hash is pinned by a test**, not merely asserted self-consistent, and checked across four + `PYTHONHASHSEED`s in fresh interpreters. Canonical form: sorted keys, no padding, `allow_nan=False` + (NaN raises rather than emitting a token no other parser reads back). +- **`RunSet` reproduces the 810-run ensemble** — same count, same order, same case IDs, verified + against the golden case at index 121. This is the strongest available evidence that the axis model + is faithful to what this project actually does, and it is why `LIST` exists: the site axis covaries + latitude, T, p and H₂O, and its cross product is not physically meaningful. +- **Model validation is mirrored where it is cheap** — the `DT`/`dt_couple` divisibility rule, the + 40/80/160 bin grids, `background_evolves` accepting only `false` (SCIENCE-5). Each mirror cites the + model line it copies. ADR-002's third motivating problem was that a form cannot learn what is valid + without importing most of the model; this is the answer to it. + +**Divergences from the plan, stated rather than absorbed** + +- `max_sim_time` is **optional**, not required. Task 0.6 says both limits are required; but simulated + time is already bounded by `schedule.duration_days`, so a required second copy would be redundant, + and inventing a default ceiling would be a fabricated number. It is an optional *lower* ceiling. + `max_wall_time_s` is required and defaults to 3600 s (ASSUMPTION-4). +- `DilutionRegime.CONSTANT` is spelled `"constant"`, while the model spells it `""`. An empty string + is not a usable dropdown key. This is the **only** enum value that is not the model's own string, + it is flagged at the point of deviation, and task 0.4's equivalence test must cover it explicitly. + +**Deliberately not done** — no physics: `plume_volume_cm3` and `so2_initial_pptv` are declared with +`derived_from` and left unresolved (0.3 resolves, 0.5 derives). No species-name validation: the +species list belongs to the model, so `studio/modelio` validates at the seam. No preset library: the +paper ensemble's axes live in the test, and earn a home in `studio/` when 0.4 or 0.7 needs them. + +**Found while writing it:** `resolve_path` initially accepted a path naming a whole group. The test +caught it. Groups are now rejected — they have no unit, no provenance and no node in the dependency +graph 0.3 builds from leaf paths. + +**One interpreter-level assumption**, recorded in `hashing.py`: float formatting via `repr` has been +the shortest round-tripping decimal since Python 3.1, so the canonical form is stable across the +versions this project supports. The pinned-hash test is what would catch that changing. + +--- + +### 2026-08-13 — Branching: `studio/dev` becomes the integration branch + +Not a task; a workflow decision taken after task 0.1 merged (#59). + +Studio tasks now branch off **`studio/dev`** and PR into it; `studio/dev` merges into `main` at +phase boundaries. `main` therefore sees Studio in reviewed batches rather than one task at a time, +while the model and viz work continues on `main` untouched. `studio-ci.yml` runs on pushes to both +branches, so a merge is verified and not just the PR that preceded it. + +This amends the "trunk-based" line in `studio/CLAUDE.md` rather than leaving the documented workflow +disagreeing with the actual one. Task 0.1 pre-dates the change and went straight into `main`. + +Two operational notes, both learned the hard way while landing #59: + +- `gh pr create` defaults to the repository's default branch. Studio PRs must pass + `--base studio/dev` explicitly. +- **Merge `main` into `studio/dev` regularly.** A conflicted PR is not merely blocked, it is + silently *untested*: GitHub cannot build the merge ref, so no workflow runs at all and the PR + shows no checks rather than a failure. --- diff --git a/docs/studio/plan/PHASE_0.md b/docs/studio/plan/PHASE_0.md index f85d69e..3d7deb9 100644 --- a/docs/studio/plan/PHASE_0.md +++ b/docs/studio/plan/PHASE_0.md @@ -1,4 +1,4 @@ -# Phase 0 — Skeleton and vertical slice +# Phase 0 — Skeleton and vertical slice · **COMPLETE** (2026-08-17) **Goal:** one hardcoded end-to-end path, real from top to bottom, however narrow. @@ -27,7 +27,7 @@ Delivered as above, with two things worth stating plainly rather than leaving im --- -## 0.2 — `studio/schema` v0 · **review gate** +## 0.2 — `studio/schema` v0 · **review gate** · *implemented, awaiting review* (issue #61) - `SciField(unit=…, range=…, provenance=…, cite=…, derived_from=[…])` over `Field(json_schema_extra=…)`. @@ -45,9 +45,15 @@ Delivered as above, with two things worth stating plainly rather than leaving im **Request review explicitly before building on it.** +Delivered as specified, with two divergences recorded in `PROGRESS.md` rather than absorbed +silently: `max_sim_time` is optional (simulated time is already bounded by `schedule.duration_days`, +and a required second bound would need an invented default), and `DilutionRegime.CONSTANT` is +spelled `"constant"` where the model spells it `""` — the only enum value that is not the model's +own string, and one the 0.4 equivalence test must cover explicitly. + --- -## 0.3 — Dependency-graph engine and override semantics +## 0.3 — Dependency-graph engine and override semantics · *done* (issue #66) The mechanism that makes "go back and edit stage 1 without losing your stage 6 choices" correct by construction. It is a data-model problem, not a UI problem. @@ -62,7 +68,7 @@ construction. It is a data-model problem, not a UI problem. --- -## 0.4 — `studio/modelio` seam + `RunSummary` +## 0.4 — `studio/modelio` seam + `RunSummary` · *done* (issue #68) - `to_scenario(RunConfig) -> CoupledScenario` — the single conversion point, and the **only** package permitted to import `coupled`. @@ -77,16 +83,20 @@ construction. It is a data-model problem, not a UI problem. --- -## 0.5 — `studio/science` derivations +## 0.5 — `studio/science` derivations · *done* (issue #64) One cited, tested implementation of each derivation that currently exists several times over. **Consolidate:** -- V₀ and mass → initial concentration — five copies today (`run_ensemble.py:41-46`, - `run_dilution_d1_clean.py:61`, `run_60day.py:37`, `viz/bake_plume_dynamics.py:63`, `run_boxsize.py`), - with two different V₀ values. -- dN/dlogDp and bin edges — four copies, two mid-point expressions - (`run_ensemble.py:146-149` vs `run_dilution_d1_clean.py:588-590`). +- V₀ and mass → initial concentration — **six** copies today, not five: `run_ensemble.py:41-46,95`, + `run_60day.py:37`, `make_rf_runs.py:44`, `viz/bake_plume_dynamics.py:62-63`, + `coupled/run_dilution_d1_clean.py:61,130` (note: at `coupled/`, not `coupled/paper_ensemble/`), + and `run_boxsize.py:43`. Two different V₀ values — a 15 km track and a 30 km one, a factor of two. +- dN/dlogDp and bin edges — four copies, written two ways. **Corrected by 0.5:** the two + expressions (`run_ensemble.py:147-149` vs `coupled/run_dilution_d1_clean.py:587-589`) are + *algebraically identical*, not two conventions — `10**(0.5*(log a + log b)) == sqrt(a*b)` — and + measured agreement on an 80-bin grid is to a few ULP (≤ 7e-16 relative). The consolidation is + still worth doing; the divergence claim was not accurate. **Reuse, do not rewrite:** `coupled/dilution.py:62` `volume_ratio` and `:74` `kdil_from_regime` (both tested); `air_number_density` (`stratchem-jax/config.py:55`); `coupled/aerosol_props.py`; @@ -102,7 +112,7 @@ Avogadro mismatch at the gas/TOMAS seam. Studio inherits it and does not silentl --- -## 0.6 — `studio/runner` + job lifecycle +## 0.6 — `studio/runner` + job lifecycle · *done* (issue #72) `JobRunner` Protocol; `LocalSubprocessRunner` launching `python -m studio.cli.run` with the thread-pinning environment from `launch_parallel.py:26-30`. Lifecycle @@ -115,7 +125,7 @@ input file. `run_coupled` prints rather than logs, so stdout is captured as the --- -## 0.7 — Golden-file harness · *do this early* +## 0.7 — Golden-file harness · *done* (measurement #70/#74, harness #79) **First task is a measurement, not an assertion.** Re-run two archived cases at today's submodule SHAs and record the observed per-quantity deviation in diff --git a/pyproject.toml b/pyproject.toml index cb697dd..76e6642 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -51,6 +51,7 @@ studio = [ # studio linting/typing; separate from `dev` so the model's suite is unaffected studio-dev = [ "ruff>=0.5", + "httpx2>=0.1", # starlette's TestClient dependency; plain httpx is deprecated for it "mypy>=1.10", "black>=24.3", "hypothesis>=6.100", # property tests for unit round-trips across the valid T-p range @@ -59,6 +60,11 @@ studio-dev = [ # the models themselves live in the tuvx-jax / stratchem-jax / tomas-jax git submodules # (put on sys.path by coupled/model_bridge.py and coupled/tomas_bridge.py); this project # installs the coupling layer + shared dependencies. +[project.scripts] +# The scripted path is first-class (ADR-002): a sweep launched from a terminal goes through the same +# schema, resolver, store and runner as one launched from the web. +plume-studio = "studio.cli.main:app" + [tool.setuptools.packages.find] where = ["."] include = ["coupled*", "studio*"] @@ -79,8 +85,11 @@ markers = [ # ---------------------------------------------------------------------------------------------- [tool.ruff] line-length = 100 -target-version = "py311" -src = ["studio"] +# Studio's floor is 3.12, not the model's >=3.10: the lockfile is compiled for 3.12, CI runs 3.12, +# and numpy's bundled type stubs use 3.12-only `type` statements, so mypy cannot check this package +# against anything older. All three tools state the same floor deliberately. +target-version = "py312" +src = ["."] extend-exclude = ["studio/web"] [tool.ruff.lint] @@ -98,14 +107,17 @@ ignore = ["ANN401"] [tool.ruff.lint.per-file-ignores] "studio/tests/**" = ["ANN"] +# B008 flags function calls in argument defaults, which is exactly how typer declares options. +# Scoped to the CLI rather than disabled globally, where it catches real mutable-default bugs. +"studio/cli/main.py" = ["B008"] [tool.black] line-length = 100 -target-version = ["py311"] +target-version = ["py312"] extend-exclude = "studio/web" [tool.mypy] -python_version = "3.11" +python_version = "3.12" # --strict is applied to studio/schema and studio/science on the command line and in CI; the settings # here are the baseline for everything else under studio/. files = ["studio"] @@ -118,5 +130,19 @@ check_untyped_defs = true [[tool.mypy.overrides]] # The model is untyped and is not ours to annotate; studio/modelio is the only importer (ADR-001). -module = ["coupled.*", "config", "tomas_jax.*", "tuvx_photolysis.*"] +# `follow_imports = "silent"` matters as much as ignore_missing_imports: once studio/modelio imports +# `coupled`, mypy would otherwise report errors from the MODEL's source (its own untyped deps -- +# yaml, scipy, the flat `aerosol` module), which are not Studio's to fix and would make the check +# unusable. Studio's own use of those imports is still checked. +module = [ + "coupled.*", + "config", + "aerosol", + "background_aerosol_distribution", + "tomas_jax.*", + "tuvx_photolysis.*", + "yaml", + "scipy.*", +] ignore_missing_imports = true +follow_imports = "silent" diff --git a/studio/CLAUDE.md b/studio/CLAUDE.md index 3471074..79a198d 100644 --- a/studio/CLAUDE.md +++ b/studio/CLAUDE.md @@ -36,13 +36,13 @@ looks right, presented with the same confidence as a computed result, is worse t - **No shortcuts around the main code path.** Do not special-case tests, do not bypass validation, do not stub the model to make a UI demo work. If a demo needs fake data it is clearly labelled synthetic and lives in a fixture. -- **Type hints everywhere.** `mypy --strict` on `studio/schema` and `studio/science`. +- **Type hints everywhere.** `mypy --strict` on `studio/schema`, `studio/science` and `studio/resolve`. - **Known bugs are issues, not TODO comments.** If a TODO is unavoidable it references an issue number. ## Package boundaries — enforced by tests -- `studio/schema` and `studio/science` import nothing from the API, the database, the web layer, or - `coupled`. They must be usable from a bare Python session. Importing `coupled` pulls in JAX. +- `studio/schema`, `studio/science` and `studio/resolve` import nothing from the API, the database, + the web layer, or `coupled`. They must be usable from a bare Python session. Importing `coupled` pulls in JAX. - **`studio/modelio` is the only package permitted to import `coupled`.** - Enforced by `studio/tests/unit/test_import_boundaries.py`, both at runtime and statically. @@ -73,7 +73,7 @@ cp studio/.env.example .env # then edit; .env i ```bash pytest studio/tests -m tier_a # fast; what CI runs pytest studio/tests -m tier_b # full-case golden reproduction; nightly/manual -mypy --strict studio/schema studio/science +mypy --strict studio/schema studio/science studio/resolve ruff check studio/ && black --check studio/ ``` @@ -100,17 +100,41 @@ just documented in [`CAVEATS.md`](../docs/studio/CAVEATS.md): ## Git workflow -- Trunk-based, short-lived branches: `feat/-`, `fix/…`, `docs/…`, `chore/…`. -- Every branch corresponds to an issue. Labels: `phase-0`…`phase-n`, `science`, `blocking`, - `architecture`, `frontend`, `backend`, `data`, `testing`, `docs`; a milestone per phase. +**`studio/dev` is the integration branch. Studio PRs target it, never `main`.** Branch off +`studio/dev`, and pass the base explicitly — `gh pr create --base studio/dev` — because `gh` +defaults to the repository's default branch, which is `main`. + +``` +main ────●────────────●──────────● viz + model work, plus batched Studio merges + \ \ / +studio/dev ●──●──●────────●──●──● integration; CI runs on pushes here + \ \ \ + task 0.2 ─● \ \ one task → one branch → one PR → studio/dev + task 0.3 ────● \ + task 0.4 ────────────────────● +``` + +- `studio/dev` → `main` at phase boundaries, or sooner when something there is needed by the model + side. **Merge `main` into `studio/dev` regularly** — the model and viz work moves independently, + and a long-lived branch that never pulls is how you get a conflicted merge nobody wants to do. + A conflicted PR is also silently untested: GitHub cannot build the merge ref, so no workflow runs + at all. +- Task 0.1 pre-dates this and went straight into `main` (#59). Everything from 0.2 on goes through + `studio/dev`. +- Short-lived task branches: `feat/-`, `fix/…`, `docs/…`, `chore/…`. +- Every branch corresponds to an issue. Labels: `studio` plus `science`, `blocking`, `architecture`, + `frontend`, `backend`, `data`, `testing`, `docs`. **Do not use the `phase-N` labels** — those are + the coupled model's phases (issues #11–#28), not Studio's; name the Studio phase in the text. - Conventional commits. Small, coherent commits; no "wip" on shared branches. - **One task, one PR.** No scope creep. -- **Model-side changes to `coupled/` go in their own PR**, with their own tests — never buried inside - an app feature. -- Annotated tag at each phase completion (`v0.1.0-phase0`) with release notes. +- **Model-side changes to `coupled/` go in their own PR against `main`**, with their own tests — + never buried inside an app feature, and never routed through `studio/dev`. +- Annotated tag at each phase completion (`v0.1.0-phase0`) with release notes, cut from `main` after + the phase's `studio/dev` → `main` merge. ## PR checklist +- [ ] Base branch is `studio/dev` (not `main`) - [ ] Linked issue; scope matches - [ ] New/changed schema fields carry unit, range, description, default, provenance - [ ] Any new `[ASSUMPTION]` added to `docs/studio/ASSUMPTIONS.md` diff --git a/studio/__init__.py b/studio/__init__.py index 96dc3d9..197d7c1 100644 --- a/studio/__init__.py +++ b/studio/__init__.py @@ -8,10 +8,10 @@ Package boundaries -- enforced by ``studio/tests/unit/test_import_boundaries.py``: -* ``studio.schema`` and ``studio.science`` must import NOTHING from the API, the database, the web - layer, or ``coupled``. Importing the model (``coupled.driver`` / ``coupled.tomas_bridge``) pulls - in JAX and TOMAS, which an API validating a form on every keystroke cannot afford. Both packages - must be usable from a bare Python session. +* ``studio.schema``, ``studio.science`` and ``studio.resolve`` must import NOTHING from the API, the + database, the web layer, or ``coupled``. Importing the model (``coupled.driver`` / + ``coupled.tomas_bridge``) pulls in JAX and TOMAS, which an API resolving a config on every + keystroke cannot afford. All three must be usable from a bare Python session. * ``studio.modelio`` is the ONLY package permitted to import ``coupled``. It is the seam that would be severed if Studio were ever extracted to its own repository. diff --git a/studio/api/__init__.py b/studio/api/__init__.py index bbcf6be..9889688 100644 --- a/studio/api/__init__.py +++ b/studio/api/__init__.py @@ -35,4 +35,6 @@ from __future__ import annotations -__all__: list[str] = [] +from studio.api.app import create_app + +__all__ = ["create_app"] diff --git a/studio/api/app.py b/studio/api/app.py new file mode 100644 index 0000000..54109d3 --- /dev/null +++ b/studio/api/app.py @@ -0,0 +1,277 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The FastAPI application: the web half of the Phase-0 slice. + +Every endpoint here is a thin shell over code the CLI already uses -- the schema, the resolver, +``studio.service``, the store. The API adds HTTP and progress streaming and nothing else, which is +what makes "identical rows from either path" (ADR-002) true rather than aspirational. + +**Progress is pushed, not polled** (spec 7.3). ``/api/events/runs/{id}`` is a Server-Sent Events +stream; it works without a broker because this process owns the worker pool (ADR-007). Job state is +read from the **database** rather than the pool, so the stream is correct for a run this process did +not submit and survives a restart of the one that did. + +**Auth is delegated to a reverse proxy** and there is no user model (ASSUMPTION-3, BLOCKING-2). This +is also the first component in the repository that serves anything, so it is the first place the +private-repo / public-page boundary can be enforced by code: it binds to localhost by default, and +nothing here writes to ``gh-pages`` or any public location. +""" + +from __future__ import annotations + +import asyncio +import json +import os +from collections.abc import AsyncIterator +from pathlib import Path +from typing import Any + +from fastapi import FastAPI, HTTPException +from fastapi.responses import FileResponse, HTMLResponse +from pydantic import BaseModel, Field +from sqlalchemy import select +from sse_starlette.sse import EventSourceResponse + +from studio.resolve import InconsistentConfigError, ResolvedConfig, resolve +from studio.runner import LocalSubprocessRunner +from studio.schema import RunConfig, run_config_json_schema +from studio.service import create_pending_run, finalise, prepare, submit +from studio.store import create_run_set, session_scope +from studio.store.models import JobRow, RunRow, RunSummaryRow + +#: Where runs, artefacts and the database live. Overridable for tests and for a deployment that +#: keeps its data somewhere other than the working directory. +STUDIO_HOME_ENV = "STUDIO_HOME" + +#: How often the SSE stream re-reads job state. 0.5 s is well under human reaction time and far +#: above the cost of one indexed SELECT; the alternative -- pushing from the worker thread -- would +#: only work for runs this process submitted. +_POLL_INTERVAL_S = 0.5 + +_STATIC = Path(__file__).resolve().parent / "static" + + +class SubmitRequest(BaseModel): + """A configuration to run. Partial: anything omitted takes the schema's default.""" + + config: dict[str, Any] = Field(default_factory=dict) + label: str = "" + + +class ResolveRequest(BaseModel): + config: dict[str, Any] = Field(default_factory=dict) + + +def create_app(home: Path | None = None, database: str | None = None) -> FastAPI: + """Build the application. + + A factory rather than a module-level singleton so tests get an isolated home and database + without monkeypatching, and so a deployment can run two instances against different data. + """ + root = Path(home or os.environ.get(STUDIO_HOME_ENV, "var/studio")) + factory, store = prepare(database, root / "runs") + runner = LocalSubprocessRunner(root / "runs" / "jobs") + + app = FastAPI( + title="Plume Studio", + version="0.1.0", + summary="Configure, run and compare coupled SAI plume box-model simulations.", + ) + app.state.factory = factory + app.state.store = store + app.state.runner = runner + + @app.get("/", response_class=HTMLResponse, include_in_schema=False) + def index() -> HTMLResponse: + return HTMLResponse((_STATIC / "index.html").read_text(encoding="utf-8")) + + @app.get("/api/schema") + def get_schema() -> dict[str, Any]: + """The JSON Schema the UI generates its form from (ADR-002). + + Nothing in the UI may invent a field that does not exist here, which is why the form is + driven by this rather than by a hand-written list that would drift. + """ + return run_config_json_schema() + + @app.post("/api/config/resolve") + def resolve_config(request: ResolveRequest) -> dict[str, Any]: + """Apply the derivations and return the resolved config, its identity, and any stale fields. + + Cheap by construction: no model import, no JAX (see the import-boundary tests), so the UI + can call this on every edit. + """ + try: + resolved = resolve(RunConfig.model_validate(request.config)) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + return { + "config": resolved.config.model_dump(mode="json"), + "config_hash": resolved.config.config_hash(), + "stale_fields": list(resolved.stale_fields), + "derived": { + "plume_volume_cm3": resolved.config.injection.plume_volume_cm3, + "so2_initial_pptv": resolved.config.injection.so2_initial_pptv, + }, + } + + @app.post("/api/runs", status_code=202) + async def create_run_endpoint(request: SubmitRequest) -> dict[str, Any]: + """Submit a run and return immediately with its identity. + + 202, not 200: the run has been accepted and is not finished. The client follows + ``/api/events/runs/{id}`` for progress rather than holding a request open for four minutes. + """ + try: + resolved: ResolvedConfig = resolve(RunConfig.model_validate(request.config)) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + try: + with session_scope(factory) as session: + run_set = create_run_set(session, label=request.label or "web") + run_id = create_pending_run( + factory, run_set=run_set, config=resolved, label=request.label + ) + job_id, runner_job_id = submit( + factory, runner, run_id=run_id, config=resolved, label=request.label + ) + except InconsistentConfigError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + + # Finalising blocks on the subprocess, so it goes to a worker thread rather than the event + # loop: a four-minute await here would stall every other request, including the SSE stream + # reporting on this very run. + asyncio.get_running_loop().run_in_executor( + None, + lambda: finalise( + factory, + store, + runner, + run_id=run_id, + job_id=job_id, + runner_job_id=runner_job_id, + ), + ) + return { + "run_id": run_id, + "config_hash": resolved.config.config_hash(), + "state": "queued", + } + + @app.get("/api/runs") + def list_runs(limit: int = 50) -> list[dict[str, Any]]: + """Recent runs, newest first.""" + with session_scope(factory) as session: + rows = session.scalars( + select(RunRow).order_by(RunRow.created_at.desc()).limit(limit) + ).all() + return [_run_brief(session, row) for row in rows] + + @app.get("/api/runs/{run_id}") + def get_run(run_id: str) -> dict[str, Any]: + with session_scope(factory) as session: + run = session.get(RunRow, run_id) + if run is None: + raise HTTPException(status_code=404, detail=f"no run {run_id!r}") + detail = _run_brief(session, run) + detail["provenance"] = run.provenance + detail["artifacts"] = [ + { + "kind": artifact.kind, + "size_bytes": artifact.size_bytes, + "sha256": artifact.sha256, + } + for artifact in run.artifacts + ] + detail["transitions"] = [ + {"state": t.state, "at": t.at.isoformat(), "detail": t.detail} + for job in run.jobs + for t in job.transitions + ] + return detail + + @app.get("/api/runs/{run_id}/summary") + def get_summary(run_id: str) -> dict[str, Any]: + """The RunSummary. Comparison views and figures read this, never the raw npz (ADR-004).""" + with session_scope(factory) as session: + row = session.get(RunSummaryRow, run_id) + if row is None: + raise HTTPException(status_code=404, detail=f"run {run_id!r} has no summary yet") + return dict(row.summary) + + @app.get("/api/runs/{run_id}/artifacts/{kind}") + def get_artifact(run_id: str, kind: str) -> FileResponse: + """Serve a stored artefact from the store, by kind.""" + from studio.store import artifact_for + + with session_scope(factory) as session: + row = artifact_for(session, run_id, kind) + if row is None: + raise HTTPException(status_code=404, detail=f"run {run_id!r} has no {kind!r}") + path, media_type, filename = row.path, row.content_type, Path(row.path).name + return FileResponse(store.open_path(path), media_type=media_type, filename=filename) + + @app.get("/api/events/runs/{run_id}") + async def stream_run(run_id: str) -> EventSourceResponse: + """Progress over SSE, read from the database rather than the worker pool. + + From the database on purpose: the stream is then correct for a run submitted by the CLI or + by a previous instance of this process, and does not depend on the job being in *this* + pool's memory. + """ + + async def events() -> AsyncIterator[dict[str, str]]: + last: str | None = None + while True: + with session_scope(factory) as session: + run = session.get(RunRow, run_id) + if run is None: + yield {"event": "error", "data": json.dumps({"detail": "unknown run"})} + return + state = run.jobs[-1].state if run.jobs else "pending" + payload = _run_brief(session, run) + if state != last: + yield {"event": "state", "data": json.dumps(payload)} + last = state + if state in {"succeeded", "failed", "cancelled", "terminated_on_limit"}: + return + await asyncio.sleep(_POLL_INTERVAL_S) + + return EventSourceResponse(events()) + + return app + + +def _run_brief(session: Any, run: RunRow) -> dict[str, Any]: + """The shape every endpoint returns for a run. One function, so they cannot disagree.""" + summary = session.get(RunSummaryRow, run.id) + job: JobRow | None = run.jobs[-1] if run.jobs else None + return { + "run_id": run.id, + "label": run.label, + "config_hash": run.config_hash, + "created_at": run.created_at.isoformat(), + "reproducible": run.reproducible, + "state": job.state if job else "pending", + "exit_code": job.exit_code if job else None, + "detail": job.detail if job else "", + "termination": summary.termination if summary else None, + "headline": ( + { + "final_so2_pptv": summary.final_so2_pptv, + "peak_h2so4_pptv": summary.peak_h2so4_pptv, + "peak_number_cm3": summary.peak_number_cm3, + "final_surface_area": summary.final_surface_area, + "flags": summary.flags, + } + if summary + else None + ), + } + + +#: Module-level app for ``uvicorn studio.api.app:app``. Built from the environment, so a deployment +#: needs no code; tests call ``create_app`` with their own home instead. +app = create_app() + +__all__ = ["STUDIO_HOME_ENV", "app", "create_app"] diff --git a/studio/api/static/index.html b/studio/api/static/index.html new file mode 100644 index 0000000..6c5b6d6 --- /dev/null +++ b/studio/api/static/index.html @@ -0,0 +1,309 @@ + + + + + +Plume Studio + + + + +
+

Plume Studio

+

Configure, run and inspect coupled SAI plume box-model simulations. + Phase 0 slice — a subset of the schema, the whole pipeline.

+
+ +
+
+

Configure

+
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+ + +
+ +
+
+
Plume volume V₀
—
+
Initial SO₂
—
+
+

—

+

Derived by the server, not the form — the same resolver the CLI uses. + Identity is the hash; two runs differing only in label are one computation.

+
+

+
+ +
+

Runs

+

No runs yet.

+
+
+
+ + + + diff --git a/studio/cli/main.py b/studio/cli/main.py new file mode 100644 index 0000000..8673f31 --- /dev/null +++ b/studio/cli/main.py @@ -0,0 +1,221 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``plume-studio`` -- the scripted path to everything the UI can do. + +**Scripted ensembles must not require the browser** (ADR-002). This is not a convenience wrapper +over the API: it goes through the same schema, resolver, store and runner, so a sweep launched from +a terminal and one launched from the web produce identical rows and identical provenance. + +Verbs stay recognisable to anyone who has used the existing runners, which share a +``plan | one | run `` shape (``run_ensemble.py:174``): + + plume-studio run config.yaml --out runs/ # one run, end to end + plume-studio sweep sweep.yaml --plan # expand axes, print N, submit NOTHING + plume-studio sweep sweep.yaml --out runs/ # expand and submit + plume-studio status # persisted state, from any process + +``--plan`` exists because deciding to spend 810 x 4.6 minutes should take a second command. It is +the same reflex the existing runners encode, and it prints what would run without creating a row. + +``status`` reads from the database rather than from memory, which is the point: job state outlives +the process that submitted it (ADR-007), so a crashed CLI does not orphan a running job. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +import typer +import yaml + +from studio.resolve import ResolvedConfig, resolve +from studio.schema import RunConfig, RunSet +from studio.service import prepare, submit_and_record +from studio.store import ( + create_db_engine, + create_run_set, + database_url, + session_factory, + session_scope, +) + +app = typer.Typer( + add_completion=False, + no_args_is_help=True, + help="Configure, run and inspect coupled SAI plume box-model simulations.", +) + + +def _load_mapping(path: Path) -> dict[str, Any]: + """Read a YAML or JSON document, or exit with a message naming the file. + + YAML by suffix, not by sniffing: a file called ``.json`` that happens to parse as YAML is still + a mistake worth reporting. + """ + if not path.is_file(): + typer.echo(f"[studio] no such file: {path}", err=True) + raise typer.Exit(code=2) + text = path.read_text(encoding="utf-8") + try: + data = json.loads(text) if path.suffix.lower() == ".json" else yaml.safe_load(text) + except (json.JSONDecodeError, yaml.YAMLError) as exc: + typer.echo( + f"[studio] {path} is not valid {path.suffix.lstrip('.') or 'YAML'}: {exc}", err=True + ) + raise typer.Exit(code=2) from exc + if not isinstance(data, dict): + typer.echo(f"[studio] {path} must contain a mapping, got {type(data).__name__}", err=True) + raise typer.Exit(code=2) + return data + + +def _resolved_config(path: Path) -> ResolvedConfig: + """Load a RunConfig document and resolve its derived fields.""" + try: + return resolve(RunConfig.model_validate(_load_mapping(path))) + except ValueError as exc: + typer.echo(f"[studio] {path} is not a valid configuration:\n{exc}", err=True) + raise typer.Exit(code=2) from exc + + +def _run_set(path: Path) -> RunSet: + try: + return RunSet.model_validate(_load_mapping(path)) + except ValueError as exc: + typer.echo(f"[studio] {path} is not a valid sweep:\n{exc}", err=True) + raise typer.Exit(code=2) from exc + + +@app.command() +def run( + config: Path = typer.Argument(..., help="RunConfig as YAML or JSON"), + out: Path = typer.Option(Path("var/studio/runs"), "--out", help="where jobs and artefacts go"), + label: str = typer.Option( + "", "--label", help="human-readable name; identity is still the hash" + ), + wait: bool = typer.Option(True, "--wait/--no-wait", help="block until the run finishes"), + database: str | None = typer.Option(None, "--database", help="override STUDIO_DATABASE_URL"), + dry_run: bool = typer.Option( + False, "--dry-run", help="resolve and print the identity; touch nothing" + ), +) -> None: + """Run one configuration, end to end.""" + resolved = _resolved_config(config) + typer.echo(f"[studio] config hash {resolved.config.config_hash()}") + typer.echo(f"[studio] initial SO2 {resolved.config.injection.so2_initial_pptv:.6e} pptv") + if dry_run: + typer.echo("[studio] --dry-run: nothing written, nothing submitted") + return + + from studio.runner import LocalSubprocessRunner + + factory, store = prepare(database, out) + runner = LocalSubprocessRunner(out / "jobs") + try: + with session_scope(factory) as session: + run_set_row = create_run_set(session, label=label or config.stem) + run_id, state = submit_and_record( + factory, store, runner, config=resolved, run_set=run_set_row, label=label, wait=wait + ) + finally: + runner.shutdown() + typer.echo(f"[studio] run {run_id} -> {state}") + if state != "succeeded" and wait: + raise typer.Exit(code=1) + + +@app.command() +def sweep( + sweep_file: Path = typer.Argument(..., help="RunSet as YAML or JSON: base config plus axes"), + out: Path = typer.Option(Path("var/studio/runs"), "--out", help="where jobs and artefacts go"), + plan: bool = typer.Option( + False, "--plan", help="print what would run and submit NOTHING (mirrors run_ensemble plan)" + ), + database: str | None = typer.Option(None, "--database", help="override STUDIO_DATABASE_URL"), +) -> None: + """Expand a sweep into runs, and submit them unless asked to plan.""" + run_set = _run_set(sweep_file) + typer.echo(f"[studio] {run_set.size()} run(s) from {len(run_set.axes)} axis/axes") + expanded = run_set.expand() + if plan: + for index, item in enumerate(expanded): + typer.echo(f" [{index}] {item.label or '(unlabelled)'} {item.config_hash[:12]}") + typer.echo("[studio] --plan: nothing submitted") + return + + from studio.runner import LocalSubprocessRunner + + factory, store = prepare(database, out) + runner = LocalSubprocessRunner(out / "jobs") + failures = 0 + try: + with session_scope(factory) as session: + run_set_row = create_run_set( + session, + label=sweep_file.stem, + axes=run_set.model_dump(mode="json")["axes"], + ) + for item in expanded: + run_id, state = submit_and_record( + factory, + store, + runner, + config=resolve(item.config), + run_set=run_set_row, + label=item.label, + ) + typer.echo(f"[studio] {item.label or run_id} -> {state}") + failures += state != "succeeded" + finally: + runner.shutdown() + if failures: + typer.echo(f"[studio] {failures} run(s) did not succeed", err=True) + raise typer.Exit(code=1) + + +@app.command() +def status( + run_id: str = typer.Argument(..., help="run id, as printed by `run` or `sweep`"), + database: str | None = typer.Option(None, "--database", help="override STUDIO_DATABASE_URL"), +) -> None: + """Show a run's persisted state. + + Reads the database, not memory: job state outlives the process that submitted it, so this works + from a different shell, after a restart, or while the run is still going. + """ + from studio.store.models import RunRow + + factory = session_factory(create_db_engine(database_url(database))) + with session_scope(factory) as session: + run = session.get(RunRow, run_id) + if run is None: + typer.echo(f"[studio] no run {run_id!r} in this database", err=True) + raise typer.Exit(code=2) + typer.echo(f"run {run.id}") + typer.echo(f"label {run.label or '(none)'}") + typer.echo(f"config {run.config_hash}") + typer.echo(f"created {run.created_at.isoformat()}") + reproducible = {True: "yes", False: "NO — a checkout was dirty", None: "unknown"} + typer.echo(f"reproducible {reproducible[run.reproducible]}") + for job in run.jobs: + typer.echo(f"job {job.id} {job.state} exit={job.exit_code}") + for transition in job.transitions: + typer.echo( + f" {transition.at.isoformat()} {transition.state} {transition.detail}" + ) + for artifact in run.artifacts: + typer.echo( + f"artifact {artifact.kind:10s} {artifact.size_bytes:>10,d} B {artifact.path}" + ) + + +def main() -> int: + app() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/studio/cli/run.py b/studio/cli/run.py new file mode 100644 index 0000000..33b9f32 --- /dev/null +++ b/studio/cli/run.py @@ -0,0 +1,70 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``python -m studio.cli.run `` -- the subprocess entry point. + +Deliberately thin. ``run_coupled`` is a library call with no ``__main__`` of its own (BLOCKING-3), +so something has to be the process that ``LocalSubprocessRunner`` launches, and this is it. All the +work is in ``studio.modelio.execute``; everything here is argument handling and exit codes. + +Exit codes are the runner's only structured signal, so they mean something specific: + +* ``0`` -- the model ran and the outputs were written. +* ``2`` -- the input could not be read or validated. Nothing was run. +* ``1`` -- the model raised. The traceback is on stderr, which the runner captures in full so the + failure can be diagnosed **without re-running it**. + +Nothing is caught and turned into a plausible result. A run that fails must look failed. +""" + +from __future__ import annotations + +import argparse +import sys +import traceback +from pathlib import Path + +from studio.resolve import ResolvedConfig + +#: Exit code for a bad or unreadable input, distinguished from a model failure so the runner can +#: tell "we never started" from "it broke". +EXIT_BAD_INPUT = 2 +EXIT_MODEL_FAILURE = 1 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="python -m studio.cli.run", + description="Run one resolved Plume Studio configuration and write its outputs.", + ) + parser.add_argument( + "input", type=Path, help="resolved config JSON (a serialised ResolvedConfig)" + ) + parser.add_argument( + "outdir", type=Path, help="directory to write state.npz and summary.json into" + ) + args = parser.parse_args(argv) + + try: + config = ResolvedConfig.model_validate_json(args.input.read_text(encoding="utf-8")) + config.require_consistent() + except Exception as exc: + print(f"[studio] cannot run {args.input}: {type(exc).__name__}: {exc}", file=sys.stderr) + return EXIT_BAD_INPUT + + # Imported here, not at module scope: it pulls in the model and therefore JAX, and a bad input + # should fail in milliseconds rather than after a second of imports. + from studio.modelio.execute import run_and_write + + try: + written = run_and_write(config, args.outdir) + except Exception: + traceback.print_exc() + return EXIT_MODEL_FAILURE + + for name, path in written.items(): + print(f"[studio] wrote {name}: {path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/modelio/__init__.py b/studio/modelio/__init__.py index 8f5a26e..f5675b3 100644 --- a/studio/modelio/__init__.py +++ b/studio/modelio/__init__.py @@ -5,31 +5,66 @@ THE ONLY PACKAGE PERMITTED TO IMPORT ``coupled`` (ADR-001). If Studio is ever extracted to its own repository, this is the seam to sever. -Contents (task 0.4, not yet implemented): +* ``scenario.py`` -- ``to_scenario(RunConfig) -> CoupledScenario``, the single conversion point and + therefore the single place unit conversion or name translation happens (ADR-003). +* ``summary.py`` -- ``RunSummary``, the versioned reduction of a run. Comparison plots and figures + read this; they never read the raw npz (ADR-004). It does not import ``coupled`` -- it only reads + arrays -- but reading the model's output format is this package's job. -* ``to_scenario(RunConfig) -> CoupledScenario`` -- the single conversion point, and therefore the - single place unit conversion happens (ADR-003). -* ``RunSummary`` -- the versioned, queryable reduction of a run: scalar time series, final size - distribution, integrated diagnostics, termination reason, flags, conservation residuals. - Comparison plots and figures read this; they never read the raw npz (ADR-004). - -The equivalence test is the point of this package: ``to_scenario`` applied to a ``RunConfig`` -describing case ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` must produce a ``CoupledScenario`` +The equivalence test is the point of ``scenario.py``: ``to_scenario`` applied to a ``RunConfig`` +describing case ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` produces a ``CoupledScenario`` FIELD-FOR-FIELD IDENTICAL to ``run_ensemble.build_scenario()`` for that case. That proves the schema -is a faithful superset before anything is run. +is a faithful superset before anything is run, which is cheaper and sharper than discovering it from +a diverging result days later. Traps this package exists to contain (see docs/studio/CAVEATS.md), each an assertion here: * ``state.npz`` carries BOTH dry and wet quantities: ``dp_mid_um`` / ``dNdlogDp`` are dry; - ``SA`` / ``radius_cm`` are wet. Every RunSummary array declares its basis. + ``SA`` / ``radius_cm`` are wet. Every RunSummary series declares its basis. * Never reconstruct the time axis as ``i * DT``. Outer intervals snap to the terminator, so the mean step is ~592 s against a nominal 600 s -- ~1.4% drift, about 0.5 days by day 36. Use stored ``t``. * Any resampling is UNIFORM in time. A coarsening grid aliases morning particle-number spikes by up to 8x. * Index gas species by NAME using the npz's own ``species`` list, never by position. (An existing analysis script hardcodes ``_SO2, _SO3, _H2SO4 = 32, 34, 35``; that is the failure to avoid.) + +**``to_scenario`` is deliberately NOT re-exported here.** Import it as +``from studio.modelio.scenario import to_scenario``. Re-exporting it would make every comparison +view that reads a ``RunSummary`` pay for the model it is not using. + +Measured, because the cost is not where one would guess: importing ``studio.modelio.scenario`` +takes ~0.13 s and pulls in no JAX at all. The first ``to_scenario()`` CALL takes ~1.05 s, because +``CoupledScenario.__post_init__`` imports ``coupled.tomas_bridge`` to validate ``background_dist`` +(``coupled_scenario.py:196``) and that is what loads JAX. Subsequent calls are free. So the expense +belongs to constructing a scenario, not to importing this package -- which is precisely why an API +that validates a form on every keystroke must stay on the schema side of this seam. Task 0.8 tracks +making that import lazy in the model. """ from __future__ import annotations -__all__: list[str] = [] +from studio.modelio.summary import ( + SECONDS_PER_DAY, + SUMMARY_SCHEMA_VERSION, + Basis, + ConservationCheck, + RunSummary, + Series, + SizeDistribution, + SummaryFlag, + TerminationReason, + summarise_state_npz, +) + +__all__ = [ + "SECONDS_PER_DAY", + "SUMMARY_SCHEMA_VERSION", + "Basis", + "ConservationCheck", + "RunSummary", + "Series", + "SizeDistribution", + "SummaryFlag", + "TerminationReason", + "summarise_state_npz", +] diff --git a/studio/modelio/execute.py b/studio/modelio/execute.py new file mode 100644 index 0000000..750eb0b --- /dev/null +++ b/studio/modelio/execute.py @@ -0,0 +1,150 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Running the model and writing what it produced. + +Lives at the seam because it calls ``run_coupled``. The runner (``studio/runner``) knows about +processes, limits and lifecycle; this module knows about the model. Neither imports the other's +concerns. + +Two things about the model shape the design here, both from BLOCKING-3: + +* ``run_coupled`` is a **library call with no ``__main__``**. It returns arrays in memory and writes + nothing. So the caller writes the npz, and ``studio/cli/run.py`` exists to be the process. +* It **prints** its diagnostics rather than logging them, so stdout is the run's log stream and the + runner captures it as one. + +The npz key set mirrors ``run_ensemble.py:150-156``, which is the canonical one every existing +figure script reads. The size-distribution reduction, though, goes through ``studio.science`` rather +than being inlined a fifth time -- that consolidation is exactly what task 0.5 was for. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import numpy as np + +from studio.modelio.scenario import to_scenario +from studio.modelio.summary import RunSummary, TerminationReason, summarise_state_npz +from studio.resolve import ResolvedConfig +from studio.science import bin_midpoints_um, dn_dlogdp + +#: Tolerance for "the run reached the end of its requested duration", in seconds. One output step is +#: 600 s nominal, so half a step is comfortably inside the noise of a completed run and far outside +#: an early stop. +_COMPLETION_TOLERANCE_S = 300.0 + + +def run_and_write(config: ResolvedConfig, out_dir: Path) -> dict[str, Path]: + """Run the model and write ``state.npz`` and ``summary.json`` into ``out_dir``. + + Returns the paths written. Raises whatever the model raises: a failed run is a failed run, and + the runner records the traceback from stderr rather than this function inventing a result. + """ + from coupled.driver import run_coupled + from coupled.tomas_bridge import _bad + + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + scenario = to_scenario(config) + + requested_s = float(config.config.schedule.duration_days) * 86400.0 + stop_condition = _max_sim_time_stop(config) + + times, states, aerosol, final_state, size_dist, photolysis = run_coupled( + scenario, + return_aerosol=True, + return_state=True, + return_size_dist=True, + return_photolysis=True, + stop_condition=stop_condition, + ) + + from config import IDX, air_number_density # the model's flat modules, via the seam + + edges_um = _bad._xk_to_dp_um(np.asarray(final_state.xk)) + state_path = out_dir / "state.npz" + np.savez( + state_path, + t=times, + x=states, + species=list(IDX), + M=air_number_density(scenario.P, scenario.T), + SA=aerosol["SA"], + radius_cm=aerosol["radius_cm"], + h2so4wp=aerosol["h2so4wp"], + particulate_S=aerosol["particulate_S"], + T=aerosol["T"], + n_cm3=size_dist["n_cm3"], + Dp_m=size_dist["Dp_m"], + dp_mid_um=bin_midpoints_um(edges_um), + dNdlogDp=dn_dlogdp(size_dist["n_cm3"], edges_um), + V_ratio=_volume_ratio(times, scenario), + total_n=size_dist["n_cm3"].sum(axis=1), + J_tmid=photolysis["t_mid"], + J=photolysis["J"], + J_equations=photolysis["equations"], + ) + + termination = ( + TerminationReason.COMPLETED + if float(times[-1]) >= requested_s - _COMPLETION_TOLERANCE_S + else TerminationReason.TERMINATED_ON_LIMIT + ) + summary = summarise_state_npz( + state_path, + label=config.config.config_hash()[:12], + config_hash=config.config.config_hash(), + termination=termination, + ) + summary_path = summary.write(out_dir / "summary.json") + return {"state": state_path, "summary": summary_path} + + +def _max_sim_time_stop(config: ResolvedConfig) -> Any: + """A ``stop_condition`` enforcing ``termination.max_sim_time_days``, or ``None``. + + **Single-parameter, taking the model's diagnostics dict.** The model dispatches on the + callback's DECLARED ARITY (``coupled/driver.py``, ``_adapt_stop_condition``): + + =========================== ========================================== + ``def stop(diag)`` the current shape + ``def stop(t1, wet_SA)`` legacy, accepted with a ``DeprecationWarning`` + ``def stop(*args)`` **TypeError** -- it matches both, so it is ambiguous + =========================== ========================================== + + Raising on ``*args`` is right: guessing which shape a variadic callback wanted would be a + silent wrong answer. It does mean the obvious "works with either" spelling is the one that does + not, so the arity here is part of the contract rather than an implementation detail -- see the + tests that pin it. + + Only ``diag["t"]`` is read today. The dict also carries ``SA``, ``N_total`` and every gas + species by name (``diag["gas"]["SO2"]``), which is what makes SO2- or number-based termination + criteria possible when the schema grows them -- the spec's ``termination.criteria[]`` needed + exactly this. + """ + limit_days = config.config.termination.max_sim_time_days + if limit_days is None: + return None + limit_s = float(limit_days) * 86400.0 + + def stop(diag: Mapping[str, Any]) -> bool: + return float(diag["t"]) >= limit_s + + return stop + + +def _volume_ratio(times: np.ndarray, scenario: Any) -> np.ndarray: + """V(t)/V0 for the run, from the model's own tested implementation. + + Reused rather than reimplemented: ``coupled.dilution.volume_ratio`` is tested, and a second copy + in ``studio/science`` is exactly what task 0.5 removed elsewhere. + """ + from coupled import dilution + + return dilution.volume_ratio(times, scenario.dilution_regime) + + +__all__ = ["RunSummary", "run_and_write"] diff --git a/studio/modelio/provenance.py b/studio/modelio/provenance.py new file mode 100644 index 0000000..2959e95 --- /dev/null +++ b/studio/modelio/provenance.py @@ -0,0 +1,219 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""What produced a result: config identity, app version, model version (ADR-006). + +The requirement is that **any figure traces to an exact configuration, model version and input +dataset checksum**. What existed before this module was a hand-composed case ID -- good design for a +fixed factorial, and genuinely useful, but it captures the *axes* rather than the resolved +configuration: everything held fixed across the 810-run ensemble (``ion_pair_rate=30``, +``day_of_year=172``, the whole background composition) is invisible in it, so two ensembles +differing only in a "fixed" value collide. + +This is **the one place Studio shells out to git**, and it is deliberately strict about it: + +* "Not a git checkout" **raises**. An empty SHA in a provenance record is worse than no record at + all, because it looks like an answer (ADR-005). +* **A dirty working tree is recorded and flags the run.** Uncommitted changes mean the SHA does not + describe the code that ran, and that is exactly the case where someone later wants to know. +* Submodule SHAs are read individually. The SANDBOX SHA alone does not pin the model: the three + submodules are what contain it (ADR-001), and a submodule pointer that has moved without a commit + here is invisible in the parent SHA. + +The record is written **before execution begins** -- a run that dies in minute three still has its +provenance -- and is never mutated. An edited config is a new config and a new run (ADR-004). + +Note what this module does NOT do: it does not import ``coupled``. Model *identity* is a question +about the checkout, not about the model's API, so recording it costs nothing. +""" + +from __future__ import annotations + +import subprocess +from datetime import UTC, datetime +from pathlib import Path +from typing import Any, Final + +from pydantic import BaseModel, ConfigDict, Field + +import studio +from studio.resolve import ResolvedConfig + +#: The submodules that, together with the SANDBOX SHA, pin the model exactly (ADR-001). +MODEL_SUBMODULES: Final[tuple[str, ...]] = ("tuvx-jax", "stratchem-jax", "tomas-jax") + +#: Bumped if the record's shape changes, so a stored record is never read under new semantics. +PROVENANCE_SCHEMA_VERSION: Final = "0.1.0" + + +class NotAGitCheckoutError(RuntimeError): + """The repository root is not a git checkout, so the model cannot be pinned. + + Raised rather than recording an empty SHA: a provenance record that cannot say what ran is not a + provenance record, and one that says ``""`` looks like it can. + """ + + +class GitCheckout(BaseModel): + """The state of one checkout: its commit, and whether it was clean.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + path: str + commit: str + #: True when ``git status --porcelain`` was non-empty. The commit then does NOT describe the + #: code that ran, which is precisely when someone will want to know. + dirty: bool + #: The porcelain output, truncated. Enough to see WHAT was uncommitted without storing a diff. + dirty_files: tuple[str, ...] = () + + +class ProvenanceRecord(BaseModel): + """Everything needed to say what produced a result. Written before the run, never mutated.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + schema_version: str = PROVENANCE_SCHEMA_VERSION + recorded_at: datetime + #: Identity of the resolved config (ADR-006). Doubles as the cache key. + config_hash: str + #: The orchestration layer's version. The model has none -- it is pinned by SHA alone. + studio_version: str + #: The SANDBOX checkout: the coupling layer plus the paper pipeline. + sandbox: GitCheckout + #: The three submodules that contain the model itself. + submodules: dict[str, GitCheckout] + #: Input datasets consulted, by identifier -> checksum. Empty in Phase 0: the schema carries no + #: dataset inputs yet. Present and empty rather than omitted, so its absence is never ambiguous. + datasets: dict[str, str] = Field(default_factory=dict) + #: The RESOLVED, post-derivation parameter set -- what the model actually received, not what the + #: user typed. This is the field that makes the record self-contained. + resolved_config: dict[str, Any] + #: Any derived field the user overrode, with the value that was in force. + overrides: dict[str, Any] = Field(default_factory=dict) + + @property + def is_reproducible(self) -> bool: + """True when every checkout was clean, so the SHAs fully describe the code that ran.""" + return not self.sandbox.dirty and not any(sub.dirty for sub in self.submodules.values()) + + @property + def dirty_checkouts(self) -> tuple[str, ...]: + """Names of checkouts with uncommitted changes. Empty when the run is reproducible.""" + names = ["SANDBOX"] if self.sandbox.dirty else [] + names.extend(name for name, sub in sorted(self.submodules.items()) if sub.dirty) + return tuple(names) + + def write(self, path: Path) -> Path: + """Write as JSON. Called before the run starts.""" + path.write_text(self.model_dump_json(indent=2), encoding="utf-8") + return path + + @classmethod + def read(cls, path: Path) -> ProvenanceRecord: + return cls.model_validate_json(path.read_text(encoding="utf-8")) + + +def _git(repo_root: Path, *args: str) -> str: + """Run git in ``repo_root`` and return stdout, or raise. + + Raises: + NotAGitCheckoutError: If git fails for any reason -- not installed, not a checkout, a broken + submodule. All of them mean the same thing here: the code that ran cannot be identified. + """ + try: + proc = subprocess.run( + ["git", *args], + cwd=str(repo_root), + capture_output=True, + text=True, + check=False, + ) + except FileNotFoundError as exc: # git itself is missing + raise NotAGitCheckoutError( + f"git is not available, so the model cannot be pinned for a run in {repo_root}. " + f"Provenance is required (ADR-006); it is not optional metadata." + ) from exc + if proc.returncode != 0: + raise NotAGitCheckoutError( + f"`git {' '.join(args)}` failed in {repo_root}: {proc.stderr.strip() or 'no output'}. " + f"A provenance record with no commit is not a provenance record, so this raises rather " + f"than recording an empty SHA." + ) + return proc.stdout.strip() + + +def describe_checkout(path: Path, *, dirty_file_limit: int = 20) -> GitCheckout: + """The commit and cleanliness of the checkout at ``path``. + + ``git status --porcelain`` rather than ``diff --quiet`` because it also reports untracked + files, and an untracked module that a run imported is exactly the kind of thing that makes a + SHA a lie. + """ + commit = _git(path, "rev-parse", "HEAD") + status = _git(path, "status", "--porcelain") + lines = tuple(line.strip() for line in status.splitlines() if line.strip()) + return GitCheckout( + path=str(path), + commit=commit, + dirty=bool(lines), + dirty_files=lines[:dirty_file_limit], + ) + + +def repository_root() -> Path: + """The SANDBOX root, derived from this file's location rather than the working directory. + + Deliberately not ``Path.cwd()``: a run submitted by an API process started anywhere at all must + still record the checkout that the code came from. + """ + return Path(studio.__file__).resolve().parent.parent + + +def record_for( + config: ResolvedConfig, + *, + repo_root: Path | None = None, + datasets: dict[str, str] | None = None, +) -> ProvenanceRecord: + """Build the record for a config, at submit time. + + Raises: + InconsistentConfigError: If the config has stale overrides. Recording provenance for a + config whose numbers do not follow from each other would give an inconsistent run a + respectable-looking pedigree. + NotAGitCheckoutError: If any checkout cannot be identified. + """ + config.require_consistent() + root = repo_root or repository_root() + submodules = {} + for name in MODEL_SUBMODULES: + path = root / name + if not (path / ".git").exists(): + raise NotAGitCheckoutError( + f"submodule {name} is not checked out at {path} (`git submodule update --init`). " + f"The SANDBOX SHA alone does not pin the model: the three submodules are what " + f"contain it (ADR-001)." + ) + submodules[name] = describe_checkout(path) + return ProvenanceRecord( + recorded_at=datetime.now(UTC), + config_hash=config.config.config_hash(), + studio_version=studio.__version__, + sandbox=describe_checkout(root), + submodules=submodules, + datasets=dict(datasets or {}), + resolved_config=config.config.model_dump(mode="json"), + overrides={path: record.value for path, record in sorted(config.overrides.items())}, + ) + + +__all__ = [ + "MODEL_SUBMODULES", + "PROVENANCE_SCHEMA_VERSION", + "GitCheckout", + "NotAGitCheckoutError", + "ProvenanceRecord", + "describe_checkout", + "record_for", + "repository_root", +] diff --git a/studio/modelio/scenario.py b/studio/modelio/scenario.py new file mode 100644 index 0000000..97b1516 --- /dev/null +++ b/studio/modelio/scenario.py @@ -0,0 +1,128 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunConfig`` -> ``CoupledScenario``. The single conversion point. + +This module is the seam. It is the ONLY place in Studio that imports ``coupled`` (ADR-001), and +therefore the only place where units are converted or names are translated (ADR-003). Everything +above it works in the schema's vocabulary; everything below is the model's. + +The mapping is deliberately dull. Canonical units are the model's native units precisely so this +function does not have to do arithmetic -- almost every field is passed through unchanged, and the +handful of places where something *does* happen are the interesting ones: + +* ``DilutionRegime.CONSTANT`` -> ``""``. The model spells "constant rate, no V(t) curve" as an empty + string, which cannot be a usable dropdown key. The only enum value that is not the model's own. +* ``background.so2_pptv`` is merged into ``dilution_background``, because the model has one field + for "what the plume relaxes toward" and the schema separates the background's SO2 from any other + species overrides. +* ``injection.so2_initial_pptv`` becomes the ``SO2`` entry of ``concentrations``. It is a DERIVED + field, so a config that has not been through ``studio.resolve`` will have ``None`` there -- which + raises here rather than reaching the model as a missing species. + +Fields the schema does not carry are left at the model's own defaults, exactly as +``run_ensemble.build_scenario`` leaves them: ``SA``, ``Yn2o5``, ``opt``, the ``micro_*`` step +controls, the aerosol placement, and ``output_dir``. Adding them to the schema before anything needs +them would be modelling the model rather than the runs. +""" + +from __future__ import annotations + +from typing import Any + +from coupled.coupled_scenario import CoupledScenario, Switches +from studio.resolve import ResolvedConfig +from studio.schema import DilutionRegime, RunConfig + +#: Schema regime -> the model's ``dilution_regime`` string. Only CONSTANT differs; the rest are +#: identical by construction, and the test asserts that rather than trusting it. +_REGIME_TO_MODEL: dict[DilutionRegime, str] = { + DilutionRegime.CONSTANT: "", + DilutionRegime.D1: "D1", + DilutionRegime.D2: "D2", + DilutionRegime.D3: "D3", + DilutionRegime.D5: "D5", + DilutionRegime.BURST: "burst", +} + + +def to_scenario(config: RunConfig | ResolvedConfig) -> CoupledScenario: + """Build the model's input object from a resolved Studio config. + + Args: + config: A ``RunConfig`` whose derived fields have been resolved, or the ``ResolvedConfig`` + that resolved them. Passing the latter is preferred: it carries the stale list, and this + function refuses to convert an inconsistent config. + + Raises: + InconsistentConfigError: If a ``ResolvedConfig`` has stale overrides. Converting one would + hand the model a set of numbers that do not follow from each other, and the result would + look like any other run. + ValueError: If a derived field is still unresolved. The alternative -- defaulting it -- is + exactly the silent fallback ADR-005 forbids. + """ + if isinstance(config, ResolvedConfig): + config.require_consistent() + run = config.config + else: + run = config + + so2_pptv = run.injection.so2_initial_pptv + if so2_pptv is None: + raise ValueError( + "injection.so2_initial_pptv is unresolved. It is a DERIVED field: run the config " + "through studio.resolve.resolve() before converting it, rather than letting the model " + "start with no SO2." + ) + + concentrations: dict[str, float] = {**run.background.gas_pptv, "SO2": so2_pptv} + dilution_background: dict[str, float] = { + "SO2": run.background.so2_pptv, + **run.dilution.background_overrides_pptv, + } + + return CoupledScenario( + T=run.site.temperature_k, + P=run.site.pressure_mbar, + WTR=run.site.h2o_ppmv, + latitude=run.site.latitude_deg, + longitude=run.site.longitude_deg, + day_of_year=run.schedule.day_of_year, + start_utc_hour=run.schedule.start_utc_hour, + days=run.schedule.duration_days, + DT=run.numerics.output_dt_s, + dt_couple=run.numerics.couple_dt_s, + photolysis=run.chemistry.photolysis.value, + tomas_nbins=run.microphysics.n_bins, + background_dist=run.background.aerosol.value, + dilution_regime=_REGIME_TO_MODEL[run.dilution.regime], + dilution_rate=run.dilution.rate_per_s, + dilution_zero_species=tuple(run.dilution.zero_species), + dilution_background=dilution_background, + ion_pair_rate=run.microphysics.ion_pair_rate, + so2_ho2_rate=run.chemistry.so2_ho2_rate, + condensation_alpha=run.microphysics.condensation_alpha, + nucleation_rate_scale=run.microphysics.nucleation_rate_scale, + coag_kernel_scale=run.microphysics.coag_kernel_scale, + switches=Switches( + sulfur=run.switches.sulfur, + nucleation=run.switches.nucleation, + condensation=run.switches.condensation, + coagulation=run.switches.coagulation, + aerosol_to_j=run.switches.aerosol_to_j, + heating_to_t=run.switches.heating_to_t, + dilution=run.switches.dilution, + ), + concentrations=concentrations, + ) + + +def scenario_as_dict(scenario: CoupledScenario) -> dict[str, Any]: + """``CoupledScenario`` -> plain dict, for the equivalence test and for provenance records. + + Thin wrapper over the model's own ``to_dict`` so callers need not import ``coupled`` to compare + or serialise a scenario -- which is the entire point of this package existing. + """ + return scenario.to_dict() + + +__all__ = ["scenario_as_dict", "to_scenario"] diff --git a/studio/modelio/summary.py b/studio/modelio/summary.py new file mode 100644 index 0000000..a21ef69 --- /dev/null +++ b/studio/modelio/summary.py @@ -0,0 +1,320 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSummary`` -- the versioned, queryable reduction of a run. + +Comparison views and figures read this; they never read the raw ``state.npz`` (ADR-004). The npz is +3.6 MB per case, carries 36 gas species and an (n_times, n_bins) size distribution, and knows +nothing about which of its arrays are wet and which are dry. A summary that fixes those three +things is what makes 810 runs comparable without opening 810 archives. + +**Every array declares its basis.** In ``state.npz``, ``dp_mid_um`` and ``dNdlogDp`` are DRY +diameters while ``SA`` and ``radius_cm`` are WET -- the same file, no labelling, and the difference +is a factor of a few in radius at stratospheric humidity. Here it is a required field on every +series, so a plot axis cannot be labelled by guesswork. + +Two more traps encoded rather than documented: + +* **Species are indexed by NAME** from the npz's own ``species`` list. An existing analysis script + hard-codes ``_SO2, _SO3, _H2SO4 = 32, 34, 35``; if the mechanism ever gains a species, that script + silently plots the wrong one. +* **The time axis comes from the stored ``t``**, never from ``i * DT``. Outer steps snap to the + terminator, so the mean step is ~592 s against a nominal 600 s -- about half a day of drift by day + 36. + +This module does not import ``coupled``; it only reads arrays. It lives here because reading the +model's output format is the model seam's job, not because it needs the model. +""" + +from __future__ import annotations + +from enum import StrEnum +from pathlib import Path +from typing import Any + +import numpy as np +from pydantic import BaseModel, ConfigDict, Field + +#: Bumped whenever the reduction changes shape or meaning. Stored in every summary so a comparison +#: view can refuse to plot two runs reduced under different rules rather than plotting them anyway. +SUMMARY_SCHEMA_VERSION = "0.1.0" + +#: Seconds per day, for the time axis. Named rather than inline (studio/CLAUDE.md). +SECONDS_PER_DAY = 86400.0 + + +class Basis(StrEnum): + """Whether a quantity is on a dry or an ambient (wet) basis. Required on every series.""" + + #: Particle without its water. ``dp_mid_um`` and ``dNdlogDp`` in ``state.npz``. + DRY = "dry" + #: Includes condensed water at ambient conditions. ``SA`` and ``radius_cm`` in ``state.npz``. + WET = "wet" + #: Not a particle-size quantity, so the distinction does not apply (gas mixing ratios, T, V/V0). + NOT_APPLICABLE = "not_applicable" + + +class TerminationReason(StrEnum): + """Why a run stopped. Never inferred from the data.""" + + COMPLETED = "completed" + FAILED = "failed" + CANCELLED = "cancelled" + TERMINATED_ON_LIMIT = "terminated_on_limit" + #: No record exists. The archived ensemble predates provenance capture (ADR-006), so a summary + #: built from one of its npz files is UNKNOWN rather than assumed to have completed. + UNKNOWN = "unknown" + + +class SummaryFlag(StrEnum): + """Machine-readable caveats that must travel with the numbers.""" + + #: Reduced from an archived npz with no provenance record: no config hash, no model version. + NO_PROVENANCE_RECORD = "no_provenance_record" + #: Dilution was active, so the box is an open system and sulfur is not expected to be conserved. + OPEN_SYSTEM_DILUTION = "open_system_dilution" + #: The run stopped on a limit rather than converging. Never presented as a converged result. + STOPPED_ON_LIMIT = "stopped_on_limit" + + +class Series(BaseModel): + """One scalar time series, with everything needed to plot it honestly.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + values: tuple[float, ...] + unit: str + basis: Basis + description: str + + +class SizeDistribution(BaseModel): + """The final size spectrum. Dry diameters, and it says so.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + diameter_um: tuple[float, ...] + dn_dlogdp_cm3: tuple[float, ...] + number_cm3: tuple[float, ...] + basis: Basis = Basis.DRY + #: Total number concentration, i.e. sum of the per-bin counts -- not of dN/dlogDp. + total_number_cm3: float + + +class ConservationCheck(BaseModel): + """A budget check, or an explicit statement that it does not apply. + + The honest part is ``status``. In-box sulfur is **not** expected to be conserved when dilution + is on: the box is an open system and entrainment removes plume sulfur while adding background + sulfur. Reporting a large "residual" for a diluting run would be reporting the dilution, and + reporting a small one would mean something had gone wrong. So the check is computed only when + the run is closed, and otherwise says why not -- rather than producing a number that invites a + reader to draw a conclusion from it. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + status: str + #: ``(S_end - S_start) / S_start`` for a closed box; ``None`` when not applicable. + relative_residual: float | None = None + initial_value: float | None = None + final_value: float | None = None + unit: str = "molec cm^-3" + reason: str = "" + + +class RunSummary(BaseModel): + """The reduction of one run. Versioned, and self-describing about basis and provenance.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + schema_version: str = SUMMARY_SCHEMA_VERSION + #: Identity of the config that produced this run (ADR-006). ``None`` for archived runs, which + #: have no provenance record -- flagged rather than left ambiguous. + config_hash: str | None = None + label: str | None = None + termination: TerminationReason = TerminationReason.UNKNOWN + flags: tuple[SummaryFlag, ...] = () + time_days: tuple[float, ...] = () + series: dict[str, Series] = Field(default_factory=dict) + final_size_distribution: SizeDistribution | None = None + sulfur_conservation: ConservationCheck | None = None + + def write(self, path: Path) -> Path: + """Write as JSON next to the run's ``state.npz``.""" + path.write_text(self.model_dump_json(indent=2), encoding="utf-8") + return path + + @classmethod + def read(cls, path: Path) -> RunSummary: + """Read one back, validating it against this version of the schema.""" + return cls.model_validate_json(path.read_text(encoding="utf-8")) + + +#: Gas species reduced to mixing-ratio series, by NAME. Extending this list is the supported way to +#: add a series; indexing by position is how the existing analysis scripts got it wrong. +_GAS_SERIES: tuple[tuple[str, str], ...] = ( + ("SO2", "sulfur dioxide, the injected species"), + ("SO3", "sulfur trioxide, the intermediate"), + ("H2SO4", "gas-phase sulfuric acid, the condensable"), + ("OH", "hydroxyl radical, the oxidant that starts the chain"), + ("HO2", "hydroperoxyl radical"), + ("O3", "ozone"), +) + +#: Aerosol/environment series carried straight through, with the basis each one is actually on. +_DIRECT_SERIES: tuple[tuple[str, str, Basis, str], ...] = ( + ("SA", "um^2 cm^-3", Basis.WET, "aerosol surface area density (wet)"), + ("radius_cm", "cm", Basis.WET, "effective particle radius (wet)"), + ("h2so4wp", "1", Basis.NOT_APPLICABLE, "H2SO4 weight fraction of the aerosol"), + ("particulate_S", "molec cm^-3", Basis.NOT_APPLICABLE, "sulfur held in the particle phase"), + ("T", "K", Basis.NOT_APPLICABLE, "box temperature"), + ("total_n", "cm^-3", Basis.DRY, "total particle number concentration"), + ("V_ratio", "1", Basis.NOT_APPLICABLE, "plume volume expansion V(t)/V0"), +) + +#: Species whose number density IS its sulfur content -- each carries exactly one S atom. Used for +#: the closed-box budget check. ``particulate_S`` is added separately; it is already a sulfur count. +_SULFUR_GAS_SPECIES = ("SO2", "SO3", "H2SO4") + + +def summarise_state_npz( + path: Path, + *, + label: str | None = None, + config_hash: str | None = None, + termination: TerminationReason = TerminationReason.UNKNOWN, +) -> RunSummary: + """Reduce a ``state.npz`` to a :class:`RunSummary`. + + ``termination`` is an argument rather than something inferred from the arrays: the npz records + what happened to the state, not why the loop stopped. A run that hit a wall-clock limit and one + that finished look identical here, and guessing would be the difference between "converged" and + "cut short" (ADR-005). + """ + with np.load(path, allow_pickle=True) as archive: + data = {key: archive[key] for key in archive.files} + + missing = {"t", "x", "species", "M"} - set(data) + if missing: + raise ValueError(f"{path} is missing required arrays {sorted(missing)}") + + # From the STORED t, never i * DT: outer steps snap to the terminator (~592 s vs 600 s nominal). + time_s = np.asarray(data["t"], dtype=np.float64) + air_number_density = float(data["M"]) + species = [str(name) for name in data["species"]] + state = np.asarray(data["x"], dtype=np.float64) + + series: dict[str, Series] = {} + for name, description in _GAS_SERIES: + if name not in species: + continue # a mechanism without this species is not an error, just fewer series + column = state[:, species.index(name)] # BY NAME. Never by position. + series[name] = Series( + values=tuple(column / air_number_density * 1.0e12), + unit="pptv", + basis=Basis.NOT_APPLICABLE, + description=description, + ) + for key, unit, basis, description in _DIRECT_SERIES: + if key in data: + series[key] = Series( + values=tuple(np.asarray(data[key], dtype=np.float64)), + unit=unit, + basis=basis, + description=description, + ) + + flags: list[SummaryFlag] = [] + if config_hash is None: + flags.append(SummaryFlag.NO_PROVENANCE_RECORD) + if termination is TerminationReason.TERMINATED_ON_LIMIT: + flags.append(SummaryFlag.STOPPED_ON_LIMIT) + + diluting = "V_ratio" in data and float(np.max(np.asarray(data["V_ratio"]))) > 1.0 + if diluting: + flags.append(SummaryFlag.OPEN_SYSTEM_DILUTION) + + return RunSummary( + config_hash=config_hash, + label=label, + termination=termination, + flags=tuple(flags), + time_days=tuple(time_s / SECONDS_PER_DAY), + series=series, + final_size_distribution=_final_size_distribution(data), + sulfur_conservation=_sulfur_budget(data, species, state, diluting=diluting), + ) + + +def _final_size_distribution(data: dict[str, Any]) -> SizeDistribution | None: + """The last spectrum. ``dp_mid_um`` and ``dNdlogDp`` are DRY (see the module docstring).""" + if not {"dp_mid_um", "dNdlogDp", "n_cm3"} <= set(data): + return None + diameters = np.asarray(data["dp_mid_um"], dtype=np.float64) + dn_dlogdp = np.asarray(data["dNdlogDp"], dtype=np.float64)[-1] + counts = np.asarray(data["n_cm3"], dtype=np.float64)[-1] + return SizeDistribution( + diameter_um=tuple(diameters), + dn_dlogdp_cm3=tuple(dn_dlogdp), + number_cm3=tuple(counts), + total_number_cm3=float(counts.sum()), + ) + + +def _sulfur_budget( + data: dict[str, Any], species: list[str], state: np.ndarray, *, diluting: bool +) -> ConservationCheck: + """In-box sulfur at the start and end, and a residual only when the box is closed.""" + present = [name for name in _SULFUR_GAS_SPECIES if name in species] + if not present: + return ConservationCheck( + status="not_applicable", + reason="the mechanism carries none of SO2, SO3, H2SO4", + ) + total = np.zeros(state.shape[0], dtype=np.float64) + for name in present: + total = total + state[:, species.index(name)] + if "particulate_S" in data: + total = total + np.asarray(data["particulate_S"], dtype=np.float64) + initial, final = float(total[0]), float(total[-1]) + + if diluting: + return ConservationCheck( + status="not_applicable", + initial_value=initial, + final_value=final, + reason=( + "dilution was active (V(t)/V0 > 1), so the box is an open system: entrainment " + "removes plume sulfur and adds background sulfur. A residual here would be a " + "measure of the dilution, not of conservation. Start and end values are reported " + "so the decay is visible." + ), + ) + if initial == 0.0: + return ConservationCheck( + status="not_applicable", + initial_value=initial, + final_value=final, + reason="no sulfur at t = 0, so a relative residual is undefined", + ) + return ConservationCheck( + status="computed", + relative_residual=(final - initial) / initial, + initial_value=initial, + final_value=final, + reason="closed box (no dilution): total sulfur should be conserved", + ) + + +__all__ = [ + "SECONDS_PER_DAY", + "SUMMARY_SCHEMA_VERSION", + "Basis", + "ConservationCheck", + "RunSummary", + "Series", + "SizeDistribution", + "SummaryFlag", + "TerminationReason", + "summarise_state_npz", +] diff --git a/studio/requirements.lock b/studio/requirements.lock index 0ca74f1..fcc7d44 100644 --- a/studio/requirements.lock +++ b/studio/requirements.lock @@ -22,6 +22,7 @@ anyio==4.14.2 \ --hash=sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494 \ --hash=sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f # via + # httpx2 # sse-starlette # starlette # watchfiles @@ -415,7 +416,13 @@ greenlet==3.5.5 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or h11==0.16.0 \ --hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \ --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86 - # via uvicorn + # via + # httpcore2 + # uvicorn +httpcore2==2.10.0 ; sys_platform != 'emscripten' \ + --hash=sha256:13c0cc3d1919d4f28457f60cd2c2abe04113a8af184ccf1142811beba936f9dc \ + --hash=sha256:7df06cfb34070cae4f7c89be69dc1095eca138e9704ceffb98d25c1912ab6f01 + # via httpx2 httptools==0.8.0 \ --hash=sha256:0770728beb05094c809b98e814edff5fef69d26ad7d21185f2f6d5884a0ba683 \ --hash=sha256:0ea897f0c729581ebf72131a438a7932d9b14efef72d75ada966700cac3caaeb \ @@ -468,6 +475,14 @@ httptools==0.8.0 \ --hash=sha256:f256d6ce930c52ca1cb2a960b7da03548c454e7d28b06059ad41bfe789036ce0 \ --hash=sha256:fe2a4c95aeba2209434e7b31172da572846cae8ca0bf1e7013e61b99fbbf5e72 # via uvicorn +httpx2==2.10.0 \ + --hash=sha256:5e3194a432701e1cc6f69a8b1b2fa199ef907013fede8d9a09a2c5b7b8141a18 \ + --hash=sha256:8741d7329fe2c7885fc9ceb61c8217acfb87a85f75723714b89ebf7ad7196338 + # via sandbox-coupled (pyproject.toml) +httpx2-jsfetch==1.0 ; sys_platform == 'emscripten' \ + --hash=sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60 \ + --hash=sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32 + # via httpx2 hypothesis==6.165.5 \ --hash=sha256:05e7e8288b2f5fbb34a30b45c9df72a5ce9da0d5ce90705c76b28dda75aac984 \ --hash=sha256:0a5004c3fe761b642ca556abf4551bc9a94d190320bcf7ce118cf8b792eaf71c \ @@ -535,7 +550,9 @@ hypothesis==6.165.5 \ idna==3.18 \ --hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \ --hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848 - # via anyio + # via + # anyio + # httpx2 iniconfig==2.3.0 \ --hash=sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730 \ --hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12 @@ -1753,6 +1770,12 @@ starlette==1.6.0 \ # via # fastapi # sse-starlette +truststore==0.10.4 ; sys_platform != 'emscripten' \ + --hash=sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301 \ + --hash=sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981 + # via + # httpcore2 + # httpx2 typer==0.27.1 \ --hash=sha256:53150287edd11baeb4e4722c8e394fcdf8181c0ae89485cba8d25c778d5edd56 \ --hash=sha256:a79bef8469a79c45498e7b814ecf8d603cc7644e9acbd9e19cac0334240b18df @@ -1768,6 +1791,7 @@ typing-extensions==4.16.0 \ # fastapi # flexcache # flexparser + # httpx2 # lineax # mypy # optimistix diff --git a/studio/resolve/__init__.py b/studio/resolve/__init__.py new file mode 100644 index 0000000..253e718 --- /dev/null +++ b/studio/resolve/__init__.py @@ -0,0 +1,63 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The dependency graph and override semantics (task 0.3). + +What makes "go back and edit stage 1 without losing your stage 6 choices" correct by construction. +It is a data-model problem, not a UI problem, so it lives here and the UI merely renders the result. + +Three pieces: + +* ``graph.py`` -- the DAG, built from the schema's ``derived_from`` metadata. Pure graph work: no + values, no derivations, no physics. +* ``registry.py`` -- which function in ``studio/science`` computes which derived field, with the + binding checked against the schema rather than trusted. +* ``resolver.py`` -- resolution in topological order, the auto / user_override / stale states, and + the two explicit ways to settle a stale field. + +**Why a package of its own.** ``studio/schema`` is data and stays free of computation; +``studio/science`` is computation and stays free of the config model. Resolution is the composition +of the two, and giving it a name keeps that layering visible -- schema and science remain usable, +and testable, without it. + +Like schema and science, this package must not import ``coupled``, the API or the database: the API +resolves a config on every keystroke, and a JAX import on that path would be unaffordable. Enforced +by ``studio/tests/unit/test_import_boundaries.py``. +""" + +from __future__ import annotations + +from studio.resolve.graph import CyclicDependencyError, DependencyGraph, schema_derived_fields +from studio.resolve.registry import DERIVATIONS, Derivation, derivation_for +from studio.resolve.resolver import ( + ChangedInput, + InconsistentConfigError, + OverrideRecord, + ResolvedConfig, + StaleField, + accept_derived, + apply_change, + downstream_of, + keep_override, + resolve, + set_override, +) + +__all__ = [ + "DERIVATIONS", + "ChangedInput", + "CyclicDependencyError", + "DependencyGraph", + "Derivation", + "InconsistentConfigError", + "OverrideRecord", + "ResolvedConfig", + "StaleField", + "accept_derived", + "apply_change", + "derivation_for", + "downstream_of", + "keep_override", + "resolve", + "schema_derived_fields", + "set_override", +] diff --git a/studio/resolve/graph.py b/studio/resolve/graph.py new file mode 100644 index 0000000..d890266 --- /dev/null +++ b/studio/resolve/graph.py @@ -0,0 +1,160 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The dependency graph, built from ``derived_from`` metadata. + +Pure graph work: no config values, no derivation functions, no physics. Given the schema's +``derived_from`` edges it answers three questions -- what depends on this, in what order must things +be computed, and is the graph even acyclic. + +Kept separate from the resolver because the two fail differently and at different times. A malformed +graph is a **schema** bug that exists the moment the metadata is written, and should be found by a +test that never touches a config; a wrong recomputation is a **resolution** bug that needs values to +show up. Mixing them would mean a cycle in the metadata first manifesting as a hang while resolving +someone's run. + +Edges point from a dependency to its dependent (``site.temperature_k`` -> +``injection.so2_initial_pptv``), so "downstream" means "what must be recomputed when this changes", +which is the only direction the override machinery ever asks about. +""" + +from __future__ import annotations + +from collections import deque +from collections.abc import Iterable, Mapping + +from studio.schema import RunConfig, field_catalogue +from studio.schema.fields import Provenance + + +class CyclicDependencyError(ValueError): + """A ``derived_from`` cycle. Raised at graph construction, never worked around. + + A cycle means the schema claims a field is computed from something that is computed from it. No + resolution order exists, and the plausible-looking alternatives -- iterate to a fixed point, or + break the cycle at an arbitrary edge -- would both produce numbers that depend on where the + engine happened to start. + """ + + +class DependencyGraph: + """Immutable DAG over dotted field paths. + + Built from ``{node: (its dependencies)}``. Nodes with no dependencies are primary fields; the + rest are derived. + """ + + def __init__(self, dependencies: Mapping[str, Iterable[str]]) -> None: + self._dependencies: dict[str, tuple[str, ...]] = { + node: tuple(deps) for node, deps in dependencies.items() + } + self._dependents: dict[str, list[str]] = {node: [] for node in self._dependencies} + for node, deps in self._dependencies.items(): + for dep in deps: + if dep not in self._dependencies: + raise ValueError( + f"{node!r} depends on {dep!r}, which is not a node in the graph. Every " + f"path named in derived_from must be a real field." + ) + self._dependents[dep].append(node) + self._order = self._topological_order() + + @classmethod + def from_schema(cls, model_cls: type[RunConfig] = RunConfig) -> DependencyGraph: + """Build the graph from the schema's own metadata -- the only source of edges (ADR-002).""" + catalogue = field_catalogue(model_cls) + return cls( + { + path: tuple(meta["derived_from"]) if meta["derived_from"] else () + for path, meta in catalogue.items() + } + ) + + @property + def nodes(self) -> tuple[str, ...]: + """Every field path, in topological order (dependencies before dependents).""" + return self._order + + def dependencies_of(self, path: str) -> tuple[str, ...]: + """The paths ``path`` is computed from. Empty for a primary field.""" + self._check_known(path) + return self._dependencies[path] + + def dependents_of(self, path: str) -> tuple[str, ...]: + """The paths computed DIRECTLY from ``path``. One hop only; see :meth:`downstream_of`.""" + self._check_known(path) + return tuple(self._dependents[path]) + + def downstream_of(self, paths: Iterable[str]) -> tuple[str, ...]: + """Every field reachable from ``paths``, in topological order. The recompute set. + + Transitive on purpose: editing a plume dimension changes ``plume_volume_cm3``, which + changes ``so2_initial_pptv``. A one-hop answer would leave the second stale and silently + wrong -- and it is the second one that reaches the model. + + The seeds themselves are NOT included; the result is what must be recomputed, and a field + the user just set is not recomputed from itself. + """ + seeds = list(paths) + for path in seeds: + self._check_known(path) + seen: set[str] = set() + queue = deque(seeds) + while queue: + for dependent in self._dependents[queue.popleft()]: + if dependent not in seen: + seen.add(dependent) + queue.append(dependent) + return tuple(node for node in self._order if node in seen) + + def derived_nodes(self) -> tuple[str, ...]: + """Fields with at least one dependency, in topological order.""" + return tuple(node for node in self._order if self._dependencies[node]) + + def _check_known(self, path: str) -> None: + if path not in self._dependencies: + raise ValueError(f"unknown field path {path!r}") + + def _topological_order(self) -> tuple[str, ...]: + """Kahn's algorithm. Ties broken alphabetically so the order is deterministic. + + Determinism matters beyond tidiness: resolution order is observable through which error + surfaces first when several derivations would fail, and a run-to-run reshuffle would make + that irreproducible. + """ + remaining = {node: len(deps) for node, deps in self._dependencies.items()} + ready = deque(sorted(node for node, count in remaining.items() if count == 0)) + order: list[str] = [] + while ready: + node = ready.popleft() + order.append(node) + newly_ready = [] + for dependent in self._dependents[node]: + remaining[dependent] -= 1 + if remaining[dependent] == 0: + newly_ready.append(dependent) + for dependent in sorted(newly_ready): + ready.append(dependent) + if len(order) != len(self._dependencies): + cyclic = sorted(node for node, count in remaining.items() if count > 0) + raise CyclicDependencyError( + f"derived_from contains a cycle involving {cyclic}. No resolution order exists; " + f"fix the metadata rather than breaking the cycle at an arbitrary edge." + ) + return tuple(order) + + +def schema_derived_fields(model_cls: type[RunConfig] = RunConfig) -> tuple[str, ...]: + """Paths the schema marks ``DERIVED``, in topological order. + + Deliberately derived from the PROVENANCE, not from "has dependencies": the two must agree, and + ``studio/tests/unit/test_resolve_registry.py`` asserts that they do. If they ever diverge, that + is a schema bug -- a field with inputs but not marked derived would never be recomputed. + """ + catalogue = field_catalogue(model_cls) + graph = DependencyGraph.from_schema(model_cls) + return tuple( + path for path in graph.nodes if catalogue[path]["provenance"] == Provenance.DERIVED.value + ) + + +__all__ = ["CyclicDependencyError", "DependencyGraph", "schema_derived_fields"] diff --git a/studio/resolve/registry.py b/studio/resolve/registry.py new file mode 100644 index 0000000..055181b --- /dev/null +++ b/studio/resolve/registry.py @@ -0,0 +1,123 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Which function computes which derived field. + +The schema says a field is derived and what from; ``studio/science`` says how. This module is the +one place those two are bound together, and the binding is checked rather than trusted: a test +asserts that every ``DERIVED`` field in the schema has an entry here, and that each entry's declared +inputs are **exactly** the field's ``derived_from``. + +That check is the whole point of the module. Without it, a field could declare inputs the derivation +ignores -- so editing one of them would mark things stale and recompute to the same number -- or a +derivation could read a value the graph does not know about, so editing THAT one would silently +leave a stale result behind. Both are the kind of wrong that looks right. + +Derivations take a mapping of ``{dotted path: value}`` and index it by full path. Positional +arguments would be shorter and would eventually pass ``pressure`` where ``temperature`` belongs. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from typing import Any, Final + +from studio.science import ( + SO2_MOLAR_MASS_G_PER_MOL, + initial_mixing_ratio_pptv, + plume_volume_cm3, +) + + +@dataclass(frozen=True) +class Derivation: + """How to compute one derived field. + + Attributes: + inputs: Dotted paths this derivation reads. Must equal the field's ``derived_from``. + fn: Takes ``{path: value}`` for exactly ``inputs`` and returns the value. + summary: One line for the UI, explaining what the recomputation did. + """ + + inputs: tuple[str, ...] + fn: Callable[[Mapping[str, Any]], Any] + summary: str + + def compute(self, values: Mapping[str, Any]) -> Any: + """Run the derivation, checking that it was handed exactly what it declared. + + Raises: + KeyError: If an input is missing. It means the resolver and this table disagree, which + is a bug in one of them and never something to paper over with a default. + """ + missing = [path for path in self.inputs if path not in values] + if missing: + raise KeyError(f"derivation is missing declared inputs {missing}") + return self.fn({path: values[path] for path in self.inputs}) + + +def _plume_volume(values: Mapping[str, Any]) -> float: + return plume_volume_cm3( + length_m=values["injection.plume_length_m"], + width_m=values["injection.plume_width_m"], + height_m=values["injection.plume_height_m"], + ) + + +def _so2_initial_pptv(values: Mapping[str, Any]) -> float: + """Note this reads the DERIVED ``plume_volume_cm3``, not the three dimensions. + + That is what makes it a chained derivation and the reason resolution runs in topological order: + if the volume were recomputed after this, this would use the previous one and be quietly stale. + """ + return initial_mixing_ratio_pptv( + mass_kg=values["injection.so2_mass_kg"], + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=values["injection.plume_volume_cm3"], + pressure_mbar=values["site.pressure_mbar"], + temperature_k=values["site.temperature_k"], + ) + + +#: Derived field path -> how to compute it. Completeness against the schema is enforced by test. +DERIVATIONS: Final[dict[str, Derivation]] = { + "injection.plume_volume_cm3": Derivation( + inputs=( + "injection.plume_length_m", + "injection.plume_width_m", + "injection.plume_height_m", + ), + fn=_plume_volume, + summary="V0 = length x width x height", + ), + "injection.so2_initial_pptv": Derivation( + inputs=( + "injection.so2_mass_kg", + "injection.plume_volume_cm3", + "site.temperature_k", + "site.pressure_mbar", + ), + fn=_so2_initial_pptv, + summary="injected mass -> number density -> mixing ratio at this site's air density", + ), +} + + +def derivation_for(path: str) -> Derivation: + """The derivation for ``path``. + + Raises: + NotImplementedError: If the schema marks a field derived and nothing here computes it. The + field would otherwise stay ``None`` all the way to the model seam, where it would fail + far from its cause -- or worse, be defaulted (ADR-005). + """ + try: + return DERIVATIONS[path] + except KeyError: + raise NotImplementedError( + f"no derivation registered for {path!r}. The schema marks it DERIVED, so something " + f"must compute it: add it here, with the function itself in studio/science." + ) from None + + +__all__ = ["DERIVATIONS", "Derivation", "derivation_for"] diff --git a/studio/resolve/resolver.py b/studio/resolve/resolver.py new file mode 100644 index 0000000..a25aa3b --- /dev/null +++ b/studio/resolve/resolver.py @@ -0,0 +1,275 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Resolution and override semantics. + +This is the mechanism behind "go back and edit stage 1 without losing your stage 6 choices". It is +a data-model property, not a UI trick: the rules below hold for the CLI and the API equally, and +nothing above this layer needs to reimplement them. + +Three states a derived field can be in: + +* **auto** -- nobody has touched it, so it is recomputed silently when anything upstream changes. +* **user_override** -- somebody typed a value. It is NEVER overwritten by a recomputation. +* **user_override + stale** -- an override whose inputs have since changed, so the value no longer + follows from the rest of the config. + +Staleness is defined against a fingerprint: when an override is set, the upstream values at that +moment are recorded with it. A field is stale when those recorded values differ from the current +ones. That makes staleness a property of the config alone -- no edit history, no ordering +assumptions -- and it is what lets the UI show *the old value, the newly-derived value, and what +changed between them* rather than a bare warning. + +The user then chooses, and both choices are explicit: + +* :func:`accept_derived` -- drop the override, go back to auto. +* :func:`keep_override` -- keep the value and re-anchor its fingerprint to the current inputs. The + field stops being stale because the user has said, knowingly, that their value still applies. + +**A config with stale fields is never quietly persisted.** :meth:`ResolvedConfig.require_consistent` +raises, and the stale list travels with the object so that anything which does persist one has to +carry the list too. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + +from studio.resolve.graph import DependencyGraph, schema_derived_fields +from studio.resolve.registry import derivation_for +from studio.schema import RunConfig + + +class InconsistentConfigError(ValueError): + """A config with stale overrides was asked to behave as if it were consistent. + + Raised rather than resolved automatically, because both resolutions -- discard the user's value + or ignore the changed input -- are decisions only the user can make. + """ + + +class ChangedInput(BaseModel): + """One input whose value moved since an override was anchored.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + path: str + was: Any + now: Any + + +class OverrideRecord(BaseModel): + """A user-supplied value for a derived field, with the inputs it was anchored against.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + value: Any + #: Upstream values when the override was set or last re-anchored. Compared exactly: these are + #: floats copied from the same config, so a tolerance would only hide a real change. + inputs: dict[str, Any] = Field(default_factory=dict) + + +class StaleField(BaseModel): + """An override that no longer follows from the config, and everything needed to decide. + + Carries the newly-derived value as well as the current one, because "this is stale" without + "here is what it would be" leaves the user to recompute it by hand -- which is how a wrong value + gets kept. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + path: str + current_value: Any + derived_value: Any + changed_inputs: tuple[ChangedInput, ...] + summary: str + + +class ResolvedConfig(BaseModel): + """A ``RunConfig`` with its derived fields filled in, plus how they got there.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + config: RunConfig + #: Derived path -> the user's value and its anchor. Absent means auto. + overrides: dict[str, OverrideRecord] = Field(default_factory=dict) + #: Overrides whose inputs have moved. Empty means internally consistent. + stale: tuple[StaleField, ...] = () + + @property + def is_consistent(self) -> bool: + """True when every derived field either was recomputed or is an anchored override.""" + return not self.stale + + @property + def stale_fields(self) -> tuple[str, ...]: + """Just the paths, for a log line or an error message.""" + return tuple(entry.path for entry in self.stale) + + def require_consistent(self) -> None: + """Raise unless the config is internally consistent. + + Called before anything that treats the config as a description of a run -- submission, + hashing for identity, writing it beside results. A stale config still HAS a hash, and that + is exactly the trap: it would be a stable identity for a set of numbers that do not follow + from each other. + """ + if self.stale: + details = "; ".join( + f"{entry.path} = {entry.current_value!r} but now derives to {entry.derived_value!r}" + for entry in self.stale + ) + raise InconsistentConfigError( + f"config has {len(self.stale)} stale override(s): {details}. Resolve each with " + f"accept_derived() or keep_override() before using this config." + ) + + def value_at(self, path: str) -> Any: + """The current value at a dotted path.""" + return _get(self.config.model_dump(mode="python"), path) + + +def resolve( + config: RunConfig, + overrides: Mapping[str, OverrideRecord] | None = None, + *, + graph: DependencyGraph | None = None, +) -> ResolvedConfig: + """Compute every derived field, honouring overrides, and report what has gone stale. + + Derived fields are visited in topological order, so a derivation that reads another derived + field (``so2_initial_pptv`` reads ``plume_volume_cm3``) sees the recomputed value rather than + the previous one. + """ + graph = graph or DependencyGraph.from_schema() + held = dict(overrides or {}) + payload = config.model_dump(mode="python") + stale: list[StaleField] = [] + + for path in schema_derived_fields(): + derivation = derivation_for(path) + inputs = {name: _get(payload, name) for name in derivation.inputs} + derived_value = derivation.compute(inputs) + record = held.get(path) + if record is None: + _set(payload, path, derived_value) + continue + _set(payload, path, record.value) + changed = tuple( + ChangedInput(path=name, was=record.inputs[name], now=value) + for name, value in inputs.items() + if name in record.inputs and record.inputs[name] != value + ) + if changed or set(record.inputs) != set(inputs): + stale.append( + StaleField( + path=path, + current_value=record.value, + derived_value=derived_value, + changed_inputs=changed, + summary=derivation.summary, + ) + ) + return ResolvedConfig( + config=RunConfig.model_validate(payload), overrides=held, stale=tuple(stale) + ) + + +def apply_change(resolved: ResolvedConfig, path: str, value: Any) -> ResolvedConfig: + """Set ``path`` to ``value`` and recompute the downstream closure. + + A change to a DERIVED field is an override -- that is what a user typing into a computed box + means -- so it is routed to :func:`set_override` rather than being silently recomputed away on + the next edit. + """ + graph = DependencyGraph.from_schema() + if path in set(schema_derived_fields()): + return set_override(resolved, path, value, graph=graph) + graph.dependents_of(path) # validates the path against the schema, and raises if unknown + payload = resolved.config.model_dump(mode="python") + _set(payload, path, value) + return resolve(RunConfig.model_validate(payload), resolved.overrides, graph=graph) + + +def set_override( + resolved: ResolvedConfig, path: str, value: Any, *, graph: DependencyGraph | None = None +) -> ResolvedConfig: + """Pin ``path`` to a user-supplied ``value``, anchored to the config's current inputs. + + Anchoring at the moment of the override is what makes it non-stale now and detectably stale + later. + """ + graph = graph or DependencyGraph.from_schema() + derived = set(schema_derived_fields()) + if path not in derived: + raise ValueError( + f"{path!r} is not a derived field, so it cannot be overridden -- set it directly with " + f"apply_change(). Derived fields: {sorted(derived)}" + ) + payload = resolved.config.model_dump(mode="python") + inputs = {name: _get(payload, name) for name in derivation_for(path).inputs} + overrides = dict(resolved.overrides) + overrides[path] = OverrideRecord(value=value, inputs=inputs) + return resolve(resolved.config, overrides, graph=graph) + + +def accept_derived(resolved: ResolvedConfig, path: str) -> ResolvedConfig: + """Drop the override at ``path``; the field goes back to auto and is recomputed.""" + if path not in resolved.overrides: + raise ValueError(f"{path!r} is not overridden, so there is nothing to accept") + overrides = {key: record for key, record in resolved.overrides.items() if key != path} + return resolve(resolved.config, overrides) + + +def keep_override(resolved: ResolvedConfig, path: str) -> ResolvedConfig: + """Keep the user's value at ``path`` and re-anchor it to the current inputs. + + The field stops being stale because the user has confirmed it still applies -- knowingly, which + is the difference between this and never having flagged it. + """ + record = resolved.overrides.get(path) + if record is None: + raise ValueError(f"{path!r} is not overridden, so there is nothing to keep") + payload = resolved.config.model_dump(mode="python") + inputs = {name: _get(payload, name) for name in derivation_for(path).inputs} + overrides = dict(resolved.overrides) + overrides[path] = OverrideRecord(value=record.value, inputs=inputs) + return resolve(resolved.config, overrides) + + +def downstream_of(paths: Iterable[str]) -> tuple[str, ...]: + """Fields that must be recomputed when ``paths`` change. Convenience over the schema graph.""" + return DependencyGraph.from_schema().downstream_of(paths) + + +def _get(payload: Mapping[str, Any], path: str) -> Any: + cursor: Any = payload + for part in path.split("."): + cursor = cursor[part] + return cursor + + +def _set(payload: dict[str, Any], path: str, value: Any) -> None: + parts = path.split(".") + cursor: dict[str, Any] = payload + for part in parts[:-1]: + cursor = cursor[part] + cursor[parts[-1]] = value + + +__all__ = [ + "ChangedInput", + "InconsistentConfigError", + "OverrideRecord", + "ResolvedConfig", + "StaleField", + "accept_derived", + "apply_change", + "downstream_of", + "keep_override", + "resolve", + "set_override", +] diff --git a/studio/runner/__init__.py b/studio/runner/__init__.py index 5e9dff5..a8612e9 100644 --- a/studio/runner/__init__.py +++ b/studio/runner/__init__.py @@ -31,4 +31,32 @@ from __future__ import annotations -__all__: list[str] = [] +from studio.runner.base import ( + TERMINAL_STATES, + CloudBatchRunner, + InvalidTransitionError, + JobRecord, + JobRegistry, + JobRunner, + JobState, + NotImplementedRunner, + SlurmRunner, + Transition, +) +from studio.runner.local import DEFAULT_MAX_WORKERS, THREAD_PINNING, LocalSubprocessRunner + +__all__ = [ + "DEFAULT_MAX_WORKERS", + "TERMINAL_STATES", + "THREAD_PINNING", + "CloudBatchRunner", + "InvalidTransitionError", + "JobRecord", + "JobRegistry", + "JobRunner", + "JobState", + "LocalSubprocessRunner", + "NotImplementedRunner", + "SlurmRunner", + "Transition", +] diff --git a/studio/runner/base.py b/studio/runner/base.py new file mode 100644 index 0000000..d7350a8 --- /dev/null +++ b/studio/runner/base.py @@ -0,0 +1,295 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The job lifecycle and the ``JobRunner`` interface. + +**Nothing above this interface may assume an execution backend** (ADR-008). Today there is one +implementation, ``LocalSubprocessRunner``; Slurm and cloud-batch adapters do not exist and raise +rather than degrade. That constraint is why the lifecycle lives here as data rather than inside the +local runner: a scheduler-backed runner would report the same states and the same transitions. + +Every transition is timestamped, and the record keeps all of them rather than just the current +state. "It failed" is not debuggable; "QUEUED at 14:02:11, RUNNING at 14:02:11, FAILED at 14:06:48 +with exit code 1" is. The same reasoning drives what a finished job keeps on disk: the resolved +input, the captured stdout, the captured stderr and the exit code, so **a failed run can be +diagnosed without re-running it** -- which matters when re-running costs four minutes and the +failure is intermittent. + +``run_coupled`` prints diagnostics to stdout rather than logging them (``[gas] dt0 stall...``, +``[coupled] NOTE:...``, ``[stop]...``), so stdout IS the run's log stream and is captured as such. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from datetime import UTC, datetime +from enum import StrEnum +from pathlib import Path +from typing import Protocol, runtime_checkable + +from pydantic import BaseModel, ConfigDict, Field + + +class JobState(StrEnum): + """Where a job is. The terminal states are distinguished on purpose. + + ``FAILED`` and ``TERMINATED_ON_LIMIT`` are not the same thing and must never be collapsed: the + first means the model could not produce a result, the second means it was still going when we + stopped it. A run stopped on a limit has partial output that may look complete, so it is flagged + and **never presented as converged**. + """ + + DRAFT = "draft" + QUEUED = "queued" + RUNNING = "running" + SUCCEEDED = "succeeded" + FAILED = "failed" + CANCELLED = "cancelled" + TERMINATED_ON_LIMIT = "terminated_on_limit" + + +#: States from which no further transition is possible. +TERMINAL_STATES = frozenset( + { + JobState.SUCCEEDED, + JobState.FAILED, + JobState.CANCELLED, + JobState.TERMINATED_ON_LIMIT, + } +) + +#: The only transitions the lifecycle allows. Enforced rather than documented: a job that went +#: RUNNING -> QUEUED, or that reported SUCCEEDED twice, means the runner lost track of a process, +#: and silently accepting it would make the record a story rather than a log. +_ALLOWED: dict[JobState, frozenset[JobState]] = { + JobState.DRAFT: frozenset({JobState.QUEUED, JobState.CANCELLED}), + JobState.QUEUED: frozenset({JobState.RUNNING, JobState.CANCELLED, JobState.FAILED}), + JobState.RUNNING: frozenset( + { + JobState.SUCCEEDED, + JobState.FAILED, + JobState.CANCELLED, + JobState.TERMINATED_ON_LIMIT, + } + ), + JobState.SUCCEEDED: frozenset(), + JobState.FAILED: frozenset(), + JobState.CANCELLED: frozenset(), + JobState.TERMINATED_ON_LIMIT: frozenset(), +} + + +class InvalidTransitionError(ValueError): + """An illegal lifecycle transition. Raised, never tolerated.""" + + +def _now() -> datetime: + """Timezone-aware UTC. Naive timestamps compare wrongly across a DST boundary.""" + return datetime.now(UTC) + + +class Transition(BaseModel): + """One state change, with when and why.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + state: JobState + at: datetime + detail: str = "" + + +class JobRecord(BaseModel): + """Everything known about one job. Immutable; a transition produces a new record. + + Immutable because this is the audit trail. A record that can be edited in place is a record that + can quietly disagree with what happened. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + job_id: str + #: Identity of the config being run (ADR-006). Links the job to its inputs and its cache entry. + config_hash: str + label: str = "" + state: JobState = JobState.DRAFT + transitions: tuple[Transition, ...] = () + #: Where the resolved input, the logs and the output live. Present from submission, so a job + #: that dies early still says where to look. + work_dir: Path | None = None + input_path: Path | None = None + #: The provenance record (ADR-006), written at submit time -- before execution -- so a run that + #: dies in minute three of four still says exactly what produced it. + provenance_path: Path | None = None + stdout_path: Path | None = None + stderr_path: Path | None = None + exit_code: int | None = None + #: Set when the job ends for any reason; a short human-facing explanation. + detail: str = "" + + @property + def is_terminal(self) -> bool: + return self.state in TERMINAL_STATES + + def _first_time(self, state: JobState) -> datetime | None: + """When the job FIRST entered ``state``, or ``None`` if it never did.""" + for transition in self.transitions: + if transition.state is state: + return transition.at + return None + + @property + def submitted_at(self) -> datetime | None: + return self._first_time(JobState.QUEUED) + + @property + def started_at(self) -> datetime | None: + return self._first_time(JobState.RUNNING) + + @property + def ended_at(self) -> datetime | None: + for transition in reversed(self.transitions): + if transition.state in TERMINAL_STATES: + return transition.at + return None + + @property + def duration_s(self) -> float | None: + """Wall-clock from RUNNING to the terminal state, or ``None`` if it has not run yet.""" + started, ended = self.started_at, self.ended_at + if started is None or ended is None: + return None + return (ended - started).total_seconds() + + def transition_to(self, state: JobState, detail: str = "", **updates: object) -> JobRecord: + """Return a new record in ``state``. + + Raises: + InvalidTransitionError: If the lifecycle does not allow it. + """ + if state not in _ALLOWED[self.state]: + allowed = sorted(s.value for s in _ALLOWED[self.state]) + raise InvalidTransitionError( + f"job {self.job_id} cannot go {self.state.value} -> {state.value}; " + f"allowed from {self.state.value}: {allowed or ['(terminal)']}" + ) + return self.model_copy( + update={ + "state": state, + "transitions": ( + *self.transitions, + Transition(state=state, at=_now(), detail=detail), + ), + "detail": detail or self.detail, + **updates, + } + ) + + +@runtime_checkable +class JobRunner(Protocol): + """How Studio executes runs. The only assumption anything above may make. + + Deliberately small. ``submit`` takes a resolved config and returns a record; everything else + operates on a job id. A Slurm implementation would satisfy this without any caller changing. + """ + + def submit(self, config: object, *, label: str = "") -> JobRecord: + """Queue a run. Returns immediately with a record in QUEUED.""" + ... + + def poll(self, job_id: str) -> JobRecord: + """Current record, advancing the state if the process has finished or exceeded its limit.""" + ... + + def cancel(self, job_id: str) -> JobRecord: + """Stop a queued or running job. Terminal jobs are returned unchanged.""" + ... + + def artifacts(self, job_id: str) -> Sequence[Path]: + """Files this job produced, if any.""" + ... + + +class NotImplementedRunner: + """Base for backends that do not exist yet (ADR-008). + + Present so that "Slurm is not supported" is a class you can point at rather than a gap in a + dispatch table. Every method raises with the same message; nothing degrades to local execution, + because a job silently running somewhere other than where it was sent is worse than an error. + """ + + backend_name = "unimplemented" + + def _raise(self) -> None: + raise NotImplementedError( + f"the {self.backend_name} execution backend is not implemented. ADR-008 records local " + f"subprocesses as the only supported backend; nothing falls back to local execution, " + f"because a job running somewhere other than where it was sent is worse than an error." + ) + + def submit(self, config: object, *, label: str = "") -> JobRecord: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + def poll(self, job_id: str) -> JobRecord: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + def cancel(self, job_id: str) -> JobRecord: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + def artifacts(self, job_id: str) -> Sequence[Path]: + self._raise() + raise AssertionError("unreachable") # pragma: no cover + + +class SlurmRunner(NotImplementedRunner): + """Not implemented (ADR-008, BLOCKING-1).""" + + backend_name = "Slurm" + + +class CloudBatchRunner(NotImplementedRunner): + """Not implemented (ADR-008, BLOCKING-1).""" + + backend_name = "cloud-batch" + + +class JobRegistry(BaseModel): + """In-memory job records, keyed by id. + + Phase 0 keeps this in the process that owns the worker pool (ADR-007). ADR-004 puts job state in + the database so a handle survives an API restart; this class is the seam that will move there, + and it is deliberately dumb so that swap is a swap rather than a rewrite. + """ + + model_config = ConfigDict(extra="forbid") + + records: dict[str, JobRecord] = Field(default_factory=dict) + + def put(self, record: JobRecord) -> JobRecord: + self.records[record.job_id] = record + return record + + def get(self, job_id: str) -> JobRecord: + try: + return self.records[job_id] + except KeyError: + raise KeyError(f"unknown job id {job_id!r}") from None + + def __contains__(self, job_id: object) -> bool: + return job_id in self.records + + +__all__ = [ + "TERMINAL_STATES", + "CloudBatchRunner", + "InvalidTransitionError", + "JobRecord", + "JobRegistry", + "JobRunner", + "JobState", + "NotImplementedRunner", + "SlurmRunner", + "Transition", +] diff --git a/studio/runner/local.py b/studio/runner/local.py new file mode 100644 index 0000000..2349557 --- /dev/null +++ b/studio/runner/local.py @@ -0,0 +1,270 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``LocalSubprocessRunner`` -- the only execution backend that exists (ADR-008). + +Each run is one thread-pinned subprocess. The pinning is not incidental: setting +``OMP/OPENBLAS/MKL/VECLIB/NUMEXPR_NUM_THREADS=1`` and disabling XLA's multithreaded Eigen is what +gives ~N times the throughput for N workers on this workload -- **not** ``vmap``. It is copied from +``coupled/paper_ensemble/launch_parallel.py:26-30``, which is the configuration the 810-run ensemble +was actually produced with. + +Why a subprocess at all: ``run_coupled`` is a library call with no ``__main__`` of its own, it +returns arrays in memory and writes nothing, and it prints its diagnostics. So something has to be +the process, that something is ``python -m studio.cli.run``, and its stdout is the run's log. + +What is on disk when a job ends, whatever the outcome: + +* ``input.json`` -- the RESOLVED config that was actually run +* ``provenance.json`` -- what produced it: config hash, app version, SANDBOX and submodule SHAs, + and whether any checkout was dirty (ADR-006). Written BEFORE the process starts. +* ``stdout.log`` / ``stderr.log`` -- captured in full +* the exit code and every state transition, in the record + +That set is chosen so a failure can be diagnosed **without re-running it**, which matters when a +re-run costs minutes and the failure is intermittent. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import uuid +from collections.abc import Sequence +from concurrent.futures import Future, ThreadPoolExecutor +from pathlib import Path +from typing import Any, Final + +from studio.modelio.provenance import record_for +from studio.resolve import ResolvedConfig +from studio.runner.base import JobRecord, JobRegistry, JobState + +#: Single-thread pinning, copied from ``launch_parallel.py:26-30``. This is what makes N concurrent +#: runs ~N times the throughput; without it they fight over cores and each one gets slower. +THREAD_PINNING: Final[dict[str, str]] = { + "OMP_NUM_THREADS": "1", + "OPENBLAS_NUM_THREADS": "1", + "MKL_NUM_THREADS": "1", + "VECLIB_MAXIMUM_THREADS": "1", + "NUMEXPR_NUM_THREADS": "1", + "XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false", +} + +#: Default concurrent subprocesses (ASSUMPTION-4). Each run is single-threaded by the pinning above, +#: so this is a core count rather than a guess about memory. Existing practice is 10. +DEFAULT_MAX_WORKERS = 4 + +#: Grace period between SIGTERM and SIGKILL when stopping a run, in seconds. Long enough for Python +#: to unwind and flush the log; short enough that a wedged process does not hold a worker. +_TERMINATE_GRACE_S = 5.0 + + +class LocalSubprocessRunner: + """Run jobs as local subprocesses from a bounded pool. + + Args: + work_root: Directory under which each job gets ``//``. + max_workers: Concurrent subprocesses (ASSUMPTION-4). + entry_module: The module launched with ``-m``. Overridable so the LIFECYCLE can be tested + without a four-minute model run -- the default is the real path, and nothing in this + class branches on the value. It is a parameter, not a test hook. + repo_root: Which checkout to record in each run's provenance. Defaults to the one this code + came from, which is what a local runner should record. It is a parameter because + "which checkout produced this?" is a real question a runner has to answer -- a worker + executing code from elsewhere would answer it differently -- and because CI has no + submodules, so the tests point it at a synthetic checkout. **It does not weaken the + guarantee**: a run still cannot start unless the checkout it names can be pinned. + python_executable: Interpreter for the subprocess; defaults to the current one, so a job + inherits the environment that submitted it rather than whatever is first on PATH. + """ + + def __init__( + self, + work_root: Path, + *, + max_workers: int = DEFAULT_MAX_WORKERS, + entry_module: str = "studio.cli.run", + python_executable: str | None = None, + repo_root: Path | None = None, + ) -> None: + if max_workers < 1: + raise ValueError(f"max_workers must be >= 1, got {max_workers}") + self.work_root = Path(work_root) + self.work_root.mkdir(parents=True, exist_ok=True) + self.max_workers = max_workers + self.entry_module = entry_module + self.python_executable = python_executable or sys.executable + self.repo_root = repo_root + self.registry = JobRegistry() + self._pool = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="studio-run") + self._processes: dict[str, subprocess.Popen[bytes]] = {} + self._futures: dict[str, Future[None]] = {} + + # -- submission ------------------------------------------------------------------------ + + def submit(self, config: ResolvedConfig, *, label: str = "") -> JobRecord: + """Write the resolved input, queue the run, return immediately. + + Raises: + InconsistentConfigError: If the config has stale overrides. A run started from one would + produce results that do not follow from their own inputs, and nothing downstream + could tell. + """ + config.require_consistent() + job_id = uuid.uuid4().hex[:12] + work_dir = self.work_root / job_id + work_dir.mkdir(parents=True, exist_ok=False) + + input_path = work_dir / "input.json" + input_path.write_text(config.model_dump_json(indent=2), encoding="utf-8") + + # Provenance BEFORE execution (ADR-006). Deliberately not in a try/except: if the model + # cannot be pinned, the run must not start. A result whose origin is unknown is worth less + # than no result, because it looks like the others. + provenance = record_for(config, repo_root=self.repo_root) + provenance_path = provenance.write(work_dir / "provenance.json") + + record = JobRecord( + job_id=job_id, + config_hash=config.config.config_hash(), + label=label, + work_dir=work_dir, + input_path=input_path, + provenance_path=provenance_path, + stdout_path=work_dir / "stdout.log", + stderr_path=work_dir / "stderr.log", + ).transition_to(JobState.QUEUED, detail=f"queued for {self.entry_module}") + self.registry.put(record) + + max_wall_time_s = config.config.termination.max_wall_time_s + self._futures[job_id] = self._pool.submit(self._execute, job_id, max_wall_time_s) + return record + + # -- execution ------------------------------------------------------------------------- + + def _execute(self, job_id: str, max_wall_time_s: float) -> None: + """Run one job to completion. Runs on a pool thread; never raises into the pool.""" + record = self.registry.get(job_id) + if record.state is JobState.CANCELLED: + return # cancelled while queued + work_dir = record.work_dir + assert work_dir is not None and record.stdout_path and record.stderr_path + + command = [ + self.python_executable, + "-m", + self.entry_module, + str(record.input_path), + str(work_dir), + ] + env = {**os.environ, **THREAD_PINNING} + try: + with ( + record.stdout_path.open("wb") as stdout, + record.stderr_path.open("wb") as stderr, + ): + process = subprocess.Popen(command, stdout=stdout, stderr=stderr, env=env) + self._processes[job_id] = process + self.registry.put( + self.registry.get(job_id).transition_to( + JobState.RUNNING, detail=" ".join(command) + ) + ) + try: + exit_code = process.wait(timeout=max_wall_time_s) + except subprocess.TimeoutExpired: + self._stop(process) + self._finish( + job_id, + JobState.TERMINATED_ON_LIMIT, + detail=( + f"exceeded max_wall_time_s = {max_wall_time_s}; partial output is " + f"NOT a converged result" + ), + exit_code=process.returncode, + ) + return + except Exception as exc: + self._finish(job_id, JobState.FAILED, detail=f"{type(exc).__name__}: {exc}") + return + finally: + self._processes.pop(job_id, None) + + current = self.registry.get(job_id) + if current.state is JobState.CANCELLED: + return + if exit_code == 0: + self._finish(job_id, JobState.SUCCEEDED, detail="completed", exit_code=exit_code) + else: + self._finish( + job_id, + JobState.FAILED, + detail=( + f"exit code {exit_code}; see {record.stderr_path.name} in {work_dir} -- the " + f"resolved input and both log streams are kept so this needs no re-run" + ), + exit_code=exit_code, + ) + + def _finish( + self, job_id: str, state: JobState, *, detail: str, exit_code: int | None = None + ) -> None: + record = self.registry.get(job_id) + if record.is_terminal: + return + self.registry.put(record.transition_to(state, detail=detail, exit_code=exit_code)) + + @staticmethod + def _stop(process: subprocess.Popen[Any]) -> None: + """SIGTERM, then SIGKILL if it will not go. Python gets a chance to flush its log first.""" + process.terminate() + try: + process.wait(timeout=_TERMINATE_GRACE_S) + except subprocess.TimeoutExpired: + process.kill() + process.wait() + + # -- inspection ------------------------------------------------------------------------ + + def poll(self, job_id: str) -> JobRecord: + """The current record. State is advanced by the worker thread, not by polling. + + Polling is therefore free and side-effect-free, which matters because the API pushes + progress over SSE (spec 7.3) and would otherwise poll this on a timer per connected client. + """ + return self.registry.get(job_id) + + def cancel(self, job_id: str) -> JobRecord: + """Stop a queued or running job. A terminal job is returned unchanged, not an error.""" + record = self.registry.get(job_id) + if record.is_terminal: + return record + process = self._processes.get(job_id) + if process is not None: + self._stop(process) + cancelled = record.transition_to(JobState.CANCELLED, detail="cancelled by request") + return self.registry.put(cancelled) + + def artifacts(self, job_id: str) -> Sequence[Path]: + """Files this job produced, sorted. Includes the logs and the resolved input on failure.""" + record = self.registry.get(job_id) + if record.work_dir is None or not record.work_dir.is_dir(): + return () + return tuple(sorted(p for p in record.work_dir.iterdir() if p.is_file())) + + def wait(self, job_id: str, timeout: float | None = None) -> JobRecord: + """Block until the job reaches a terminal state. For the CLI and for tests, not the API.""" + future = self._futures.get(job_id) + if future is not None: + future.result(timeout=timeout) + return self.registry.get(job_id) + + def shutdown(self, *, cancel_running: bool = False) -> None: + """Stop accepting work; optionally stop what is already running.""" + if cancel_running: + for job_id in list(self._processes): + self.cancel(job_id) + self._pool.shutdown(wait=True) + + +__all__ = ["DEFAULT_MAX_WORKERS", "THREAD_PINNING", "LocalSubprocessRunner"] diff --git a/studio/schema/__init__.py b/studio/schema/__init__.py index 6b05083..fd4aeaa 100644 --- a/studio/schema/__init__.py +++ b/studio/schema/__init__.py @@ -5,23 +5,111 @@ Defined once here in Pydantic v2, exported as JSON Schema, consumed by the web client for form generation and validation. Nothing in the UI may invent a field that does not exist here. -Contents (task 0.2, not yet implemented): +Contents: -* ``SciField`` -- the field-metadata carrier: canonical unit, valid range or enum, label, - description, default, PROVENANCE of that default, citation, and ``derived_from``. -* ``RunConfig`` -- one validated simulation description; versioned, canonically serialisable, - hashable. -* ``RunSet`` + axes (GRID / ZIP / LIST) -- the primary user-facing object. A single run is a RunSet - with zero axes; there is no separate N=1 code path. -* Canonical JSON serialisation and stable SHA-256 hashing (ADR-006). +* ``SciField`` (``fields.py``) -- the field-metadata carrier: canonical unit, valid range, label, + description, default, PROVENANCE of that default, citation, ``derived_from``, and any caveat that + must travel with the value. Inconsistent metadata raises at import time. +* ``RunConfig`` (``config.py``) -- one validated simulation description; versioned, canonically + serialisable, hashable. Minimum viable for the Phase-0 slice: what the golden case + ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` needs, plus the provenance and derivation inputs the + model has no field for. +* ``RunSet`` + axes (``runset.py``) -- the primary user-facing object. A single run is a RunSet with + zero axes; there is no separate N = 1 code path. +* Canonical JSON and stable SHA-256 (``hashing.py``) -- run identity and cache key (ADR-006). +* JSON Schema export and the flat field catalogue (``export.py``). -Canonical units are the MODEL's native units (mbar, ppm, pptv, K, s, um^2/cm^3), not SI -- see +Canonical units are the MODEL's native units (mbar, ppmv, pptv, K, s, um^2/cm^3), not SI -- see ADR-003 and ASSUMPTION-1. ``pint`` is used for display conversion at the presentation boundary and for property tests, never inside the model interface. +**No physics happens here.** Derived fields declare what they are computed from and stay unset; the +dependency-graph engine is task 0.3 and the derivations are task 0.5. + This package must not import ``coupled``, the API, or the database (see ``studio/__init__.py``). """ from __future__ import annotations -__all__: list[str] = [] +from studio.schema.config import ( + PAPER_BACKGROUND_GAS_PPTV, + SCHEMA_VERSION, + Background, + Chemistry, + Dilution, + Injection, + Microphysics, + Numerics, + ProcessSwitches, + RunConfig, + Schedule, + SchemaModel, + Site, + Termination, +) +from studio.schema.enums import AxisKind, BackgroundAerosol, DilutionRegime, PhotolysisMode +from studio.schema.export import ( + SCHEMA_ID, + field_catalogue, + iter_leaf_fields, + run_config_json_schema, +) +from studio.schema.fields import EXTENSION_KEY, Provenance, SciField, field_metadata +from studio.schema.hashing import ( + CANONICAL_FORM_VERSION, + canonical_json, + canonical_payload, + config_hash, + short_hash, +) +from studio.schema.runset import ( + Axis, + AxisPoint, + ExpandedRun, + RunSet, + apply_assignments, + resolve_path, +) +from studio.schema.units import PINT_EXPRESSION, Unit + +__all__ = [ + "CANONICAL_FORM_VERSION", + "EXTENSION_KEY", + "PAPER_BACKGROUND_GAS_PPTV", + "PINT_EXPRESSION", + "SCHEMA_ID", + "SCHEMA_VERSION", + "Axis", + "AxisKind", + "AxisPoint", + "Background", + "BackgroundAerosol", + "Chemistry", + "Dilution", + "DilutionRegime", + "ExpandedRun", + "Injection", + "Microphysics", + "Numerics", + "PhotolysisMode", + "ProcessSwitches", + "Provenance", + "RunConfig", + "RunSet", + "Schedule", + "SchemaModel", + "SciField", + "Site", + "Termination", + "Unit", + "apply_assignments", + "canonical_json", + "canonical_payload", + "config_hash", + "field_catalogue", + "field_metadata", + "iter_leaf_fields", + "resolve_path", + "run_config_json_schema", + "short_hash", +] diff --git a/studio/schema/config.py b/studio/schema/config.py new file mode 100644 index 0000000..7cdefb4 --- /dev/null +++ b/studio/schema/config.py @@ -0,0 +1,751 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunConfig`` -- one validated simulation description (ADR-002). + +Deliberately MINIMUM VIABLE: it covers exactly what the Phase-0 slice needs, which is everything +``run_ensemble.build_scenario()`` sets for the golden case +``30N_20km__sabr220__D2med__a1p0__nuc1__cg1``, plus the provenance and derivation inputs the model +has no field for. Stages 4-7 of the spec are NOT modelled here; doing that before anything runs is +how a config layer ends up describing a model that does not exist. + +Defaults are the paper ensemble's configuration, not the model's, wherever the two differ -- Phase +0's job is to reproduce existing trusted runs (ASSUMPTION-5). Each such field records which it is, +so "why is this 30 and not 0?" has an answer in the schema rather than in someone's memory. + +Three things this module deliberately does NOT do: + +* **No physics.** Fields marked ``DERIVED`` declare what they are computed from and stay unset. The + dependency-graph engine is task 0.3 and the derivations are task 0.5; a plausible number computed + here would be exactly the failure mode ``studio/CLAUDE.md`` forbids. +* **No species-name validation.** Whether ``"HCl"`` is a species is a question for the model's own + ``IDX``, and ``studio.schema`` may not import the model. ``studio/modelio`` validates names at the + seam, where the answer actually lives. +* **No unit conversion.** Values are in canonical (= model-native) units already (ADR-003). + +Validation that the model performs in ``CoupledScenario.__post_init__`` is MIRRORED here where it is +cheap to do so -- the DT/dt_couple divisibility rule especially -- because ADR-002's third +motivating problem is that a form cannot today learn what is valid without importing most of the +model. +Mirrored rules cite the model line they mirror; if the model's rule changes, the citation is how you +find this one. +""" + +from __future__ import annotations + +from typing import Annotated, Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from studio.schema.enums import BackgroundAerosol, DilutionRegime, PhotolysisMode +from studio.schema.fields import Provenance, SciField +from studio.schema.units import Unit + +#: The schema's own version. Bumped on any change to field names, semantics or defaults, because +#: those change the config hash and therefore run identity (ADR-006). Old configs are never silently +#: reinterpreted under new semantics. +SCHEMA_VERSION = "0.2.0" + +#: Stratospheric background gas composition [pptv] used by the 810-run ensemble +#: (``run_ensemble.py:56-57``). Module-level so the default is one object with one source, and so a +#: test can compare against it without reaching into a field default. +PAPER_BACKGROUND_GAS_PPTV: dict[str, float] = { + "O2": 2.1e11, + "O3": 1.18e6, + "OH": 0.5, + "HO2": 3.0, + "NO": 450.0, + "NO2": 450.0, + "HCl": 777.0, + "ClONO2": 127.0, + "HNO3": 5000.0, +} + + +class SchemaModel(BaseModel): + """Base for every schema model: frozen, and unknown keys are an error. + + ``frozen`` because a submitted config is immutable (ADR-004) -- an edit produces a new config + and a new run, which is what makes ``config_hash`` a meaningful identity. ``extra="forbid"`` + because a typo'd key that is silently accepted is a config that does not describe the run. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + +class Site(SchemaModel): + """Where the box is, and the thermodynamic state it sits in. + + Phase 0 takes T and p as user input. Phase 1 derives them from climatology at a chosen + (lat, lon, altitude-or-tropopause-relative) point -- which is why they are primary fields now + and become ``derived_from`` targets later, not the other way round. + """ + + latitude_deg: float = SciField( + default=30.0, + unit=Unit.DEGREE, + ge=-90.0, + le=90.0, + label="Latitude", + description="Box latitude; drives the solar zenith angle and therefore photolysis.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:62 (LAT_ALT '30N_20km')", + examples=[30.0, 60.0], + ) + longitude_deg: float = SciField( + default=0.0, + unit=Unit.DEGREE, + ge=-180.0, + le=180.0, + label="Longitude", + description=( + "Box longitude. Only affects the solar zenith angle via local solar time; at 0 deg, " + "UTC and local solar time coincide, which is why the ensemble's 00:00 local release " + "is expressed as start_utc_hour = 0." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:98", + ) + temperature_k: float = SciField( + default=210.0, + unit=Unit.KELVIN, + gt=0.0, + label="Temperature", + description=( + "Box temperature. The box is isobaric and ISOTHERMAL: the temperature feedback is " + "refused while the radiative calculation has no longwave component, so this value " + "holds for the whole run. See switches.heating_to_t and SCIENCE-4 (issue #56)." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Site: 210 K, 55 hPa)", + examples=[210.0, 213.0], + ) + pressure_mbar: float = SciField( + default=55.0, + unit=Unit.MBAR, + gt=0.0, + label="Pressure", + description=( + "Box pressure, the model's native pressure unit (mbar == hPa). Also sets the box " + "altitude used to place the aerosol in the TUV-x radiation column." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Site: 210 K, 55 hPa)", + examples=[55.0, 120.0], + ) + h2o_ppmv: float = SciField( + default=6.9104, + unit=Unit.PPMV, + ge=0.0, + label="Water vapour", + description=( + "Water vapour mixing ratio. The ensemble value is RH = 3% precomputed at 210 K / " + "55 hPa; it is a decimal in native units on purpose (ADR-003) -- do not round-trip it " + "through SI." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:62 (LAT_ALT WTR column, RH = 3%)", + caveat=( + "Reanalysis stratospheric water vapour is biased dry, so ERA5 is not the recommended " + "source for this field when Phase 1 lands (BLOCKING-5)." + ), + ) + + +class Schedule(SchemaModel): + """When the release happens and how long the box is integrated.""" + + day_of_year: int = SciField( + default=172, + unit=Unit.DIMENSIONLESS, + ge=1, + le=366, + label="Day of year", + description="Day of year of the release; sets the solar declination.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (day 172, ~21 June)", + ) + start_utc_hour: float = SciField( + default=0.0, + unit=Unit.HOUR, + ge=0.0, + lt=24.0, + label="Release hour (UTC)", + description=( + "UTC hour of release. The paper ensemble releases at 00:00 LOCAL SOLAR time and sets " + "this to 0 with longitude 0, where the two coincide. At any other longitude they do " + "not, and the distinction is the user's to make." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:98", + ) + duration_days: int = SciField( + default=10, + unit=Unit.DAY, + ge=1, + label="Duration", + description=( + "Simulated duration. Measured cost: ~3-5 min for 10 days at 80 bins, ~30-40 min for " + "60 days (BLOCKING-4)." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:99", + examples=[10, 60], + ) + + +class Injection(SchemaModel): + """What is released, and into what volume. + + ASSUMPTION-5, and it must be said in the UI too: **V0 does not enter the dynamics**. The model + is intensive and volume-invariant (``coupled/tests/test_boxvol_invariance.py``); the geometry + below exists only to turn an injected mass into an initial concentration. Presenting it as a + plume shape that the physics responds to would be a lie of layout. + + What t = 0 means -- engine exit plane or post-vortex-breakup -- is SCIENCE-2 (issue #54) and is + the most consequential open question in the project, because it moves the initial concentration + by orders of magnitude. + """ + + so2_mass_kg: float = SciField( + default=1000.0, + unit=Unit.KILOGRAM, + gt=0.0, + label="SO2 released", + description=( + "Mass of SO2 released into the initial plume volume. The ensemble uses 1 tonne." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Injection: 1 t)", + ) + plume_length_m: float = SciField( + default=15000.0, + unit=Unit.METRE, + gt=0.0, + label="Plume length", + description="Along-track length of the initial plume volume.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:45 (10 m x 10 m x 15 km)", + caveat="Only sets the initial concentration; the dynamics are volume-invariant.", + examples=[15000.0, 30000.0], + ) + plume_width_m: float = SciField( + default=10.0, + unit=Unit.METRE, + gt=0.0, + label="Plume width", + description="Cross-track width of the initial plume volume.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:45 (10 m x 10 m x 15 km)", + caveat="Only sets the initial concentration; the dynamics are volume-invariant.", + ) + plume_height_m: float = SciField( + default=10.0, + unit=Unit.METRE, + gt=0.0, + label="Plume height", + description="Vertical extent of the initial plume volume.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:45 (10 m x 10 m x 15 km)", + caveat="Only sets the initial concentration; the dynamics are volume-invariant.", + ) + plume_volume_cm3: float | None = SciField( + default=None, + unit=Unit.CM3, + label="Plume volume V0", + description=( + "Initial plume volume. Computed, not entered: the ensemble's 10 m x 10 m x 15 km gives " + "1.5e12 cm^3. Left unresolved by the schema -- task 0.5 owns the one cited " + "implementation, replacing the five copies that exist today with two different values." + ), + provenance=Provenance.DERIVED, + derived_from=[ + "injection.plume_length_m", + "injection.plume_width_m", + "injection.plume_height_m", + ], + ) + so2_initial_pptv: float | None = SciField( + default=None, + unit=Unit.PPTV, + label="Initial SO2", + description=( + "Initial plume SO2 mixing ratio. The ensemble fixes the NUMBER DENSITY " + "(6.27e15 molec cm^-3) and lets the pptv follow from the air density at this site, so " + "this depends on temperature and pressure as well as on mass and volume." + ), + provenance=Provenance.DERIVED, + derived_from=[ + "injection.so2_mass_kg", + "injection.plume_volume_cm3", + "site.temperature_k", + "site.pressure_mbar", + ], + ) + + +class Background(SchemaModel): + """The air the plume is diluted into, and the aerosol it entrains.""" + + aerosol: BackgroundAerosol = SciField( + default=BackgroundAerosol.SABR_220, + unit=Unit.DIMENSIONLESS, + label="Background aerosol", + description=( + "Background aerosol size distribution seeded into the initial TOMAS state and " + "entrained thereafter. The lognormal sets are digitized from source plots, not " + "published parameters." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_dilution_parameters.md (Background: SABRE aged air)", + ) + so2_pptv: float = SciField( + default=20.0, + unit=Unit.PPTV, + ge=0.0, + label="Background SO2", + description=( + "SO2 mixing ratio of the entrained background air. The plume's SO2 relaxes toward this " + "value rather than toward zero." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_dilution_parameters.md (Background: SO2 20 pptv)", + examples=[20.0, 100.0], + ) + gas_pptv: dict[str, float] = SciField( + default_factory=lambda: dict(PAPER_BACKGROUND_GAS_PPTV), + unit=Unit.PPTV, + label="Background gas composition", + description=( + "Initial plume gas composition, which is also the entrained background composition. " + "Species omitted start at zero. Names are validated at the model seam " + "(studio/modelio), not here, because the species list belongs to the model." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:56-57 (_BG_GAS_PPT)", + caveat=( + "Standing decision (Ali, 2026-07-08, run_ensemble.py:49-55): production runs should " + "initialise from a SPUN-UP control run, not this static list, which is retained only " + "to reproduce the existing 810-run ensemble." + ), + ) + + +class Dilution(SchemaModel): + """Plume expansion and entrainment of background air.""" + + regime: DilutionRegime = SciField( + default=DilutionRegime.D2, + unit=Unit.DIMENSIONLESS, + label="Dilution regime", + description=( + "Volume-expansion regime V(t)/V0 (Schumann et al. 1998 form). CONSTANT uses " + "dilution.rate_per_s instead of a curve." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_dilution_parameters.md (Med Kz (D2, default))", + ) + rate_per_s: float = SciField( + default=1.157e-6, + unit=Unit.PER_SECOND, + ge=0.0, + label="Constant dilution rate", + description=( + "First-order relaxation rate toward the background. IGNORED unless regime is CONSTANT; " + "the model ignores it silently, so a UI must grey it out rather than imply it applies." + ), + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:102", + ) + zero_species: tuple[str, ...] = SciField( + default=(), + unit=Unit.DIMENSIONLESS, + label="Species zeroed in the background", + description=( + "Gas species set to zero in the entrained background air; all others keep their " + "initial value. Empty means the background is the full initial composition." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:100 (dilution_zero_species=())", + ) + background_overrides_pptv: dict[str, float] = SciField( + default_factory=dict, + unit=Unit.PPTV, + label="Background overrides", + description=( + "Explicit background mixing ratios applied AFTER zero_species (an override wins over a " + "zero). background.so2_pptv is merged in here by studio/modelio, so SO2 need not be " + "repeated; this field is for any OTHER species." + ), + provenance=Provenance.CONVENTION, + ) + background_evolves: Literal[False] = SciField( + default=False, + unit=Unit.DIMENSIONLESS, + label="Background evolves", + description=( + "Whether the entrained background reservoir evolves photochemically. The model is " + "one-box with a STATIC background (driver.py:261-276), so False is the only accepted " + "value and any other fails validation rather than being quietly ignored." + ), + provenance=Provenance.CONVENTION, + caveat="Whether one box with a spun-up IC suffices is SCIENCE-5 (issue #57).", + ) + + +class Microphysics(SchemaModel): + """TOMAS sectional microphysics: resolution and the three sensitivity multipliers.""" + + n_bins: Literal[40, 80, 160] = SciField( + default=80, + unit=Unit.COUNT, + label="Size bins", + description=( + "TOMAS size resolution over a FIXED dry Dp range of 1.7 nm - 17.5 um. The mass " + "ratio is 2**(40/n_bins), so the top boundary is pinned; d_min/d_max/mass_doubling " + "are not selectable, whatever the spec implies." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (TOMAS, 80 size bins)", + ) + condensation_alpha: float = SciField( + default=1.0, + unit=Unit.DIMENSIONLESS, + gt=0.0, + le=1.0, + label="Condensation alpha", + description="Fuchs-Sutugin mass-accommodation coefficient for H2SO4 condensation.", + provenance=Provenance.PAPER_ENSEMBLE, + source=( + "coupled/paper_ensemble/TABLE_microphysics_parameters.md (Varied: alpha 0.5, **1.0**)" + ), + examples=[0.5, 1.0], + ) + nucleation_rate_scale: float = SciField( + default=1.0, + unit=Unit.DIMENSIONLESS, + ge=0.0, + label="Nucleation rate scale", + description=( + "Free multiplier on the Dunne et al. (2016) binary H2SO4-H2O nucleation rate (neutral " + "and ion-induced channels alike). 0 disables nucleation." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source=( + "coupled/paper_ensemble/TABLE_microphysics_parameters.md " + "(Varied: 0.01x, **1x**, 100x)" + ), + examples=[0.01, 1.0, 100.0], + ) + coag_kernel_scale: float = SciField( + default=1.0, + unit=Unit.DIMENSIONLESS, + ge=0.0, + label="Coagulation kernel scale", + description=( + "Free multiplier on the Brownian coagulation kernel (Fuchs non-continuum corrected)." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/TABLE_microphysics_parameters.md (Varied: 0.5x, **1x**, 2x)", + examples=[0.5, 1.0, 2.0], + ) + ion_pair_rate: float = SciField( + default=30.0, + unit=Unit.PER_CM3_PER_S, + ge=0.0, + label="Ion pair production rate", + description=( + "Ion-pair production rate feeding the Dunne (2016) ion-induced nucleation channels. " + "0 disables the ion-induced channels; the neutral ones are unaffected." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source=( + "coupled/paper_ensemble/TABLE_microphysics_parameters.md " + "(30 ion pairs cm^-3 s^-1, galactic cosmic rays at ~20 km)" + ), + caveat=( + "The MODEL defaults this to 0.0, which disables ion-induced nucleation entirely; the " + "ensemble value of 30 is used here instead. It is a bare constant with no derivation: " + "a cited function of altitude, latitude and solar-cycle phase is task 0.5, and until " + "one is agreed it stays a fixed number rather than a computed-looking one." + ), + ) + + +class Chemistry(SchemaModel): + """Gas-phase chemistry and photolysis.""" + + photolysis: PhotolysisMode = SciField( + default=PhotolysisMode.TUVX, + unit=Unit.DIMENSIONLESS, + label="Photolysis", + description=( + "Photolysis driver. Also gates the sulfur chain in the model today: SO2->SO3->H2SO4 is " + "active only when this is not REFERENCE." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:99 (photolysis='tuvx')", + ) + so2_ho2_rate: float = SciField( + default=1.0e-18, + unit=Unit.CM3_PER_MOLEC_PER_S, + ge=0.0, + label="SO2 + HO2 rate constant", + description="Rate constant for SO2 + HO2 -> SO3 + OH. Active only in the sulfur chain.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:130", + cite="JPL 19-5, reaction I34", + caveat=( + "JPL gives only an UPPER LIMIT (~1e-18) for this reaction and recommends no products, " + "so this is a deliberate sensitivity knob, not a measured rate. 0 removes the channel; " + "1e-18/1e-17/1e-16 scan the plausible range." + ), + examples=[0.0, 1e-18, 1e-17, 1e-16], + ) + + +class Numerics(SchemaModel): + """Time stepping. The two steps are coupled by a divisibility rule the model enforces.""" + + output_dt_s: float = SciField( + default=600.0, + unit=Unit.SECOND, + gt=0.0, + label="Output step", + description=( + "Interval at which state is recorded. NOTE for anyone reading results: outer intervals " + "snap to the terminator, so the ACTUAL mean step is ~592 s against this nominal 600 s. " + "Never reconstruct the time axis as i * dt; use the stored t." + ), + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:80", + ) + couple_dt_s: float = SciField( + default=600.0, + unit=Unit.SECOND, + gt=0.0, + label="Coupling step", + description=( + "Outer operator-split step: the cadence at which TUV-x J and aerosol optics are " + "recomputed and frozen. Within it, the gas/TOMAS/dilution coupling is resolved on an " + "adaptive micro-step, so this does not have to be small." + ), + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:85", + examples=[300.0, 600.0], + ) + + @model_validator(mode="after") + def _check_step_divisibility(self) -> Numerics: + """Mirror of ``coupled/coupled_scenario.py:200-204``. + + Mirrored rather than deferred so a form can reject the combination without importing the + model (ADR-002). If the model's rule changes, this citation is how you find this copy. + """ + if self.couple_dt_s > self.output_dt_s: + raise ValueError( + f"couple_dt_s ({self.couple_dt_s}) must be <= output_dt_s ({self.output_dt_s})" + ) + ratio = self.output_dt_s / self.couple_dt_s + if abs(ratio - round(ratio)) > 1e-9: + raise ValueError( + f"output_dt_s ({self.output_dt_s}) must be an integer multiple of couple_dt_s " + f"({self.couple_dt_s}); got a ratio of {ratio}" + ) + return self + + +class ProcessSwitches(SchemaModel): + """Per-process on/off flags. Mirrors ``coupled.coupled_scenario.Switches``.""" + + sulfur: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Sulfur chain", + description="Gas-phase SO2 -> SO3 -> H2SO4 chain.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:42", + ) + nucleation: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Nucleation", + description="TOMAS binary H2SO4-H2O nucleation.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:43", + ) + condensation: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Condensation", + description="TOMAS condensation of H2SO4 onto existing particles.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:44", + ) + coagulation: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Coagulation", + description="TOMAS Brownian coagulation.", + provenance=Provenance.MODEL_DEFAULT, + source="coupled/coupled_scenario.py:45", + ) + aerosol_to_j: bool = SciField( + default=False, + unit=Unit.DIMENSIONLESS, + label="Aerosol -> photolysis", + description="Feed the box aerosol's optics back into the TUV-x radiation field.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:106 (aerosol_to_j=False)", + caveat=( + "The MODEL defaults this on; the ensemble runs it off. The aerosol radiator in the " + "TUV-x port is approximate -- an exact one is deferred (see the repository's " + "validation status)." + ), + ) + heating_to_t: Literal[False] = SciField( + default=False, + unit=Unit.DIMENSIONLESS, + label="Radiative heating -> T", + description=( + "Let radiative heating change the box temperature. FALSE IS THE ONLY ACCEPTED VALUE: " + "the radiative calculation has no longwave component, so there is no temperature " + "feedback to enable. True fails validation rather than being quietly ignored." + ), + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:106 (heating_to_t=False)", + caveat=( + "Decision (Ali, 2026-08-13): heating and buoyancy are OUT OF SCOPE for this model, " + "not pending features. Longwave radiation is absent from the radiative calculation " + "(AD-5.4), so the heating term is shortwave-only: enabling it does not make the box " + "thermodynamics more complete, it makes them one-sided, giving a ~+1.2 K / 10 d drift " + "that is an artefact of the missing cooling. Answering whether the box should heat, or " + "rise, needs a DIFFERENT model with longwave radiation and plume dynamics. The MODEL " + "still defaults this on and every science script turns it off; Studio refuses it. " + "See SCIENCE-4 (issue #56)." + ), + ) + dilution: bool = SciField( + default=True, + unit=Unit.DIMENSIONLESS, + label="Dilution", + description="Plume dilution and entrainment of background gas and aerosol.", + provenance=Provenance.PAPER_ENSEMBLE, + source="coupled/paper_ensemble/run_ensemble.py:106 (dilution=True)", + ) + + +class Termination(SchemaModel): + """Limits that stop a run. A run stopped by one is never presented as converged.""" + + max_wall_time_s: float = SciField( + default=3600.0, + unit=Unit.SECOND, + gt=0.0, + label="Max wall time", + description=( + "Wall-clock cap enforced by the runner (task 0.6). 3600 s covers the measured 3-5 min " + "for a 10-day/80-bin case and 30-40 min for 60 days (BLOCKING-4), with headroom." + ), + provenance=Provenance.CONVENTION, + source="docs/studio/ASSUMPTIONS.md ASSUMPTION-4", + ) + max_sim_time_days: float | None = SciField( + default=None, + unit=Unit.DAY, + gt=0.0, + label="Max simulated time", + description=( + "Optional early stop in SIMULATED time. None means the run is bounded by " + "schedule.duration_days alone, which is the normal case -- this is not a second copy " + "of the duration, it is a lower ceiling for a run you expect to cut short." + ), + provenance=Provenance.CONVENTION, + caveat=( + "The model's stop_condition callback receives only (t1_seconds, wet_SA), so " + "multi-quantity termination criteria (e.g. on SO2 or number) are NOT available and " + "must raise rather than be approximated. See the capability register in " + "OPEN_QUESTIONS.md." + ), + ) + + +def _group(model: type[SchemaModel], description: str) -> Any: + """A field holding a nested group of scientific fields. + + Groups carry no unit, range or provenance of their own -- their leaves do -- so they use plain + ``Field``. The metadata-completeness test knows this and checks the leaves. + + Every group is defaultable, which is what makes ``RunConfig()`` with no arguments the paper + ensemble's golden case rather than a validation error. That property is load-bearing: it is the + starting point a form opens on, and the base of a RunSet. + """ + return Field(default_factory=model, description=description) + + +class RunConfig(SchemaModel): + """One simulation, fully described. + + Identity is the SHA-256 of the canonical JSON of THIS object (ADR-006), so anything that changes + the result belongs here and anything that does not must stay out. In particular there is no + ``label``, ``notes`` or ``output_dir`` field: a run's name is a property of the run, not of the + physics, and two runs whose only difference is a name are the same computation. + """ + + schema_version: Literal["0.2.0"] = SciField( + default=SCHEMA_VERSION, + unit=Unit.DIMENSIONLESS, + label="Schema version", + description=( + "Version of this schema. Part of the hashed identity: old configs are never silently " + "reinterpreted under new semantics." + ), + provenance=Provenance.CONVENTION, + ) + site: Site = _group(Site, "Location and thermodynamic state of the box.") + schedule: Schedule = _group(Schedule, "Release time and simulated duration.") + injection: Injection = _group(Injection, "What is released, and into what initial volume.") + background: Background = _group( + Background, "Composition and aerosol of the air being entrained." + ) + dilution: Dilution = _group(Dilution, "Plume expansion and entrainment.") + microphysics: Microphysics = _group( + Microphysics, "TOMAS resolution and sensitivity multipliers." + ) + chemistry: Chemistry = _group(Chemistry, "Gas-phase chemistry and photolysis.") + numerics: Numerics = _group(Numerics, "Time stepping.") + switches: ProcessSwitches = _group(ProcessSwitches, "Per-process on/off flags.") + termination: Termination = _group(Termination, "Limits that stop a run.") + + def canonical_json(self) -> str: + """Canonical JSON serialisation. See ``studio.schema.hashing``.""" + from studio.schema.hashing import canonical_json + + return canonical_json(self) + + def config_hash(self) -> str: + """Stable SHA-256 over the canonical JSON -- this config's identity (ADR-006).""" + from studio.schema.hashing import config_hash + + return config_hash(self) + + +#: Convenience alias for annotating "a path into a RunConfig", e.g. ``"microphysics.n_bins"``. +ConfigPath = Annotated[str, "dotted path into RunConfig"] + +__all__ = [ + "PAPER_BACKGROUND_GAS_PPTV", + "SCHEMA_VERSION", + "Background", + "Chemistry", + "ConfigPath", + "Dilution", + "Injection", + "Microphysics", + "Numerics", + "ProcessSwitches", + "RunConfig", + "Schedule", + "SchemaModel", + "Site", + "Termination", +] diff --git a/studio/schema/enums.py b/studio/schema/enums.py new file mode 100644 index 0000000..ec60d97 --- /dev/null +++ b/studio/schema/enums.py @@ -0,0 +1,104 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Closed choice sets, with the model's own string values wherever one exists. + +Where a member's value equals the model's string, ``studio/modelio`` passes it through unchanged and +there is nothing to get wrong. Exactly ONE member deviates -- ``DilutionRegime.CONSTANT`` -- and it +says so at the point of deviation, because that is where the 0.4 equivalence test has to account +for it. +""" + +from __future__ import annotations + +from enum import StrEnum + + +class PhotolysisMode(StrEnum): + """Photolysis driver. Values match ``coupled.coupled_scenario.PHOTOLYSIS_MODES`` exactly. + + Note the sulfur chain is gated on this being non-``reference`` in the model today + (``Env.sulfur_chain = photolysis != "reference"``), so it is not purely a radiation choice. + """ + + #: Fixed 45-degree J, on by day / off by night. Reproduces the original MATLAB model. + REFERENCE = "reference" + #: Reference J scaled by the real solar zenith angle. + SZA = "sza" + #: Absolute per-reaction J from the TUV-x port at the box altitude. The paper ensemble's choice. + TUVX = "tuvx" + + +class DilutionRegime(StrEnum): + """Plume volume-expansion regime (Schumann et al. 1998 form; ``coupled/dilution.py:39``). + + The four constant-Kz regimes share ``V(t)/V0 = max(1, t^0.8)`` for t <= 1e4 s and + ``1585 * exp[k (t - 1e4)^(3/2)]`` after, differing only in the turbulent-growth coefficient k. + ``BURST`` replaces the single exponential with a three-stage sequence (Kz = 10 m^2 s^-1 for + ~14 h), continuous at the breakpoints. See ``TABLE_dilution_parameters.md``. + """ + + #: Constant first-order rate from ``dilution.rate_per_s`` instead of a V(t) curve. + #: THE ONE VALUE THAT IS NOT THE MODEL'S STRING: the model spells this ``""`` (empty + #: ``dilution_regime``), which cannot be a sane dropdown key. ``studio/modelio`` maps + #: ``CONSTANT -> ""``, and the 0.4 equivalence test must cover this case explicitly. + CONSTANT = "constant" + #: Low Kz, k = 2.811e-9. + D1 = "D1" + #: Medium Kz, k = 8.89e-9. The paper ensemble's default regime. + D2 = "D2" + #: High Kz, k = 2.811e-8. + D3 = "D3" + #: Very high Kz, k = 5.33e-8. + D5 = "D5" + #: Transient burst of turbulence; three-stage, continuous at the breakpoints. + BURST = "burst" + + +class BackgroundAerosol(StrEnum): + """Background aerosol size distribution seeded into the initial TOMAS state. + + Values match ``coupled.tomas_bridge.BACKGROUND_MODES`` keys, plus ``redcircles`` (the tabulated + loader, which is the model's default and is not in that dict). + + The lognormal mode sets are DIGITIZED from source plots, and the number concentrations are + chosen so each mode's peak dN/dlogDp matches the value read off the plot -- the most reliable + digitized feature. They are approximations of a figure, not published parameters, which is why + every member below carries where it came from. + + Wet-vs-dry matters here and is per-dataset rather than declared: ``AER_GEO`` and ``CESM_G6_AMB`` + are specified at AMBIENT conditions and skip the STP->ambient factor on seeding, the others are + at STP. Making that an explicit field is SCIENCE-3 (issue #55). + """ + + #: Marianna's tabulated distribution. The model's default. + REDCIRCLES = "redcircles" + #: SABRE young air (high N2O), one mode, peak dN/dlogDp ~1000 cm^-3. + SABR_330 = "sabr_330" + #: SABRE mid air (310-320 ppbv N2O), peak ~320 cm^-3. + SABR_310 = "sabr_310" + #: SABRE aged air (220-230 ppbv N2O), Dg = 0.12 um, sigma_g = 1.6. The paper ensemble's clean + #: background, and the one used by the golden case. + SABR_220 = "sabr_220" + #: CESM G6 SAI, three modes, read at STP. + CESM_G6 = "cesm_g6" + #: CESM G6 with the source plot read as AMBIENT. Kept separate so ``CESM_G6`` stays + #: reproducible. + CESM_G6_AMB = "cesm_g6_amb" + #: AER 2D geoengineered stratosphere (Pierce et al., 5 Mt-S/yr, 95 nm case), ambient basis. + AER_GEO = "aer_geo" + + +class AxisKind(StrEnum): + """How a ``RunSet`` axis combines with the others. See ``studio/schema/runset.py``.""" + + #: Crossed with every other GRID/LIST axis (Cartesian product). + GRID = "grid" + #: Advanced in lockstep with the other ZIP axes; the zipped group is then crossed with the rest. + ZIP = "zip" + #: Like GRID, but each point sets SEVERAL fields at once -- a covarying group, e.g. the paper + #: ensemble's site axis, where latitude, T, p and H2O move together and only certain + #: combinations are physically meaningful. + LIST = "list" + + +__all__ = ["AxisKind", "BackgroundAerosol", "DilutionRegime", "PhotolysisMode"] diff --git a/studio/schema/export.py b/studio/schema/export.py new file mode 100644 index 0000000..1b1f271 --- /dev/null +++ b/studio/schema/export.py @@ -0,0 +1,76 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""JSON Schema export and the flat field catalogue. + +The web client generates its form from the exported JSON Schema (ADR-002) -- **nothing in the UI may +invent a field that does not exist here**. Pydantic emits the structure; ``SciField``'s metadata +rides along under ``x-studio`` because that is how ``json_schema_extra`` works, so the export needs +no parallel serialisation path that could drift from the models. + +``field_catalogue()`` is the flat view: dotted path -> metadata, for every leaf. It is what a +"what does this parameter mean, and where did its default come from?" panel reads, what the +dependency-graph engine (task 0.3) will walk to build the DAG from ``derived_from``, and what the +metadata-completeness test iterates. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +from pydantic import BaseModel + +from studio.schema.config import SCHEMA_VERSION, RunConfig +from studio.schema.fields import field_metadata + +#: Stable identifier for the exported schema. Versioned with the schema itself so a client can tell +#: which one it is holding. +SCHEMA_ID = f"https://reflective.org/studio/schemas/run-config/{SCHEMA_VERSION}.json" + + +def iter_leaf_fields( + model_cls: type[BaseModel] = RunConfig, prefix: str = "" +) -> Iterator[tuple[str, Any]]: + """Yield ``(dotted_path, FieldInfo)`` for every leaf field, descending into nested groups.""" + for name, info in model_cls.model_fields.items(): + path = f"{prefix}{name}" + annotation = info.annotation + if isinstance(annotation, type) and issubclass(annotation, BaseModel): + yield from iter_leaf_fields(annotation, prefix=f"{path}.") + else: + yield path, info + + +def field_catalogue(model_cls: type[BaseModel] = RunConfig) -> dict[str, dict[str, Any]]: + """Flat ``{dotted path: metadata}`` for every leaf field. + + The metadata is ``SciField``'s ``x-studio`` block plus the field's ``description``, ``default`` + and ``required`` flag -- everything needed to render and explain one input. + """ + catalogue: dict[str, dict[str, Any]] = {} + for path, info in iter_leaf_fields(model_cls): + entry = field_metadata(info) + entry["description"] = info.description + entry["required"] = info.is_required() + if not info.is_required(): + default = info.get_default(call_default_factory=True, validated_data=None) + entry["default"] = default.value if hasattr(default, "value") else default + catalogue[path] = entry + return catalogue + + +def run_config_json_schema() -> dict[str, Any]: + """The JSON Schema the web client consumes. + + ``by_alias=False`` because the schema has no aliases and the field names ARE the paths used by + ``RunSet`` axes; a client that reads a path here can use it there unchanged. + """ + schema = RunConfig.model_json_schema(by_alias=False, mode="serialization") + schema["$schema"] = "https://json-schema.org/draft/2020-12/schema" + schema["$id"] = SCHEMA_ID + schema["title"] = "Plume Studio run configuration" + schema["x-studio-schema-version"] = SCHEMA_VERSION + return schema + + +__all__ = ["SCHEMA_ID", "field_catalogue", "iter_leaf_fields", "run_config_json_schema"] diff --git a/studio/schema/fields.py b/studio/schema/fields.py new file mode 100644 index 0000000..ef93121 --- /dev/null +++ b/studio/schema/fields.py @@ -0,0 +1,169 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``SciField`` -- the field-metadata carrier the whole schema is built from (ADR-002). + +A schema field is not just a type and a default. It is a scientific quantity, and the thing that +makes it usable in a form -- or defensible in a paper -- is the metadata around it: what unit it is +in, what range is valid, and above all WHERE ITS DEFAULT CAME FROM. + +That last part is the point. The defaults in this project are not arbitrary: they are the paper +ensemble's configuration, recorded in ``TABLE_microphysics_parameters.md`` and +``TABLE_dilution_parameters.md``, or the model's own defaults in ``coupled/coupled_scenario.py``, or +values with a literature citation. A default with no recorded source is exactly the failure mode +``studio/CLAUDE.md`` exists to prevent, so ``provenance`` is REQUIRED and its consistency rules are +enforced at import time -- a bad field definition raises when the module is imported, not when a run +produces a quietly wrong number. + +Metadata lands in the exported JSON Schema under the ``x-studio`` key, which is what the web client +reads to generate a form. Everything below is data; nothing here computes physics. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from enum import StrEnum +from typing import Any, Final + +from pydantic import Field +from pydantic.fields import FieldInfo + +from studio.schema.units import Unit + + +class Provenance(StrEnum): + """Where a field's default came from. Required on every field; no default value. + + The distinction that matters is between a value someone MEASURED or PUBLISHED and a value + someone CHOSE. Both are legitimate; conflating them is not. + """ + + #: The model's own default, unchanged. ``source`` cites the file:line. + MODEL_DEFAULT = "model_default" + #: The paper ensemble's configuration. ``source`` cites the TABLE_*.md row or the runner line. + PAPER_ENSEMBLE = "paper_ensemble" + #: From the literature. ``cite`` is required. + LITERATURE = "literature" + #: A chosen convention with no external source -- an interface decision, not a scientific claim. + CONVENTION = "convention" + #: Computed from other fields. ``derived_from`` is required; the value is not supplied by hand. + DERIVED = "derived" + #: No defensible default exists. The user must supply it; there is no fallback (ADR-005). + USER_REQUIRED = "user_required" + + +#: JSON Schema extension key holding Studio's metadata. Namespaced with the conventional `x-` +#: prefix so a generic JSON Schema validator ignores it. +EXTENSION_KEY: Final = "x-studio" + +_SENTINEL: Final = object() + + +def SciField( + *, + unit: Unit, + description: str, + provenance: Provenance, + default: Any = _SENTINEL, + default_factory: Any = None, + label: str | None = None, + source: str | None = None, + cite: str | None = None, + derived_from: Sequence[str] = (), + caveat: str | None = None, + ge: float | None = None, + le: float | None = None, + gt: float | None = None, + lt: float | None = None, + examples: Sequence[Any] | None = None, +) -> Any: + """A pydantic field carrying Studio's scientific metadata. + + Args: + unit: Canonical unit (ADR-003). Use ``Unit.DIMENSIONLESS`` for pure scale factors. + description: What the quantity IS -- enough for someone who has not read the model. + provenance: Where the default came from. See ``Provenance``. + default: The default value. Omit for a required field. + default_factory: For mutable defaults (dicts, tuples), as in pydantic. + label: Short human-readable name for a form. Defaults to the field name at export time. + source: File:line or document reference backing the default. + cite: Literature citation. Required when ``provenance`` is ``LITERATURE``. + derived_from: Dotted paths this field is computed from. Required when ``DERIVED``, and + forbidden otherwise -- it is what the dependency graph in task 0.3 is built from. + caveat: A warning that must travel with the value into the UI (e.g. a one-sided physics + approximation). Surfaced, never hidden. + ge, le, gt, lt: Validity bounds, passed to pydantic AND recorded in the metadata. + examples: Illustrative values, e.g. the levels this field takes in the paper ensemble. + + Raises: + ValueError: If the metadata is internally inconsistent. Raised at import time, on purpose. + """ + if provenance is Provenance.LITERATURE and not cite: + raise ValueError("provenance=LITERATURE requires `cite`; a citation is the whole claim") + if provenance in (Provenance.MODEL_DEFAULT, Provenance.PAPER_ENSEMBLE) and not source: + raise ValueError( + f"provenance={provenance.value} requires `source` (file:line or TABLE_*.md row) -- " + f"the point of these two values is that the default is traceable" + ) + if provenance is Provenance.DERIVED: + if not derived_from: + raise ValueError("provenance=DERIVED requires `derived_from`") + if default is not _SENTINEL and default is not None: + raise ValueError( + "a DERIVED field must not carry a hand-written default; it is computed from " + f"{list(derived_from)} by the dependency-graph engine (task 0.3)" + ) + elif derived_from: + raise ValueError( + f"`derived_from` is only meaningful with provenance=DERIVED, got {provenance.value}" + ) + if provenance is Provenance.USER_REQUIRED and ( + default is not _SENTINEL or default_factory is not None + ): + raise ValueError( + "provenance=USER_REQUIRED means there is no defensible default, so it must not have one" + ) + if not description.strip(): + raise ValueError("description is required and must not be blank") + + extra: dict[str, Any] = { + "unit": unit.value, + "provenance": provenance.value, + "derived_from": list(derived_from), + } + for key, value in (("label", label), ("source", source), ("cite", cite), ("caveat", caveat)): + if value is not None: + extra[key] = value + bounds = { + name: v for name, v in (("ge", ge), ("le", le), ("gt", gt), ("lt", lt)) if v is not None + } + if bounds: + extra["range"] = bounds + + kwargs: dict[str, Any] = { + "description": description, + "json_schema_extra": {EXTENSION_KEY: extra}, + **bounds, + } + if examples is not None: + kwargs["examples"] = list(examples) + if default_factory is not None: + kwargs["default_factory"] = default_factory + elif default is not _SENTINEL: + kwargs["default"] = default + return Field(**kwargs) + + +def field_metadata(info: FieldInfo) -> dict[str, Any]: + """Studio metadata for a pydantic field, or ``{}`` if it was not declared with ``SciField``. + + Used by the metadata-completeness test and by the JSON Schema export; a caller that gets ``{}`` + is looking at a field that bypassed ``SciField``, which is a bug rather than a special case. + """ + extra = info.json_schema_extra + if not isinstance(extra, dict): + return {} + meta = extra.get(EXTENSION_KEY, {}) + return dict(meta) if isinstance(meta, dict) else {} + + +__all__ = ["EXTENSION_KEY", "Provenance", "SciField", "field_metadata"] diff --git a/studio/schema/hashing.py b/studio/schema/hashing.py new file mode 100644 index 0000000..6d81291 --- /dev/null +++ b/studio/schema/hashing.py @@ -0,0 +1,99 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Canonical serialisation and the config hash (ADR-006). + +A run's identity is the SHA-256 of the canonical JSON of its ``RunConfig``. That hash is also the +cache key, the golden-fixture key and half of the idempotence check, so **a hash that drifts +silently invalidates everything at once** -- and does it quietly, which is worse. Hence a canonical +form pinned by explicit rules and a test that asserts a known hash rather than merely asserting +self-consistency. + +Canonical form: + +1. ``model_dump(mode="json")`` -- enums become their string values, tuples become lists. +2. ``sort_keys=True`` -- insertion order cannot leak into identity. This is what makes two configs + built by different code paths hash the same. +3. ``separators=(",", ":")`` -- no incidental whitespace. +4. ``ensure_ascii=False`` with a UTF-8 encode -- one representation per string, not two. +5. ``allow_nan=False`` -- ``NaN``/``Infinity`` are not JSON and are not a valid configuration + either. This RAISES rather than emitting a non-standard token (ADR-005). + +Float formatting is Python's ``repr``, which has produced the shortest round-tripping decimal since +3.1 and is therefore stable across the versions this project supports. That is the one assumption +here that is a property of the interpreter rather than of this module, and the pinned-hash test is +what would catch it changing. + +There is no hashing anywhere else in the repository -- this is the first -- so nothing constrains +the choice except the need for it to never change silently. +""" + +from __future__ import annotations + +import hashlib +import json +from typing import Any + +from pydantic import BaseModel + +#: Named so a future change is a visible migration rather than an invisible one. If the canonical +#: form ever has to change, bump this, bump the schema version, and re-baseline the fixtures on +#: purpose -- never quietly. +CANONICAL_FORM_VERSION = 1 + + +def canonical_payload(model: BaseModel) -> dict[str, Any]: + """The JSON-mode dict that gets serialised. + + Exposed for tests, and for debugging a hash change: diffing two payloads says which field moved. + """ + payload = model.model_dump(mode="json") + if not isinstance(payload, dict): # pragma: no cover -- pydantic models always dump to a dict + raise TypeError(f"expected a dict from model_dump, got {type(payload).__name__}") + return payload + + +def canonical_json(model: BaseModel) -> str: + """Canonical JSON string for ``model``. + + Raises: + ValueError: If the config contains NaN or Infinity, which are neither valid JSON nor a valid + configuration. Failing here beats writing a token no other parser will read back. + """ + try: + return json.dumps( + canonical_payload(model), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ) + except ValueError as exc: + raise ValueError( + f"{type(model).__name__} is not canonically serialisable: {exc}. NaN and Infinity are " + f"not valid JSON and not a valid configuration; fix the value rather than the encoder." + ) from exc + + +def config_hash(model: BaseModel) -> str: + """Stable SHA-256 (hex) over the canonical JSON of ``model``.""" + return hashlib.sha256(canonical_json(model).encode("utf-8")).hexdigest() + + +def short_hash(model: BaseModel, length: int = 12) -> str: + """First ``length`` hex characters of the config hash, for display and directory names. + + Display only. Twelve hex characters is ~48 bits, fine for a human reading a list and not fine as + an identity; equality checks use the full hash. + """ + if not 4 <= length <= 64: + raise ValueError(f"length must be in [4, 64], got {length}") + return config_hash(model)[:length] + + +__all__ = [ + "CANONICAL_FORM_VERSION", + "canonical_json", + "canonical_payload", + "config_hash", + "short_hash", +] diff --git a/studio/schema/runset.py b/studio/schema/runset.py new file mode 100644 index 0000000..6778089 --- /dev/null +++ b/studio/schema/runset.py @@ -0,0 +1,283 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSet`` -- the primary user-facing object. A single run is the N = 1 case of it. + +**There is no separate single-run code path.** A run with no sweep is a ``RunSet`` with zero axes, +and it goes through exactly the same expansion. Bolting sweeps on later would mean rewriting the +config layer, the results schema and every comparison view, which is why this exists in the first +version of the schema rather than the third. + +The shape is taken from what the paper ensemble already does. Its 810 cases are +``itertools.product`` over six axes (``run_ensemble.py:82``), and its case IDs are the axis LABELS +joined by ``__`` (``:84``) -- e.g. ``30N_20km__sabr220__D2med__a1p0__nuc1__cg1``. That naming is +genuinely good design for a fixed factorial and it is preserved here: an axis point carries a label, +and the expanded run's label is the join. Identity is still the config hash (ADR-006); the label is +for people, and for the existing directory layout. + +Three axis kinds: + +* ``GRID`` -- crossed with every other GRID/LIST axis. +* ``LIST`` -- also crossed, but each point assigns SEVERAL fields at once. The paper ensemble's site + axis is exactly this: ``("30N_20km", 30.0, 210.0, 55.0, 6.9104)`` moves latitude, T, p and H2O + together, and the intermediate combinations are not physically meaningful. +* ``ZIP`` -- advanced in lockstep with the other ZIP axes; the zipped group is then crossed with the + GRID/LIST axes as a single pseudo-axis, positioned where the first ZIP axis was declared. + +Ordering is deterministic and matches ``itertools.product``: axes vary right-to-left, the LAST axis +fastest. This is not an implementation detail -- it is what makes an expansion reproducible and what +lets a RunSet reproduce the existing ensemble's case order exactly. +""" + +from __future__ import annotations + +import itertools +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from studio.schema.config import RunConfig, SchemaModel +from studio.schema.enums import AxisKind + + +def resolve_path(model_cls: type[BaseModel], path: str) -> None: + """Validate that ``path`` names a real LEAF field, or raise. + + Fails loud and early on both ways of getting it wrong: an axis over ``"microphysics.n_bin"`` is + a typo that would otherwise surface as a sweep whose axis silently never varied, and an axis + over ``"microphysics"`` would assign a whole group at once -- which the model would accept but + which has no unit, no provenance and no place in the dependency graph task 0.3 builds from leaf + paths. Groups are containers; values live on leaves. + """ + parts = path.split(".") + if not all(parts): + raise ValueError(f"malformed path {path!r}") + current: type[BaseModel] = model_cls + for i, part in enumerate(parts): + fields = getattr(current, "model_fields", None) + if fields is None or part not in fields: + known = sorted(fields) if fields else [] + where = ".".join(parts[:i]) or model_cls.__name__ + raise ValueError(f"unknown field {part!r} in {where}; known fields: {known}") + annotation = fields[part].annotation + is_group = isinstance(annotation, type) and issubclass(annotation, BaseModel) + if i < len(parts) - 1: + if not is_group: + raise ValueError( + f"{'.'.join(parts[: i + 1])} is a leaf field; {path!r} tries to descend into it" + ) + current = annotation + elif is_group: + raise ValueError( + f"{path!r} is a group of fields, not a leaf; assign its leaves individually " + f"({', '.join(f'{path}.{name}' for name in sorted(annotation.model_fields))})" + ) + + +def apply_assignments(config: RunConfig, assignments: dict[str, Any]) -> RunConfig: + """Return a new ``RunConfig`` with ``assignments`` applied. + + Re-validates through the model rather than mutating: configs are frozen (ADR-004), and an axis + value that violates a bound or a cross-field rule must fail here, at expansion time, rather than + at submit time for run 407 of 810. + """ + payload = config.model_dump(mode="python") + for path, value in assignments.items(): + resolve_path(RunConfig, path) + parts = path.split(".") + cursor: dict[str, Any] = payload + for part in parts[:-1]: + cursor = cursor[part] + cursor[parts[-1]] = value + return RunConfig.model_validate(payload) + + +class AxisPoint(SchemaModel): + """One level of an axis: a short label and the field assignments it stands for.""" + + label: str = Field( + description=( + "Short token used to build the run label, e.g. 'sabr220' or 'a1p0'. Kept terse because " + "it becomes part of a directory name, following the existing ensemble's convention." + ), + min_length=1, + ) + assignments: dict[str, Any] = Field( + description="Dotted RunConfig paths to values, applied together as one point.", + min_length=1, + ) + + +class Axis(SchemaModel): + """One dimension of a sweep.""" + + name: str = Field(description="Axis name, for display and for error messages.", min_length=1) + kind: AxisKind = Field(default=AxisKind.GRID, description="How this axis combines with others.") + points: tuple[AxisPoint, ...] = Field(description="The levels of this axis.", min_length=1) + + @model_validator(mode="after") + def _check_points(self) -> Axis: + labels = [point.label for point in self.points] + duplicates = sorted({label for label in labels if labels.count(label) > 1}) + if duplicates: + raise ValueError( + f"axis {self.name!r} has duplicate point labels {duplicates}; labels become run " + f"labels and directory names, so they must be unique within an axis" + ) + for point in self.points: + for path in point.assignments: + resolve_path(RunConfig, path) + if self.kind is not AxisKind.LIST: + multi = [p.label for p in self.points if len(p.assignments) > 1] + if multi: + raise ValueError( + f"axis {self.name!r} is {self.kind.value.upper()} but points {multi} assign " + f"more than one field; a covarying group is what LIST is for" + ) + paths = {path for point in self.points for path in point.assignments} + if len(paths) > 1: + raise ValueError( + f"axis {self.name!r} varies {sorted(paths)}; a {self.kind.value.upper()} axis " + f"varies exactly one field. Use LIST for a covarying group." + ) + return self + + @classmethod + def over( + cls, + name: str, + path: str, + levels: dict[str, Any], + kind: AxisKind = AxisKind.GRID, + ) -> Axis: + """Build a single-field axis from ``{label: value}``. The common case, spelled short.""" + return cls( + name=name, + kind=kind, + points=tuple( + AxisPoint(label=label, assignments={path: value}) for label, value in levels.items() + ), + ) + + +class ExpandedRun(BaseModel): + """One concrete run produced by expanding a ``RunSet``.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + #: Axis point labels joined by ``__``, matching the existing ensemble's case IDs. Empty for a + #: RunSet with no axes -- identity is the hash, so an unlabelled run is not an anonymous one. + label: str + #: The axis point label per axis name, so a comparison view can group by axis without re-parsing + #: the label string (``_tokens()`` in make_paper_candidate_plots.py:92 exists because that + #: re-parsing is otherwise necessary). + coordinates: dict[str, str] + config: RunConfig + + @property + def config_hash(self) -> str: + """This run's identity (ADR-006).""" + return self.config.config_hash() + + +class RunSet(SchemaModel): + """A base configuration plus the axes to sweep over it. + + ``expand()`` is the only way to get runs out, including when there are no axes. + """ + + base: RunConfig = Field( + default_factory=RunConfig, description="The configuration every run starts from." + ) + axes: tuple[Axis, ...] = Field( + default=(), description="Sweep axes; empty means a single run (N = 1)." + ) + + @model_validator(mode="after") + def _check_axes(self) -> RunSet: + names = [axis.name for axis in self.axes] + duplicates = sorted({name for name in names if names.count(name) > 1}) + if duplicates: + raise ValueError(f"duplicate axis names {duplicates}") + zipped = [axis for axis in self.axes if axis.kind is AxisKind.ZIP] + if zipped: + lengths = {axis.name: len(axis.points) for axis in zipped} + if len(set(lengths.values())) > 1: + raise ValueError( + f"ZIP axes are advanced in lockstep and must have equal length, got {lengths}" + ) + assigned: dict[str, str] = {} + for axis in self.axes: + for path in {p for point in axis.points for p in point.assignments}: + if path in assigned and assigned[path] != axis.name: + raise ValueError( + f"axes {assigned[path]!r} and {axis.name!r} both assign {path!r}; the " + f"result would depend on axis order, so it is rejected rather than ordered" + ) + assigned[path] = axis.name + return self + + def size(self) -> int: + """Number of runs ``expand()`` will produce, WITHOUT building any of them. + + The existing runners' ``plan`` verb prints a count before committing compute; this is the + equivalent, and it stays cheap no matter how large the sweep is. + """ + crossed = [axis for axis in self.axes if axis.kind is not AxisKind.ZIP] + zipped = [axis for axis in self.axes if axis.kind is AxisKind.ZIP] + total = 1 + for axis in crossed: + total *= len(axis.points) + if zipped: + total *= len(zipped[0].points) + return total + + def expand(self) -> list[ExpandedRun]: + """Every run in this set, in a deterministic order (last axis varies fastest).""" + groups, order = self._axis_groups() + runs: list[ExpandedRun] = [] + for combination in itertools.product(*groups): + assignments: dict[str, Any] = {} + coordinates: dict[str, str] = {} + for axes_in_group, points in zip(order, combination, strict=True): + for axis, point in zip(axes_in_group, points, strict=True): + assignments.update(point.assignments) + coordinates[axis.name] = point.label + label = "__".join( + coordinates[axis.name] for axis in self.axes if axis.name in coordinates + ) + runs.append( + ExpandedRun( + label=label, + coordinates=coordinates, + config=apply_assignments(self.base, assignments) if assignments else self.base, + ) + ) + return runs + + def _axis_groups(self) -> tuple[list[list[tuple[AxisPoint, ...]]], list[list[Axis]]]: + """Axes as product operands, preserving declaration order. + + Each operand is a list of "point tuples": one point per axis in that group. Crossed axes + form single-axis groups; all ZIP axes form ONE group whose points advance together, placed + where the first ZIP axis was declared. + """ + zipped = [axis for axis in self.axes if axis.kind is AxisKind.ZIP] + groups: list[list[tuple[AxisPoint, ...]]] = [] + order: list[list[Axis]] = [] + zip_emitted = False + for axis in self.axes: + if axis.kind is AxisKind.ZIP: + if zip_emitted: + continue + zip_emitted = True + groups.append( + [tuple(points) for points in zip(*(a.points for a in zipped), strict=True)] + ) + order.append(zipped) + else: + groups.append([(point,) for point in axis.points]) + order.append([axis]) + return groups, order + + +__all__ = ["Axis", "AxisPoint", "ExpandedRun", "RunSet", "apply_assignments", "resolve_path"] diff --git a/studio/schema/units.py b/studio/schema/units.py new file mode 100644 index 0000000..77502af --- /dev/null +++ b/studio/schema/units.py @@ -0,0 +1,80 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The canonical unit registry. + +Canonical units are the MODEL's native units, not SI (ADR-003, ASSUMPTION-1): mbar, ppmv, pptv, K, +s, um^2 cm^-3, cm^-3 s^-1. The reason is float identity at the model seam -- the existing ensemble's +constants are decimals in native units (``WTR = 6.9104`` ppm at ``run_ensemble.py:62``) and +round-tripping them through SI is not guaranteed to return the same float64. Reproducing trusted +runs is worth more than SI purity here. + +Every dimensioned schema field declares one of these symbols. The set is CLOSED: a unit that is not +listed cannot be used, which turns a typo (``"ppvt"``) into an import-time error instead of a +silently unconvertible field. + +``pint`` is used for display conversion at the presentation boundary and for round-trip property +tests -- never inside the model interface, where ``studio/modelio`` hands ``CoupledScenario`` plain +floats already in native units. + +Three units are deliberately NOT pint-parseable, and say so rather than being faked: + +* ``ppmv`` / ``pptv`` are mixing ratios by VOLUME (mole fraction). pint would treat a bare + ``1e-12`` as dimensionless, losing the by-volume convention, and the conversion to a number + density depends on T and p -- which is a derivation (``studio/science``), not a unit conversion. +* ``molec`` is a count of molecules. pint has no such unit; ``cm^3 molec^-1 s^-1`` is the standard + bimolecular rate-constant unit and is carried symbolically. +""" + +from __future__ import annotations + +from enum import StrEnum +from typing import Final + + +class Unit(StrEnum): + """Canonical unit symbols. The string value is what appears in the exported JSON Schema.""" + + DIMENSIONLESS = "1" + KELVIN = "K" + MBAR = "mbar" + PPMV = "ppmv" + PPTV = "pptv" + DEGREE = "degree" + SECOND = "s" + DAY = "d" + HOUR = "h" + METRE = "m" + KILOGRAM = "kg" + CM3 = "cm^3" + PER_CM3_PER_S = "cm^-3 s^-1" + CM3_PER_MOLEC_PER_S = "cm^3 molec^-1 s^-1" + UM2_PER_CM3 = "um^2 cm^-3" + PER_SECOND = "s^-1" + COUNT = "count" + + +#: Canonical symbol -> the equivalent ``pint`` expression, or ``None`` where no faithful one exists. +#: A ``None`` here is a statement that the quantity carries a convention pint cannot represent, not +#: an omission -- see the module docstring. Checked exhaustively by the unit tests, so adding a +#: member to ``Unit`` without adding it here is a test failure rather than a runtime surprise. +PINT_EXPRESSION: Final[dict[Unit, str | None]] = { + Unit.DIMENSIONLESS: "dimensionless", + Unit.KELVIN: "kelvin", + Unit.MBAR: "millibar", + Unit.PPMV: None, # mole fraction x 1e6; by-volume convention, T/p-dependent to a number density + Unit.PPTV: None, # mole fraction x 1e12; likewise + Unit.DEGREE: "degree", + Unit.SECOND: "second", + Unit.DAY: "day", + Unit.HOUR: "hour", + Unit.METRE: "meter", + Unit.KILOGRAM: "kilogram", + Unit.CM3: "centimeter ** 3", + Unit.PER_CM3_PER_S: "1 / centimeter ** 3 / second", + Unit.CM3_PER_MOLEC_PER_S: None, # `molec` is a molecule count; pint has no such unit + Unit.UM2_PER_CM3: "micrometer ** 2 / centimeter ** 3", + Unit.PER_SECOND: "1 / second", + Unit.COUNT: None, # a plain count of things (size bins); dimensionless but not a ratio +} + +__all__ = ["PINT_EXPRESSION", "Unit"] diff --git a/studio/science/__init__.py b/studio/science/__init__.py index c67e544..4f13ee7 100644 --- a/studio/science/__init__.py +++ b/studio/science/__init__.py @@ -2,33 +2,86 @@ # SPDX-License-Identifier: Apache-2.0 """Scientific derivations: plume volume, initial concentration, size-distribution reductions, GCR. -Purpose is consolidation as much as new code. Several of these derivations already exist in the -repository three to five times over, with drifting constants -- one implementation each, cited and -tested (task 0.5): +Consolidation as much as new code. Each derivation below existed several times over in this +repository, and this is the one cited, tested implementation: -* V0 and injected mass -> initial concentration. Five copies today, with two different V0 values - (``run_ensemble.py:41-46`` uses a 15 km track, ``run_dilution_d1_clean.py:61`` a 30 km one). -* dN/dlogDp and bin diameter edges. Four copies, two mid-point expressions. +* **Plume volume and injected mass -> initial concentration** (``plume.py``). Six copies today, and + they disagree: the ensemble uses a 15 km track and the D1 flagship a 30 km one -- a factor of two + in V0 for the same injected mass. Which is right depends on what t = 0 means (SCIENCE-2, #54). +* **dN/dlogDp and bin mid-points** (``size_distribution.py``). Four copies, written two ways which + turn out to be algebraically identical -- measured agreement to a few ULP. +* **Air number density** (``air.py``). A deliberate MIRROR of the model's, because this package may + not import the model; a test asserts they agree exactly. +* **GCR ion-pair rate** (``gcr.py``). Raises ``NotImplementedError``: the value in use is uncited + and the honest thing is to say so (SCIENCE-6, #63). -Reused rather than rewritten -- these are already correct and tested: +**Reused, not rewritten.** These are already implemented and tested in the model, so Studio reaches +them through ``studio/modelio`` (the only package allowed to import ``coupled``) rather than keeping +a second copy here: -* ``coupled.dilution.volume_ratio`` / ``kdil_from_regime`` (plume expansion V(t)/V0 and k_dil) -* ``config.air_number_density`` (stratchem-jax) -* ``coupled.aerosol_props`` (surface area, effective wet radius, H2SO4 weight percent) -* ``coupled.units`` (mass <-> number density; note the DELIBERATE ~0.036% Avogadro mismatch at the - gas/TOMAS seam documented at ``coupled/units.py:18-22`` -- inherited, not silently "corrected") +* ``coupled.dilution.volume_ratio`` / ``kdil_from_regime`` -- plume expansion V(t)/V0 and k_dil +* ``coupled.aerosol_props`` -- surface area, effective wet radius, H2SO4 weight percent +* ``coupled.units`` -- number density <-> mass per grid cell -Genuinely new: a galactic-cosmic-ray ion-pair parameterisation. Today ``ion_pair_rate`` is a bare -30.0 with no derivation. The default should not be zero and should depend on altitude, latitude and -solar-cycle phase -- but absent an agreed citation it raises ``NotImplementedError`` rather than -returning a plausible number (ADR-005). - -No magic numbers: physical constants live in one module with sources, and any numeric literal in -scientific code needs a named constant and a citation. +Every physical constant lives in ``constants.py`` with its source, including the two that are +deliberately the model's rounded values rather than the best-known ones -- Studio inherits the +model's constants so that Phase 0 can reproduce its runs, and says so at the point of use. This package must not import ``coupled``, the API, or the database (see ``studio/__init__.py``). """ from __future__ import annotations -__all__: list[str] = [] +from studio.science.air import air_number_density +from studio.science.constants import ( + AIR_NUMBER_DENSITY_COEFF, + AVOGADRO, + AVOGADRO_GAS_MODEL, + AVOGADRO_SEAM_RELATIVE_DIFFERENCE, + CM3_PER_M3, + G_PER_KG, + H2SO4_MOLAR_MASS_G_PER_MOL, + MBAR_TO_TORR, + PPMV_PER_MOLE_FRACTION, + PPTV_PER_MOLE_FRACTION, + SO2_MOLAR_MASS_G_PER_MOL, +) +from studio.science.gcr import ( + MODEL_DEFAULT_ION_PAIR_RATE, + PAPER_ENSEMBLE_ION_PAIR_RATE, + ion_pair_production_rate, +) +from studio.science.plume import ( + initial_mixing_ratio_pptv, + injected_number_density, + number_density_to_pptv, + plume_volume_cm3, + pptv_to_number_density, +) +from studio.science.size_distribution import bin_midpoints_um, dlog10_dp, dn_dlogdp + +__all__ = [ + "AIR_NUMBER_DENSITY_COEFF", + "AVOGADRO", + "AVOGADRO_GAS_MODEL", + "AVOGADRO_SEAM_RELATIVE_DIFFERENCE", + "CM3_PER_M3", + "G_PER_KG", + "H2SO4_MOLAR_MASS_G_PER_MOL", + "MBAR_TO_TORR", + "MODEL_DEFAULT_ION_PAIR_RATE", + "PAPER_ENSEMBLE_ION_PAIR_RATE", + "PPMV_PER_MOLE_FRACTION", + "PPTV_PER_MOLE_FRACTION", + "SO2_MOLAR_MASS_G_PER_MOL", + "air_number_density", + "bin_midpoints_um", + "dlog10_dp", + "dn_dlogdp", + "initial_mixing_ratio_pptv", + "injected_number_density", + "ion_pair_production_rate", + "number_density_to_pptv", + "plume_volume_cm3", + "pptv_to_number_density", +] diff --git a/studio/science/air.py b/studio/science/air.py new file mode 100644 index 0000000..5877ab9 --- /dev/null +++ b/studio/science/air.py @@ -0,0 +1,38 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Air number density. + +This is the one function here that duplicates a model function rather than reusing it, and the +duplication is forced: ``studio.science`` may not import ``coupled`` or the model's flat modules +(ADR-001), because the packages must be usable from a bare Python session and importing the model +costs a JAX import. + +So it is a MIRROR, not a fork. It is one line, it cites the line it mirrors, and +``studio/tests/unit/test_science_air.py`` asserts that the two agree exactly whenever the +``stratchem-jax`` submodule is checked out. If the model's relation ever changes, that test fails -- +which is the property that makes a mirror acceptable and a quiet copy not. +""" + +from __future__ import annotations + +from studio.science.constants import AIR_NUMBER_DENSITY_COEFF, MBAR_TO_TORR + + +def air_number_density(pressure_mbar: float, temperature_k: float) -> float: + """Air number density M [molec cm^-3] at ``pressure_mbar`` and ``temperature_k``. + + Mirrors ``stratchem-jax/config.py:55`` (``M = 9.65e18*P/T*conv``), itself a port of the MATLAB + ``runconcs_het.m``. The coefficient folds in the mbar->torr conversion. + + Raises: + ValueError: On a non-positive pressure or temperature. There is no sensible fallback: a + zero temperature is a division by zero and a negative pressure is not a state (ADR-005). + """ + if pressure_mbar <= 0.0: + raise ValueError(f"pressure must be > 0 mbar, got {pressure_mbar}") + if temperature_k <= 0.0: + raise ValueError(f"temperature must be > 0 K, got {temperature_k}") + return AIR_NUMBER_DENSITY_COEFF * pressure_mbar / temperature_k * MBAR_TO_TORR + + +__all__ = ["air_number_density"] diff --git a/studio/science/constants.py b/studio/science/constants.py new file mode 100644 index 0000000..f5ac1ae --- /dev/null +++ b/studio/science/constants.py @@ -0,0 +1,83 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Physical constants, in one place, each with its source. + +``studio/CLAUDE.md``: *no magic numbers -- physical constants live in one module with sources, and +any numeric literal in scientific code needs a named constant and a citation.* + +Two of the values here are deliberately NOT their best-known values, and that is the interesting +part of this module. The model uses rounded constants in places, and Studio's job in Phase 0 is to +reproduce the model's existing runs exactly. A silent "correction" here would shift every derived +initial concentration by a small amount and make it impossible to tell a Studio bug from a model +change -- the same reasoning that made the canonical units the model's native ones (ADR-003). + +Each such value says so, in place, so the choice is visible at the point of use rather than +discoverable by whoever eventually diffs a result. +""" + +from __future__ import annotations + +from typing import Final + +#: Avogadro constant [molecules / mol]. CODATA 2019 exact value, and what ``coupled/units.py:23`` +#: uses so its gas<->TOMAS bridge is self-consistent with TOMAS. The paper ensemble uses this same +#: value for the injected-mass conversion (``run_ensemble.py:39``), so Studio must too. +AVOGADRO: Final = 6.02214076e23 + +#: The GAS-PHASE model's Avogadro constant -- rounded (``mechanism.khet``, +#: ``config.air_number_density``). Recorded, never used here. Molecule<->mole conversions therefore +#: differ by ~0.036 % between the two halves of the model. ``coupled/units.py:18-22`` documents this +#: as a conscious seam, not drift; Studio INHERITS it and does not silently reconcile it. Anything +#: that needs the gas side's convention must say so explicitly. +AVOGADRO_GAS_MODEL: Final = 6.02e23 + +#: Relative size of that seam, as a fraction. Stated as a number so a tolerance can cite it rather +#: than hard-coding "about 4e-4" somewhere downstream. +AVOGADRO_SEAM_RELATIVE_DIFFERENCE: Final = (AVOGADRO - AVOGADRO_GAS_MODEL) / AVOGADRO + +#: Molar mass of SO2 [g / mol] AS THE MODEL USES IT (``run_ensemble.py:44``). The true value is +#: 64.066 g/mol; the ensemble's 64.0 is a 0.10 % difference. Kept rounded because the golden runs +#: were produced with it -- see the module docstring. +SO2_MOLAR_MASS_G_PER_MOL: Final = 64.0 + +#: Molar mass of H2SO4 [g / mol] as the model uses it (``coupled/run_dilution_d1_clean.py:598``). +#: True value 98.079; the 0.08 % difference is inherited for the same reason. +H2SO4_MOLAR_MASS_G_PER_MOL: Final = 98.0 + +#: Coefficient of the air-number-density relation, [molec K / (cm^3 torr)]. From the MATLAB +#: ``runconcs_het.m`` line ``M = 9.65e18*P/T*conv``, ported at ``stratchem-jax/config.py:55``. +#: It is the ideal-gas law with the torr/kelvin units folded in; it is NOT independently derived +#: here, because the whole point is to match what the gas model uses. +AIR_NUMBER_DENSITY_COEFF: Final = 9.65e18 + +#: Millibar -> torr, the ``conv`` factor inside that same MATLAB expression +#: (``stratchem-jax/config.py:52``). +MBAR_TO_TORR: Final = 760.0 / 1013.25 + +#: Cubic centimetres per cubic metre. Named because plume geometry is entered in metres and the +#: model's volumes are in cm^3, and that factor of 1e6 is exactly the kind of literal that ends up +#: wrong once. +CM3_PER_M3: Final = 1.0e6 + +#: Parts per trillion by volume per unit mole fraction. +PPTV_PER_MOLE_FRACTION: Final = 1.0e12 + +#: Parts per million by volume per unit mole fraction. +PPMV_PER_MOLE_FRACTION: Final = 1.0e6 + +#: Grams per kilogram. +G_PER_KG: Final = 1000.0 + +__all__ = [ + "AIR_NUMBER_DENSITY_COEFF", + "AVOGADRO", + "AVOGADRO_GAS_MODEL", + "AVOGADRO_SEAM_RELATIVE_DIFFERENCE", + "CM3_PER_M3", + "G_PER_KG", + "H2SO4_MOLAR_MASS_G_PER_MOL", + "MBAR_TO_TORR", + "PPMV_PER_MOLE_FRACTION", + "PPTV_PER_MOLE_FRACTION", + "SO2_MOLAR_MASS_G_PER_MOL", +] diff --git a/studio/science/gcr.py b/studio/science/gcr.py new file mode 100644 index 0000000..57c32de --- /dev/null +++ b/studio/science/gcr.py @@ -0,0 +1,77 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Galactic-cosmic-ray ion-pair production rate. + +**This module deliberately does not compute anything.** It is the clearest case in the project of +the rule in ``studio/CLAUDE.md``: a number that is needed, that has no agreed source, and that would +be trivially easy to fabricate convincingly. + +What exists today is a bare ``30.0`` cm^-3 s^-1 in the paper ensemble (``run_ensemble.py:102``), +described in ``TABLE_microphysics_parameters.md`` as "galactic cosmic rays at ~20 km" with no +citation; and the model's own default of ``0.0``, which switches off ion-induced nucleation +altogether (``coupled/coupled_scenario.py:117``). So the two available values are an uncited +constant and a value that disables a physical process. + +The rate genuinely varies -- roughly a factor of two over the solar cycle, and strongly with +latitude (geomagnetic cutoff) and altitude. A function of those three arguments is the right shape. +Inventing its coefficients is not, because the result would arrive with the same confidence as a +computed one and feed the most sensitive part of the system: the ion-induced channels of Dunne et +al. (2016) nucleation. + +So :func:`ion_pair_production_rate` raises, and :data:`PAPER_ENSEMBLE_ION_PAIR_RATE` is available +for anyone who wants the ensemble's constant *as a constant*, with its provenance attached. + +Tracked as SCIENCE-6, issue #63. +""" + +from __future__ import annotations + +from typing import Final + +#: The paper ensemble's fixed value [ion pairs cm^-3 s^-1] (``run_ensemble.py:102``; +#: ``TABLE_microphysics_parameters.md``, "30 ion pairs cm^-3 s^-1, galactic cosmic rays at ~20 km"). +#: Uncited. Use it to reproduce the ensemble, not as a general-purpose value -- it is a single +#: number for a quantity that varies with altitude, latitude and solar-cycle phase. +PAPER_ENSEMBLE_ION_PAIR_RATE: Final = 30.0 + +#: The MODEL's default, which disables the ion-induced nucleation channels entirely +#: (``coupled/coupled_scenario.py:117``). Recorded so that "the default is 0" is discoverable here +#: rather than surprising someone whose nucleation quietly lost a channel. +MODEL_DEFAULT_ION_PAIR_RATE: Final = 0.0 + +_SCIENCE_6 = ( + "SCIENCE-6 (issue #63): the GCR ion-pair production rate has no agreed parameterisation. " + "The paper ensemble uses a fixed, uncited 30.0 cm^-3 s^-1 at ~20 km and the model defaults to " + "0.0, which disables ion-induced nucleation entirely." +) + + +def ion_pair_production_rate( + altitude_km: float, latitude_deg: float, solar_cycle_phase: float +) -> float: + """Ion-pair production rate [cm^-3 s^-1]. **Not implemented, by decision.** + + Args: + altitude_km: Box altitude. + latitude_deg: Geographic latitude; the geomagnetic cutoff rigidity, and therefore the + ionisation rate, is a strong function of it. + solar_cycle_phase: Phase in [0, 1], 0 at solar minimum (maximum GCR flux). + + Raises: + NotImplementedError: Always. Absent a citable parameterisation, returning a plausible number + would be worse than failing: it would look computed. Use + :data:`PAPER_ENSEMBLE_ION_PAIR_RATE` explicitly if you want the ensemble's constant. + """ + raise NotImplementedError( + f"ion_pair_production_rate(altitude_km={altitude_km}, latitude_deg={latitude_deg}, " + f"solar_cycle_phase={solar_cycle_phase}) is not implemented. {_SCIENCE_6} Set " + f"microphysics.ion_pair_rate explicitly -- studio.science.gcr." + f"PAPER_ENSEMBLE_ION_PAIR_RATE is that constant with its provenance attached." + ) + + +__all__ = [ + "MODEL_DEFAULT_ION_PAIR_RATE", + "PAPER_ENSEMBLE_ION_PAIR_RATE", + "ion_pair_production_rate", +] diff --git a/studio/science/plume.py b/studio/science/plume.py new file mode 100644 index 0000000..eeaccd3 --- /dev/null +++ b/studio/science/plume.py @@ -0,0 +1,140 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Initial plume volume, and injected mass -> initial concentration. + +**The most-duplicated derivation in the repository.** Six copies today, and they do not agree: + +=================================================== ========================================== +``coupled/paper_ensemble/run_ensemble.py:41-46,95`` V0 = 10 m x 10 m x **15 km**; 1 t SO2 -> + 6.273063291666667e15 molec cm^-3 -> pptv +``coupled/paper_ensemble/run_60day.py:37`` the same number, HARD-CODED (bit-identical, + verified) rather than derived +``coupled/paper_ensemble/make_rf_runs.py:44`` V0 = 1.5e12 cm^3, restated +``coupled/viz/bake_plume_dynamics.py:62-63`` V0 = 1.5e12 cm^3, restated, used to convert + concentration back to SO2-equivalent tonnes +``coupled/run_dilution_d1_clean.py:61,130`` V0 = 10 m x 10 m x **30 km** -- twice the + volume -- and the injection is specified + pptv-FIRST (2.9e9 pptv, "~1.7 t"), with the + mass computed back from it +``coupled/paper_ensemble/run_boxsize.py:43`` scales the initial concentration by a volume + factor, which is how the volume-invariance + sweep is done +=================================================== ========================================== + +The divergence that matters is the **15 km vs 30 km track**: a factor of two in V0, and therefore a +factor of two in initial concentration for the same injected mass. Which is right depends on what +t = 0 means, which is SCIENCE-2 (issue #54) and unresolved. Phase 0 follows the 810-run ensemble +(ASSUMPTION-5); this module implements the derivation, not the choice of inputs. + +**V0 does not enter the dynamics.** The model is intensive and volume-invariant +(``coupled/tests/test_boxvol_invariance.py``); the box-size sweep works purely by scaling the +initial concentration. So this derivation exists to turn a mass into a concentration and for no +other reason, and a UI must not imply that plume geometry feeds the physics. +""" + +from __future__ import annotations + +from studio.science.air import air_number_density +from studio.science.constants import ( + AVOGADRO, + CM3_PER_M3, + G_PER_KG, + PPTV_PER_MOLE_FRACTION, +) + + +def plume_volume_cm3(length_m: float, width_m: float, height_m: float) -> float: + """Initial plume volume V0 [cm^3] from a rectangular track. + + The ensemble's 10 m x 10 m x 15 km gives 1.5e12 cm^3 (``run_ensemble.py:45``). + + Raises: + ValueError: On a non-positive dimension -- a zero-volume plume divides by zero downstream. + """ + for name, value in (("length", length_m), ("width", width_m), ("height", height_m)): + if value <= 0.0: + raise ValueError(f"plume {name} must be > 0 m, got {value}") + return length_m * width_m * height_m * CM3_PER_M3 + + +def injected_number_density( + mass_kg: float, molar_mass_g_per_mol: float, volume_cm3: float +) -> float: + """Number density [molec cm^-3] of ``mass_kg`` of a species spread through ``volume_cm3``. + + ``n = (mass / M_w) * N_A / V``. Mirrors ``run_ensemble.py:46`` exactly, including its use of the + CODATA Avogadro constant rather than the gas model's rounded one -- see + ``studio/science/constants.py`` on that seam. + + Raises: + ValueError: On non-positive mass, molar mass or volume. + """ + if mass_kg <= 0.0: + raise ValueError(f"injected mass must be > 0 kg, got {mass_kg}") + if molar_mass_g_per_mol <= 0.0: + raise ValueError(f"molar mass must be > 0 g/mol, got {molar_mass_g_per_mol}") + if volume_cm3 <= 0.0: + raise ValueError(f"plume volume must be > 0 cm^3, got {volume_cm3}") + return mass_kg * G_PER_KG / molar_mass_g_per_mol * AVOGADRO / volume_cm3 + + +def number_density_to_pptv( + number_density_molec_cm3: float, air_number_density_molec_cm3: float +) -> float: + """Number density [molec cm^-3] -> mixing ratio [pptv] at a given air number density. + + Raises: + ValueError: On a non-positive air number density, or a negative number density. + """ + if air_number_density_molec_cm3 <= 0.0: + raise ValueError( + f"air number density must be > 0 molec/cm^3, got {air_number_density_molec_cm3}" + ) + if number_density_molec_cm3 < 0.0: + raise ValueError(f"number density must be >= 0, got {number_density_molec_cm3}") + return number_density_molec_cm3 / air_number_density_molec_cm3 * PPTV_PER_MOLE_FRACTION + + +def pptv_to_number_density(pptv: float, air_number_density_molec_cm3: float) -> float: + """Mixing ratio [pptv] -> number density [molec cm^-3]. + + Inverse of :func:`number_density_to_pptv`. Needed because one existing run specifies its + injection pptv-FIRST and computes the mass back (``coupled/run_dilution_d1_clean.py:130``); + both directions are real workflows. + """ + if air_number_density_molec_cm3 <= 0.0: + raise ValueError( + f"air number density must be > 0 molec/cm^3, got {air_number_density_molec_cm3}" + ) + if pptv < 0.0: + raise ValueError(f"mixing ratio must be >= 0 pptv, got {pptv}") + return pptv / PPTV_PER_MOLE_FRACTION * air_number_density_molec_cm3 + + +def initial_mixing_ratio_pptv( + *, + mass_kg: float, + molar_mass_g_per_mol: float, + volume_cm3: float, + pressure_mbar: float, + temperature_k: float, +) -> float: + """The full chain: injected mass -> initial mixing ratio [pptv]. Mirrors ``run_ensemble.py:95``. + + Note what the ensemble actually fixes: the injected **number density** is the same at every + site, and the pptv follows from the local air density. So the same 1 t release is a different + mixing ratio at 55 hPa than at 120 hPa, and the mixing ratio is the derived quantity -- not the + other way round. Keyword-only because five positional floats in a row is a units bug waiting to + happen. + """ + number_density = injected_number_density(mass_kg, molar_mass_g_per_mol, volume_cm3) + return number_density_to_pptv(number_density, air_number_density(pressure_mbar, temperature_k)) + + +__all__ = [ + "initial_mixing_ratio_pptv", + "injected_number_density", + "number_density_to_pptv", + "plume_volume_cm3", + "pptv_to_number_density", +] diff --git a/studio/science/size_distribution.py b/studio/science/size_distribution.py new file mode 100644 index 0000000..13dbbda --- /dev/null +++ b/studio/science/size_distribution.py @@ -0,0 +1,98 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Size-distribution reductions: bin mid-points, dlog10Dp, and dN/dlogDp. + +Four copies exist in the repository, written two ways: + +* ``run_ensemble.py:147-149`` and ``analyses/.../make_background_overlays.py:41-44`` + -- ``dp_mid = 10**(0.5*(log10(edges[:-1]) + log10(edges[1:])))``, + ``dlogdp = log10(edges)[1:] - log10(edges)[:-1]`` +* ``coupled/run_dilution_d1_clean.py:587-589`` + -- ``dp_mid = sqrt(edges[:-1]*edges[1:])``, ``dlogdp = log10(edges[1:]/edges[:-1])`` + +**These are the same quantity.** ``10**(0.5*(log a + log b)) == sqrt(a*b)`` and +``log b - log a == log(b/a)`` identically; the plan's note that the repository uses "two different +mid-point expressions" is, on inspection, two spellings of one expression. Measured on an 80-bin +TOMAS-like grid they differ by <= 7e-16 (mid-point) and <= 5e-15 (dlog10Dp) relative -- a few ULP of +float64 rounding, not a modelling difference. That is worth stating plainly, because "there are two +conventions in the code" would otherwise become a thing people believe and work around. + +This module implements the geometric-mean form (``sqrt(a*b)``): fewer operations, and no +intermediate logarithm to round. The equivalence is asserted by test, so the choice cannot +silently start mattering. + +The real trap in this area is not the formula. It is that **``dp_mid_um`` and ``dNdlogDp`` in +``state.npz`` are DRY diameters** while ``SA`` and ``radius_cm`` in the same file are WET -- see +``docs/studio/CAVEATS.md``. This module computes numbers; it does not know which basis its inputs +are on, so its callers must, and ``RunSummary`` (task 0.4) has to declare it per array. +""" + +from __future__ import annotations + +import numpy as np +import numpy.typing as npt + +FloatArray = npt.NDArray[np.float64] + + +def _validated_edges(edges_um: npt.ArrayLike) -> FloatArray: + """Bin edges as a float array, or raise. Strictly increasing and positive.""" + edges = np.asarray(edges_um, dtype=np.float64) + if edges.ndim != 1: + raise ValueError(f"bin edges must be 1-D, got shape {edges.shape}") + if edges.size < 2: + raise ValueError(f"need at least 2 bin edges to define a bin, got {edges.size}") + if not np.all(np.isfinite(edges)): + raise ValueError("bin edges must all be finite") + if np.any(edges <= 0.0): + raise ValueError("bin edges must be > 0 um (the grid is logarithmic in diameter)") + if not np.all(np.diff(edges) > 0.0): + raise ValueError("bin edges must be strictly increasing") + return edges + + +def bin_midpoints_um(edges_um: npt.ArrayLike) -> FloatArray: + """Geometric mid-point diameter of each bin [um]. + + Geometric, not arithmetic: the TOMAS grid is logarithmic in mass (ratio ``2**(40/n_bins)``), so + the arithmetic mean of two edges is not the centre of the bin on the axis these are plotted on. + """ + edges = _validated_edges(edges_um) + return np.sqrt(edges[:-1] * edges[1:]) + + +def dlog10_dp(edges_um: npt.ArrayLike) -> FloatArray: + """Width of each bin in log10(diameter): the ``dlogDp`` a size distribution is normalised by.""" + edges = _validated_edges(edges_um) + return np.log10(edges[1:] / edges[:-1]) + + +def dn_dlogdp(number_per_cm3: npt.ArrayLike, edges_um: npt.ArrayLike) -> FloatArray: + """Normalise per-bin number concentration [cm^-3] to dN/dlogDp [cm^-3]. + + Accepts a single spectrum ``(n_bins,)`` or a time series ``(n_times, n_bins)``; the last axis is + the size axis, matching ``state.npz``'s ``n_cm3``. + + Normalising is what makes bins comparable across grids: raw per-bin counts on a 40-bin grid and + an 80-bin grid are not the same curve, and only the normalised form is. + + Raises: + ValueError: If the bin count does not match the edges. This is the mistake worth catching -- + passing ``n_bins + 1`` edges is right and passing ``n_bins`` is a silent off-by-one that + numpy would broadcast into a plausible-looking wrong answer. + """ + counts = np.asarray(number_per_cm3, dtype=np.float64) + widths = dlog10_dp(edges_um) + if counts.ndim not in (1, 2): + raise ValueError(f"number concentration must be 1-D or 2-D, got shape {counts.shape}") + if counts.shape[-1] != widths.size: + raise ValueError( + f"got {counts.shape[-1]} bins but {widths.size + 1} edges define {widths.size} bins; " + f"edges must have exactly one more element than bins" + ) + if np.any(counts < 0.0): + raise ValueError("number concentration must be >= 0 cm^-3") + return counts / widths + + +__all__ = ["FloatArray", "bin_midpoints_um", "dlog10_dp", "dn_dlogdp"] diff --git a/studio/service.py b/studio/service.py new file mode 100644 index 0000000..58da77b --- /dev/null +++ b/studio/service.py @@ -0,0 +1,219 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Submitting a run: the one flow the CLI and the API both use. + +Extracted the moment there were two callers. A second copy of "resolve, record provenance, persist, +submit, record every transition, store the artefacts" would drift within a week, and the drift would +be invisible -- both paths would keep working, and only their database rows would disagree. ADR-002 +requires that a sweep launched from a terminal and one launched from the web produce identical rows +and identical provenance; that is a property of there being *one function*, not of two being written +carefully. + +The two callers differ in what they do *around* this -- the CLI blocks and prints, the API returns a +handle and streams progress -- and neither difference reaches in here. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any + +from sqlalchemy.orm import sessionmaker + +from studio.modelio.provenance import record_for +from studio.resolve import ResolvedConfig +from studio.runner import LocalSubprocessRunner +from studio.store import ( + LocalDirectoryStore, + create_db_engine, + create_run, + database_url, + record_artifact, + record_job, + record_summary, + record_transition, + session_factory, + session_scope, + upgrade_to_head, +) +from studio.store.models import JobRow, RunRow, RunSetRow + +#: How often a running job's state is copied into the database. Short against a 4-minute run, long +#: against a single SELECT. +_FOLLOW_INTERVAL_S = 0.2 + +#: Artefacts recorded for every run, by kind -> filename in the job's work directory. A fixed set +#: rather than "whatever the directory contains", so a missing one is a visible gap rather than a +#: silently shorter list. +ARTIFACTS: dict[str, str] = { + "input": "input.json", + "provenance": "provenance.json", + "state": "state.npz", + "summary": "summary.json", + "stdout": "stdout.log", + "stderr": "stderr.log", +} + + +def prepare(database: str | None, out: Path) -> tuple[sessionmaker[Any], LocalDirectoryStore]: + """Migrate the database and open the artefact store. + + Idempotent, so both entry points can call it unconditionally at startup: a fresh clone must not + need a manual ``alembic upgrade`` before the first run. + """ + url = database_url(database) + upgrade_to_head(url) + return session_factory(create_db_engine(url)), LocalDirectoryStore(Path(out) / "artifacts") + + +def _require(row: Any, what: str, key: str) -> Any: + """Return ``row``, or raise saying what went missing. + + ``Session.get`` returns ``None`` for a row that is not there, and passing that on would fail + several frames later as an ``AttributeError`` about ``None``. Here it means the database changed + underneath a run that is mid-flight -- rare, and worth naming precisely when it happens. + """ + if row is None: + raise RuntimeError( + f"{what} {key!r} vanished from the database while its run was in progress; " + f"the run may have been deleted concurrently" + ) + return row + + +def create_pending_run( + factory: sessionmaker[Any], + *, + run_set: RunSetRow, + config: ResolvedConfig, + label: str = "", +) -> str: + """Record the run and its provenance **before** anything is submitted. + + Provenance first, deliberately (ADR-006): if the model cannot be pinned, the run must not start, + and a run that dies in its first second still says exactly what produced it. + """ + provenance = record_for(config) + with session_scope(factory) as session: + run = create_run( + session, + run_set=run_set, + config=config, + label=label, + provenance=provenance.model_dump(mode="json"), + ) + return str(run.id) + + +def submit( + factory: sessionmaker[Any], + runner: LocalSubprocessRunner, + *, + run_id: str, + config: ResolvedConfig, + label: str = "", +) -> tuple[str, str]: + """Submit a recorded run. Returns ``(job_id, runner_job_id)``. + + Returns both because they are different things: the database's job id is durable and survives a + restart, while the runner's is a handle into this process's pool. + """ + record = runner.submit(config, label=label) + with session_scope(factory) as session: + run_row = _require(session.get(RunRow, run_id), "run", run_id) + job = record_job( + session, + run=run_row, + state=record.state.value, + work_dir=record.work_dir, + detail=record.detail, + ) + return str(job.id), record.job_id + + +def finalise( + factory: sessionmaker[Any], + store: LocalDirectoryStore, + runner: LocalSubprocessRunner, + *, + run_id: str, + job_id: str, + runner_job_id: str, +) -> str: + """Follow a job to completion, persisting each transition **as it happens**, then store its + artefacts. Returns the final state. + + Polling the runner rather than only waiting, because the database is what the API streams from + (a stream reading the worker pool would be blind to runs submitted by the CLI or by a previous + process). Writing only at the end would leave a job showing ``queued`` for its whole four + minutes and then jumping to ``succeeded`` -- technically a trail, useless as progress. + + ``_FOLLOW_INTERVAL_S`` is short relative to a run and long relative to a SELECT; SQLite has one + writer, and each write here is a single short transaction. + """ + seen = 1 # the queued transition is already persisted by submit() + while True: + record = runner.poll(runner_job_id) + new_transitions = record.transitions[seen:] + if new_transitions: + with session_scope(factory) as session: + job_row = _require(session.get(JobRow, job_id), "job", job_id) + for transition in new_transitions: + record_transition( + session, + job=job_row, + state=transition.state.value, + detail=transition.detail, + exit_code=record.exit_code if transition.state is record.state else None, + ) + seen = len(record.transitions) + if record.is_terminal: + break + time.sleep(_FOLLOW_INTERVAL_S) + + final = runner.wait(runner_job_id) + with session_scope(factory) as session: + run_row = _require(session.get(RunRow, run_id), "run", run_id) + work_dir = Path(final.work_dir or ".") + for kind, filename in ARTIFACTS.items(): + source = work_dir / filename + if source.is_file(): + record_artifact(session, run=run_row, kind=kind, source=source, store=store) + summary_path = work_dir / "summary.json" + if summary_path.is_file(): + record_summary( + session, run=run_row, summary=json.loads(summary_path.read_text(encoding="utf-8")) + ) + return str(final.state.value) + + +def submit_and_record( + factory: sessionmaker[Any], + store: LocalDirectoryStore, + runner: LocalSubprocessRunner, + *, + config: ResolvedConfig, + run_set: RunSetRow, + label: str = "", + wait: bool = True, +) -> tuple[str, str]: + """The whole flow, for a caller that just wants a run to happen. Returns ``(run_id, state)``.""" + run_id = create_pending_run(factory, run_set=run_set, config=config, label=label) + job_id, runner_job_id = submit(factory, runner, run_id=run_id, config=config, label=label) + if not wait: + return run_id, "queued" + return run_id, finalise( + factory, store, runner, run_id=run_id, job_id=job_id, runner_job_id=runner_job_id + ) + + +__all__ = [ + "ARTIFACTS", + "create_pending_run", + "finalise", + "prepare", + "submit", + "submit_and_record", +] diff --git a/studio/store/__init__.py b/studio/store/__init__.py new file mode 100644 index 0000000..faf93a3 --- /dev/null +++ b/studio/store/__init__.py @@ -0,0 +1,85 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Persistence for run metadata (ADR-004, ADR-007). + +Three stores, matched to three shapes. This package is the relational one: run sets, runs, configs, +jobs and pointers to output. **Arrays live on disk behind ``artifacts.ArtifactStore``**, never in +the database, and climatology is a separate Phase-1 concern. + +* ``models.py`` -- the tables. No ``AUTOINCREMENT``, no reliance on SQLite type affinity, + timezone-aware timestamps: moving to Postgres must be a connection-string change. +* ``engine.py`` -- engine and short-transaction session scope; SQLite gets ``foreign_keys=ON`` and + WAL, because otherwise its constraints would be documentation while Postgres enforced them. +* ``artifacts.py`` -- the storage interface. A directory today; MinIO or S3 later without touching + callers. Checksums are computed on write so "still the file that was written" is checkable. +* ``repository.py`` -- the operations, and no update path for a config: immutability is structural. +* ``migrations/`` -- Alembic from the first migration, so the second one is routine. + +This package must not import ``coupled``. +""" + +from __future__ import annotations + +from studio.store.artifacts import ArtifactStore, LocalDirectoryStore, StoredArtifact, sha256_of +from studio.store.engine import ( + DATABASE_URL_ENV, + create_db_engine, + database_url, + session_factory, + session_scope, +) +from studio.store.migrate import current_revision, upgrade_to_head +from studio.store.models import ( + Base, + DatasetVersionRow, + JobRow, + JobTransitionRow, + ResultArtifactRow, + RunConfigRow, + RunRow, + RunSetRow, + RunSummaryRow, +) +from studio.store.repository import ( + artifact_for, + create_run, + create_run_set, + ensure_config, + record_artifact, + record_job, + record_summary, + record_transition, + runs_for_config, +) + +__all__ = [ + "DATABASE_URL_ENV", + "ArtifactStore", + "Base", + "DatasetVersionRow", + "JobRow", + "JobTransitionRow", + "LocalDirectoryStore", + "ResultArtifactRow", + "RunConfigRow", + "RunRow", + "RunSetRow", + "RunSummaryRow", + "StoredArtifact", + "artifact_for", + "create_db_engine", + "create_run", + "create_run_set", + "current_revision", + "database_url", + "ensure_config", + "record_artifact", + "record_job", + "record_summary", + "record_transition", + "runs_for_config", + "session_factory", + "session_scope", + "sha256_of", + "upgrade_to_head", +] diff --git a/studio/store/alembic.ini b/studio/store/alembic.ini new file mode 100644 index 0000000..baa9848 --- /dev/null +++ b/studio/store/alembic.ini @@ -0,0 +1,47 @@ +# Alembic configuration for Plume Studio's run-metadata database. +# +# alembic -c studio/store/alembic.ini upgrade head +# alembic -c studio/store/alembic.ini revision --autogenerate -m "what changed" +# +# The URL is deliberately NOT set here: it comes from STUDIO_DATABASE_URL (or the local SQLite +# default) via studio.store.engine, so one configuration works for a developer machine, CI, and a +# Postgres deployment without editing a file that is committed. +[alembic] +script_location = studio/store/migrations +prepend_sys_path = . +# explicit, or alembic warns and falls back to splitting on spaces, commas and colons +path_separator = os +file_template = %%(rev)s_%%(slug)s + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARNING +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARNING +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s diff --git a/studio/store/artifacts.py b/studio/store/artifacts.py new file mode 100644 index 0000000..c1bf447 --- /dev/null +++ b/studio/store/artifacts.py @@ -0,0 +1,119 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Artefact storage, behind an interface (ADR-004). + +"Object storage" in Phase 0 is a directory on disk. MinIO or S3 substitutes without touching +callers, which is why every path a caller sees is **relative to the store root** -- an absolute path +baked into the database would be a deployment detail that outlives the deployment. + +The store computes a SHA-256 as it copies. That is what makes "this artefact is still the file that +was written" a checkable claim rather than an assumption, and it costs one pass over a file that is +being read anyway. +""" + +from __future__ import annotations + +import hashlib +import shutil +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol, runtime_checkable + +#: Read in 1 MiB blocks: large enough that the syscall overhead disappears, small enough that a +#: 3.6 MB npz never sits in memory twice. +_CHUNK = 1024 * 1024 + + +@dataclass(frozen=True) +class StoredArtifact: + """What the database records about one stored file. Never the bytes.""" + + key: str + size_bytes: int + sha256: str + content_type: str + + +@runtime_checkable +class ArtifactStore(Protocol): + """Where run outputs live. Nothing above this may assume a filesystem.""" + + def put(self, run_id: str, kind: str, source: Path) -> StoredArtifact: + """Copy ``source`` into the store under ``run_id``, returning what to record.""" + ... + + def open_path(self, key: str) -> Path: + """Resolve a stored key to a readable path.""" + ... + + def exists(self, key: str) -> bool: ... + + +def sha256_of(path: Path) -> str: + """Streaming SHA-256 of a file.""" + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(_CHUNK): + digest.update(chunk) + return digest.hexdigest() + + +#: Extension -> content type, for the artefacts Studio actually writes. Deliberately small: a +#: general mimetype lookup would guess, and a wrong content type on an npz is worse than none. +_CONTENT_TYPES = { + ".npz": "application/x-npz", + ".json": "application/json", + ".log": "text/plain", + ".png": "image/png", +} + + +class LocalDirectoryStore: + """The Phase-0 implementation: a directory tree, one subdirectory per run.""" + + def __init__(self, root: Path) -> None: + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + + def put(self, run_id: str, kind: str, source: Path) -> StoredArtifact: + """Copy in, hash, and return the record. + + Raises: + FileNotFoundError: If the source is missing. A run that was supposed to produce an + artefact and did not is a failure to surface, not a row to omit. + """ + source = Path(source) + if not source.is_file(): + raise FileNotFoundError( + f"artefact {kind!r} for run {run_id} is missing at {source}; a run that did not " + f"produce it should fail rather than be recorded without it" + ) + destination = self.root / run_id / source.name + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, destination) + return StoredArtifact( + key=str(destination.relative_to(self.root)), + size_bytes=destination.stat().st_size, + sha256=sha256_of(destination), + content_type=_CONTENT_TYPES.get(source.suffix, "application/octet-stream"), + ) + + def open_path(self, key: str) -> Path: + path = self.root / key + if not path.is_file(): + raise FileNotFoundError(f"artefact {key!r} is recorded but missing from {self.root}") + return path + + def exists(self, key: str) -> bool: + return (self.root / key).is_file() + + def verify(self, key: str, expected_sha256: str) -> bool: + """Whether the stored bytes still hash to what was recorded. + + The reason the checksum is stored at all: silent corruption and a helpfully "tidied" + directory look identical from the database. + """ + return self.exists(key) and sha256_of(self.open_path(key)) == expected_sha256 + + +__all__ = ["ArtifactStore", "LocalDirectoryStore", "StoredArtifact", "sha256_of"] diff --git a/studio/store/engine.py b/studio/store/engine.py new file mode 100644 index 0000000..43a3683 --- /dev/null +++ b/studio/store/engine.py @@ -0,0 +1,98 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Engine and session factory. + +SQLite in Phase 0; a Postgres URL substitutes without code changes, which is the whole point of +putting Alembic in from the first migration (ADR-007). + +Two SQLite-specific pragmas are set **for SQLite only**, and neither is a workaround leaking into +the models: + +* ``foreign_keys=ON`` -- SQLite ignores foreign keys unless asked, so without this the constraints + in ``models.py`` would be documentation on that backend and enforced on Postgres. Silent + divergence between environments is worse than either behaviour. +* ``journal_mode=WAL`` -- readers stop blocking the writer. ADR-007 flags SQLite's single-writer + model as a real constraint, and the API reads job state far more often than it writes. +""" + +from __future__ import annotations + +import os +from collections.abc import Iterator +from contextlib import contextmanager +from pathlib import Path +from typing import Any + +from sqlalchemy import Engine, create_engine, event +from sqlalchemy.orm import Session, sessionmaker + +#: Read at import of a session factory, never at module import, so a test can set it first. +DATABASE_URL_ENV = "STUDIO_DATABASE_URL" + +#: Where a local SQLite database lives when nothing says otherwise. Matches studio/.env.example. +DEFAULT_SQLITE_PATH = Path("var/studio/studio.db") + + +def database_url(url: str | None = None) -> str: + """Resolve the database URL: explicit argument, then environment, then the local default.""" + if url: + return url + from_env = os.environ.get(DATABASE_URL_ENV) + if from_env: + return from_env + DEFAULT_SQLITE_PATH.parent.mkdir(parents=True, exist_ok=True) + return f"sqlite:///{DEFAULT_SQLITE_PATH}" + + +def create_db_engine(url: str | None = None, *, echo: bool = False) -> Engine: + """An engine with SQLite's footguns disarmed.""" + resolved = database_url(url) + engine = create_engine(resolved, echo=echo, future=True) + if resolved.startswith("sqlite"): + + @event.listens_for(engine, "connect") + def _sqlite_pragmas(dbapi_connection: Any, _record: Any) -> None: + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.execute("PRAGMA journal_mode=WAL") + cursor.close() + + return engine + + +def session_factory(engine: Engine) -> sessionmaker[Session]: + """Sessions that do not expire objects on commit. + + ``expire_on_commit=False`` because callers read attributes off returned rows after the + transaction closes; the alternative is a lazy reload per attribute, which on SQLite means + re-acquiring the very lock ADR-007 warns about. + """ + return sessionmaker(bind=engine, expire_on_commit=False, future=True) + + +@contextmanager +def session_scope(factory: sessionmaker[Session]) -> Iterator[Session]: + """A transaction that commits on success and rolls back on any exception. + + Short by design: SQLite has one writer, so a session held open across a model run would block + every other job's state update. + """ + session = factory() + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + finally: + session.close() + + +__all__ = [ + "DATABASE_URL_ENV", + "DEFAULT_SQLITE_PATH", + "create_db_engine", + "database_url", + "session_factory", + "session_scope", +] diff --git a/studio/store/migrate.py b/studio/store/migrate.py new file mode 100644 index 0000000..d3c89f2 --- /dev/null +++ b/studio/store/migrate.py @@ -0,0 +1,50 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Running migrations from code, so nothing depends on the ``alembic`` CLI being on PATH. + +``Base.metadata.create_all`` is deliberately **not** used anywhere, including in tests. It would +produce a schema that no migration ever created, so the migrations would be exercised for the first +time on someone's real database -- which is the failure mode Alembic-from-the-first-migration exists +to prevent (ADR-007). Tests upgrade to head like everything else. +""" + +from __future__ import annotations + +from pathlib import Path + +from alembic import command +from alembic.config import Config +from alembic.migration import MigrationContext +from sqlalchemy import Engine + +#: The committed configuration; the URL is supplied at call time, never from the file. +ALEMBIC_INI = Path(__file__).resolve().parent / "alembic.ini" +MIGRATIONS_DIR = Path(__file__).resolve().parent / "migrations" + + +def alembic_config(url: str) -> Config: + """An Alembic config pointed at this package's migrations and the given URL.""" + config = Config(str(ALEMBIC_INI)) + config.set_main_option("script_location", str(MIGRATIONS_DIR)) + config.set_main_option("sqlalchemy.url", url) + return config + + +def upgrade_to_head(url: str) -> None: + """Bring a database up to the latest revision. Idempotent.""" + command.upgrade(alembic_config(url), "head") + + +def current_revision(engine: Engine) -> str | None: + """The revision a database is at, or ``None`` if it has never been migrated.""" + with engine.connect() as connection: + return MigrationContext.configure(connection).get_current_revision() + + +__all__ = [ + "ALEMBIC_INI", + "MIGRATIONS_DIR", + "alembic_config", + "current_revision", + "upgrade_to_head", +] diff --git a/studio/store/migrations/env.py b/studio/store/migrations/env.py new file mode 100644 index 0000000..f0a5fbe --- /dev/null +++ b/studio/store/migrations/env.py @@ -0,0 +1,63 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Alembic environment. + +The URL comes from ``studio.store.engine.database_url`` rather than ``alembic.ini`` so that one +committed configuration serves a developer machine, CI and a Postgres deployment -- the difference +is ``STUDIO_DATABASE_URL``, not an edited file. + +``render_as_batch=True`` is set for SQLite only. SQLite cannot ALTER most things in place, so +Alembic rebuilds the table; without it, the first migration that drops a column would fail on the +Phase-0 backend and pass on Postgres. That divergence is exactly what ADR-007 says to avoid. +""" + +from __future__ import annotations + +from logging.config import fileConfig + +from alembic import context + +from studio.store.engine import create_db_engine +from studio.store.models import Base + +config = context.config +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +target_metadata = Base.metadata + + +def run_migrations_offline() -> None: + """Emit SQL without a connection, for review or for a DBA to apply.""" + url = config.get_main_option("sqlalchemy.url") or None + from studio.store.engine import database_url + + context.configure( + url=database_url(url), + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + compare_type=True, + ) + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + engine = create_db_engine(config.get_main_option("sqlalchemy.url") or None) + with engine.connect() as connection: + context.configure( + connection=connection, + target_metadata=target_metadata, + compare_type=True, + render_as_batch=connection.dialect.name == "sqlite", + ) + with context.begin_transaction(): + context.run_migrations() + engine.dispose() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/studio/store/migrations/script.py.mako b/studio/store/migrations/script.py.mako new file mode 100644 index 0000000..ee40d2c --- /dev/null +++ b/studio/store/migrations/script.py.mako @@ -0,0 +1,24 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Created: ${create_date} +""" +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} +branch_labels = ${repr(branch_labels)} +depends_on = ${repr(depends_on)} + + +def upgrade() -> None: + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + ${downgrades if downgrades else "pass"} diff --git a/studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py b/studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py new file mode 100644 index 0000000..f4481b5 --- /dev/null +++ b/studio/store/migrations/versions/1427673e9a31_initial_run_metadata_schema.py @@ -0,0 +1,215 @@ +"""initial run metadata schema + +Revision ID: 1427673e9a31 +Revises: +Created: 2026-08-14 16:27:10.579709 +""" + +from __future__ import annotations + +import sqlalchemy as sa +from alembic import op + +revision = "1427673e9a31" +down_revision = None +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.create_table( + "dataset_version", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("identifier", sa.String(length=255), nullable=False), + sa.Column("version", sa.String(length=64), nullable=False), + sa.Column("sha256", sa.String(length=64), nullable=False), + sa.Column("source", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint("identifier", "version", name="uq_dataset_identifier_version"), + ) + with op.batch_alter_table("dataset_version", schema=None) as batch_op: + batch_op.create_index( + batch_op.f("ix_dataset_version_identifier"), ["identifier"], unique=False + ) + + op.create_table( + "run_config", + sa.Column("config_hash", sa.String(length=64), nullable=False), + sa.Column("schema_version", sa.String(length=32), nullable=False), + sa.Column("resolved_config", sa.JSON(), nullable=False), + sa.Column("overrides", sa.JSON(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint("config_hash"), + ) + op.create_table( + "run_set", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("label", sa.String(length=255), nullable=False), + sa.Column("axes", sa.JSON(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("owner", sa.String(length=255), nullable=True), + sa.PrimaryKeyConstraint("id"), + ) + op.create_table( + "run", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("run_set_id", sa.String(length=64), nullable=False), + sa.Column("config_hash", sa.String(length=64), nullable=False), + sa.Column("label", sa.String(length=255), nullable=False), + sa.Column("derived_from_run_id", sa.String(length=64), nullable=True), + sa.Column("provenance", sa.JSON(), nullable=True), + sa.Column("reproducible", sa.Integer(), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["config_hash"], + ["run_config.config_hash"], + ), + sa.ForeignKeyConstraint( + ["derived_from_run_id"], + ["run.id"], + ), + sa.ForeignKeyConstraint( + ["run_set_id"], + ["run_set.id"], + ), + sa.PrimaryKeyConstraint("id"), + ) + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_run_config_hash"), ["config_hash"], unique=False) + batch_op.create_index( + "ix_run_config_hash_created", ["config_hash", "created_at"], unique=False + ) + batch_op.create_index(batch_op.f("ix_run_run_set_id"), ["run_set_id"], unique=False) + + op.create_table( + "job", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("state", sa.String(length=32), nullable=False), + sa.Column("backend", sa.String(length=64), nullable=False), + sa.Column("work_dir", sa.Text(), nullable=True), + sa.Column("exit_code", sa.Integer(), nullable=True), + sa.Column("detail", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("id"), + ) + with op.batch_alter_table("job", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_job_run_id"), ["run_id"], unique=False) + batch_op.create_index(batch_op.f("ix_job_state"), ["state"], unique=False) + + op.create_table( + "result_artifact", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("kind", sa.String(length=32), nullable=False), + sa.Column("path", sa.Text(), nullable=False), + sa.Column("size_bytes", sa.Integer(), nullable=False), + sa.Column("sha256", sa.String(length=64), nullable=False), + sa.Column("content_type", sa.String(length=64), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint("run_id", "kind", name="uq_artifact_run_kind"), + ) + with op.batch_alter_table("result_artifact", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_result_artifact_run_id"), ["run_id"], unique=False) + + op.create_table( + "run_dataset", + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("dataset_id", sa.String(length=64), nullable=False), + sa.ForeignKeyConstraint( + ["dataset_id"], + ["dataset_version.id"], + ), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("run_id", "dataset_id"), + ) + op.create_table( + "run_summary", + sa.Column("run_id", sa.String(length=64), nullable=False), + sa.Column("schema_version", sa.String(length=32), nullable=False), + sa.Column("termination", sa.String(length=32), nullable=False), + sa.Column("flags", sa.JSON(), nullable=False), + sa.Column("final_so2_pptv", sa.Float(), nullable=True), + sa.Column("peak_h2so4_pptv", sa.Float(), nullable=True), + sa.Column("peak_number_cm3", sa.Float(), nullable=True), + sa.Column("final_surface_area", sa.Float(), nullable=True), + sa.Column("summary", sa.JSON(), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["run_id"], + ["run.id"], + ), + sa.PrimaryKeyConstraint("run_id"), + ) + with op.batch_alter_table("run_summary", schema=None) as batch_op: + batch_op.create_index( + batch_op.f("ix_run_summary_termination"), ["termination"], unique=False + ) + + op.create_table( + "job_transition", + sa.Column("id", sa.String(length=64), nullable=False), + sa.Column("job_id", sa.String(length=64), nullable=False), + sa.Column("state", sa.String(length=32), nullable=False), + sa.Column("at", sa.DateTime(timezone=True), nullable=False), + sa.Column("detail", sa.Text(), nullable=False), + sa.ForeignKeyConstraint( + ["job_id"], + ["job.id"], + ), + sa.PrimaryKeyConstraint("id"), + ) + with op.batch_alter_table("job_transition", schema=None) as batch_op: + batch_op.create_index(batch_op.f("ix_job_transition_job_id"), ["job_id"], unique=False) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("job_transition", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_job_transition_job_id")) + + op.drop_table("job_transition") + with op.batch_alter_table("run_summary", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_run_summary_termination")) + + op.drop_table("run_summary") + op.drop_table("run_dataset") + with op.batch_alter_table("result_artifact", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_result_artifact_run_id")) + + op.drop_table("result_artifact") + with op.batch_alter_table("job", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_job_state")) + batch_op.drop_index(batch_op.f("ix_job_run_id")) + + op.drop_table("job") + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_run_run_set_id")) + batch_op.drop_index("ix_run_config_hash_created") + batch_op.drop_index(batch_op.f("ix_run_config_hash")) + + op.drop_table("run") + op.drop_table("run_set") + op.drop_table("run_config") + with op.batch_alter_table("dataset_version", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_dataset_version_identifier")) + + op.drop_table("dataset_version") + # ### end Alembic commands ### diff --git a/studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py b/studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py new file mode 100644 index 0000000..fff0e77 --- /dev/null +++ b/studio/store/migrations/versions/c0236fb572d2_reproducible_is_a_boolean.py @@ -0,0 +1,36 @@ +"""reproducible is a boolean + +Revision ID: c0236fb572d2 +Revises: 1427673e9a31 +Created: 2026-08-14 17:16:21.643335 +""" + +from __future__ import annotations + +import sqlalchemy as sa +from alembic import op + +revision = "c0236fb572d2" +down_revision = "1427673e9a31" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.alter_column( + "reproducible", existing_type=sa.INTEGER(), type_=sa.Boolean(), existing_nullable=True + ) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("run", schema=None) as batch_op: + batch_op.alter_column( + "reproducible", existing_type=sa.Boolean(), type_=sa.INTEGER(), existing_nullable=True + ) + + # ### end Alembic commands ### diff --git a/studio/store/models.py b/studio/store/models.py new file mode 100644 index 0000000..c92495f --- /dev/null +++ b/studio/store/models.py @@ -0,0 +1,294 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Relational models for run metadata (ADR-004). + +**The database stores pointers, never arrays.** Raw output is object storage keyed by run id -- a +directory on disk today, MinIO or S3 later without touching callers -- and what lives here is the +path, size, checksum and content type. A 3.6 MB npz per run times 810 runs is not a database's job. + +Two rules from ADR-004 are structural rather than conventional: + +* **Configs are immutable once submitted.** ``run_config`` is insert-only, keyed by ``config_hash``: + the same config submitted twice is the same row, and an edited config is a *different* row and a + *different* run. There is deliberately no update path -- see ``repository.py``, which offers + get-or-create and nothing else. +* **Results are never overwritten.** An artefact row is written once per run. + +Portability is a requirement, not an aspiration (ADR-007): moving to Postgres must be a +connection-string change. So: + +* **No ``AUTOINCREMENT``.** Primary keys are explicit strings -- content hashes where a natural one + exists, UUID hex otherwise. This also makes a run's identity meaningful rather than positional, + and lets a row be written before the database has ever seen it. +* **No reliance on SQLite's type affinity.** Every column has a real type, and timestamps go + through :class:`UtcDateTime` -- because ``DateTime(timezone=True)`` alone returns *naive* + datetimes on SQLite and aware ones on Postgres, which is a comparison bug that only appears in + production. +* **JSON columns via SQLAlchemy's portable ``JSON``**, which is native on both backends. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import ( + Boolean, + DateTime, + Float, + ForeignKey, + Index, + Integer, + String, + Text, + UniqueConstraint, +) +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship +from sqlalchemy.types import JSON, TypeDecorator + + +class UtcDateTime(TypeDecorator[datetime]): + """A timestamp that is timezone-aware on **every** backend. + + ``DateTime(timezone=True)`` is not enough. Postgres stores the offset and returns an aware + datetime; **SQLite stores a string and hands back a naive one**, so the same code compares + correctly on one backend and silently wrongly on the other -- the exact cross-backend divergence + ADR-007 says to avoid, and the reason this class exists rather than a convention that everyone + remembers to call ``.replace(tzinfo=UTC)``. + + Naive input **raises**: a caller who does not know their own timezone cannot be given one by + guessing (ADR-005). + """ + + impl = DateTime(timezone=True) + cache_ok = True + + def process_bind_param(self, value: datetime | None, dialect: Any) -> datetime | None: + if value is None: + return None + if value.tzinfo is None: + raise ValueError( + f"naive datetime {value!r} cannot be stored: its timezone is unknown, and assuming " + f"one would be wrong on every machine but the one that wrote it. Use " + f"datetime.now(UTC)." + ) + return value.astimezone(UTC) + + def process_result_value(self, value: datetime | None, dialect: Any) -> datetime | None: + if value is None: + return None + return value if value.tzinfo is not None else value.replace(tzinfo=UTC) + + +class Base(DeclarativeBase): + """Declarative base. Alembic's autogenerate compares against this metadata.""" + + +#: Identifier column width: a SHA-256 hex digest is 64, a UUID hex 32. Fixed rather than unbounded +#: so Postgres gets a sensible column and the intent is visible. +_ID = String(64) + + +class RunSetRow(Base): + """A sweep. A single run is the N = 1 case of one (see ``studio.schema.runset``).""" + + __tablename__ = "run_set" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + label: Mapped[str] = mapped_column(String(255), default="") + #: The axes, as submitted. Stored whole because a RunSet is small and re-expanding it must give + #: exactly the runs that were created, not today's interpretation of the axes. + axes: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + #: Nullable owner, present from the first migration so adopting tenancy later is a backfill + #: rather than a migration of every query (BLOCKING-2 / ASSUMPTION-3). + owner: Mapped[str | None] = mapped_column(String(255), nullable=True) + + runs: Mapped[list[RunRow]] = relationship( + back_populates="run_set", cascade="all, delete-orphan" + ) + + +class RunConfigRow(Base): + """A resolved configuration, keyed by its own hash. **Insert-only.** + + The primary key IS the identity (ADR-006): the same configuration submitted from the CLI and + from the API is one row, which is what makes the hash usable as a cache key. Two runs of the + same config share this row and differ only in their run id. + """ + + __tablename__ = "run_config" + + config_hash: Mapped[str] = mapped_column(_ID, primary_key=True) + schema_version: Mapped[str] = mapped_column(String(32)) + #: The resolved, post-derivation parameter set -- what the model actually received. + resolved_config: Mapped[dict[str, Any]] = mapped_column(JSON) + #: Derived fields the user overrode, with the value in force. + overrides: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + +class RunRow(Base): + """One simulation: its config, its lineage, and where its outputs live.""" + + __tablename__ = "run" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + run_set_id: Mapped[str] = mapped_column(_ID, ForeignKey("run_set.id"), index=True) + config_hash: Mapped[str] = mapped_column(_ID, ForeignKey("run_config.config_hash"), index=True) + #: The ensemble-style case label (``30N_20km__sabr220__…``). A label, never an identity -- it + #: captures the axes rather than the resolved configuration, so two sweeps differing only in a + #: "fixed" value would collide on it (ADR-006). + label: Mapped[str] = mapped_column(String(255), default="") + #: Lineage for an edited config: this run supersedes that one (ADR-004). Never a mutation. + derived_from_run_id: Mapped[str | None] = mapped_column( + _ID, ForeignKey("run.id"), nullable=True + ) + #: The provenance record (ADR-006) as written at submit time, stored verbatim. + provenance: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True) + #: False when any checkout was dirty: the SHAs do not describe the code that ran. + reproducible: Mapped[bool | None] = mapped_column(Boolean, nullable=True) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + run_set: Mapped[RunSetRow] = relationship(back_populates="runs") + jobs: Mapped[list[JobRow]] = relationship(back_populates="run", cascade="all, delete-orphan") + artifacts: Mapped[list[ResultArtifactRow]] = relationship( + back_populates="run", cascade="all, delete-orphan" + ) + + __table_args__ = (Index("ix_run_config_hash_created", "config_hash", "created_at"),) + + +class JobRow(Base): + """Execution of a run. State lives here, not in memory, so a handle survives an API restart.""" + + __tablename__ = "job" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), index=True) + #: Current lifecycle state (``studio.runner.JobState``). Stored as its string value so the + #: database is readable without the enum, and a new state is a data change not a schema one. + state: Mapped[str] = mapped_column(String(32), index=True) + backend: Mapped[str] = mapped_column(String(64), default="local-subprocess") + work_dir: Mapped[str | None] = mapped_column(Text, nullable=True) + exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True) + detail: Mapped[str] = mapped_column(Text, default="") + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + updated_at: Mapped[datetime] = mapped_column(UtcDateTime) + + run: Mapped[RunRow] = relationship(back_populates="jobs") + transitions: Mapped[list[JobTransitionRow]] = relationship( + back_populates="job", cascade="all, delete-orphan", order_by="JobTransitionRow.at" + ) + + +class JobTransitionRow(Base): + """One timestamped state change. The audit trail, kept rather than collapsed to a status. + + "It failed" is not debuggable; "QUEUED 14:02:11, RUNNING 14:02:11, FAILED 14:06:48 exit 1" is. + """ + + __tablename__ = "job_transition" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + job_id: Mapped[str] = mapped_column(_ID, ForeignKey("job.id"), index=True) + state: Mapped[str] = mapped_column(String(32)) + at: Mapped[datetime] = mapped_column(UtcDateTime) + detail: Mapped[str] = mapped_column(Text, default="") + + job: Mapped[JobRow] = relationship(back_populates="transitions") + + +class ResultArtifactRow(Base): + """A pointer to one output file. **The bytes are not here.** + + Path, size, checksum and content type -- enough to find it, verify it, and notice when it has + gone missing. ``sha256`` is what makes "the file is still the one that was written" checkable + rather than assumed. + """ + + __tablename__ = "result_artifact" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), index=True) + #: ``state``, ``summary``, ``provenance``, ``stdout``, ``stderr``, ``input``. + kind: Mapped[str] = mapped_column(String(32)) + #: Relative to the artifact store's root, never absolute: the root moves between deployments. + path: Mapped[str] = mapped_column(Text) + size_bytes: Mapped[int] = mapped_column(Integer) + sha256: Mapped[str] = mapped_column(String(64)) + content_type: Mapped[str] = mapped_column(String(64), default="application/octet-stream") + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + run: Mapped[RunRow] = relationship(back_populates="artifacts") + + __table_args__ = (UniqueConstraint("run_id", "kind", name="uq_artifact_run_kind"),) + + +class DatasetVersionRow(Base): + """An input dataset consulted by a run, with its checksum (ADR-006). + + Empty in Phase 0 -- the schema carries no dataset inputs yet -- but present from the first + migration, because "which ERA5 product was this run built on?" is a question Phase 1 must be + able to answer about runs made before it existed. + """ + + __tablename__ = "dataset_version" + + id: Mapped[str] = mapped_column(_ID, primary_key=True) + identifier: Mapped[str] = mapped_column(String(255), index=True) + version: Mapped[str] = mapped_column(String(64)) + sha256: Mapped[str] = mapped_column(String(64)) + source: Mapped[str] = mapped_column(Text, default="") + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + __table_args__ = ( + UniqueConstraint("identifier", "version", name="uq_dataset_identifier_version"), + ) + + +class RunDatasetRow(Base): + """Which datasets a run consulted. Many-to-many, so a dataset is recorded once.""" + + __tablename__ = "run_dataset" + + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), primary_key=True) + dataset_id: Mapped[str] = mapped_column(_ID, ForeignKey("dataset_version.id"), primary_key=True) + + +class RunSummaryRow(Base): + """The versioned reduction (ADR-004), stored so comparison views never open the raw npz. + + Kept as JSON rather than shredded into columns: it is read whole, its shape is versioned, and + normalising it would make ``schema_version`` a migration problem instead of a field. + """ + + __tablename__ = "run_summary" + + run_id: Mapped[str] = mapped_column(_ID, ForeignKey("run.id"), primary_key=True) + schema_version: Mapped[str] = mapped_column(String(32)) + termination: Mapped[str] = mapped_column(String(32), index=True) + flags: Mapped[list[str]] = mapped_column(JSON, default=list) + #: Headline scalars promoted out of the JSON for querying and sorting without deserialising: + #: "show me every run where peak number exceeded X" must not require reading 810 blobs. + final_so2_pptv: Mapped[float | None] = mapped_column(Float, nullable=True) + peak_h2so4_pptv: Mapped[float | None] = mapped_column(Float, nullable=True) + peak_number_cm3: Mapped[float | None] = mapped_column(Float, nullable=True) + final_surface_area: Mapped[float | None] = mapped_column(Float, nullable=True) + summary: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column(UtcDateTime) + + +__all__ = [ + "Base", + "DatasetVersionRow", + "JobRow", + "JobTransitionRow", + "ResultArtifactRow", + "RunConfigRow", + "RunDatasetRow", + "RunRow", + "RunSetRow", + "RunSummaryRow", + "UtcDateTime", +] diff --git a/studio/store/repository.py b/studio/store/repository.py new file mode 100644 index 0000000..778a8d7 --- /dev/null +++ b/studio/store/repository.py @@ -0,0 +1,259 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The operations the CLI, the API and the runner need -- and deliberately no others. + +**There is no update path for a config.** ADR-004 says a submitted config is immutable and an edit +produces a new config and a new run; a repository that offered ``update_config`` would make that a +convention people remember rather than a property of the system. :func:`ensure_config` is +get-or-create, keyed by the config's own hash, and is the only way a config enters the database. + +Everything here takes a ``Session`` rather than opening its own. Transaction boundaries belong to +the caller, because SQLite has a single writer (ADR-007) and only the caller knows what else belongs +in the same short transaction. +""" + +from __future__ import annotations + +import uuid +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from studio.resolve import ResolvedConfig +from studio.store.artifacts import ArtifactStore +from studio.store.models import ( + JobRow, + JobTransitionRow, + ResultArtifactRow, + RunConfigRow, + RunRow, + RunSetRow, + RunSummaryRow, +) + + +def _now() -> datetime: + return datetime.now(UTC) + + +def _new_id() -> str: + return uuid.uuid4().hex + + +def ensure_config(session: Session, config: ResolvedConfig) -> RunConfigRow: + """Get-or-create the config row. **The only way a config enters the database.** + + Idempotent by construction: the primary key is the config's own hash, so submitting the same + configuration from the CLI and from the API converges on one row rather than creating a second + identity for the same computation. + + Raises: + InconsistentConfigError: If the config has stale overrides. Persisting one would give an + inconsistent set of numbers a permanent identity. + """ + config.require_consistent() + config_hash = config.config.config_hash() + existing = session.get(RunConfigRow, config_hash) + if existing is not None: + return existing + row = RunConfigRow( + config_hash=config_hash, + schema_version=config.config.schema_version, + resolved_config=config.config.model_dump(mode="json"), + overrides={path: record.value for path, record in sorted(config.overrides.items())}, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def create_run_set( + session: Session, + *, + label: str = "", + axes: dict[str, Any] | None = None, + owner: str | None = None, +) -> RunSetRow: + """Create a sweep. A single run is the N = 1 case of one, with no axes.""" + row = RunSetRow(id=_new_id(), label=label, axes=axes or {}, owner=owner, created_at=_now()) + session.add(row) + session.flush() + return row + + +def create_run( + session: Session, + *, + run_set: RunSetRow, + config: ResolvedConfig, + label: str = "", + provenance: dict[str, Any] | None = None, + derived_from_run_id: str | None = None, +) -> RunRow: + """Create a run, ensuring its config exists first. + + ``provenance`` is the record written at submit time (ADR-006), stored verbatim. ``reproducible`` + is derived from it here rather than recomputed later, because it is a property of the moment the + run started, not of the checkout as it stands now. + """ + config_row = ensure_config(session, config) + reproducible: bool | None = None + if provenance is not None: + dirty = bool(provenance.get("sandbox", {}).get("dirty")) or any( + bool(sub.get("dirty")) for sub in provenance.get("submodules", {}).values() + ) + reproducible = not dirty + row = RunRow( + id=_new_id(), + run_set_id=run_set.id, + config_hash=config_row.config_hash, + label=label, + derived_from_run_id=derived_from_run_id, + provenance=provenance, + reproducible=reproducible, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def record_job( + session: Session, + *, + run: RunRow, + state: str, + backend: str = "local-subprocess", + work_dir: Path | str | None = None, + detail: str = "", +) -> JobRow: + """Create a job in its first state, with the matching transition.""" + now = _now() + job = JobRow( + id=_new_id(), + run_id=run.id, + state=state, + backend=backend, + work_dir=str(work_dir) if work_dir is not None else None, + detail=detail, + created_at=now, + updated_at=now, + ) + session.add(job) + session.add(JobTransitionRow(id=_new_id(), job_id=job.id, state=state, at=now, detail=detail)) + session.flush() + return job + + +def record_transition( + session: Session, *, job: JobRow, state: str, detail: str = "", exit_code: int | None = None +) -> JobRow: + """Advance a job and append the transition. + + Appends rather than replaces: the current state is a convenience column, and the transition list + is the record. A job whose state went backwards is visible here instead of being overwritten. + """ + now = _now() + job.state = state + job.detail = detail or job.detail + job.updated_at = now + if exit_code is not None: + job.exit_code = exit_code + session.add(JobTransitionRow(id=_new_id(), job_id=job.id, state=state, at=now, detail=detail)) + session.flush() + return job + + +def record_artifact( + session: Session, *, run: RunRow, kind: str, source: Path, store: ArtifactStore +) -> ResultArtifactRow: + """Store a file and record the pointer. The bytes never enter the database. + + Raises: + FileNotFoundError: If the file is missing -- surfaced rather than recorded as an absent row. + """ + stored = store.put(run.id, kind, Path(source)) + row = ResultArtifactRow( + id=_new_id(), + run_id=run.id, + kind=kind, + path=stored.key, + size_bytes=stored.size_bytes, + sha256=stored.sha256, + content_type=stored.content_type, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def record_summary(session: Session, *, run: RunRow, summary: dict[str, Any]) -> RunSummaryRow: + """Store the RunSummary, promoting four headline scalars into columns. + + Promoted so "every run whose peak number exceeded X" is a query rather than 810 + deserialisations. The scalars are read from the summary rather than recomputed, so the column + and the JSON can never disagree. + """ + series = summary.get("series", {}) + + def last(name: str) -> float | None: + values = series.get(name, {}).get("values") or [] + return float(values[-1]) if values else None + + def peak(name: str) -> float | None: + values = series.get(name, {}).get("values") or [] + return float(max(values)) if values else None + + row = RunSummaryRow( + run_id=run.id, + schema_version=str(summary.get("schema_version", "")), + termination=str(summary.get("termination", "unknown")), + flags=list(summary.get("flags", [])), + final_so2_pptv=last("SO2"), + peak_h2so4_pptv=peak("H2SO4"), + peak_number_cm3=peak("total_n"), + final_surface_area=last("SA"), + summary=summary, + created_at=_now(), + ) + session.add(row) + session.flush() + return row + + +def runs_for_config(session: Session, config_hash: str) -> list[RunRow]: + """Every run of one configuration, newest first. The cache lookup (ADR-006). + + Identical hash is necessary but **not sufficient** to reuse a result: the caller must also + compare the model version in each run's provenance, which is why this returns the runs rather + than a verdict. + """ + statement = ( + select(RunRow).where(RunRow.config_hash == config_hash).order_by(RunRow.created_at.desc()) + ) + return list(session.scalars(statement)) + + +def artifact_for(session: Session, run_id: str, kind: str) -> ResultArtifactRow | None: + statement = select(ResultArtifactRow).where( + ResultArtifactRow.run_id == run_id, ResultArtifactRow.kind == kind + ) + return session.scalars(statement).one_or_none() + + +__all__ = [ + "artifact_for", + "create_run", + "create_run_set", + "ensure_config", + "record_artifact", + "record_job", + "record_summary", + "record_transition", + "runs_for_config", +] diff --git a/studio/tests/conftest.py b/studio/tests/conftest.py index 0e7ecca..68a6ce8 100644 --- a/studio/tests/conftest.py +++ b/studio/tests/conftest.py @@ -13,6 +13,7 @@ from __future__ import annotations +import subprocess from pathlib import Path import pytest @@ -44,3 +45,64 @@ def paper_ensemble_runs(repo_root: Path) -> Path: f"docs/studio/adr/ADR-009-golden-file-strategy.md" ) return runs + + +def _git(path: Path, *args: str) -> None: + """Run git in ``path`` with an identity, so commits work on a machine with no global config.""" + subprocess.run( + [ + "git", + "-c", + "user.email=test@example.invalid", + "-c", + "user.name=Test", + "-c", + "commit.gpgsign=false", + *args, + ], + cwd=str(path), + check=True, + capture_output=True, + ) + + +def _make_repo(path: Path, filename: str = "file.txt") -> Path: + """A real git repo with one commit.""" + path.mkdir(parents=True, exist_ok=True) + _git(path, "init", "--quiet") + (path / filename).write_text("content\n", encoding="utf-8") + _git(path, "add", filename) + _git(path, "commit", "--quiet", "-m", "initial") + return path + + +@pytest.fixture +def fake_sandbox(tmp_path: Path) -> Path: + """A SANDBOX-shaped tree: a root repo with the three model submodules REGISTERED as such. + + Shared because both the provenance tests and the runner tests need a pinnable checkout, and CI + checks out no submodules -- so pointing at a synthetic one is what lets those tests run + everywhere rather than skipping in CI. + + Registered rather than merely nested: a nested repo the parent does not know about shows up in + ``git status --porcelain`` as untracked, so the parent reads dirty. + ``protocol.file.allow=always`` is required for local-path submodules (CVE-2022-39253); safe + here, since the "remote" is a directory the test just created. + """ + from studio.modelio.provenance import MODEL_SUBMODULES + + root = _make_repo(tmp_path / "SANDBOX") + for name in MODEL_SUBMODULES: + origin = _make_repo(tmp_path / "origins" / name) + _git( + root, + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "--quiet", + str(origin), + name, + ) + _git(root, "commit", "--quiet", "-m", "add submodules") + return root diff --git a/studio/tests/fixtures/__init__.py b/studio/tests/fixtures/__init__.py new file mode 100644 index 0000000..d8c6981 --- /dev/null +++ b/studio/tests/fixtures/__init__.py @@ -0,0 +1,3 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Fixture modules launched as real subprocesses by the runner tests.""" diff --git a/studio/tests/fixtures/fake_run.py b/studio/tests/fixtures/fake_run.py new file mode 100644 index 0000000..6d4595a --- /dev/null +++ b/studio/tests/fixtures/fake_run.py @@ -0,0 +1,56 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""A stand-in for ``studio.cli.run`` with the same argv shape and no model. + +``LocalSubprocessRunner`` takes ``entry_module`` as a parameter so its lifecycle -- queueing, +timeouts, cancellation, log capture, exit codes -- can be exercised in milliseconds instead of the +three to five minutes a real 10-day case costs. Nothing in the runner branches on the value; the +default is the real entry point, and the real one is exercised separately by +``test_a_bad_input_file_exits_distinctly_from_a_model_failure``. + +Behaviour comes from the ``STUDIO_FAKE_RUN`` environment variable, as JSON. An environment variable +rather than a file in the work directory because the subprocess can start before a file written +after ``submit()`` lands -- a race that would make these tests flaky in exactly the way process +tests usually are. + +Modes: ``ok`` (print and exit 0), ``fail`` (print to stderr and exit 1), ``sleep`` (sleep, to be +killed by the wall-clock limit or by cancellation), ``dump_env`` (print the thread-pinning variables +this process actually received). +""" + +from __future__ import annotations + +import json +import os +import sys +import time + +#: The pinning the runner is supposed to apply. Imported rather than restated so that the test +#: asserting it cannot pass against a stale copy of the list. +from studio.runner.local import THREAD_PINNING + + +def main(argv: list[str] | None = None) -> int: + argv = sys.argv[1:] if argv is None else argv + if len(argv) != 2: + print(f"usage: fake_run ; got {argv}", file=sys.stderr) + return 2 + + directive = json.loads(os.environ.get("STUDIO_FAKE_RUN", '{"mode": "ok"}')) + mode = directive.get("mode", "ok") + + if mode == "dump_env": + print(json.dumps({key: os.environ.get(key) for key in THREAD_PINNING})) + return 0 + if mode == "sleep": + time.sleep(float(directive.get("seconds", 30))) + return 0 + if mode == "fail": + print(directive.get("message", "failed"), file=sys.stderr) + return 1 + print(directive.get("message", "[studio] fake run complete")) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/tests/golden/REFERENCE_TOLERANCES.md b/studio/tests/golden/REFERENCE_TOLERANCES.md new file mode 100644 index 0000000..c040e01 --- /dev/null +++ b/studio/tests/golden/REFERENCE_TOLERANCES.md @@ -0,0 +1,323 @@ +# Reference tolerances — measured, not assumed + +**Measured:** 2026-08-13 · **Task:** Phase 0 / 0.7 (issue #70) · **Decides:** +[ASSUMPTION-2](../../../docs/studio/ASSUMPTIONS.md#assumption-2--the-archived-statenpz-files-are-the-golden-reference-at-a-tolerance-yet-to-be-measured) +· **Per:** [ADR-009](../../../docs/studio/adr/ADR-009-golden-file-strategy.md) + +This file records **what the deviation is**, not what anyone hoped it would be. No assertions were +written in this pass; Tier-B tolerances are chosen from these numbers and must cite this file. + +--- + +## Verdict + +**Reproduction is close, but it is not bit-for-bit.** + +Every physically meaningful quantity in both cases agrees with the archive to **≤ 3.4e-12**, and every +headline endpoint (final SO₂, peak/final H₂SO₄, peak/final particle number, peak/final wet surface +area, particulate sulfur, final size distribution) to **≤ 2.1e-12**. That is float64 round-off +territory — 1–4 decimal digits above machine epsilon after 1461 coupled intervals — not a physics +change. The archived ensemble **can** serve as a golden reference. + +It cannot serve as a *bit-exact* one: only ~31 % of the gas state-vector elements and ~1 % of the +aerosol samples reproduce exactly. An `array_equal` / `atol=0` golden test would fail on arrival. + +Two supporting measurements make the interpretation firm rather than hopeful: + +1. **The model is bit-for-bit deterministic today.** Case 121 was run twice in this environment; all + 18 stored arrays compared `array_equal == True`. So the ~1e-12 residual is *drift between the + archive's environment and today's*, not run-to-run noise. Re-running a case at fixed SHAs is + reproducible to the last bit, which is what makes a tight tolerance defensible. +2. **The deviation does not grow with time or with regime.** The worst per-quantity deviations land + at day 1.3, 3.2, 5.7, 8.4, 8.8, 9.3, 9.8 — scattered, not accumulating — and the `burst` / + `sabr330` case is if anything *quieter* than the `D2med` / `sabr220` one. That is the signature of + round-off, not of a diverging integration. + +--- + +## Provenance of this measurement + +| | | +| --- | --- | +| SANDBOX commit | `8c12422721debf9c8f5bba7bfe485df9c0e9bce7` (branch `feat/70-golden-tolerances`, off `studio/dev`) | +| `stratchem-jax` | `19fec0fafc35a5cae2a184ae5b1849236e2ca315` (`heads/main`) | +| `tomas-jax` | `39535ea021f9fe189dcfece5f3eb1167538d4503` (`remotes/origin/feat/marianna-dilution`) | +| `tuvx-jax` | `06f6777a73703fa607d3c87556f436c129f60cdf` (`heads/main`) | +| Platform | macOS 26.2, arm64, CPU only (`jax.devices() == [CpuDevice(id=0)]`) | +| Python / JAX / jaxlib | 3.12.12 / 0.11.0 / 0.11.0 | +| diffrax / numpy / scipy | 0.7.2 / 2.5.2 / 1.18.0 | +| Environment | `studio/requirements.lock`, `uv pip install -e . --no-deps` | +| Thread pinning | `OMP/OPENBLAS/MKL/VECLIB/NUMEXPR_NUM_THREADS=1`, `XLA_FLAGS=--xla_cpu_multi_thread_eigen=false` (`coupled/paper_ensemble/launch_parallel.py:26-30`) | + +Note the submodule SHAs above are **where the reference was measured**, not where the archive was +produced — the latter was never recorded (ADR-006), and this gap cannot be retrofitted. + +### What the archive is, and what changed since + +The two archived `state.npz` files carry mtimes of **2026-07-04 03:53** (case 121) and +**2026-07-04 06:07** (case 67). Changes landing after that date and reachable from today's SHAs: + +- `coupled/driver.py` — `bd289e9` (2026-07-08), the day-12.139 fix: SciPy-BDF fallback removed, + fail-fast probe budget, sticky `first_step=1e-2` retry. Its commit message claims the change + "lives on the failure branch only, so all existing 10-day ensemble results are unchanged". + **This measurement is consistent with that claim** — a 10-day run never reaches t = 2²⁰ s + (day 12.14), so the retry branch is never taken, and the observed deviation is round-off, not a + solver-path change. +- `stratchem-jax` `19fec0f` and `tuvx-jax` `06f6777` (both 2026-07-15) — standalone packaging only. +- `tomas-jax` `39535ea` (2026-07-04) — wires `coag_kernel_scale` into `coag_euler_step`. This one + may straddle the archive run itself. **Both cases here use `cg1` (scale = 1.0)**, for which the + wiring is a no-op, so it cannot be the source of the residual. A `cg0p5` or `cg2` case is *not* + covered by this measurement and should be checked separately before being used as a golden case. + +The residual therefore most plausibly comes from the toolchain (JAX/XLA/LLVM codegen: fused +multiply-add and reduction-order choices differ across versions), not from repository code. + +--- + +## Cases run + +| # | index | case_id | why | +| --- | --- | --- | --- | +| 1 | 121 | `30N_20km__sabr220__D2med__a1p0__nuc1__cg1` | the designated golden case: mid dilution, clean `sabr220` background | +| 2 | 67 | `30N_20km__sabr330__burst__a1p0__nuc1__cg1` | **contrast on two axes** — `burst` dilution (the fastest early transient in the ensemble, the regime ADR-009 flags for 8× aliasing) and the loaded `sabr330` background instead of the clean one. If the residual were regime-dependent, this is where it would show. | + +Command (from the SANDBOX root, thread-pinned as above): + +```bash +python -m coupled.paper_ensemble.run_ensemble one 121 +python -m coupled.paper_ensemble.run_ensemble one 67 +``` + +Both produced `steps = 1461`, a time axis **bit-identical** to the archive (`t` and `V_ratio` both +100 % equal), an identical species list and identical dry diameter bins — so nothing below is +confounded by a shifted grid. + +The CAVEATS note on the time axis is confirmed rather than assumed: the archived `t` spans +0 → 864000 s in 1461 samples with **41 distinct step sizes** ranging 0.56 s to 600 s and a **mean of +591.78 s**, not the nominal 600. Reconstructing time as `i × DT` would misplace day 10 by ~0.5 % +here and far more on longer runs. Every "at day" below comes from the stored `t`. + +### Wall clock + +| run | seconds | conditions | +| --- | --- | --- | +| case 121 | 287.3 | two cases concurrently, 1 thread each | +| case 67 | 275.2 | two cases concurrently, 1 thread each | +| case 121, repeat | 272.3 | alone | + +~4.6 min per 10-day / 80-bin case. Consistent with ADR-009's 3–5 min budget; Tier B of 4–6 cases is +~20–30 min serial, less in parallel. + +--- + +## Measured deviations + +Relative deviation is `|fresh − archived| / |archived|`, evaluated only where the archived series +exceeds **1e-6 × its own peak** (see "The one trap"). "max over t" is over all 1461 samples; +"at day" locates it using the **stored `t`**, never `i × DT`. + +Produced by: + +```bash +python studio/tests/golden/measure_deviation.py \ + --archive /Users/ali/Documents/GitHub/gas-phase-chemistry/SANDBOX/coupled/paper_ensemble/runs +``` + +### Case 121 — `30N_20km__sabr220__D2med__a1p0__nuc1__cg1` + +| quantity | endpoint / extremum | max over t | at day | +| --- | --- | --- | --- | +| SO₂, final [pptv] | 2.01e-14 | 4.54e-14 | 7.24 | +| H₂SO₄, peak [pptv] | 2.97e-14 | 3.38e-12 | 1.34 | +| H₂SO₄, final [pptv] | 1.19e-14 | 3.38e-12 | 1.34 | +| total N, peak [cm⁻³] | 1.80e-15 | 3.85e-13 | 1.34 | +| total N, final [cm⁻³] | 1.00e-14 | 3.85e-13 | 1.34 | +| wet SA, peak [µm² cm⁻³] | 1.73e-15 | 2.73e-14 | 8.40 | +| wet SA, final [µm² cm⁻³] | 9.01e-15 | 2.73e-14 | 8.40 | +| particulate S, peak | 5.52e-15 | 2.25e-14 | 8.80 | +| particulate S, final | 1.74e-14 | 2.25e-14 | 8.80 | +| wet radius, final [cm] | 1.05e-14 | 1.76e-14 | 1.34 | +| H₂SO₄ wt %, final | 2.16e-16 | 6.49e-16 | 8.55 | +| **final size dist** `n_cm3`, per bin | 2.04e-12 | — | 10.00 | +| **final size dist** `dNdlogDp`, per bin | 2.04e-12 | — | 10.00 | +| photolysis `J`, all 23 reactions × all steps | 1.09e-13 | — | — | + +Final size distribution: worst bin at **Dp_dry = 0.4611 µm**, 50 of 80 bins above the floor, +L2-relative deviation of the whole final profile **1.76e-13**. Integrated `sum(n_cm3)` final agrees +to 1.00e-14. + +Worst gas species (floored): `Cl2` 1.14e-11 @ day 0.354, then `H2SO4` 3.38e-12 @ day 1.340, +`Cl` 1.85e-12, `NO2` 1.67e-12, `NO` 9.20e-13. + +Bit-identical fraction: `t` 100 %, `V_ratio` 100 %, `T` 100 %, `J` 58.5 %, `h2so4wp` 48.7 %, +`x` 31.7 %, `radius_cm` 1.9 %, `SA` 1.6 %, `particulate_S` 1.4 %, `total_n` 1.1 %, +`dNdlogDp` 0.85 %, `n_cm3` 0.84 %. + +### Case 67 — `30N_20km__sabr330__burst__a1p0__nuc1__cg1` + +| quantity | endpoint / extremum | max over t | at day | +| --- | --- | --- | --- | +| SO₂, final [pptv] | 2.01e-14 | 3.16e-14 | 3.24 | +| H₂SO₄, peak [pptv] | 0.00e+00 | 4.44e-13 | 5.74 | +| H₂SO₄, final [pptv] | 2.74e-14 | 4.44e-13 | 5.74 | +| total N, peak [cm⁻³] | 7.43e-16 | 1.05e-13 | 9.30 | +| total N, final [cm⁻³] | 9.35e-15 | 1.05e-13 | 9.30 | +| wet SA, peak [µm² cm⁻³] | 3.39e-15 | 2.44e-14 | 9.44 | +| wet SA, final [µm² cm⁻³] | 2.15e-14 | 2.44e-14 | 9.44 | +| particulate S, peak | 5.02e-16 | 2.17e-14 | 9.84 | +| particulate S, final | 2.07e-14 | 2.17e-14 | 9.84 | +| wet radius, final [cm] | 1.92e-15 | 1.36e-14 | 8.22 | +| H₂SO₄ wt %, final | 2.16e-16 | 8.68e-16 | 1.36 | +| **final size dist** `n_cm3`, per bin | 1.91e-12 | — | 10.00 | +| **final size dist** `dNdlogDp`, per bin | 1.91e-12 | — | 10.00 | +| photolysis `J`, all 23 reactions × all steps | 1.09e-13 | — | — | + +Final size distribution: worst bin at **Dp_dry = 0.1027 µm**, 54 of 80 bins above the floor, +L2-relative deviation **1.53e-13**. Integrated `sum(n_cm3)` final agrees to 9.35e-15. + +Worst gas species (floored): `Cl2` 5.56e-12 @ day 0.347, then `Cl` 7.00e-13, `NO` 6.11e-13, +`H2SO4` 4.44e-13, `ClO` 1.79e-13. + +Bit-identical fraction: `x` 31.5 %, `particulate_S` 2.5 %, `SA` 0.9 %, `total_n` 0.75 %, +`n_cm3` 0.73 %. + +### Determinism control (same environment, two runs of case 121) + +All 18 stored arrays `array_equal == True`. Wall clock 287.3 s vs 272.3 s. The gas/microphysics path +carries no seed and no nondeterministic reduction on this platform. + +--- + +## The one trap: unguarded relative error explodes on near-zero species + +Without the 1e-6 floor, the worst relative deviation across the gas state vector is **4.24e+04** +(case 121) / **3.81e+03** (case 67). Both are `O1D`: + +``` +case 121, day 2.167: archived -4.850e-39 fresh 2.057e-34 rel 4.242e+04 +case 67, day 0.104: archived -3.698e-36 fresh -1.411e-32 rel 3.814e+03 +``` + +`O1D` peaks at ~3.1 molec cm⁻³ and collapses to O(1e-35) — *and to small negative values* — at night. +`O` behaves the same way (`min_nonzero` 2.2e-299). These are solver residuals oscillating about zero, +not physics: the absolute difference is ~1e-34 molec cm⁻³ against a species peak of 3 molec cm⁻³. + +**Consequence for the assertions written next:** a golden test that computes relative error over the +raw `x` array without a magnitude floor will fail by four orders of magnitude for a difference of +1e-34 molec cm⁻³. Either floor at a fraction of the series peak (as here) or assert per species on an +absolute floor. Do not "fix" this by widening a global tolerance to 1e5 — that would make the test +meaningless for every real species. + +--- + +## Recommended tolerances for the Tier-B assertions (not yet written) + +Proposed, each with the rationale ADR-009 requires. All are ~50–100× the measured deviation, which +leaves headroom for a toolchain bump without admitting a physics change (the smallest physically +interesting change in any of these quantities is ≫ 1e-6 relative). + +| assertion target | proposed `rtol` | rationale | +| --- | --- | --- | +| headline endpoints — final SO₂, peak/final H₂SO₄, peak/final N, peak/final wet SA, particulate S | `1e-12` | measured ≤ 2.9e-14 across both cases; 1e-12 is ~35× headroom and still ~6 orders below any meaningful physics change | +| any stored series, max over time | `1e-10` | measured worst 3.4e-12 (`H2SO4`, case 121, day 1.34); 1e-10 covers the `Cl2` trace-species worst case of 1.1e-11 with ~10× margin | +| final size distribution, per bin, bins above 1e-6 × peak | `1e-10` | measured 2.0e-12 worst bin; per-bin conditioning is worse than integrated N, so it gets its own (looser) number | +| photolysis `J` | `1e-11` | measured 1.09e-13, identical in both cases — TUV-x is the most reproducible part of the pipeline | +| time axis `t`, `V_ratio`, `T` | **exact** | measured bit-identical in both cases; these are analytic, not integrated, so any drift is a real bug | +| dry bin edges `dp_mid_um` | `1e-15` | **not exact — corrected 2026-08-14.** Bit-identical only when the fresh run is produced by `run_ensemble`. A run produced by Studio differs in 44 of 80 bins by up to **8.1e-16**, because task 0.5 adopted `sqrt(a*b)` where `run_ensemble` writes `10**(0.5*(log10 a + log10 b))` — algebraically identical, differently rounded. Asserting `exact` here would pass against the old pipeline and fail against every Studio run, looking like a physics regression over a spelling difference. | +| near-zero species (`O1D`, `O`, and any series below 1e-6 × peak) | **excluded** | see "The one trap"; assert an absolute floor instead if coverage is wanted | + +These numbers describe **this environment**. A JAX/jaxlib bump is the most likely thing to move them, +and the correct response is to re-run this measurement and update this file — not to widen a +tolerance in a test file. + +## Independently reproduced (2026-08-14) + +The golden case was re-run a second time by a different route — through `python -m studio.cli.run` +(the task-0.6 entry point) rather than `run_ensemble` — and compared against the archive again: + +| quantity | first measurement | independent re-run | +| --- | --- | --- | +| SO₂ final | 2.01e-14 | 2.01e-14 | +| H₂SO₄ final / peak | 1.19e-14 / 2.97e-14 | 1.19e-14 / 2.98e-14 | +| total N final / peak | 1.00e-14 / 1.80e-15 | 1.00e-14 / 1.80e-15 | +| wet SA final / peak | 9.01e-15 / 1.73e-15 | 9.01e-15 / 1.73e-15 | +| particulate S final / peak | 1.74e-14 / 5.52e-15 | 1.74e-14 / 5.52e-15 | +| final size dist, worst bin | 2.04e-12 | 2.04e-12 | +| gas elements exactly equal | ~31 % | 31.7 % | + +`t` and `V_ratio` bit-identical, as first measured. **`dp_mid_um` was not**, which is what produced +the correction in the table above — and it could only surface via the Studio pipeline, which did not +exist on the branch where the first measurement was taken. `dNdlogDp` inherits that difference at +3.2e-13, comfortably inside its own `1e-10`, so only the `dp_mid_um` row needed changing. + +## What this comparison assumes about the archive — settled 2026-08-14 + +The archived files carry **no provenance record**, so the code that produced them is not knowable +from the data (that gap is exactly what ADR-006 closes going forward and explicitly cannot close +retroactively). This comparison therefore rests on an assumption, and it is written down rather than +left implicit: + +> **`coupled/paper_ensemble/runs/` is the uniform 810-run factorial produced by `run_ensemble.py`'s +> axis tables, and is a valid reproduction reference.** Confirmed by Ali, 2026-08-14. + +**Every other archive directory is excluded**, because they were produced differently — some +initialise from a spun-up control run (`runs_60day/` uses `frank_control_ic.json`), some predate that +practice, some carry different configurations: + +`runs_60day` · `runs_bgstop` · `runs_bgstop_ctrl` · `runs_boxsize` · `runs_geo` · `runs_no_sai` · +`runs_special` · `runs_start_time` · `runs_start_time_geo` + +Reproducing one of those from the axis tables would compare two different computations, where a pass +is luck and a failure means nothing. `paper_cases.py` only maps the factorial's case IDs, so this +exclusion is structural rather than a convention someone has to remember. + +Supporting evidence, for the record: all 810 `state.npz` in `runs/` were written on **2026-07-04** +within one 02:18–06:53 window, and six cases spanning four dilution regimes and both backgrounds +reproduce to ≤ 4.1e-14 on every endpoint. Mtimes are not provenance, but a heterogeneous set would +not behave that way. + +--- + +## Six-case measurement, 2026-08-14 + +The original measurement covered two cases. All six curated Tier-B cases have since been re-run and +compared (the four dilution regimes on `sabr220`, plus `D2med` and `burst` on `sabr330`): + +| case | worst endpoint | worst series | `t`/`V_ratio`/`T` | +| --- | --- | --- | --- | +| D1low · sabr220 | 4.1e-14 | 1.6e-11 | bit-identical | +| D2med · sabr220 | 3.0e-14 | 1.6e-11 | bit-identical | +| D3high · sabr220 | 1.7e-14 | 1.6e-11 | bit-identical | +| burst · sabr220 | 3.5e-14 | 1.6e-11 | bit-identical | +| D2med · sabr330 | 2.4e-14 | 1.5e-11 | bit-identical | +| burst · sabr330 | 3.7e-14 | 7.4e-12 | bit-identical | + +Every endpoint is ≤ 4.1e-14 against a `1e-12` tolerance; every series ≤ 1.6e-11 against `1e-10`. +Wall clock 256–293 s per case. + +**Correction to the timing claim.** The two-case measurement noted the worst H₂SO₄ deviation at day +1.34 and this file previously read as though deviation were tied to the early nucleation burst. +Across six cases the worst-deviation days are **5.83, 1.34, 9.27, 2.15, 3.03, 5.74** — no common +feature. The deviation is a flat ~1e-14 baseline with occasional spikes; it is float round-off +scattered through the run, **not** accumulation and not burst-timed. + +**Where the worst per-bin size-distribution deviation sits.** All four `sabr220` cases give an +*identical* 1.63e-11, at the same cell — t = 0.042 d, bin 5 (Dp 3.2 nm), 4.218 counts — while the two +`sabr330` cases peak late and elsewhere (t = 7.34 d bin 6; t = 8.41 d bin 11). The common thread is +not timing but **sparsity**: every one of them is a bin holding 1.7–4.2 particles cm⁻³. It is +inherited from `n_cm3`, not from the `dlogdp` normalisation — those divisors agree to 6.1e-15. + +Regenerate the figures behind these numbers with `plot_fidelity.py` in this directory. + +--- + +## Not covered by this measurement + +- Only `cg1` was measured; `cg0p5` / `cg2` cases may be affected by `tomas-jax` `39535ea` landing + around the archive date. Verify before adopting one as a golden case. +- Only `30N_20km` and only 10-day / 80-bin runs. The 60-day runs cross the day-12.14 boundary where + the sticky-retry branch *is* taken, and are expected to differ from any pre-`bd289e9` archive. +- Only the two dilution regimes above; `D1low`, `D3high`, `D5vhigh` and the `cesm` background are + unmeasured. +- Only this machine, single-threaded. Deviations on CI's Linux runners are unmeasured, and Tier B + does not run there. diff --git a/studio/tests/golden/__init__.py b/studio/tests/golden/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/studio/tests/golden/fixtures/tier_a_short_run.npz b/studio/tests/golden/fixtures/tier_a_short_run.npz new file mode 100644 index 0000000..e61f433 Binary files /dev/null and b/studio/tests/golden/fixtures/tier_a_short_run.npz differ diff --git a/studio/tests/golden/make_fixture.py b/studio/tests/golden/make_fixture.py new file mode 100644 index 0000000..2c1a6f9 --- /dev/null +++ b/studio/tests/golden/make_fixture.py @@ -0,0 +1,158 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Build the Tier-A golden fixture: a short run, reduced on a UNIFORM stride. + + python -m studio.tests.golden.make_fixture # writes tier_a_short_run.npz + python -m studio.tests.golden.make_fixture --check # reports drift, writes nothing + +**Uniform stride, not an adaptive or thinned grid.** A coarsening grid aliases the morning +particle-number spike by up to 8x (ADR-009, and ``CAVEATS.md``), so a fixture built on one would +encode the aliasing and then assert it forever. Every ``STRIDE``-th stored sample, and the last one +so the endpoint is always present. + +The fixture is committed because it is small (tens of kB) and because Tier A must not depend on the +3.6 MB archive, which is gitignored and absent on a fresh clone. Regenerating it is a deliberate act +with a recorded reason -- ``--check`` exists so drift can be measured without overwriting the +reference by accident. +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path +from typing import Any + +import numpy as np + +from studio.resolve import resolve +from studio.schema import RunConfig +from studio.tests.golden.tolerances import ( + RTOL_HEADLINE, + RTOL_SERIES, + worst_relative_deviation, +) + +#: Where the fixture lives. Next to the tests that read it. +FIXTURE_PATH = Path(__file__).parent / "fixtures" / "tier_a_short_run.npz" + +#: 1 day, 40 bins: the cheapest run that still exercises the whole pipeline -- gas chemistry, TUV-x +#: photolysis, all three microphysics processes and dilution. ~21 s on an M-series CPU. +TIER_A_DAYS = 1 +TIER_A_BINS = 40 + +#: Keep every 4th sample. 1 day at 600 s nominal is ~147 stored samples, so this is 38 -- enough +#: for a series comparison to be meaningful, small enough to commit. +STRIDE = 4 + +#: Arrays reduced along time. Everything a Tier-A assertion compares. +_TIME_SERIES_KEYS = ( + "t", + "x", + "SA", + "radius_cm", + "h2so4wp", + "particulate_S", + "T", + "V_ratio", + "total_n", + "n_cm3", + "dNdlogDp", +) + +#: Arrays that are not time series and are stored whole. +_WHOLE_KEYS = ("species", "M", "dp_mid_um", "J_equations") + +#: Photolysis is on the INTERVAL axis (n_times - 1), not the sample axis, so it strides separately. +#: Included because ``REFERENCE_TOLERANCES.md`` measured it as the most reproducible part of the +#: pipeline (1.09e-13), which makes any drift there a strong signal rather than noise. +_INTERVAL_SERIES_KEYS = ("J_tmid", "J") + + +def tier_a_config() -> Any: + """The Tier-A case: the golden case's defaults, shortened and coarsened.""" + payload = RunConfig().model_dump() + payload["schedule"]["duration_days"] = TIER_A_DAYS + payload["microphysics"]["n_bins"] = TIER_A_BINS + return resolve(RunConfig.model_validate(payload)) + + +def run_tier_a_case(out_dir: Path) -> Path: + """Run the case and return the path to its ``state.npz``.""" + from studio.modelio.execute import run_and_write + + return run_and_write(tier_a_config(), out_dir)["state"] + + +def reduce_state(npz_path: Path) -> dict[str, np.ndarray]: + """Uniform-stride reduction of a ``state.npz``, keeping the final sample.""" + with np.load(npz_path, allow_pickle=True) as archive: + data = {key: archive[key] for key in archive.files} + n_times = len(data["t"]) + keep = sorted(set(range(0, n_times, STRIDE)) | {n_times - 1}) + reduced: dict[str, np.ndarray] = {"kept_indices": np.asarray(keep), "n_times_full": n_times} + for key in _TIME_SERIES_KEYS: + reduced[key] = np.asarray(data[key])[keep] + n_intervals = len(data["J_tmid"]) + keep_intervals = sorted(set(range(0, n_intervals, STRIDE)) | {n_intervals - 1}) + reduced["kept_intervals"] = np.asarray(keep_intervals) + for key in _INTERVAL_SERIES_KEYS: + reduced[key] = np.asarray(data[key])[keep_intervals] + for key in _WHOLE_KEYS: + reduced[key] = np.asarray(data[key]) + return reduced + + +def write_fixture(out_dir: Path) -> Path: + """Run the case, reduce it, write the fixture.""" + state = run_tier_a_case(out_dir) + reduced = reduce_state(state) + FIXTURE_PATH.parent.mkdir(parents=True, exist_ok=True) + np.savez_compressed(FIXTURE_PATH, **reduced) + size_kb = FIXTURE_PATH.stat().st_size / 1024 + print( + f"wrote {FIXTURE_PATH} ({size_kb:.0f} kB): {len(reduced['kept_indices'])} of " + f"{reduced['n_times_full']} samples, stride {STRIDE}" + ) + return FIXTURE_PATH + + +def check_against_fixture(out_dir: Path) -> int: + """Re-run and report drift against the committed fixture. Writes nothing.""" + if not FIXTURE_PATH.is_file(): + print(f"no fixture at {FIXTURE_PATH}; run without --check first", file=sys.stderr) + return 2 + fresh = reduce_state(run_tier_a_case(out_dir)) + with np.load(FIXTURE_PATH, allow_pickle=True) as archive: + reference = {key: archive[key] for key in archive.files} + species = [str(name) for name in reference["species"]] + print(f"{'quantity':22s} {'worst rel':>12s} tolerance") + for name, rtol in (("SO2", RTOL_HEADLINE), ("H2SO4", RTOL_HEADLINE)): + index = species.index(name) + worst = worst_relative_deviation(fresh["x"][:, index], reference["x"][:, index]) + print(f"{name:22s} {worst:12.3e} {rtol:.0e}") + for key in ("total_n", "SA", "particulate_S", "dNdlogDp"): + worst = worst_relative_deviation(fresh[key], reference[key]) + print(f"{key:22s} {worst:12.3e} {RTOL_SERIES:.0e}") + return 0 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--check", action="store_true", help="re-run and report drift; do not overwrite the fixture" + ) + parser.add_argument( + "--work-dir", type=Path, default=None, help="where to run (default: a tempdir)" + ) + args = parser.parse_args() + + import tempfile + + with tempfile.TemporaryDirectory(prefix="studio-golden-") as tmp: + out_dir = args.work_dir or Path(tmp) + return check_against_fixture(out_dir) if args.check else (write_fixture(out_dir) and 0) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/tests/golden/measure_all_cases.py b/studio/tests/golden/measure_all_cases.py new file mode 100644 index 0000000..8cfc93e --- /dev/null +++ b/studio/tests/golden/measure_all_cases.py @@ -0,0 +1,98 @@ +"""Re-run the curated Tier-B cases and dump every deviation to JSON, incrementally. + + python -m studio.tests.golden.measure_all_cases + +This is the tool behind the numbers in ``REFERENCE_TOLERANCES.md``. It is not a test: it asserts +nothing, so re-measuring never "fails" -- it reports, and a human decides whether the new numbers +are acceptable. + +**Incremental on purpose.** Six 10-day cases is ~28 minutes and an interrupted run used to leave +nothing behind; each case now writes its results the moment it finishes, and an already-present run +is reused rather than repeated. Outputs live in the work directory you name, not a temp dir, so they +survive to be re-analysed. +""" + +import json +import sys +import time +from pathlib import Path + +import numpy as np + +from studio.modelio.execute import run_and_write +from studio.tests.golden.paper_cases import TIER_B_CASES, config_for_case +from studio.tests.golden.tolerances import ( + endpoint_deviations, + worst_relative_deviation, +) + +ARCHIVE = Path( + "/Users/ali/Documents/GitHub/gas-phase-chemistry/SANDBOX/coupled/paper_ensemble/runs" +) +OUT = Path(sys.argv[1]) +OUT.mkdir(parents=True, exist_ok=True) +RESULTS = OUT / "deviations.json" +GAS = ("SO2", "SO3", "H2SO4", "OH") +SERIES = ("SA", "radius_cm", "h2so4wp", "particulate_S", "total_n") + +results = json.loads(RESULTS.read_text()) if RESULTS.is_file() else {} +for case in TIER_B_CASES: + if case in results: + print(f"[skip] {case} already measured", flush=True) + continue + state = OUT / case / "state.npz" + t0 = time.perf_counter() + if not state.is_file(): + print(f"[run ] {case} ...", flush=True) + state = run_and_write(config_for_case(case), OUT / case)["state"] + wall = time.perf_counter() - t0 + + with ( + np.load(state, allow_pickle=True) as f, + np.load(ARCHIVE / case / "state.npz", allow_pickle=True) as a, + ): + fresh = {k: f[k] for k in f.files} + arch = {k: a[k] for k in a.files} + species = [str(s) for s in arch["species"]] + entry = {"wall_s": wall, "endpoints": {}, "series": {}, "time_days": arch["t"].tolist()} + + for name in GAS: + i = species.index(name) + entry["endpoints"][name] = endpoint_deviations(fresh["x"][:, i], arch["x"][:, i]) + entry["series"][name] = worst_relative_deviation(fresh["x"][:, i], arch["x"][:, i]) + # deviation vs time, floored the same way the harness floors it + ref = np.abs(arch["x"][:, i]) + peak = float(np.nanmax(ref)) + ok = ref > peak * 1e-6 + dev = np.where( + ok, np.abs(fresh["x"][:, i] - arch["x"][:, i]) / np.where(ok, ref, 1), np.nan + ) + entry.setdefault("dev_vs_time", {})[name] = dev.tolist() + for key in SERIES: + entry["endpoints"][key] = endpoint_deviations(fresh[key], arch[key]) + entry["series"][key] = worst_relative_deviation(fresh[key], arch[key]) + entry["series"]["dNdlogDp"] = worst_relative_deviation(fresh["dNdlogDp"], arch["dNdlogDp"]) + entry["exact"] = {k: bool(np.array_equal(fresh[k], arch[k])) for k in ("t", "V_ratio", "T")} + # the near-zero trap, as data: unfloored relative error against series magnitude + unfloored = {} + for name in ("O1D", "O", "SO2", "H2SO4"): + if name in species: + i = species.index(name) + ref = np.abs(arch["x"][:, i]) + peak = float(np.nanmax(ref)) + nz = ref > 0 + rel = np.abs(fresh["x"][:, i] - arch["x"][:, i])[nz] / ref[nz] + unfloored[name] = { + "peak": peak, + "worst_unfloored": float(np.max(rel)) if rel.size else 0.0, + "worst_floored": worst_relative_deviation(fresh["x"][:, i], arch["x"][:, i]), + } + entry["near_zero"] = unfloored + + results[case] = entry + RESULTS.write_text(json.dumps(results, indent=1)) + print( + f"[done] {case} {wall:6.1f}s worst series {max(entry['series'].values()):.2e}", flush=True + ) + +print(f"\n{len(results)}/{len(TIER_B_CASES)} cases measured -> {RESULTS}") diff --git a/studio/tests/golden/measure_deviation.py b/studio/tests/golden/measure_deviation.py new file mode 100644 index 0000000..b6b1f84 --- /dev/null +++ b/studio/tests/golden/measure_deviation.py @@ -0,0 +1,208 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Measure the per-quantity deviation between a freshly-run case and the archived ``state.npz``. + +**This is a measurement tool, not a test.** It contains no assertions and no tolerances of its own; +it prints what the deviation *is*. The numbers it produced are recorded in +[`REFERENCE_TOLERANCES.md`](REFERENCE_TOLERANCES.md) together with the SHAs they were taken at +(ADR-009: measure before asserting). Tier-B assertions, when written, cite that file. + +Usage (from the SANDBOX root, with the studio venv): + + python -m coupled.paper_ensemble.run_ensemble one 121 # produce the fresh run + python studio/tests/golden/measure_deviation.py 30N_20km__sabr220__D2med__a1p0__nuc1__cg1 \\ + --archive [--fresh ] + +``--archive`` must point at a **read-only** copy of the archived ensemble: those outputs are not +regenerable at their original provenance (ADR-006) and this script never writes to that tree. + +Relative-error floor +-------------------- +Relative error is evaluated only where the archived series exceeds ``FLOOR_FRAC`` times its own +peak. Without a floor the metric is dominated by night-time O(1e-35) O1D/O values that oscillate +about zero — solver noise on a species whose peak is ~3 molec/cm3. See REFERENCE_TOLERANCES.md, +"The one trap". +""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +import numpy as np + +#: Samples below this fraction of the archived series peak are excluded from the relative metric. +FLOOR_FRAC = 1e-6 + + +def _rel_series(fresh: np.ndarray, arch: np.ndarray, t: np.ndarray) -> tuple[float, float]: + """Max relative deviation over the time axis, and the day at which it occurs.""" + peak = np.nanmax(np.abs(arch)) + mask = np.abs(arch) > FLOOR_FRAC * peak + rel = np.full(arch.shape, np.nan) + rel[mask] = np.abs(fresh[mask] - arch[mask]) / np.abs(arch[mask]) + i = int(np.nanargmax(rel)) + return float(np.nanmax(rel)), float(t[i] / 86400.0) + + +def _rel_scalar(fresh: float, arch: float) -> float: + if arch == 0.0: + return 0.0 if fresh == 0.0 else float("nan") + return abs(fresh - arch) / abs(arch) + + +def compare(arch_npz: Path, fresh_npz: Path) -> None: + a = np.load(arch_npz, allow_pickle=True) + f = np.load(fresh_npz, allow_pickle=True) + + if a["t"].shape != f["t"].shape: + raise ValueError( + f"time-grid length differs: archived {a['t'].shape} vs fresh {f['t'].shape}" + ) + t = a["t"] + print(f"time axis identical: {np.array_equal(a['t'], f['t'])}") + print(f"species list identical: {list(a['species']) == list(f['species'])}") + + print("\n-- bit-for-bit per stored array --") + for k in a.files: + A, F = a[k], f[k] + eq = A.shape == F.shape and np.array_equal(A, F) + if eq or A.dtype.kind in "US": + print(f" {k}: equal={eq}") + else: + frac = 100.0 * (A == F).sum() / A.size + print( + f" {k}: equal=False bit-identical {frac:.2f}% " + f"max|abs diff|={np.nanmax(np.abs(F.astype(float) - A.astype(float))):.3e}" + ) + + M = float(a["M"]) + ix = {s: i for i, s in enumerate(list(a["species"]))} + so2_a, so2_f = a["x"][:, ix["SO2"]] / M * 1e12, f["x"][:, ix["SO2"]] / M * 1e12 + h_a, h_f = a["x"][:, ix["H2SO4"]] / M * 1e12, f["x"][:, ix["H2SO4"]] / M * 1e12 + + rows: list[tuple[str, float, float, float]] = [] + + def add(name: str, fv: float, av: float, series: tuple[np.ndarray, np.ndarray]) -> None: + r, day = _rel_series(series[0], series[1], t) + rows.append((name, _rel_scalar(fv, av), r, day)) + + add("SO2, final [pptv]", float(so2_f[-1]), float(so2_a[-1]), (so2_f, so2_a)) + add("H2SO4, peak [pptv]", float(h_f.max()), float(h_a.max()), (h_f, h_a)) + add("H2SO4, final [pptv]", float(h_f[-1]), float(h_a[-1]), (h_f, h_a)) + add( + "total N, peak [cm-3]", + float(f["total_n"].max()), + float(a["total_n"].max()), + (f["total_n"], a["total_n"]), + ) + add( + "total N, final [cm-3]", + float(f["total_n"][-1]), + float(a["total_n"][-1]), + (f["total_n"], a["total_n"]), + ) + add( + "wet SA, peak [um2 cm-3]", + float(np.nanmax(f["SA"])), + float(np.nanmax(a["SA"])), + (f["SA"], a["SA"]), + ) + add("wet SA, final [um2 cm-3]", float(f["SA"][-1]), float(a["SA"][-1]), (f["SA"], a["SA"])) + add( + "particulate S, peak", + float(np.nanmax(f["particulate_S"])), + float(np.nanmax(a["particulate_S"])), + (f["particulate_S"], a["particulate_S"]), + ) + add( + "particulate S, final", + float(f["particulate_S"][-1]), + float(a["particulate_S"][-1]), + (f["particulate_S"], a["particulate_S"]), + ) + add( + "wet radius, final [cm]", + float(f["radius_cm"][-1]), + float(a["radius_cm"][-1]), + (f["radius_cm"], a["radius_cm"]), + ) + add( + "H2SO4 wt%, final", + float(f["h2so4wp"][-1]), + float(a["h2so4wp"][-1]), + (f["h2so4wp"], a["h2so4wp"]), + ) + + print("\n-- per-quantity relative deviation --") + print(f"{'quantity':30s} {'endpoint':>10s} {'max over t':>12s} {'at day':>8s}") + for nm, sc, se, day in rows: + print(f"{nm:30s} {sc:10.2e} {se:12.2e} {day:8.2f}") + + print("\n-- final size distribution (dry bins) --") + for key in ("n_cm3", "dNdlogDp"): + av, fv = a[key][-1], f[key][-1] + mask = np.abs(av) > FLOOR_FRAC * np.abs(av).max() + rel = np.abs(fv[mask] - av[mask]) / np.abs(av[mask]) + j = int(np.argmax(rel)) + bins = np.where(mask)[0] + print( + f" {key}: max per-bin rel dev {rel.max():.2e} at Dp_dry=" + f"{a['dp_mid_um'][bins[j]]:.4g} um ({mask.sum()}/{len(av)} bins above floor); " + f"L2 rel {np.linalg.norm(fv - av) / np.linalg.norm(av):.2e}" + ) + + print("\n-- photolysis J --") + Ja, Jf = a["J"], f["J"] + mask = np.abs(Ja) > FLOOR_FRAC * np.abs(Ja).max() + rel = np.abs(Jf[mask] - Ja[mask]) / np.abs(Ja[mask]) + print(f" max rel dev {rel.max():.2e} over {mask.sum()} entries above floor") + + print("\n-- gas species, worst 5 (floored) --") + worst = [] + for s, j in ix.items(): + av, fv = a["x"][:, j], f["x"][:, j] + if np.abs(av).max() == 0.0: + continue + r, day = _rel_series(fv, av, t) + worst.append((r, s, day, float(np.abs(av).max()))) + worst.sort(reverse=True) + for r, s, day, pk in worst[:5]: + print(f" {s:8s} {r:10.2e} at day {day:6.3f} (series peak {pk:.3e} molec/cm3)") + + print("\n-- unguarded worst over the whole gas state vector (shows why the floor exists) --") + xa, xf = a["x"], f["x"] + nz = np.abs(xa) > 0 + ru = np.zeros_like(xa) + ru[nz] = np.abs(xf[nz] - xa[nz]) / np.abs(xa[nz]) + i, j = np.unravel_index(int(ru.argmax()), ru.shape) + sp = list(a["species"])[j] + print( + f" {ru.max():.3e} in {sp} at day {t[i] / 86400.0:.3f}: " + f"archived {xa[i, j]:.3e} vs fresh {xf[i, j]:.3e} " + f"(series peak {np.abs(xa[:, j]).max():.3e} molec/cm3)" + ) + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("case_id") + p.add_argument( + "--archive", + required=True, + type=Path, + help="read-only root holding /state.npz for the archived ensemble", + ) + p.add_argument( + "--fresh", + type=Path, + default=Path(__file__).resolve().parents[3] / "coupled/paper_ensemble/runs", + help="root holding the freshly-produced /state.npz", + ) + args = p.parse_args() + print(f"=== {args.case_id} ===") + compare(args.archive / args.case_id / "state.npz", args.fresh / args.case_id / "state.npz") + + +if __name__ == "__main__": + main() diff --git a/studio/tests/golden/paper_cases.py b/studio/tests/golden/paper_cases.py new file mode 100644 index 0000000..5a7d57f --- /dev/null +++ b/studio/tests/golden/paper_cases.py @@ -0,0 +1,133 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Case ID -> ``RunConfig``, for the Tier-B archive comparison. + +The 810-run ensemble names each case by its axis levels joined with ``__`` -- +``30N_20km__sabr220__D2med__a1p0__nuc1__cg1`` -- and that name IS the parameter set +(``run_ensemble.py:84``). This module reverses it, so a Tier-B test can say "reproduce this archived +directory" and get the config that produced it. + +Lives under ``studio/tests/`` rather than in ``studio/``: task 0.2 deliberately kept the paper +ensemble's axes out of the package, on the grounds that they would earn a home there when something +needed them. Tier B needs them *as test data*, which is not the same as needing a preset library, so +they stay here until production asks. + +Token tables are copied from ``run_ensemble.py:61-76`` and checked against it by +``test_tier_b_archive.py`` -- a token that stops matching is a case that would silently be run with +the wrong parameters and compared against the right archive. +""" + +from __future__ import annotations + +from typing import Final + +from studio.resolve import ResolvedConfig, resolve +from studio.schema import BackgroundAerosol, DilutionRegime, RunConfig + +#: site token -> (latitude, T [K], p [mbar], H2O [ppmv]). ``run_ensemble.py:61-65``. +SITES: Final[dict[str, tuple[float, float, float, float]]] = { + "30N_20km": (30.0, 210.0, 55.0, 6.9104), + "60N_15km": (60.0, 210.0, 120.0, 3.1673), + "30N_20km_213K": (30.0, 213.0, 55.0, 10.1834), +} + +#: background token -> (aerosol distribution, background SO2 [pptv]). ``run_ensemble.py:67-71``. +BACKGROUNDS: Final[dict[str, tuple[BackgroundAerosol, float]]] = { + "sabr330": (BackgroundAerosol.SABR_330, 20.0), + "sabr220": (BackgroundAerosol.SABR_220, 20.0), + "cesm": (BackgroundAerosol.CESM_G6, 100.0), +} + +#: dilution token -> regime. ``run_ensemble.py:72-73``. +DILUTIONS: Final[dict[str, DilutionRegime]] = { + "D1low": DilutionRegime.D1, + "D2med": DilutionRegime.D2, + "D3high": DilutionRegime.D3, + "burst": DilutionRegime.BURST, + "D5vhigh": DilutionRegime.D5, +} + +#: condensation alpha, nucleation scale, coagulation scale. ``run_ensemble.py:74-76``. +STICKING: Final[dict[str, float]] = {"a0p5": 0.5, "a1p0": 1.0} +NUCLEATION: Final[dict[str, float]] = {"nuc0p01": 0.01, "nuc1": 1.0, "nuc100": 100.0} +COAGULATION: Final[dict[str, float]] = {"cg0p5": 0.5, "cg1": 1.0, "cg2": 2.0} + +#: The ensemble's fixed values, which are the schema's defaults: 80 bins, day 172, 00:00, 10 days, +#: ion pair rate 30, SO2+HO2 1e-18, aerosol->J and heating off. Asserted, not assumed, by +#: ``test_tier_b_archive.py``. +ENSEMBLE_BINS: Final = 80 +ENSEMBLE_DAYS: Final = 10 + + +def config_for_case(case_id: str) -> ResolvedConfig: + """The resolved config that produced the archived directory ``case_id``. + + Raises: + ValueError: On an unknown token or the wrong number of them. A mistyped case would otherwise + be run with default parameters and compared against a real archive, which fails in a way + that looks like a physics regression. + """ + tokens = case_id.split("__") + if len(tokens) != 6: + raise ValueError( + f"case id {case_id!r} has {len(tokens)} tokens, expected 6: " + f"site__background__dilution__sticking__nucleation__coag" + ) + site, background, dilution, sticking, nucleation, coagulation = tokens + for token, table, what in ( + (site, SITES, "site"), + (background, BACKGROUNDS, "background"), + (dilution, DILUTIONS, "dilution"), + (sticking, STICKING, "sticking"), + (nucleation, NUCLEATION, "nucleation"), + (coagulation, COAGULATION, "coagulation"), + ): + if token not in table: + raise ValueError(f"unknown {what} token {token!r}; known: {sorted(table)}") + + latitude, temperature, pressure, h2o = SITES[site] + aerosol, background_so2 = BACKGROUNDS[background] + payload = RunConfig().model_dump() + payload["site"].update( + latitude_deg=latitude, + temperature_k=temperature, + pressure_mbar=pressure, + h2o_ppmv=h2o, + ) + payload["background"].update(aerosol=aerosol.value, so2_pptv=background_so2) + payload["dilution"]["regime"] = DILUTIONS[dilution].value + payload["microphysics"].update( + condensation_alpha=STICKING[sticking], + nucleation_rate_scale=NUCLEATION[nucleation], + coag_kernel_scale=COAGULATION[coagulation], + n_bins=ENSEMBLE_BINS, + ) + payload["schedule"]["duration_days"] = ENSEMBLE_DAYS + return resolve(RunConfig.model_validate(payload)) + + +#: The curated Tier-B set: D1/D2/D3/burst against the clean and loaded backgrounds (ADR-009 asks for +#: 4-6). All are ``cg1`` deliberately -- ``REFERENCE_TOLERANCES.md`` records that cg0p5/cg2 may +#: straddle the tomas-jax commit that wired ``coag_kernel_scale`` through, so adopting one as a +#: golden case needs its own measurement first. +TIER_B_CASES: Final[tuple[str, ...]] = ( + "30N_20km__sabr220__D1low__a1p0__nuc1__cg1", + "30N_20km__sabr220__D2med__a1p0__nuc1__cg1", + "30N_20km__sabr220__D3high__a1p0__nuc1__cg1", + "30N_20km__sabr220__burst__a1p0__nuc1__cg1", + "30N_20km__sabr330__D2med__a1p0__nuc1__cg1", + "30N_20km__sabr330__burst__a1p0__nuc1__cg1", +) + +__all__ = [ + "BACKGROUNDS", + "COAGULATION", + "DILUTIONS", + "ENSEMBLE_BINS", + "ENSEMBLE_DAYS", + "NUCLEATION", + "SITES", + "STICKING", + "TIER_B_CASES", + "config_for_case", +] diff --git a/studio/tests/golden/plot_fidelity.py b/studio/tests/golden/plot_fidelity.py new file mode 100644 index 0000000..1e235e9 --- /dev/null +++ b/studio/tests/golden/plot_fidelity.py @@ -0,0 +1,348 @@ +"""Plots of reproduction fidelity: how closely today's code reproduces the 2026-07 archive. + +Not a performance benchmark. Each figure answers one question the tolerance table cannot: + +1. headroom -- how far every quantity sits below the tolerance that was MEASURED for it +2. timing -- WHEN the deviation peaks (the answer is the nucleation burst, not the endpoint) +3. the floor -- why near-zero species are excluded, in one glance + +Palette: the dataviz reference categorical order, validated (worst adjacent CVD dE 9.1 protan, +normal-vision 22.9). Two hues fall below 3:1 against the surface, which obliges visible labels +rather than colour alone -- so every series is direct-labelled. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +SURFACE = "#fcfcfb" +INK = "#0b0b0b" +INK_2 = "#52514e" +GRID = "#e1e0d9" +# validated categorical order, slots 1-4 +SERIES = {"SO2": "#2a78d6", "SO3": "#eb6834", "H2SO4": "#1baf7a", "OH": "#eda100"} +STATUS_BAD = "#e34948" +SEQ = "#2a78d6" + +plt.rcParams.update( + { + "font.family": "Helvetica", + "font.size": 9, + "axes.spines.top": False, + "axes.spines.right": False, + "axes.edgecolor": INK_2, + "axes.labelcolor": INK, + "text.color": INK, + "xtick.color": INK_2, + "ytick.color": INK_2, + "axes.grid": True, + "grid.color": GRID, + "grid.linewidth": 0.6, + "axes.axisbelow": True, + "figure.facecolor": SURFACE, + "axes.facecolor": SURFACE, + "savefig.facecolor": SURFACE, + } +) + +RTOL_HEADLINE, RTOL_SERIES, RTOL_SIZE = 1e-12, 1e-10, 1e-10 +SHORT = { + "30N_20km__sabr220__D1low__a1p0__nuc1__cg1": "D1 low", + "30N_20km__sabr220__D2med__a1p0__nuc1__cg1": "D2 med", + "30N_20km__sabr220__D3high__a1p0__nuc1__cg1": "D3 high", + "30N_20km__sabr220__burst__a1p0__nuc1__cg1": "burst", + "30N_20km__sabr330__D2med__a1p0__nuc1__cg1": "D2 med · sabr330", + "30N_20km__sabr330__burst__a1p0__nuc1__cg1": "burst · sabr330", +} + + +def _fig_headroom(data: dict, out: Path) -> Path: + """Worst deviation per quantity across all cases, against the tolerance measured for it. + + Dot plot on a log axis: the job is magnitude against a threshold, so one hue, and the + threshold is a line rather than a second colour. + """ + endpoint_rows, series_rows = [], [] + for entry in data.values(): + for name, devs in entry["endpoints"].items(): + for label, value in devs.items(): + endpoint_rows.append((f"{name} {label}", value)) + for name, value in entry["series"].items(): + series_rows.append((f"{name}", value)) + + def collapse(rows): + worst: dict[str, float] = {} + for name, value in rows: + worst[name] = max(worst.get(name, 0.0), value) + return sorted(worst.items(), key=lambda kv: kv[1]) + + endpoints, series = collapse(endpoint_rows), collapse(series_rows) + fig, axes = plt.subplots( + 1, 2, figsize=(11.6, 5.9), gridspec_kw={"width_ratios": [1, 1], "wspace": 0.40} + ) + for ax, rows, tol, title, sub in ( + ( + axes[0], + endpoints, + RTOL_HEADLINE, + "Endpoints — final and peak values", + "what a result is read for", + ), + ( + axes[1], + series, + RTOL_SERIES, + "Series — worst over the whole run", + "every stored sample, floored at 1e-6 x peak", + ), + ): + labels = [name for name, _ in rows] + values = np.array([max(v, 1e-17) for _, v in rows]) + y = np.arange(len(rows)) + ax.hlines(y, 1e-17, values, color=SEQ, linewidth=2, alpha=0.35) + ax.scatter(values, y, s=44, color=SEQ, zorder=3, edgecolor=SURFACE, linewidth=1.2) + ax.axvline(tol, color=STATUS_BAD, linewidth=1.6, linestyle="--", zorder=2) + ax.text( + tol * 0.85, + len(rows) - 0.05, + f"measured tolerance {tol:.0e} ", + color=STATUS_BAD, + fontsize=8.5, + va="bottom", + ha="right", + ) + for yi, value in zip(y, values, strict=True): + ax.text(value * 1.35, yi, f"{value:.1e}", va="center", fontsize=7.6, color=INK_2) + ax.set_yticks(y, labels, fontsize=8.5) + ax.set_xscale("log") + ax.set_xlim(1e-17, tol * 60) + ax.set_ylim(-0.8, len(rows) + 0.35) + ax.set_xlabel("worst relative deviation vs the archive") + ax.set_title(title, fontsize=10.5, loc="left", pad=14) + ax.text(0, 1.015, sub, transform=ax.transAxes, fontsize=8.4, color=INK_2) + fig.suptitle( + "Reproduction fidelity: every quantity sits orders of magnitude inside its tolerance", + fontsize=12, + x=0.005, + ha="left", + y=0.995, + ) + fig.text( + 0.012, + 0.015, + f"{len(data)} cases from coupled/paper_ensemble/runs/ (all 810 files written 2026-07-04), " + f"re-run today, configs rebuilt from run_ensemble.py's axis tables.\n" + f"CAVEAT: those files carry no provenance record, so the code that produced them is not " + f"known — only assumed to be the axis tables.\n" + f"Other archive directories (runs_60day, runs_bgstop*, runs_geo, …) were produced " + f"differently and are excluded from this comparison.\n" + f"Bit-for-bit reproduction is false (~31% of gas state elements differ). Tolerances were " + f"measured before they were asserted.", + fontsize=7.8, + color=INK_2, + linespacing=1.7, + ) + # explicit margins rather than tight_layout + bbox_inches="tight": the two fight over the + # multi-line caption and the axis labels lose + fig.subplots_adjust(left=0.135, right=0.985, top=0.855, bottom=0.26, wspace=0.42) + fig.savefig(out, dpi=200) + plt.close(fig) + return out + + +def _fig_timing(data: dict, out: Path) -> Path: + """Deviation against simulated time, one panel per case. Small multiples, 4 species.""" + cases = list(data) + ncols = 3 + nrows = int(np.ceil(len(cases) / ncols)) + fig, axes = plt.subplots( + nrows, ncols, figsize=(11.5, 3.1 * nrows), sharex=True, sharey=True, squeeze=False + ) + for index, case in enumerate(cases): + ax = axes[index // ncols][index % ncols] + entry = data[case] + days = np.array(entry["time_days"]) / 86400.0 + for name, colour in SERIES.items(): + dev = np.array(entry["dev_vs_time"][name], dtype=float) + ax.plot(days, dev, color=colour, linewidth=1.5, label=name, solid_capstyle="round") + worst_name = max( + entry["series"], key=lambda k: entry["series"].get(k, 0) if k in SERIES else 0 + ) + dev = np.array(entry["dev_vs_time"][worst_name], dtype=float) + peak_index = int(np.nanargmax(dev)) + ax.scatter( + [days[peak_index]], + [dev[peak_index]], + s=40, + color=SERIES[worst_name], + edgecolor=SURFACE, + linewidth=1.4, + zorder=4, + ) + late = days[peak_index] > 0.65 * float(days[-1]) + ax.annotate( + f"{worst_name} {dev[peak_index]:.1e}\nday {days[peak_index]:.2f}", + (days[peak_index], dev[peak_index]), + textcoords="offset points", + xytext=(-8 if late else 8, 6), + ha="right" if late else "left", + fontsize=7.8, + color=INK_2, + ) + ax.axhline(RTOL_SERIES, color=STATUS_BAD, linewidth=1.3, linestyle="--") + ax.set_yscale("log") + ax.set_ylim(1e-18, 1e-8) + ax.set_title(SHORT.get(case, case), fontsize=10, loc="left") + if index % ncols == 0: + ax.set_ylabel("relative deviation") + if index // ncols == nrows - 1: + ax.set_xlabel("simulated time (days)") + for spare in range(len(cases), nrows * ncols): + axes[spare // ncols][spare % ncols].axis("off") + handles = [ + plt.Line2D([], [], color=colour, linewidth=2, label=name) for name, colour in SERIES.items() + ] + handles.append( + plt.Line2D( + [], [], color=STATUS_BAD, linewidth=1.3, linestyle="--", label="series tolerance 1e-10" + ) + ) + fig.legend( + handles=handles, + loc="upper right", + ncols=5, + frameon=False, + fontsize=9, + bbox_to_anchor=(0.995, 1.0), + ) + + def _peak_day(entry: dict) -> float: + dev = np.array(entry["dev_vs_time"]["H2SO4"], dtype=float) + return float(np.array(entry["time_days"])[int(np.nanargmax(dev))] / 86400.0) + + peaks = ", ".join(f"{_peak_day(entry):.1f}" for entry in data.values()) + fig.suptitle( + "Deviation is scattered round-off, not accumulation — it does not grow with time", + fontsize=12, + x=0.005, + ha="left", + y=1.005, + ) + fig.text( + 0.005, + 0.975, + f"A baseline near 1e-14 for the whole run with occasional spikes to ~1e-12, three orders " + f"below the series tolerance. Worst-deviation days across the six cases: {peaks} — no " + f"common feature, so this is float round-off rather than a diverging integration.", + fontsize=8.4, + color=INK_2, + ) + fig.tight_layout(rect=(0, 0, 1, 0.935)) + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + return out + + +def _fig_floor(data: dict, out: Path) -> Path: + """Why near-zero species are excluded: the same comparison, floored and unfloored.""" + names, unfloored, floored, peaks = [], [], [], [] + for name in ("O1D", "O", "SO2", "H2SO4"): + values = [entry["near_zero"][name] for entry in data.values() if name in entry["near_zero"]] + if not values: + continue + names.append(name) + unfloored.append(max(v["worst_unfloored"] for v in values)) + floored.append(max(v["worst_floored"] for v in values)) + peaks.append(max(v["peak"] for v in values)) + + y = np.arange(len(names)) + fig, ax = plt.subplots(figsize=(9.6, 3.5)) + ax.hlines(y, np.maximum(floored, 1e-17), unfloored, color=GRID, linewidth=3, zorder=1) + ax.scatter( + np.maximum(floored, 1e-17), + y, + s=52, + color=SEQ, + zorder=3, + edgecolor=SURFACE, + linewidth=1.2, + label="floored — what the harness compares", + ) + ax.scatter( + unfloored, + y, + s=52, + color=STATUS_BAD, + zorder=3, + edgecolor=SURFACE, + linewidth=1.2, + label="unfloored — every sample, including noise about zero", + ) + for yi, (low, high, peak) in enumerate(zip(floored, unfloored, peaks, strict=True)): + ax.text( + max(low, 1e-17) * 0.5, + yi, + f"{max(low, 0):.0e}", + va="center", + ha="right", + fontsize=8, + color=SEQ, + ) + ax.text(high * 2.0, yi, f"{high:.0e}", va="center", fontsize=8, color=STATUS_BAD) + ax.text(1e-16, yi + 0.32, f"peak {peak:.1e} molec cm$^{{-3}}$", fontsize=7.4, color=INK_2) + ax.axvline(RTOL_SERIES, color=INK_2, linewidth=1.2, linestyle=":") + ax.text(RTOL_SERIES, len(names) - 0.45, " series tolerance 1e-10", fontsize=8, color=INK_2) + ax.set_yticks(y, names, fontsize=10) + ax.set_xscale("log") + ax.set_xlim(1e-17, 1e7) + ax.set_ylim(-0.6, len(names) - 0.25) + ax.set_xlabel("worst relative deviation") + ax.legend(loc="upper right", frameon=False, fontsize=8.6, bbox_to_anchor=(1.0, 0.92)) + ax.set_title( + "The near-zero trap: why O1D and O are excluded rather than merely toleranced", + fontsize=11.5, + loc="left", + pad=12, + ) + fig.text( + 0.005, + -0.06, + "O1D spends the night at ~1e-35 molec cm$^{-3}$ oscillating about zero — the archived " + "value is literally negative. Unfloored, its relative error reaches 1e+4 over an absolute " + "difference of 1e-34.", + fontsize=8.2, + color=INK_2, + wrap=True, + ) + fig.tight_layout() + fig.savefig(out, dpi=200, bbox_inches="tight") + plt.close(fig) + return out + + +def main() -> int: + bench = Path(sys.argv[1]) + data = json.loads((bench / "deviations.json").read_text()) + out_dir = bench / "figures" + out_dir.mkdir(exist_ok=True) + written = [ + _fig_headroom(data, out_dir / "fidelity_headroom.png"), + _fig_timing(data, out_dir / "fidelity_timing.png"), + _fig_floor(data, out_dir / "fidelity_near_zero_floor.png"), + ] + for path in written: + print(f"wrote {path} ({path.stat().st_size / 1024:.0f} kB)") + print(f"\n{len(data)} cases plotted") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/studio/tests/golden/test_tier_a_short_run.py b/studio/tests/golden/test_tier_a_short_run.py new file mode 100644 index 0000000..d58f413 --- /dev/null +++ b/studio/tests/golden/test_tier_a_short_run.py @@ -0,0 +1,183 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Tier A: one short real run against a committed fixture. + +This is the tier that catches **drift in Studio's own pipeline** -- a changed derivation, a +reordered operator split, a dependency bump that moves the solver. It compares against a fixture +this project generated (``make_fixture.py``), not against the 2026-07 archive; reproducing the +archive is Tier B's job and costs ~4.6 min per case. + +**Honest limitation, stated because it undercuts the plan's intent:** the plan calls Tier A "CI, +seconds", but CI does not check out the private submodules, so the model cannot run there. In CI +this module **skips**, and Tier A there is the pure schema/units/DAG/hash/expansion tests; locally +it runs. Fixing that means giving CI a deploy key for the submodules, which is its own change -- +tracked rather than quietly ignored. + +Cost when it does run: ~19 s for the run plus the comparison. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np +import pytest + +from studio.tests.golden.make_fixture import ( + FIXTURE_PATH, + STRIDE, + TIER_A_BINS, + TIER_A_DAYS, + reduce_state, + run_tier_a_case, + tier_a_config, +) +from studio.tests.golden.tolerances import ( + EXACT_ARRAYS, + RTOL_BIN_EDGES, + RTOL_PHOTOLYSIS, + RTOL_SERIES, + RTOL_SIZE_DISTRIBUTION, + assert_exact, + assert_headline_matches, + assert_series_matches, +) + +#: Series compared sample-by-sample at the series tolerance. +_SERIES_KEYS = ("SA", "radius_cm", "h2so4wp", "particulate_S", "total_n") + +#: Gas species compared at the headline tolerance. The sulfur chain plus its oxidant: what a result +#: is actually read for. Near-zero species are excluded by the floor in ``tolerances.py``. +_GAS_KEYS = ("SO2", "SO3", "H2SO4", "OH") + + +@pytest.fixture(scope="module") +def reference() -> dict[str, Any]: + """The committed fixture, or skip if it has not been generated.""" + if not FIXTURE_PATH.is_file(): + pytest.skip( + f"no Tier-A fixture at {FIXTURE_PATH}; generate it with " + f"`python -m studio.tests.golden.make_fixture`" + ) + with np.load(FIXTURE_PATH, allow_pickle=True) as archive: + return {key: archive[key] for key in archive.files} + + +@pytest.fixture(scope="module") +def fresh(repo_root: Path, tmp_path_factory: pytest.TempPathFactory) -> dict[str, Any]: + """Re-run the Tier-A case now and reduce it the same way. ~19 s.""" + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip( + "model submodules not checked out (`git submodule update --init`); the Tier-A golden " + "run needs them. CI does not check them out, so this skips there -- see the module " + "docstring." + ) + out_dir = tmp_path_factory.mktemp("tier_a_golden") + return reduce_state(run_tier_a_case(out_dir)) + + +@pytest.mark.tier_a +def test_the_case_is_what_the_fixture_was_built_from(reference: dict[str, Any]) -> None: + """Guard against comparing a re-run of one case against a fixture built from another. + + Cheap, and it fails clearly. Without it, changing ``TIER_A_DAYS`` would produce a shape mismatch + deep inside a series comparison instead of saying "the fixture is stale". + """ + config = tier_a_config().config + assert config.schedule.duration_days == TIER_A_DAYS + assert config.microphysics.n_bins == TIER_A_BINS + assert len(reference["dp_mid_um"]) == TIER_A_BINS + + # the reduction rule, restated: every STRIDE-th sample plus the last one + n_full = int(reference["n_times_full"]) + expected = sorted(set(range(0, n_full, STRIDE)) | {n_full - 1}) + assert list(reference["kept_indices"]) == expected + + +@pytest.mark.tier_a +def test_the_time_axis_is_bit_identical(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """``t`` is built from the terminator schedule, not integrated. Any drift is a real bug.""" + assert_exact("t", fresh["t"], reference["t"]) + + +@pytest.mark.tier_a +@pytest.mark.parametrize("key", EXACT_ARRAYS) +def test_analytic_arrays_are_bit_identical( + key: str, fresh: dict[str, Any], reference: dict[str, Any] +) -> None: + """``t``, ``V_ratio`` and ``T``: analytic, so exact equality is the right assertion.""" + assert_exact(key, fresh[key], reference[key]) + + +@pytest.mark.tier_a +@pytest.mark.parametrize("species", _GAS_KEYS) +def test_gas_species_reproduce( + species: str, fresh: dict[str, Any], reference: dict[str, Any] +) -> None: + names = [str(name) for name in reference["species"]] + index = names.index(species) # BY NAME, never by position + # Endpoints at the headline tolerance, the series at its own looser one -- two different rows of + # REFERENCE_TOLERANCES.md. Conflating them is what broke Tier B's first real run. + assert_headline_matches(species, fresh["x"][:, index], reference["x"][:, index]) + assert_series_matches( + f"{species} (series)", fresh["x"][:, index], reference["x"][:, index], RTOL_SERIES + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize("key", _SERIES_KEYS) +def test_aerosol_series_reproduce( + key: str, fresh: dict[str, Any], reference: dict[str, Any] +) -> None: + assert_series_matches(key, fresh[key], reference[key], RTOL_SERIES) + + +@pytest.mark.tier_a +def test_the_size_distribution_reproduces(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """Per bin, over the whole stored series -- not just the final spectrum.""" + assert_series_matches( + "dNdlogDp", fresh["dNdlogDp"], reference["dNdlogDp"], RTOL_SIZE_DISTRIBUTION + ) + assert_series_matches("n_cm3", fresh["n_cm3"], reference["n_cm3"], RTOL_SIZE_DISTRIBUTION) + + +@pytest.mark.tier_a +def test_photolysis_reproduces(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """J is the most reproducible part of the pipeline (measured 1.09e-13), so drift here is signal. + + Compared per reaction, not summed: a compensating pair of errors across two reactions would + survive a total and is exactly the kind of thing this tier exists to catch. + """ + assert_exact("J_tmid", fresh["J_tmid"], reference["J_tmid"]) + equations = [str(name) for name in reference["J_equations"]] + for index, equation in enumerate(equations): + assert_series_matches( + f"J[{equation}]", fresh["J"][:, index], reference["J"][:, index], RTOL_PHOTOLYSIS + ) + + +@pytest.mark.tier_a +def test_the_dry_bin_edges_reproduce(fresh: dict[str, Any], reference: dict[str, Any]) -> None: + """Not exact: Studio and the archive spell the geometric mean differently (see tolerances.py). + + Within Studio's own pipeline they should agree bit-for-bit, so this passing at 1e-15 rather than + exactly would itself be information -- the tolerance is the one measured against the archive. + """ + assert_series_matches("dp_mid_um", fresh["dp_mid_um"], reference["dp_mid_um"], RTOL_BIN_EDGES) + + +@pytest.mark.tier_a +def test_the_run_is_still_physically_recognisable(fresh: dict[str, Any]) -> None: + """A sanity floor under the tolerances: a comparison can only be meaningful if the run happened. + + All four assertions would hold for any correct run of this case, and none of them would hold for + a run that silently did nothing -- which is the failure a tolerance-based test cannot see. + """ + names = [str(name) for name in fresh["species"]] + so2 = fresh["x"][:, names.index("SO2")] + h2so4 = fresh["x"][:, names.index("H2SO4")] + assert so2[-1] < so2[0], "SO2 must be consumed" + assert h2so4.max() > 0.0, "H2SO4 must be produced" + assert fresh["total_n"].max() > 0.0, "particles must form" + assert fresh["V_ratio"][-1] > 1.0, "the plume must expand" diff --git a/studio/tests/golden/test_tier_b_archive.py b/studio/tests/golden/test_tier_b_archive.py new file mode 100644 index 0000000..045e5f0 --- /dev/null +++ b/studio/tests/golden/test_tier_b_archive.py @@ -0,0 +1,199 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Tier B: reproduce the archived 810-run ensemble, at the measured tolerance. + +**Nightly or manual, never in CI.** Six 10-day cases at ~4.6 min each is ~28 minutes, and the +archive it compares against is gitignored, so a fresh clone does not have it. Run it explicitly: + + pytest studio/tests/golden -m tier_b + +What it asserts is **reproduction of results produced in July 2026 by a different toolchain**, +which is a different claim from Tier A's "Studio has not drifted since its own fixture". +Bit-for-bit is already known to be false -- ~31 % of gas state elements differ -- so every tolerance +here comes from ``REFERENCE_TOLERANCES.md``, measured before it was asserted. + +**A failure here is not automatically a regression.** A JAX or diffrax bump moves these numbers; the +correct response is to re-run ``measure_deviation.py``, update the record with the new SHAs, and +decide whether the new deviation is acceptable -- not to widen a constant until the test passes. The +assertion messages say so at the point of failure, where the temptation is. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import numpy as np +import pytest + +from studio.tests.golden.paper_cases import ( + ENSEMBLE_BINS, + ENSEMBLE_DAYS, + TIER_B_CASES, + config_for_case, +) +from studio.tests.golden.tolerances import ( + EXACT_ARRAYS, + RTOL_HEADLINE, + RTOL_SERIES, + RTOL_SIZE_DISTRIBUTION, + assert_exact, + endpoint_deviations, + worst_relative_deviation, +) + +#: Gas species compared at the headline tolerance, by name. +_GAS_KEYS = ("SO2", "SO3", "H2SO4", "OH") + +#: Aerosol series compared at the series tolerance. +_SERIES_KEYS = ("SA", "radius_cm", "h2so4wp", "particulate_S", "total_n") + + +@pytest.fixture(scope="module") +def archive_root(repo_root: Path) -> Path: + """The archived ensemble, or skip. Read-only: this is irreplaceable reference data.""" + runs = repo_root / "coupled" / "paper_ensemble" / "runs" + if not runs.is_dir(): + pytest.skip( + f"archived ensemble not present at {runs} (gitignored and regenerable); Tier B " + f"reproduces it and cannot run without it" + ) + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + return runs + + +@pytest.fixture(scope="module") +def reproduced( + archive_root: Path, tmp_path_factory: pytest.TempPathFactory +) -> dict[str, dict[str, Any]]: + """Re-run every curated case once, and pair each with its archived counterpart. + + Module-scoped because each case costs ~4.6 min: running them once and sharing the arrays across + assertions is the difference between 28 minutes and several hours. + """ + from studio.modelio.execute import run_and_write + + out_root = tmp_path_factory.mktemp("tier_b_golden") + paired: dict[str, dict[str, Any]] = {} + for case_id in TIER_B_CASES: + archived_path = archive_root / case_id / "state.npz" + if not archived_path.is_file(): + continue # a case absent from this machine's archive is a skip, not a failure + fresh_path = run_and_write(config_for_case(case_id), out_root / case_id)["state"] + with ( + np.load(fresh_path, allow_pickle=True) as fresh, + np.load(archived_path, allow_pickle=True) as archived, + ): + paired[case_id] = { + "fresh": {key: fresh[key] for key in fresh.files}, + "archived": {key: archived[key] for key in archived.files}, + } + if not paired: + pytest.skip(f"none of {list(TIER_B_CASES)} is present in {archive_root}") + return paired + + +@pytest.mark.tier_b +def test_the_curated_set_covers_the_dilution_regimes_and_both_backgrounds() -> None: + """ADR-009 asks for 4-6 cases across D1/D2/D3/burst x sabr220/sabr330. Assert the set, cheaply. + + Pure — it needs neither the model nor the archive, so a mis-curated set is caught in + milliseconds rather than 28 minutes in. + """ + assert 4 <= len(TIER_B_CASES) <= 6 + regimes = {config_for_case(case).config.dilution.regime.value for case in TIER_B_CASES} + backgrounds = {config_for_case(case).config.background.aerosol.value for case in TIER_B_CASES} + assert {"D1", "D2", "D3", "burst"} <= regimes + assert {"sabr_220", "sabr_330"} <= backgrounds + for case in TIER_B_CASES: + config = config_for_case(case).config + assert config.microphysics.n_bins == ENSEMBLE_BINS + assert config.schedule.duration_days == ENSEMBLE_DAYS + assert config.microphysics.coag_kernel_scale == 1.0, ( + "the curated set stays on cg1 until the coag_kernel_scale coverage gap recorded in " + "REFERENCE_TOLERANCES.md is measured" + ) + + +@pytest.mark.tier_b +def test_every_case_reproduces(reproduced: dict[str, dict[str, Any]]) -> None: + """One test over all cases, reporting every deviation before failing. + + Deliberately not parametrised per case: after 28 minutes of compute, "SO2 failed in case 3" is + much less useful than the whole table. A per-case failure would also hide whether the deviation + is systematic or specific to one regime, which is the first thing to want to know. + """ + failures: list[str] = [] + #: every deviation, not only the ones that exceed: 28 minutes of compute should produce a + #: measurement, not just a verdict. Printed below so a passing run still reports numbers. + observed: dict[str, tuple[float, float]] = {} + for case_id, pair in sorted(reproduced.items()): + fresh, archived = pair["fresh"], pair["archived"] + species = [str(name) for name in archived["species"]] + + for key in EXACT_ARRAYS: + try: + assert_exact(f"{case_id}/{key}", fresh[key], archived[key]) + except AssertionError as exc: + failures.append(str(exc)) + + # Two tolerances, from two different rows of the record: the endpoints a result is read + # for (1e-12) and the series they come from (1e-10). Applying the endpoint number to a + # whole series is the mistake this harness made on its first real run. + for name in _GAS_KEYS: + index = species.index(name) # BY NAME + series_fresh, series_archived = fresh["x"][:, index], archived["x"][:, index] + for label, deviation in endpoint_deviations(series_fresh, series_archived).items(): + observed[f"{case_id}/{name} ({label})"] = (deviation, RTOL_HEADLINE) + if deviation > RTOL_HEADLINE: + failures.append( + f"{case_id}/{name} {label}: {deviation:.3e} > {RTOL_HEADLINE:.0e}" + ) + worst = worst_relative_deviation(series_fresh, series_archived) + observed[f"{case_id}/{name} (series)"] = (worst, RTOL_SERIES) + if worst > RTOL_SERIES: + failures.append(f"{case_id}/{name} series: {worst:.3e} > {RTOL_SERIES:.0e}") + + for key in _SERIES_KEYS: + for label, deviation in endpoint_deviations(fresh[key], archived[key]).items(): + observed[f"{case_id}/{key} ({label})"] = (deviation, RTOL_HEADLINE) + if deviation > RTOL_HEADLINE: + failures.append( + f"{case_id}/{key} {label}: {deviation:.3e} > {RTOL_HEADLINE:.0e}" + ) + worst = worst_relative_deviation(fresh[key], archived[key]) + observed[f"{case_id}/{key} (series)"] = (worst, RTOL_SERIES) + if worst > RTOL_SERIES: + failures.append(f"{case_id}/{key} series: {worst:.3e} > {RTOL_SERIES:.0e}") + + worst = worst_relative_deviation(fresh["dNdlogDp"], archived["dNdlogDp"]) + observed[f"{case_id}/dNdlogDp (per bin)"] = (worst, RTOL_SIZE_DISTRIBUTION) + if worst > RTOL_SIZE_DISTRIBUTION: + failures.append(f"{case_id}/dNdlogDp: {worst:.3e} > {RTOL_SIZE_DISTRIBUTION:.0e}") + + print(f"\n{'quantity':52s} {'worst rel':>11s} tolerance") + for label, (deviation, tolerance) in sorted(observed.items()): + print(f"{label:52s} {deviation:11.3e} {tolerance:.0e}") + + assert not failures, ( + "Tier-B reproduction deviates beyond the MEASURED tolerances:\n " + + "\n ".join(failures) + + "\n\nThese tolerances were measured, not chosen (see REFERENCE_TOLERANCES.md in this " + "directory). A toolchain bump moves them: re-run measure_deviation.py, update that record " + "with the new SHAs, and decide whether the new deviation is acceptable. Do not widen the " + "constants to make this pass." + ) + + +@pytest.mark.tier_b +def test_the_runs_are_physically_recognisable(reproduced: dict[str, dict[str, Any]]) -> None: + """The floor under the tolerances: a tolerance test cannot tell that a run did nothing.""" + for case_id, pair in sorted(reproduced.items()): + fresh = pair["fresh"] + species = [str(name) for name in fresh["species"]] + so2 = fresh["x"][:, species.index("SO2")] + assert so2[-1] < so2[0], f"{case_id}: SO2 must be consumed" + assert fresh["x"][:, species.index("H2SO4")].max() > 0.0, f"{case_id}: H2SO4 must form" + assert fresh["total_n"].max() > 0.0, f"{case_id}: particles must form" + assert fresh["V_ratio"][-1] > 1.0, f"{case_id}: the plume must expand" diff --git a/studio/tests/golden/tolerances.py b/studio/tests/golden/tolerances.py new file mode 100644 index 0000000..e70f260 --- /dev/null +++ b/studio/tests/golden/tolerances.py @@ -0,0 +1,173 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The measured tolerances, as code, and the comparison that applies them. + +**Every number here was measured before it was asserted** and is traceable to +``REFERENCE_TOLERANCES.md`` in this directory, which records the SHAs, the environment and the +per-quantity deviations it came from (task 0.7, issue #70). None of them is a value somebody chose +because a test failed. + +The rule that follows: **if an assertion here starts failing, re-run the measurement and update the +record — do not widen the number.** A tolerance widened to make a test pass is a test that no longer +tests anything, and this file exists to make that a visible edit rather than a quiet one. + +Two subtleties the measurement turned up, both encoded below rather than left to whoever writes the +next assertion: + +* **Near-zero series must be floored.** Unguarded relative error reaches 4.24e+04 on night-time + ``O1D``/``O`` at O(1e-35) molec cm^-3 oscillating about zero -- the archived value is literally + negative on a species that peaks around 3 molec cm^-3. Comparing only samples above + ``1e-6 x the series' own peak`` is what makes the comparison mean anything. +* **``dp_mid_um`` is not exact.** The archive writes ``10**(0.5*(log10 a + log10 b))``; Studio + writes ``sqrt(a*b)`` (task 0.5, deliberately). Algebraically identical, ~8e-16 apart in float64. +""" + +from __future__ import annotations + +from typing import Final + +import numpy as np +import numpy.typing as npt + +#: Final and peak values of the quantities a result is actually read for. Measured <= 2.9e-14 across +#: both reference cases; 1e-12 is ~35x headroom and still ~6 orders below any meaningful physics +#: change, so a real regression cannot hide under it. +RTOL_HEADLINE: Final = 1e-12 + +#: Any stored series, compared sample by sample. Measured worst 3.4e-12 (H2SO4, day 1.34); 1e-10 +#: covers the worst trace species (Cl2, 1.1e-11) with ~10x margin. +RTOL_SERIES: Final = 1e-10 + +#: Per-bin size distribution. Per-bin conditioning is worse than integrated number, so it gets its +#: own looser number; measured worst bin 2.0e-12. +RTOL_SIZE_DISTRIBUTION: Final = 1e-10 + +#: Photolysis J. Measured 1.09e-13 and identical in both reference cases -- TUV-x is the most +#: reproducible part of the pipeline. +RTOL_PHOTOLYSIS: Final = 1e-11 + +#: Dry bin mid-points. NOT exact: see the module docstring. A few ULP, measured 8.1e-16. +RTOL_BIN_EDGES: Final = 1e-15 + +#: Arrays measured bit-identical, because they are analytic rather than integrated. Any difference +#: at all is a real bug, so these are compared exactly and deliberately have no tolerance. +EXACT_ARRAYS: Final = ("t", "V_ratio", "T") + +#: A sample is compared only if it exceeds this fraction of its own series' peak. Below it, relative +#: error is meaningless (see the module docstring); an absolute-floor assertion would be the way to +#: cover those, and is not attempted here rather than being faked. +NEAR_ZERO_FRACTION_OF_PEAK: Final = 1e-6 + +#: Species excluded outright: they spend most of the run at O(1e-35) and oscillate about zero, so +#: even the floor above leaves too few comparable samples to mean anything. +EXCLUDED_SPECIES: Final = ("O1D", "O") + +FloatArray = npt.NDArray[np.float64] + + +def worst_relative_deviation( + fresh: npt.ArrayLike, reference: npt.ArrayLike, *, floor_by_peak: bool = True +) -> float: + """Worst relative deviation between two series, ignoring samples below the near-zero floor. + + Returns 0.0 when nothing is comparable, rather than NaN: a series entirely below its own floor + carries no information either way, and propagating NaN into an assertion would fail for the + wrong reason. + """ + a = np.asarray(reference, dtype=np.float64) + b = np.asarray(fresh, dtype=np.float64) + if a.shape != b.shape: + raise ValueError(f"shape mismatch: reference {a.shape} vs fresh {b.shape}") + magnitude = np.abs(a) + if floor_by_peak: + peak = float(np.nanmax(magnitude)) if magnitude.size else 0.0 + comparable = magnitude > peak * NEAR_ZERO_FRACTION_OF_PEAK + else: + comparable = magnitude > 0.0 + comparable &= np.isfinite(a) & np.isfinite(b) + if not np.any(comparable): + return 0.0 + return float(np.max(np.abs(b[comparable] - a[comparable]) / magnitude[comparable])) + + +def assert_series_matches( + name: str, fresh: npt.ArrayLike, reference: npt.ArrayLike, rtol: float +) -> None: + """Compare a series, or raise with the measured number and where to look. + + The message names the tolerance's provenance on purpose: the first instinct on a failure here is + to widen the number, and the right response is to re-measure. + """ + worst = worst_relative_deviation(fresh, reference) + if worst > rtol: + raise AssertionError( + f"{name}: worst relative deviation {worst:.3e} exceeds {rtol:.0e}.\n" + f"This tolerance was MEASURED (see REFERENCE_TOLERANCES.md in this directory), not " + f"chosen. Re-run the measurement and update that record with the new SHAs -- do not " + f"widen this number to make the test pass." + ) + + +def endpoint_deviations(fresh: npt.ArrayLike, reference: npt.ArrayLike) -> dict[str, float]: + """Relative deviation of the FINAL and PEAK values of a series. + + Separate from :func:`worst_relative_deviation` because the record gives these two a tighter + tolerance than the series they come from -- ``1e-12`` against ``1e-10``. Conflating them is an + easy mistake with a misleading symptom: it looks like a reproduction failure when it is a test + reading the wrong row. + """ + a = np.asarray(reference, dtype=np.float64) + b = np.asarray(fresh, dtype=np.float64) + if a.shape != b.shape: + raise ValueError(f"shape mismatch: reference {a.shape} vs fresh {b.shape}") + out: dict[str, float] = {} + for label, reference_value, fresh_value in ( + ("final", float(a[-1]), float(b[-1])), + ("peak", float(np.nanmax(a)), float(np.nanmax(b))), + ): + if reference_value == 0.0: + continue # a zero endpoint has no relative deviation; the series check still covers it + out[label] = abs(fresh_value - reference_value) / abs(reference_value) + return out + + +def assert_headline_matches(name: str, fresh: npt.ArrayLike, reference: npt.ArrayLike) -> None: + """Final and peak at ``RTOL_HEADLINE``. What a result is actually read for.""" + for label, deviation in endpoint_deviations(fresh, reference).items(): + if deviation > RTOL_HEADLINE: + raise AssertionError( + f"{name} ({label}): relative deviation {deviation:.3e} exceeds " + f"{RTOL_HEADLINE:.0e}. This is the ENDPOINT tolerance; the series that produced it " + f"has its own, looser one ({RTOL_SERIES:.0e}). Both were measured -- see " + f"REFERENCE_TOLERANCES.md in this directory." + ) + + +def assert_exact(name: str, fresh: npt.ArrayLike, reference: npt.ArrayLike) -> None: + """Bit-for-bit, for the analytic arrays. Any difference is a real bug.""" + a = np.asarray(reference) + b = np.asarray(fresh) + if not np.array_equal(a, b): + differing = int(np.sum(a != b)) if a.shape == b.shape else -1 + raise AssertionError( + f"{name} is expected to be bit-identical (analytic, not integrated) but " + f"{differing} element(s) differ. Worst relative deviation " + f"{worst_relative_deviation(b, a, floor_by_peak=False):.3e}." + ) + + +__all__ = [ + "EXACT_ARRAYS", + "EXCLUDED_SPECIES", + "NEAR_ZERO_FRACTION_OF_PEAK", + "RTOL_BIN_EDGES", + "RTOL_HEADLINE", + "RTOL_PHOTOLYSIS", + "RTOL_SERIES", + "RTOL_SIZE_DISTRIBUTION", + "assert_exact", + "assert_headline_matches", + "assert_series_matches", + "endpoint_deviations", + "worst_relative_deviation", +] diff --git a/studio/tests/unit/test_api.py b/studio/tests/unit/test_api.py new file mode 100644 index 0000000..08d4eff --- /dev/null +++ b/studio/tests/unit/test_api.py @@ -0,0 +1,252 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The API, and the page it serves. + +Most of this needs neither the model nor a run: the endpoints that matter for the form -- schema, +resolve, list, fetch -- are cheap by design, because the UI calls ``/api/config/resolve`` on every +keystroke and a JAX import on that path would be unaffordable. + +**The SSE test runs a real uvicorn server in a thread.** ``TestClient`` serialises requests, so a +stream opened against it does not observe a state change made concurrently -- when I first checked +progress under ``TestClient`` it reported only the terminal state, which looked exactly like a +broken stream and was not one. A claim as central as "progress is pushed" (spec 7.3) has to be +tested against something that can actually push. +""" + +from __future__ import annotations + +import json +import socket +import threading +import time +from pathlib import Path +from typing import Any + +import pytest +from fastapi.testclient import TestClient + +from studio.api.app import create_app +from studio.resolve import resolve +from studio.schema import RunConfig +from studio.store import create_run, create_run_set, record_job, record_transition, session_scope + + +@pytest.fixture +def api(tmp_path: Path) -> Any: + """An app with its own home and database, built by the factory rather than monkeypatched.""" + app = create_app(home=tmp_path / "home", database=f"sqlite:///{tmp_path / 'studio.db'}") + with TestClient(app) as client: + yield client, app + + +@pytest.mark.tier_a +def test_the_page_is_served(api: Any) -> None: + client, _ = api + response = client.get("/") + assert response.status_code == 200 + assert "Plume Studio" in response.text + assert "/api/config/resolve" in response.text, "the page must talk to the real resolver" + + +@pytest.mark.tier_a +def test_the_schema_endpoint_is_the_schema(api: Any) -> None: + """Nothing in the UI may invent a field that does not exist here (ADR-002).""" + from studio.schema import SCHEMA_VERSION + + client, _ = api + schema = client.get("/api/schema").json() + assert schema["x-studio-schema-version"] == SCHEMA_VERSION + assert "Site" in schema["$defs"] + assert schema["$defs"]["Site"]["properties"]["temperature_k"]["x-studio"]["unit"] == "K" + + +@pytest.mark.tier_a +def test_resolve_returns_the_derived_values_and_the_identity(api: Any) -> None: + """The form shows what the server derived, never what the browser guessed.""" + client, _ = api + response = client.post( + "/api/config/resolve", json={"config": {"schedule": {"duration_days": 1}}} + ) + assert response.status_code == 200 + body = response.json() + expected = resolve(RunConfig.model_validate({"schedule": {"duration_days": 1}})) + assert body["config_hash"] == expected.config.config_hash() + assert body["derived"]["plume_volume_cm3"] == 1.5e12 + assert body["derived"]["so2_initial_pptv"] == pytest.approx(3.309115922996412e9, rel=1e-15) + assert body["stale_fields"] == [] + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + "config", + [ + {"site": {"temperature_k": -5}}, + {"microphysics": {"n_bins": 100}}, + {"switches": {"heating_to_t": True}}, + {"site": {"temprature_k": 210}}, + ], +) +def test_an_invalid_config_is_422_not_a_500(api: Any, config: dict[str, Any]) -> None: + """Validation failures are the schema working, so they are reported as client errors. + + The heating case is the interesting one: it is refused because the model cannot represent the + physics (SCIENCE-4), and that refusal has to reach the browser rather than crashing the server. + """ + client, _ = api + assert client.post("/api/config/resolve", json={"config": config}).status_code == 422 + assert client.post("/api/runs", json={"config": config}).status_code == 422 + + +@pytest.mark.tier_a +def test_unknown_things_are_404(api: Any) -> None: + client, _ = api + assert client.get("/api/runs/nope").status_code == 404 + assert client.get("/api/runs/nope/summary").status_code == 404 + assert client.get("/api/runs/nope/artifacts/state").status_code == 404 + + +@pytest.mark.tier_a +def test_runs_start_empty_and_list_what_exists(api: Any, tmp_path: Path) -> None: + client, app = api + assert client.get("/api/runs").json() == [] + + with session_scope(app.state.factory) as session: + run_set = create_run_set(session, label="direct") + run = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + label="written directly", + provenance={"sandbox": {"dirty": False}, "submodules": {}}, + ) + record_job(session, run=run, state="queued") + + listed = client.get("/api/runs").json() + assert len(listed) == 1 + assert listed[0]["label"] == "written directly" + assert listed[0]["state"] == "queued" + assert listed[0]["reproducible"] is True, "a JSON boolean, not 0/1" + + +@pytest.mark.tier_a +def test_the_run_detail_carries_provenance_and_transitions(api: Any) -> None: + """What the page needs to say honestly what produced a result (ADR-006).""" + client, app = api + with session_scope(app.state.factory) as session: + run_set = create_run_set(session) + run = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + provenance={"sandbox": {"commit": "a" * 40, "dirty": True}, "submodules": {}}, + ) + job = record_job(session, run=run, state="queued") + record_transition(session, job=job, state="running", detail="launched") + run_id = run.id + + detail = client.get(f"/api/runs/{run_id}").json() + assert detail["reproducible"] is False + assert detail["provenance"]["sandbox"]["commit"] == "a" * 40 + assert [t["state"] for t in detail["transitions"]] == ["queued", "running"] + + +def _free_port() -> int: + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return int(sock.getsockname()[1]) + + +@pytest.mark.tier_a +def test_progress_is_pushed_as_it_happens(tmp_path: Path) -> None: + """A real server, a real SSE client, and a state change made while the stream is open. + + This is the test that would have caught a stream reporting only the terminal state. It uses no + model: the stream reads job state from the DATABASE, so writing a transition from the test is + exactly what a running job does -- and is also why the stream works for runs submitted by the + CLI or by a previous process. + """ + import httpx2 as httpx + import uvicorn + + database = f"sqlite:///{tmp_path / 'studio.db'}" + app = create_app(home=tmp_path / "home", database=database) + with session_scope(app.state.factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig()), label="streamed") + job = record_job(session, run=run, state="queued") + run_id, job_id = run.id, job.id + + port = _free_port() + server = uvicorn.Server( + uvicorn.Config(app, host="127.0.0.1", port=port, log_level="error", lifespan="off") + ) + thread = threading.Thread(target=server.run, daemon=True) + thread.start() + deadline = time.monotonic() + 15 + while not server.started and time.monotonic() < deadline: + time.sleep(0.05) + assert server.started, "uvicorn did not start" + + def advance() -> None: + """Move the job on while the stream is open, as a running job would.""" + from studio.store.models import JobRow + + for state in ("running", "succeeded"): + time.sleep(0.6) + with session_scope(app.state.factory) as session: + record_transition(session, job=session.get(JobRow, job_id), state=state) + + try: + threading.Thread(target=advance, daemon=True).start() + seen: list[str] = [] + with httpx.Client(timeout=20.0) as client: + with client.stream( + "GET", f"http://127.0.0.1:{port}/api/events/runs/{run_id}" + ) as stream: + for line in stream.iter_lines(): + if line.startswith("data:"): + seen.append(json.loads(line[5:])["state"]) + if seen and seen[-1] == "succeeded": + break + finally: + server.should_exit = True + thread.join(timeout=10) + + assert seen[0] == "queued", "the stream must report the state it finds, not only changes" + assert "running" in seen, "an intermediate state must arrive while the run is in flight" + assert seen[-1] == "succeeded" + + +@pytest.mark.tier_a +def test_the_stream_reports_an_unknown_run_rather_than_hanging(api: Any) -> None: + """A client asking about a run that does not exist gets an error event and a closed stream.""" + client, _ = api + with client.stream("GET", "/api/events/runs/nope") as stream: + events = [line for line in stream.iter_lines() if line.startswith(("event:", "data:"))] + assert any("error" in line for line in events) + + +@pytest.mark.tier_a +def test_a_submitted_run_appears_immediately_with_202(api: Any, repo_root: Path) -> None: + """Submission returns a handle, not a finished run: 202 means accepted, not complete. + + Needs the model, because provenance pins the submodules before anything starts (ADR-006). + """ + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + + client, _ = api + response = client.post( + "/api/runs", + json={ + "config": {"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}, + "label": "accepted", + }, + ) + assert response.status_code == 202 + body = response.json() + assert body["state"] == "queued" + assert client.get(f"/api/runs/{body['run_id']}").json()["label"] == "accepted" + assert ( + client.get(f"/api/runs/{body['run_id']}").json()["config_hash"] == body["config_hash"] + ), "the identity in the response is the one that was stored" diff --git a/studio/tests/unit/test_cli.py b/studio/tests/unit/test_cli.py new file mode 100644 index 0000000..5d71281 --- /dev/null +++ b/studio/tests/unit/test_cli.py @@ -0,0 +1,203 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The CLI: the scripted path to everything the UI can do (ADR-002). + +Most of these need neither the model nor a long run -- ``--plan`` and ``--dry-run`` exist precisely +so that deciding to spend compute is a separate act from spending it, and that makes them cheap to +test. The one end-to-end test runs the real model for one simulated day and skips without the +submodules, like every other model-touching test. + +No test hooks: the CLI is exercised through its real arguments, and the seams it uses +(``--database``, ``--out``) are ones a real deployment uses too. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import yaml +from typer.testing import CliRunner + +from studio.cli.main import app +from studio.schema import RunConfig + +runner = CliRunner() + + +@pytest.fixture +def config_file(tmp_path: Path) -> Path: + path = tmp_path / "config.yaml" + path.write_text( + yaml.safe_dump({"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}), + encoding="utf-8", + ) + return path + + +@pytest.fixture +def sweep_file(tmp_path: Path) -> Path: + path = tmp_path / "sweep.yaml" + path.write_text( + yaml.safe_dump( + { + "base": {"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}, + "axes": [ + { + "name": "nucleation", + "kind": "grid", + "points": [ + { + "label": "nuc1", + "assignments": {"microphysics.nucleation_rate_scale": 1.0}, + }, + { + "label": "nuc100", + "assignments": {"microphysics.nucleation_rate_scale": 100.0}, + }, + ], + } + ], + } + ), + encoding="utf-8", + ) + return path + + +@pytest.mark.tier_a +def test_a_dry_run_prints_the_identity_and_touches_nothing( + config_file: Path, tmp_path: Path +) -> None: + """Identity before compute: the hash is knowable without running anything.""" + result = runner.invoke( + app, ["run", str(config_file), "--dry-run", "--out", str(tmp_path / "o")] + ) + assert result.exit_code == 0, result.output + assert "config hash" in result.output + assert "nothing written, nothing submitted" in result.output + assert not (tmp_path / "o").exists(), "--dry-run must not create the output tree" + + +@pytest.mark.tier_a +def test_the_dry_run_hash_is_the_schema_hash(config_file: Path) -> None: + """The CLI must not have its own idea of identity.""" + from studio.resolve import resolve + + expected = resolve( + RunConfig.model_validate({"schedule": {"duration_days": 1}, "microphysics": {"n_bins": 40}}) + ).config.config_hash() + result = runner.invoke(app, ["run", str(config_file), "--dry-run"]) + assert expected in result.output + + +@pytest.mark.tier_a +def test_plan_lists_the_runs_and_submits_nothing(sweep_file: Path, tmp_path: Path) -> None: + """Mirrors ``run_ensemble.py``'s ``plan`` verb: deciding to spend compute is its own command.""" + result = runner.invoke(app, ["sweep", str(sweep_file), "--plan", "--out", str(tmp_path / "o")]) + assert result.exit_code == 0, result.output + assert "2 run(s) from 1 axis/axes" in result.output + assert "nuc1" in result.output and "nuc100" in result.output + assert "nothing submitted" in result.output + assert not (tmp_path / "o").exists() + + +@pytest.mark.tier_a +def test_planned_runs_have_distinct_identities(sweep_file: Path) -> None: + """A sweep whose points collided would silently run the same case twice.""" + result = runner.invoke(app, ["sweep", str(sweep_file), "--plan"]) + hashes = [line.split()[-1] for line in result.output.splitlines() if line.startswith(" [")] + assert len(hashes) == len(set(hashes)) == 2 + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("contents", "message"), + [ + ("not: [valid", "not valid"), + ("- a\n- b\n", "must contain a mapping"), + ("site:\n temperature_k: -5\n", "not a valid configuration"), + ], +) +def test_a_bad_config_file_exits_two_with_a_message( + tmp_path: Path, contents: str, message: str +) -> None: + """Exit 2 is "we never started", distinct from a model failure (exit 1).""" + path = tmp_path / "bad.yaml" + path.write_text(contents, encoding="utf-8") + result = runner.invoke(app, ["run", str(path), "--dry-run"]) + assert result.exit_code == 2 + assert message in result.output + + +@pytest.mark.tier_a +def test_a_missing_file_names_itself(tmp_path: Path) -> None: + result = runner.invoke(app, ["run", str(tmp_path / "absent.yaml"), "--dry-run"]) + assert result.exit_code == 2 + assert "no such file" in result.output + + +@pytest.mark.tier_a +def test_status_of_an_unknown_run_exits_two(tmp_path: Path) -> None: + """Distinguishes "no such run" from "a run with nothing to report".""" + from studio.store import upgrade_to_head + + url = f"sqlite:///{tmp_path / 'studio.db'}" + upgrade_to_head(url) + result = runner.invoke(app, ["status", "nope", "--database", url]) + assert result.exit_code == 2 + assert "no run" in result.output + + +@pytest.mark.tier_a +def test_a_run_is_persisted_and_readable_from_a_new_process( + config_file: Path, tmp_path: Path, repo_root: Path +) -> None: + """The vertical slice's CLI half, end to end, with the real model. ~20 s. + + ``status`` runs against a session opened after the run finished -- the point being that job + state lives in the database rather than in the submitting process's memory (ADR-007), so a + crashed or exited CLI leaves a readable record rather than an orphan. + """ + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + + url = f"sqlite:///{tmp_path / 'studio.db'}" + out = tmp_path / "runs" + result = runner.invoke( + app, ["run", str(config_file), "--out", str(out), "--database", url, "--label", "slice"] + ) + assert result.exit_code == 0, result.output + assert "succeeded" in result.output + + from sqlalchemy import select + + from studio.store import create_db_engine, session_factory, session_scope + from studio.store.models import RunRow + + factory = session_factory(create_db_engine(url)) + with session_scope(factory) as session: + run = session.scalars(select(RunRow)).one() + run_id = run.id + assert run.label == "slice" + assert run.provenance is not None, "provenance is recorded at submit (ADR-006)" + assert set(run.provenance["submodules"]), "the model is pinned, not just the app" + kinds = {artifact.kind for artifact in run.artifacts} + assert {"input", "provenance", "state", "summary", "stdout"} <= kinds + (job,) = run.jobs + assert job.state == "succeeded" + assert [t.state for t in job.transitions] == [ + "queued", + "running", + "succeeded", + ], "the persisted trail must be as complete as the runner's, or persisting it is pointless" + + status = runner.invoke(app, ["status", run_id, "--database", url]) + assert status.exit_code == 0, status.output + assert run_id in status.output + assert "succeeded" in status.output + assert "state" in status.output and "summary" in status.output + + summary_row = json.loads((out / "artifacts" / run_id / "summary.json").read_text()) + assert summary_row["config_hash"] == run.config_hash, "the summary carries the same identity" diff --git a/studio/tests/unit/test_config_hash.py b/studio/tests/unit/test_config_hash.py new file mode 100644 index 0000000..1d14f2b --- /dev/null +++ b/studio/tests/unit/test_config_hash.py @@ -0,0 +1,160 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The config hash must not drift. Ever, silently. + +``config_hash`` is run identity, the cache key and the golden-fixture key at once (ADR-006). If it +changes for a reason nobody noticed, every cached result stops matching and every fixture starts +missing -- and the symptom is "everything recomputes", which reads as a performance problem rather +than a correctness one. + +So the pinned hash below is a deliberate tripwire. If a change to the schema alters it, that is +correct and expected: bump ``SCHEMA_VERSION`` and update the constant IN THE SAME COMMIT, with the +reason in the message. What must never happen is the value changing without anyone deciding it +should. +""" + +from __future__ import annotations + +import json +import math +import subprocess +import sys +import textwrap + +import pytest + +from studio.schema import RunConfig, canonical_json, canonical_payload, config_hash, short_hash +from studio.schema.hashing import CANONICAL_FORM_VERSION + +#: SHA-256 of the canonical JSON of ``RunConfig()`` -- the paper ensemble's golden case, which is +#: also the schema's default configuration. Tied to SCHEMA_VERSION 0.2.0 and canonical form 1. +#: Moved from ...46cbe3 when 0.2.0 refused the temperature feedback: the VALUE of heating_to_t did +#: not change (False either way), but schema_version is part of the hashed payload, which is what +#: makes "old configs are never silently reinterpreted under new semantics" true rather than stated. +GOLDEN_DEFAULT_HASH = "e9d207b91d74d45076433cdd25b0f3b59f365dbefd4f8c1b47afe1fdc2373ab4" + + +@pytest.mark.tier_a +def test_default_config_hash_is_pinned() -> None: + """Tolerance: exact. A hash is either the same or it is a different config (ADR-006).""" + assert CANONICAL_FORM_VERSION == 1, "canonical form changed; the pinned hash must be re-derived" + assert config_hash(RunConfig()) == GOLDEN_DEFAULT_HASH, ( + "the default RunConfig's hash changed. If you meant to change the schema, bump " + "SCHEMA_VERSION and update GOLDEN_DEFAULT_HASH in this commit, with the reason in the " + "message. If you did not, something altered a default silently." + ) + + +@pytest.mark.tier_a +def test_hash_is_independent_of_dict_insertion_order() -> None: + """Two configs built by different code paths must hash the same. + + The API builds a config from a JSON body, the CLI from a YAML file, a sweep from + ``apply_assignments`` -- three insertion orders for the same run. If order leaked into the hash, + the same computation would get three identities and the cache would never hit. + """ + forward = RunConfig() + payload = forward.model_dump(mode="python") + shuffled = {key: payload[key] for key in reversed(list(payload))} + shuffled["background"] = { + key: shuffled["background"][key] for key in reversed(list(shuffled["background"])) + } + shuffled["background"]["gas_pptv"] = { + key: shuffled["background"]["gas_pptv"][key] + for key in reversed(list(shuffled["background"]["gas_pptv"])) + } + reversed_config = RunConfig.model_validate(shuffled) + + assert canonical_json(reversed_config) == canonical_json(forward) + assert config_hash(reversed_config) == config_hash(forward) + + +@pytest.mark.tier_a +def test_hash_is_stable_across_interpreters_and_hash_seeds() -> None: + """Run in fresh interpreters with different PYTHONHASHSEEDs and compare. + + Python randomises string hashing per process by default. Any dependence of the canonical form on + set or dict iteration influenced by that randomisation would make the hash vary between runs -- + invisible in one process, and the reason this check spawns real ones. It is also the closest a + single-version CI can get to the "stable across Python versions" requirement; the pinned + constant above covers the rest. + """ + probe = textwrap.dedent(""" + from studio.schema import RunConfig, config_hash + print(config_hash(RunConfig())) + """) + hashes = set() + for seed in ("0", "1", "12345", "random"): + proc = subprocess.run( + [sys.executable, "-c", probe], + capture_output=True, + text=True, + env={"PYTHONHASHSEED": seed, "PATH": "/usr/bin:/bin"}, + cwd=str(__import__("pathlib").Path(__file__).resolve().parents[3]), + ) + assert proc.returncode == 0, proc.stderr + hashes.add(proc.stdout.strip()) + assert hashes == {GOLDEN_DEFAULT_HASH}, f"hash varied across hash seeds: {hashes}" + + +@pytest.mark.tier_a +def test_a_changed_value_changes_the_hash() -> None: + """The other half of identity: different configs must not collide. + + Uses the smallest change that matters scientifically -- a nucleation scale of 1 vs 1.0000001 -- + because a hash that only notices large edits is worse than none. + """ + base = RunConfig() + tweaked = base.model_copy( + update={ + "microphysics": base.microphysics.model_copy( + update={"nucleation_rate_scale": 1.0000001} + ) + } + ) + assert config_hash(tweaked) != config_hash(base) + + +@pytest.mark.tier_a +def test_canonical_json_is_sorted_compact_and_parseable() -> None: + """The canonical form's rules, asserted rather than assumed.""" + text = canonical_json(RunConfig()) + assert ", " not in text and '": ' not in text, "canonical JSON must not contain padding spaces" + parsed = json.loads(text) + assert list(parsed) == sorted(parsed), "top-level keys must be sorted" + assert list(parsed["site"]) == sorted(parsed["site"]), "nested keys must be sorted too" + assert parsed == canonical_payload(RunConfig()) + + +@pytest.mark.tier_a +def test_enums_serialise_as_their_model_string() -> None: + """The hashed payload carries the model's own strings, so a config is readable as what it is.""" + parsed = json.loads(canonical_json(RunConfig())) + assert parsed["chemistry"]["photolysis"] == "tuvx" + assert parsed["background"]["aerosol"] == "sabr_220" + assert parsed["dilution"]["regime"] == "D2" + + +@pytest.mark.tier_a +def test_non_finite_values_raise_rather_than_serialising() -> None: + """NaN is not JSON, and it is not a configuration either (ADR-005). + + ``json.dumps`` would happily emit the non-standard ``NaN`` token, which then fails to parse in + every other language. Better to refuse at the boundary. + """ + base = RunConfig() + nan_config = base.model_copy( + update={"site": base.site.model_copy(update={"temperature_k": math.nan})} + ) + with pytest.raises(ValueError, match="not canonically serialisable"): + canonical_json(nan_config) + + +@pytest.mark.tier_a +def test_short_hash_is_a_prefix_and_bounded() -> None: + """Display-only, and it says so by refusing silly lengths.""" + config = RunConfig() + assert config_hash(config).startswith(short_hash(config)) + assert len(short_hash(config)) == 12 + with pytest.raises(ValueError, match=r"\[4, 64\]"): + short_hash(config, length=2) diff --git a/studio/tests/unit/test_import_boundaries.py b/studio/tests/unit/test_import_boundaries.py index d0117c7..f2f06d0 100644 --- a/studio/tests/unit/test_import_boundaries.py +++ b/studio/tests/unit/test_import_boundaries.py @@ -2,10 +2,10 @@ # SPDX-License-Identifier: Apache-2.0 """The package boundaries from ADR-001, enforced rather than documented. -``studio.schema`` and ``studio.science`` must be usable from a bare Python session: no ``coupled``, -no JAX, no database, no web framework. This is not tidiness. ``coupled.tomas_bridge`` and -``coupled.driver`` import JAX (and set ``jax_enable_x64``) at module scope, and an API that -validates a form on every keystroke cannot pay a JAX import. +``studio.schema``, ``studio.science`` and ``studio.resolve`` must be usable from a bare Python +session: no ``coupled``, no JAX, no database, no web framework. This is not tidiness. +``coupled.tomas_bridge`` and ``coupled.driver`` import JAX (and set ``jax_enable_x64``) at module +scope, and an API that validates a form on every keystroke cannot pay a JAX import. Two complementary checks: @@ -36,7 +36,8 @@ STUDIO_ROOT = REPO_ROOT / "studio" #: Packages that must stay importable without the model. See ``studio/__init__.py``. -CLEAN_PACKAGES = ["studio.schema", "studio.science"] +#: ``studio.resolve`` is here because the API resolves a config on every keystroke. +CLEAN_PACKAGES = ["studio.schema", "studio.science", "studio.resolve"] #: The single package permitted to import ``coupled`` (ADR-001). Adding to this set requires #: amending ADR-001 in the same change. @@ -103,7 +104,7 @@ def test_clean_packages_do_not_import_the_model(package: str) -> None: leaked = _forbidden_imports_after(package) assert leaked == [], ( f"{package} imported {leaked}, breaking the boundary in ADR-001. " - f"studio.schema and studio.science must be usable from a bare Python session; " + f"{CLEAN_PACKAGES} must be usable from a bare Python session; " f"only {sorted(MODEL_SEAM_PACKAGES)} may import `coupled`." ) @@ -115,7 +116,12 @@ def test_only_the_model_seam_imports_coupled() -> None: Catches imports hidden inside function bodies, which the runtime check above cannot see. """ offenders = _modules_importing_coupled() - stray = {module for module in offenders if module not in MODEL_SEAM_PACKAGES} + stray = { + module + for module in offenders + # a submodule of the seam (studio.modelio.scenario) is the seam; a sibling package is not + if not any(module == seam or module.startswith(f"{seam}.") for seam in MODEL_SEAM_PACKAGES) + } assert stray == set(), ( f"{sorted(stray)} import `coupled`, but ADR-001 names {sorted(MODEL_SEAM_PACKAGES)} as the " f"only model seam. Move the call behind studio.modelio, or amend ADR-001 in this change." diff --git a/studio/tests/unit/test_modelio_equivalence.py b/studio/tests/unit/test_modelio_equivalence.py new file mode 100644 index 0000000..da7f432 --- /dev/null +++ b/studio/tests/unit/test_modelio_equivalence.py @@ -0,0 +1,175 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The equivalence test: the schema is a faithful superset of what the ensemble actually ran. + +``to_scenario`` applied to the golden ``RunConfig`` must produce a ``CoupledScenario`` +**field-for-field identical** to ``run_ensemble.build_scenario()`` for case +``30N_20km__sabr220__D2med__a1p0__nuc1__cg1``. Tolerance: **exact**, on every field, including the +floats -- this is not a physics comparison, it is a claim that two code paths build the same object, +and "close" would mean one of them is doing arithmetic the other is not. + +Proving it here is much cheaper and sharper than discovering it from a diverging result: a mismatch +names the field, now, instead of showing up as a 3 % difference in particle number after a 4-minute +run and a day of bisection. + +These tests need the model, so they need the submodules checked out. CI does not check them out +(see ``.github/workflows/studio-ci.yml``), so this skips there and runs locally -- the same +asymmetry as the ``air_number_density`` mirror check, and worth the same caution. +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path +from typing import Any + +import pytest + +from studio.resolve import resolve, set_override +from studio.schema import BackgroundAerosol, DilutionRegime, PhotolysisMode, RunConfig + +#: The axis values for the golden case, exactly as ``run_ensemble.py:61-76`` spells them. +GOLDEN_AXES: dict[str, Any] = { + "lat_alt": ("30N_20km", 30.0, 210.0, 55.0, 6.9104), + "background": ("sabr220", "sabr_220", 20.0), + "dilution": ("D2med", "D2"), + "sticking": ("a1p0", 1.0), + "nucleation": ("nuc1", 1.0), + "coag": ("cg1", 1.0), +} + + +@pytest.fixture(scope="module") +def build_scenario(repo_root: Path) -> Any: + """``run_ensemble.build_scenario``, or skip if the model is not checked out.""" + if not (repo_root / "stratchem-jax" / "config.py").is_file(): + pytest.skip( + "model submodules not checked out (`git submodule update --init`); the equivalence " + "test compares against run_ensemble.build_scenario, which needs them" + ) + from coupled.paper_ensemble.run_ensemble import build_scenario as builder + + return builder + + +@pytest.fixture(scope="module") +def studio_scenario() -> Any: + from studio.modelio.scenario import to_scenario + + return to_scenario(resolve(RunConfig())) + + +@pytest.mark.tier_a +def test_the_golden_case_is_the_schemas_default(build_scenario: Any) -> None: + """``RunConfig()`` with no arguments IS the golden case; nothing has to be set up to get it.""" + reference = build_scenario(GOLDEN_AXES) + config = RunConfig() + assert config.site.temperature_k == reference.T + assert config.site.pressure_mbar == reference.P + assert config.background.aerosol is BackgroundAerosol.SABR_220 + assert config.dilution.regime is DilutionRegime.D2 + assert config.chemistry.photolysis is PhotolysisMode.TUVX + + +@pytest.mark.tier_a +def test_to_scenario_is_field_for_field_identical( + build_scenario: Any, studio_scenario: Any +) -> None: + """The whole point of the task. Every field, exact equality, no exceptions.""" + reference = dataclasses.asdict(build_scenario(GOLDEN_AXES)) + produced = dataclasses.asdict(studio_scenario) + + assert set(produced) == set(reference), "the two scenarios have different field sets" + differing = { + key: (reference[key], produced[key]) for key in reference if produced[key] != reference[key] + } + assert differing == {}, ( + "to_scenario diverges from run_ensemble.build_scenario for the golden case " + f"(reference, produced): {differing}. The schema is meant to be a faithful superset; " + f"either the mapping is wrong or the schema default is." + ) + + +@pytest.mark.tier_a +def test_the_initial_so2_matches_to_the_last_bit(build_scenario: Any, studio_scenario: Any) -> None: + """Called out separately because it is the one value Studio DERIVES rather than passes through. + + Everything else is a copy; this one goes mass -> number density -> mixing ratio through + ``studio.science`` while the ensemble does the same arithmetic inline. Exact equality is the + evidence that consolidating that derivation changed nothing. + """ + reference = build_scenario(GOLDEN_AXES) + assert studio_scenario.concentrations["SO2"] == reference.concentrations["SO2"] + assert studio_scenario.concentrations == reference.concentrations + + +@pytest.mark.tier_a +def test_the_constant_regime_maps_to_the_models_empty_string(build_scenario: Any) -> None: + """The one enum value that is not the model's own string (``enums.py``). + + Asserted against a real ``CoupledScenario`` rather than against the mapping table, because the + model validates ``dilution_regime`` in ``__post_init__`` -- if ``""`` ever stopped being the + spelling, this fails here rather than in a run. + """ + from studio.modelio.scenario import to_scenario + + config = RunConfig.model_validate( + {**RunConfig().model_dump(), "dilution": {"regime": "constant"}} + ) + assert to_scenario(resolve(config)).dilution_regime == "" + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("regime", "expected"), + [("D1", "D1"), ("D2", "D2"), ("D3", "D3"), ("D5", "D5"), ("burst", "burst")], +) +def test_every_other_regime_passes_through_unchanged( + build_scenario: Any, regime: str, expected: str +) -> None: + """The claim that the enum values ARE the model's strings, checked rather than trusted.""" + from studio.modelio.scenario import to_scenario + + config = RunConfig.model_validate({**RunConfig().model_dump(), "dilution": {"regime": regime}}) + assert to_scenario(resolve(config)).dilution_regime == expected + + +@pytest.mark.tier_a +def test_an_unresolved_config_is_refused(build_scenario: Any) -> None: + """A config that never went through the resolver has ``so2_initial_pptv = None``. + + Handing that to the model would start the plume with no SO2 -- a run that completes and means + nothing. It raises instead (ADR-005). + """ + from studio.modelio.scenario import to_scenario + + with pytest.raises(ValueError, match="so2_initial_pptv is unresolved"): + to_scenario(RunConfig()) + + +@pytest.mark.tier_a +def test_a_stale_config_is_refused(build_scenario: Any) -> None: + """Converting a stale config would hand the model numbers that do not follow from each other.""" + from studio.modelio.scenario import to_scenario + from studio.resolve import InconsistentConfigError, apply_change + + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError): + to_scenario(stale) + + +@pytest.mark.tier_a +def test_a_resolved_override_reaches_the_model(build_scenario: Any) -> None: + """The other half: an override the user has settled must actually arrive.""" + from studio.modelio.scenario import to_scenario + from studio.resolve import keep_override + + overridden = set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9) + assert ( + to_scenario(keep_override(overridden, "injection.so2_initial_pptv")).concentrations["SO2"] + == 5.0e9 + ) diff --git a/studio/tests/unit/test_modelio_summary.py b/studio/tests/unit/test_modelio_summary.py new file mode 100644 index 0000000..fa18c24 --- /dev/null +++ b/studio/tests/unit/test_modelio_summary.py @@ -0,0 +1,236 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSummary``, against a synthetic archive and against a real one. + +The synthetic cases run everywhere, including CI, and are where the traps are tested deliberately: +shuffled species order, a non-uniform time axis, a closed box vs a diluting one. The real-archive +case runs only where the 810-run ensemble exists and checks that the reduction survives contact with +an actual 3.6 MB file. + +Note ``summary.py`` does not import ``coupled``: it reads arrays. So all of this runs without the +model, which is why these tests are not skipped in CI while the equivalence tests are. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest + +from studio.modelio import ( + SUMMARY_SCHEMA_VERSION, + Basis, + RunSummary, + SummaryFlag, + TerminationReason, + summarise_state_npz, +) + +GOLDEN_CASE = "30N_20km__sabr220__D2med__a1p0__nuc1__cg1" + +#: The mechanism's species list, in a deliberately awkward order: SO2/SO3/H2SO4 are NOT at the +#: positions the existing analysis scripts hard-code (32, 34, 35). +SPECIES = ("H2SO4", "OH", "SO2", "O3", "HO2", "SO3") + + +def _write_npz( + path: Path, + *, + n_times: int = 5, + n_bins: int = 4, + diluting: bool = True, + time_s: np.ndarray | None = None, +) -> Path: + """A synthetic ``state.npz`` with the real file's key set and shapes.""" + rng = np.random.default_rng(20260813) # seeded: any stochastic component records its seed + time_s = np.arange(n_times, dtype=np.float64) * 592.0 if time_s is None else time_s + state = np.zeros((n_times, len(SPECIES)), dtype=np.float64) + state[:, SPECIES.index("SO2")] = np.linspace(6.0e15, 1.0e15, n_times) + state[:, SPECIES.index("SO3")] = np.linspace(0.0, 1.0e10, n_times) + state[:, SPECIES.index("H2SO4")] = np.linspace(0.0, 5.0e13, n_times) + state[:, SPECIES.index("OH")] = rng.uniform(1e5, 1e7, n_times) + state[:, SPECIES.index("HO2")] = rng.uniform(1e6, 1e8, n_times) + state[:, SPECIES.index("O3")] = np.full(n_times, 1.18e12) + edges = np.geomspace(1.7e-3, 17.5, n_bins + 1) + counts = rng.uniform(1.0, 100.0, (n_times, n_bins)) + np.savez( + path, + t=time_s, + x=state, + species=np.array(SPECIES), + M=np.float64(1.8956916099773243e18), + SA=np.linspace(2.0, 40.0, n_times), + radius_cm=np.full(n_times, 1.2e-5), + h2so4wp=np.full(n_times, 0.72), + # closes the closed-box budget exactly: SO2 1.0e15 + SO3 1.0e10 + H2SO4 5.0e13 + this + # == the initial 6.0e15 + particulate_S=np.linspace(0.0, 6.0e15 - 1.0e15 - 1.0e10 - 5.0e13, n_times), + T=np.full(n_times, 210.0), + n_cm3=counts, + Dp_m=np.tile(np.sqrt(edges[:-1] * edges[1:]) * 1e-6, (n_times, 1)), + dp_mid_um=np.sqrt(edges[:-1] * edges[1:]), + dNdlogDp=counts / np.log10(edges[1:] / edges[:-1]), + V_ratio=np.linspace(1.0, 1500.0, n_times) if diluting else np.ones(n_times), + total_n=counts.sum(axis=1), + ) + return path + + +@pytest.fixture +def synthetic_npz(tmp_path: Path) -> Path: + return _write_npz(tmp_path / "state.npz") + + +@pytest.mark.tier_a +def test_species_are_indexed_by_name_not_position(synthetic_npz: Path) -> None: + """The trap, tested head-on. + + ``SPECIES`` puts SO2 at index 2 and H2SO4 at index 0 -- nothing like the 32/34/35 an existing + analysis script hard-codes. A summary that indexed by position would report ozone as SO2 here, + and would report something plausible rather than crashing. + """ + summary = summarise_state_npz(synthetic_npz) + m_air = 1.8956916099773243e18 + assert summary.series["SO2"].values[0] == pytest.approx(6.0e15 / m_air * 1e12, rel=1e-12) + assert summary.series["O3"].values[0] == pytest.approx(1.18e12 / m_air * 1e12, rel=1e-12) + assert summary.series["SO2"].values[-1] < summary.series["SO2"].values[0] + + +@pytest.mark.tier_a +def test_the_time_axis_comes_from_the_stored_t(tmp_path: Path) -> None: + """Never ``i * DT``. Outer steps snap to the terminator, so the mean step is ~592 s, not 600. + + Over a 36-day run that is about half a day of drift -- enough to put a diurnal feature on the + wrong side of local noon. + """ + irregular = np.array([0.0, 592.0, 1184.0, 1776.0, 2368.0]) # 4 x 592, not 4 x 600 + summary = summarise_state_npz(_write_npz(tmp_path / "state.npz", time_s=irregular)) + np.testing.assert_allclose(summary.time_days, irregular / 86400.0, rtol=0.0, atol=0.0) + nominal = np.arange(5) * 600.0 / 86400.0 + assert summary.time_days[-1] != pytest.approx(nominal[-1]), "a nominal grid would differ here" + + +@pytest.mark.tier_a +def test_every_series_declares_its_basis(synthetic_npz: Path) -> None: + """Wet vs dry is not optional metadata; it is a factor of a few in radius at 55 hPa.""" + summary = summarise_state_npz(synthetic_npz) + assert summary.series["SA"].basis is Basis.WET + assert summary.series["radius_cm"].basis is Basis.WET + assert summary.series["total_n"].basis is Basis.DRY + assert summary.series["SO2"].basis is Basis.NOT_APPLICABLE + assert summary.final_size_distribution is not None + assert summary.final_size_distribution.basis is Basis.DRY + for name, series in summary.series.items(): + assert series.unit, f"{name} has no unit" + assert series.description, f"{name} has no description" + + +@pytest.mark.tier_a +def test_the_final_size_distribution_is_the_last_step(synthetic_npz: Path) -> None: + with np.load(synthetic_npz) as archive: + expected = archive["n_cm3"][-1] + distribution = summarise_state_npz(synthetic_npz).final_size_distribution + assert distribution is not None + np.testing.assert_allclose(distribution.number_cm3, expected, rtol=0.0) + assert distribution.total_number_cm3 == pytest.approx(float(expected.sum()), rel=1e-15) + assert len(distribution.diameter_um) == len(distribution.dn_dlogdp_cm3) == len(expected) + + +@pytest.mark.tier_a +def test_a_diluting_run_reports_no_conservation_residual(synthetic_npz: Path) -> None: + """The box is an open system, so a residual would measure the dilution, not conservation. + + Reporting a number here would invite a reader to conclude something from it. The start and end + values are still reported, so the decay is visible without being dressed up as a budget error. + """ + check = summarise_state_npz(synthetic_npz).sulfur_conservation + assert check is not None + assert check.status == "not_applicable" + assert check.relative_residual is None + assert "open system" in check.reason + assert check.initial_value is not None and check.final_value is not None + assert SummaryFlag.OPEN_SYSTEM_DILUTION in summarise_state_npz(synthetic_npz).flags + + +@pytest.mark.tier_a +def test_a_closed_box_gets_a_real_residual(tmp_path: Path) -> None: + """With V(t)/V0 == 1 throughout, sulfur should be conserved and the residual means something. + + The synthetic archive is built so gas + particulate sulfur closes exactly: 6.0e15 at t = 0, and + SO2 1.0e15 + SO3 1.0e10 + H2SO4 5.0e13 + particulate 4.94999e15 at the end. Tolerance 1e-12 + relative -- float64 summation noise on five terms, not a physical tolerance, because the + quantity being checked is arithmetic rather than physics. + """ + check = summarise_state_npz( + _write_npz(tmp_path / "state.npz", diluting=False) + ).sulfur_conservation + assert check is not None + assert check.status == "computed" + assert check.relative_residual == pytest.approx(0.0, abs=1e-12) + assert ( + SummaryFlag.OPEN_SYSTEM_DILUTION + not in summarise_state_npz(_write_npz(tmp_path / "closed.npz", diluting=False)).flags + ) + + +@pytest.mark.tier_a +def test_termination_is_recorded_never_inferred(synthetic_npz: Path) -> None: + """The npz says what the state did, not why the loop stopped. Guessing would be the difference + between "converged" and "cut short".""" + assert summarise_state_npz(synthetic_npz).termination is TerminationReason.UNKNOWN + stopped = summarise_state_npz(synthetic_npz, termination=TerminationReason.TERMINATED_ON_LIMIT) + assert SummaryFlag.STOPPED_ON_LIMIT in stopped.flags + + +@pytest.mark.tier_a +def test_a_summary_without_provenance_says_so(synthetic_npz: Path) -> None: + """The archived ensemble has no config hash (ADR-006). That is a flag, not a blank field.""" + assert SummaryFlag.NO_PROVENANCE_RECORD in summarise_state_npz(synthetic_npz).flags + with_hash = summarise_state_npz(synthetic_npz, config_hash="abc123", label=GOLDEN_CASE) + assert SummaryFlag.NO_PROVENANCE_RECORD not in with_hash.flags + assert with_hash.config_hash == "abc123" + assert with_hash.label == GOLDEN_CASE + + +@pytest.mark.tier_a +def test_a_summary_round_trips_through_json(synthetic_npz: Path, tmp_path: Path) -> None: + """It is written next to state.npz and read back by comparison views; both directions matter.""" + original = summarise_state_npz(synthetic_npz, label="case", config_hash="abc123") + path = original.write(tmp_path / "summary.json") + restored = RunSummary.read(path) + assert restored == original + assert restored.schema_version == SUMMARY_SCHEMA_VERSION + + +@pytest.mark.tier_a +def test_a_truncated_archive_raises(tmp_path: Path) -> None: + """Missing arrays are a corrupted run, not a run with fewer series.""" + path = tmp_path / "state.npz" + np.savez(path, t=np.zeros(3), species=np.array(SPECIES)) + with pytest.raises(ValueError, match="missing required arrays"): + summarise_state_npz(path) + + +@pytest.mark.tier_a +def test_summarising_a_real_archived_run(paper_ensemble_runs: Path) -> None: + """The golden case, straight from the 810-run ensemble. Skips where the archive is absent.""" + path = paper_ensemble_runs / GOLDEN_CASE / "state.npz" + if not path.is_file(): + pytest.skip(f"{path} not present") + summary = summarise_state_npz(path, label=GOLDEN_CASE) + + assert len(summary.time_days) == 1461 + assert summary.time_days[0] == 0.0 + assert summary.time_days[-1] == pytest.approx(10.0, rel=1e-12), "a 10-day run" + assert summary.series["SO2"].values[-1] < summary.series["SO2"].values[0], "SO2 is consumed" + assert max(summary.series["H2SO4"].values) > 0.0, "H2SO4 is produced" + assert summary.final_size_distribution is not None + assert len(summary.final_size_distribution.diameter_um) == 80 + assert summary.termination is TerminationReason.UNKNOWN + assert set(summary.flags) >= { + SummaryFlag.NO_PROVENANCE_RECORD, + SummaryFlag.OPEN_SYSTEM_DILUTION, + } + assert summary.sulfur_conservation is not None + assert summary.sulfur_conservation.status == "not_applicable" diff --git a/studio/tests/unit/test_provenance.py b/studio/tests/unit/test_provenance.py new file mode 100644 index 0000000..8582eef --- /dev/null +++ b/studio/tests/unit/test_provenance.py @@ -0,0 +1,224 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Provenance records (ADR-006). + +These build **real git repositories** in a temp directory rather than mocking ``subprocess``. +The module is a thin shell around git's behaviour, so a mocked git would test the mock: whether +``status --porcelain`` reports an untracked file, whether a missing ``.git`` fails as expected, +whether ``rev-parse`` in a fresh repo with no commits errors — those are the questions, and only +git answers them. + +Cost: ~1 s for a handful of ``git init`` calls. Cheap enough for Tier A, and it means the dirty-tree +and not-a-checkout paths are genuinely exercised rather than asserted about a stub. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from studio.modelio.provenance import ( + MODEL_SUBMODULES, + PROVENANCE_SCHEMA_VERSION, + NotAGitCheckoutError, + ProvenanceRecord, + describe_checkout, + record_for, + repository_root, +) +from studio.resolve import ResolvedConfig, apply_change, resolve, set_override +from studio.schema import RunConfig + + +@pytest.fixture +def resolved() -> ResolvedConfig: + return resolve(RunConfig()) + + +@pytest.mark.tier_a +def test_the_record_carries_every_field_adr_006_requires( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """config hash, app version, SANDBOX SHA, three submodule SHAs, datasets, resolved config.""" + record = record_for(resolved, repo_root=fake_sandbox) + + assert record.schema_version == PROVENANCE_SCHEMA_VERSION + assert record.config_hash == resolved.config.config_hash() + assert record.studio_version # studio.__version__, whatever it currently is + assert len(record.sandbox.commit) == 40 + assert set(record.submodules) == set(MODEL_SUBMODULES) + assert all(len(sub.commit) == 40 for sub in record.submodules.values()) + assert record.datasets == {}, "present and empty, so its absence is never ambiguous" + assert record.recorded_at.tzinfo is not None, "a naive timestamp compares wrongly across zones" + + +@pytest.mark.tier_a +def test_the_recorded_config_is_the_resolved_one( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """ "What the model actually received", not what the user typed. + + The derived fields are the test: a record of the user's inputs would have ``None`` here, and + would not let anyone reconstruct the run. + """ + record = record_for(resolved, repo_root=fake_sandbox) + injection = record.resolved_config["injection"] + assert injection["plume_volume_cm3"] == 1.5e12 + assert injection["so2_initial_pptv"] == pytest.approx(3.309115922996412e9, rel=1e-15) + assert record.resolved_config["schema_version"] == resolved.config.schema_version + + +@pytest.mark.tier_a +def test_an_override_is_recorded_with_the_value_in_force(fake_sandbox: Path) -> None: + """A user-supplied derived value must be visible as an override, not silently indistinguishable + from a computed one.""" + from studio.resolve import keep_override + + overridden = keep_override( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "injection.so2_initial_pptv", + ) + record = record_for(overridden, repo_root=fake_sandbox) + assert record.overrides == {"injection.so2_initial_pptv": 5.0e9} + assert record.resolved_config["injection"]["so2_initial_pptv"] == 5.0e9 + + +@pytest.mark.tier_a +def test_a_clean_tree_is_reproducible(fake_sandbox: Path, resolved: ResolvedConfig) -> None: + record = record_for(resolved, repo_root=fake_sandbox) + assert record.is_reproducible + assert record.dirty_checkouts == () + + +@pytest.mark.tier_a +def test_a_dirty_sandbox_is_recorded_and_flags_the_run( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """The SHA no longer describes the code that ran -- exactly when someone wants to know.""" + (fake_sandbox / "file.txt").write_text("modified\n", encoding="utf-8") + record = record_for(resolved, repo_root=fake_sandbox) + + assert record.sandbox.dirty + assert record.sandbox.dirty_files, "the record must say WHAT was uncommitted" + assert not record.is_reproducible + assert record.dirty_checkouts == ("SANDBOX",) + + +@pytest.mark.tier_a +def test_an_untracked_file_counts_as_dirty(fake_sandbox: Path, resolved: ResolvedConfig) -> None: + """``status --porcelain`` rather than ``diff --quiet``, on purpose. + + An untracked module that a run imported is exactly the kind of thing that makes a SHA a lie, and + ``git diff`` would not see it. + """ + (fake_sandbox / "scratch_module.py").write_text("x = 1\n", encoding="utf-8") + assert record_for(resolved, repo_root=fake_sandbox).sandbox.dirty + + +@pytest.mark.tier_a +def test_a_dirty_submodule_flags_the_run_and_the_parent( + fake_sandbox: Path, resolved: ResolvedConfig +) -> None: + """The model lives in the submodules; a dirty one means the model that ran is not any commit. + + **Both** checkouts are flagged, and that is git being helpful rather than the record being + imprecise: a registered submodule with a dirty working tree also shows up in the PARENT's + ``status --porcelain`` as modified, because the parent's recorded submodule pointer no longer + describes what is on disk. So an edited submodule cannot hide behind a clean-looking SANDBOX. + """ + (fake_sandbox / "stratchem-jax" / "file.txt").write_text("edited\n", encoding="utf-8") + record = record_for(resolved, repo_root=fake_sandbox) + assert record.submodules["stratchem-jax"].dirty + assert not record.is_reproducible + assert record.dirty_checkouts == ("SANDBOX", "stratchem-jax") + assert not record.submodules["tuvx-jax"].dirty, "only the edited submodule is dirty" + + +@pytest.mark.tier_a +def test_not_a_git_checkout_raises(tmp_path: Path, resolved: ResolvedConfig) -> None: + """An empty SHA looks like an answer, so this refuses to produce one (ADR-005).""" + plain = tmp_path / "not-a-repo" + plain.mkdir() + for name in MODEL_SUBMODULES: + (plain / name).mkdir() + with pytest.raises(NotAGitCheckoutError): + record_for(resolved, repo_root=plain) + + +@pytest.mark.tier_a +def test_a_missing_submodule_raises(tmp_path: Path, resolved: ResolvedConfig) -> None: + """The SANDBOX SHA alone does not pin the model (ADR-001), so a missing submodule is fatal.""" + from studio.tests.conftest import _make_repo + + root = _make_repo(tmp_path / "partial") + _make_repo(root / "tuvx-jax") # the other two are absent + with pytest.raises(NotAGitCheckoutError, match="submodule"): + record_for(resolved, repo_root=root) + + +@pytest.mark.tier_a +def test_a_repo_with_no_commits_raises(tmp_path: Path, resolved: ResolvedConfig) -> None: + """``rev-parse HEAD`` has nothing to report, which is a failure rather than an empty string.""" + from studio.tests.conftest import _git + + root = tmp_path / "empty" + root.mkdir() + _git(root, "init", "--quiet") + with pytest.raises(NotAGitCheckoutError): + describe_checkout(root) + + +@pytest.mark.tier_a +def test_a_stale_config_is_refused(fake_sandbox: Path) -> None: + """Recording provenance for an inconsistent config would give it a respectable pedigree.""" + from studio.resolve import InconsistentConfigError + + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError): + record_for(stale, repo_root=fake_sandbox) + + +@pytest.mark.tier_a +def test_the_record_is_immutable(fake_sandbox: Path, resolved: ResolvedConfig) -> None: + """Written once, before the run, never mutated (ADR-006).""" + record = record_for(resolved, repo_root=fake_sandbox) + with pytest.raises(ValueError, match="frozen"): + record.config_hash = "tampered" # type: ignore[misc] + + +@pytest.mark.tier_a +def test_the_record_round_trips_through_json( + fake_sandbox: Path, resolved: ResolvedConfig, tmp_path: Path +) -> None: + original = record_for(resolved, repo_root=fake_sandbox) + restored = ProvenanceRecord.read(original.write(tmp_path / "provenance.json")) + assert restored == original + assert restored.is_reproducible == original.is_reproducible + + +@pytest.mark.tier_a +def test_the_repository_root_is_derived_from_the_package_not_the_cwd() -> None: + """An API process started anywhere must still record the checkout the code came from.""" + root = repository_root() + assert (root / "studio").is_dir() + assert (root / "coupled").is_dir() + + +@pytest.mark.tier_a +def test_this_checkout_can_be_pinned(resolved: ResolvedConfig) -> None: + """The real repository, not a synthetic one: submodules present, SHAs readable. + + Skips where the submodules are absent, which is the same condition every other model-touching + test skips on -- including in CI. + """ + root = repository_root() + if not all((root / name / ".git").exists() for name in MODEL_SUBMODULES): + pytest.skip("model submodules not checked out (`git submodule update --init`)") + record = record_for(resolved, repo_root=root) + assert len(record.sandbox.commit) == 40 + assert set(record.submodules) == set(MODEL_SUBMODULES) diff --git a/studio/tests/unit/test_resolve_graph.py b/studio/tests/unit/test_resolve_graph.py new file mode 100644 index 0000000..977742c --- /dev/null +++ b/studio/tests/unit/test_resolve_graph.py @@ -0,0 +1,131 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The dependency graph, on synthetic graphs and on the real schema. + +Most of these run on hand-built graphs rather than on ``RunConfig``. That is deliberate: the +properties under test (transitivity, ordering, cycle detection) are properties of the algorithm, and +the schema currently has exactly one chain of length two. A graph engine tested only against the +shape it happens to be used with is a graph engine that breaks the first time the schema grows. +""" + +from __future__ import annotations + +import pytest + +from studio.resolve import CyclicDependencyError, DependencyGraph, schema_derived_fields + +#: a -> b -> d, a -> c -> d, and an isolated node. +DIAMOND = {"a": (), "b": ("a",), "c": ("a",), "d": ("b", "c"), "lonely": ()} + + +@pytest.mark.tier_a +def test_dependents_are_one_hop() -> None: + graph = DependencyGraph(DIAMOND) + assert set(graph.dependents_of("a")) == {"b", "c"} + assert graph.dependents_of("d") == () + assert graph.dependencies_of("d") == ("b", "c") + assert graph.dependencies_of("a") == () + + +@pytest.mark.tier_a +def test_downstream_is_transitive_and_excludes_the_seed() -> None: + """The recompute set. Transitive, because a one-hop answer leaves the far end silently stale.""" + graph = DependencyGraph(DIAMOND) + assert graph.downstream_of(["a"]) == ("b", "c", "d") + assert graph.downstream_of(["b"]) == ("d",) + assert graph.downstream_of(["d"]) == () + assert graph.downstream_of(["lonely"]) == () + + +@pytest.mark.tier_a +def test_downstream_of_several_seeds_is_the_union_without_duplicates() -> None: + """Editing two fields at once must not recompute the shared descendant twice.""" + graph = DependencyGraph(DIAMOND) + assert graph.downstream_of(["b", "c"]) == ("d",) + + +@pytest.mark.tier_a +def test_downstream_is_in_topological_order() -> None: + """Ordering is the contract: a dependent must never be computed before its dependency.""" + graph = DependencyGraph({"x": (), "mid": ("x",), "far": ("mid",)}) + assert graph.downstream_of(["x"]) == ("mid", "far") + + +@pytest.mark.tier_a +def test_topological_order_is_deterministic() -> None: + """Ties broken alphabetically, so two runs of the same engine order identically.""" + first = DependencyGraph(DIAMOND).nodes + second = DependencyGraph(dict(reversed(list(DIAMOND.items())))).nodes + assert first == second == ("a", "lonely", "b", "c", "d") + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + "graph", + [ + {"a": ("a",)}, # self-edge + {"a": ("b",), "b": ("a",)}, # two-cycle + {"a": ("c",), "b": ("a",), "c": ("b",)}, # three-cycle + {"ok": (), "a": ("b",), "b": ("a",)}, # cycle alongside a healthy node + ], +) +def test_cycles_raise_rather_than_hang(graph: dict[str, tuple[str, ...]]) -> None: + """No fixed-point iteration, no arbitrary edge-breaking: both would make the result depend on + where the engine started.""" + with pytest.raises(CyclicDependencyError, match="cycle"): + DependencyGraph(graph) + + +@pytest.mark.tier_a +def test_a_dependency_on_an_unknown_node_raises() -> None: + """A ``derived_from`` naming a field that does not exist is a typo that drops an edge.""" + with pytest.raises(ValueError, match="not a node in the graph"): + DependencyGraph({"a": ("ghost",)}) + + +@pytest.mark.tier_a +def test_querying_an_unknown_path_raises() -> None: + graph = DependencyGraph(DIAMOND) + with pytest.raises(ValueError, match="unknown field path"): + graph.dependents_of("nope") + with pytest.raises(ValueError, match="unknown field path"): + graph.downstream_of(["nope"]) + + +@pytest.mark.tier_a +def test_the_real_schema_graph_is_acyclic_and_complete() -> None: + """Building it is the assertion: a cycle or a dangling ``derived_from`` raises here.""" + graph = DependencyGraph.from_schema() + from studio.schema import field_catalogue + + assert set(graph.nodes) == set(field_catalogue()) + + +@pytest.mark.tier_a +def test_the_schemas_derived_chain() -> None: + """``so2_initial_pptv`` depends on ``plume_volume_cm3``, which is itself derived. + + Pinned because it is the case that makes topological order matter rather than be decoration: a + resolver that recomputed in declaration order could use last round's volume. + """ + graph = DependencyGraph.from_schema() + assert schema_derived_fields() == ( + "injection.plume_volume_cm3", + "injection.so2_initial_pptv", + ) + assert "injection.plume_volume_cm3" in graph.dependencies_of("injection.so2_initial_pptv") + assert graph.downstream_of(["injection.plume_length_m"]) == ( + "injection.plume_volume_cm3", + "injection.so2_initial_pptv", + ) + assert graph.downstream_of(["site.temperature_k"]) == ("injection.so2_initial_pptv",) + + +@pytest.mark.tier_a +def test_primary_fields_have_no_dependencies() -> None: + """Everything the user types is a root; only DERIVED fields have inputs.""" + graph = DependencyGraph.from_schema() + derived = set(schema_derived_fields()) + for path in graph.nodes: + if path not in derived: + assert graph.dependencies_of(path) == (), f"{path} is primary but has dependencies" diff --git a/studio/tests/unit/test_resolve_registry.py b/studio/tests/unit/test_resolve_registry.py new file mode 100644 index 0000000..b3c4919 --- /dev/null +++ b/studio/tests/unit/test_resolve_registry.py @@ -0,0 +1,106 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The registry must agree with the schema, exactly. + +Two failure modes this guards, both of which produce a config that looks resolved and is not: + +* a field declares an input the derivation ignores -- editing it marks things stale and recomputes + to the same number, so the UI reports a change that did not happen; +* a derivation reads a value the schema does not list -- editing THAT one recomputes nothing, and + the stale result reaches the model. +""" + +from __future__ import annotations + +import pytest + +from studio.resolve import DERIVATIONS, derivation_for, schema_derived_fields +from studio.schema import field_catalogue + + +@pytest.mark.tier_a +def test_every_derived_field_has_a_derivation() -> None: + """A DERIVED field with nothing to compute it stays None all the way to the model seam.""" + missing = [path for path in schema_derived_fields() if path not in DERIVATIONS] + assert ( + missing == [] + ), f"schema fields marked DERIVED with no entry in studio/resolve/registry.py: {missing}" + + +@pytest.mark.tier_a +def test_no_derivation_exists_for_a_field_the_schema_does_not_derive() -> None: + """The reverse direction: a stale registry entry would never run and would rot unnoticed.""" + extra = [path for path in DERIVATIONS if path not in set(schema_derived_fields())] + assert ( + extra == [] + ), f"registered derivations for fields the schema does not mark DERIVED: {extra}" + + +@pytest.mark.tier_a +def test_declared_inputs_match_derived_from_exactly() -> None: + """Order-insensitive, but membership must be identical -- this is the DAG's correctness.""" + catalogue = field_catalogue() + for path in schema_derived_fields(): + schema_inputs = set(catalogue[path]["derived_from"]) + registry_inputs = set(derivation_for(path).inputs) + assert registry_inputs == schema_inputs, ( + f"{path}: schema says it derives from {sorted(schema_inputs)}, the registry reads " + f"{sorted(registry_inputs)}. The graph and the computation must agree or the recompute " + f"set is wrong in one direction or the other." + ) + + +@pytest.mark.tier_a +def test_an_unregistered_field_raises_rather_than_returning_none() -> None: + with pytest.raises(NotImplementedError, match="no derivation registered"): + derivation_for("site.temperature_k") + + +@pytest.mark.tier_a +def test_a_derivation_refuses_inputs_it_did_not_declare() -> None: + """A missing input means resolver and registry disagree; never default it away.""" + derivation = derivation_for("injection.plume_volume_cm3") + with pytest.raises(KeyError, match="missing declared inputs"): + derivation.compute({"injection.plume_length_m": 15000.0}) + + +@pytest.mark.tier_a +def test_derivations_produce_the_same_values_as_studio_science_directly() -> None: + """The registry is a binding, not a second implementation. Tolerance: exact.""" + from studio.science import ( + SO2_MOLAR_MASS_G_PER_MOL, + initial_mixing_ratio_pptv, + plume_volume_cm3, + ) + + volume = derivation_for("injection.plume_volume_cm3").compute( + { + "injection.plume_length_m": 15000.0, + "injection.plume_width_m": 10.0, + "injection.plume_height_m": 10.0, + } + ) + assert volume == plume_volume_cm3(15000.0, 10.0, 10.0) + + pptv = derivation_for("injection.so2_initial_pptv").compute( + { + "injection.so2_mass_kg": 1000.0, + "injection.plume_volume_cm3": volume, + "site.temperature_k": 210.0, + "site.pressure_mbar": 55.0, + } + ) + assert pptv == initial_mixing_ratio_pptv( + mass_kg=1000.0, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=volume, + pressure_mbar=55.0, + temperature_k=210.0, + ) + + +@pytest.mark.tier_a +def test_every_derivation_carries_a_summary() -> None: + """The UI has to say what it recomputed and why; an empty string is not an explanation.""" + for path in schema_derived_fields(): + assert derivation_for(path).summary.strip(), f"{path} has no summary" diff --git a/studio/tests/unit/test_resolve_resolver.py b/studio/tests/unit/test_resolve_resolver.py new file mode 100644 index 0000000..5346d4b --- /dev/null +++ b/studio/tests/unit/test_resolve_resolver.py @@ -0,0 +1,274 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Resolution and override semantics -- the "edit stage 1 without losing stage 6" guarantee. + +The load-bearing test is ``test_an_edit_changes_exactly_the_downstream_closure``: it captures every +field before and after an edit and asserts the set that moved is EXACTLY the edited field plus its +closure. Both failure directions matter and both are silent -- recomputing too little leaves a stale +number that reaches the model, recomputing too much quietly discards something the user set. +""" + +from __future__ import annotations + +import subprocess +import sys +import textwrap +from pathlib import Path +from typing import Any + +import pytest + +from studio.resolve import ( + InconsistentConfigError, + accept_derived, + apply_change, + keep_override, + resolve, + set_override, +) +from studio.schema import RunConfig, field_catalogue + +SO2_PPTV = "injection.so2_initial_pptv" +VOLUME = "injection.plume_volume_cm3" + + +def _flat(resolved: Any) -> dict[str, Any]: + """Every leaf value, by path, for before/after comparison.""" + return {path: resolved.value_at(path) for path in field_catalogue()} + + +@pytest.mark.tier_a +def test_a_fresh_config_resolves_its_derived_fields() -> None: + """The schema ships them unset (0.2 declares, 0.3 resolves); this is where they get values.""" + assert RunConfig().injection.plume_volume_cm3 is None + resolved = resolve(RunConfig()) + assert resolved.config.injection.plume_volume_cm3 == 1.5e12 + assert resolved.config.injection.so2_initial_pptv == pytest.approx( + 3.309115922996412e9, rel=1e-15 + ) + assert resolved.is_consistent + assert resolved.overrides == {} + + +@pytest.mark.tier_a +def test_chained_derivations_resolve_in_order() -> None: + """``so2_initial_pptv`` must see the NEW volume, not the previous one. + + Halving the track length halves V0 and therefore doubles nothing -- it halves the concentration. + A resolver running in declaration order rather than topological order would return the old + value here, which is why this is asserted numerically rather than structurally. + """ + resolved = apply_change(resolve(RunConfig()), "injection.plume_length_m", 30000.0) + assert resolved.config.injection.plume_volume_cm3 == 3.0e12 + assert resolved.config.injection.so2_initial_pptv == pytest.approx( + 1.654557961498206e9, rel=1e-15 + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("path", "value", "expected_changed"), + [ + ("injection.plume_length_m", 30000.0, {VOLUME, SO2_PPTV}), + ("injection.plume_width_m", 20.0, {VOLUME, SO2_PPTV}), + ("site.temperature_k", 213.0, {SO2_PPTV}), + ("site.pressure_mbar", 120.0, {SO2_PPTV}), + ("injection.so2_mass_kg", 2000.0, {SO2_PPTV}), + ("microphysics.n_bins", 40, set()), + ("chemistry.so2_ho2_rate", 1e-16, set()), + ("switches.aerosol_to_j", True, set()), # heating_to_t cannot be True (schema 0.2.0) + ], +) +def test_an_edit_changes_exactly_the_downstream_closure( + path: str, value: Any, expected_changed: set[str] +) -> None: + """Not "at least" and not "at most". Exactly. + + Under-recomputing leaves a stale number that reaches the model; over-recomputing silently + discards a value the user chose. The last three cases matter as much as the first five: editing + a field with no dependents must move nothing else at all. + """ + before_resolved = resolve(RunConfig()) + before = _flat(before_resolved) + after = _flat(apply_change(before_resolved, path, value)) + changed = {key for key in before if before[key] != after[key]} + assert changed == expected_changed | {path} + + +@pytest.mark.tier_a +def test_an_override_is_never_overwritten_by_a_recomputation() -> None: + """The stage-6 choice survives the stage-1 edit. This is the whole point of the task.""" + overridden = set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9) + assert overridden.config.injection.so2_initial_pptv == 5.0e9 + assert overridden.is_consistent, "an override anchored to the current inputs is not stale" + + after_edit = apply_change(overridden, "site.temperature_k", 213.0) + assert after_edit.config.injection.so2_initial_pptv == 5.0e9 + + +@pytest.mark.tier_a +def test_a_moved_input_marks_the_override_stale_with_both_values() -> None: + """ "Stale" without "here is what it would be" leaves the user to recompute by hand.""" + overridden = set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9) + after_edit = apply_change(overridden, "site.temperature_k", 213.0) + + assert after_edit.stale_fields == (SO2_PPTV,) + (entry,) = after_edit.stale + assert entry.current_value == 5.0e9 + assert entry.derived_value == pytest.approx(3.3563890076106462e9, rel=1e-15) + assert entry.summary + assert [(c.path, c.was, c.now) for c in entry.changed_inputs] == [ + ("site.temperature_k", 210.0, 213.0) + ] + + +@pytest.mark.tier_a +def test_an_unrelated_edit_does_not_make_an_override_stale() -> None: + """Only a change to one of ITS inputs counts. Flagging on every edit would train users to + dismiss the flag.""" + overridden = set_override(resolve(RunConfig()), VOLUME, 3.0e12) + after = apply_change(overridden, "site.temperature_k", 213.0) + assert after.is_consistent + assert after.config.injection.plume_volume_cm3 == 3.0e12 + + +@pytest.mark.tier_a +def test_a_downstream_auto_field_uses_the_overridden_value() -> None: + """An override is the value in force, so anything computed from it must use it.""" + overridden = set_override(resolve(RunConfig()), VOLUME, 3.0e12) + assert overridden.config.injection.so2_initial_pptv == pytest.approx( + 1.654557961498206e9, rel=1e-15 + ), "so2_initial_pptv must be computed from the overridden V0, not the geometric one" + + +@pytest.mark.tier_a +def test_accept_derived_drops_the_override_and_recomputes() -> None: + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + accepted = accept_derived(stale, SO2_PPTV) + assert accepted.is_consistent + assert accepted.overrides == {} + assert accepted.config.injection.so2_initial_pptv == pytest.approx( + 3.3563890076106462e9, rel=1e-15 + ) + + +@pytest.mark.tier_a +def test_keep_override_re_anchors_and_clears_staleness() -> None: + """The user has said, knowingly, that their value still applies -- that is the difference + between this and never having flagged it.""" + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + kept = keep_override(stale, SO2_PPTV) + assert kept.is_consistent + assert kept.config.injection.so2_initial_pptv == 5.0e9 + assert kept.overrides[SO2_PPTV].inputs["site.temperature_k"] == 213.0 + + # ...and it goes stale again on the NEXT change, rather than being permanently silenced + assert apply_change(kept, "site.temperature_k", 220.0).stale_fields == (SO2_PPTV,) + + +@pytest.mark.tier_a +def test_a_stale_config_refuses_to_pass_as_consistent() -> None: + """A stale config still has a hash, and that is the trap: a stable identity for numbers that do + not follow from each other.""" + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + with pytest.raises(InconsistentConfigError, match="stale override"): + stale.require_consistent() + assert resolve(RunConfig()).require_consistent() is None + + +@pytest.mark.tier_a +def test_the_stale_list_travels_with_the_config() -> None: + """Serialising must carry the stale list, or a persisted config loses the fact that it is + inconsistent -- exactly what the plan forbids.""" + stale = apply_change( + set_override(resolve(RunConfig()), SO2_PPTV, 5.0e9), "site.temperature_k", 213.0 + ) + restored = type(stale).model_validate_json(stale.model_dump_json()) + assert restored.stale_fields == (SO2_PPTV,) + assert restored.overrides[SO2_PPTV].value == 5.0e9 + with pytest.raises(InconsistentConfigError): + restored.require_consistent() + + +@pytest.mark.tier_a +def test_editing_a_derived_field_directly_is_an_override() -> None: + """A user typing into a computed box means "I want this value", not "recompute me away".""" + edited = apply_change(resolve(RunConfig()), VOLUME, 2.0e12) + assert edited.overrides[VOLUME].value == 2.0e12 + assert edited.config.injection.plume_volume_cm3 == 2.0e12 + + +@pytest.mark.tier_a +def test_overriding_a_primary_field_is_refused() -> None: + """Primary fields have no derivation to be stale against; the concept does not apply.""" + with pytest.raises(ValueError, match="not a derived field"): + set_override(resolve(RunConfig()), "site.temperature_k", 999.0) + + +@pytest.mark.tier_a +def test_settling_a_field_that_is_not_overridden_is_refused() -> None: + resolved = resolve(RunConfig()) + with pytest.raises(ValueError, match="nothing to accept"): + accept_derived(resolved, SO2_PPTV) + with pytest.raises(ValueError, match="nothing to keep"): + keep_override(resolved, SO2_PPTV) + + +@pytest.mark.tier_a +def test_an_unknown_path_raises() -> None: + with pytest.raises(ValueError, match="unknown field"): + apply_change(resolve(RunConfig()), "site.temprature_k", 210.0) + + +@pytest.mark.tier_a +def test_an_invalid_value_is_rejected_by_the_schema_during_resolution() -> None: + """Resolution does not bypass validation: bounds still apply to an edited value.""" + with pytest.raises(ValueError, match="condensation_alpha"): + apply_change(resolve(RunConfig()), "microphysics.condensation_alpha", 1.5) + + +@pytest.mark.tier_a +def test_a_degenerate_input_is_caught_by_the_schema_before_the_derivation_runs() -> None: + """A zero plume dimension is rejected at validation, not deep in the arithmetic. + + Both layers refuse it -- ``plume_volume_cm3`` raises on a non-positive dimension too (see + ``test_science_plume.py``) -- but the schema's ``gt=0`` fires first, which is the better place: + the error names the field the user typed in rather than a function they have never heard of. + The derivation's own check remains as the guard for any caller that does not come through the + schema. + """ + with pytest.raises(ValueError, match="plume_length_m"): + apply_change(resolve(RunConfig()), "injection.plume_length_m", 0.0) + + +@pytest.mark.tier_a +def test_resolution_does_not_import_the_model() -> None: + """The API resolves on every keystroke; a JAX import on that path would be unaffordable. + + Run in a FRESH interpreter, not in-process. ``test_import_boundaries.py`` covers the same ground + for imports; this one covers the CALL, because a lazy import inside ``resolve()`` would slip + past an import-time check. In-process it would prove nothing either way: another test module in + this session imports ``studio.modelio.scenario``, which is allowed to reach the model, and that + alone would put ``coupled`` in ``sys.modules``. + """ + probe = textwrap.dedent(""" + import sys + from studio.resolve import resolve + from studio.schema import RunConfig + resolve(RunConfig()) + print(sorted({"coupled", "jax", "jaxlib"} & {m.split(".")[0] for m in sys.modules})) + """) + proc = subprocess.run( + [sys.executable, "-c", probe], + capture_output=True, + text=True, + cwd=str(Path(__file__).resolve().parents[3]), + ) + assert proc.returncode == 0, proc.stderr + assert proc.stdout.strip() == "[]", f"resolving pulled in {proc.stdout.strip()}" diff --git a/studio/tests/unit/test_runner.py b/studio/tests/unit/test_runner.py new file mode 100644 index 0000000..b46ad9d --- /dev/null +++ b/studio/tests/unit/test_runner.py @@ -0,0 +1,422 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The job lifecycle and the local subprocess runner. + +These launch **real subprocesses** -- the runner's whole job is process management, and a mocked +``Popen`` would test the mock. What they do not launch is the model: ``entry_module`` points at a +small fixture module that exits, fails, or sleeps on command. That is a parameter of the runner +rather than a test hook: nothing in ``LocalSubprocessRunner`` branches on its value, and the default +is the real entry point. + +The tests that matter most are the ones about **what survives a failure**. A run that dies at minute +three of four must leave enough behind to diagnose it without paying those three minutes again. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import pytest + +from studio.resolve import ResolvedConfig, apply_change, resolve, set_override +from studio.runner import ( + THREAD_PINNING, + InvalidTransitionError, + JobRecord, + JobState, + LocalSubprocessRunner, + SlurmRunner, +) +from studio.schema import RunConfig + +#: A stand-in for ``studio.cli.run``: same argv shape, controllable outcome, no model. +FIXTURE_MODULE = "studio.tests.fixtures.fake_run" + + +@pytest.fixture +def resolved() -> ResolvedConfig: + return resolve(RunConfig()) + + +@pytest.fixture +def runner(tmp_path: Path, fake_sandbox: Path) -> LocalSubprocessRunner: + """A runner over the fixture entry point, recording provenance for a synthetic checkout. + + ``repo_root=fake_sandbox`` because submitting now writes a provenance record, which requires a + pinnable checkout -- and CI checks out no submodules. Pointing at a synthetic checkout is what + lets these tests run in CI rather than skipping; the production default is the real one. + """ + made = LocalSubprocessRunner( + tmp_path / "jobs", + max_workers=2, + entry_module=FIXTURE_MODULE, + repo_root=fake_sandbox, + ) + yield made + made.shutdown(cancel_running=True) + + +class TestLifecycle: + """Transitions are data, and illegal ones raise.""" + + @pytest.mark.tier_a + def test_the_happy_path_records_every_step(self) -> None: + record = ( + JobRecord(job_id="j", config_hash="h") + .transition_to(JobState.QUEUED) + .transition_to(JobState.RUNNING) + .transition_to(JobState.SUCCEEDED) + ) + assert [t.state for t in record.transitions] == [ + JobState.QUEUED, + JobState.RUNNING, + JobState.SUCCEEDED, + ] + assert record.is_terminal + assert record.submitted_at and record.started_at and record.ended_at + assert record.duration_s is not None and record.duration_s >= 0.0 + + @pytest.mark.tier_a + @pytest.mark.parametrize( + ("from_state", "to_state"), + [ + (JobState.DRAFT, JobState.RUNNING), # never ran the queue + (JobState.RUNNING, JobState.QUEUED), # backwards + (JobState.SUCCEEDED, JobState.RUNNING), # terminal + (JobState.FAILED, JobState.SUCCEEDED), # rewriting history + ], + ) + def test_illegal_transitions_raise(self, from_state: JobState, to_state: JobState) -> None: + """A job that appears to move backwards means the runner lost track of a process. + + Accepting it silently would turn the record from a log into a story. + """ + record = JobRecord(job_id="j", config_hash="h").model_copy(update={"state": from_state}) + with pytest.raises(InvalidTransitionError): + record.transition_to(to_state) + + @pytest.mark.tier_a + def test_a_record_is_immutable(self) -> None: + """It is an audit trail; one that can be edited in place can disagree with what happened.""" + record = JobRecord(job_id="j", config_hash="h") + with pytest.raises(ValueError, match="frozen"): + record.state = JobState.RUNNING # type: ignore[misc] + assert record.transition_to(JobState.QUEUED) is not record + + @pytest.mark.tier_a + def test_terminated_on_limit_is_not_failed(self) -> None: + """Two different things: "could not produce a result" vs "we stopped it mid-flight". + + Collapsing them would let a partial run be read as a converged one. + """ + assert JobState.TERMINATED_ON_LIMIT != JobState.FAILED + record = ( + JobRecord(job_id="j", config_hash="h") + .transition_to(JobState.QUEUED) + .transition_to(JobState.RUNNING) + .transition_to(JobState.TERMINATED_ON_LIMIT, detail="exceeded max_wall_time_s") + ) + assert record.is_terminal + assert "max_wall_time" in record.detail + + +class TestLocalSubprocessRunner: + @pytest.mark.tier_a + def test_a_successful_run_reaches_succeeded( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + record = runner.submit(resolved, label="ok") + assert record.state is JobState.QUEUED + final = runner.wait(record.job_id, timeout=30) + assert final.state is JobState.SUCCEEDED + assert final.exit_code == 0 + assert final.config_hash == resolved.config.config_hash() + + @pytest.mark.tier_a + def test_the_resolved_input_is_written_before_the_run( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Written at submit, not completion, so a job that dies at once still has its input.""" + record = runner.submit(resolved) + assert record.input_path is not None and record.input_path.is_file() + restored = ResolvedConfig.model_validate_json(record.input_path.read_text()) + assert restored.config.config_hash() == resolved.config.config_hash() + runner.wait(record.job_id, timeout=30) + + @pytest.mark.tier_a + def test_a_failed_run_keeps_everything_needed_to_diagnose_it( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The point of the whole design: no re-run required. + + A four-minute case that fails intermittently must not have to be reproduced to be + understood, so the exit code, both log streams and the exact input are all on disk. + """ + _directive(monkeypatch, mode="fail", message="synthetic failure") + record = runner.submit(resolved, label="boom") + final = runner.wait(record.job_id, timeout=30) + + assert final.state is JobState.FAILED + assert final.exit_code == 1 + assert final.stderr_path is not None + assert "synthetic failure" in final.stderr_path.read_text() + assert final.input_path is not None and final.input_path.is_file() + names = {path.name for path in runner.artifacts(final.job_id)} + assert {"input.json", "stdout.log", "stderr.log"} <= names + + @pytest.mark.tier_a + def test_stdout_is_captured_as_the_log_stream( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """``run_coupled`` prints rather than logs, so stdout IS the run log.""" + _directive(monkeypatch, mode="ok", message="[coupled] NOTE: something happened") + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + assert final.stdout_path is not None + assert "[coupled] NOTE: something happened" in final.stdout_path.read_text() + + @pytest.mark.tier_a + def test_exceeding_the_wall_clock_limit_is_terminated_not_failed( + self, runner: LocalSubprocessRunner, monkeypatch: pytest.MonkeyPatch + ) -> None: + """And the record says so, so nothing downstream reads the partial output as converged.""" + config = resolve( + RunConfig.model_validate( + {**RunConfig().model_dump(), "termination": {"max_wall_time_s": 1.0}} + ) + ) + _directive(monkeypatch, mode="sleep", seconds=30) + record = runner.submit(config, label="slow") + final = runner.wait(record.job_id, timeout=60) + + assert final.state is JobState.TERMINATED_ON_LIMIT + assert "max_wall_time_s" in final.detail + assert "NOT a converged result" in final.detail + + @pytest.mark.tier_a + def test_cancelling_a_running_job_stops_it( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + _directive(monkeypatch, mode="sleep", seconds=30) + record = runner.submit(resolved) + _wait_for_state(runner, record.job_id, JobState.RUNNING) + cancelled = runner.cancel(record.job_id) + assert cancelled.state is JobState.CANCELLED + assert runner.poll(record.job_id).state is JobState.CANCELLED + + @pytest.mark.tier_a + def test_cancelling_a_finished_job_is_not_an_error( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Cancelling something that already finished is a race, not a mistake by the caller.""" + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + assert runner.cancel(final.job_id).state is final.state + + @pytest.mark.tier_a + def test_a_stale_config_is_refused_at_submission(self, runner: LocalSubprocessRunner) -> None: + """Nothing downstream could tell that the numbers did not follow from each other.""" + from studio.resolve import InconsistentConfigError + + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError): + runner.submit(stale) + + @pytest.mark.tier_a + def test_polling_an_unknown_job_raises(self, runner: LocalSubprocessRunner) -> None: + with pytest.raises(KeyError, match="unknown job id"): + runner.poll("nope") + + @pytest.mark.tier_a + def test_jobs_get_separate_work_directories( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Two runs of the SAME config must not share a directory and overwrite each other.""" + first = runner.submit(resolved) + second = runner.submit(resolved) + assert first.work_dir != second.work_dir + assert first.config_hash == second.config_hash, "same config, same identity" + for job in (first, second): + runner.wait(job.job_id, timeout=30) + + @pytest.mark.tier_a + def test_the_subprocess_gets_the_thread_pinning_environment( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """This pinning -- not vmap -- is what gives ~N times the throughput for N workers. + + Asserted by having the fixture module dump its own environment, because a runner that + *intends* to pin threads and does not would only show up as everything being slow. + """ + _directive(monkeypatch, mode="dump_env") + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + assert final.stdout_path is not None + reported = json.loads(final.stdout_path.read_text()) + assert {key: reported.get(key) for key in THREAD_PINNING} == THREAD_PINNING + + @pytest.mark.tier_a + def test_a_bad_input_file_exits_distinctly_from_a_model_failure( + self, tmp_path: Path, resolved: ResolvedConfig + ) -> None: + """Exit code 2 means "never started"; 1 means "the model raised". The runner needs both. + + Runs the REAL entry point (``studio.cli.run``), because this is its contract, and a bad + input is rejected before any model import -- so it costs milliseconds, not a JAX load. + """ + import subprocess + + bad = tmp_path / "bad.json" + bad.write_text('{"config": {"site": {"temperature_k": -5}}}', encoding="utf-8") + proc = subprocess.run( + [sys.executable, "-m", "studio.cli.run", str(bad), str(tmp_path / "out")], + capture_output=True, + text=True, + cwd=str(Path(__file__).resolve().parents[3]), + ) + assert proc.returncode == 2, proc.stderr + assert "cannot run" in proc.stderr + + +class TestUnimplementedBackends: + @pytest.mark.tier_a + def test_slurm_raises_rather_than_falling_back_to_local(self) -> None: + """A job running somewhere other than where it was sent is worse than an error (ADR-008).""" + with pytest.raises(NotImplementedError, match="Slurm"): + SlurmRunner().submit(None) + with pytest.raises(NotImplementedError, match="not implemented"): + SlurmRunner().poll("x") + + +def _directive(monkeypatch: pytest.MonkeyPatch, **directive: object) -> None: + """Tell the fixture module what to do, BEFORE the job is submitted. + + Via the environment rather than a file in the work directory: the subprocess can start before a + file written after ``submit()`` lands, which is exactly how process tests become flaky. + """ + monkeypatch.setenv("STUDIO_FAKE_RUN", json.dumps(directive)) + + +def _wait_for_state( + runner: LocalSubprocessRunner, job_id: str, state: JobState, timeout: float = 30.0 +) -> None: + import time + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if runner.poll(job_id).state is state: + return + time.sleep(0.02) + raise AssertionError(f"job {job_id} never reached {state}") + + +class TestMaxSimTimeStopCondition: + """The ``stop_condition`` this layer hands the model, and the shape it must have. + + Pure-function tests: no model run, but they pin the coupling that PR #73 exposed. The model + dispatches on the callback's DECLARED ARITY and raises ``TypeError`` on ``*args`` -- correctly, + since a variadic callback matches both shapes and guessing would be a silent wrong answer. That + makes the arity part of this function's contract rather than an implementation detail, so it is + asserted here where a change is cheap to notice. + """ + + @pytest.mark.tier_a + def test_no_limit_means_no_stop_condition(self) -> None: + """``None`` is not a callback that never fires; it is no callback at all.""" + from studio.modelio.execute import _max_sim_time_stop + + assert _max_sim_time_stop(resolve(RunConfig())) is None + + @pytest.mark.tier_a + def test_the_callback_takes_exactly_one_parameter(self) -> None: + """The diagnostics-dict shape. Not ``*args``, which the model rejects as ambiguous. + + Arity is part of the contract, not an implementation detail: two parameters still work but + emit a ``DeprecationWarning``, and a variadic callback raises ``TypeError``. Asserted here + because it is cheap to notice and expensive to discover from a run. + """ + import inspect + + from studio.modelio.execute import _max_sim_time_stop + + stop = _max_sim_time_stop(_with_sim_limit(2.0)) + assert stop is not None + parameters = list(inspect.signature(stop).parameters.values()) + assert len(parameters) == 1 + assert parameters[0].kind is inspect.Parameter.POSITIONAL_OR_KEYWORD + + @pytest.mark.tier_a + def test_it_fires_exactly_at_the_limit(self) -> None: + """Boundary included: at the limit the run has reached its cap, not almost reached it.""" + from studio.modelio.execute import _max_sim_time_stop + + stop = _max_sim_time_stop(_with_sim_limit(2.0)) + assert stop is not None + two_days_s = 2.0 * 86400.0 + assert stop({"t": two_days_s - 1.0, "SA": 12.0}) is False + assert stop({"t": two_days_s, "SA": 12.0}) is True + assert stop({"t": two_days_s + 1.0, "SA": 12.0}) is True + + +def _with_sim_limit(days: float) -> ResolvedConfig: + payload = RunConfig().model_dump() + payload["termination"]["max_sim_time_days"] = days + return resolve(RunConfig.model_validate(payload)) + + +class TestProvenanceIsWrittenAtSubmit: + """A run's provenance must exist before it can fail (ADR-006).""" + + @pytest.mark.tier_a + def test_provenance_exists_the_moment_submit_returns( + self, runner: LocalSubprocessRunner, resolved: ResolvedConfig + ) -> None: + """Before the process finishes -- so a run that dies in minute three still has it. + + Asserted against the record returned by ``submit()``, not after ``wait()``: the point is the + ordering, and checking afterwards would pass even if it were written at completion. + """ + from studio.modelio.provenance import ProvenanceRecord + + record = runner.submit(resolved, label="prov") + assert record.provenance_path is not None + assert record.provenance_path.is_file(), "written at submit, not at completion" + + provenance = ProvenanceRecord.read(record.provenance_path) + assert provenance.config_hash == resolved.config.config_hash() == record.config_hash + assert set(provenance.submodules) # the model is pinned, not just the app + runner.wait(record.job_id, timeout=30) + + @pytest.mark.tier_a + def test_a_failed_run_still_has_its_provenance( + self, + runner: LocalSubprocessRunner, + resolved: ResolvedConfig, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The case that matters: four minutes in, exit 1, and someone asks what produced it.""" + _directive(monkeypatch, mode="fail", message="synthetic failure") + record = runner.submit(resolved) + final = runner.wait(record.job_id, timeout=30) + + assert final.state is JobState.FAILED + assert final.provenance_path is not None and final.provenance_path.is_file() + assert "provenance.json" in {p.name for p in runner.artifacts(final.job_id)} diff --git a/studio/tests/unit/test_runset.py b/studio/tests/unit/test_runset.py new file mode 100644 index 0000000..c7ba508 --- /dev/null +++ b/studio/tests/unit/test_runset.py @@ -0,0 +1,344 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``RunSet`` expansion, checked against the sweep this project actually ran. + +The strongest available evidence that the axis model is faithful is that it reproduces the paper +ensemble: 810 cases, the same order, and the same case-ID labels that name the directories on disk +today. That is what ``test_reproduces_the_paper_ensemble`` does, and it is why the axis kinds are +shaped the way they are -- ``LIST`` exists because the site axis covaries latitude, T, p and H2O. + +The axis definitions live here rather than in the package: task 0.2 is the schema, not a library of +presets. If task 0.4 or 0.7 needs them, that is when they earn a home in ``studio/``. +""" + +from __future__ import annotations + +import pytest + +from studio.schema import ( + Axis, + AxisKind, + AxisPoint, + BackgroundAerosol, + DilutionRegime, + RunConfig, + RunSet, + apply_assignments, +) + +#: The six axes of ``coupled/paper_ensemble/run_ensemble.py:61-76``, in declaration order. +#: Labels are the ensemble's own tokens, so the expanded labels are its case IDs verbatim. +PAPER_AXES = ( + Axis( + name="site", + # latitude, T, p and H2O move together; the cross product is not physically meaningful + kind=AxisKind.LIST, + points=( + AxisPoint( + label="30N_20km", + assignments={ + "site.latitude_deg": 30.0, + "site.temperature_k": 210.0, + "site.pressure_mbar": 55.0, + "site.h2o_ppmv": 6.9104, + }, + ), + AxisPoint( + label="60N_15km", + assignments={ + "site.latitude_deg": 60.0, + "site.temperature_k": 210.0, + "site.pressure_mbar": 120.0, + "site.h2o_ppmv": 3.1673, + }, + ), + AxisPoint( + label="30N_20km_213K", + assignments={ + "site.latitude_deg": 30.0, + "site.temperature_k": 213.0, + "site.pressure_mbar": 55.0, + "site.h2o_ppmv": 10.1834, + }, + ), + ), + ), + Axis( + name="background", + kind=AxisKind.LIST, # aerosol distribution and background SO2 co-vary + points=( + AxisPoint( + label="sabr330", + assignments={ + "background.aerosol": BackgroundAerosol.SABR_330, + "background.so2_pptv": 20.0, + }, + ), + AxisPoint( + label="sabr220", + assignments={ + "background.aerosol": BackgroundAerosol.SABR_220, + "background.so2_pptv": 20.0, + }, + ), + AxisPoint( + label="cesm", + assignments={ + "background.aerosol": BackgroundAerosol.CESM_G6, + "background.so2_pptv": 100.0, + }, + ), + ), + ), + Axis.over( + "dilution", + "dilution.regime", + { + "D1low": DilutionRegime.D1, + "D2med": DilutionRegime.D2, + "D3high": DilutionRegime.D3, + "burst": DilutionRegime.BURST, + "D5vhigh": DilutionRegime.D5, + }, + ), + Axis.over("sticking", "microphysics.condensation_alpha", {"a0p5": 0.5, "a1p0": 1.0}), + Axis.over( + "nucleation", + "microphysics.nucleation_rate_scale", + {"nuc0p01": 0.01, "nuc1": 1.0, "nuc100": 100.0}, + ), + Axis.over("coag", "microphysics.coag_kernel_scale", {"cg0p5": 0.5, "cg1": 1.0, "cg2": 2.0}), +) + +#: The case the golden tests key on, and its index in the ensemble's ordering +#: (site 0, background 1, dilution 1, sticking 1, nucleation 1, coag 1 under itertools.product). +GOLDEN_CASE_ID = "30N_20km__sabr220__D2med__a1p0__nuc1__cg1" +GOLDEN_CASE_INDEX = 121 + + +@pytest.mark.tier_a +def test_a_single_run_is_a_runset_with_no_axes() -> None: + """N = 1 goes through the same expansion as N = 810. There is no second code path.""" + runs = RunSet().expand() + assert len(runs) == 1 + assert runs[0].config == RunConfig() + assert runs[0].label == "" + assert runs[0].coordinates == {} + + +@pytest.mark.tier_a +def test_reproduces_the_paper_ensemble() -> None: + """810 cases, in the ensemble's order, with the ensemble's case IDs. + + Compared against the labels rather than against ``run_ensemble.all_cases()`` directly: importing + that module pulls in ``coupled`` and therefore JAX, which Tier A must stay clear of. The tokens + below are copied from ``run_ensemble.py:61-76``, so a divergence in either direction shows up. + """ + runset = RunSet(axes=PAPER_AXES) + assert runset.size() == 810 == 3 * 3 * 5 * 2 * 3 * 3 + runs = runset.expand() + assert len(runs) == runset.size(), "size() must agree with expand() without building anything" + + assert runs[0].label == "30N_20km__sabr330__D1low__a0p5__nuc0p01__cg0p5" + assert runs[-1].label == "30N_20km_213K__cesm__D5vhigh__a1p0__nuc100__cg2" + assert len({run.label for run in runs}) == 810, "case IDs must be unique" + + golden = runs[GOLDEN_CASE_INDEX] + assert golden.label == GOLDEN_CASE_ID + assert golden.coordinates == { + "site": "30N_20km", + "background": "sabr220", + "dilution": "D2med", + "sticking": "a1p0", + "nucleation": "nuc1", + "coag": "cg1", + } + + +@pytest.mark.tier_a +def test_the_golden_case_resolves_to_the_ensembles_values() -> None: + """Spot-check the resolved config against ``run_ensemble.build_scenario`` for the golden case. + + This is not the equivalence proof -- that is task 0.4, field-for-field against a real + ``CoupledScenario``. It is the cheap version that catches an axis wired to the wrong path now, + rather than after the model seam exists. + """ + config = RunSet(axes=PAPER_AXES).expand()[GOLDEN_CASE_INDEX].config + assert config.site.latitude_deg == 30.0 + assert config.site.temperature_k == 210.0 + assert config.site.pressure_mbar == 55.0 + assert config.site.h2o_ppmv == 6.9104 + assert config.background.aerosol is BackgroundAerosol.SABR_220 + assert config.background.so2_pptv == 20.0 + assert config.dilution.regime is DilutionRegime.D2 + assert config.microphysics.condensation_alpha == 1.0 + assert config.microphysics.nucleation_rate_scale == 1.0 + assert config.microphysics.coag_kernel_scale == 1.0 + # unswept values stay at the ensemble's fixed configuration + assert config.microphysics.n_bins == 80 + assert config.microphysics.ion_pair_rate == 30.0 + assert config.schedule.day_of_year == 172 + assert config.chemistry.so2_ho2_rate == 1e-18 + assert config.switches.aerosol_to_j is False + assert config.switches.heating_to_t is False + + +@pytest.mark.tier_a +def test_expanded_runs_have_distinct_identities() -> None: + """Different parameters, different hashes -- on a real sweep, not a two-element toy.""" + runs = RunSet( + axes=( + Axis.over( + "nuc", "microphysics.nucleation_rate_scale", {"lo": 0.01, "mid": 1.0, "hi": 100.0} + ), + Axis.over("coag", "microphysics.coag_kernel_scale", {"a": 0.5, "b": 1.0, "c": 2.0}), + ) + ).expand() + assert len({run.config_hash for run in runs}) == len(runs) == 9 + + +@pytest.mark.tier_a +def test_grid_order_varies_the_last_axis_fastest() -> None: + """Ordering is part of the contract: it is what makes an expansion reproducible.""" + runs = RunSet( + axes=( + Axis.over("a", "microphysics.condensation_alpha", {"a1": 0.5, "a2": 1.0}), + Axis.over("b", "microphysics.coag_kernel_scale", {"b1": 0.5, "b2": 1.0, "b3": 2.0}), + ) + ).expand() + assert [run.label for run in runs] == [ + "a1__b1", + "a1__b2", + "a1__b3", + "a2__b1", + "a2__b2", + "a2__b3", + ] + + +@pytest.mark.tier_a +def test_zip_axes_advance_in_lockstep_and_cross_with_the_grid() -> None: + """ZIP pairs values instead of crossing them; the pair is then crossed with GRID axes.""" + runs = RunSet( + axes=( + Axis.over( + "site_t", "site.temperature_k", {"cold": 210.0, "warm": 213.0}, kind=AxisKind.ZIP + ), + Axis.over( + "site_p", "site.pressure_mbar", {"low": 55.0, "high": 120.0}, kind=AxisKind.ZIP + ), + Axis.over("nuc", "microphysics.nucleation_rate_scale", {"n1": 1.0, "n2": 100.0}), + ) + ).expand() + assert [run.label for run in runs] == [ + "cold__low__n1", + "cold__low__n2", + "warm__high__n1", + "warm__high__n2", + ] + assert (runs[0].config.site.temperature_k, runs[0].config.site.pressure_mbar) == (210.0, 55.0) + assert (runs[2].config.site.temperature_k, runs[2].config.site.pressure_mbar) == (213.0, 120.0) + + +@pytest.mark.tier_a +def test_zip_axes_of_unequal_length_are_rejected() -> None: + """Silently truncating to the shorter axis would drop runs the user asked for.""" + with pytest.raises(ValueError, match="equal length"): + RunSet( + axes=( + Axis.over("t", "site.temperature_k", {"a": 210.0, "b": 213.0}, kind=AxisKind.ZIP), + Axis.over("p", "site.pressure_mbar", {"x": 55.0}, kind=AxisKind.ZIP), + ) + ) + + +@pytest.mark.tier_a +def test_two_axes_assigning_the_same_field_are_rejected() -> None: + """The result would depend on axis order, so it is refused rather than silently ordered.""" + with pytest.raises(ValueError, match="both assign"): + RunSet( + axes=( + Axis.over("a", "microphysics.n_bins", {"lo": 40}), + Axis.over("b", "microphysics.n_bins", {"hi": 80}), + ) + ) + + +@pytest.mark.tier_a +def test_a_grid_axis_may_not_covary_two_fields() -> None: + """Covariation is what LIST is for; allowing it on GRID would make the kind meaningless.""" + with pytest.raises(ValueError, match="LIST"): + Axis( + name="site", + kind=AxisKind.GRID, + points=( + AxisPoint( + label="a", assignments={"site.temperature_k": 210.0, "site.pressure_mbar": 55.0} + ), + ), + ) + + +@pytest.mark.tier_a +def test_duplicate_point_labels_are_rejected() -> None: + """Labels become directory names; two runs cannot share one.""" + with pytest.raises(ValueError, match="duplicate point labels"): + Axis( + name="nuc", + points=( + AxisPoint(label="x", assignments={"microphysics.nucleation_rate_scale": 1.0}), + AxisPoint(label="x", assignments={"microphysics.nucleation_rate_scale": 2.0}), + ), + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("path", "message"), + [ + ("microphysics.n_bin", "unknown field"), + ("microphysics", "is a group of fields"), # a group has no unit, provenance or DAG node + ("site.temperature_k.value", "leaf field"), + ("nonexistent.thing", "unknown field"), + ], +) +def test_bad_axis_paths_are_rejected_at_construction(path: str, message: str) -> None: + """A typo'd path would otherwise produce a sweep whose axis silently never varied.""" + with pytest.raises(ValueError, match=message): + Axis(name="typo", points=(AxisPoint(label="x", assignments={path: 1.0}),)) + + +@pytest.mark.tier_a +def test_axis_values_are_validated_at_expansion_time() -> None: + """Fail on run 1 of 810, not on run 407. + + An out-of-range level is a mistake in the sweep definition; discovering it hours in, after + compute has been spent, is the expensive way to find out. + """ + runset = RunSet( + axes=(Axis.over("alpha", "microphysics.condensation_alpha", {"ok": 1.0, "bad": 1.5}),) + ) + with pytest.raises(ValueError, match="condensation_alpha"): + runset.expand() + + +@pytest.mark.tier_a +def test_apply_assignments_leaves_the_base_untouched() -> None: + """Configs are frozen and expansion must not alias them (ADR-004).""" + base = RunConfig() + changed = apply_assignments(base, {"microphysics.n_bins": 40}) + assert changed.microphysics.n_bins == 40 + assert base.microphysics.n_bins == 80 + assert changed.config_hash() != base.config_hash() + + +@pytest.mark.tier_a +def test_cross_field_rules_still_apply_to_swept_values() -> None: + """A sweep cannot slip past validation that a hand-written config would hit. + + ``couple_dt_s`` must divide ``output_dt_s`` (mirroring coupled_scenario.py:200-204), including + when it arrives from an axis. + """ + runset = RunSet(axes=(Axis.over("dt", "numerics.couple_dt_s", {"bad": 450.0}),)) + with pytest.raises(ValueError, match="integer multiple"): + runset.expand() diff --git a/studio/tests/unit/test_schema_export.py b/studio/tests/unit/test_schema_export.py new file mode 100644 index 0000000..a79f20d --- /dev/null +++ b/studio/tests/unit/test_schema_export.py @@ -0,0 +1,176 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""JSON Schema export and round-tripping. + +The export is the contract with the web client (ADR-002): the form is generated from it, so a field +that loses its metadata on the way out is a field the UI cannot explain. Round-tripping is the +contract with everything else -- the CLI reads YAML, the API reads JSON, the runner writes the +resolved config next to the results, and all three must reconstruct the same object with the same +hash. +""" + +from __future__ import annotations + +import json + +import pytest + +from studio.schema import ( + EXTENSION_KEY, + SCHEMA_ID, + SCHEMA_VERSION, + RunConfig, + config_hash, + field_catalogue, + run_config_json_schema, +) + + +@pytest.mark.tier_a +def test_export_is_json_serialisable_and_identifies_itself() -> None: + """A client must be able to tell which schema version it is holding.""" + schema = run_config_json_schema() + json.dumps(schema) # raises if anything in the export is not JSON + assert schema["$id"] == SCHEMA_ID + assert SCHEMA_VERSION in schema["$id"] + assert schema["x-studio-schema-version"] == SCHEMA_VERSION + assert schema["$schema"].startswith("https://json-schema.org/") + + +@pytest.mark.tier_a +def test_metadata_survives_the_export() -> None: + """Unit, provenance and source must reach the client, or the form cannot explain a field.""" + schema = run_config_json_schema() + site = schema["$defs"]["Site"]["properties"] + temperature = site["temperature_k"][EXTENSION_KEY] + assert temperature["unit"] == "K" + assert temperature["provenance"] == "paper_ensemble" + assert "TABLE_microphysics_parameters.md" in temperature["source"] + assert temperature["range"] == {"gt": 0.0} + assert site["temperature_k"]["description"] + + +@pytest.mark.tier_a +def test_every_leaf_field_appears_in_the_catalogue_with_a_default() -> None: + """``RunConfig()`` must be constructible with no arguments -- it is the form's opening state. + + The only fields without a default are the derived ones, which are unresolved by design. + """ + catalogue = field_catalogue() + assert len(catalogue) >= 40 + missing_default = [path for path, meta in catalogue.items() if "default" not in meta] + assert missing_default == [], f"fields with no default: {missing_default}" + derived_unset = [ + path + for path, meta in catalogue.items() + if meta["provenance"] == "derived" and meta["default"] is not None + ] + assert derived_unset == [] + + +@pytest.mark.tier_a +def test_catalogue_paths_are_the_same_paths_runset_axes_use() -> None: + """One vocabulary. A path read from the schema must be usable as an axis path unchanged.""" + from studio.schema import resolve_path + + for path in field_catalogue(): + resolve_path(RunConfig, path) + + +@pytest.mark.tier_a +def test_json_round_trip_preserves_identity() -> None: + """Serialise, parse, revalidate: same object, same hash.""" + original = RunConfig() + restored = RunConfig.model_validate_json(original.model_dump_json()) + assert restored == original + assert config_hash(restored) == config_hash(original) + + +@pytest.mark.tier_a +def test_round_trip_survives_a_non_default_config() -> None: + """Defaults round-tripping proves little; a config with every group altered proves more.""" + original = RunConfig.model_validate( + { + "site": { + "latitude_deg": -60.0, + "longitude_deg": 175.0, + "temperature_k": 213.0, + "pressure_mbar": 120.0, + "h2o_ppmv": 3.1673, + }, + "schedule": {"day_of_year": 355, "start_utc_hour": 13.5, "duration_days": 60}, + "injection": {"so2_mass_kg": 2500.0, "plume_length_m": 30000.0}, + "background": {"aerosol": "cesm_g6", "so2_pptv": 100.0, "gas_pptv": {"O3": 1.2e6}}, + "dilution": {"regime": "constant", "rate_per_s": 2.0e-6, "zero_species": ("SO2",)}, + "microphysics": {"n_bins": 160, "condensation_alpha": 0.5, "ion_pair_rate": 0.0}, + "chemistry": {"photolysis": "sza", "so2_ho2_rate": 1e-16}, + "numerics": {"output_dt_s": 1200.0, "couple_dt_s": 300.0}, + "switches": {"aerosol_to_j": True}, # heating_to_t is refused (schema 0.2.0) + "termination": {"max_wall_time_s": 60.0, "max_sim_time_days": 5.0}, + } + ) + restored = RunConfig.model_validate_json(original.model_dump_json()) + assert restored == original + assert config_hash(restored) == config_hash(original) + assert config_hash(restored) != config_hash(RunConfig()) + + +@pytest.mark.tier_a +def test_unknown_fields_are_rejected() -> None: + """A typo'd key is a config that does not describe the run; accepting it silently is worse.""" + payload = RunConfig().model_dump() + payload["site"]["temprature_k"] = 210.0 + with pytest.raises(ValueError, match="temprature_k"): + RunConfig.model_validate(payload) + + +@pytest.mark.tier_a +def test_configs_are_immutable() -> None: + """Identity is a hash of the content, so content that can change under it is a bug (ADR-004).""" + config = RunConfig() + with pytest.raises(ValueError, match="frozen"): + config.site.temperature_k = 250.0 # type: ignore[misc] + + +@pytest.mark.tier_a +def test_background_evolves_accepts_only_false() -> None: + """SCIENCE-5: the model's background is static, so True must fail rather than be ignored.""" + payload = RunConfig().model_dump() + payload["dilution"]["background_evolves"] = True + with pytest.raises(ValueError, match="background_evolves"): + RunConfig.model_validate(payload) + + +@pytest.mark.tier_a +def test_the_temperature_feedback_cannot_be_enabled() -> None: + """Decision (Ali, 2026-08-13): no temperature feedback while longwave radiation is missing. + + The model's heating term is shortwave-only, so enabling it does not make the thermodynamics + more complete -- it makes them one-sided, and the resulting ~+1.2 K / 10 d drift is an artefact + of the absent cooling. Refused outright rather than defaulted off, so it cannot be turned on by + a form, a YAML file or a sweep axis without the schema changing first. + """ + payload = RunConfig().model_dump() + payload["switches"]["heating_to_t"] = True + with pytest.raises(ValueError, match="heating_to_t"): + RunConfig.model_validate(payload) + assert RunConfig().switches.heating_to_t is False + + +@pytest.mark.tier_a +def test_a_sweep_cannot_enable_the_temperature_feedback_either() -> None: + """The axis path is the one that would slip past a UI-level guard.""" + from studio.schema import Axis, RunSet + + runset = RunSet(axes=(Axis.over("heating", "switches.heating_to_t", {"on": True}),)) + with pytest.raises(ValueError, match="heating_to_t"): + runset.expand() + + +@pytest.mark.tier_a +def test_bin_count_is_restricted_to_the_grids_the_model_has() -> None: + """40/80/160 are the only TOMAS grids; 100 must fail here, not inside tomas_bridge.""" + payload = RunConfig().model_dump() + payload["microphysics"]["n_bins"] = 100 + with pytest.raises(ValueError, match="n_bins"): + RunConfig.model_validate(payload) diff --git a/studio/tests/unit/test_schema_metadata.py b/studio/tests/unit/test_schema_metadata.py new file mode 100644 index 0000000..18605db --- /dev/null +++ b/studio/tests/unit/test_schema_metadata.py @@ -0,0 +1,263 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Field metadata is a guarantee, not an intention. + +The PR checklist says "new/changed schema fields carry unit, range, description, default, +provenance". A checklist is a request for attention; this module makes it a build failure, so a +field added in six months without a recorded source cannot merge. + +The provenance rules themselves are enforced inside ``SciField`` at import time. What is tested here +is that they FIRE -- a validation rule nobody has ever seen reject anything is a rule you do not +know works. +""" + +from __future__ import annotations + +import pytest +from pint import UndefinedUnitError, UnitRegistry + +from studio.schema import ( + PINT_EXPRESSION, + Provenance, + RunConfig, + SciField, + Unit, + field_catalogue, + iter_leaf_fields, +) +from studio.schema.fields import field_metadata + +CATALOGUE = field_catalogue() + +#: Units carrying a convention pint cannot express. Kept here, spelled out, rather than as "anything +#: mapping to None" -- so that mapping a unit to None BY MISTAKE fails this test instead of joining +#: an exemption list silently. See studio/schema/units.py for why each one is here. +NON_PINT_UNITS = {Unit.PPMV, Unit.PPTV, Unit.CM3_PER_MOLEC_PER_S, Unit.COUNT} + + +@pytest.mark.tier_a +def test_every_leaf_field_has_complete_metadata() -> None: + """Unit, description and provenance on every field; no exceptions, no exemption list.""" + incomplete = {} + for path, info in iter_leaf_fields(): + meta = field_metadata(info) + missing = [key for key in ("unit", "provenance") if not meta.get(key)] + if not info.description: + missing.append("description") + if missing: + incomplete[path] = missing + assert incomplete == {}, ( + f"fields missing metadata: {incomplete}. Every schema field is declared with SciField, " + f"which requires unit, description and provenance -- see studio/schema/fields.py." + ) + + +@pytest.mark.tier_a +def test_declared_units_are_in_the_canonical_registry() -> None: + """A unit string outside the registry is a typo; the registry is closed on purpose (ADR-003).""" + known = {unit.value for unit in Unit} + unknown = {path: meta["unit"] for path, meta in CATALOGUE.items() if meta["unit"] not in known} + assert unknown == {}, f"units outside the canonical registry: {unknown}; known: {sorted(known)}" + + +@pytest.mark.tier_a +def test_pint_parses_every_unit_that_claims_to_be_parseable() -> None: + """The registry's pint expressions must actually parse, and the exemptions must be deliberate. + + pint is a declared dependency precisely so display conversion is possible; an expression that + does not parse would only be discovered at the presentation boundary, in front of a user. + """ + registry = UnitRegistry() + assert set(PINT_EXPRESSION) == set(Unit), ( + "every Unit member needs an entry in PINT_EXPRESSION (a pint expression, or None with a " + "stated reason); missing: " + f"{sorted(u.value for u in set(Unit) - set(PINT_EXPRESSION))}" + ) + for unit, expression in PINT_EXPRESSION.items(): + if expression is None: + assert unit in NON_PINT_UNITS, ( + f"{unit.value} maps to None but is not one of the documented non-pint units " + f"{sorted(u.value for u in NON_PINT_UNITS)}. If it genuinely cannot be expressed, " + f"say why in units.py and add it there." + ) + continue + try: + registry.Unit(expression) + except UndefinedUnitError as exc: # pragma: no cover -- the failure path is the point + pytest.fail(f"pint cannot parse {expression!r} for unit {unit.value!r}: {exc}") + + +@pytest.mark.tier_a +def test_sourced_provenance_carries_a_source() -> None: + """MODEL_DEFAULT and PAPER_ENSEMBLE mean "traceable"; without a source they mean nothing.""" + unsourced = { + path: meta["provenance"] + for path, meta in CATALOGUE.items() + if meta["provenance"] in {Provenance.MODEL_DEFAULT.value, Provenance.PAPER_ENSEMBLE.value} + and not meta.get("source") + } + assert unsourced == {}, f"defaults claiming a source but not giving one: {unsourced}" + + +@pytest.mark.tier_a +def test_derived_fields_declare_their_inputs_and_stay_unresolved() -> None: + """Derived fields are declarations, not computations (task 0.3 resolves them, 0.5 derives them). + + A derived field arriving with a value would mean physics happened in the schema layer, which is + the single thing studio/CLAUDE.md is most emphatic about. + """ + derived = {p: m for p, m in CATALOGUE.items() if m["provenance"] == Provenance.DERIVED.value} + assert derived, "expected at least the V0 and initial-concentration derivations to be declared" + for path, meta in derived.items(): + assert meta["derived_from"], f"{path} is DERIVED but declares no inputs" + assert meta.get("default") is None, ( + f"{path} is DERIVED but ships a value ({meta['default']!r}); it must stay unresolved " + f"until the dependency-graph engine computes it" + ) + for source_path in meta["derived_from"]: + assert source_path in CATALOGUE, ( + f"{path} derives from {source_path!r}, which is not a field. The dependency graph " + f"in task 0.3 is built from these paths, so a stale one silently drops an edge." + ) + + +@pytest.mark.tier_a +def test_non_derived_fields_declare_no_inputs() -> None: + """``derived_from`` on a primary field would put a phantom edge in the task-0.3 DAG.""" + strays = { + path: meta["derived_from"] + for path, meta in CATALOGUE.items() + if meta["provenance"] != Provenance.DERIVED.value and meta["derived_from"] + } + assert strays == {}, f"non-derived fields declaring derived_from: {strays}" + + +@pytest.mark.tier_a +def test_bounded_quantities_declare_their_range() -> None: + """Physical quantities that cannot take any float must say so, and the bound must be enforced. + + Checked against a hand-listed set rather than "all floats": some quantities genuinely are + unbounded, and a test that demanded bounds everywhere would be satisfied by meaningless ones. + """ + must_be_bounded = { + "site.latitude_deg", + "site.longitude_deg", + "site.temperature_k", + "site.pressure_mbar", + "site.h2o_ppmv", + "schedule.day_of_year", + "schedule.start_utc_hour", + "schedule.duration_days", + "injection.so2_mass_kg", + "microphysics.condensation_alpha", + "microphysics.nucleation_rate_scale", + "microphysics.ion_pair_rate", + "chemistry.so2_ho2_rate", + "numerics.output_dt_s", + "numerics.couple_dt_s", + "termination.max_wall_time_s", + } + unbounded = {path for path in must_be_bounded if not CATALOGUE[path].get("range")} + assert unbounded == set(), f"quantities with no declared range: {sorted(unbounded)}" + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("path", "value"), + [ + ("site.latitude_deg", 91.0), + ("site.temperature_k", 0.0), + ("site.pressure_mbar", -1.0), + ("schedule.day_of_year", 367), + ("schedule.start_utc_hour", 24.0), + ("microphysics.condensation_alpha", 1.5), + ("microphysics.nucleation_rate_scale", -1.0), + ("chemistry.so2_ho2_rate", -1e-18), + ], +) +def test_declared_ranges_are_actually_enforced(path: str, value: float) -> None: + """A declared range that pydantic does not enforce is documentation, not validation.""" + group, field = path.split(".") + base = RunConfig() + payload = base.model_dump() + payload[group][field] = value + with pytest.raises(ValueError, match=field): + RunConfig.model_validate(payload) + + +@pytest.mark.tier_a +def test_caveats_survive_into_the_catalogue() -> None: + """A caveat exists to reach the user; losing it in export would defeat the point. + + The heating switch is the case that matters most: shortwave-only heating produces a one-sided + warm drift, and enabling it without that warning is how someone reports a temperature trend as + a result. + """ + assert "one-sided" in CATALOGUE["switches.heating_to_t"]["caveat"] + assert "UPPER LIMIT" in CATALOGUE["chemistry.so2_ho2_rate"]["caveat"] + assert "spun-up" in CATALOGUE["background.gas_pptv"]["caveat"].lower() + + +@pytest.mark.tier_a +class TestSciFieldRejectsInconsistentMetadata: + """The import-time rules in ``SciField``, exercised. Each of these once looked reasonable.""" + + def test_literature_without_citation(self) -> None: + with pytest.raises(ValueError, match="requires `cite`"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.LITERATURE, + default=1.0, + ) + + def test_model_default_without_source(self) -> None: + with pytest.raises(ValueError, match="requires `source`"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.MODEL_DEFAULT, + default=1.0, + ) + + def test_derived_without_inputs(self) -> None: + with pytest.raises(ValueError, match="requires `derived_from`"): + SciField(unit=Unit.KELVIN, description="x", provenance=Provenance.DERIVED) + + def test_derived_with_a_hand_written_default(self) -> None: + with pytest.raises(ValueError, match="must not carry a hand-written default"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.DERIVED, + derived_from=["site.temperature_k"], + default=210.0, + ) + + def test_derived_from_on_a_primary_field(self) -> None: + with pytest.raises(ValueError, match="only meaningful with provenance=DERIVED"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.CONVENTION, + derived_from=["site.temperature_k"], + default=1.0, + ) + + def test_user_required_with_a_default(self) -> None: + with pytest.raises(ValueError, match="must not have one"): + SciField( + unit=Unit.KELVIN, + description="x", + provenance=Provenance.USER_REQUIRED, + default=1.0, + ) + + def test_blank_description(self) -> None: + with pytest.raises(ValueError, match="description is required"): + SciField( + unit=Unit.KELVIN, + description=" ", + provenance=Provenance.CONVENTION, + default=1.0, + ) diff --git a/studio/tests/unit/test_science_air.py b/studio/tests/unit/test_science_air.py new file mode 100644 index 0000000..30fc8fd --- /dev/null +++ b/studio/tests/unit/test_science_air.py @@ -0,0 +1,101 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""``air_number_density`` is a mirror of the model's. This is what makes that acceptable. + +``studio.science`` may not import the model (ADR-001), so this one relation is duplicated. A +duplicate is only safe if divergence is detectable, so the test below imports the model's own +implementation and asserts EXACT agreement across the parameter range the runs actually use. + +It skips -- rather than fails -- when the ``stratchem-jax`` submodule is not checked out, because +its absence says nothing about the code under test. CI does not check out the private submodules +(see ``.github/workflows/studio-ci.yml``), so in CI this is a skip and locally it is a real check. +That asymmetry is deliberate but worth knowing: a divergence would be caught on a developer machine, +not by CI. +""" + +from __future__ import annotations + +import subprocess +import sys +import textwrap +from pathlib import Path + +import pytest + +from studio.science import air_number_density + +#: The T-p corners the ensemble and the flagship runs use: 55 hPa / 210 K and 213 K (30N 20 km), +#: 120 hPa / 210 K (60N 15 km), 55 hPa / 215 K (the D1 clean run). +RUN_CONDITIONS = [(55.0, 210.0), (55.0, 213.0), (120.0, 210.0), (55.0, 215.0)] + + +@pytest.mark.tier_a +def test_known_value_at_the_golden_site() -> None: + """55 hPa, 210 K -> 1.8956916099773243e18 molec cm^-3. + + Tolerance 1e-12 relative: the value is one multiply-divide chain on exact constants, so the only + admissible difference is float64 rounding. Quoted to full precision on purpose -- a rounded + literal here would have passed while hiding a wrong constant. + """ + assert air_number_density(55.0, 210.0) == pytest.approx(1.8956916099773243e18, rel=1e-12) + + +@pytest.mark.tier_a +def test_scales_as_the_ideal_gas_law() -> None: + """Linear in pressure, inverse in temperature -- the property, not just a value.""" + assert air_number_density(110.0, 210.0) == pytest.approx( + 2.0 * air_number_density(55.0, 210.0), rel=1e-15 + ) + assert air_number_density(55.0, 420.0) == pytest.approx( + 0.5 * air_number_density(55.0, 210.0), rel=1e-15 + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize(("pressure", "temperature"), [(0.0, 210.0), (-1.0, 210.0)]) +def test_non_physical_pressure_raises(pressure: float, temperature: float) -> None: + with pytest.raises(ValueError, match="pressure must be > 0"): + air_number_density(pressure, temperature) + + +@pytest.mark.tier_a +@pytest.mark.parametrize(("pressure", "temperature"), [(55.0, 0.0), (55.0, -3.0)]) +def test_non_physical_temperature_raises(pressure: float, temperature: float) -> None: + """Zero kelvin would be a division by zero; ``inf`` is not a useful answer (ADR-005).""" + with pytest.raises(ValueError, match="temperature must be > 0"): + air_number_density(pressure, temperature) + + +@pytest.mark.tier_a +def test_agrees_exactly_with_the_models_own_implementation(repo_root: Path) -> None: + """The mirror check. Tolerance: exact -- same formula and constants, so any difference is real. + + Runs in a subprocess with ``stratchem-jax`` on ``sys.path``: importing the model's flat + ``config`` module in-process would leave a ``config`` in ``sys.modules`` for every test that + follows, which is exactly the kind of cross-test contamination the import-boundary test warns + about. + """ + stratchem = repo_root / "stratchem-jax" + if not (stratchem / "config.py").is_file(): + pytest.skip( + f"stratchem-jax submodule not checked out at {stratchem} " + f"(`git submodule update --init`); the mirror check needs the model's own version" + ) + probe = textwrap.dedent(f""" + import sys + sys.path.insert(0, {str(stratchem)!r}) + from config import air_number_density as model_impl + print("\\n".join(repr(model_impl(p, t)) for p, t in {RUN_CONDITIONS!r})) + """) + proc = subprocess.run( + [sys.executable, "-c", probe], capture_output=True, text=True, cwd=str(repo_root) + ) + assert proc.returncode == 0, f"could not evaluate the model's implementation:\n{proc.stderr}" + + model_values = [float(line) for line in proc.stdout.split()] + studio_values = [air_number_density(p, t) for p, t in RUN_CONDITIONS] + assert model_values == studio_values, ( + "studio.science.air_number_density has diverged from stratchem-jax/config.py:55. " + "It is a deliberate mirror (studio.science may not import the model); if the model's " + "relation changed, change this one in the same commit and say so." + ) diff --git a/studio/tests/unit/test_science_gcr.py b/studio/tests/unit/test_science_gcr.py new file mode 100644 index 0000000..66609f5 --- /dev/null +++ b/studio/tests/unit/test_science_gcr.py @@ -0,0 +1,67 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The GCR module refuses to compute, and these tests hold it to that. + +An unimplemented derivation is only safe if it STAYS unimplemented until someone supplies a +citation. The obvious future failure is a well-meaning change that makes +``ion_pair_production_rate`` return "something reasonable" so a form stops erroring -- these tests +are what would fail in that PR, with the reason attached. +""" + +from __future__ import annotations + +import pytest + +from studio.science import ( + MODEL_DEFAULT_ION_PAIR_RATE, + PAPER_ENSEMBLE_ION_PAIR_RATE, + ion_pair_production_rate, +) + + +@pytest.mark.tier_a +def test_the_parameterisation_refuses_to_guess() -> None: + """No citation, no number (ADR-005). The error must say why and where it is tracked.""" + with pytest.raises(NotImplementedError) as excinfo: + ion_pair_production_rate(altitude_km=20.0, latitude_deg=30.0, solar_cycle_phase=0.5) + message = str(excinfo.value) + assert "SCIENCE-6" in message + assert "#63" in message + assert "PAPER_ENSEMBLE_ION_PAIR_RATE" in message, ( + "the error must point at the constant that DOES have provenance, or the next person " + "will invent a value rather than find it" + ) + + +@pytest.mark.tier_a +def test_it_refuses_for_every_input_including_the_ensembles_own_conditions() -> None: + """~20 km / 30N is exactly where the uncited 30.0 came from, and it is not special-cased. + + Returning the known value at the known point and raising elsewhere would be the most tempting + version of this mistake: it would look like a working function with gaps. + """ + for altitude, latitude, phase in [(20.0, 30.0, 0.0), (15.0, 60.0, 1.0), (20.0, 30.0, 0.5)]: + with pytest.raises(NotImplementedError): + ion_pair_production_rate( + altitude_km=altitude, latitude_deg=latitude, solar_cycle_phase=phase + ) + + +@pytest.mark.tier_a +def test_the_two_documented_constants_are_what_the_code_uses() -> None: + """The ensemble's value and the model's default, both recorded, neither silently preferred.""" + assert PAPER_ENSEMBLE_ION_PAIR_RATE == 30.0 + assert MODEL_DEFAULT_ION_PAIR_RATE == 0.0 + + +@pytest.mark.tier_a +def test_the_schema_default_matches_the_ensemble_constant() -> None: + """One value, two places: the schema default and this constant must not drift apart. + + ``microphysics.ion_pair_rate`` defaults to the ensemble's 30.0 (ASSUMPTION-5) rather than the + model's 0.0, and that choice is only defensible while both sides agree on what the ensemble + used. + """ + from studio.schema import RunConfig + + assert RunConfig().microphysics.ion_pair_rate == PAPER_ENSEMBLE_ION_PAIR_RATE diff --git a/studio/tests/unit/test_science_plume.py b/studio/tests/unit/test_science_plume.py new file mode 100644 index 0000000..7e29530 --- /dev/null +++ b/studio/tests/unit/test_science_plume.py @@ -0,0 +1,158 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""The injected-mass -> initial-concentration chain, against the numbers the ensemble actually ran. + +Tolerance for the golden values: **exact**. These are not measurements being approximated, they are +the same arithmetic on the same constants, and the ensemble's own code hard-codes one of them +(``run_60day.py:37``) -- so anything other than bit-equality means Studio's chain differs from the +model's, which is precisely what this task exists to prevent. +""" + +from __future__ import annotations + +import pytest + +from studio.science import ( + SO2_MOLAR_MASS_G_PER_MOL, + air_number_density, + initial_mixing_ratio_pptv, + injected_number_density, + number_density_to_pptv, + plume_volume_cm3, + pptv_to_number_density, +) + +#: ``run_ensemble.py:45`` -- 10 m x 10 m x 15 km. +ENSEMBLE_V0_CM3 = 1.5e12 + +#: ``run_ensemble.py:46`` computes this, and ``run_60day.py:37`` hard-codes it. Verified +#: bit-identical between the two, which is why it can be asserted exactly. +ENSEMBLE_SO2_NUMBER_DENSITY = 6.273063291666667e15 + +#: 1 tonne, the ensemble's release (``TABLE_microphysics_parameters.md``). +ENSEMBLE_SO2_MASS_KG = 1000.0 + + +@pytest.mark.tier_a +def test_plume_volume_matches_the_ensemble_geometry() -> None: + """10 m x 10 m x 15 km == 1.5e12 cm^3, exactly.""" + assert plume_volume_cm3(15000.0, 10.0, 10.0) == ENSEMBLE_V0_CM3 + + +@pytest.mark.tier_a +def test_injected_number_density_reproduces_the_ensemble_exactly() -> None: + """The number the whole 810-run ensemble was initialised with.""" + assert ( + injected_number_density(ENSEMBLE_SO2_MASS_KG, SO2_MOLAR_MASS_G_PER_MOL, ENSEMBLE_V0_CM3) + == ENSEMBLE_SO2_NUMBER_DENSITY + ) + + +@pytest.mark.tier_a +def test_the_full_chain_reproduces_the_golden_cases_initial_so2() -> None: + """Mass -> number density -> pptv at the golden case's 210 K / 55 hPa. + + What the ensemble fixes is the number DENSITY, so the mixing ratio is the derived quantity and + differs between sites for the same injected mass. That is asserted below, because it is the + part people get backwards. + """ + pptv = initial_mixing_ratio_pptv( + mass_kg=ENSEMBLE_SO2_MASS_KG, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=ENSEMBLE_V0_CM3, + pressure_mbar=55.0, + temperature_k=210.0, + ) + expected = ENSEMBLE_SO2_NUMBER_DENSITY / air_number_density(55.0, 210.0) * 1e12 + assert pptv == expected + assert pptv == pytest.approx(3.309115922996412e9, rel=1e-15) + + +@pytest.mark.tier_a +def test_the_same_mass_gives_a_different_mixing_ratio_at_a_different_site() -> None: + """60N / 15 km (120 hPa) is denser air, so 1 t of SO2 is a SMALLER mixing ratio there.""" + at_20km = initial_mixing_ratio_pptv( + mass_kg=ENSEMBLE_SO2_MASS_KG, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=ENSEMBLE_V0_CM3, + pressure_mbar=55.0, + temperature_k=210.0, + ) + at_15km = initial_mixing_ratio_pptv( + mass_kg=ENSEMBLE_SO2_MASS_KG, + molar_mass_g_per_mol=SO2_MOLAR_MASS_G_PER_MOL, + volume_cm3=ENSEMBLE_V0_CM3, + pressure_mbar=120.0, + temperature_k=210.0, + ) + assert at_15km < at_20km + assert at_20km / at_15km == pytest.approx(120.0 / 55.0, rel=1e-12) + + +@pytest.mark.tier_a +def test_the_two_track_lengths_in_the_repository_differ_by_exactly_two() -> None: + """The one divergence that matters, pinned as a fact rather than left as a memory. + + ``run_ensemble.py:45`` uses a 15 km track and ``coupled/run_dilution_d1_clean.py:61`` a 30 km + one. Same injected mass, half the concentration. Which is correct depends on SCIENCE-2 (#54). + """ + ensemble = plume_volume_cm3(15000.0, 10.0, 10.0) + d1_flagship = plume_volume_cm3(30000.0, 10.0, 10.0) + assert d1_flagship == 2.0 * ensemble + assert injected_number_density(1000.0, SO2_MOLAR_MASS_G_PER_MOL, d1_flagship) == 0.5 * ( + injected_number_density(1000.0, SO2_MOLAR_MASS_G_PER_MOL, ensemble) + ) + + +@pytest.mark.tier_a +def test_pptv_round_trips_through_number_density() -> None: + """Both directions are real workflows: the D1 run specifies pptv and computes the mass back.""" + m_air = air_number_density(55.0, 215.0) + original = 2.9e9 # coupled/run_dilution_d1_clean.py:69 + assert pptv_to_number_density(original, m_air) / m_air * 1e12 == pytest.approx( + original, rel=1e-15 + ) + assert number_density_to_pptv(pptv_to_number_density(original, m_air), m_air) == pytest.approx( + original, rel=1e-15 + ) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("kwargs", "message"), + [ + ({"length_m": 0.0, "width_m": 10.0, "height_m": 10.0}, "length must be > 0"), + ({"length_m": 15000.0, "width_m": -1.0, "height_m": 10.0}, "width must be > 0"), + ({"length_m": 15000.0, "width_m": 10.0, "height_m": 0.0}, "height must be > 0"), + ], +) +def test_degenerate_geometry_raises(kwargs: dict[str, float], message: str) -> None: + """A zero-volume plume is a division by zero one step later; catch it where it is meaningful.""" + with pytest.raises(ValueError, match=message): + plume_volume_cm3(**kwargs) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("mass", "molar_mass", "volume", "message"), + [ + (0.0, 64.0, 1.5e12, "mass must be > 0"), + (-1.0, 64.0, 1.5e12, "mass must be > 0"), + (1000.0, 0.0, 1.5e12, "molar mass must be > 0"), + (1000.0, 64.0, 0.0, "volume must be > 0"), + ], +) +def test_injection_inputs_are_validated( + mass: float, molar_mass: float, volume: float, message: str +) -> None: + with pytest.raises(ValueError, match=message): + injected_number_density(mass, molar_mass, volume) + + +@pytest.mark.tier_a +def test_conversions_reject_a_non_physical_air_density() -> None: + """No default, no fallback: an air density of zero has no meaningful mixing ratio (ADR-005).""" + with pytest.raises(ValueError, match="air number density must be > 0"): + number_density_to_pptv(1e15, 0.0) + with pytest.raises(ValueError, match="air number density must be > 0"): + pptv_to_number_density(1e9, -1.0) diff --git a/studio/tests/unit/test_science_size_distribution.py b/studio/tests/unit/test_science_size_distribution.py new file mode 100644 index 0000000..bb3e898 --- /dev/null +++ b/studio/tests/unit/test_science_size_distribution.py @@ -0,0 +1,123 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Size-distribution reductions, and the claim that the repository's "two conventions" are one. + +The headline test here is ``test_the_repositorys_two_spellings_are_the_same_quantity``: it measures +the difference between the two forms in the code rather than asserting they are equivalent on paper, +because "these are the same" is the kind of statement that is easy to believe and expensive to be +wrong about. +""" + +from __future__ import annotations + +import numpy as np +import pytest + +from studio.science import bin_midpoints_um, dlog10_dp, dn_dlogdp + +#: A TOMAS-like 80-bin grid: dry Dp from 1.7 nm to 17.5 um, geometric (the real grid's mass ratio is +#: 2**(40/n_bins), which is geometric in diameter too). Built here rather than imported from the +#: model because Tier A must not pay a JAX import; the property under test is grid-shape-independent +#: and is also checked on a deliberately irregular grid below. +EDGES_UM = np.geomspace(1.7e-3, 17.5, 81) + + +@pytest.mark.tier_a +def test_the_repositorys_two_spellings_are_the_same_quantity() -> None: + """``10**(0.5*(log a + log b))`` vs ``sqrt(a*b)``, and ``log b - log a`` vs ``log(b/a)``. + + Tolerance 1e-14 relative, against a measured ~7e-16 (mid-point) and ~5e-15 (dlog10Dp) on this + grid: a few ULP of float64, i.e. rounding, not a modelling difference. The plan's note that the + repository carries "two different mid-point expressions" is a misreading of two spellings of one + expression, and this is the evidence for saying so. + """ + log_edges = np.log10(EDGES_UM) + run_ensemble_mid = 10 ** (0.5 * (log_edges[:-1] + log_edges[1:])) # run_ensemble.py:148 + run_ensemble_dlog = log_edges[1:] - log_edges[:-1] # run_ensemble.py:148 + + np.testing.assert_allclose(bin_midpoints_um(EDGES_UM), run_ensemble_mid, rtol=1e-14, atol=0.0) + np.testing.assert_allclose(dlog10_dp(EDGES_UM), run_ensemble_dlog, rtol=1e-14, atol=0.0) + + +@pytest.mark.tier_a +def test_equivalence_holds_on_an_irregular_grid_too() -> None: + """Not an artefact of a perfectly geometric grid: same check where bin widths vary wildly.""" + edges = np.array([1e-3, 2e-3, 5e-3, 1e-2, 3e-1, 1.0, 17.5]) + log_edges = np.log10(edges) + np.testing.assert_allclose( + bin_midpoints_um(edges), 10 ** (0.5 * (log_edges[:-1] + log_edges[1:])), rtol=1e-14 + ) + np.testing.assert_allclose(dlog10_dp(edges), log_edges[1:] - log_edges[:-1], rtol=1e-14) + + +@pytest.mark.tier_a +def test_midpoint_is_geometric_not_arithmetic() -> None: + """On a log axis the arithmetic mean is not the centre, and the difference is not small. + + For a bin spanning a decade the two differ by ~28 %, which would be visible as a shifted mode + diameter in every size-distribution figure. + """ + edges = np.array([0.1, 1.0]) + assert bin_midpoints_um(edges)[0] == pytest.approx(np.sqrt(0.1), rel=1e-15) + arithmetic = 0.55 + assert abs(bin_midpoints_um(edges)[0] - arithmetic) / arithmetic > 0.25 + + +@pytest.mark.tier_a +def test_dn_dlogdp_normalises_by_bin_width() -> None: + """The defining property: equal counts in unequal bins are NOT an equal dN/dlogDp.""" + edges = np.array([1.0, 10.0, 1000.0]) # widths 1 and 2 in log10 + result = dn_dlogdp(np.array([100.0, 100.0]), edges) + np.testing.assert_allclose(result, [100.0, 50.0], rtol=1e-15) + + +@pytest.mark.tier_a +def test_dn_dlogdp_handles_a_time_series() -> None: + """``state.npz`` stores ``n_cm3`` as (n_times, n_bins); the size axis is last.""" + counts = np.tile(np.linspace(1.0, 80.0, 80), (5, 1)) + result = dn_dlogdp(counts, EDGES_UM) + assert result.shape == (5, 80) + np.testing.assert_allclose(result[0], counts[0] / dlog10_dp(EDGES_UM), rtol=1e-15) + + +@pytest.mark.tier_a +def test_integrating_dn_dlogdp_recovers_the_total_number() -> None: + """sum(dN/dlogDp * dlogDp) == sum(N) -- the conservation check behind the normalisation.""" + counts = np.linspace(0.5, 40.0, 80) + recovered = float(np.sum(dn_dlogdp(counts, EDGES_UM) * dlog10_dp(EDGES_UM))) + assert recovered == pytest.approx(float(counts.sum()), rel=1e-14) + + +@pytest.mark.tier_a +def test_off_by_one_in_the_edge_count_raises() -> None: + """The mistake worth catching: n edges for n bins broadcasts into a plausible wrong answer.""" + with pytest.raises(ValueError, match="edges must have exactly one more element"): + dn_dlogdp(np.ones(80), EDGES_UM[:-1]) + with pytest.raises(ValueError, match="edges must have exactly one more element"): + dn_dlogdp(np.ones(79), EDGES_UM) + + +@pytest.mark.tier_a +@pytest.mark.parametrize( + ("edges", "message"), + [ + (np.array([1.0]), "at least 2 bin edges"), + (np.array([[1.0, 2.0], [3.0, 4.0]]), "must be 1-D"), + (np.array([0.0, 1.0]), "must be > 0 um"), + (np.array([-1.0, 1.0]), "must be > 0 um"), + (np.array([1.0, 0.5, 2.0]), "strictly increasing"), + (np.array([1.0, 1.0]), "strictly increasing"), + (np.array([1.0, np.nan]), "must all be finite"), + ], +) +def test_malformed_edges_raise(edges: np.ndarray, message: str) -> None: + """A logarithmic grid has preconditions; violating them silently yields NaN, not an error.""" + with pytest.raises(ValueError, match=message): + bin_midpoints_um(edges) + + +@pytest.mark.tier_a +def test_negative_counts_raise() -> None: + """A negative number concentration is a corrupted input, not a small one.""" + with pytest.raises(ValueError, match="must be >= 0"): + dn_dlogdp(np.array([1.0, -1.0]), np.array([1.0, 2.0, 3.0])) diff --git a/studio/tests/unit/test_store.py b/studio/tests/unit/test_store.py new file mode 100644 index 0000000..e53193c --- /dev/null +++ b/studio/tests/unit/test_store.py @@ -0,0 +1,323 @@ +# Copyright (C) 2026 University Corporation for Atmospheric Research +# SPDX-License-Identifier: Apache-2.0 +"""Persistence: the schema, the immutability rules, and the model/migration agreement. + +Every test upgrades a real SQLite database **through the migrations**, never via +``Base.metadata.create_all``. Creating tables straight from the models would test a schema no +migration ever produced, leaving the migrations to be exercised for the first time on someone's real +database -- which is the failure Alembic-from-the-first-migration exists to prevent. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from sqlalchemy import inspect + +from studio.resolve import apply_change, resolve, set_override +from studio.schema import RunConfig +from studio.store import ( + LocalDirectoryStore, + artifact_for, + create_db_engine, + create_run, + create_run_set, + current_revision, + ensure_config, + record_artifact, + record_job, + record_summary, + record_transition, + runs_for_config, + session_factory, + session_scope, + sha256_of, + upgrade_to_head, +) + +EXPECTED_TABLES = { + "run_set", + "run", + "run_config", + "job", + "job_transition", + "result_artifact", + "dataset_version", + "run_dataset", + "run_summary", +} + + +@pytest.fixture +def database(tmp_path: Path): + """A migrated SQLite database and its session factory.""" + url = f"sqlite:///{tmp_path / 'studio.db'}" + upgrade_to_head(url) + engine = create_db_engine(url) + yield engine, session_factory(engine), url + engine.dispose() + + +@pytest.fixture +def store(tmp_path: Path) -> LocalDirectoryStore: + return LocalDirectoryStore(tmp_path / "artifacts") + + +@pytest.mark.tier_a +def test_migrating_creates_every_table(database) -> None: + engine, _, _ = database + assert EXPECTED_TABLES <= set(inspect(engine).get_table_names()) + assert current_revision(engine) is not None + + +@pytest.mark.tier_a +def test_upgrading_twice_is_idempotent(database) -> None: + """Startup must be able to call this unconditionally.""" + engine, _, url = database + before = current_revision(engine) + upgrade_to_head(url) + assert current_revision(engine) == before + + +@pytest.mark.tier_a +def test_the_models_and_the_migration_agree(database) -> None: + """The drift test: autogenerate against a migrated database must find nothing to do. + + Without it, a column added to ``models.py`` without a migration works on every developer machine + (where the table was created from the models by some other path) and fails on the first real + deployment. Here it fails immediately, in seconds. + """ + from alembic.autogenerate import compare_metadata + from alembic.migration import MigrationContext + + engine, _, _ = database + from studio.store.models import Base + + with engine.connect() as connection: + context = MigrationContext.configure(connection, opts={"compare_type": True}) + diff = compare_metadata(context, Base.metadata) + assert diff == [], ( + f"models.py and the migrations have drifted: {diff}. Generate a migration " + f"(`alembic -c studio/store/alembic.ini revision --autogenerate`) rather than editing the " + f"models alone." + ) + + +@pytest.mark.tier_a +def test_the_same_config_is_one_row(database) -> None: + """Identity is the hash (ADR-006): CLI and API submissions converge rather than duplicating.""" + _, factory, _ = database + config = resolve(RunConfig()) + with session_scope(factory) as session: + first = ensure_config(session, config) + second = ensure_config(session, resolve(RunConfig())) + assert first.config_hash == second.config_hash + assert first is second + + +@pytest.mark.tier_a +def test_an_edited_config_is_a_different_row(database) -> None: + """Configs are immutable: an edit produces a new config, never a mutation (ADR-004).""" + _, factory, _ = database + original = resolve(RunConfig()) + edited = apply_change(original, "site.temperature_k", 213.0) + with session_scope(factory) as session: + first = ensure_config(session, original) + second = ensure_config(session, edited) + assert first.config_hash != second.config_hash + assert first.resolved_config["site"]["temperature_k"] == 210.0 + assert second.resolved_config["site"]["temperature_k"] == 213.0 + + +@pytest.mark.tier_a +def test_the_repository_offers_no_way_to_update_a_config() -> None: + """Immutability is structural, not a convention someone has to remember.""" + import studio.store.repository as repository + + forbidden = [ + name + for name in dir(repository) + if any(word in name.lower() for word in ("update", "delete", "overwrite")) + and not name.startswith("_") + ] + assert forbidden == [], f"repository exposes mutation helpers: {forbidden}" + + +@pytest.mark.tier_a +def test_a_stale_config_is_refused(database) -> None: + """Persisting one would give an inconsistent set of numbers a permanent identity.""" + from studio.resolve import InconsistentConfigError + + _, factory, _ = database + stale = apply_change( + set_override(resolve(RunConfig()), "injection.so2_initial_pptv", 5.0e9), + "site.temperature_k", + 213.0, + ) + with pytest.raises(InconsistentConfigError), session_scope(factory) as session: + ensure_config(session, stale) + + +@pytest.mark.tier_a +def test_a_run_records_its_provenance_and_reproducibility(database) -> None: + """``reproducible`` describes the moment the run started, so it is stored, not recomputed.""" + _, factory, _ = database + provenance = { + "sandbox": {"commit": "a" * 40, "dirty": True}, + "submodules": {"tuvx-jax": {"commit": "b" * 40, "dirty": False}}, + } + with session_scope(factory) as session: + run_set = create_run_set(session, label="sweep") + run = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + label="case", + provenance=provenance, + ) + assert run.reproducible is False + assert run.provenance["sandbox"]["commit"] == "a" * 40 + + clean = create_run( + session, + run_set=run_set, + config=resolve(RunConfig()), + provenance={"sandbox": {"dirty": False}, "submodules": {}}, + ) + assert clean.reproducible is True + + +@pytest.mark.tier_a +def test_job_transitions_are_appended_not_replaced(database) -> None: + """The transition list is the record; the state column is a convenience.""" + _, factory, _ = database + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + job = record_job(session, run=run, state="queued", work_dir="/tmp/x") + record_transition(session, job=job, state="running", detail="launched") + record_transition(session, job=job, state="failed", detail="exit 1", exit_code=1) + + assert job.state == "failed" + assert job.exit_code == 1 + assert [t.state for t in job.transitions] == ["queued", "running", "failed"] + assert all(t.at.tzinfo is not None for t in job.transitions), "timestamps must be aware" + + +@pytest.mark.tier_a +def test_an_artifact_stores_a_pointer_and_a_checksum(database, store, tmp_path: Path) -> None: + """The bytes stay on disk; the database gets a path, a size and a hash.""" + _, factory, _ = database + source = tmp_path / "state.npz" + source.write_bytes(b"not really an npz, but bytes are bytes") + + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + row = record_artifact(session, run=run, kind="state", source=source, store=store) + run_id = run.id + + assert row.sha256 == sha256_of(source) + assert row.size_bytes == source.stat().st_size + assert not Path(row.path).is_absolute(), "paths are relative to the store root" + assert store.verify(row.path, row.sha256) + with session_scope(factory) as session: + assert artifact_for(session, run_id, "state").sha256 == row.sha256 + + +@pytest.mark.tier_a +def test_a_missing_artifact_raises_rather_than_recording_an_absence(database, store) -> None: + """A run that was supposed to produce an artefact and did not is a failure to surface.""" + _, factory, _ = database + with pytest.raises(FileNotFoundError), session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + record_artifact( + session, run=run, kind="state", source=Path("/nonexistent.npz"), store=store + ) + + +@pytest.mark.tier_a +def test_corruption_is_detectable(database, store, tmp_path: Path) -> None: + """Why the checksum is stored: silent corruption and a tidied directory look the same.""" + _, factory, _ = database + source = tmp_path / "summary.json" + source.write_text('{"schema_version": "0.1.0"}', encoding="utf-8") + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + row = record_artifact(session, run=run, kind="summary", source=source, store=store) + stored = store.open_path(row.path) + key, digest = row.path, row.sha256 + + assert store.verify(key, digest) + stored.write_text('{"schema_version": "tampered"}', encoding="utf-8") + assert not store.verify(key, digest) + + +@pytest.mark.tier_a +def test_a_summary_promotes_headline_scalars_for_querying(database) -> None: + """So "every run where peak number exceeded X" is a query, not 810 deserialisations.""" + _, factory, _ = database + summary = { + "schema_version": "0.1.0", + "termination": "completed", + "flags": ["open_system_dilution"], + "series": { + "SO2": {"values": [3.3e9, 1.7e6]}, + "H2SO4": {"values": [0.0, 15.05, 2.1]}, + "total_n": {"values": [0.0, 3.07e6, 1.2e6]}, + "SA": {"values": [2.0, 41.3]}, + }, + } + with session_scope(factory) as session: + run_set = create_run_set(session) + run = create_run(session, run_set=run_set, config=resolve(RunConfig())) + row = record_summary(session, run=run, summary=summary) + + assert row.final_so2_pptv == 1.7e6 + assert row.peak_h2so4_pptv == 15.05 + assert row.peak_number_cm3 == 3.07e6 + assert row.final_surface_area == 41.3 + assert row.termination == "completed" + assert row.summary["series"]["SO2"]["values"][-1] == row.final_so2_pptv, "column and JSON agree" + + +@pytest.mark.tier_a +def test_runs_for_a_config_are_the_cache_lookup(database) -> None: + """Identical hash is necessary but not sufficient to reuse a result, so this returns runs. + + The caller still has to compare the model version in each run's provenance -- which is why this + is not ``cached_result_for()`` returning a verdict. + """ + _, factory, _ = database + config = resolve(RunConfig()) + with session_scope(factory) as session: + run_set = create_run_set(session) + create_run(session, run_set=run_set, config=config, label="first") + create_run(session, run_set=run_set, config=config, label="second") + found = runs_for_config(session, config.config.config_hash()) + assert {run.label for run in found} == {"first", "second"} + assert len({run.id for run in found}) == 2, "same config, two distinct runs" + + +@pytest.mark.tier_a +def test_foreign_keys_are_enforced_on_sqlite(database) -> None: + """Without the pragma, SQLite ignores them and Postgres does not -- silent divergence.""" + from sqlalchemy.exc import IntegrityError + + from studio.store.models import JobRow + + _, factory, _ = database + with pytest.raises(IntegrityError), session_scope(factory) as session: + session.add( + JobRow( + id="j", + run_id="does-not-exist", + state="queued", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ) + )