studio: Phase 0 → main (nine tasks, 269 tests, exit criteria met) - #88
Open
aliakherati wants to merge 19 commits into
Open
studio: Phase 0 → main (nine tasks, 269 tests, exit criteria met)#88aliakherati wants to merge 19 commits into
aliakherati wants to merge 19 commits into
Conversation
Studio tasks branch off studio/dev and PR into it; studio/dev merges into main at phase boundaries. main sees Studio in reviewed batches while the model and viz work continues there untouched. CI runs on pushes to studio/dev as well as main, so a merge is verified and not only the PR that preceded it -- a PR is tested against its own head, and a stale one can go green and still break the branch it lands on. Amends the "trunk-based" line in studio/CLAUDE.md rather than leaving the documented workflow disagreeing with the actual one, and records the two things that cost time while landing #59: gh pr create defaults to the repository default branch, so --base studio/dev must be explicit; and a conflicted PR is not merely blocked but silently untested, because GitHub cannot build the merge ref and so runs no workflow at all. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
) studio: schema v0 -- SciField, RunConfig, RunSet and the config hash Task 0.2 (#61), the review gate. studio/schema/ in seven modules: a closed canonical unit registry, the SciField metadata carrier, the enums, RunConfig and its ten groups, RunSet with GRID/ZIP/LIST axes, canonical JSON + SHA-256, and the JSON Schema export. 41 leaf fields, 61 Tier-A tests. Provenance is required, and its rules are enforced when the module is imported rather than when a run produces a quietly wrong number: MODEL_DEFAULT and PAPER_ENSEMBLE must cite a source, LITERATURE must cite a reference, DERIVED must declare its inputs and must not ship a value. A default whose origin nobody recorded cannot be declared at all. Defaults are the paper ensemble's rather than the model's where the two differ (ASSUMPTION-5) -- ion_pair_rate is the visible case, 30 cm^-3 s^-1 against the model's 0.0, which would disable ion-induced nucleation entirely. Both are recorded on the field. RunConfig() with no arguments is therefore the golden case, which is the form's opening state and the base of every RunSet. Identity carries only what changes the result: no label, notes or output_dir field, so two runs differing in name are one computation. The hash is pinned by a test and checked across four PYTHONHASHSEEDs in fresh interpreters, because a hash that drifts silently invalidates every cache and fixture at once and reads as a performance problem. RunSet reproduces the 810-run ensemble -- same count, same order, same case IDs, checked at the golden case. That is why LIST exists: the site axis covaries latitude, T, p and H2O, and its cross product is not physically meaningful. Model validation is mirrored where it is cheap (DT/dt_couple divisibility, the 40/80/160 bin grids, background_evolves accepting only false), each mirror citing the model line it copies. ADR-002's third motivating problem was that a form cannot learn what is valid without importing most of the model; this answers it. No physics: plume_volume_cm3 and so2_initial_pptv are declared with derived_from and left unresolved for tasks 0.3 and 0.5. Two divergences from the plan are recorded in PROGRESS.md rather than absorbed: max_sim_time is optional, and DilutionRegime.CONSTANT is spelled "constant" where the model spells it "". Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…GCR (#65) studio: science derivations -- plume, size distribution, air density, GCR Task 0.5 (#64), taken before 0.3 so the dependency-graph engine has real derivations to resolve. studio/science/ in five modules, 40 new Tier-A tests (101 total). Consolidation first, new code second -- and the consolidation corrected two claims the plan made from memory. Both are now fixed in plan/PHASE_0.md: There are SIX copies of the V0 / initial-concentration derivation, not five (the missed one is make_rf_runs.py:44), and run_dilution_d1_clean.py lives at coupled/, not coupled/paper_ensemble/. The divergence between them is real and consequential: the ensemble uses a 15 km track and the D1 flagship a 30 km one, so the same injected mass gives half the concentration. Which is right depends on SCIENCE-2 (#54). Separately, run_60day.py:37's hard-coded 6.273063291666667e15 is bit-identical to what run_ensemble.py:46 computes -- a frozen copy, not a variant. The "two different mid-point expressions" are one expression. 10**(0.5*(log a + log b)) and sqrt(a*b) are algebraically identical, as are log b - log a and log(b/a); measured agreement on an 80-bin grid is to a few ULP (<= 7e-16 relative). The test measures this rather than asserting it, because "there are two conventions in the code" is the kind of belief people work around. air_number_density is a MIRROR of the model's, not a fork: studio.science may not import the model (ADR-001), so the test runs the model's own implementation in a subprocess and asserts exact agreement at the four T-p corners the runs use. Note the asymmetry -- CI does not check out the private submodules, so that check skips there and only really runs on a developer machine. gcr.py computes nothing. ion_pair_production_rate raises NotImplementedError naming SCIENCE-6 (#63), and PAPER_ENSEMBLE_ION_PAIR_RATE = 30.0 carries its provenance instead. A test asserts it refuses even at ~20 km / 30N, where the uncited 30.0 came from: returning the known value at the known point and raising elsewhere would look like a working function with gaps. Two molar masses are deliberately the model's rounded values (SO2 64.0 vs 64.066, H2SO4 98.0 vs 98.079), recorded as such in constants.py along with the ~0.036% Avogadro seam at the gas/TOMAS boundary. Studio inherits the model's constants so Phase 0 can reproduce its runs; a silent correction would make a Studio bug indistinguishable from a model change. Studio's Python floor is now stated as 3.12 in ruff, black and mypy: numpy's bundled stubs use 3.12-only type statements, so mypy --strict could not check this package against 3.11 at all. The lockfile and CI were already 3.12 and no dependency changed. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 0.3 (#66). New package studio/resolve: graph.py (the DAG built from derived_from), registry.py (which function computes which field), resolver.py (resolution, overrides, staleness). 47 new Tier-A tests, 148 total. A fourth pure package rather than a module inside studio/schema, because schema is data and stays free of computation while science is computation and stays free of the config model; resolution is the composition of the two, and naming it keeps that layering visible. It joins schema and science in the import-boundary test and under mypy --strict: the API resolves on every keystroke, so a JAX import on that path would be unaffordable. Staleness is defined against a fingerprint. Setting an override records the upstream values at that moment, and the field is stale when they no longer match. That makes staleness a property of the config alone -- no edit history, no ordering assumptions -- and it is what lets the UI present the old value, the newly-derived value and what changed between them rather than a bare warning. accept_derived and keep_override are the two ways out, both explicit; a kept override re-anchors and goes stale again on the next change rather than being silenced for good. require_consistent() raises on any stale field and the stale list survives serialisation, so a persisted config cannot lose the fact that it is inconsistent. The trap that guards: a stale config still has a hash, which would be a stable identity for numbers that do not follow from each other. The load-bearing test captures every field before and after an edit and asserts the set that moved is exactly the edited field plus its downstream closure. Both directions fail silently otherwise -- too little leaves a stale number that reaches the model, too much discards a value the user chose -- and it covers three fields with no dependents, where the expected change set is the edited field alone. The registry is checked against the schema rather than trusted: every DERIVED field has a derivation, no derivation exists for a field the schema does not derive, and declared inputs equal derived_from exactly. Cycles raise at construction rather than iterating to a fixed point or breaking an arbitrary edge, both of which would make the result depend on where the engine started. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 0.4 (#68). studio/modelio: scenario.py converts RunConfig to CoupledScenario, and summary.py reduces a state.npz to a versioned RunSummary. 23 new Tier-A tests, 171 total. The equivalence test passes: to_scenario(resolve(RunConfig())) is field-for-field identical to run_ensemble.build_scenario() for the golden case, compared as dataclasses.asdict with exact equality on every field including the floats. The schema is a faithful superset of what the ensemble ran, proven before any run rather than discovered from a diverging result. The derived SO2 initial concentration matches to the last bit, which is the evidence that consolidating that derivation in 0.5 changed nothing. to_scenario is deliberately not re-exported from the package. Measured: importing studio.modelio.scenario costs ~0.13 s and pulls in no JAX; the first CALL costs ~1.05 s, because CoupledScenario.__post_init__ imports tomas_bridge to validate background_dist (coupled_scenario.py:196). A comparison view reading a RunSummary should not pay for a model it is not using, and the expensive import should be visible at the import site. RunSummary encodes the traps rather than documenting them: every series declares whether it is on a wet or dry basis, species are indexed by name from the npz's own species list, and the time axis is the stored t. The synthetic test archive orders species so SO2 sits at index 2 -- nothing like the 32/34/35 an existing script hard-codes -- so a positional reduction would report ozone as SO2, plausibly and silently. Termination is an argument, never inferred: the npz records what the state did, not why the loop stopped. The conservation check refuses to report a number when one would mislead. With dilution on the box is an open system, so a large residual would be measuring the dilution and a small one would mean something was wrong; the check returns not_applicable with the reason and the start/end values, and a closed box gets a real residual. Also fixed: a sum() over a generator starting at integer 0 that mypy caught; mypy reporting errors from the model's own untyped source once modelio imported it (follow_imports = "silent" on the model modules -- Studio's use is still checked); and a test of mine that inspected sys.modules in-process, which is the exact mistake test_import_boundaries.py warns about and now runs in a fresh interpreter. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* studio: the runner, the job lifecycle and the first end-to-end run Task 0.6 (#72). studio/runner (base.py, local.py), studio/modelio/execute.py and studio/cli/run.py. 20 new Tier-A tests, 191 total. The first end-to-end run works: `python -m studio.cli.run input.json outdir` runs the real model and writes state.npz plus summary.json. Verified on a 1-day/40-bin case in 21 s -- SO2 3.309e9 -> 1.72e6 pptv, peak H2SO4 15.05 pptv, peak number 3.07e6 cm^-3, npz key set identical to the canonical one at run_ensemble.py:150-156, termination "completed" with a real config_hash. A Studio-created run therefore has the provenance the archived ensemble lacks. The lifecycle is data and its transitions are enforced. Every transition is timestamped and kept, because "it failed" is not debuggable and "QUEUED 14:02:11, RUNNING 14:02:11, FAILED 14:06:48 exit 1" is. An illegal transition raises: a job that appears to move backwards means the runner lost track of a process, and accepting it silently would turn the record from a log into a story. TERMINATED_ON_LIMIT is kept distinct from FAILED all the way through -- one means the model could not produce a result, the other that it was still going when we stopped it, and the second leaves partial output that can look complete. A failed run is debuggable without re-running it: the resolved input is written at submit rather than at completion, so a job that dies immediately still has it; stdout and stderr are captured in full, stdout being the log stream because run_coupled prints rather than logs; the exit code is recorded. Exit codes distinguish "never started" (2, bad input, rejected before the model is imported so it costs milliseconds) from "the model raised" (1). entry_module is a parameter rather than a test hook -- the runner launches a module by name, nothing branches on the value, the default is the real path, and the real path is exercised by the exit-code test. Slurm and cloud-batch raise instead of falling back to local execution, because a job running somewhere other than where it was sent is worse than an error. execute.py reuses coupled.dilution.volume_ratio and studio.science's size-distribution reduction rather than inlining a fifth copy, and enforces max_sim_time through a *args stop-condition that works either side of task 0.8's widening of that callback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * studio: the stop-condition shape the model will actually accept PR #73 widens run_coupled's stop_condition to a diagnostics dict and dispatches on the callback's DECLARED arity: two positional parameters is accepted as legacy, one is the new dict shape, and *args raises TypeError because it matches both. Raising on *args is right -- guessing which shape a variadic callback wanted would be a silent wrong answer -- but it means the "works with either" spelling I had reached for is the one thing that does not work. So the callback is now two-positional, which the model accepts both before and after that change, and three tests pin the arity so the coupling is visible rather than latent. Without them the break would have been silent: nothing today sets max_sim_time, so the path is unexercised until someone does. Migrating to the dict shape is worth doing once #73 is in studio/dev, since that is what makes termination criteria on SO2 or particle number possible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decision by Ali: longwave radiation is not in the radiative calculation, so there must be no temperature feedback in the model. The reasoning matters for how it is encoded. The model's heating term is shortwave-only (AD-5.4), so enabling it does not make the box thermodynamics more complete -- it makes them one-sided, and the ~+1.2 K / 10 d warm drift it produces is an artefact of the missing cooling rather than a physical response. A default of False would leave that a switch someone can flip; Literal[False] makes True fail validation, which is the same treatment dilution.background_evolves already had for the same class of reason. Two tests cover it: the direct one, and the sweep-axis path, which is the one that would slip past a guard implemented in the UI. SCHEMA_VERSION goes 0.1.0 -> 0.2.0 and the pinned config hash moves with it. The VALUE of heating_to_t did not change -- it was already False -- but schema_version is part of the hashed payload, and that is precisely what makes "old configs are never silently reinterpreted under new semantics" true rather than stated. A config written before this commit no longer hashes to a 0.2.0 identity, which is intended. Recorded where a reader of results would look: CAVEATS.md gains a top-level entry saying every run is isothermal at the configured temperature and must not be read as containing a plume-warming signal, and SCIENCE-4 is marked partly answered -- buoyant rise, sedimentation and the isobaric assumption remain open. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(studio): measure the archived-ensemble reproduction tolerance (#70) ADR-009's first golden-harness task is a measurement, not an assertion. Two archived cases re-run at today's submodule SHAs and compared per quantity against the archived state.npz: the golden case (index 121, 30N_20km__sabr220__D2med__a1p0__nuc1__cg1) and a contrast on two axes (index 67, 30N_20km__sabr330__burst__a1p0__nuc1__cg1 -- burst dilution, loaded background). Result: reproduction is CLOSE BUT NOT BIT-FOR-BIT. Every headline quantity agrees to <= 2.1e-12 and the worst deviation anywhere is 3.4e-12, but only ~31% of gas state-vector elements and ~1% of aerosol samples are exactly equal -- an atol=0 golden test would have failed on arrival. Two controls fix the interpretation: running the same case twice today is bit-identical across all 18 stored arrays (so the residual is toolchain drift, not run-to-run noise), and the worst deviations scatter across days 1.3-9.8 rather than accumulating. Also recorded: unguarded relative error over the raw gas state vector peaks at 4.2e+04, entirely on night-time O1D/O at O(1e-35) molec/cm3 oscillating about zero. Golden tests must floor by series magnitude. No assertions written in this pass, by design. Tolerances proposed for the Tier-B assertions, each with its rationale, in REFERENCE_TOLERANCES.md. ASSUMPTION-2 marked settled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * docs(studio): record the measured non-uniform time grid in the tolerance file The archived t spans 0-864000 s in 1461 samples with 41 distinct step sizes (0.56-600 s, mean 591.78 s), confirming the CAVEATS warning against reconstructing time as i x DT rather than taking it on trust. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…run (#76) studio: correct the dp_mid_um tolerance row -- not exact for a Studio run #74 proposed asserting dp_mid_um exact. That holds only when the fresh run comes from run_ensemble. A run produced by Studio differs in 44 of 80 bins by up to 8.1e-16, because task 0.5 deliberately adopted sqrt(a*b) where run_ensemble writes 10**(0.5*(log10 a + log10 b)) -- the two spellings 0.5 proved algebraically identical and measured a few ULP apart. Asserting exact equality would therefore pass against the old pipeline and fail against every run Studio itself produces, presenting as a physics regression over a rounding difference. Corrected to 1e-15; t, V_ratio and T stay exact. dNdlogDp inherits the difference at 3.2e-13, inside its own 1e-10, so no other row moved. Found by re-running the golden case through studio.cli.run as an independent check of the measurement. Every other number reproduced to the digit; both runs are now tabulated in the record, which also makes the point that a tolerance measured through one pipeline is not automatically a tolerance for another. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…squash (#77) studio: heating and buoyancy are out of scope, not pending Ali's follow-up decision: ignore buoyancy and anything heating-related, because this model cannot answer those questions -- simulating the feedback of heating needs a different model. That is a scope boundary rather than an undecided item, and the two are recorded differently on purpose. Longwave radiation is absent from the radiative calculation, so the heating term is shortwave-only and enabling it makes the thermodynamics one-sided rather than more complete. Buoyant rise follows from a heating rate this model cannot compute, so a rise velocity would be a free parameter dressed as physics. Answering either needs a model with longwave radiation and plume dynamics. The interface consequence: no buoyancy or heating-rate fields are added to the schema at all. A field for a capability the model does not have would advertise it, and the absence is the honest interface (ADR-005). SCIENCE-4 is answered for heating and buoyancy, and `numerics.box_thermodynamics.*` moves in the capability register from "see SCIENCE-4" to "not exposed, by decision". Sedimentation is deliberately left open. It is a particle-loss process, not a thermodynamic response; "we decided not to model heating" is not an argument about gravitational settling, and folding it into this decision would have quietly closed a question nobody answered. CAVEATS.md now says every run is isobaric and isothermal at the configured temperature, and that a result must not be read as containing a plume-warming signal, a lofting signal, or an altitude change. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…iagnostics dict (#78) * fix(coupled): four contained fixes — JAX-free scenario validation, diagnostics-dict stop_condition, user lognormal backgrounds, drop output_dir (#73) * refactor(coupled): hoist BACKGROUND_MODES into a JAX-free module CoupledScenario.__post_init__ validated background_dist via a function-body `from coupled.tomas_bridge import BACKGROUND_MODES`. tomas_bridge imports jax (and sets jax_enable_x64) at module scope, so merely CONSTRUCTING a scenario cost ~1.0 s and left jax resident. An API that validates a form on every keystroke cannot pay that. The mode tables move verbatim to coupled/backgrounds.py -- pure data, no numpy/jax/scipy, no coupled sibling that imports them. tomas_bridge re-exports BACKGROUND_MODES / AMBIENT_BACKGROUNDS so paper scripts and docs referring to tomas_bridge.BACKGROUND_MODES keep working; it remains where the tables are USED. coupled_scenario imports them at module scope instead. Measured with a fresh interpreter, repo root as cwd: python -c "import time,sys; t0=time.perf_counter(); from coupled.coupled_scenario import CoupledScenario; t1=time.perf_counter(); CoupledScenario(); t2=time.perf_counter(); print(t1-t0, t2-t1, 'jax' in sys.modules)" before: import 0.007 s, first construct 0.99-1.19 s, jax loaded True after: import 0.010 s, first construct 0.000 s, jax loaded False The proof is a subprocess test: an in-process `"jax" not in sys.modules` assertion is vacuous, because the rest of the suite has already imported the driver by the time any single test runs. The import in coupled_scenario is absolute (`from coupled.backgrounds import`) rather than relative because that module is also imported FLAT as `coupled_scenario`, with coupled/ on sys.path, by coupled/conftest.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(coupled): stop_condition receives a diagnostics dict The early-stop callback was called with only (t1_seconds, wet_SA), so a termination criterion could not reference SO2 or particle number (docs/studio/OPEN_QUESTIONS.md, termination.criteria[]). It now receives one dict, evaluated on the end-of-interval state: t, interval, T, SA, radius_cm, h2so4wp, particulate_S, N_total, gas `gas` is {species_name: molec/cm^3} for all 34 species, keyed by NAME so a state-vector reordering cannot silently misread it. The aerosol entries are the WET quantities the heterogeneous chemistry sees and are NaN -- never 0 -- when TOMAS is inactive, so a `SA < x` criterion cannot read "no aerosol model" as "the plume has relaxed". BOTH shapes are supported, dispatched on the callback's DECLARED arity (inspect.signature) once at run_coupled entry rather than by attempting a call and catching TypeError: one positional parameter -> the dict form, two -> the legacy (t1, wet_SA) form with a DeprecationWarning. Zero, three-plus, or a bare *args raises TypeError, because *args is compatible with both shapes and guessing would pass a dict where a float was expected. Resolving at entry means a mis-shaped callback fails immediately instead of minutes into a run. The legacy shape is kept because coupled/paper_ensemble/README.md documents it and out-of-tree run scripts use it. The two in-repo callers (run_60day.py, run_bgstop.py) are migrated to the dict form; the criterion is unchanged (same diag["SA"] / diag["t"] values as the arguments they were passed before). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(coupled): accept user-defined lognormal background modes _seed_lognormal already took arbitrary (N, Dg, sigma_g) tuples, but background_dist was restricted to the six named mode sets plus "redcircles". It now also accepts a mode list, seeded through the identical code path -- a test asserts a user list equal to a named set's own modes yields a BIT-IDENTICAL initial state. Validation lives in coupled/backgrounds.normalize_modes (still JAX-free, so scenario validation stays cheap): N > 0, Dg > 0, sigma_g > 1, exactly three numeric entries per mode, at least one mode. sigma_g == 1 is rejected specifically because log10(sigma_g) = 0 divides by zero inside the dN/dlogDp evaluation; every one of these would otherwise seed a silently degenerate background rather than raise. Lists (as YAML/JSON return them) canonicalize to a tuple of float 3-tuples, so a scenario round-trips. New field `background_modes_basis` ("stp" | "ambient"), REQUIRED for a mode list and REJECTED for a name. It is not defaulted: a bare mode list carries no number basis, the named sets carry theirs via AMBIENT_BACKGROUNDS, and the STP->ambient factor is ~0.09 at 68 mbar / 210 K -- so silently picking one is an order-of-magnitude error in the background number concentration, not a rounding difference. A test pins that factor to the two branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(coupled)!: remove the dead CoupledScenario.output_dir field The driver never read it. `run_coupled` returns arrays and writes nothing -- every caller (paper_ensemble/run_*.py, run_dilution_d1_clean.py) computes its own path and calls np.savez itself. REMOVED rather than honoured, deliberately. Honouring it would give the driver a filesystem side effect it does not have today, and would create two competing sources of truth for where a run's results live: the field, and the path the caller passes to np.savez. Studio's runner (task 0.6) owns run directories and must be the only such source. A config field that nothing reads is worse than no field, because it looks authoritative. Loading an archived config that still carries the key raises with an explanation naming the removal and what to do instead, rather than the generic "Unknown CoupledScenario keys" -- that config was valid when it was written. The archived paper runs are unaffected: they store only state.npz, no config. coupled/scenarios/coupled_default.yaml drops the key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * chore(studio): merge main into studio/dev; stop condition takes the diagnostics dict studio/dev now has #73's four model fixes. Studio's suite (196 Tier-A tests, equivalence tests included) passes against the changed model unmodified -- output_dir is gone and nothing missed it, which is the evidence that removing it was safe rather than tidy. Conflict resolutions, stated because the reasoning matters more than the diff: test_import_boundaries.py takes studio/dev's structure (three clean packages including studio.resolve, and the seam check that allows submodules of studio.modelio) with main's wording, since the __post_init__ pattern it describes is now past tense -- #73 fixed it. studio/__init__.py merged cleanly and the merge exposed a stale docstring of mine, which still claimed two clean packages after 0.3 added a third; fixed here, where it became visible. The stop condition moves to the diagnostics dict. #73 deprecated the two-argument form, so leaving it would have Studio emitting a DeprecationWarning on a normal path. Only diag["t"] is read today, but the dict also carries SA, N_total and every gas species by name, which is what makes the spec's SO2- or number-based termination criteria possible. max_sim_time enforcement is now proven end to end rather than only unit-tested: a 1-day request with max_sim_time_days = 0.5, run through studio.cli.run with -W error::DeprecationWarning, stops at t = 0.500 d and is labelled TERMINATED_ON_LIMIT with the stopped_on_limit flag. That path had never run before -- nothing set max_sim_time -- and it also confirms 0.6's termination inference, so partial output cannot be read as converged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* studio: the two-tier golden harness (task 0.7) The assertions, on the tolerances #70 measured and #76 corrected. studio/tests/golden: tolerances.py, paper_cases.py, make_fixture.py, one test module per tier. 17 new Tier-A tests (214 total) plus 3 Tier-B tests. Tier A is a real 1-day/40-bin run in 19 s against a committed 40 kB fixture -- the cheapest run that still exercises gas chemistry, TUV-x photolysis, all three microphysics processes and dilution. The fixture is a UNIFORM-stride reduction (every 4th sample plus the last) because a coarsening grid aliases the morning number spike by up to 8x, and a fixture built on one would encode the aliasing and assert it forever. It catches drift in Studio's own pipeline, which is a different claim from reproducing the archive. Tier B reproduces six curated 10-day cases from the archive in ~28 min, nightly or manual: D1/D2/D3/burst against both backgrounds, all cg1 because REFERENCE_TOLERANCES.md records that cg0p5/cg2 may straddle the tomas-jax commit that wired coag_kernel_scale through. The tolerances live in one module, each citing the measurement it came from, and the failure messages say re-measure rather than widen -- a tolerance widened to make a test pass is a test that no longer tests anything. The near-zero floor lives in the comparison rather than in each test, because unguarded relative error reaches 4.24e+04 on night-time O1D at 1e-35 molec/cm3. Photolysis is compared per reaction rather than summed, since a compensating pair of errors across two reactions survives a total; J was added to the fixture for it. Tier B reports every deviation before failing rather than being parametrised per case, because after 28 minutes the whole table beats the first failure and shows whether a deviation is systematic. Both tiers carry a physical floor -- SO2 consumed, H2SO4 produced, particles formed, plume expanded -- because a tolerance test cannot tell that a run did nothing. Honest limitation: the plan calls Tier A "CI, seconds", but CI does not check out the private submodules, so the model cannot run there and this module skips in CI. Fixing that needs a deploy key and is its own change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * studio: apply the endpoint and series tolerances to the right things Tier B's first real run against the archive failed with three exceedances -- H2SO4 in D2med at 3.377e-12, SO3 and OH in D3high at ~6e-12, all against 1e-12. It was not a reproduction failure: 3.377e-12 is essentially the 3.38e-12 the measurement itself recorded for H2SO4 max-over-time. The harness was applying the ENDPOINT tolerance to whole-SERIES comparisons, and those are two different rows of REFERENCE_TOLERANCES.md -- 1e-12 for final and peak values, 1e-10 for a series maximum. No tolerance was widened; that is what this harness's own failure messages forbid. The two numbers the record already specifies now apply to the two things they describe, through a named assert_headline_matches so the call sites read like the record's rows and the conflation is hard to repeat. Tier A carried the same mistake -- invisible there, because it compares against its own fixture where the deviation is ~0 -- and is fixed too. Tier B now reports every deviation rather than only the exceedances: 27 minutes of compute should produce a measurement, not just a verdict. The D3high series maxima (~6e-12, comfortably inside 1e-10) are data the original two-case measurement did not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* studio: provenance records, written before the run (task 0.9a) 0.9 is not one PR -- the exit criteria need FastAPI, SQLAlchemy/Alembic, a CLI, React/Vite with SSE and a figure -- so it is split, and this is the piece nothing implemented and the exit criteria depend on: a stored result with full provenance (ADR-006). A record pins config_hash, studio.__version__, the SANDBOX SHA, all three submodule SHAs, whether each checkout was dirty and which paths, and the resolved post-derivation parameter set -- what the model actually received rather than what the user typed -- plus any override with the value in force. datasets is present and empty rather than omitted so its emptiness is never ambiguous. This is the one place Studio shells out to git and it is strict: not-a-checkout, git missing, a repo with no commits, or a missing submodule all raise, because an empty SHA in a provenance record is worse than no record -- it looks like an answer. Cleanliness comes from status --porcelain rather than diff --quiet, so an untracked file counts; an untracked module that a run imported is exactly what makes a SHA a lie. The runner writes it at submit, before the process starts, so a run that dies in minute three of four still says what produced it. The test asserts against the record submit() returns rather than after wait(), since checking afterwards would pass even if it were written at completion. Verified end to end. Three times here a test failed and the code was right, each time because my expectation of git was wrong: a nested repo is not a registered submodule (the parent sees it as untracked and reads dirty, so the fixture now uses git submodule add); a dirty submodule flags both it and the parent, which is git being helpful because an edited submodule then cannot hide behind a clean-looking SANDBOX; and local-path submodules need protocol.file.allow=always. The tests build real git repositories rather than mocking subprocess -- the module is a thin shell over git's behaviour, so a mocked git would test the mock. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR * studio: the runner records a checkout it is given, so its tests run in CI CI caught what local tests could not: making provenance mandatory at submit means the runner needs a pinnable checkout, and CI checks out no submodules, so every runner submit test failed there while passing locally. LocalSubprocessRunner now takes repo_root -- which checkout to record -- defaulting to the one the code came from. This does not weaken the guarantee: a run still cannot start unless the checkout it names can be pinned. And "which checkout produced this?" is a question a runner genuinely has to answer; a worker executing code from elsewhere would answer it differently. The synthetic-checkout builder moves from the provenance tests into conftest, since the runner tests need it too. Both suites now run in CI rather than skipping: 40 tests across the two, no submodules required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…laims (#82) Ali confirmed that coupled/paper_ensemble/runs/ is a valid reproduction reference. That is now written into REFERENCE_TOLERANCES.md rather than left implicit in the harness, because it is an assumption the data cannot support on its own: those files carry no provenance record, which is the gap ADR-006 closes going forward and cannot close retroactively. Every other archive directory is excluded and the record now says why. runs_60day initialises from a spun-up control run; runs_bgstop*, runs_boxsize, runs_geo, runs_no_sai, runs_special and runs_start_time* were produced with different vintages and configurations. Rebuilding one of those from the axis tables would compare two different computations, where a pass is luck and a failure means nothing. All six curated cases are now measured rather than two: every endpoint within 4.1e-14 against 1e-12, every series within 1.6e-11 against 1e-10, t/V_ratio/T bit-identical throughout. The wider data corrected two claims. The record read as though deviation were tied to the early nucleation burst, because the two-case measurement found the worst H2SO4 deviation at day 1.34; across six cases the worst days are 5.83, 1.34, 9.27, 2.15, 3.03 and 5.74, with no common feature -- it is round-off scattered through the run rather than accumulation. And the size-distribution outlier is about sparsity rather than timing: the four sabr220 cases share an identical 1.63e-11 at one early cell while the sabr330 cases peak late and elsewhere, but every one of them is a bin holding 1.7-4.2 particles per cm3. Adds measure_all_cases.py and plot_fidelity.py as tools rather than tests. Re-measuring must never "fail" -- it reports, and a human decides whether the numbers are acceptable. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rst migration (#84) studio: persistence -- models, artefact store, and Alembic from the first migration Task 0.9b (#83). studio/store: nine tables, an artefact store behind an interface, a repository with no update path, and migrations that are exercised by every test. Immutability is structural rather than conventional. run_config is keyed by the config's own hash and ensure_config is get-or-create; a test asserts the repository exposes no update, delete or overwrite helper at all. An edited config is a different row and a different run, linked by derived_from_run_id. The database stores pointers, never arrays. Artefacts go behind an ArtifactStore protocol -- a directory today, MinIO or S3 later without touching callers -- and the row keeps a relative path, size, content type and a SHA-256 computed on write. That checksum makes "still the file that was written" checkable, since silent corruption and a tidied directory look identical from the database otherwise. A missing artefact raises rather than being recorded as an absence. Two portability decisions, both because the backends would otherwise disagree silently. UtcDateTime, a TypeDecorator, because DateTime(timezone=True) returns an aware datetime on Postgres and a NAIVE one on SQLite -- caught by a test asserting tzinfo is not None, which failed on the first run; naive input now raises rather than being assigned a guessed zone. And foreign_keys=ON for SQLite, without which its foreign keys are ignored entirely and the constraints in models.py would be documentation on the Phase-0 backend while being enforced in production. The drift test runs Alembic's compare_metadata against a migrated database and requires an empty diff, so a column added without a migration fails in seconds rather than on the first real deployment. Nothing uses Base.metadata.create_all, including the tests: every test upgrades through the migrations, so they are exercised continuously rather than for the first time on someone's database. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
studio: the CLI -- run, sweep, status (task 0.9c) Half the exit criteria: a run can be submitted from the CLI, produces a stored result with full provenance, and is readable afterwards from a different process. Verified end to end at 19 s for a 1-day/40-bin case -- resolve, provenance, persist, submit, wait, six artefacts and the summary recorded, exit succeeded -- and then `status` from a separate process after the first had exited, printing the full transition trail, every artefact, and "reproducible NO -- a checkout was dirty", which is the honest answer for a tree with uncommitted work. The CLI is not a wrapper over the API (ADR-002): it goes through the same schema, resolver, store and runner, so a sweep launched from a terminal and one launched from the web produce identical rows and identical provenance. --plan mirrors run_ensemble.py's plan verb, because deciding to spend 810 x 4.6 minutes should take a second command; it prints what would run, with each case's hash, and creates no row. Two things fixed by looking at real output rather than at tests. The persisted trail was thinner than the runner's -- the first run recorded queued -> succeeded and dropped running -- which defeats the point of persisting a trail at all, since that hides how long a job waited for a worker and whether a failed one ever started; it now copies every transition and a test asserts the sequence. And session.get() returns None, which I was passing straight into the repository where it would have failed frames later as an AttributeError about None; mypy caught it, and it now raises naming the row and the key. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d, 0.9e) (#87) Phase 0's exit criteria are met: a run can be submitted from the CLI and from the web UI, produces a stored result with full provenance, and the golden tests pass. studio/service.py was extracted the moment there were two callers. A second copy of "resolve, record provenance, persist, submit, follow, store artefacts" would drift within a week and the drift would be invisible, since both paths would keep working and only their database rows would disagree. Verified against a real uvicorn server rather than only a test client: POST returns 202, the SSE stream reports running immediately and succeeded at t+20 s, and the run lists with reproducible false and termination completed. Three bugs came from looking at output rather than at green tests. The database never showed running, because transitions were written only after a job finished -- a trail, but useless as progress -- so finalise now follows the runner and persists each transition as it happens. reproducible came back as 0 rather than false, an Integer column where a Boolean belonged; the drift test caught it the moment the model changed and the second migration took one command. And SSE looked broken under TestClient, which serialises requests: the test now runs a real server in a thread and changes state while the stream is open, the only shape that catches a stream reporting nothing until the end. Deliberate divergence from ADR-007, recorded rather than silent: the page is one self-contained HTML file rather than React + Vite. A build toolchain for a single form is machinery ahead of need, the same reasoning that kept Redis and Docker out of Phase 0, and the repository already has precedent in coupled/viz/*.html. /api/schema exists so the form can be generated when the UI outgrows one form. Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # studio/__init__.py # studio/tests/unit/test_import_boundaries.py
Nine tasks, 269 Tier-A tests. Every exit criterion met, with the narrow ones named rather than rounded up: no React (one self-contained page instead), CI cannot see the submodules so three model-facing checks run only locally, and PROGRESS.md conflicts on almost every parallel PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR
Collaborator
Author
Tier B: green — the last exit-criterion row is now verifiedAll three archive tests ran: the curated-set coverage check, How it was run, because the first attempt was invalidI originally launched this in the main working tree and then checked out The run above was pinned:
Exit criteria
Ready to merge from my side. Two notes on ordering: #89 fixes a bug in the UI this PR ships, and #90 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 →
main. Nine tasks, 269 Tier-A tests, 43 typed source files.This is a merge PR, not new work — everything here has been reviewed in #59…#87. What is new is two conflict resolutions, the phase-completion record, and verification run on the merged result.
The exit criteria, and how narrowly each is met
plume-studio run config.yaml— 19 s for 1 day / 40 binsrunning→succeeded, verified against a real uvicorn serverstate.npz, versionedRunSummary, six artefacts with checksums, and an immutable record naming the SANDBOX commit + all three submodule SHAsThe predicted conflict, resolved as recorded
The merge hit exactly the conflict flagged in the #73 review: the two
studio/files that PR edited onmain, whichstudio/devhad since moved past. Taken as recorded then —studio/dev's structure (three clean packages, the seam-submodule check) withmain's corrected wording (the__post_init__pattern it describes was fixed by #73). Both sides asserted present programmatically, because two hand-resolves earlier in this phase silently dropped content.Verified on the merged result
269Studio Tier-A tests · the model's own132tests ·ruff·black·mypy --strict. Running the model's suite matters here specifically because this merge is where the app layer and #73's model changes finally sit in one tree.What is deliberately not done
/api/schemaexists so the form can be generated when the UI outgrows one form.air_number_densitymirror. This is the weakest point in the setup — a deploy key would fix it.PROGRESS.mdconflicted on six parallel PRs, once losing a commit to a squash-merge. One file per entry would end it.Five findings from this phase that changed code rather than docs
dp_mid_umis not bit-identical for a Studio-produced run, because 0.5 deliberately changed the spelling: a tolerance measured through one pipeline is not a tolerance for another.DateTime(timezone=True)returns naive datetimes on SQLite and aware ones on Postgres — right in production, wrong in development.After this merges
An annotated
v0.1.0-phase0tag frommain, per the workflow instudio/CLAUDE.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01EaDHnPdiacsybr8Tp5WnqR