Skip to content

A run does not describe itself: results/ holds 27 files named by version, with no config, seed, conditions or git stamp — two runs of one version overwrite each other #6

Description

@dim-sous

What is wrong

results/ today (27 files, 33 MB, gitignored):

v5_regulation_activation_results.npz
v5_regulation_activation_results.png
v5_regulation_activation_scalars.json      (444 bytes)
v5_regulation_activation_stress_tests.png
v5_comparison.json
phase1_milp_smoke.png  …

Every name identifies a version, not a run. Nothing on disk records the config that produced a file, the seed, the horizon, the strategy, or the checkout it came from. Two runs of the same version silently overwrite each other. A figure cannot be traced to its numbers; the numbers cannot be traced to their conditions.

So results/ is generated but not reconstitutable — the exact folder kind Wilson et al. say must be deletable and regenerable, and it is neither.

Why this is the highest-leverage item

It is the single precondition for all three problems this repo has:

  • reproducibility — a number with no recorded conditions is not a finding (RCWS [A7]: "a run at one named condition is that condition, not a capability");
  • comparison — two runs can only be compared if each carries what it was run with;
  • a Lab — the RCWS Lab is 1,300 lines of stdlib with no logic of its own because every record already carries its config, metrics, provenance and drawable panels. Built over today's results/, a Lab has nothing to show.

What it should be

RCWS's record layer, transcribed:

  • One record per run, written by the harness at the end (and, later, on failure): kind, name, config (the whole experiment config from run_simulation takes 15 positional parameters and 9 config dataclasses from one file, with no runtime schema — every caller is welded to the core's shape #5), metrics, provenance (git sha, date, duration, dirty flag), and any drawable series it declares.
  • Records are the currency. Every figure and every printed number reads a record; the table and the plot cannot disagree (RCWS #47).
  • A record declares its own panels (Series), so a new quantity gains a figure with no renderer edit.
  • One store, one drift guard (a record whose class has moved on still lists), one records view of what the project has run.
  • Naming: a figure's stem is the target that drew it, enforced at the write chokepoint — the invariant that lets a view map figures to runs with no table.

Done when

  • run_simulation (or its wrapper) writes a record; nothing else writes to out/.
  • Existing plots read records instead of live arrays.
  • sim records (or equivalent) lists every record with kind, date, sha.
  • Proof: the record for the canonical synthetic day round-trips — same config in, same metrics out.

Phase 2 of the platform epic. Depends on #5 (the config it records) and #1 (where it lives).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    engineseams, run schema, data model

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions