Skip to content

Equivalence harness: step 0 of the control-plane plan - #398

Open
thacer13 wants to merge 6 commits into
mainfrom
equivalence-harness
Open

thacer13 wants to merge 6 commits into
mainfrom
equivalence-harness

Conversation

@thacer13

Copy link
Copy Markdown
Collaborator

Step 0 of roadmap/control-plane.md part 2 in the specs: a harness that shows whether a build behaves as another
one does, so each later step (the session crate, the interface crate, project data into the project) can be held to
"no change at all". Test code only; nothing changes in what Lemon Studio does.

What it does

It drives the app without a window the way a user does: clicks, double clicks, right clicks, drags, the wheel, keys and
modifiers, aimed at what the frame shows (only controls a click there reaches; a kind of control it has used less is
picked more, one never used yet much more). After every step it writes a fingerprint of the project, the project state
the interface still holds (tempo, pattern shown, rack groups and loops), what .lemon would store, the undo history,
whether the engine holds the project, and whether there is something to save. A trace recorded on one build replays on
another: the first step whose fingerprint differs names the input that behaves differently.

At the end of each trace it checks what needs no other build:

  • undo back to the start gives the project as opened, and redo gives the last one back;
  • the engine holds what the project says;
  • a saved copy opens as the project the save wrote, and saving that again writes the same.

What these report on purpose is listed with the reason in equiv_accepted.txt (each FL claim with its measurement).
The comparisons look through representation alone (list order, a clip's record bytes a save rewrites from its fields,
an empty pattern, a send at unity, bookkeeping).

Tools

All #[ignore] tests in crates/os-app/src/equiv.rs, run on a hidden display (the module's doc has the commands):

  • equiv_record and equiv_replay (each trace in a process of its own with its own home folder, where nothing the app
    starts can reach the desktop: Open folder, help links and the like run stand-ins that only note the call);
  • equiv_twice (record and replay on one build: what is not deterministic);
  • equiv_shrink (cuts a trace to the few steps that still break a check);
  • equiv_show (what each step lands on and changes; with OS_EQUIV_SHOW=checks the end checks, with =save a
    saved copy against the project);
  • equiv_coverage (which kinds of control the recordings showed, aimed at and changed something with).

Changes outside the harness

Clock and randomness reads go through the app's own clock (anim::now) and one seed (clip_colors::clock), so a
replay sees the same times and random colours; a project made new in a test gets a fixed start date. Test-only
accessors on the transport (what was last sent, how often).

The harness tested too

The plan's planted faults, each on a scratch build and replayed against a clean recording of 38 of the projects:

Fault Caught in By
A step click no longer marks the project changed for playback 21 of 38 traces the engine part, at that step
A new undo step stored before the newest one 38 of 38 the undo history, the undo and redo checks
A save leaves a channel's pan as the file had it 14 of 38 the save and reopen check
One engine update in three lost 38 of 38 the engine part

unsaved() is the comparison with the project as last saved, so the plan's check that it is true exactly then holds
by construction; each step's value is in the fingerprint. Ids are checked after every step: every channel, pattern,
mixer track, slot plugin, arrangement and clip has one, and no two share one.

Measured

On FL's 151 bundled projects (one 300-step trace each, about an hour on a hidden display): with the fixes of #397,
no finding off the accepted list, and a replay on the same build gives the same fingerprints for all 151; 132 kinds of
control changed something. On main as it is, 139 of the 151 traces report something (the fixes PR lists what).

🤖 Generated with Claude Code

thacer13 and others added 6 commits November 10, 2026 07:02
…lock, one seed for random colours and the randomizers

The drag, double-click, hold and tap timers read anim::now instead of Instant::now, and the random colours, the piano
roll's randomizer and the clip colours take their seed from clip_colors::clock; a project made new in a test that runs
the motions' clock starts at a fixed date. Nothing changes outside tests: anim::now is the time unless a test sets the
clock. For the equivalence harness (specs roadmap/control-plane.md part 2, step 0), which needs every run alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drives the app without a window the way a user does, on FL's bundled projects, and records each step's fingerprint (the
project, the project state the interface holds, what .lemon stores, the undo history, whether the engine holds the
project, whether there is something to save); a trace recorded on one build replays on another and names the first
input that behaves differently. At the end of each trace: undo to the start, redo to the end, the engine in step, a
saved copy opened as the project the save wrote, a second save the same, ids. What these report on purpose is listed
with its reason in equiv_accepted.txt. Tools: equiv_record, equiv_replay, equiv_twice, equiv_shrink, equiv_show,
equiv_coverage (all ignored tests; the module's doc has the commands). Test code only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… not the accepted list of the day

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…so moving that state into the project changes no fingerprint

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…part, as the fingerprint does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… after undo, until step 15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant