Skip to content

Diff step: batch the coverage runs of changed tests - #65

Merged
VirtualMachinist merged 11 commits into
mainfrom
m3/mutation-cost
Nov 20, 2026
Merged

VirtualMachinist merged 11 commits into
mainfrom
m3/mutation-cost

Conversation

@VirtualMachinist

Copy link
Copy Markdown
Member

This makes the gate's diff step cheaper without changing what it decides. The step runs the changed test files under coverage, at the head and at the merge base, to find the functions those tests cover.

What it does

Before: one Vitest-with-coverage run per changed test file at the head, and one per file at the merge base, one after another.

After:

  • The head versions run together in one run, and the merge-base copies in another, in groups of at most 100 files.
  • A group that yields nothing falls back to running file by file, as before. The file that can't run is still found and named, with the same failure reason.
  • Each merge-base copy is written beside its original, run, and removed.

What stays the same: the functions in scope, the mutate ranges, the mutants and the verdicts. The covered functions are now ordered by file and line.

Measured on the diff step alone, with the same tree and base:

Subject Before (wall) Before (CPU) After (wall) After (CPU)
A large probe diff (about 1,250 mutants) 60–88 s 102–124 s 21 s 86–88 s

The type checker was measured too

Narrowing the type checker's program to the mutated files was unsound, because 3 of 594 compile-error mutants turned into survivors. Narrowing it to the mutated files plus their importers was sound but saved nothing, and skipLibCheck saved nothing either. The checker is unchanged.

Declared exceptions

File Change
src/diff/{scope,coverage,functions}.ts and their tests The diff engine, which runs those coverage runs
WEB_TOPS.md §6.4 The wording on how the changed test files are run. This is a protocol-change, wording only
CHANGELOG.md, .web-tops/survivors.yaml A changelog line, and four equivalent-mutant entries, each with its reason

Tests pin that the batched path and the file-by-file path give the same functions, and that a failing file gives the same failure reason naming the same file.

Gate on this head

  • layers: 5 ran, 0 failed, 0 not-evaluated.
  • Mutation: 462 killed, 0 unexplained survivors. 5 are explained, 4 of them new equivalent entries.

🤖 Generated with Claude Code

VirtualMachinist and others added 6 commits November 19, 2026 22:51
The mutation layer found what each changed test file covers by running Vitest on that file alone,
at the head and again at the merge base: one process, one start-up and one pass over the coverage
include for each file, one after another. The functions it needs are those any of the files ran, so
the head versions now run in one run, and the merge-base copies in another; a group that gives
nothing (one of its files fails, or the run leaves no report) runs file by file as before, so the
file that cannot run is found and named exactly as it was. At most 100 files run at once, so a very
large diff never overruns a command line. The functions in scope, the mutate ranges and the
verdicts are the same; coveredFunctions are now ordered by file and line.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…atements

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A test that drives diffScope reaches the git, function and coverage readers, so their weak spots
are scored with the rest. Each gets a case: what a path counts as a test or as source, a coverage
entry with no location, a hit that a function only partly holds, functions with no body, TSX
against TS, what git says when it fails, the arguments of the default Vitest runs and the removal
of their directories, the name of a merge-base copy, and ranges that do not join across files.
The module-level patterns of scope.ts move into the functions that use them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ur equivalent mutants

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@VirtualMachinist VirtualMachinist added the protocol-change Changes laws, schema, profiles, thresholds, budgets or normative spec text label Nov 20, 2026
VirtualMachinist and others added 5 commits November 20, 2026 03:13
Every mutant of the diff step re-runs each test that covers it. The batching tests ran real git for
each case, and the real-Vitest pin ran three Vitest runs in-process, so a mutant cost tens of
seconds on a three-CPU runner and the macOS leg stopped at the 30-minute mutation budget. Git is
now a fake that answers from a diff text, in every test but the older ones, and the real-Vitest pin
takes both scopes in a child process, where nothing it runs counts as coverage of the diff step.
The cases are the same.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tions

The helpers the diff tests take their git from are source files to the mutation layer, so what
they say is pinned by their own test: the text of a diff, what the fake git answers, and the
directory a plain project is made in. The covered functions are listed by file and line, and a run
that took its own directory away is no failure.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ent mutants of the test helpers

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@VirtualMachinist
VirtualMachinist merged commit fb84d68 into main Nov 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

protocol-change Changes laws, schema, profiles, thresholds, budgets or normative spec text

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant