Repository navigation
Diff step: batch the coverage runs of changed tests - #65
Merged
Merged
Conversation
The mutation layer found what each changed test file covers by running Vitest on that file alone, at the head and again at the merge base: one process, one start-up and one pass over the coverage include for each file, one after another. The functions it needs are those any of the files ran, so the head versions now run in one run, and the merge-base copies in another; a group that gives nothing (one of its files fails, or the run leaves no report) runs file by file as before, so the file that cannot run is found and named exactly as it was. At most 100 files run at once, so a very large diff never overruns a command line. The functions in scope, the mutate ranges and the verdicts are the same; coveredFunctions are now ordered by file and line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…atements Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A test that drives diffScope reaches the git, function and coverage readers, so their weak spots are scored with the rest. Each gets a case: what a path counts as a test or as source, a coverage entry with no location, a hit that a function only partly holds, functions with no body, TSX against TS, what git says when it fails, the arguments of the default Vitest runs and the removal of their directories, the name of a merge-base copy, and ranges that do not join across files. The module-level patterns of scope.ts move into the functions that use them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ur equivalent mutants Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Every mutant of the diff step re-runs each test that covers it. The batching tests ran real git for each case, and the real-Vitest pin ran three Vitest runs in-process, so a mutant cost tens of seconds on a three-CPU runner and the macOS leg stopped at the 30-minute mutation budget. Git is now a fake that answers from a diff text, in every test but the older ones, and the real-Vitest pin takes both scopes in a child process, where nothing it runs counts as coverage of the diff step. The cases are the same. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tions The helpers the diff tests take their git from are source files to the mutation layer, so what they say is pinned by their own test: the text of a diff, what the fake git answers, and the directory a plain project is made in. The covered functions are listed by file and line, and a run that took its own directory away is no failure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ent mutants of the test helpers Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes the gate's diff step cheaper without changing what it decides. The step runs the changed test files under coverage, at the head and at the merge base, to find the functions those tests cover.
What it does
Before: one Vitest-with-coverage run per changed test file at the head, and one per file at the merge base, one after another.
After:
What stays the same: the functions in scope, the mutate ranges, the mutants and the verdicts. The covered functions are now ordered by file and line.
Measured on the diff step alone, with the same tree and base:
The type checker was measured too
Narrowing the type checker's program to the mutated files was unsound, because 3 of 594 compile-error mutants turned into survivors. Narrowing it to the mutated files plus their importers was sound but saved nothing, and
skipLibChecksaved nothing either. The checker is unchanged.Declared exceptions
src/diff/{scope,coverage,functions}.tsand their testsWEB_TOPS.md§6.4protocol-change, wording onlyCHANGELOG.md,.web-tops/survivors.yamlTests pin that the batched path and the file-by-file path give the same functions, and that a failing file gives the same failure reason naming the same file.
Gate on this head
layers: 5 ran, 0 failed, 0 not-evaluated.🤖 Generated with Claude Code