Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .web-tops/survivors.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,65 @@ survivors:
reason: document.toJS throws only Error instances (the YAML library), so the
fallback text is a dead branch.
owner: web-platform
- file: src/diff/coverage.ts
function: src/diff/coverage.ts::innermostHolding
mutator: ConditionalExpression
original: "span.start <= hit.start"
replacement: "true"
occurrence: 1
reason: innermostAt then keeps only spans that hold the line of the hit's start,
which are exactly the spans that start at or before it, so the first test
of the filter changes nothing.
owner: web-platform
- file: src/diff/functions.ts
function: src/diff/functions.ts::boundName
mutator: ConditionalExpression
original: "parent.initializer === node"
replacement: "true"
occurrence: 1
reason: "A function node is the child of a variable, property assignment or
property declaration only as its initializer: its other children are
names, types and decorators, none of them function nodes."
owner: web-platform
- file: src/diff/functions.ts
function: src/diff/functions.ts::declaredName
mutator: ConditionalExpression
original: "ts.isArrowFunction(node)"
replacement: "false"
occurrence: 1
reason: An arrow function has no name, so the next line gives undefined for it
as well; the test exists for the type of the node, not for the result.
owner: web-platform
- file: src/diff/scope.ts
function: src/diff/scope.ts::scopeOf
mutator: ConditionalExpression
original: "file.status !== \"deleted\""
replacement: "true"
occurrence: 1
reason: A deleted file has no added lines, so it adds no range to mutate and no
function to the changed ones, whether or not it is counted among the
sources.
owner: web-platform
- file: src/diff/test-support.ts
function: src/diff/test-support.ts::<module>
mutator: ArrayDeclaration
original: "[]"
replacement: "[\"Stryker was here\"]"
occurrence: 1
reason: The list only holds directories to remove; a name that is no directory
is removed with force and changes nothing, so a first entry in the list is
not seen.
owner: web-platform
- file: src/diff/test-support.ts
function: src/diff/test-support.ts::showOf
mutator: StringLiteral
original: "\"\""
replacement: "\"Stryker was here!\""
occurrence: 1
reason: "The fallback is the key of a file that is not in the table of
merge-base files, and any other string is not in it either: the same
failure comes back."
owner: web-platform
- file: src/gate/adapter.ts
function: src/gate/adapter.ts::classCap
mutator: StringLiteral
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ All notable changes to this project are listed here. The protocol version and th
## [Unreleased]

- Milestone M3 opens: `knip`, `secretlint` with its recommended preset and `eslint-plugin-react-hooks` are pinned exactly in §2, `package.json` and the lock, and the M3 paths are in every tool scope. §8.1 and §17 name the law fixtures `fixtures/laws/W-NN/{bad,clean}/`. (G0)
- The diff step finds what the changed test files cover with one Vitest run for their HEAD versions and one for their merge-base versions, not one run of each file alone, and runs file by file only where a group gives nothing. The functions in scope, the mutants and the verdicts are the same; the cost of the step falls with the number of changed test files.

## [0.3.0] - 2026-11-19

Expand Down
7 changes: 4 additions & 3 deletions WEB_TOPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -390,10 +390,11 @@ banned shortcuts, `!` and `as`, cheaper than handling the case. The profile key
test file at the merge base or at HEAD**, so a change that only weakens a test, or drops the call
to a function, is mutated. `webtops` computes the line ranges and passes them as Stryker `mutate`
entries, `file:start-end` (confirmed on Stryker 10.1.1). It maps each changed test file to the
functions it covers by running that file alone under Vitest, with the gate's own configuration
functions it covers by running the changed test files under Vitest, with the gate's own configuration
(the project's, coverage thresholds removed) and the gate's coverage include and exclude (§6.1),
once for the version at HEAD and once for the version at the merge base; a deleted test file runs
only the second. The functions that ran are added, with their whole ranges. Each range is then widened to the scored units it reaches into (§6.7), and a range that touches module-level code to all of the file's module-level lines, so the occurrence in a survivor's key does not depend on the diff. A HEAD version that
once for the versions at HEAD and once for the versions at the merge base, each in one run of up to 100 files (a function
is covered when any file ran it; a group whose run gives nothing runs file by file, so the file that cannot run is
found); a deleted test file is in the second only. The functions that ran are added, with their whole ranges. Each range is then widened to the scored units it reaches into (§6.7), and a range that touches module-level code to all of the file's module-level lines, so the occurrence in a survivor's key does not depend on the diff. A HEAD version that
cannot run is `fail`; a merge-base version that cannot run against HEAD adds nothing. The diff is
taken without rename detection, so a renamed file counts as new and every function in it is
judged. A pure deletion belongs to the smallest function that holds the lines on both sides of
Expand Down
254 changes: 254 additions & 0 deletions src/diff/batch.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import type { Exec } from "../gate/types.ts";
import type { CoverageOf, CoverageOfMany } from "./scope.ts";
import { diffScope } from "./scope.ts";
import { cleanRepos, diffOf, fakeGit, plainDir } from "./test-support.ts";
import type { CoverageMap, FunctionHit, ScopeResult } from "./types.ts";

// The changed test files are run together, once, and file by file only where that gives nothing: the functions in
// scope, the mutate ranges and the verdicts are those of a run of each file alone.

afterEach(cleanRepos);

const M = [
"export function f(a: number): number {",
" return a + 1;",
"}",
"export function g(a: number): number {",
" return a * 2;",
"}",
"",
].join("\n");
const STRONG = 'import { f, g } from "./m.ts";\nf(1);\ng(2);\n';
const WEAK = 'import { f } from "./m.ts";\nf(1);\n';

/** What a run covered of src/m.ts: f at lines 1 to 3, g at 4 to 6. */
const ran = (f: boolean, g: boolean): CoverageMap =>
new Map([
[
"src/m.ts",
[
{ start: 1, end: 3, hits: f ? 1 : 0 },
{ start: 4, end: 6, hits: g ? 1 : 0 },
],
],
]);

/** A project with src/m.ts and the given test files added, and the git that would say so. */
function added(...names: string[]): { cwd: string; exec: Exec } {
const cwd = plainDir({ "src/m.ts": M, ...Object.fromEntries(names.map((name) => [name, WEAK])) });
const exec = fakeGit(diffOf(names.map((path) => ({ path, kind: "added" }))));
return { cwd, exec };
}

/** src/m.test.ts weakened and src/n.test.ts added, with the strong version of the first at the merge base. */
function weakened(): { cwd: string; exec: Exec } {
const cwd = plainDir({ "src/m.ts": M, "src/m.test.ts": WEAK, "src/n.test.ts": WEAK });
const diff = diffOf([
{ path: "src/m.test.ts", kind: "modified" },
{ path: "src/n.test.ts", kind: "added" },
]);
return { cwd, exec: fakeGit(diff, { "src/m.test.ts": STRONG }) };
}

const keys = (result: ScopeResult): string[] =>
result.outcome === "fail" ? [] : result.scope.coveredFunctions.map((ref) => ref.key);

describe("changed test files run together", () => {
it("runs the head versions in one run, and none of them alone", async () => {
const together: (readonly string[])[] = [];
const alone: string[] = [];
const result = await diffScope({
...added("src/a.test.ts", "src/b.test.ts", "src/c.test.ts"),
base: "main",
coverageOfMany: (group) => {
together.push(group);
return Promise.resolve(ran(true, true));
},
coverageOf: (file) => {
alone.push(file);
return Promise.resolve(undefined);
},
});
expect(together).toEqual([["src/a.test.ts", "src/b.test.ts", "src/c.test.ts"]]);
expect(alone).toEqual([]);
expect(keys(result)).toEqual(["src/m.ts::f", "src/m.ts::g"]);
});

it("runs them one by one, in order, when the run together gives nothing, and adds up what each covered", async () => {
const alone: string[] = [];
const result = await diffScope({
...added("src/a.test.ts", "src/b.test.ts"),
base: "main",
coverageOfMany: () => Promise.resolve(undefined),
coverageOf: (file) => {
alone.push(file);
return Promise.resolve(file === "src/a.test.ts" ? ran(true, false) : ran(false, true));
},
});
expect(alone).toEqual(["src/a.test.ts", "src/b.test.ts"]);
expect(keys(result)).toEqual(["src/m.ts::f", "src/m.ts::g"]);
});

it("fails naming the first file that cannot run alone, the later ones unnamed", async () => {
const result = await diffScope({
...added("src/a.test.ts", "src/b.test.ts", "src/c.test.ts"),
base: "main",
coverageOfMany: () => Promise.resolve(undefined),
coverageOf: (file) => Promise.resolve(file === "src/a.test.ts" ? ran(true, true) : undefined),
});
expect(result).toEqual({
outcome: "fail",
reason: "could not run src/b.test.ts to find the functions it covers",
});
});

it("is the same scope as running each file alone", async () => {
const project = added("src/a.test.ts", "src/b.test.ts");
const maps: Record<string, CoverageMap> = {
"src/a.test.ts": ran(true, false),
"src/b.test.ts": ran(false, true),
};
const alone: CoverageOf = (file) => Promise.resolve(maps[file]);
const many: CoverageOfMany = () => Promise.resolve(ran(true, true));
const one = await diffScope({ ...project, base: "main", coverageOf: alone });
const all = await diffScope({
...project,
base: "main",
coverageOfMany: many,
coverageOf: alone,
});
expect(all).toEqual(one);
});

it("runs the merge-base versions together too, as copies that are gone afterwards", async () => {
const project = weakened();
const together: (readonly string[])[] = [];
const seen: string[] = [];
const result = await diffScope({
...project,
copyToken: () => "tok",
base: "main",
coverageOf: () => Promise.resolve(undefined),
coverageOfMany: (group) => {
together.push(group);
for (const file of group) seen.push(existsSync(join(project.cwd, file)) ? file : "");
return Promise.resolve(
group.some((file) => file.includes(".wt-base-")) ? ran(true, true) : ran(true, false),
);
},
});
expect(together).toEqual([["src/m.test.ts", "src/n.test.ts"], ["src/m.wt-base-tok.test.ts"]]);
expect(seen).toEqual(["src/m.test.ts", "src/n.test.ts", "src/m.wt-base-tok.test.ts"]);
expect(keys(result)).toEqual(["src/m.ts::f", "src/m.ts::g"]);
expect(existsSync(join(project.cwd, "src/m.wt-base-tok.test.ts"))).toBe(false);
});

it("holds the merge-base version in the copy, and runs a copy that fails together alone, leaving it out", async () => {
const project = weakened();
const alone: string[] = [];
const text: Record<string, string> = {};
const result = await diffScope({
...project,
copyToken: () => "tok",
base: "main",
coverageOfMany: (group) =>
Promise.resolve(
group.some((file) => file.includes(".wt-base-")) ? undefined : ran(true, false),
),
coverageOf: (file) => {
alone.push(file);
text[file] = readFileSync(join(project.cwd, file), "utf8");
return Promise.resolve(undefined);
},
});
expect(alone).toEqual(["src/m.wt-base-tok.test.ts"]);
expect(text["src/m.wt-base-tok.test.ts"]).toBe(STRONG.trim());
expect(keys(result)).toEqual(["src/m.ts::f"]);
expect(existsSync(join(project.cwd, "src/m.wt-base-tok.test.ts"))).toBe(false);
});

it("leaves a merge-base version out when its runs throw, and still removes the copy", async () => {
const project = weakened();
const result = await diffScope({
...project,
copyToken: () => "tok",
base: "main",
coverageOfMany: (group) => {
if (group.some((file) => file.includes(".wt-base-"))) throw new Error("the run died");
return Promise.resolve(ran(true, false));
},
coverageOf: () => {
throw new Error("the run died");
},
});
expect(keys(result)).toEqual(["src/m.ts::f"]);
expect(existsSync(join(project.cwd, "src/m.wt-base-tok.test.ts"))).toBe(false);
});

it("runs at most 100 files at once", async () => {
const names = Array.from({ length: 205 }, (_, at) => `src/t${String(at)}.test.ts`);
const sizes: number[] = [];
await diffScope({
...added(...names),
base: "main",
coverageOfMany: (group) => {
sizes.push(group.length);
return Promise.resolve(ran(true, false));
},
});
expect(sizes).toEqual([100, 100, 5]);
});

it("lists the functions of one file by line, though the head covered the later and the base the earlier", async () => {
const project = weakened();
const result = await diffScope({
...project,
copyToken: () => "tok",
base: "main",
coverageOf: () => Promise.resolve(undefined),
coverageOfMany: (group) =>
Promise.resolve(
group.some((file) => file.includes(".wt-base-")) ? ran(true, false) : ran(false, true),
),
});
expect(keys(result)).toEqual(["src/m.ts::f", "src/m.ts::g"]);
});

it("lists the functions by file and line, whatever order the coverage came in", async () => {
const cwd = plainDir({ "src/z.ts": M, "src/a.ts": M, "src/t.test.ts": WEAK });
const both = (path: string): [string, readonly FunctionHit[]] => [
path,
[
{ start: 4, end: 6, hits: 1 },
{ start: 1, end: 3, hits: 1 },
],
];
const result = await diffScope({
cwd,
base: "main",
exec: fakeGit(diffOf([{ path: "src/t.test.ts", kind: "added" }])),
coverageOfMany: () => Promise.resolve(new Map([both("src/z.ts"), both("src/a.ts")])),
});
expect(keys(result)).toEqual(["src/a.ts::f", "src/a.ts::g", "src/z.ts::f", "src/z.ts::g"]);
});

it("runs no group when it is given a source for one file only", async () => {
const asked: string[] = [];
const project = added("src/a.test.ts", "src/b.test.ts");
const exec: Exec = (command, args, options) => {
asked.push(command);
return project.exec(command, args, options);
};
const result = await diffScope({
cwd: project.cwd,
base: "main",
exec,
coverageOf: () => Promise.resolve(ran(true, false)),
});
expect(keys(result)).toEqual(["src/m.ts::f"]);
expect(asked.filter((command) => command.includes("vitest"))).toEqual([]);
});
});
Loading
Loading