Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/mutation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,6 @@ jobs:
ref: ${{ github.sha }}
persist-credentials: false
- uses: ./.github/actions/setup-project
with:
install: 'false'
- id: plan
run: |
matrix=$(node scripts/full-mutations.mjs plan)
Expand Down
24 changes: 18 additions & 6 deletions docs/mutation-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,10 @@ current discovery. `npm run mutation:full` remains a single fresh local campaign

Scheduled CI, full manual CI and exact-source release validation instead use the
native GitHub matrix in `.github/workflows/mutation.yml`. Node's filesystem glob
discovers every parser source; sorted whole files are assigned round-robin to at
most 11 nonempty parser jobs. The existing pure-runtime campaign occupies the
discovers every parser source. The pinned Stryker instrumenter generates the
complete current mutant inventory without running tests. A small planner joins
overlapping mutant spans into indivisible components and assigns larger components
first to the least-populated of at most 11 nonempty parser jobs. The existing pure-runtime campaign occupies the
twelfth mutation slot. This ceiling is not a measured optimum or a cap on other
CI jobs. Each parser job retains two Stryker workers, the ordinary full Vitest
suite, native parser-test discovery and a fresh Stryker coverage dry run.
Expand All @@ -31,8 +33,12 @@ event SHA. Caller inputs cannot select other code within the event's cache scope

Fresh jobs remove native working state before `--force --mutate`, because force
alone can retain out-of-scope results from an old incremental report. They never
restore or publish mutation-result caches. The collector checks every assigned
file exactly once, current source/test bytes, common configuration and declared
restore or publish mutation-result caches. Stryker has no distributed mutant-ID
selector: jobs use its native line/column ranges. A range must fully contain a
mutated AST node, so arbitrary adjacent line splits are unsafe. The planner
re-instruments each assignment and requires its exact mutant identities to match
the complete inventory, with no missing, extra or duplicate ownership. The
collector repeats this check against the unchanged raw reports, current source/test bytes, common configuration and declared
inputs, clean process completion, run/attempt/commit identity and artifact/report
digests. It applies the existing source-bound equivalent proofs and 100% adjusted
policy across unchanged raw reports, resolving test IDs within each report.
Expand All @@ -52,9 +58,15 @@ native all-skipped suites (such as an absent optional external sample) without a
filename exception; every runnable suite must appear consistently in every shard.

Ordinary PR/main runs, including cache misses, retain the single-runner path.
File workloads may be uneven; runner count alone does not establish a latency
guarantee. The short native regression in `scripts/full-mutations.integration.test.mjs`
Mutant counts are only a scheduling heuristic: individual tests and timeouts vary
widely, and a method-spanning mutant prevents further safe subdivision. The
planner uses no historical timing store, result cache, dependency graph or report
rewrite. Planning now installs the locked native instrumenter; each shard still
pays for its own ordinary suite and coverage dry run. The short native regression in `scripts/full-mutations.integration.test.mjs`
checks the real runner/collector, fresh state, changed inputs and failed evidence.
It also verifies a file split across jobs and rejects a missing mutant even when
the altered report's digest is updated. Ordinary incremental execution remains
whole-scope. See the pinned native [range containment implementation](https://github.com/stryker-mutator/stryker-js/blob/959127763e5bbe2eb65f5341a8161780d6d8a476/packages/instrumenter/src/transformers/babel-transformer.ts).

Native reuse includes killed, timeout and reviewed-equivalent outcomes. It is
not fresh execution: reused native records omit `duration`. The audit permits an
Expand Down
1 change: 1 addition & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
"@jazzer.js/core": "4.0.0",
"@typescript/native": "npm:typescript@7.0.2",
"@stryker-mutator/core": "10.1.1",
"@stryker-mutator/instrumenter": "10.1.1",
"eslint-plugin-security": "4.2.0",
"globals": "17.13.0",
"@stryker-mutator/vitest-runner": "10.1.1",
Expand Down
28 changes: 22 additions & 6 deletions scripts/full-mutations.integration.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { cp, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/pr
import { join } from 'node:path';
import { test } from 'node:test';
import { currentSources, runShard, collectShards } from './full-mutations.mjs';
import { sourceContext } from './mutation-policy.mjs';
import { sha256, sourceContext } from './mutation-policy.mjs';
import { ENCODER_TEST_NAMES } from './verify-encoder-discovery.mjs';
import { repositoryRoot } from './release-utils.mjs';

Expand All @@ -18,17 +18,16 @@ async function fixture() {
const files = {
'.gitignore': 'node_modules/\n.cache/\nreports/\n.stryker-tmp/\n',
'package.json': '{"type":"module"}',
'src/mp3/a.ts': 'export const positive = (n: number) => n > 0;\n',
'src/mp3/b.ts': 'export const negative = (n: number) => n < 0;\n',
'src/mp3/a.ts':
'export const positive = (n: number) => n > 0;\nexport const negative = (n: number) => n < 0;\n',
'vitest.config.ts':
"export default { test: { include: ['tests/**/*.test.ts'], maxWorkers: 1 } };",
'vitest.mutation.config.ts':
"export default { test: { include: ['tests/mp3/**/*.test.ts'], maxWorkers: 1 } };",
'stryker.config.json': JSON.stringify(config),
'tests/mp3/optional.test.ts': "import {it} from 'vitest'; it.skip('optional sample',()=>{});",
'tests/mp3/free-format-real.test.ts': `import {test,expect} from 'vitest';
import {positive} from '../../src/mp3/a';
import {negative} from '../../src/mp3/b';
import {positive,negative} from '../../src/mp3/a';
for(const name of ${JSON.stringify(ENCODER_TEST_NAMES)}) test(name,()=>{
for(const n of [-1,0,1]) {
expect(positive(n)).toBe(n===1);
Expand Down Expand Up @@ -68,7 +67,7 @@ async function fixture() {
}

test(
'native whole-file shards remain fresh and collection fails closed',
'native complete ranges split one file, remain fresh and collection fails closed',
{ timeout: 90000 },
async () => {
const root = await fixture();
Expand All @@ -90,10 +89,27 @@ test(
assert.equal(summary.adjustedScore, 100);
assert.equal(summary.execution.reused, 0);
assert.equal(summary.execution.shards, 2);
assert.equal(summary.perFile['src/mp3/a.ts'].total, 10);
const native = JSON.parse(
await readFile(join(directory, `full-parser-shard-${attempt}-0/mutation.json`), 'utf8'),
);
assert.equal(Object.hasOwn(native.testFiles, 'tests/mp3/optional.test.ts'), false);
const shardDirectory = join(directory, `full-parser-shard-${attempt}-0`);
const reportPath = join(shardDirectory, 'mutation.json');
const manifestPath = join(shardDirectory, 'complete.json');
const reportBytes = await readFile(reportPath);
const manifestBytes = await readFile(manifestPath);
const manifest = JSON.parse(manifestBytes);
native.files['src/mp3/a.ts'].mutants.pop();
const incomplete = JSON.stringify(native);
await writeFile(reportPath, incomplete);
await writeFile(
manifestPath,
JSON.stringify({ ...manifest, reportSha256: sha256(incomplete) }),
);
await assert.rejects(collectShards(root, directory), /native partition mutant/);
await writeFile(reportPath, reportBytes);
await writeFile(manifestPath, manifestBytes);
await assert.rejects(readFile(join(root, '.cache/mutation/native-baseline.json')), {
code: 'ENOENT',
});
Expand Down
19 changes: 8 additions & 11 deletions scripts/full-mutations.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,11 @@ import { mutationEnvironment, mutationInputs, fingerprint } from './mutation-cac
import { runnableParserTestFiles, verifyParserTestDiscovery } from './parser-test-discovery.mjs';
import { assertEncoderShardDiscovery } from './verify-encoder-discovery.mjs';
import { runMutationProcess } from './mutation-process.mjs';
import { mutationPartitions } from './mutation-partitions.mjs';
import { isMain, readJson, repositoryRoot } from './release-utils.mjs';

// Reserve the twelfth mutation runner for the independent pure-runtime campaign.
export const MAX_PARSER_SHARDS = 11;
export function partitionSources(paths) {
assert.ok(paths.length > 0, 'Empty parser source inventory');
assert.equal(new Set(paths).size, paths.length, 'Duplicate parser source');
const groups = Array.from({ length: Math.min(MAX_PARSER_SHARDS, paths.length) }, () => []);
paths.toSorted().forEach((file, index) => groups[index % groups.length].push(file));
return groups;
}

export async function currentSources(root) {
const paths = (await Array.fromAsync(glob(PARSER_GLOB, { cwd: root })))
Expand Down Expand Up @@ -67,7 +61,7 @@ async function freshTests(root, scope, env, log) {

export async function runShard(root, shard) {
const sources = await currentSources(root);
const scopes = partitionSources([...sources.keys()]);
const { scopes } = await mutationPartitions(sources, MAX_PARSER_SHARDS);
assert.ok(Number.isInteger(shard) && scopes[shard], 'Invalid parser shard');
const output = join(root, 'reports/mutation');
const lock = join(root, '.cache/mutation/run.lock');
Expand Down Expand Up @@ -179,7 +173,7 @@ export async function collectShards(root, directory) {
const output = join(root, 'reports/mutation');
await rm(output, { recursive: true, force: true });
const sources = await currentSources(root);
const scopes = partitionSources([...sources.keys()]);
const { scopes, inventories } = await mutationPartitions(sources, MAX_PARSER_SHARDS);
const expected = invocation(root);
const names = scopes.map((_, index) => artifactName(expected.attempt, index));
assert.deepEqual(
Expand All @@ -196,7 +190,7 @@ export async function collectShards(root, directory) {
const inputs = await mutationInputs(root, mutationEnvironment());
const reports = validateShardEvidence(entries, scopes, expected, inputs);
const policy = await readJson(join(root, 'mutation-equivalents.json'));
const summary = auditMutationShards(reports, policy, sources, scopes);
const summary = auditMutationShards(reports, policy, sources, scopes, inventories);
await freshTestSources(root, reports);
assertEncoderShardDiscovery(reports);
assert.equal(fingerprint(await mutationInputs(root, mutationEnvironment())), fingerprint(inputs));
Expand All @@ -221,7 +215,10 @@ if (isMain(import.meta.url)) {
const [command, argument, ...extra] = process.argv.slice(2);
assert.equal(extra.length, 0, 'Unexpected arguments');
if (command === 'plan' && argument === undefined) {
const scopes = partitionSources([...(await currentSources(repositoryRoot)).keys()]);
const { scopes } = await mutationPartitions(
await currentSources(repositoryRoot),
MAX_PARSER_SHARDS,
);
console.log(JSON.stringify({ include: scopes.map((_, shard) => ({ shard })) }));
} else if (command === 'run' && /^(0|[1-9]\d*)$/.test(argument ?? '')) {
await runShard(repositoryRoot, Number(argument));
Expand Down
14 changes: 0 additions & 14 deletions scripts/full-mutations.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,13 @@ import { join } from 'node:path';
import { tmpdir } from 'node:os';
import {
currentSources,
partitionSources,
validateShardEvidence,
assertShardTestInventory,
} from './full-mutations.mjs';
import { sha256 } from './mutation-policy.mjs';
import { fingerprint } from './mutation-cache-inputs.mjs';
import { assertEncoderShardDiscovery, ENCODER_TEST_NAMES } from './verify-encoder-discovery.mjs';

test('whole-file partitions are complete, deterministic and bounded without empty jobs', () => {
assert.throws(() => partitionSources([]), /Empty/);
assert.throws(() => partitionSources(['a', 'a']), /Duplicate/);
for (const count of [1, 2, 8, 11, 12, 25]) {
const files = Array.from({ length: count }, (_, index) => `src/mp3/${index}.ts`);
const scopes = partitionSources(files);
assert.equal(scopes.length, Math.min(11, count));
assert.ok(scopes.every((scope) => scope.length > 0));
assert.deepEqual(scopes.flat().sort(), files.toSorted());
assert.deepEqual(partitionSources(files.toReversed()), scopes);
}
});

test('native source discovery picks up new nested files and excludes declarations', async () => {
const root = await mkdtemp(join(tmpdir(), 'mp3-shard-discovery-'));
try {
Expand Down
107 changes: 107 additions & 0 deletions scripts/mutation-partitions.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
import assert from 'node:assert/strict';
import { Instrumenter } from '@stryker-mutator/instrumenter';
import nativePackage from '@stryker-mutator/instrumenter/package.json' with { type: 'json' };

const options = { excludedMutations: [], ignorers: [], plugins: null };
const logger = { debug() {}, info() {}, warn() {}, trace() {}, isDebugEnabled: () => false };
const position = (a, b) => a.line - b.line || a.column - b.column;
const lexical = (a, b) => (a < b ? -1 : Number(a > b));
const identity = ({ fileName, location, mutatorName, replacement }) =>
JSON.stringify([fileName, location.start, location.end, mutatorName, replacement]);
const identities = (mutants) => mutants.map(identity).sort();

async function instrument(files) {
const { mutants } = await new Instrumenter(logger).instrument(files, options);
assert.ok(
mutants.every((mutant) => mutant.status !== 'Ignored'),
'Ignored native mutant',
);
return mutants;
}

function components(mutants) {
const result = [];
const sorted = mutants.toSorted(
(a, b) =>
position(a.location.start, b.location.start) || position(a.location.end, b.location.end),
);
for (const mutant of sorted) {
const previous = result.at(-1);
if (previous && position(mutant.location.start, previous.end) < 0) {
if (position(mutant.location.end, previous.end) > 0) previous.end = mutant.location.end;
previous.mutants.push(mutant);
} else result.push({ fileName: mutant.fileName, ...mutant.location, mutants: [mutant] });
}
return result;
}

function reportIdentity({ fileName, location, mutatorName, replacement }) {
const point = ({ line, column }) => ({ line: line + 1, column: column + 1 });
return {
fileName,
location: { start: point(location.start), end: point(location.end) },
mutatorName,
replacement,
};
}

async function provePartition(parts, sources) {
const ranges = new Map();
for (const { fileName, start, end } of parts) {
if (!ranges.has(fileName)) ranges.set(fileName, []);
ranges.get(fileName).push({ start, end });
}
const files = [...ranges].map(([name, mutate]) => ({ name, content: sources.get(name), mutate }));
const expected = parts.flatMap((part) => part.mutants);
assert.deepEqual(
identities(await instrument(files)),
identities(expected),
'Native partition coverage differs',
);
return expected.map(reportIdentity);
}

// Native ranges include whole AST nodes. Overlapping mutant spans stay together;
// instrumenting the actual planned ranges proves that no boundary drops a mutant.
export async function mutationPartitions(sources, limit = 11) {
assert.equal(nativePackage.version, '10.1.1', 'Unsupported native instrumenter version');
assert.ok(sources instanceof Map && sources.size > 0, 'Empty parser source inventory');
assert.ok(Number.isInteger(limit) && limit > 0, 'Invalid parser shard limit');
const parts = [];
const fullInventory = [];
for (const [name, content] of [...sources].sort(([a], [b]) => lexical(a, b))) {
assert.ok(typeof content === 'string' && content.length > 0, `Empty parser source: ${name}`);
const mutants = await instrument([{ name, content, mutate: true }]);
assert.ok(mutants.length > 0, `No native mutants: ${name}`);
assert.equal(new Set(identities(mutants)).size, mutants.length, 'Duplicate native mutant');
fullInventory.push(...mutants.map(reportIdentity));
parts.push(...components(mutants));
}
parts.sort(
(a, b) =>
b.mutants.length - a.mutants.length ||
lexical(a.fileName, b.fileName) ||
position(a.start, b.start),
);
const length = Math.min(limit, parts.length);
const bins = Array.from({ length }, () => ({ parts: [], count: 0 }));
for (const part of parts) {
const bin = bins.reduce((a, b) => (a.count <= b.count ? a : b));
bin.parts.push(part);
bin.count += part.mutants.length;
}
const scopes = bins.map((bin) =>
bin.parts.map(
({ fileName, start, end }) =>
`${fileName}:${start.line + 1}:${start.column}-${end.line + 1}:${end.column}`,
),
);
const inventories = [];
for (const bin of bins) inventories.push(await provePartition(bin.parts, sources));
assert.deepEqual(
identities(inventories.flat()),
identities(fullInventory),
'Incomplete native partition union',
);
return { scopes, inventories };
}
Loading
Loading