From 4179483779f5f95536130df29f869d1574c84080 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 08:30:12 +0000 Subject: [PATCH 01/12] feat(stryker-js): add stryker audit to check dropped mutants against a kill matrix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `stryker audit --matrix ` instruments a kill-matrix run's sources without running a test, joins every mutant the default policy drops to its dominator, and judges each pair from the full matrix (killers(d) ⊆ killers(m) over named tests): Pass, Vacuous, AttributionUnverified or Fail, each with a reason code and a next action. Tests whose every kill is dropped are reported as orphaned. `--files` scopes the check to a PR's changed files. `--counts-only` reads finished reports and counts planned, CompileError, compiled and executed mutants. The decisions are pure workflows (audit-drops, count-runs) with property laws; the shell is audit-request.cell.ts, reusing instrumentSources from the instrument stage. Report fields that JSON never carries as present-undefined are optionalKey, so the generated codec laws round-trip. Over kill-matrix-420: 78 drops, 24 Pass, 51 Vacuous, 3 AttributionUnverified (the three file-hook pairs), 0 Fail, 0 orphaned. Counts over mutation-report-416: 8626 planned, 4136 CompileError. --- .changeset/stryker-audit-drops.md | 5 + packages/stryker-js/README.md | 7 + packages/stryker-js/src/Cli.schema.ts | 12 + .../audit-drops.workflow.property.test.ts | 265 ++++++++++++++ .../count-runs.workflow.property.test.ts | 83 +++++ ...oute-cli-request.workflow.property.test.ts | 2 + .../stryker-js/src/audit-drops.workflow.ts | 340 ++++++++++++++++++ packages/stryker-js/src/audit-request.cell.ts | 340 ++++++++++++++++++ packages/stryker-js/src/audit.schema.ts | 266 ++++++++++++++ packages/stryker-js/src/bin/cli-command.ts | 50 +++ .../stryker-js/src/count-runs.workflow.ts | 118 ++++++ .../src/route-cli-request.workflow.ts | 22 ++ packages/stryker-js/src/run-request.cell.ts | 6 + packages/stryker-js/src/run/instrument.ts | 46 ++- .../tests/audit.integration.test.ts | 288 +++++++++++++++ 15 files changed, 1834 insertions(+), 16 deletions(-) create mode 100644 .changeset/stryker-audit-drops.md create mode 100644 packages/stryker-js/src/__tests__/audit-drops.workflow.property.test.ts create mode 100644 packages/stryker-js/src/__tests__/count-runs.workflow.property.test.ts create mode 100644 packages/stryker-js/src/audit-drops.workflow.ts create mode 100644 packages/stryker-js/src/audit-request.cell.ts create mode 100644 packages/stryker-js/src/audit.schema.ts create mode 100644 packages/stryker-js/src/count-runs.workflow.ts create mode 100644 packages/stryker-js/tests/audit.integration.test.ts diff --git a/.changeset/stryker-audit-drops.md b/.changeset/stryker-audit-drops.md new file mode 100644 index 000000000..84b2296ee --- /dev/null +++ b/.changeset/stryker-audit-drops.md @@ -0,0 +1,5 @@ +--- +"@systemfsoftware/stryker-js": minor +--- + +New `stryker audit` subcommand. With `--matrix ` it reads a run recorded with every killer kept (`STRYKER_KILL_MATRIX=1`), finds every mutant the default mutant-set policy drops without running a test, and judges each against its dominator: `Pass`, `Vacuous`, `AttributionUnverified` or `Fail`, each with a reason code and a next action. A test whose every killed mutant is dropped is reported as orphaned. `--files` limits the check to the named files, so a pull request checks only what it changes. It exits `1` when a pair fails, a rule has no joined pair, or a test is orphaned, and `2` on unusable input. `--counts-only` counts planned, `CompileError`, compiled and executed mutants and test executions per project from finished reports. Both modes write versioned JSON to `--out` and print at most 20 detail lines. diff --git a/packages/stryker-js/README.md b/packages/stryker-js/README.md index 67db81cbe..48b62144d 100644 --- a/packages/stryker-js/README.md +++ b/packages/stryker-js/README.md @@ -171,6 +171,11 @@ pnpm exec stryker compare --baseline reports/mutation-report.json --fresh report # Annotate a pull request at each surfaced survivor pnpm exec stryker annotate --baseline .stryker-baseline.json +# Check every dropped mutant against a kill matrix, or count a finished run +pnpm exec stryker audit --matrix kill-matrix/ --projects packages/a,packages/b --out reports/audit.json +pnpm exec stryker audit --matrix kill-matrix/ --files packages/a/src/sum.ts --out reports/audit.json +pnpm exec stryker audit --counts-only --matrix mutation-report/ --out reports/counts.json + # Judge one surfaced survivor, or drive runs from an editor or an agent pnpm exec stryker feedback --not-useful --reason "logging only" pnpm exec stryker serve stdio @@ -179,6 +184,8 @@ pnpm exec stryker mcp `stryker run --since ` takes the merge base of `` and the working tree, diffs it at `--unified=0`, and intersects the changed lines with `mutate`; it falls back to a full run when the Stryker config, a test-runner config, `package.json`, or the lockfile changed. `stryker run --mutant ` admits each id against the previous report (`reports/mutation/mutation.json`), restricts the run to those mutants' files, and emits each one's status, covering tests, killing test, and a `stryker run --mutant ` reproducer. `stryker gate` exits non-zero on survivors missing from its baseline, `stryker compare` exits `1` on a status mismatch after subtracting a `--noise` file, and `stryker annotate` prints GitHub workflow commands at each surfaced survivor. +`stryker audit` checks that the default mutant-set policy loses no kill. `--matrix` is a directory holding `/stryker-incremental.json` for each `--projects` entry, recorded by a run with every killer kept (the shared config's `STRYKER_KILL_MATRIX=1` lane). The audit instruments the matrix sources without running a test, joins each dropped mutant `m` to its dominator `d` by id, and judges the pair: `Pass` when every named test that kills `d` also kills `m` (`killers-contained`) or `m` was never compiled (`dropped-mutant-carries-no-kill-signal`); `Vacuous` when `d` survived or had no coverage; `AttributionUnverified` when a killer is a file-level hook, the matrix names no test for it, or either mutant timed out; `Fail` when a named test kills `d` but not `m`, `d` never ran, `m` survived, or the ids do not match. A test whose every killed mutant is dropped is reported as orphaned. `--files` scopes the audit to the drops in the named files, so a pull request audits only what it changes. The audit exits `0` when no pair fails, no rule goes unattested and no test is orphaned, `1` otherwise, and `2` on unusable input (an unreadable or undecodable matrix, reports that record no mutant, or `--files` with `--counts-only`). `--counts-only` reads the reports alone and counts planned, `CompileError`, compiled and executed mutants and test executions per project, with ignored mutants grouped by rule. Both modes write a versioned JSON report (`schemaVersion`) to `--out`, where every pair, unattested rule and orphaned test carries a reason code and a next action, and print a summary capped at 20 detail lines. + A mutation run only starts on main CI. `stryker run` — and the programmatic `run`/`strykerCell` entry, the MSP `mutationTest` request, and the MCP `rerun_mutant` tool — refuses to start unless `GITHUB_ACTIONS` is `true`. A refused run exits non-zero and ends the machine stream with a `refused` event naming the rule (`mutation-runs-on-main-ci`). Dry runs (`--dryRunOnly`), `--version`, config and help commands, the MSP `discover` request, and the read-only MCP tools are unaffected. `stryker serve stdio|socket` speaks the Mutation Server Protocol and `stryker mcp` serves MCP over stdio. ## Output Modes diff --git a/packages/stryker-js/src/Cli.schema.ts b/packages/stryker-js/src/Cli.schema.ts index 9a1a0d502..816ff0664 100644 --- a/packages/stryker-js/src/Cli.schema.ts +++ b/packages/stryker-js/src/Cli.schema.ts @@ -106,6 +106,16 @@ const PlanRequestSchema = S.TaggedStruct('plan', { export type PlanRequest = S.Schema.Type +const AuditRequestSchema = S.TaggedStruct('audit', { + matrix: S.String, + out: S.String, + countsOnly: S.Boolean, + projects: S.Array(S.String).pipe(S.optional), + files: S.Array(S.String).pipe(S.optional), +}) + +export type AuditRequest = S.Schema.Type + export type CliRequest = | RunRequest | MergeRequest @@ -113,6 +123,7 @@ export type CliRequest = | GateRequest | AnnotateRequest | PlanRequest + | AuditRequest | ServeRouteRequest | FeedbackRouteRequest | McpRouteRequest @@ -126,6 +137,7 @@ export class CliRouteCommand extends S.TaggedClass()('CliRouteC GateRequestSchema, AnnotateRequestSchema, PlanRequestSchema, + AuditRequestSchema, ServeRouteRequestSchema, FeedbackRouteRequestSchema, McpRouteRequestSchema, diff --git a/packages/stryker-js/src/__tests__/audit-drops.workflow.property.test.ts b/packages/stryker-js/src/__tests__/audit-drops.workflow.property.test.ts new file mode 100644 index 000000000..63f678aa1 --- /dev/null +++ b/packages/stryker-js/src/__tests__/audit-drops.workflow.property.test.ts @@ -0,0 +1,265 @@ +import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' +import { describe, it } from '@systemfsoftware/vitest' +import * as Arbitrary from 'effect/Arbitrary' +import * as Arr from 'effect/Array' +import * as Match from 'effect/Match' +import * as Option from 'effect/Option' +import * as Result from 'effect/Result' +import * as S from 'effect/Schema' + +import { auditDrops, AuditDropsCommand } from '../audit-drops.workflow.js' +import type { + AuditedDrop, + AuditedPair, + DropAuditReport, + MatrixMutant, + MatrixProject, + PairVerdict, +} from '../audit.schema.js' + +const IDS = ['0000000000000000', '1111111111111111', '2222222222222222', '3333333333333333'] as const +const NAMED_TESTS = ['t1', 't2', 't3'] as const +const FILE_HOOK = 'src/a.test.ts#hook' +const MUTANT = Mutant.MutantId.make(IDS[0]) +const DOMINATOR = Mutant.MutantId.make(IDS[1]) + +const killerArb = Arbitrary.schema(S.Literals([...NAMED_TESTS, FILE_HOOK])) +const killersArb = Arbitrary.array(killerArb, { maxLength: 3 }).pipe(Arbitrary.map(Arr.dedupe)) + +const statusArb: Arbitrary.Arbitrary = Arbitrary.all({ + killed: Arbitrary.schema(S.Boolean), + status: Arbitrary.schema(Mutant.MutantStatusSchema), +}).pipe(Arbitrary.map(({ killed, status }) => (killed ? 'Killed' : status))) + +const recordArb = (id: string): Arbitrary.Arbitrary => + Arbitrary.all({ status: statusArb, killedBy: killersArb }).pipe( + Arbitrary.map(({ status, killedBy }) => ({ id, status, killedBy })), + ) + +const catalogArb: Arbitrary.Arbitrary> = Arbitrary.array( + Arbitrary.schema(S.Literals(NAMED_TESTS)), + { maxLength: 1 }, +).pipe(Arbitrary.map((unattributed) => NAMED_TESTS.filter((test) => !unattributed.includes(test)))) + +const dropOf = (mutant: Mutant.MutantId, dominator: Mutant.MutantId): AuditedDrop => ({ + project: 'p', + mutant, + subsumed: { _tag: 'Subsumed', rule: 'complement', dominators: [dominator] }, +}) + +const commandOf = (project: MatrixProject, drops: ReadonlyArray): AuditDropsCommand => + AuditDropsCommand.make({ scope: { _tag: 'Corpus' }, matrix: [project], drops }) + +interface PairScenario { + readonly mutant: MatrixMutant + readonly dominator: MatrixMutant + readonly mutantPresent: boolean + readonly dominatorPresent: boolean + readonly tests: ReadonlyArray +} + +const pairScenarioArb: Arbitrary.Arbitrary = Arbitrary.all({ + mutant: recordArb(MUTANT), + dominator: recordArb(DOMINATOR), + mutantPresent: Arbitrary.schema(S.Literals([true, true, true, false])), + dominatorPresent: Arbitrary.schema(S.Literals([true, true, true, false])), + tests: catalogArb, +}) + +const present = (record: MatrixMutant, isPresent: boolean): ReadonlyArray => isPresent ? [record] : [] + +const pairCommandOf = (scenario: PairScenario): AuditDropsCommand => + commandOf( + { + project: 'p', + mutants: [ + ...present(scenario.mutant, scenario.mutantPresent), + ...present(scenario.dominator, scenario.dominatorPresent), + ], + tests: scenario.tests, + }, + [dropOf(MUTANT, DOMINATOR)], + ) + +const reportOf = (subject: typeof auditDrops, command: AuditDropsCommand): DropAuditReport => + Result.getOrThrow(subject(command)) + +const verdictOf = (pair: AuditedPair): Option.Option => + Match.valueTags(pair, { + JoinedPair: (joined) => Option.some(joined.verdict), + UnjoinablePair: () => Option.none(), + }) + +const onlyVerdictOf = (report: DropAuditReport): Option.Option => + Option.flatMap(Arr.head(report.pairs), verdictOf) + +const isFail = (verdict: Option.Option): boolean => + Option.match(verdict, { + onNone: () => false, + onSome: (judged) => + Match.valueTags(judged, { + Fail: () => true, + Pass: () => false, + Vacuous: () => false, + AttributionUnverified: () => false, + }), + }) + +const isContainmentPass = (verdict: Option.Option): boolean => + Option.match(verdict, { + onNone: () => false, + onSome: (judged) => + Match.valueTags(judged, { + Pass: (pass) => pass.reason === 'killers-contained', + Fail: () => false, + Vacuous: () => false, + AttributionUnverified: () => false, + }), + }) + +const bothKilledAndJoined = (scenario: PairScenario): boolean => + scenario.mutantPresent && scenario.dominatorPresent && + scenario.mutant.status === 'Killed' && scenario.dominator.status === 'Killed' + +interface OrphanSlot { + readonly status: Mutant.MutantStatus + readonly killsT1: boolean + readonly dropped: boolean + readonly others: ReadonlyArray +} + +const orphanSlotArb: Arbitrary.Arbitrary = Arbitrary.all({ + status: statusArb, + killsT1: Arbitrary.schema(S.Boolean), + dropped: Arbitrary.schema(S.Boolean), + others: Arbitrary.array(Arbitrary.schema(S.Literals(['t2', 't3'])), { maxLength: 2 }).pipe(Arbitrary.map(Arr.dedupe)), +}) + +interface OrphanScenario { + readonly slots: ReadonlyArray + readonly t1Named: boolean +} + +const orphanScenarioArb: Arbitrary.Arbitrary = Arbitrary.all({ + slots: Arbitrary.all(IDS.map(() => orphanSlotArb)), + t1Named: Arbitrary.schema(S.Literals([true, true, false])), +}) + +const orphanCommandOf = (scenario: OrphanScenario): AuditDropsCommand => + commandOf( + { + project: 'p', + mutants: scenario.slots.map((slot, index) => ({ + id: IDS[index] ?? '', + status: slot.status, + killedBy: slot.killsT1 ? ['t1', ...slot.others] : slot.others, + })), + tests: scenario.t1Named ? NAMED_TESTS : ['t2', 't3'], + }, + scenario.slots.flatMap((slot, index) => + slot.dropped ? [dropOf(Mutant.MutantId.make(IDS[index] ?? ''), DOMINATOR)] : [] + ), + ) + +const t1OrphanedOf = (scenario: OrphanScenario): boolean => { + const tally = scenario.slots.reduce( + (acc, slot) => { + const killedByT1 = slot.status === 'Killed' && slot.killsT1 + return { kills: acc.kills || killedByT1, allDropped: acc.allDropped && (!killedByT1 || slot.dropped) } + }, + { kills: false, allDropped: true }, + ) + return scenario.t1Named && tally.kills && tally.allDropped +} + +const dropsArb: Arbitrary.Arbitrary> = Arbitrary.array( + Arbitrary.all({ + mutant: Arbitrary.schema(S.Literals(IDS)), + first: Arbitrary.schema(S.Literals(IDS)), + rest: Arbitrary.array(Arbitrary.schema(S.Literals(IDS)), { maxLength: 2 }), + }).pipe(Arbitrary.map(({ mutant, first, rest }) => ({ + project: 'p', + mutant: Mutant.MutantId.make(mutant), + subsumed: { + _tag: 'Subsumed' as const, + rule: 'complement' as const, + dominators: Arr.map([first, ...rest] as const, (id) => Mutant.MutantId.make(id)), + }, + }))), + { maxLength: 4 }, +) + +describe('auditDrops', () => { + it.prop( + '∀d_Drops_≡ShouldAuditOnePairWhenADropNamesADominator', + { of: [dropsArb], subject: auditDrops }, + (subject, [drops]) => + reportOf(subject, commandOf({ project: 'p', mutants: [], tests: [] }, drops)).pairs.length === + drops.reduce((count, drop) => count + drop.subsumed.dominators.length, 0), + ) + + it.prop( + '∀s_DroppedPair_≡ShouldFailWhenANamedTestKillsTheDominatorButNotTheMutant', + { of: [pairScenarioArb], subject: auditDrops }, + (subject, [scenario]) => { + const lostNamedKill = bothKilledAndJoined(scenario) && + scenario.dominator.killedBy.some((killer) => + scenario.tests.includes(killer) && !scenario.mutant.killedBy.includes(killer) + ) + return !lostNamedKill || isFail(onlyVerdictOf(reportOf(subject, pairCommandOf(scenario)))) + }, + ) + + it.prop( + '∀s_DroppedPair_≡ShouldPassContainmentOnlyWhenEveryDominatorKillerKillsTheMutant', + { of: [pairScenarioArb], subject: auditDrops }, + (subject, [scenario]) => { + const contained = bothKilledAndJoined(scenario) && scenario.dominator.killedBy.length > 0 && + scenario.dominator.killedBy.every((killer) => scenario.mutant.killedBy.includes(killer)) + return contained === isContainmentPass(onlyVerdictOf(reportOf(subject, pairCommandOf(scenario)))) + }, + ) + + it.prop( + '∀s_DroppedPair_≡ShouldFailWhenTheMutantSurvivesAKilledDominator', + { of: [pairScenarioArb], subject: auditDrops }, + (subject, [scenario]) => { + const hidesSurvivor = scenario.mutantPresent && scenario.dominatorPresent && + scenario.dominator.status === 'Killed' && + (scenario.mutant.status === 'Survived' || scenario.mutant.status === 'NoCoverage') + return !hidesSurvivor || isFail(onlyVerdictOf(reportOf(subject, pairCommandOf(scenario)))) + }, + ) + + it.prop( + '∀s_DroppedPair_≡ShouldLeaveTheRuleUnattestedWhenThePairDoesNotJoin', + { of: [pairScenarioArb], subject: auditDrops }, + (subject, [scenario]) => { + const joins = scenario.mutantPresent && scenario.dominatorPresent && + scenario.mutant.status !== 'Pending' && scenario.dominator.status !== 'Pending' + const attested = Option.match(Arr.head(reportOf(subject, pairCommandOf(scenario)).rules), { + onNone: () => false, + onSome: (rule) => Match.valueTags(rule, { Attested: () => true, Unattested: () => false }), + }) + return attested === joins + }, + ) + + it.prop( + '∀s_DroppedPair_≡ShouldFailTheAuditWhenThePairFails', + { of: [pairScenarioArb], subject: auditDrops }, + (subject, [scenario]) => { + const report = reportOf(subject, pairCommandOf(scenario)) + const failed = Match.valueTags(report, { DropAuditPassed: () => false, DropAuditFailed: () => true }) + return !isFail(onlyVerdictOf(report)) || failed + }, + ) + + it.prop( + '∀k_KillMatrix_≡ShouldReportAnOrphanWhenEveryMutantANamedTestKillsIsDropped', + { of: [orphanScenarioArb], subject: auditDrops }, + (subject, [scenario]) => + reportOf(subject, orphanCommandOf(scenario)).orphanedTests.some((orphan) => orphan.test === 't1') === + t1OrphanedOf(scenario), + ) +}) diff --git a/packages/stryker-js/src/__tests__/count-runs.workflow.property.test.ts b/packages/stryker-js/src/__tests__/count-runs.workflow.property.test.ts new file mode 100644 index 000000000..88d82f47c --- /dev/null +++ b/packages/stryker-js/src/__tests__/count-runs.workflow.property.test.ts @@ -0,0 +1,83 @@ +import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' +import { describe, it } from '@systemfsoftware/vitest' +import * as Arbitrary from 'effect/Arbitrary' +import * as Result from 'effect/Result' +import * as S from 'effect/Schema' + +import type { CountedReport, RunCounts } from '../audit.schema.js' +import { countRuns, CountRunsCommand } from '../count-runs.workflow.js' + +type CountedMutant = CountedReport['files'][string]['mutants'][number] + +const statusReasonArb: Arbitrary.Arbitrary = Arbitrary.all({ + named: Arbitrary.schema(S.Boolean), + rule: Arbitrary.schema(Mutant.IgnoreRuleId), + free: Arbitrary.schema(S.String.check(S.isMaxLength(8))), +}).pipe(Arbitrary.map(({ named, rule, free }) => (named ? `${rule}: ${free}` : free))) + +const mutantArb: Arbitrary.Arbitrary = Arbitrary.all({ + id: Arbitrary.schema(S.String.check(S.isMaxLength(4))), + status: Arbitrary.schema(Mutant.MutantStatusSchema), + statusReason: statusReasonArb, + testsCompleted: Arbitrary.schema(S.Int.check(S.isBetween({ minimum: 0, maximum: 9 }))), +}) + +const reportArb: Arbitrary.Arbitrary = Arbitrary.array( + Arbitrary.array(mutantArb, { maxLength: 4 }), + { maxLength: 3 }, +).pipe(Arbitrary.map((files) => ({ + files: Object.fromEntries(files.map((mutants, index) => [`src/f${index}.ts`, { mutants }])), +}))) + +const commandArb: Arbitrary.Arbitrary = Arbitrary.array(reportArb, { minLength: 1, maxLength: 3 }) + .pipe( + Arbitrary.map((reports) => + CountRunsCommand.make({ reports: reports.map((report, index) => ({ project: `p${index}`, report })) }) + ), + ) + +const mutantsOf = (command: CountRunsCommand): ReadonlyArray => + command.reports.flatMap((entry) => Object.values(entry.report.files).flatMap((file) => file.mutants)) + +const sumOf = (values: ReadonlyArray): number => values.reduce((sum, value) => sum + value, 0) + +const statusTotal = (counts: RunCounts): number => sumOf(Object.values(counts.statuses)) + +describe('countRuns', () => { + it.prop( + '∀c_CountRunsCommand_≡ShouldRefuseWhenNoReportRecordsAMutant', + { of: [commandArb], subject: countRuns }, + (subject, [command]) => Result.isFailure(subject(command)) === (mutantsOf(command).length === 0), + ) + + it.prop( + '∀c_CountRunsCommand_≡ShouldCountEveryRecordedMutantOnceWhenProjectsAreTotalled', + { of: [commandArb], subject: countRuns }, + (subject, [command]) => + Result.match(subject(command), { + onFailure: () => mutantsOf(command).length === 0, + onSuccess: (counts) => + counts.total.planned === mutantsOf(command).length && + statusTotal(counts.total) === counts.total.planned && + sumOf(counts.projects.map((project) => project.counts.planned)) === counts.total.planned && + sumOf(counts.projects.map((project) => project.counts.testExecutions)) === counts.total.testExecutions, + }), + ) + + it.prop( + '∀c_CountRunsCommand_≡ShouldCountAnIgnoredMutantUnderTheRuleItsReasonNames', + { of: [commandArb], subject: countRuns }, + (subject, [command]) => + Result.match(subject(command), { + onFailure: () => true, + onSuccess: (counts) => { + const ignored = mutantsOf(command).filter((mutant) => mutant.status === 'Ignored') + return counts.total.ignoredByRule.every((entry) => + entry.count === ignored.filter((mutant) => (mutant.statusReason ?? '').startsWith(`${entry.rule}: `)).length + ) && + sumOf(counts.total.ignoredByRule.map((entry) => entry.count)) + counts.total.ignoredUnrecognized === + ignored.length + }, + }), + ) +}) diff --git a/packages/stryker-js/src/__tests__/route-cli-request.workflow.property.test.ts b/packages/stryker-js/src/__tests__/route-cli-request.workflow.property.test.ts index bd4017edb..db492449d 100644 --- a/packages/stryker-js/src/__tests__/route-cli-request.workflow.property.test.ts +++ b/packages/stryker-js/src/__tests__/route-cli-request.workflow.property.test.ts @@ -13,6 +13,7 @@ const matchesRoute = (command: CliRouteCommand, tag: string): boolean => Match.tag('gate', () => tag === 'CliGateRequested'), Match.tag('annotate', () => tag === 'CliAnnotateRequested'), Match.tag('plan', () => tag === 'CliPlanRequested'), + Match.tag('audit', () => tag === 'CliAuditRequested'), Match.tag('serve', () => tag === 'CliServeRequested'), Match.tag('feedback', () => tag === 'CliFeedbackRequested'), Match.tag('mcp', () => tag === 'CliMcpRequested'), @@ -38,6 +39,7 @@ const decides = (subject: typeof routeCliRequest, command: CliRouteCommand): str Match.tag('CliGateRequested', () => 'CliGateRequested'), Match.tag('CliAnnotateRequested', () => 'CliAnnotateRequested'), Match.tag('CliPlanRequested', () => 'CliPlanRequested'), + Match.tag('CliAuditRequested', () => 'CliAuditRequested'), Match.tag('CliServeRequested', () => 'CliServeRequested'), Match.tag('CliFeedbackRequested', () => 'CliFeedbackRequested'), Match.tag('CliMcpRequested', () => 'CliMcpRequested'), diff --git a/packages/stryker-js/src/audit-drops.workflow.ts b/packages/stryker-js/src/audit-drops.workflow.ts new file mode 100644 index 000000000..2157db26c --- /dev/null +++ b/packages/stryker-js/src/audit-drops.workflow.ts @@ -0,0 +1,340 @@ +import { Workflow } from '@systemfsoftware/effect-cell-types' +import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' +import * as Arr from 'effect/Array' +import * as Boolean from 'effect/Boolean' +import * as Match from 'effect/Match' +import * as Option from 'effect/Option' +import * as Order from 'effect/Order' +import * as Record from 'effect/Record' +import * as Result from 'effect/Result' +import * as S from 'effect/Schema' + +import { + AuditedDrop, + type AuditedPair, + AuditScope, + DropAuditFailed, + DropAuditPassed, + DropAuditReport, + type FailReason, + type MatrixMutant, + MatrixProject, + type OrphanedTest, + type PairVerdict, + type PassReason, + type RuleSummary, + type UnjoinableReason, + type UnverifiedReason, + type VacuousReason, +} from './audit.schema.js' + +export class AuditDropsCommand extends S.TaggedClass()('AuditDropsCommand', { + scope: AuditScope, + drops: S.Array(AuditedDrop), + matrix: S.Array(MatrixProject), +}) { + static readonly [Workflow.InstrumentationBrand] = {} as const +} + +type Reason = + | typeof PassReason.Type + | typeof VacuousReason.Type + | typeof UnverifiedReason.Type + | typeof FailReason.Type + | typeof UnjoinableReason.Type + +const FIX_THE_RULE = "Fix the subsumption rule so it keeps this mutant, or set `mutator.mutantSetPolicy: 'full'`." +const RERUN_MATRIX = 'Re-run the kill-matrix lane on the current main, then audit again.' + +const NEXT_ACTION: { readonly [reason in Reason]: string } = { + 'killers-contained': 'None: every test that kills the dominator also kills the dropped mutant.', + 'dropped-mutant-carries-no-kill-signal': + 'None: the dropped mutant never reached a test on main (CompileError, RuntimeError or Ignored), so dropping it loses no kill.', + 'dominator-survived': + 'None for the drop. To make the pair checkable, add a test that kills the dominator: it survives on main.', + 'dominator-uncovered': + 'None for the drop. To make the pair checkable, add a test that reaches the site: no test covers it on main.', + 'killer-is-file-hook': + 'Confirm by hand that the file-level failure in `missingKillers` also fails for the dropped mutant, or move that assertion into a named test so the kill is attributed.', + 'killer-unrecorded': `The dominator is Killed with no recorded killer. ${RERUN_MATRIX}`, + 'dropped-mutant-timed-out': + "Confirm that every test in the dominator's killers times out on the dropped mutant: a Timeout records no killer.", + 'dominator-timed-out': + 'Confirm that every test the dominator times out under also kills the dropped mutant: a Timeout records no killer.', + 'killers-not-contained': + `Signal lost: the tests in \`missingKillers\` kill the dominator but not the dropped mutant. ${FIX_THE_RULE}`, + 'dropped-mutant-survives': + `Signal lost: the dropped mutant survives on main while its dominator is killed, so the drop hides a survivor. ${FIX_THE_RULE}`, + 'misidentified-pair': + 'The dominator is uncovered on main but the dropped mutant ran, so the rule paired two different sites. Fix the dominator the rule names.', + 'dominator-not-run': + 'The dominator never reached a test on main (CompileError, RuntimeError or Ignored), so it cannot stand in for the dropped mutant. Fix the rule so it names a dominator that runs.', + 'mutant-absent': `The kill matrix has no record of the dropped mutant. ${RERUN_MATRIX}`, + 'dominator-absent': `The kill matrix has no record of the dominator. ${RERUN_MATRIX}`, + 'status-unsettled': `The kill matrix holds the pair as Pending. ${RERUN_MATRIX}`, +} + +const UNATTESTED_NEXT = + `No drop of this rule joined the kill matrix, so nothing checks it. ${RERUN_MATRIX} Do not ship the rule until a pair joins.` + +const ORPHANED_NEXT = + `Signal lost: this test kills only dropped mutants, so it kills nothing once they are dropped. ${FIX_THE_RULE}` + +interface ProjectIndex { + readonly mutants: Record.ReadonlyRecord + readonly tests: ReadonlyArray +} + +interface PairKey { + readonly project: string + readonly rule: Mutant.Subsumed['rule'] + readonly mutant: Mutant.MutantId + readonly dominator: Mutant.MutantId +} + +interface JoinedFacts { + readonly mutant: MatrixMutant + readonly dominator: MatrixMutant + readonly missingKillers: ReadonlyArray + readonly tests: ReadonlyArray +} + +type Judged = Result.Result + +const EMPTY_INDEX: ProjectIndex = { mutants: {}, tests: [] } + +const pass = (reason: typeof PassReason.Type): Judged => Result.succeed({ _tag: 'Pass', reason }) +const vacuous = (reason: typeof VacuousReason.Type): Judged => Result.succeed({ _tag: 'Vacuous', reason }) +const unverified = (reason: typeof UnverifiedReason.Type): Judged => + Result.succeed({ _tag: 'AttributionUnverified', reason }) +const fail = (reason: typeof FailReason.Type): Judged => Result.succeed({ _tag: 'Fail', reason }) +const unsettled = (): Judged => Result.fail('status-unsettled') + +const indexOf = (matrix: ReadonlyArray): Record.ReadonlyRecord => + Object.fromEntries(matrix.map((project) => [ + project.project, + { mutants: Object.fromEntries(project.mutants.map((mutant) => [mutant.id, mutant])), tests: project.tests }, + ])) + +const containment = (facts: JoinedFacts): Judged => + Boolean.match(facts.dominator.killedBy.length === 0, { + onTrue: () => unverified('killer-unrecorded'), + onFalse: () => + Boolean.match(facts.missingKillers.length === 0, { + onTrue: () => pass('killers-contained'), + onFalse: () => + Boolean.match(facts.missingKillers.some((killer) => facts.tests.includes(killer)), { + onTrue: () => fail('killers-not-contained'), + onFalse: () => unverified('killer-is-file-hook'), + }), + }), + }) + +const underKilledDominator = (facts: JoinedFacts): Judged => + Match.value(facts.mutant.status).pipe( + Match.when('Killed', () => containment(facts)), + Match.when('Timeout', () => unverified('dropped-mutant-timed-out')), + Match.whenOr('Survived', 'NoCoverage', () => fail('dropped-mutant-survives')), + Match.whenOr('CompileError', 'RuntimeError', 'Ignored', () => pass('dropped-mutant-carries-no-kill-signal')), + Match.when('Pending', unsettled), + Match.exhaustive, + ) + +const underTimedOutDominator = (facts: JoinedFacts): Judged => + Match.value(facts.mutant.status).pipe( + Match.whenOr('Killed', 'Timeout', () => unverified('dominator-timed-out')), + Match.whenOr('Survived', 'NoCoverage', () => fail('dropped-mutant-survives')), + Match.whenOr('CompileError', 'RuntimeError', 'Ignored', () => pass('dropped-mutant-carries-no-kill-signal')), + Match.when('Pending', unsettled), + Match.exhaustive, + ) + +const underUncoveredDominator = (facts: JoinedFacts): Judged => + Match.value(facts.mutant.status).pipe( + Match.whenOr('NoCoverage', 'CompileError', 'RuntimeError', 'Ignored', () => vacuous('dominator-uncovered')), + Match.whenOr('Killed', 'Survived', 'Timeout', () => fail('misidentified-pair')), + Match.when('Pending', unsettled), + Match.exhaustive, + ) + +const settledOr = (facts: JoinedFacts, judged: () => Judged): Judged => + Boolean.match(facts.mutant.status === 'Pending', { onTrue: unsettled, onFalse: judged }) + +const judge = (facts: JoinedFacts): Judged => + Match.value(facts.dominator.status).pipe( + Match.when('Killed', () => underKilledDominator(facts)), + Match.when('Timeout', () => underTimedOutDominator(facts)), + Match.when('Survived', () => settledOr(facts, () => vacuous('dominator-survived'))), + Match.when('NoCoverage', () => underUncoveredDominator(facts)), + Match.whenOr('CompileError', 'RuntimeError', 'Ignored', () => settledOr(facts, () => fail('dominator-not-run'))), + Match.when('Pending', unsettled), + Match.exhaustive, + ) + +const factsOf = (index: ProjectIndex, mutant: MatrixMutant, dominator: MatrixMutant): JoinedFacts => ({ + mutant, + dominator, + missingKillers: dominator.killedBy.filter((killer) => !mutant.killedBy.includes(killer)), + tests: index.tests, +}) + +const joinedOf = (index: ProjectIndex, key: PairKey): Result.Result => + Option.match(Record.get(index.mutants, key.mutant), { + onNone: () => Result.fail('mutant-absent'), + onSome: (mutant) => + Option.match(Record.get(index.mutants, key.dominator), { + onNone: () => Result.fail('dominator-absent'), + onSome: (dominator) => Result.succeed(factsOf(index, mutant, dominator)), + }), + }) + +const unjoinablePair = (key: PairKey, reason: typeof UnjoinableReason.Type): AuditedPair => ({ + _tag: 'UnjoinablePair', + ...key, + reason, + next: NEXT_ACTION[reason], +}) + +const joinedPair = (key: PairKey, facts: JoinedFacts, verdict: PairVerdict): AuditedPair => ({ + _tag: 'JoinedPair', + ...key, + mutantStatus: facts.mutant.status, + dominatorStatus: facts.dominator.status, + missingKillers: facts.missingKillers, + verdict, + next: NEXT_ACTION[verdict.reason], +}) + +const auditPair = (indexes: Record.ReadonlyRecord, key: PairKey): AuditedPair => + Result.match(joinedOf(Option.getOrElse(Record.get(indexes, key.project), () => EMPTY_INDEX), key), { + onFailure: (reason) => unjoinablePair(key, reason), + onSuccess: (facts) => + Result.match(judge(facts), { + onFailure: (reason) => unjoinablePair(key, reason), + onSuccess: (verdict) => joinedPair(key, facts, verdict), + }), + }) + +const pairKeysOf = (drop: AuditedDrop): ReadonlyArray => + drop.subsumed.dominators.map((dominator) => ({ + project: drop.project, + rule: drop.subsumed.rule, + mutant: drop.mutant, + dominator, + })) + +const pairOrder: Order.Order = Order.combineAll([ + Order.mapInput(Order.String, (pair: AuditedPair) => pair.project), + Order.mapInput(Order.String, (pair: AuditedPair) => pair.mutant), + Order.mapInput(Order.String, (pair: AuditedPair) => pair.dominator), +]) + +type Tally = 'pass' | 'vacuous' | 'attributionUnverified' | 'fail' | 'unjoinable' + +const tallyOf = (pair: AuditedPair): Tally => + Match.valueTags(pair, { + UnjoinablePair: (): Tally => 'unjoinable', + JoinedPair: (joined) => + Match.valueTags(joined.verdict, { + Pass: (): Tally => 'pass', + Vacuous: (): Tally => 'vacuous', + AttributionUnverified: (): Tally => 'attributionUnverified', + Fail: (): Tally => 'fail', + }), + }) + +const tallyCount = (pairs: ReadonlyArray, tally: Tally): number => + pairs.filter((pair) => tallyOf(pair) === tally).length + +const ruleSummaryOf = ( + drops: Arr.NonEmptyReadonlyArray, + pairs: ReadonlyArray, +): RuleSummary => { + const counts = { + rule: Arr.headNonEmpty(drops).subsumed.rule, + drops: drops.length, + pass: tallyCount(pairs, 'pass'), + vacuous: tallyCount(pairs, 'vacuous'), + attributionUnverified: tallyCount(pairs, 'attributionUnverified'), + fail: tallyCount(pairs, 'fail'), + unjoinable: tallyCount(pairs, 'unjoinable'), + } + return Boolean.match(pairs.length > counts.unjoinable, { + onTrue: (): RuleSummary => ({ _tag: 'Attested', ...counts }), + onFalse: (): RuleSummary => ({ _tag: 'Unattested', ...counts, reason: 'no-drop-joined', next: UNATTESTED_NEXT }), + }) +} + +const isUnattested = (rule: RuleSummary): boolean => + Match.valueTags(rule, { Attested: () => false, Unattested: () => true }) + +const orphansOf = (project: MatrixProject, drops: ReadonlyArray): ReadonlyArray => { + const dropped: ReadonlyArray = drops.filter((drop) => drop.project === project.project).map((drop) => + drop.mutant + ) + const kills = project.mutants + .filter((mutant) => mutant.status === 'Killed') + .flatMap((mutant) => mutant.killedBy.map((test) => ({ test, mutant: mutant.id }))) + .filter((kill) => project.tests.includes(kill.test)) + return Object.entries(Arr.groupBy(kills, (kill) => kill.test)).flatMap(([test, killedBy]) => + Boolean.match(killedBy.every((kill) => dropped.includes(kill.mutant)), { + onTrue: (): ReadonlyArray => [{ + project: project.project, + test, + killed: Arr.map(killedBy, (kill) => kill.mutant), + reason: 'every-killed-mutant-dropped', + next: ORPHANED_NEXT, + }], + onFalse: () => [], + }) + ) +} + +const orphanOrder: Order.Order = Order.combine( + Order.mapInput(Order.String, (orphan: OrphanedTest) => orphan.project), + Order.mapInput(Order.String, (orphan: OrphanedTest) => orphan.test), +) + +const decide = (command: AuditDropsCommand): DropAuditReport => { + const indexes = indexOf(command.matrix) + const pairs = Arr.sort(command.drops.flatMap(pairKeysOf).map((key) => auditPair(indexes, key)), pairOrder) + const pairsByRule = Arr.groupBy(pairs, (pair) => pair.rule) + const rules = Arr.sort( + Object.values(Arr.groupBy(command.drops, (drop) => drop.subsumed.rule)).map((drops) => + ruleSummaryOf( + drops, + Option.getOrElse(Option.fromUndefinedOr(pairsByRule[Arr.headNonEmpty(drops).subsumed.rule]), () => []), + ) + ), + Order.mapInput(Order.String, (rule: RuleSummary) => rule.rule), + ) + const orphanedTests = Arr.sort(command.matrix.flatMap((project) => orphansOf(project, command.drops)), orphanOrder) + const fields = { + schemaVersion: '1' as const, + scope: command.scope, + projects: command.matrix.map((project) => ({ + project: project.project, + matrixMutants: project.mutants.length, + drops: command.drops.filter((drop) => drop.project === project.project).length, + })), + rules, + pairs, + orphanedTests, + } + const failures = { + pairs: tallyCount(pairs, 'fail'), + unattestedRules: rules.filter(isUnattested).length, + orphanedTests: orphanedTests.length, + } + return Boolean.match(failures.pairs + failures.unattestedRules + failures.orphanedTests === 0, { + onTrue: (): DropAuditReport => DropAuditPassed.make(fields), + onFalse: (): DropAuditReport => DropAuditFailed.make({ ...fields, failures }), + }) +} + +export const auditDrops = Workflow.make({ + command: AuditDropsCommand, + decision: DropAuditReport, + error: S.Never, + decide: (command): Result.Result => Result.succeed(decide(command)), +}) diff --git a/packages/stryker-js/src/audit-request.cell.ts b/packages/stryker-js/src/audit-request.cell.ts new file mode 100644 index 000000000..82610cc4d --- /dev/null +++ b/packages/stryker-js/src/audit-request.cell.ts @@ -0,0 +1,340 @@ +import { Cell } from '@systemfsoftware/effect-cell-types' +import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' +import * as Arr from 'effect/Array' +import * as Boolean from 'effect/Boolean' +import type * as Console from 'effect/Console' +import * as Effect from 'effect/Effect' +import * as FileSystem from 'effect/FileSystem' +import * as Layer from 'effect/Layer' +import * as Match from 'effect/Match' +import * as Option from 'effect/Option' +import * as Path from 'effect/Path' +import * as Result from 'effect/Result' +import * as S from 'effect/Schema' + +import { auditDrops, AuditDropsCommand } from './audit-drops.workflow.js' +import { + type AuditedDrop, + type AuditedPair, + AuditFailed, + AuditInputUnusable, + type AuditReport, + AuditReportJson, + type AuditScope, + CountedReportJson, + type CountsReport, + type DropAuditReport, + type KillMatrixReport, + KillMatrixReportJson, + type MatrixProject, + type NothingCounted, + type RunCounts, +} from './audit.schema.js' +import { countRuns, CountRunsCommand } from './count-runs.workflow.js' +import { readProjectCell } from './read-project.cell.js' +import type { HostServices } from './run/host.service.js' +import { instrumentSources } from './run/instrument.js' +import { loadConfigCell } from './run/load-config.cell.js' +import { prepareForInstrumentCell } from './run/plan-prepare.cell.js' +import type { PrepareForInstrument } from './run/prepare.js' +import { RunEnvironment } from './run/RunEnvironment.service.js' +import type { EnginePorts } from './run/StageServices.service.js' + +export interface AuditRequest { + readonly matrix: string + readonly out: string + readonly countsOnly: boolean + readonly projects?: ReadonlyArray | undefined + readonly files?: ReadonlyArray | undefined +} + +export interface AuditRequestInput { + readonly request: AuditRequest + readonly channel: AuditChannel +} + +export interface AuditChannel { + readonly environment: { + readonly basePath: string + readonly host: HostServices + readonly console: Console.Console + } +} + +const REPORT_FILE = 'stryker-incremental.json' +const DETAIL_LINES = 20 +const DECODE_DETAIL_CHARS = 300 + +const UNREADABLE_NEXT = + 'Pass --matrix the directory holding /stryker-incremental.json for every --projects entry, such as a downloaded kill-matrix or mutation-report artifact.' +const UNDECODABLE_NEXT = + 'Audit drops against a report written by a kill-matrix run (STRYKER_KILL_MATRIX=1), which records every killer and the test catalog.' +const FILES_WITH_COUNTS_NEXT = 'Drop --files: --counts-only counts every record of the reports.' + +type AuditFailure = AuditFailed | AuditInputUnusable | NothingCounted + +const prepareStageCell = Cell.andThen(Cell.andThen(loadConfigCell, readProjectCell), prepareForInstrumentCell) + +const readReport = ( + schema: S.Codec, + file: string, +): Effect.Effect => + Effect.flatMap(FileSystem.FileSystem, (fs) => + fs.readFileString(file).pipe( + Effect.mapError(() => + AuditInputUnusable.make({ code: 'matrix-unreadable', detail: `cannot read ${file}`, next: UNREADABLE_NEXT }) + ), + Effect.flatMap((text) => + Effect.fromResult(Result.mapError( + S.decodeResult(schema)(text), + (error) => + AuditInputUnusable.make({ + code: 'matrix-undecodable', + detail: `cannot decode ${file}: ${error.message.replace(/\s+/gu, ' ').slice(0, DECODE_DETAIL_CHARS)}`, + next: UNDECODABLE_NEXT, + }), + )) + ), + )) + +const projectsOf = (projects: ReadonlyArray | undefined): ReadonlyArray => + Option.getOrElse(Option.filter(Option.fromUndefinedOr(projects), Arr.isReadonlyArrayNonEmpty), () => ['.']) + +const scopeOf = (path: Path.Path, basePath: string, files: ReadonlyArray | undefined): AuditScope => + Option.match(Option.filter(Option.fromUndefinedOr(files), Arr.isReadonlyArrayNonEmpty), { + onNone: (): AuditScope => ({ _tag: 'Corpus' }), + onSome: (present): AuditScope => ({ + _tag: 'Files', + files: Arr.map(present, (file) => path.relative(basePath, path.resolve(basePath, file))), + }), + }) + +const inScope = (scope: AuditScope, path: Path.Path, project: string, key: string): boolean => + Match.value(scope).pipe( + Match.tag('Corpus', () => true), + Match.tag('Files', ({ files }) => files.includes(path.join(project, key))), + Match.exhaustive, + ) + +const withDefaultPolicy = (prepared: PrepareForInstrument): PrepareForInstrument => ({ + ...prepared, + options: { ...prepared.options, mutator: { ...prepared.options.mutator, mutantSetPolicy: 'default' } }, +}) + +const dropOf = (project: string, mutant: Mutant.Mutant): ReadonlyArray => + Option.match(Option.filter(Option.fromUndefinedOr(mutant.subsumption), S.is(Mutant.Subsumed)), { + onNone: () => [], + onSome: (subsumed) => [{ project, mutant: mutant.id, subsumed }], + }) + +const matrixProjectOf = (project: string, report: KillMatrixReport): MatrixProject => ({ + project, + mutants: Object.values(report.files).flatMap((file) => + file.mutants.map((mutant) => ({ id: mutant.id, status: mutant.status, killedBy: mutant.killedBy ?? [] })) + ), + tests: Object.values(report.testFiles).flatMap((file) => file.tests.map((test) => test.id)), +}) + +interface AuditedProject { + readonly drops: ReadonlyArray + readonly matrix: MatrixProject +} + +const auditProject = ( + request: AuditRequest, + channel: AuditChannel, + scope: AuditScope, + directory: string, +): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + const basePath = channel.environment.basePath + const report = yield* readReport( + KillMatrixReportJson, + path.resolve(basePath, request.matrix, directory, REPORT_FILE), + ) + const files = Object.entries(report.files) + .filter(([key]) => inScope(scope, path, directory, key)) + .map(([key, file]) => ({ name: path.resolve(report.projectRoot, key), content: file.source, mutate: true })) + const drops = yield* Effect.scoped(Effect.gen(function*() { + const project = yield* fs.realPath(path.resolve(basePath, directory)) + const env = { ...channel.environment.host.env, basePath: project } + const context = yield* Layer.build(RunEnvironment.stage(env, channel.environment.host.events)) + const prepared = yield* Cell.provideContext(prepareStageCell, context).run({ + cliOptions: {}, + targetMutatePatterns: undefined, + }) + const instrumented = yield* instrumentSources(withDefaultPolicy(prepared), files) + return instrumented.mutants.flatMap((mutant) => dropOf(directory, mutant)) + })).pipe(Effect.orDie) + return { drops, matrix: matrixProjectOf(directory, report) } + }) + +const inProjectDirectory = (directory: string, body: Effect.Effect): Effect.Effect => + Effect.suspend(() => { + const previous = globalThis.process.cwd() + return Effect.acquireUseRelease( + Effect.sync(() => globalThis.process.chdir(directory)), + () => body, + () => Effect.sync(() => globalThis.process.chdir(previous)), + ) + }) + +const writeReport = ( + basePath: string, + out: string, + report: AuditReport, +): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + const target = path.resolve(basePath, out) + const text = yield* S.encodeEffect(AuditReportJson)(report) + yield* fs.makeDirectory(path.dirname(target), { recursive: true }) + yield* fs.writeFileString(target, `${text}\n`) + return target + }).pipe(Effect.orDie) + +const verdictLabel = (verdict: DropAuditReport['pairs'][number]): string => + Match.valueTags(verdict, { + UnjoinablePair: (unjoinable) => `Unjoinable ${unjoinable.reason}`, + JoinedPair: (joined) => + Match.valueTags(joined.verdict, { + Pass: (pass) => `Pass ${pass.reason}`, + Vacuous: (vacuous) => `Vacuous ${vacuous.reason}`, + AttributionUnverified: (unverified) => `AttributionUnverified ${unverified.reason}`, + Fail: (fail) => `Fail ${fail.reason}`, + }), + }) + +const pairLine = (pair: AuditedPair): string => + `${verdictLabel(pair)}: ${pair.mutant} dropped for ${pair.dominator} (${pair.project})` + +const listedRankOf = (pair: AuditedPair): Option.Option => + Match.value(pair).pipe( + Match.tag('UnjoinablePair', () => Option.some(1)), + Match.tag('JoinedPair', (joined) => + Match.value(joined.verdict).pipe( + Match.tag('Fail', () => Option.some(0)), + Match.tag('AttributionUnverified', () => Option.some(2)), + Match.tag('Pass', () => Option.none()), + Match.tag('Vacuous', () => Option.none()), + Match.exhaustive, + )), + Match.exhaustive, + ) + +const detailLinesOf = (report: DropAuditReport): ReadonlyArray => { + const ranked = report.pairs.flatMap((pair) => + Option.toArray(Option.map(listedRankOf(pair), (rank) => ({ rank, pair }))) + ) + const failing = ranked.filter((entry) => entry.rank === 0).map((entry) => pairLine(entry.pair)) + const orphans = report.orphanedTests.map((orphan) => + `Orphaned ${orphan.reason}: test ${orphan.test} (${orphan.project}) kills only ${orphan.killed.length} dropped mutant(s)` + ) + const rest = ranked.filter((entry) => entry.rank > 0).toSorted((a, b) => a.rank - b.rank).map((entry) => + pairLine(entry.pair) + ) + return [...failing, ...orphans, ...rest] +} + +const attestation = (rule: DropAuditReport['rules'][number]): string => + Match.valueTags(rule, { Attested: () => 'Attested', Unattested: (unattested) => `Unattested (${unattested.reason})` }) + +const ruleLine = (rule: DropAuditReport['rules'][number]): string => + `rule ${rule.rule}: ${rule.drops} drop(s); ${rule.pass} Pass, ${rule.vacuous} Vacuous, ${rule.attributionUnverified} AttributionUnverified, ${rule.fail} Fail, ${rule.unjoinable} Unjoinable; ${ + attestation(rule) + }` + +const outcomeLabel = (report: DropAuditReport): string => + Match.valueTags(report, { DropAuditPassed: () => 'passed', DropAuditFailed: () => 'FAILED' }) + +const truncatedOf = (details: ReadonlyArray): ReadonlyArray => + Boolean.match(details.length > DETAIL_LINES, { + onTrue: () => [...details.slice(0, DETAIL_LINES), `... ${details.length - DETAIL_LINES} more`], + onFalse: () => details, + }) + +const dropSummaryOf = (report: DropAuditReport, target: string): ReadonlyArray => { + const drops = report.projects.reduce((sum, project) => sum + project.drops, 0) + return [ + `stryker audit: ${ + outcomeLabel(report) + }, ${drops} drop(s) over ${report.projects.length} project(s), ${report.pairs.length} pair(s)`, + ...report.rules.map(ruleLine), + ...truncatedOf(detailLinesOf(report)), + `report: ${target}`, + ] +} + +const countsLine = (label: string, counts: RunCounts): string => + `${label}: ${counts.planned} planned, ${counts.statuses.CompileError} CompileError (${ + (counts.compileErrorShare * 100).toFixed(1) + }%), ${counts.compiled} compiled, ${counts.executed} executed, ${counts.testExecutions} test executions` + +const countsSummaryOf = (report: CountsReport, target: string): ReadonlyArray => [ + ...report.projects.map((project) => countsLine(project.project, project.counts)), + countsLine('total', report.total), + `report: ${target}`, +] + +const printAll = (channel: AuditChannel, lines: ReadonlyArray): Effect.Effect => + Effect.sync(() => lines.forEach((line) => channel.environment.console.log(line))) + +const auditDropList = ( + request: AuditRequest, + channel: AuditChannel, +): Effect.Effect => + Effect.gen(function*() { + const path = yield* Path.Path + const basePath = channel.environment.basePath + const scope = scopeOf(path, basePath, request.files) + const audited = yield* Effect.forEach( + projectsOf(request.projects), + (directory) => inProjectDirectory(directory, auditProject(request, channel, scope, directory)), + { concurrency: 1 }, + ) + const report = yield* Effect.fromResult(auditDrops(AuditDropsCommand.make({ + scope, + drops: audited.flatMap((project) => project.drops), + matrix: audited.map((project) => project.matrix), + }))) + const target = yield* writeReport(basePath, request.out, report) + yield* printAll(channel, dropSummaryOf(report, target)) + yield* Match.valueTags(report, { + DropAuditPassed: () => Effect.void, + DropAuditFailed: (failed) => Effect.fail(AuditFailed.make({ failures: failed.failures, report: target })), + }) + }) + +const countReports = ( + request: AuditRequest, + channel: AuditChannel, +): Effect.Effect => + Effect.gen(function*() { + const path = yield* Path.Path + const basePath = channel.environment.basePath + yield* Effect.when( + Effect.fail(AuditInputUnusable.make({ + code: 'files-with-counts-only', + detail: '--files scopes the drop audit, which --counts-only does not run', + next: FILES_WITH_COUNTS_NEXT, + })), + Effect.succeed(Option.isSome(Option.filter(Option.fromUndefinedOr(request.files), Arr.isReadonlyArrayNonEmpty))), + ) + const reports = yield* Effect.forEach(projectsOf(request.projects), (project) => + Effect.map( + readReport(CountedReportJson, path.resolve(basePath, request.matrix, project, REPORT_FILE)), + (report) => ({ project, report }), + )) + const counts = yield* Effect.fromResult(countRuns(CountRunsCommand.make({ reports }))) + const target = yield* writeReport(basePath, request.out, counts) + yield* printAll(channel, countsSummaryOf(counts, target)) + }) + +export const auditRequest = ({ request, channel }: AuditRequestInput): Effect.Effect => + Boolean.match(request.countsOnly, { + onTrue: () => countReports(request, channel), + onFalse: () => auditDropList(request, channel), + }) diff --git a/packages/stryker-js/src/audit.schema.ts b/packages/stryker-js/src/audit.schema.ts new file mode 100644 index 000000000..5581ff3be --- /dev/null +++ b/packages/stryker-js/src/audit.schema.ts @@ -0,0 +1,266 @@ +import { Mutant, type Plugin } from '@systemfsoftware/stryker-js-plugin-interface' +import * as S from 'effect/Schema' + +export const AuditSchemaVersion = S.Literal('1') + +const Count = S.Int.check(S.isGreaterThanOrEqualTo(0)) + +export const KillMatrixReport = S.Struct({ + projectRoot: S.String, + files: S.Record( + S.String, + S.Struct({ + source: S.String, + mutants: S.Array(S.Struct({ + id: S.String, + status: Mutant.MutantStatusSchema, + killedBy: S.String.pipe(S.Array, S.optionalKey), + })), + }), + ), + testFiles: S.Record(S.String, S.Struct({ tests: S.Array(S.Struct({ id: S.String })) })), +}) +export type KillMatrixReport = typeof KillMatrixReport.Type +export const KillMatrixReportJson = S.fromJsonString(KillMatrixReport) + +export const CountedReport = S.Struct({ + files: S.Record( + S.String, + S.Struct({ + mutants: S.Array(S.Struct({ + id: S.String, + status: Mutant.MutantStatusSchema, + statusReason: S.String.pipe(S.optionalKey), + testsCompleted: Count.pipe(S.optionalKey), + })), + }), + ), + costs: S.Record(S.String, S.Struct({ actualMs: S.NullOr(S.Finite) })).pipe(S.optionalKey), +}) +export type CountedReport = typeof CountedReport.Type +export const CountedReportJson = S.fromJsonString(CountedReport) + +export const AuditScope = S.Union([ + S.TaggedStruct('Corpus', {}), + S.TaggedStruct('Files', { files: S.NonEmptyArray(S.String) }), +]) +export type AuditScope = typeof AuditScope.Type + +export const PassReason = S.Literals(['killers-contained', 'dropped-mutant-carries-no-kill-signal']) +export const VacuousReason = S.Literals(['dominator-survived', 'dominator-uncovered']) +export const UnverifiedReason = S.Literals([ + 'killer-is-file-hook', + 'killer-unrecorded', + 'dropped-mutant-timed-out', + 'dominator-timed-out', +]) +export const FailReason = S.Literals([ + 'killers-not-contained', + 'dropped-mutant-survives', + 'misidentified-pair', + 'dominator-not-run', +]) +export const UnjoinableReason = S.Literals(['mutant-absent', 'dominator-absent', 'status-unsettled']) + +export const PairVerdict = S.Union([ + S.TaggedStruct('Pass', { reason: PassReason }), + S.TaggedStruct('Vacuous', { reason: VacuousReason }), + S.TaggedStruct('AttributionUnverified', { reason: UnverifiedReason }), + S.TaggedStruct('Fail', { reason: FailReason }), +]) +export type PairVerdict = typeof PairVerdict.Type + +const PairFields = { + project: S.String, + rule: Mutant.Subsumed.fields.rule, + mutant: Mutant.MutantId, + dominator: Mutant.MutantId, + next: S.String, +} + +export const JoinedPair = S.TaggedStruct('JoinedPair', { + ...PairFields, + mutantStatus: Mutant.MutantStatusSchema, + dominatorStatus: Mutant.MutantStatusSchema, + missingKillers: S.Array(S.String), + verdict: PairVerdict, +}) +export type JoinedPair = typeof JoinedPair.Type + +export const UnjoinablePair = S.TaggedStruct('UnjoinablePair', { + ...PairFields, + reason: UnjoinableReason, +}) +export type UnjoinablePair = typeof UnjoinablePair.Type + +export const AuditedPair = S.Union([JoinedPair, UnjoinablePair]) +export type AuditedPair = typeof AuditedPair.Type + +const RuleCounts = { + rule: Mutant.Subsumed.fields.rule, + drops: Count, + pass: Count, + vacuous: Count, + attributionUnverified: Count, + fail: Count, + unjoinable: Count, +} + +export const RuleSummary = S.Union([ + S.TaggedStruct('Attested', RuleCounts), + S.TaggedStruct('Unattested', { ...RuleCounts, reason: S.Literal('no-drop-joined'), next: S.String }), +]) +export type RuleSummary = typeof RuleSummary.Type + +export const OrphanedTest = S.Struct({ + project: S.String, + test: S.String, + killed: S.NonEmptyArray(S.String), + reason: S.Literal('every-killed-mutant-dropped'), + next: S.String, +}) +export type OrphanedTest = typeof OrphanedTest.Type + +export const ProjectDrops = S.Struct({ + project: S.String, + matrixMutants: Count, + drops: Count, +}) + +const DropAuditFields = { + schemaVersion: AuditSchemaVersion, + scope: AuditScope, + projects: S.Array(ProjectDrops), + rules: S.Array(RuleSummary), + pairs: S.Array(AuditedPair), + orphanedTests: S.Array(OrphanedTest), +} + +export const AuditFailures = S.Struct({ pairs: Count, unattestedRules: Count, orphanedTests: Count }) + +const AuditReportTypeId: unique symbol = Symbol.for('@systemfsoftware/stryker-js/AuditReport') + +export class DropAuditPassed extends S.TaggedClass()('DropAuditPassed', DropAuditFields) { + readonly [AuditReportTypeId] = AuditReportTypeId +} + +export class DropAuditFailed extends S.TaggedClass()('DropAuditFailed', { + ...DropAuditFields, + failures: AuditFailures, +}) { + readonly [AuditReportTypeId] = AuditReportTypeId +} + +export const DropAuditReport = S.Union([DropAuditPassed, DropAuditFailed]) +export type DropAuditReport = typeof DropAuditReport.Type + +export const MatrixMutant = S.Struct({ + id: S.String, + status: Mutant.MutantStatusSchema, + killedBy: S.Array(S.String), +}) +export type MatrixMutant = typeof MatrixMutant.Type + +export const MatrixProject = S.Struct({ + project: S.String, + mutants: S.Array(MatrixMutant), + tests: S.Array(S.String), +}) +export type MatrixProject = typeof MatrixProject.Type + +export const AuditedDrop = S.Struct({ + project: S.String, + mutant: Mutant.MutantId, + subsumed: Mutant.Subsumed, +}) +export type AuditedDrop = typeof AuditedDrop.Type + +export const StatusCounts = S.Struct({ + Killed: Count, + Survived: Count, + NoCoverage: Count, + CompileError: Count, + RuntimeError: Count, + Timeout: Count, + Ignored: Count, + Pending: Count, +}) +export type StatusCounts = typeof StatusCounts.Type + +export const RunCounts = S.Struct({ + planned: Count, + statuses: StatusCounts, + ignoredByRule: S.Array(S.Struct({ rule: Mutant.IgnoreRuleId, count: Count })), + ignoredUnrecognized: Count, + compileErrorShare: S.Finite, + compiled: Count, + executed: Count, + testExecutions: Count, + compileErrorCheckerMs: S.Finite, +}) +export type RunCounts = typeof RunCounts.Type + +export class CountsReport extends S.TaggedClass()('Counts', { + schemaVersion: AuditSchemaVersion, + projects: S.Array(S.Struct({ project: S.String, counts: RunCounts })), + total: RunCounts, +}) { + readonly [AuditReportTypeId] = AuditReportTypeId +} + +export const AuditReport = S.Union([DropAuditPassed, DropAuditFailed, CountsReport]) +export type AuditReport = typeof AuditReport.Type +export const AuditReportJson = S.fromJsonString(AuditReport) + +export const CountedProject = S.Struct({ project: S.String, report: CountedReport }) + +export class NothingCounted extends S.TaggedError()('NothingCounted', { + projects: S.Array(S.String), +}) { + readonly exitClass = 'ConfigError' satisfies Plugin.ExitClass + readonly code = 'no-mutant-recorded' as const + + get reason(): string { + return [ + `stryker audit (${this.code}): the incremental reports of ${this.projects.join(', ')} record no mutant.`, + "Pass --matrix the directory of a finished run's reports and --projects the directories it ran.", + ].join(' ') + } + + override get message(): string { + return this.reason + } +} + +export class AuditFailed extends S.TaggedError()('AuditFailed', { + failures: AuditFailures, + report: S.String, +}) { + readonly exitClass = 'VerdictFail' satisfies Plugin.ExitClass + + override get message(): string { + return [ + `stryker audit: dropping these mutants loses kill signal: ${this.failures.pairs} failed pair(s),`, + `${this.failures.unattestedRules} unattested rule(s), ${this.failures.orphanedTests} orphaned test(s).`, + `Each entry in ${this.report} carries its reason code and next action.`, + ].join(' ') + } +} + +export const AuditInputCode = S.Literals(['matrix-unreadable', 'matrix-undecodable', 'files-with-counts-only']) + +export class AuditInputUnusable extends S.TaggedError()('AuditInputUnusable', { + code: AuditInputCode, + detail: S.String, + next: S.String, +}) { + readonly exitClass = 'ConfigError' satisfies Plugin.ExitClass + + get reason(): string { + return `stryker audit (${this.code}): ${this.detail}. ${this.next}` + } + + override get message(): string { + return this.reason + } +} diff --git a/packages/stryker-js/src/bin/cli-command.ts b/packages/stryker-js/src/bin/cli-command.ts index 176457f44..ec179fd30 100644 --- a/packages/stryker-js/src/bin/cli-command.ts +++ b/packages/stryker-js/src/bin/cli-command.ts @@ -437,6 +437,35 @@ const planOptions = { ), } +const auditOptions = { + matrix: Flag.String('matrix').pipe( + Flag.withDescription( + 'The directory holding /stryker-incremental.json for each project: a kill-matrix artifact for the drop audit, any finished run for --counts-only.', + ), + ), + out: Flag.String('out').pipe( + Flag.withDescription('Write the versioned audit JSON to this file.'), + Flag.withDefault('reports/audit.json'), + ), + countsOnly: Flag.map(optional(Flag.Boolean('counts-only')), absentWhenFalse).pipe( + Flag.withDescription('Count planned, CompileError, compiled and executed mutants from the reports alone.'), + ), + projects: Flag.String('projects').pipe( + Flag.withDescription( + 'A comma separated list of project directories to audit; defaults to the current working directory.', + ), + Flag.map(splitOnComma), + optional, + ), + files: Flag.String('files').pipe( + Flag.withDescription( + 'A comma separated list of files, relative to the working directory, to scope the drop audit to; defaults to every file in the matrix.', + ), + Flag.map(splitOnComma), + optional, + ), +} + const serveOptions = { port: Flag.Int('port').pipe( Flag.withDescription('The port the socket channel listens on. Required for the `socket` channel.'), @@ -633,6 +662,26 @@ export const makeStrykerCommand = ({ environment, recordAnswer }: { ), ) + const auditCommand = Command.make('audit', auditOptions, (config) => + runRequestCell.run({ + route: CliRouteCommand.make({ + route: { + _tag: 'audit', + matrix: config.matrix, + out: config.out, + countsOnly: config.countsOnly === true, + projects: Option.getOrUndefined(config.projects), + files: Option.getOrUndefined(config.files), + }, + }), + options: {}, + environment, + }).pipe(Effect.provideService(Console.Console, environment.console), Effect.flatMap(recordAnswer))).pipe( + Command.withDescription( + 'Check every mutant the default policy drops against a kill matrix, so no test loses its only kill, or count a finished run', + ), + ) + const serveCommand = Command.make('serve', { ...serveOptions, ...serveArgs }, (config) => runRequestCell.run({ route: CliRouteCommand.make({ @@ -730,6 +779,7 @@ export const makeStrykerCommand = ({ environment, recordAnswer }: { gateCommand, annotateCommand, planCommand, + auditCommand, serveCommand, feedbackCommand, mcpCommand, diff --git a/packages/stryker-js/src/count-runs.workflow.ts b/packages/stryker-js/src/count-runs.workflow.ts new file mode 100644 index 000000000..b73d707c7 --- /dev/null +++ b/packages/stryker-js/src/count-runs.workflow.ts @@ -0,0 +1,118 @@ +import { Workflow } from '@systemfsoftware/effect-cell-types' +import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' +import * as Arr from 'effect/Array' +import * as Boolean from 'effect/Boolean' +import * as Number from 'effect/Number' +import * as Option from 'effect/Option' +import * as Order from 'effect/Order' +import * as Record from 'effect/Record' +import * as Result from 'effect/Result' +import * as S from 'effect/Schema' + +import { + CountedProject, + type CountedReport, + CountsReport, + NothingCounted, + type RunCounts, + type StatusCounts, +} from './audit.schema.js' + +export class CountRunsCommand extends S.TaggedClass()('CountRunsCommand', { + reports: S.Array(CountedProject), +}) { + static readonly [Workflow.InstrumentationBrand] = {} as const +} + +type CountedMutant = CountedReport['files'][string]['mutants'][number] + +interface Counted { + readonly mutant: CountedMutant + readonly checkerMs: number +} + +const decodeIgnoreReason = S.decodeOption(Mutant.IgnoreStatusReason) + +const checkerMsOf = (report: CountedReport, id: string): number => + Option.getOrElse( + Option.flatMap( + Option.flatMap(Option.fromUndefinedOr(report.costs), (costs) => Record.get(costs, id)), + (cost) => Option.fromNullOr(cost.actualMs), + ), + () => 0, + ) + +const countedOf = (report: CountedReport): ReadonlyArray => + Object.values(report.files).flatMap((file) => + file.mutants.map((mutant) => ({ mutant, checkerMs: checkerMsOf(report, mutant.id) })) + ) + +const statusCountsOf = (mutants: ReadonlyArray): StatusCounts => { + const count = (status: Mutant.MutantStatus): number => mutants.filter((mutant) => mutant.status === status).length + return { + Killed: count('Killed'), + Survived: count('Survived'), + NoCoverage: count('NoCoverage'), + CompileError: count('CompileError'), + RuntimeError: count('RuntimeError'), + Timeout: count('Timeout'), + Ignored: count('Ignored'), + Pending: count('Pending'), + } +} + +const ignoreRuleOf = (mutant: CountedMutant): Option.Option => + Option.map(Option.flatMap(Option.fromUndefinedOr(mutant.statusReason), decodeIgnoreReason), (reason) => reason.ruleId) + +const runCountsOf = (reports: ReadonlyArray): RunCounts => { + const counted = reports.flatMap(countedOf) + const mutants = counted.map((entry) => entry.mutant) + const statuses = statusCountsOf(mutants) + const ignoreRules = mutants.filter((mutant) => mutant.status === 'Ignored').map(ignoreRuleOf) + const recognized = Arr.getSomes(ignoreRules) + const planned = mutants.length + return { + planned, + statuses, + ignoredByRule: Arr.sort( + Object.values(Arr.groupBy(recognized, (rule) => rule)).map((members) => ({ + rule: members[0], + count: members.length, + })), + Order.mapInput(Order.String, (entry: { readonly rule: string }) => entry.rule), + ), + ignoredUnrecognized: ignoreRules.length - recognized.length, + compileErrorShare: Boolean.match(planned === 0, { + onTrue: () => 0, + onFalse: () => statuses.CompileError / planned, + }), + compiled: planned - statuses.CompileError - statuses.Ignored - statuses.Pending, + executed: statuses.Killed + statuses.Survived + statuses.Timeout + statuses.RuntimeError, + testExecutions: Number.sumAll( + mutants.map((mutant) => Option.getOrElse(Option.fromUndefinedOr(mutant.testsCompleted), () => 0)), + ), + compileErrorCheckerMs: Number.sumAll( + counted.filter((entry) => entry.mutant.status === 'CompileError').map((entry) => entry.checkerMs), + ), + } +} + +const decide = (command: CountRunsCommand): Result.Result => { + const total = runCountsOf(command.reports.map((entry) => entry.report)) + return Boolean.match(total.planned === 0, { + onTrue: () => Result.fail(NothingCounted.make({ projects: command.reports.map((entry) => entry.project) })), + onFalse: () => + Result.succeed(CountsReport.make({ + schemaVersion: '1', + projects: command.reports.map((entry) => ({ project: entry.project, counts: runCountsOf([entry.report]) })), + total, + })), + }) +} + +export const countRuns = Workflow.make({ + command: CountRunsCommand, + decision: CountsReport, + error: NothingCounted, + decide, +}) diff --git a/packages/stryker-js/src/route-cli-request.workflow.ts b/packages/stryker-js/src/route-cli-request.workflow.ts index 7ccb5b763..b1deb4641 100644 --- a/packages/stryker-js/src/route-cli-request.workflow.ts +++ b/packages/stryker-js/src/route-cli-request.workflow.ts @@ -89,6 +89,16 @@ export class CliPlanRequested extends S.TaggedClass()('CliPlan readonly [CliRouteDecisionTypeId] = CliRouteDecisionTypeId } +export class CliAuditRequested extends S.TaggedClass()('CliAuditRequested', { + matrix: S.String, + out: S.String, + countsOnly: S.Boolean, + projects: S.Array(S.String).pipe(S.optional), + files: S.Array(S.String).pipe(S.optional), +}) { + readonly [CliRouteDecisionTypeId] = CliRouteDecisionTypeId +} + export class CliServeRequested extends S.TaggedClass()('CliServeRequested', { channel: ServeChannelSchema, port: S.optional(S.Int), @@ -121,6 +131,7 @@ export type CliRouteDecision = | CliGateRequested | CliAnnotateRequested | CliPlanRequested + | CliAuditRequested | CliServeRequested | CliFeedbackRequested | CliMcpRequested @@ -139,6 +150,7 @@ export const routeCliRequest = Workflow.make({ CliGateRequested, CliAnnotateRequested, CliPlanRequested, + CliAuditRequested, CliServeRequested, CliFeedbackRequested, CliMcpRequested, @@ -175,6 +187,16 @@ export const routeCliRequest = Workflow.make({ full: plan.full, }), )), + Match.tag('audit', (audit) => + Result.succeed( + CliAuditRequested.make({ + matrix: audit.matrix, + out: audit.out, + countsOnly: audit.countsOnly, + projects: audit.projects, + files: audit.files, + }), + )), Match.tag('serve', (serve) => Result.succeed( CliServeRequested.make({ channel: serve.channel, port: serve.port, address: serve.address }), diff --git a/packages/stryker-js/src/run-request.cell.ts b/packages/stryker-js/src/run-request.cell.ts index b613c16e8..fcc0dec29 100644 --- a/packages/stryker-js/src/run-request.cell.ts +++ b/packages/stryker-js/src/run-request.cell.ts @@ -18,6 +18,8 @@ import * as S from 'effect/Schema' import type { SchemaError } from 'effect/Schema' import { type Admitted } from './admit-survivors-run.workflow.js' +import { auditRequest } from './audit-request.cell.js' +import type { AuditFailed, AuditInputUnusable, NothingCounted } from './audit.schema.js' import { Baseline } from './Baseline.schema.js' import { BudgetExceeded, budgetGate, BudgetGateCommand, BudgetInputUnusable } from './budget-gate.workflow.js' import { BudgetBaseline } from './BudgetBaseline.schema.js' @@ -116,6 +118,9 @@ export type CliFailure = | BudgetExceeded | BudgetInputUnusable | AnnotationsUnusable + | AuditFailed + | AuditInputUnusable + | NothingCounted | ShardPlanInvalid | ShardUnknown | ShardChildFailed @@ -651,6 +656,7 @@ export const runRequestCell = Sandwich.named(SpanTaxonomy.Spans.runRequest.name) CliGateRequested: (gate, channel) => gateReport(gate, channel).pipe(Effect.tapError(remediateGateRefusal)), CliAnnotateRequested: (annotate, channel) => annotateReport(annotate, channel), CliPlanRequested: (plan, channel) => planRequest({ request: plan, channel }), + CliAuditRequested: (audit, channel) => auditRequest({ request: audit, channel }), CliFeedbackRequested: (feedback, channel) => feedbackRoute(feedback, channel), CliMcpRequested: (_, channel) => Layer.launch(mcpServerLayer({ basePath: channel.environment.basePath })).pipe(Effect.scoped, Effect.orDie), diff --git a/packages/stryker-js/src/run/instrument.ts b/packages/stryker-js/src/run/instrument.ts index bca628cb5..2cc275289 100644 --- a/packages/stryker-js/src/run/instrument.ts +++ b/packages/stryker-js/src/run/instrument.ts @@ -81,6 +81,35 @@ const withInstrumentedFiles = ( ), ) +const instrumentWith = ( + command: PrepareForInstrument, + filesToMutate: ReadonlyArray, +): Effect.Effect => + Instrument.instrument([...filesToMutate], { + ignorers: [...command.ignorers], + excludedMutations: [...command.mutatorSelection.excludedMutations], + mutantSetPolicy: command.options.mutator.mutantSetPolicy, + mutators: Mutator.selectMutators( + Mutator.registryOf( + command.mutatorCatalogs, + command.loadedPlugins.mutators.map(({ contribution }) => contribution), + ), + command.mutatorSelection.optInMutations, + ), + }, command.formatRegistry).pipe( + Effect.mapError((cause) => StageError.make({ stage: 'instrument', reason: 'Instrumenter failed', cause })), + ) + +export const instrumentSources: { + ( + filesToMutate: ReadonlyArray, + ): (command: PrepareForInstrument) => Effect.Effect + ( + command: PrepareForInstrument, + filesToMutate: ReadonlyArray, + ): Effect.Effect +} = dual(2, instrumentWith) + export const instrumentFiles = Effect.fnUntraced(function*( command: PrepareForInstrument, ): Effect.fn.Return< @@ -101,22 +130,7 @@ export const instrumentFiles = Effect.fnUntraced(function*( StageError.make({ stage: 'instrument', reason: 'Failed to read files to mutate', cause }) ), ) - - const { excludedMutations, optInMutations } = command.mutatorSelection - const instrumentResult = yield* Instrument.instrument(filesToMutate, { - ignorers: [...command.ignorers], - excludedMutations: [...excludedMutations], - mutantSetPolicy: command.options.mutator.mutantSetPolicy, - mutators: Mutator.selectMutators( - Mutator.registryOf( - command.mutatorCatalogs, - command.loadedPlugins.mutators.map(({ contribution }) => contribution), - ), - optInMutations, - ), - }, command.formatRegistry).pipe( - Effect.mapError((cause) => StageError.make({ stage: 'instrument', reason: 'Instrumenter failed', cause })), - ) + const instrumentResult = yield* instrumentSources(command, filesToMutate) return { filesToMutate, diff --git a/packages/stryker-js/tests/audit.integration.test.ts b/packages/stryker-js/tests/audit.integration.test.ts new file mode 100644 index 000000000..4f44ac8f5 --- /dev/null +++ b/packages/stryker-js/tests/audit.integration.test.ts @@ -0,0 +1,288 @@ +import { Gherkin, Given, it, makeFeature, Then, When } from '@systemfsoftware/effect-gherkin-spec' +import { Engine } from '@systemfsoftware/stryker-js' +import * as Arr from 'effect/Array' +import * as Effect from 'effect/Effect' +import * as Fiber from 'effect/Fiber' +import * as FileSystem from 'effect/FileSystem' +import * as Option from 'effect/Option' +import * as Path from 'effect/Path' +import * as ChildProcess from 'effect/process/ChildProcess' +import * as ChildProcessSpawner from 'effect/process/ChildProcessSpawner' +import * as S from 'effect/Schema' +import * as Scope from 'effect/Scope' +import * as Stream from 'effect/Stream' + +const Feature = makeFeature({ it }) + +const PACKAGE_ROOT = decodeURIComponent(new URL('..', import.meta.url).pathname) +const STRYKER_BIN = decodeURIComponent(new URL('../dist/main.mjs', import.meta.url).pathname) + +const CONSUMER_PACKAGE = '{ "name": "audit-consumer", "type": "module", "private": true }\n' +const CONFIG = `export default { + testRunner: 'command', + commandRunner: { command: 'true' }, + mutate: ['src/**/*.js'], + coverageAnalysis: 'off', + concurrency: 1, + reporters: ['clear-text'], +} +` +const SOURCE_FILE = 'src/sign.js' +const SOURCE = 'export const isPositive = (n) => n > 0\n' +const NAMED_TEST = 'test/sign.test.js#isPositive keeps zero out' +const OTHER_TEST = 'test/sign.test.js#isPositive admits one' +const MATRIX_DIR = 'matrix' +const AUDIT_OUT = 'reports/audit.json' + +interface MatrixRecord { + readonly id: string + readonly status: string + readonly killedBy: ReadonlyArray +} + +interface AuditRun { + readonly exitCode: number + readonly stdout: string + readonly stderr: string +} + +const decodeDropAudit = S.decodeUnknownEffect(S.fromJsonString(S.Struct({ + pairs: S.Array(S.Struct({ + mutant: S.String, + dominator: S.String, + verdict: S.optional(S.Struct({ reason: S.String })), + reason: S.optional(S.String), + })), +}))) + +const decodeCounts = S.decodeUnknownEffect(S.fromJsonString(S.Struct({ + total: S.Struct({ planned: S.Int, compiled: S.Int, statuses: S.Struct({ CompileError: S.Int, Killed: S.Int }) }), +}))) + +const matrixJson = (root: string, mutants: ReadonlyArray): string => + `${ + JSON.stringify({ + projectRoot: root, + files: { [SOURCE_FILE]: { source: SOURCE, mutants } }, + testFiles: { 'test/sign.test.js': { tests: [{ id: NAMED_TEST }, { id: OTHER_TEST }] } }, + }) + }\n` + +const prepareProject = (): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + const binaryPresent = yield* fs.exists(STRYKER_BIN) + yield* Effect.when( + Effect.die(new Error('dist/main.mjs is missing - build the package first')), + Effect.succeed(!binaryPresent), + ) + const root = yield* fs.realPath(yield* fs.makeTempDirectoryScoped({ prefix: 'stryker-audit-' })) + yield* fs.makeDirectory(path.join(root, 'node_modules', '@systemfsoftware'), { recursive: true }) + yield* fs.symlink(PACKAGE_ROOT, path.join(root, 'node_modules', '@systemfsoftware', 'stryker-js')) + yield* fs.makeDirectory(path.join(root, 'src')) + yield* fs.makeDirectory(path.join(root, MATRIX_DIR)) + yield* fs.writeFileString(path.join(root, 'package.json'), CONSUMER_PACKAGE) + yield* fs.writeFileString(path.join(root, 'stryker.config.mjs'), CONFIG) + yield* fs.writeFileString(path.join(root, SOURCE_FILE), SOURCE) + return root + }).pipe(Effect.orDie) + +const writeMatrix = ( + root: string, + mutants: ReadonlyArray, +): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + yield* fs.writeFileString(path.join(root, MATRIX_DIR, 'stryker-incremental.json'), matrixJson(root, mutants)) + }).pipe(Effect.orDie) + +const runAudit = ( + root: string, + args: ReadonlyArray, +): Effect.Effect => + Effect.scoped( + Effect.gen(function*() { + const spawner = yield* ChildProcessSpawner.ChildProcessSpawner + const handle = yield* spawner.spawn( + ChildProcess.make(globalThis.process.execPath, [STRYKER_BIN, 'audit', '--out', AUDIT_OUT, ...args], { + cwd: root, + stdin: 'ignore', + stdout: 'pipe', + stderr: 'pipe', + env: { STRYKER_MODE: 'human', NO_COLOR: '1' }, + extendEnv: true, + }), + ) + const printedOut = yield* Effect.forkScoped(handle.stdout.pipe(Stream.decodeText, Stream.mkString)) + const printedErr = yield* Effect.forkScoped(handle.stderr.pipe(Stream.decodeText, Stream.mkString)) + const exitCode = yield* handle.exitCode + const [out, err] = yield* Effect.all([Fiber.join(printedOut), Fiber.join(printedErr)]) + return { exitCode: Number(exitCode), stdout: out, stderr: err } + }), + ).pipe(Effect.orDie) + +const readAuditText = (root: string): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + return yield* fs.readFileString(path.join(root, AUDIT_OUT)) + }).pipe(Effect.orDie) + +interface AuditedPairView { + readonly mutant: string + readonly dominator: string + readonly verdict?: { readonly reason: string } | undefined + readonly reason?: string | undefined +} + +interface DropAuditView { + readonly pairs: ReadonlyArray +} + +interface AuditedRun { + readonly ran: AuditRun + readonly audit: DropAuditView +} + +interface DroppedPair { + readonly mutant: string + readonly dominator: string +} + +type AuditServices = FileSystem.FileSystem | Path.Path | ChildProcessSpawner.ChildProcessSpawner + +const readDropAudit = (root: string): Effect.Effect => + Effect.flatMap(readAuditText(root), decodeDropAudit).pipe(Effect.orDie) + +const discoverDroppedPair = (root: string): Effect.Effect => + Effect.gen(function*() { + yield* writeMatrix(root, []) + yield* runAudit(root, ['--matrix', MATRIX_DIR]) + const audit = yield* readDropAudit(root) + return yield* Option.match(Arr.head(audit.pairs), { + onNone: () => Effect.die(new Error('the audit named no dropped mutant for the relational comparison')), + onSome: ({ mutant, dominator }) => Effect.succeed({ mutant, dominator }), + }) + }) + +const auditWith = ( + root: string, + pair: DroppedPair, + killers: { readonly mutant: ReadonlyArray; readonly dominator: ReadonlyArray }, +): Effect.Effect => + Effect.gen(function*() { + yield* writeMatrix(root, [ + { id: pair.mutant, status: 'Killed', killedBy: killers.mutant }, + { id: pair.dominator, status: 'Killed', killedBy: killers.dominator }, + ]) + const ran = yield* runAudit(root, ['--matrix', MATRIX_DIR]) + const audit = yield* readDropAudit(root) + return { ran, audit } + }) + +const reasonsOf = (audit: DropAuditView): ReadonlyArray => + audit.pairs.map((pair) => pair.verdict?.reason ?? pair.reason ?? '') + +Feature('Auditing the mutants the default policy drops against a kill matrix', { timeout: 180_000 }) + .withLayer(Engine.nodePlatformLayer) + .live('the built stryker binary instruments the matrix sources and joins every drop in a real Node process') + .body(({ scenario }) => { + scenario( + 'A drop whose dominator is killed by a named test that spares the dropped mutant fails the audit', + Gherkin.Do.pipe( + Given('a project whose one relational comparison drops its complement')('root', () => prepareProject()), + Given('the drop and its dominator as the audit names them')('pair', (s) => discoverDroppedPair(s.root)), + When('the matrix records a named test that kills only the dominator')( + 'audited', + (s) => auditWith(s.root, s.pair, { mutant: [OTHER_TEST], dominator: [NAMED_TEST, OTHER_TEST] }), + ), + Then('the process exits 1 and the pair fails because the killers are not contained')((s, expect) => + expect({ + exitCode: s.audited.ran.exitCode, + reasons: reasonsOf(s.audited.audit), + }).toStrictEqual({ exitCode: 1, reasons: ['killers-not-contained'] }) + ), + ), + ) + + scenario( + 'A drop killed by every test that kills its dominator passes the audit', + Gherkin.Do.pipe( + Given('a project whose one relational comparison drops its complement')('root', () => prepareProject()), + Given('the drop and its dominator as the audit names them')('pair', (s) => discoverDroppedPair(s.root)), + When('the matrix records the same named test killing both')( + 'audited', + (s) => auditWith(s.root, s.pair, { mutant: [NAMED_TEST], dominator: [NAMED_TEST] }), + ), + Then('the process exits 0 and the pair passes on contained killers')((s, expect) => + expect({ + exitCode: s.audited.ran.exitCode, + reasons: reasonsOf(s.audited.audit), + }).toStrictEqual({ exitCode: 0, reasons: ['killers-contained'] }) + ), + ), + ) + + scenario( + 'A matrix that records neither mutant leaves the rule unattested and fails the audit', + Gherkin.Do.pipe( + Given('a project whose one relational comparison drops its complement')('root', () => prepareProject()), + Given('a matrix holding the source but no mutant')('matrix', (s) => writeMatrix(s.root, [])), + When('stryker audits the drops')('ran', (s) => runAudit(s.root, ['--matrix', MATRIX_DIR])), + When('the audit report is read back')('audit', (s) => readDropAudit(s.root)), + Then('the process exits 1 because the matrix has no record of the dropped mutant')((s, expect) => + expect({ exitCode: s.ran.exitCode, reasons: reasonsOf(s.audit) }).toStrictEqual({ + exitCode: 1, + reasons: ['mutant-absent'], + }) + ), + ), + ) + + scenario( + 'Counting a finished run reads planned, CompileError and compiled from the reports alone', + Gherkin.Do.pipe( + Given('a project')('root', () => prepareProject()), + Given('a report with one Killed, one CompileError and one Ignored mutant')( + 'matrix', + (s) => + writeMatrix(s.root, [ + { id: '1a1a1a1a1a1a1a1a', status: 'Killed', killedBy: [NAMED_TEST] }, + { id: '2b2b2b2b2b2b2b2b', status: 'CompileError', killedBy: [] }, + { id: '3c3c3c3c3c3c3c3c', status: 'Ignored', killedBy: [] }, + ]), + ), + When('stryker audits with --counts-only')( + 'ran', + (s) => runAudit(s.root, ['--counts-only', '--matrix', MATRIX_DIR]), + ), + When('the counts are read back')( + 'counts', + (s) => Effect.flatMap(readAuditText(s.root), decodeCounts).pipe(Effect.orDie), + ), + Then('the process exits 0 and only the Killed mutant counts as compiled')((s, expect) => + expect({ exitCode: s.ran.exitCode, counts: s.counts }).toStrictEqual({ + exitCode: 0, + counts: { total: { planned: 3, compiled: 1, statuses: { CompileError: 1, Killed: 1 } } }, + }) + ), + ), + ) + + scenario( + 'A matrix directory without the project report is an input failure that names the fix', + Gherkin.Do.pipe( + Given('a project with an empty matrix directory')('root', () => prepareProject()), + When('stryker audits against it')('ran', (s) => runAudit(s.root, ['--matrix', MATRIX_DIR])), + Then('the process exits 2 and stderr names the unreadable matrix and where it should point')((s, expect) => + expect({ + exitCode: s.ran.exitCode, + namesTheCode: s.ran.stderr.includes('matrix-unreadable'), + namesTheFix: s.ran.stderr.includes('Pass --matrix the directory'), + }).toStrictEqual({ exitCode: 2, namesTheCode: true, namesTheFix: true }) + ), + ), + ) + }) From 22d4f0c8739c164e3b8497ed2fff68cdd78a543a Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 08:37:15 +0000 Subject: [PATCH 02/12] refactor(stryker-js): simplify the drop audit's tallies and lookups One reduce per rule tallies the pair verdicts; test catalogs and dropped ids become record lookups; drops are grouped by project once; the Reason union derives from the verdict schemas; the cell reuses PlanChannel; six schemas used only inside audit.schema.ts lose their export. The audit JSON over kill-matrix-420 and the counts over mutation-report-416 are byte-identical before and after. --- .../stryker-js/src/audit-drops.workflow.ts | 85 ++++++++++--------- packages/stryker-js/src/audit-request.cell.ts | 14 +-- packages/stryker-js/src/audit.schema.ts | 14 ++- 3 files changed, 56 insertions(+), 57 deletions(-) diff --git a/packages/stryker-js/src/audit-drops.workflow.ts b/packages/stryker-js/src/audit-drops.workflow.ts index 2157db26c..e9a262e1c 100644 --- a/packages/stryker-js/src/audit-drops.workflow.ts +++ b/packages/stryker-js/src/audit-drops.workflow.ts @@ -36,12 +36,7 @@ export class AuditDropsCommand extends S.TaggedClass()('Audit static readonly [Workflow.InstrumentationBrand] = {} as const } -type Reason = - | typeof PassReason.Type - | typeof VacuousReason.Type - | typeof UnverifiedReason.Type - | typeof FailReason.Type - | typeof UnjoinableReason.Type +type Reason = PairVerdict['reason'] | typeof UnjoinableReason.Type const FIX_THE_RULE = "Fix the subsumption rule so it keeps this mutant, or set `mutator.mutantSetPolicy: 'full'`." const RERUN_MATRIX = 'Re-run the kill-matrix lane on the current main, then audit again.' @@ -82,7 +77,7 @@ const ORPHANED_NEXT = interface ProjectIndex { readonly mutants: Record.ReadonlyRecord - readonly tests: ReadonlyArray + readonly tests: Record.ReadonlyRecord } interface PairKey { @@ -96,12 +91,12 @@ interface JoinedFacts { readonly mutant: MatrixMutant readonly dominator: MatrixMutant readonly missingKillers: ReadonlyArray - readonly tests: ReadonlyArray + readonly tests: Record.ReadonlyRecord } type Judged = Result.Result -const EMPTY_INDEX: ProjectIndex = { mutants: {}, tests: [] } +const EMPTY_INDEX: ProjectIndex = { mutants: {}, tests: {} } const pass = (reason: typeof PassReason.Type): Judged => Result.succeed({ _tag: 'Pass', reason }) const vacuous = (reason: typeof VacuousReason.Type): Judged => Result.succeed({ _tag: 'Vacuous', reason }) @@ -110,10 +105,16 @@ const unverified = (reason: typeof UnverifiedReason.Type): Judged => const fail = (reason: typeof FailReason.Type): Judged => Result.succeed({ _tag: 'Fail', reason }) const unsettled = (): Judged => Result.fail('status-unsettled') +const lookupOf = (keys: ReadonlyArray): Record.ReadonlyRecord => + Object.fromEntries(keys.map((key) => [key, true])) + const indexOf = (matrix: ReadonlyArray): Record.ReadonlyRecord => Object.fromEntries(matrix.map((project) => [ project.project, - { mutants: Object.fromEntries(project.mutants.map((mutant) => [mutant.id, mutant])), tests: project.tests }, + { + mutants: Object.fromEntries(project.mutants.map((mutant) => [mutant.id, mutant])), + tests: lookupOf(project.tests), + }, ])) const containment = (facts: JoinedFacts): Judged => @@ -123,7 +124,7 @@ const containment = (facts: JoinedFacts): Judged => Boolean.match(facts.missingKillers.length === 0, { onTrue: () => pass('killers-contained'), onFalse: () => - Boolean.match(facts.missingKillers.some((killer) => facts.tests.includes(killer)), { + Boolean.match(facts.missingKillers.some((killer) => Record.has(facts.tests, killer)), { onTrue: () => fail('killers-not-contained'), onFalse: () => unverified('killer-is-file-hook'), }), @@ -243,22 +244,22 @@ const tallyOf = (pair: AuditedPair): Tally => }), }) -const tallyCount = (pairs: ReadonlyArray, tally: Tally): number => - pairs.filter((pair) => tallyOf(pair) === tally).length +type TallyCounts = { readonly [tally in Tally]: number } + +const NO_TALLIES: TallyCounts = { pass: 0, vacuous: 0, attributionUnverified: 0, fail: 0, unjoinable: 0 } + +const tallyCountsOf = (pairs: ReadonlyArray): TallyCounts => + pairs.reduce((counts, pair) => { + const tally = tallyOf(pair) + return { ...counts, [tally]: counts[tally] + 1 } + }, NO_TALLIES) const ruleSummaryOf = ( - drops: Arr.NonEmptyReadonlyArray, + rule: Mutant.Subsumed['rule'], + drops: number, pairs: ReadonlyArray, ): RuleSummary => { - const counts = { - rule: Arr.headNonEmpty(drops).subsumed.rule, - drops: drops.length, - pass: tallyCount(pairs, 'pass'), - vacuous: tallyCount(pairs, 'vacuous'), - attributionUnverified: tallyCount(pairs, 'attributionUnverified'), - fail: tallyCount(pairs, 'fail'), - unjoinable: tallyCount(pairs, 'unjoinable'), - } + const counts = { rule, drops, ...tallyCountsOf(pairs) } return Boolean.match(pairs.length > counts.unjoinable, { onTrue: (): RuleSummary => ({ _tag: 'Attested', ...counts }), onFalse: (): RuleSummary => ({ _tag: 'Unattested', ...counts, reason: 'no-drop-joined', next: UNATTESTED_NEXT }), @@ -268,16 +269,18 @@ const ruleSummaryOf = ( const isUnattested = (rule: RuleSummary): boolean => Match.valueTags(rule, { Attested: () => false, Unattested: () => true }) -const orphansOf = (project: MatrixProject, drops: ReadonlyArray): ReadonlyArray => { - const dropped: ReadonlyArray = drops.filter((drop) => drop.project === project.project).map((drop) => - drop.mutant - ) +const orphansOf = ( + project: MatrixProject, + index: ProjectIndex, + dropped: ReadonlyArray, +): ReadonlyArray => { + const droppedIds = lookupOf(dropped.map((drop) => drop.mutant)) const kills = project.mutants .filter((mutant) => mutant.status === 'Killed') .flatMap((mutant) => mutant.killedBy.map((test) => ({ test, mutant: mutant.id }))) - .filter((kill) => project.tests.includes(kill.test)) + .filter((kill) => Record.has(index.tests, kill.test)) return Object.entries(Arr.groupBy(kills, (kill) => kill.test)).flatMap(([test, killedBy]) => - Boolean.match(killedBy.every((kill) => dropped.includes(kill.mutant)), { + Boolean.match(killedBy.every((kill) => Record.has(droppedIds, kill.mutant)), { onTrue: (): ReadonlyArray => [{ project: project.project, test, @@ -299,30 +302,36 @@ const decide = (command: AuditDropsCommand): DropAuditReport => { const indexes = indexOf(command.matrix) const pairs = Arr.sort(command.drops.flatMap(pairKeysOf).map((key) => auditPair(indexes, key)), pairOrder) const pairsByRule = Arr.groupBy(pairs, (pair) => pair.rule) + const dropsByProject = Arr.groupBy(command.drops, (drop) => drop.project) + const dropsOf = (project: MatrixProject): ReadonlyArray => + Option.getOrElse(Record.get(dropsByProject, project.project), () => []) const rules = Arr.sort( - Object.values(Arr.groupBy(command.drops, (drop) => drop.subsumed.rule)).map((drops) => - ruleSummaryOf( - drops, - Option.getOrElse(Option.fromUndefinedOr(pairsByRule[Arr.headNonEmpty(drops).subsumed.rule]), () => []), - ) - ), + Object.values(Arr.groupBy(command.drops, (drop) => drop.subsumed.rule)).map((drops) => { + const rule = Arr.headNonEmpty(drops).subsumed.rule + return ruleSummaryOf(rule, drops.length, Option.getOrElse(Option.fromUndefinedOr(pairsByRule[rule]), () => [])) + }), Order.mapInput(Order.String, (rule: RuleSummary) => rule.rule), ) - const orphanedTests = Arr.sort(command.matrix.flatMap((project) => orphansOf(project, command.drops)), orphanOrder) + const orphanedTests = Arr.sort( + command.matrix.flatMap((project) => + orphansOf(project, Option.getOrElse(Record.get(indexes, project.project), () => EMPTY_INDEX), dropsOf(project)) + ), + orphanOrder, + ) const fields = { schemaVersion: '1' as const, scope: command.scope, projects: command.matrix.map((project) => ({ project: project.project, matrixMutants: project.mutants.length, - drops: command.drops.filter((drop) => drop.project === project.project).length, + drops: dropsOf(project).length, })), rules, pairs, orphanedTests, } const failures = { - pairs: tallyCount(pairs, 'fail'), + pairs: tallyCountsOf(pairs).fail, unattestedRules: rules.filter(isUnattested).length, orphanedTests: orphanedTests.length, } diff --git a/packages/stryker-js/src/audit-request.cell.ts b/packages/stryker-js/src/audit-request.cell.ts index 82610cc4d..f96f17b79 100644 --- a/packages/stryker-js/src/audit-request.cell.ts +++ b/packages/stryker-js/src/audit-request.cell.ts @@ -2,7 +2,6 @@ import { Cell } from '@systemfsoftware/effect-cell-types' import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' import * as Arr from 'effect/Array' import * as Boolean from 'effect/Boolean' -import type * as Console from 'effect/Console' import * as Effect from 'effect/Effect' import * as FileSystem from 'effect/FileSystem' import * as Layer from 'effect/Layer' @@ -31,8 +30,8 @@ import { type RunCounts, } from './audit.schema.js' import { countRuns, CountRunsCommand } from './count-runs.workflow.js' +import { type PlanChannel } from './plan-request.cell.js' import { readProjectCell } from './read-project.cell.js' -import type { HostServices } from './run/host.service.js' import { instrumentSources } from './run/instrument.js' import { loadConfigCell } from './run/load-config.cell.js' import { prepareForInstrumentCell } from './run/plan-prepare.cell.js' @@ -40,27 +39,20 @@ import type { PrepareForInstrument } from './run/prepare.js' import { RunEnvironment } from './run/RunEnvironment.service.js' import type { EnginePorts } from './run/StageServices.service.js' -export interface AuditRequest { +interface AuditRequest { readonly matrix: string readonly out: string readonly countsOnly: boolean readonly projects?: ReadonlyArray | undefined readonly files?: ReadonlyArray | undefined } +type AuditChannel = PlanChannel export interface AuditRequestInput { readonly request: AuditRequest readonly channel: AuditChannel } -export interface AuditChannel { - readonly environment: { - readonly basePath: string - readonly host: HostServices - readonly console: Console.Console - } -} - const REPORT_FILE = 'stryker-incremental.json' const DETAIL_LINES = 20 const DECODE_DETAIL_CHARS = 300 diff --git a/packages/stryker-js/src/audit.schema.ts b/packages/stryker-js/src/audit.schema.ts index 5581ff3be..5bff80361 100644 --- a/packages/stryker-js/src/audit.schema.ts +++ b/packages/stryker-js/src/audit.schema.ts @@ -1,7 +1,7 @@ import { Mutant, type Plugin } from '@systemfsoftware/stryker-js-plugin-interface' import * as S from 'effect/Schema' -export const AuditSchemaVersion = S.Literal('1') +const AuditSchemaVersion = S.Literal('1') const Count = S.Int.check(S.isGreaterThanOrEqualTo(0)) @@ -78,20 +78,18 @@ const PairFields = { next: S.String, } -export const JoinedPair = S.TaggedStruct('JoinedPair', { +const JoinedPair = S.TaggedStruct('JoinedPair', { ...PairFields, mutantStatus: Mutant.MutantStatusSchema, dominatorStatus: Mutant.MutantStatusSchema, missingKillers: S.Array(S.String), verdict: PairVerdict, }) -export type JoinedPair = typeof JoinedPair.Type -export const UnjoinablePair = S.TaggedStruct('UnjoinablePair', { +const UnjoinablePair = S.TaggedStruct('UnjoinablePair', { ...PairFields, reason: UnjoinableReason, }) -export type UnjoinablePair = typeof UnjoinablePair.Type export const AuditedPair = S.Union([JoinedPair, UnjoinablePair]) export type AuditedPair = typeof AuditedPair.Type @@ -121,7 +119,7 @@ export const OrphanedTest = S.Struct({ }) export type OrphanedTest = typeof OrphanedTest.Type -export const ProjectDrops = S.Struct({ +const ProjectDrops = S.Struct({ project: S.String, matrixMutants: Count, drops: Count, @@ -136,7 +134,7 @@ const DropAuditFields = { orphanedTests: S.Array(OrphanedTest), } -export const AuditFailures = S.Struct({ pairs: Count, unattestedRules: Count, orphanedTests: Count }) +const AuditFailures = S.Struct({ pairs: Count, unattestedRules: Count, orphanedTests: Count }) const AuditReportTypeId: unique symbol = Symbol.for('@systemfsoftware/stryker-js/AuditReport') @@ -247,7 +245,7 @@ export class AuditFailed extends S.TaggedError()('AuditFailed', { } } -export const AuditInputCode = S.Literals(['matrix-unreadable', 'matrix-undecodable', 'files-with-counts-only']) +const AuditInputCode = S.Literals(['matrix-unreadable', 'matrix-undecodable', 'files-with-counts-only']) export class AuditInputUnusable extends S.TaggedError()('AuditInputUnusable', { code: AuditInputCode, From 6782ef63416630d7965fdbb5374be1e5e5e84373 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 09:05:26 +0000 Subject: [PATCH 03/12] refactor(stryker-js): share the per-project chdir loop between plan and audit projectsOf and the cwd swap lived twice (plan-request.cell.ts and audit-request.cell.ts). project-directory.adapter.ts now owns both as forEachProjectDirectory, a dual over the --projects entries run with concurrency 1 because process.chdir is process-wide. --- packages/stryker-js/src/plan-request.cell.ts | 21 ++-------- .../src/project-directory.adapter.ts | 38 +++++++++++++++++++ 2 files changed, 42 insertions(+), 17 deletions(-) create mode 100644 packages/stryker-js/src/project-directory.adapter.ts diff --git a/packages/stryker-js/src/plan-request.cell.ts b/packages/stryker-js/src/plan-request.cell.ts index 4a421371b..5e1f4e9b6 100644 --- a/packages/stryker-js/src/plan-request.cell.ts +++ b/packages/stryker-js/src/plan-request.cell.ts @@ -26,6 +26,7 @@ import { DryRunCoverageReused } from './dry-run-reuse.workflow.js' import { CompileErrorProbeSchema, CostsFieldSchema } from './plan-request.schema.js' import { type PlannedMutant, planShards, PlanShardsCommand } from './plan-shards.workflow.js' import type { LoadedPlugins } from './Plugins.schema.js' +import { forEachProjectDirectory } from './project-directory.adapter.js' import { readProjectCell } from './read-project.cell.js' import { requireDryRun, type RequireDryRunDecision } from './require-dry-run.workflow.js' import { dryRunChoiceOf, requireDryRunCommandOf } from './run/dry-run-choice.js' @@ -386,13 +387,6 @@ const writePlan = ( }) }).pipe(Effect.orDie) -const projectsOf = (projects: ReadonlyArray | undefined): ReadonlyArray => - Option.getOrElse( - Option.filter(Option.map(Option.fromUndefinedOr(projects), (present) => [...present]), (present) => - present.length > 0), - () => ['.'], - ) - const labelBaseOf = (path: Path.Path, basePath: string, out: string | undefined): string => Option.getOrElse( Option.map(Option.fromUndefinedOr(out), (present) => path.dirname(path.resolve(basePath, present))), @@ -423,17 +417,10 @@ const planReuseRowOf = (entry: ProjectPlan): RunEvent.PlanProjectReuse => export const planRequest = ({ request, channel }: PlanRequestInput): Effect.Effect => Effect.gen(function*() { const path = yield* Path.Path - const previous = globalThis.process.cwd() const labelBase = labelBaseOf(path, channel.environment.basePath, request.out) - const planned = yield* Effect.forEach( - projectsOf(request.projects), - (directory) => - Effect.acquireUseRelease( - Effect.sync(() => globalThis.process.chdir(directory)), - () => planProject(request, channel, labelBase, directory), - () => Effect.sync(() => globalThis.process.chdir(previous)), - ), - { concurrency: 1 }, + const planned = yield* forEachProjectDirectory( + request.projects, + (directory) => planProject(request, channel, labelBase, directory), ) const scheduled = planned.flatMap((entry) => entry.mutants.map((mutant) => ({ project: entry.label, ...mutant }))) const dryRunCosts: Record = Object.fromEntries( diff --git a/packages/stryker-js/src/project-directory.adapter.ts b/packages/stryker-js/src/project-directory.adapter.ts new file mode 100644 index 000000000..359929264 --- /dev/null +++ b/packages/stryker-js/src/project-directory.adapter.ts @@ -0,0 +1,38 @@ +import * as Arr from 'effect/Array' +import * as Effect from 'effect/Effect' +import { dual } from 'effect/Function' +import * as Option from 'effect/Option' + +export const projectsOf = (projects: ReadonlyArray | undefined): ReadonlyArray => + Option.getOrElse(Option.filter(Option.fromUndefinedOr(projects), Arr.isReadonlyArrayNonEmpty), () => ['.']) + +const inProjectDirectory = (directory: string, body: Effect.Effect): Effect.Effect => + Effect.suspend(() => { + const previous = globalThis.process.cwd() + return Effect.acquireUseRelease( + Effect.sync(() => globalThis.process.chdir(directory)), + () => body, + () => Effect.sync(() => globalThis.process.chdir(previous)), + ) + }) + +/** + * Runs `body` once per `--projects` entry, one at a time, with the process + * working directory switched to that entry, because config discovery reads + * the working directory. No entries means the working directory alone. + */ +export const forEachProjectDirectory: { + ( + body: (directory: string) => Effect.Effect, + ): (projects: ReadonlyArray | undefined) => Effect.Effect, E, R> + ( + projects: ReadonlyArray | undefined, + body: (directory: string) => Effect.Effect, + ): Effect.Effect, E, R> +} = dual(2, ( + projects: ReadonlyArray | undefined, + body: (directory: string) => Effect.Effect, +): Effect.Effect, E, R> => + Effect.forEach(projectsOf(projects), (directory) => inProjectDirectory(directory, body(directory)), { + concurrency: 1, + })) From c52edf5801b48a88d209e73bd466adb29e35fa67 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 09:05:27 +0000 Subject: [PATCH 04/12] fix(stryker-js): refuse audits that cannot see a lost kill A kill matrix now carries its mutantSetPolicy and the audit refuses anything but 'full' (matrix-not-full, exit 2): an ordinary run never runs the mutants the default policy drops, so it passed every drop. A --files entry the matrix does not record is refused (files-outside-matrix) instead of auditing nothing and exiting 0; scoped paths are normalised against the base path. The CLI integration test adds the orphan exit-1 contract, the policy refusal, a scoped pass and the unmatched-file refusal. Over kill-matrix-420 the audit JSON is byte-identical; mutation-report-416 is now refused. --- packages/stryker-js/README.md | 4 +- packages/stryker-js/src/audit-request.cell.ts | 91 +++++++++++++------ packages/stryker-js/src/audit.schema.ts | 11 ++- .../tests/audit.integration.test.ts | 89 +++++++++++++++++- 4 files changed, 162 insertions(+), 33 deletions(-) diff --git a/packages/stryker-js/README.md b/packages/stryker-js/README.md index 48b62144d..bfa9db8b1 100644 --- a/packages/stryker-js/README.md +++ b/packages/stryker-js/README.md @@ -173,7 +173,7 @@ pnpm exec stryker annotate --baseline .stryker-baseline.json # Check every dropped mutant against a kill matrix, or count a finished run pnpm exec stryker audit --matrix kill-matrix/ --projects packages/a,packages/b --out reports/audit.json -pnpm exec stryker audit --matrix kill-matrix/ --files packages/a/src/sum.ts --out reports/audit.json +pnpm exec stryker audit --matrix kill-matrix/ --projects packages/a --files packages/a/src/sum.ts --out reports/audit.json pnpm exec stryker audit --counts-only --matrix mutation-report/ --out reports/counts.json # Judge one surfaced survivor, or drive runs from an editor or an agent @@ -184,7 +184,7 @@ pnpm exec stryker mcp `stryker run --since ` takes the merge base of `` and the working tree, diffs it at `--unified=0`, and intersects the changed lines with `mutate`; it falls back to a full run when the Stryker config, a test-runner config, `package.json`, or the lockfile changed. `stryker run --mutant ` admits each id against the previous report (`reports/mutation/mutation.json`), restricts the run to those mutants' files, and emits each one's status, covering tests, killing test, and a `stryker run --mutant ` reproducer. `stryker gate` exits non-zero on survivors missing from its baseline, `stryker compare` exits `1` on a status mismatch after subtracting a `--noise` file, and `stryker annotate` prints GitHub workflow commands at each surfaced survivor. -`stryker audit` checks that the default mutant-set policy loses no kill. `--matrix` is a directory holding `/stryker-incremental.json` for each `--projects` entry, recorded by a run with every killer kept (the shared config's `STRYKER_KILL_MATRIX=1` lane). The audit instruments the matrix sources without running a test, joins each dropped mutant `m` to its dominator `d` by id, and judges the pair: `Pass` when every named test that kills `d` also kills `m` (`killers-contained`) or `m` was never compiled (`dropped-mutant-carries-no-kill-signal`); `Vacuous` when `d` survived or had no coverage; `AttributionUnverified` when a killer is a file-level hook, the matrix names no test for it, or either mutant timed out; `Fail` when a named test kills `d` but not `m`, `d` never ran, `m` survived, or the ids do not match. A test whose every killed mutant is dropped is reported as orphaned. `--files` scopes the audit to the drops in the named files, so a pull request audits only what it changes. The audit exits `0` when no pair fails, no rule goes unattested and no test is orphaned, `1` otherwise, and `2` on unusable input (an unreadable or undecodable matrix, reports that record no mutant, or `--files` with `--counts-only`). `--counts-only` reads the reports alone and counts planned, `CompileError`, compiled and executed mutants and test executions per project, with ignored mutants grouped by rule. Both modes write a versioned JSON report (`schemaVersion`) to `--out`, where every pair, unattested rule and orphaned test carries a reason code and a next action, and print a summary capped at 20 detail lines. +`stryker audit` checks that the default mutant-set policy loses no kill. `--matrix` is a directory holding `/stryker-incremental.json` for each `--projects` entry, recorded by a run with every killer kept (the shared config's `STRYKER_KILL_MATRIX=1` lane); a report written under any policy but `full` is refused, because an ordinary run never runs the mutants the default policy drops. The audit instruments the matrix sources without running a test, joins each dropped mutant `m` to its dominator `d` by id, and judges the pair: `Pass` when every named test that kills `d` also kills `m` (`killers-contained`) or `m` was never compiled (`dropped-mutant-carries-no-kill-signal`); `Vacuous` when `d` survived or had no coverage; `AttributionUnverified` when a killer is a file-level hook, the matrix names no test for it, or either mutant timed out; `Fail` when a named test kills `d` but not `m`, `d` never ran, `m` survived, or the ids do not match. A test whose every killed mutant is dropped is reported as orphaned. `--files` scopes the audit to the drops in the named files, each written as `<--projects entry>/`, so a pull request audits only what it changes; a named file the matrix does not record is refused. The audit exits `0` when no pair fails, no rule goes unattested and no test is orphaned, `1` otherwise, and `2` on unusable input (an unreadable, undecodable or not-`full` matrix, a `--files` entry outside the matrix, reports that record no mutant, or `--files` with `--counts-only`). `--counts-only` reads the reports alone and counts planned, `CompileError`, compiled and executed mutants and test executions per project, with ignored mutants grouped by rule. Both modes write a versioned JSON report (`schemaVersion`) to `--out`, where every pair, unattested rule and orphaned test carries a reason code and a next action, and print a summary capped at 20 detail lines. A mutation run only starts on main CI. `stryker run` — and the programmatic `run`/`strykerCell` entry, the MSP `mutationTest` request, and the MCP `rerun_mutant` tool — refuses to start unless `GITHUB_ACTIONS` is `true`. A refused run exits non-zero and ends the machine stream with a `refused` event naming the rule (`mutation-runs-on-main-ci`). Dry runs (`--dryRunOnly`), `--version`, config and help commands, the MSP `discover` request, and the read-only MCP tools are unaffected. `stryker serve stdio|socket` speaks the Mutation Server Protocol and `stryker mcp` serves MCP over stdio. diff --git a/packages/stryker-js/src/audit-request.cell.ts b/packages/stryker-js/src/audit-request.cell.ts index f96f17b79..4d108bc5e 100644 --- a/packages/stryker-js/src/audit-request.cell.ts +++ b/packages/stryker-js/src/audit-request.cell.ts @@ -31,6 +31,7 @@ import { } from './audit.schema.js' import { countRuns, CountRunsCommand } from './count-runs.workflow.js' import { type PlanChannel } from './plan-request.cell.js' +import { forEachProjectDirectory, projectsOf } from './project-directory.adapter.js' import { readProjectCell } from './read-project.cell.js' import { instrumentSources } from './run/instrument.js' import { loadConfigCell } from './run/load-config.cell.js' @@ -61,6 +62,10 @@ const UNREADABLE_NEXT = 'Pass --matrix the directory holding /stryker-incremental.json for every --projects entry, such as a downloaded kill-matrix or mutation-report artifact.' const UNDECODABLE_NEXT = 'Audit drops against a report written by a kill-matrix run (STRYKER_KILL_MATRIX=1), which records every killer and the test catalog.' +const NOT_FULL_NEXT = + 'Audit drops against a kill-matrix run (STRYKER_KILL_MATRIX=1), which runs every mutant and records every killer; an ordinary run never runs the mutants the default policy drops.' +const OUTSIDE_MATRIX_NEXT = + 'Pass --files only source files the matrix records, as <--projects entry>/; a file the matrix does not record has no kill data to audit.' const FILES_WITH_COUNTS_NEXT = 'Drop --files: --counts-only counts every record of the reports.' type AuditFailure = AuditFailed | AuditInputUnusable | NothingCounted @@ -89,8 +94,18 @@ const readReport = ( ), )) -const projectsOf = (projects: ReadonlyArray | undefined): ReadonlyArray => - Option.getOrElse(Option.filter(Option.fromUndefinedOr(projects), Arr.isReadonlyArrayNonEmpty), () => ['.']) +const readMatrix = (file: string): Effect.Effect => + readReport(KillMatrixReportJson, file).pipe( + Effect.filterOrFail( + (report) => report.mutantSetPolicy === 'full', + (report) => + AuditInputUnusable.make({ + code: 'matrix-not-full', + detail: `${file} was written with mutator.mutantSetPolicy '${report.mutantSetPolicy}', not 'full'`, + next: NOT_FULL_NEXT, + }), + ), + ) const scopeOf = (path: Path.Path, basePath: string, files: ReadonlyArray | undefined): AuditScope => Option.match(Option.filter(Option.fromUndefinedOr(files), Arr.isReadonlyArrayNonEmpty), { @@ -101,13 +116,36 @@ const scopeOf = (path: Path.Path, basePath: string, files: ReadonlyArray }), }) -const inScope = (scope: AuditScope, path: Path.Path, project: string, key: string): boolean => +const inScope = (scope: AuditScope, scopedPath: string): boolean => Match.value(scope).pipe( Match.tag('Corpus', () => true), - Match.tag('Files', ({ files }) => files.includes(path.join(project, key))), + Match.tag('Files', ({ files }) => files.includes(scopedPath)), + Match.exhaustive, + ) + +const unmatchedFilesOf = (scope: AuditScope, scopedFiles: ReadonlyArray): ReadonlyArray => + Match.value(scope).pipe( + Match.tag('Corpus', () => []), + Match.tag('Files', ({ files }) => files.filter((file) => !scopedFiles.includes(file))), Match.exhaustive, ) +const refuseUnmatchedFiles = ( + scope: AuditScope, + scopedFiles: ReadonlyArray, +): Effect.Effect => { + const unmatched = unmatchedFilesOf(scope, scopedFiles) + return Boolean.match(Arr.isReadonlyArrayNonEmpty(unmatched), { + onTrue: () => + Effect.fail(AuditInputUnusable.make({ + code: 'files-outside-matrix', + detail: `no --projects entry's matrix records ${unmatched.join(', ')}`, + next: OUTSIDE_MATRIX_NEXT, + })), + onFalse: () => Effect.void, + }) +} + const withDefaultPolicy = (prepared: PrepareForInstrument): PrepareForInstrument => ({ ...prepared, options: { ...prepared.options, mutator: { ...prepared.options.mutator, mutantSetPolicy: 'default' } }, @@ -130,6 +168,7 @@ const matrixProjectOf = (project: string, report: KillMatrixReport): MatrixProje interface AuditedProject { readonly drops: ReadonlyArray readonly matrix: MatrixProject + readonly scopedFiles: ReadonlyArray } const auditProject = ( @@ -142,13 +181,19 @@ const auditProject = ( const fs = yield* FileSystem.FileSystem const path = yield* Path.Path const basePath = channel.environment.basePath - const report = yield* readReport( - KillMatrixReportJson, - path.resolve(basePath, request.matrix, directory, REPORT_FILE), - ) - const files = Object.entries(report.files) - .filter(([key]) => inScope(scope, path, directory, key)) - .map(([key, file]) => ({ name: path.resolve(report.projectRoot, key), content: file.source, mutate: true })) + const report = yield* readMatrix(path.resolve(basePath, request.matrix, directory, REPORT_FILE)) + const scoped = Object.entries(report.files) + .map(([key, file]) => ({ + key, + file, + scopedPath: path.relative(basePath, path.resolve(basePath, directory, key)), + })) + .filter((entry) => inScope(scope, entry.scopedPath)) + const files = scoped.map(({ key, file }) => ({ + name: path.resolve(report.projectRoot, key), + content: file.source, + mutate: true, + })) const drops = yield* Effect.scoped(Effect.gen(function*() { const project = yield* fs.realPath(path.resolve(basePath, directory)) const env = { ...channel.environment.host.env, basePath: project } @@ -160,17 +205,11 @@ const auditProject = ( const instrumented = yield* instrumentSources(withDefaultPolicy(prepared), files) return instrumented.mutants.flatMap((mutant) => dropOf(directory, mutant)) })).pipe(Effect.orDie) - return { drops, matrix: matrixProjectOf(directory, report) } - }) - -const inProjectDirectory = (directory: string, body: Effect.Effect): Effect.Effect => - Effect.suspend(() => { - const previous = globalThis.process.cwd() - return Effect.acquireUseRelease( - Effect.sync(() => globalThis.process.chdir(directory)), - () => body, - () => Effect.sync(() => globalThis.process.chdir(previous)), - ) + return { + drops, + matrix: matrixProjectOf(directory, report), + scopedFiles: scoped.map((entry) => entry.scopedPath), + } }) const writeReport = ( @@ -282,11 +321,11 @@ const auditDropList = ( const path = yield* Path.Path const basePath = channel.environment.basePath const scope = scopeOf(path, basePath, request.files) - const audited = yield* Effect.forEach( - projectsOf(request.projects), - (directory) => inProjectDirectory(directory, auditProject(request, channel, scope, directory)), - { concurrency: 1 }, + const audited = yield* forEachProjectDirectory( + request.projects, + (directory) => auditProject(request, channel, scope, directory), ) + yield* refuseUnmatchedFiles(scope, audited.flatMap((project) => project.scopedFiles)) const report = yield* Effect.fromResult(auditDrops(AuditDropsCommand.make({ scope, drops: audited.flatMap((project) => project.drops), diff --git a/packages/stryker-js/src/audit.schema.ts b/packages/stryker-js/src/audit.schema.ts index 5bff80361..3739d2898 100644 --- a/packages/stryker-js/src/audit.schema.ts +++ b/packages/stryker-js/src/audit.schema.ts @@ -1,4 +1,4 @@ -import { Mutant, type Plugin } from '@systemfsoftware/stryker-js-plugin-interface' +import { Mutant, Options, type Plugin } from '@systemfsoftware/stryker-js-plugin-interface' import * as S from 'effect/Schema' const AuditSchemaVersion = S.Literal('1') @@ -7,6 +7,7 @@ const Count = S.Int.check(S.isGreaterThanOrEqualTo(0)) export const KillMatrixReport = S.Struct({ projectRoot: S.String, + mutantSetPolicy: Options.MutantSetPolicy, files: S.Record( S.String, S.Struct({ @@ -245,7 +246,13 @@ export class AuditFailed extends S.TaggedError()('AuditFailed', { } } -const AuditInputCode = S.Literals(['matrix-unreadable', 'matrix-undecodable', 'files-with-counts-only']) +const AuditInputCode = S.Literals([ + 'matrix-unreadable', + 'matrix-undecodable', + 'matrix-not-full', + 'files-outside-matrix', + 'files-with-counts-only', +]) export class AuditInputUnusable extends S.TaggedError()('AuditInputUnusable', { code: AuditInputCode, diff --git a/packages/stryker-js/tests/audit.integration.test.ts b/packages/stryker-js/tests/audit.integration.test.ts index 4f44ac8f5..3e9f3d92c 100644 --- a/packages/stryker-js/tests/audit.integration.test.ts +++ b/packages/stryker-js/tests/audit.integration.test.ts @@ -53,16 +53,18 @@ const decodeDropAudit = S.decodeUnknownEffect(S.fromJsonString(S.Struct({ verdict: S.optional(S.Struct({ reason: S.String })), reason: S.optional(S.String), })), + orphanedTests: S.Array(S.Struct({ test: S.String })), }))) const decodeCounts = S.decodeUnknownEffect(S.fromJsonString(S.Struct({ total: S.Struct({ planned: S.Int, compiled: S.Int, statuses: S.Struct({ CompileError: S.Int, Killed: S.Int }) }), }))) -const matrixJson = (root: string, mutants: ReadonlyArray): string => +const matrixJson = (root: string, mutants: ReadonlyArray, mutantSetPolicy: string): string => `${ JSON.stringify({ projectRoot: root, + mutantSetPolicy, files: { [SOURCE_FILE]: { source: SOURCE, mutants } }, testFiles: { 'test/sign.test.js': { tests: [{ id: NAMED_TEST }, { id: OTHER_TEST }] } }, }) @@ -91,11 +93,15 @@ const prepareProject = (): Effect.Effect, + mutantSetPolicy = 'full', ): Effect.Effect => Effect.gen(function*() { const fs = yield* FileSystem.FileSystem const path = yield* Path.Path - yield* fs.writeFileString(path.join(root, MATRIX_DIR, 'stryker-incremental.json'), matrixJson(root, mutants)) + yield* fs.writeFileString( + path.join(root, MATRIX_DIR, 'stryker-incremental.json'), + matrixJson(root, mutants, mutantSetPolicy), + ) }).pipe(Effect.orDie) const runAudit = ( @@ -139,6 +145,7 @@ interface AuditedPairView { interface DropAuditView { readonly pairs: ReadonlyArray + readonly orphanedTests: ReadonlyArray<{ readonly test: string }> } interface AuditedRun { @@ -171,13 +178,14 @@ const auditWith = ( root: string, pair: DroppedPair, killers: { readonly mutant: ReadonlyArray; readonly dominator: ReadonlyArray }, + scope: ReadonlyArray = [], ): Effect.Effect => Effect.gen(function*() { yield* writeMatrix(root, [ { id: pair.mutant, status: 'Killed', killedBy: killers.mutant }, { id: pair.dominator, status: 'Killed', killedBy: killers.dominator }, ]) - const ran = yield* runAudit(root, ['--matrix', MATRIX_DIR]) + const ran = yield* runAudit(root, ['--matrix', MATRIX_DIR, ...scope]) const audit = yield* readDropAudit(root) return { ran, audit } }) @@ -271,6 +279,81 @@ Feature('Auditing the mutants the default policy drops against a kill matrix', { ), ) + scenario( + 'A named test that kills only the dropped mutant is orphaned and fails the audit though the pair passes', + Gherkin.Do.pipe( + Given('a project whose one relational comparison drops its complement')('root', () => prepareProject()), + Given('the drop and its dominator as the audit names them')('pair', (s) => discoverDroppedPair(s.root)), + When('the matrix records a second named test that kills the dropped mutant alone')( + 'audited', + (s) => auditWith(s.root, s.pair, { mutant: [NAMED_TEST, OTHER_TEST], dominator: [NAMED_TEST] }), + ), + Then('the process exits 1, the pair passes and the second test is reported as orphaned')((s, expect) => + expect({ + exitCode: s.audited.ran.exitCode, + reasons: reasonsOf(s.audited.audit), + orphans: s.audited.audit.orphanedTests.map((orphan) => orphan.test), + }).toStrictEqual({ exitCode: 1, reasons: ['killers-contained'], orphans: [OTHER_TEST] }) + ), + ), + ) + + scenario( + 'A report written under the default policy is refused as a kill matrix', + Gherkin.Do.pipe( + Given('a project')('root', () => prepareProject()), + Given('a report recorded under the default mutant-set policy')( + 'matrix', + (s) => writeMatrix(s.root, [{ id: '1a1a1a1a1a1a1a1a', status: 'Killed', killedBy: [NAMED_TEST] }], 'default'), + ), + When('stryker audits against it')('ran', (s) => runAudit(s.root, ['--matrix', MATRIX_DIR])), + Then('the process exits 2 and stderr names the policy refusal and the kill-matrix lane')((s, expect) => + expect({ + exitCode: s.ran.exitCode, + namesTheCode: s.ran.stderr.includes('matrix-not-full'), + namesTheFix: s.ran.stderr.includes('STRYKER_KILL_MATRIX=1'), + }).toStrictEqual({ exitCode: 2, namesTheCode: true, namesTheFix: true }) + ), + ), + ) + + scenario( + "Scoping the audit to a file the matrix records judges that file's drops", + Gherkin.Do.pipe( + Given('a project whose one relational comparison drops its complement')('root', () => prepareProject()), + Given('the drop and its dominator as the audit names them')('pair', (s) => discoverDroppedPair(s.root)), + When('the audit is scoped with --files to the source that holds the drop')( + 'audited', + (s) => auditWith(s.root, s.pair, { mutant: [NAMED_TEST], dominator: [NAMED_TEST] }, ['--files', SOURCE_FILE]), + ), + Then('the process exits 0 and the scoped pair passes on contained killers')((s, expect) => + expect({ + exitCode: s.audited.ran.exitCode, + reasons: reasonsOf(s.audited.audit), + }).toStrictEqual({ exitCode: 0, reasons: ['killers-contained'] }) + ), + ), + ) + + scenario( + 'Scoping the audit to a file the matrix does not record is an input failure', + Gherkin.Do.pipe( + Given('a project')('root', () => prepareProject()), + Given('a matrix holding only the project source')('matrix', (s) => writeMatrix(s.root, [])), + When('stryker audits with --files naming another file')( + 'ran', + (s) => runAudit(s.root, ['--matrix', MATRIX_DIR, '--files', 'src/other.js']), + ), + Then('the process exits 2 and stderr names the file outside the matrix')((s, expect) => + expect({ + exitCode: s.ran.exitCode, + namesTheCode: s.ran.stderr.includes('files-outside-matrix'), + namesTheFile: s.ran.stderr.includes('src/other.js'), + }).toStrictEqual({ exitCode: 2, namesTheCode: true, namesTheFile: true }) + ), + ), + ) + scenario( 'A matrix directory without the project report is an input failure that names the fix', Gherkin.Do.pipe( From 30792acdbec23f3cf0ed2ca195defa500d6aa336 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 09:06:49 +0000 Subject: [PATCH 05/12] test(stryker-js): pin what an audit Count refuses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generated laws only prove acceptance. The in-source refusal law draws finite numbers plus the boundaries (-1, -0, 0.5, MAX_SAFE_INTEGER + 1, NaN, ±Infinity) and accepts exactly the non-negative safe integers; its first run shrank to 2^53, which S.Int refuses as unsafe. --- packages/stryker-js/src/audit.schema.ts | 31 +++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/packages/stryker-js/src/audit.schema.ts b/packages/stryker-js/src/audit.schema.ts index 3739d2898..12324e535 100644 --- a/packages/stryker-js/src/audit.schema.ts +++ b/packages/stryker-js/src/audit.schema.ts @@ -269,3 +269,34 @@ export class AuditInputUnusable extends S.TaggedError()('Aud return this.reason } } + +if (import.meta.vitest !== void 0) { + const { it } = await import('@systemfsoftware/vitest') + const Arr = await import('effect/Array') + const Result = await import('effect/Result') + + const boundaries: ReadonlyArray = [ + -1, + -0, + 0, + 0.5, + 1, + Number.MAX_SAFE_INTEGER, + Number.MAX_SAFE_INTEGER + 1, + Number.NaN, + Infinity, + -Infinity, + ] + + const decodesCount = (value: number): boolean => Result.isSuccess(S.decodeResult(Count)(value)) + + it.prop( + '∀n_CountRefusal_≡AcceptsExactlyTheNonNegativeSafeIntegers', + { of: [S.Finite], subject: decodesCount }, + (subject, [drawn]) => + Arr.every( + Arr.prepend(boundaries, drawn), + (value) => subject(value) === (Number.isSafeInteger(value) && value >= 0), + ), + ) +} From ac51ff729009ec7625d1a938ad9e5dc0435bbf09 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 09:10:29 +0000 Subject: [PATCH 06/12] test(stryker-js): count a default-policy report, as an ordinary run writes it --counts-only reads a normal Mutation report, so the matrix-not-full refusal must not reach it. The counts scenario now feeds a report written under the default policy. Over mutation-report-416 the counts stay 8626 planned and 4136 CompileError, exit 0. --- packages/stryker-js/tests/audit.integration.test.ts | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/stryker-js/tests/audit.integration.test.ts b/packages/stryker-js/tests/audit.integration.test.ts index 3e9f3d92c..943e53d14 100644 --- a/packages/stryker-js/tests/audit.integration.test.ts +++ b/packages/stryker-js/tests/audit.integration.test.ts @@ -253,14 +253,16 @@ Feature('Auditing the mutants the default policy drops against a kill matrix', { 'Counting a finished run reads planned, CompileError and compiled from the reports alone', Gherkin.Do.pipe( Given('a project')('root', () => prepareProject()), - Given('a report with one Killed, one CompileError and one Ignored mutant')( + Given( + "an ordinary run's report, under the default policy, with one Killed, one CompileError and one Ignored mutant", + )( 'matrix', (s) => writeMatrix(s.root, [ { id: '1a1a1a1a1a1a1a1a', status: 'Killed', killedBy: [NAMED_TEST] }, { id: '2b2b2b2b2b2b2b2b', status: 'CompileError', killedBy: [] }, { id: '3c3c3c3c3c3c3c3c', status: 'Ignored', killedBy: [] }, - ]), + ], 'default'), ), When('stryker audits with --counts-only')( 'ran', From beae43c15d2496c27e0898b84e3dc89bfe6b51bd Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 12:01:50 +0000 Subject: [PATCH 07/12] ci: audit default-policy drops against main's kill matrix on every PR A drop-audit job in CI runs stryker audit from the checkout's build against the newest kill-matrix- artifact from main. A pull request audits the drops in the files it changes; one that changes the instrumenter or a stryker config, a push to main and a dispatch audit the whole corpus. A missing matrix (matrix-missing) or one main has moved past for a touched file (matrix-stale) fails with a next action; the job summary carries verdict counts and annotations point at the uploaded audit.json. --- .github/workflows/ci.yml | 283 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 283 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a1adc8787..f81d0804c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -141,3 +141,286 @@ jobs: path: e2e-telemetry.tar.gz retention-days: 7 if-no-files-found: error + + drop-audit: + if: github.event_name != 'push' || github.event.head_commit.author.username != 'github-actions[bot]' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + actions: read + env: + GH_TOKEN: ${{ github.token }} + MATRIX_DIR: kill-matrix + AUDIT_OUT: reports/audit.json + steps: + - uses: actions/checkout@v7 + with: + fetch-depth: ${{ github.event_name == 'pull_request' && 2 || 1 }} + - uses: pnpm/action-setup@v6 + - uses: actions/setup-node@v7 + with: + node-version: 24 + cache: pnpm + + - id: matrix + name: Download main's newest kill matrix + run: | + found='' + for page in 1 2 3 4 5; do + found=$(gh api "repos/$GITHUB_REPOSITORY/actions/artifacts?per_page=100&page=$page" --jq ' + [.artifacts[] + | select((.name | test("^kill-matrix-[0-9]+$")) and (.expired | not) and .workflow_run.head_branch == "main")] + | sort_by(.name | ltrimstr("kill-matrix-") | tonumber) | last // empty + | "\(.name) \(.workflow_run.id) \(.workflow_run.head_sha)"') + [ -n "$found" ] && break + done + next='Dispatch the Kill matrix workflow on main (gh workflow run kill-matrix.yml --ref main), wait for its collect job to publish kill-matrix-, then re-run this job.' + t='`' + if [ -z "$found" ]; then + echo "::error title=Drop audit has no kill matrix (matrix-missing)::No unexpired kill-matrix- artifact from a main run exists. $next" + echo "## Drop audit"$'\n\n'"**Failed: ${t}matrix-missing${t}.** No unexpired ${t}kill-matrix-${t} artifact from a main run exists. $next" >> "$GITHUB_STEP_SUMMARY" + exit 1 + fi + read -r name run sha <<< "$found" + gh run download "$run" --name "$name" --dir "$MATRIX_DIR" + projects=$(find "$MATRIX_DIR" -name stryker-incremental.json -printf '%h\n' | sed "s|^$MATRIX_DIR/||" | sort | paste -sd, -) + if [ -z "$projects" ]; then + echo "::error title=Drop audit has no kill matrix (matrix-missing)::Artifact $name of run $run holds no /stryker-incremental.json. $next" + echo "## Drop audit"$'\n\n'"**Failed: ${t}matrix-missing${t}.** Artifact ${t}$name${t} holds no ${t}/stryker-incremental.json${t}. $next" >> "$GITHUB_STEP_SUMMARY" + exit 1 + fi + { + echo "name=$name" + echo "sha=$sha" + echo "url=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$run" + echo "projects=$projects" + } >> "$GITHUB_OUTPUT" + + - id: scope + name: Scope the audit + env: + EVENT: ${{ github.event_name }} + MATRIX_NAME: ${{ steps.matrix.outputs.name }} + MATRIX_SHA: ${{ steps.matrix.outputs.sha }} + MATRIX_PROJECTS: ${{ steps.matrix.outputs.projects }} + run: | + node --input-type=module - <<'NODE' + import { execFileSync } from 'node:child_process' + import { appendFileSync, readFileSync } from 'node:fs' + + const { EVENT, MATRIX_DIR, MATRIX_NAME, MATRIX_SHA, MATRIX_PROJECTS, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY } = process.env + const projects = MATRIX_PROJECTS.split(',') + const output = (fields) => + appendFileSync(GITHUB_OUTPUT, Object.entries(fields).map(([key, value]) => `${key}=${value}\n`).join('')) + const corpus = (why) => output({ mode: 'corpus', projects: projects.join(','), files: '', scope: `the whole corpus, ${why}` }) + + if (EVENT !== 'pull_request') { + corpus(`on ${EVENT}`) + process.exit(0) + } + + const git = (...args) => execFileSync('git', args, { encoding: 'utf8', maxBuffer: 1 << 28 }) + git('fetch', '--quiet', '--no-tags', '--depth=1', 'origin', '+refs/heads/main:refs/remotes/origin/main') + const onMain = (path) => { + try { + return git('show', `refs/remotes/origin/main:${path}`) + } catch { + return undefined + } + } + // HEAD is the pull request's merge commit; HEAD^1 is the base branch it merges into. + const changed = git('diff', '--name-only', '--no-renames', 'HEAD^1', 'HEAD').split('\n').filter(Boolean) + const deciders = ['packages/stryker-js-instrumenter/', 'packages/toolchain/stryker-config/', ...projects.map((project) => `${project}/stryker.config.`)] + const decidingChanges = changed.filter((path) => deciders.some((prefix) => path.startsWith(prefix))) + const reports = new Map(projects.map((project) => [project, JSON.parse(readFileSync(`${MATRIX_DIR}/${project}/stryker-incremental.json`, 'utf8'))])) + const recorded = changed.flatMap((path) => { + const project = projects.find((candidate) => path.startsWith(`${candidate}/`)) + const file = project === undefined ? undefined : reports.get(project).files?.[path.slice(project.length + 1)] + return file === undefined ? [] : [{ path, project, source: file.source }] + }) + + // The audit instruments the sources the matrix recorded. When main has changed a file this pull + // request touches since the matrix was recorded, the audit would judge code main no longer has. + const stale = recorded.filter((file) => onMain(file.path) !== file.source).map((file) => file.path) + if (stale.length > 0) { + const next = `Wait for a Kill matrix run on a main commit that has those changes (or dispatch one: gh workflow run kill-matrix.yml --ref main), then re-run this job.` + const shown = stale.slice(0, 10).join(', ') + (stale.length > 10 ? ` and ${stale.length - 10} more` : '') + console.log(`::error title=Drop audit matrix is stale (matrix-stale)::Main changed ${shown} after ${MATRIX_NAME} was recorded at ${MATRIX_SHA.slice(0, 9)}. ${next}`) + appendFileSync(GITHUB_STEP_SUMMARY, [ + '## Drop audit', + '', + `**Failed: \`matrix-stale\`.** \`${MATRIX_NAME}\` was recorded at main \`${MATRIX_SHA.slice(0, 9)}\`, and main has changed ${stale.length} file(s) this pull request touches since then, so the matrix does not describe them. ${next}`, + '', + ...stale.slice(0, 20).map((path) => `- \`${path}\``), + '', + ].join('\n')) + process.exit(1) + } + + if (decidingChanges.length > 0) { + corpus(`because this pull request changes what decides drops (${decidingChanges.slice(0, 3).join(', ')}${decidingChanges.length > 3 ? ', ...' : ''})`) + } else if (recorded.length > 0) { + output({ + mode: 'files', + projects: [...new Set(recorded.map((file) => file.project))].join(','), + files: recorded.map((file) => file.path).join(','), + scope: `the ${recorded.length} file(s) this pull request changes that the matrix records`, + }) + } else { + output({ mode: 'none', projects: '', files: '', scope: '' }) + appendFileSync(GITHUB_STEP_SUMMARY, [ + '## Drop audit', + '', + `**Passed: nothing to audit.** This pull request changes no file that \`${MATRIX_NAME}\` (main \`${MATRIX_SHA.slice(0, 9)}\`) records, and nothing that decides drops (${deciders.slice(0, 2).join(', ')} or a project's stryker config).`, + '', + ].join('\n')) + } + NODE + + - if: steps.scope.outputs.mode != 'none' + uses: ./.github/actions/released-tarballs + - id: build-key + if: steps.scope.outputs.mode != 'none' + name: Content key of the build + env: + KEY: mutation-build-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'turbo.json', 'packages/**', 'test/e2e-core/**') }} + run: echo "key=$KEY" >> "$GITHUB_OUTPUT" + - if: steps.scope.outputs.mode != 'none' + run: pnpm install --frozen-lockfile + - name: Restore the build the mutation lane cached + if: steps.scope.outputs.mode != 'none' + uses: actions/cache/restore@v6 + with: + path: .turbo/cache + key: ${{ steps.build-key.outputs.key }} + restore-keys: | + mutation-build-${{ runner.os }}- + - name: Build the CLI and the audited projects + if: steps.scope.outputs.mode != 'none' + env: + SCOPE_PROJECTS: ${{ steps.scope.outputs.projects }} + run: | + mapfile -t filters < <(tr ',' '\n' <<< "packages/stryker-js,$SCOPE_PROJECTS" | sort -u | sed 's|^|--filter=./|') + pnpm exec turbo run build --output-logs=errors-only --log-order=stream "${filters[@]}" + + - id: audit + name: stryker audit + if: steps.scope.outputs.mode != 'none' + env: + SCOPE_PROJECTS: ${{ steps.scope.outputs.projects }} + SCOPE_FILES: ${{ steps.scope.outputs.files }} + run: | + args=(audit --matrix "$MATRIX_DIR" --projects "$SCOPE_PROJECTS" --out "$AUDIT_OUT") + if [ -n "$SCOPE_FILES" ]; then args+=(--files "$SCOPE_FILES"); fi + status=0 + node packages/stryker-js/dist/main.mjs "${args[@]}" 2> audit.stderr || status=$? + cat audit.stderr >&2 + echo "status=$status" >> "$GITHUB_OUTPUT" + + - id: upload + if: steps.audit.outputs.status == '0' || steps.audit.outputs.status == '1' + uses: actions/upload-artifact@v7 + with: + name: drop-audit-${{ github.run_id }} + path: ${{ env.AUDIT_OUT }} + if-no-files-found: error + retention-days: 14 + + - name: Report the audit + if: steps.audit.outputs.status != '' + env: + AUDIT_STATUS: ${{ steps.audit.outputs.status }} + ARTIFACT_URL: ${{ steps.upload.outputs.artifact-url }} + SCOPE: ${{ steps.scope.outputs.scope }} + MATRIX_NAME: ${{ steps.matrix.outputs.name }} + MATRIX_SHA: ${{ steps.matrix.outputs.sha }} + MATRIX_URL: ${{ steps.matrix.outputs.url }} + run: | + node --input-type=module - <<'NODE' + import { appendFileSync, existsSync, readFileSync } from 'node:fs' + + const { AUDIT_STATUS, ARTIFACT_URL, AUDIT_OUT, SCOPE, MATRIX_DIR, MATRIX_NAME, MATRIX_SHA, MATRIX_URL, GITHUB_STEP_SUMMARY } = process.env + const status = Number(AUDIT_STATUS) + const escape = (text) => String(text).replaceAll('%', '%25').replaceAll('\r', '%0D').replaceAll('\n', '%0A') + const property = (text) => escape(text).replaceAll(':', '%3A').replaceAll(',', '%2C') + const error = (title, message, at) => + console.log(`::error ${at === undefined ? '' : `file=${property(at.file)},line=${at.line},`}title=${property(title)}::${escape(message)}`) + const summary = (lines) => appendFileSync(GITHUB_STEP_SUMMARY, [...lines, ''].join('\n')) + const header = ['## Drop audit', '', `Scope: ${SCOPE}.`, '', `Matrix: \`${MATRIX_NAME}\`, main \`${MATRIX_SHA.slice(0, 9)}\` (${MATRIX_URL}).`, ''] + + if (!existsSync(AUDIT_OUT)) { + const lines = readFileSync('audit.stderr', 'utf8').split('\n') + const refusal = lines.find((text) => text.includes('stryker audit (')) + const exitLine = lines.find((text) => /^exit \d+ \(/.test(text)) + const reason = refusal !== undefined + ? refusal.slice(refusal.indexOf('stryker audit (')) + : `stryker audit exited ${status} without writing ${AUDIT_OUT}: ${exitLine ?? 'no exit line on stderr'}. Read the "stryker audit" step log.` + error('Drop audit could not run', reason) + summary([...header, `**Failed (exit ${status}): the audit did not produce a report.** ${reason}`]) + process.exit(status === 0 ? 1 : status) + } + + const report = JSON.parse(readFileSync(AUDIT_OUT, 'utf8')) + const full = `Full report: ${ARTIFACT_URL} (${AUDIT_OUT} in artifact drop-audit-${process.env.GITHUB_RUN_ID}).` + const matrices = new Map() + const locate = (project, id) => { + if (!matrices.has(project)) { + matrices.set(project, JSON.parse(readFileSync(`${MATRIX_DIR}/${project}/stryker-incremental.json`, 'utf8'))) + } + for (const [key, file] of Object.entries(matrices.get(project).files)) { + const mutant = file.mutants.find((candidate) => candidate.id === id) + if (mutant !== undefined) return { file: `${project}/${key}`, line: mutant.location.start.line } + } + return undefined + } + + const verdictOf = (pair) => pair._tag === 'UnjoinablePair' ? 'Unjoinable' : pair.verdict._tag + const reasonOf = (pair) => pair._tag === 'UnjoinablePair' ? pair.reason : pair.verdict.reason + const tally = { Pass: 0, Vacuous: 0, AttributionUnverified: 0, Fail: 0, Unjoinable: 0 } + for (const pair of report.pairs) tally[verdictOf(pair)] += 1 + const failed = report.pairs.filter((pair) => verdictOf(pair) === 'Fail') + const unverified = report.pairs.filter((pair) => verdictOf(pair) === 'AttributionUnverified') + const unattested = report.rules.filter((rule) => rule._tag === 'Unattested') + const orphans = report.orphanedTests + + for (const pair of failed.slice(0, 20)) { + error( + `Drop audit: Fail ${pair.verdict.reason}`, + `Mutant ${pair.mutant} is dropped for ${pair.dominator} (rule ${pair.rule}). ${pair.next} ${full}`, + locate(pair.project, pair.mutant), + ) + } + for (const rule of unattested) { + error(`Drop audit: rule ${rule.rule} unattested`, `${rule.drops} drop(s), none joined the matrix (${rule.reason}). ${rule.next} ${full}`) + } + for (const orphan of orphans.slice(0, 20)) { + error('Drop audit: test left without a kill', `${orphan.test} (${orphan.project}) kills only dropped mutants: ${orphan.killed.join(', ')}. ${orphan.next} ${full}`) + } + if (status !== 0) { + error('Drop audit failed', `${tally.Fail} Fail, ${unattested.length} unattested rule(s), ${orphans.length} test(s) left without a kill. ${full}`) + } + + const detail = [ + ...failed.map((pair) => `- Fail \`${pair.verdict.reason}\`: \`${pair.mutant}\` dropped for \`${pair.dominator}\` (${pair.project}). ${pair.next}`), + ...unattested.map((rule) => `- Unattested rule \`${rule.rule}\`: ${rule.drops} drop(s). ${rule.next}`), + ...orphans.map((orphan) => `- Orphaned test \`${orphan.test}\` (${orphan.project}). ${orphan.next}`), + ...unverified.map((pair) => `- AttributionUnverified \`${reasonOf(pair)}\`: \`${pair.mutant}\` dropped for \`${pair.dominator}\` (${pair.project})`), + ] + summary([ + ...header, + `**${status === 0 ? 'Passed' : `Failed (exit ${status})`}**: ${report.pairs.length} drop pair(s) over ${report.projects.length} project(s).`, + '', + '| Verdict | Count |', + '| --- | ---: |', + ...Object.entries(tally).map(([verdict, count]) => `| ${verdict} | ${count} |`), + `| Unattested rules | ${unattested.length} |`, + `| Tests left without a kill | ${orphans.length} |`, + '', + ...detail.slice(0, 20), + ...(detail.length > 20 ? [`- ... and ${detail.length - 20} more in the full report`] : []), + '', + full, + ]) + process.exit(status) + NODE From 8c13e7821b6edd04827d1e4e2d51f5d47c7525b7 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Sat, 10 Oct 2026 12:40:20 +0000 Subject: [PATCH 08/12] fix(ci): skip the drop-audit upload and report when the audit did not run A skipped step's output is null, and Actions' loose equality coerces null to 0, so status == '0' held on a nothing-to-audit pull request: the upload ran, found no audit.json and failed the job (run 38052728998). Gate both steps on steps.audit.outcome, which is a string. --- .github/workflows/ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f81d0804c..e5a77e7b4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -319,7 +319,7 @@ jobs: echo "status=$status" >> "$GITHUB_OUTPUT" - id: upload - if: steps.audit.outputs.status == '0' || steps.audit.outputs.status == '1' + if: steps.audit.outcome == 'success' && steps.audit.outputs.status != '2' uses: actions/upload-artifact@v7 with: name: drop-audit-${{ github.run_id }} @@ -328,7 +328,7 @@ jobs: retention-days: 14 - name: Report the audit - if: steps.audit.outputs.status != '' + if: steps.audit.outcome == 'success' env: AUDIT_STATUS: ${{ steps.audit.outputs.status }} ARTIFACT_URL: ${{ steps.upload.outputs.artifact-url }} From 293e649b05a159039a31052ba33cdb5763ea07b3 Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Tue, 13 Oct 2026 03:45:11 +0000 Subject: [PATCH 09/12] ci(repo): audit the whole corpus on every run and find the kill matrix by run drop-audit drops its scope step: the audit instruments the sources the kill matrix recorded, so a pull request changes the verdict only through the audit, the plugins it loads or a project's config, and a scope list can miss a judge change. Every PR and non-bot push to main now audits the whole corpus (plan KTD17). The matrix is found through the newest 100 completed main runs of mutation.yml and kill-matrix.yml instead of a five-page scan of the repository-wide artifact list, which grows by 1200-2300 artifacts a day (kill-matrix-455 was on page 6 five hours after upload). Review fixes: the lookup step runs under bash -eo pipefail, so a failed API read fails the step instead of reporting matrix-missing; GH_TOKEN is scoped to that step; the build-cache key is hashed before install, as the Mutation lane hashes it, so it can match; the upload guard reads AUDIT_OUT. The audit's stdout, uncoloured, is the step summary. The 2026-10-13 plan supersedes the 2026-10-09 one: PR-A and PR-B are merged, #288 is PR-W, and the root owns this judgment-surface change (CONST-E9, CONST-G5). --- .github/workflows/ci.yml | 306 ++++-------------- ...0-13-0342-feat-mutant-subsumption-plan.md} | 89 +++-- 2 files changed, 107 insertions(+), 288 deletions(-) rename docs/plans/{2026-10-09-2024-feat-mutant-subsumption-plan.md => 2026-10-13-0342-feat-mutant-subsumption-plan.md} (90%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b06e133ce..ffc7ccc66 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -709,148 +709,62 @@ jobs: contents: read actions: read env: - GH_TOKEN: ${{ github.token }} MATRIX_DIR: kill-matrix AUDIT_OUT: reports/audit.json steps: - uses: actions/checkout@v7 - with: - fetch-depth: ${{ github.event_name == 'pull_request' && 2 || 1 }} - - uses: pnpm/action-setup@v6 - - uses: actions/setup-node@v7 - with: - node-version: 24 - cache: pnpm - + - id: build-key + name: Key of the Mutation lane's build cache, hashed before install as that lane hashes it + env: + KEY: mutation-build-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'turbo.json', 'packages/**', 'test/e2e-core/**') }} + run: echo "key=$KEY" >> "$GITHUB_OUTPUT" + # Looked up through main's runs of the workflows that publish it: the repository-wide artifact + # list grows by over a thousand entries a day, so a newest-first scan of it misses the matrix. - id: matrix name: Download main's newest kill matrix + shell: bash + env: + GH_TOKEN: ${{ github.token }} run: | - found='' - for page in 1 2 3 4 5; do - found=$(gh api "repos/$GITHUB_REPOSITORY/actions/artifacts?per_page=100&page=$page" --jq ' - [.artifacts[] - | select((.name | test("^kill-matrix-[0-9]+$")) and (.expired | not) and .workflow_run.head_branch == "main" and .workflow_run.head_repository_id == .workflow_run.repository_id)] - | sort_by(.name | ltrimstr("kill-matrix-") | tonumber) | last // empty - | "\(.name) \(.workflow_run.id) \(.workflow_run.head_sha)"') - [ -n "$found" ] && break - done - next='Dispatch the Kill matrix workflow on main (gh workflow run kill-matrix.yml --ref main), wait for its collect job to publish kill-matrix-, then re-run this job.' - t='`' - if [ -z "$found" ]; then - echo "::error title=Drop audit has no kill matrix (matrix-missing)::No unexpired kill-matrix- artifact from a main run exists. $next" - echo "## Drop audit"$'\n\n'"**Failed: ${t}matrix-missing${t}.** No unexpired ${t}kill-matrix-${t} artifact from a main run exists. $next" >> "$GITHUB_STEP_SUMMARY" + next='Run the kill-matrix lane on main (gh workflow run mutation.yml --ref main -f lane=kill-matrix), wait for its collect job to publish kill-matrix-, then re-run this job.' + missing() { + echo "::error title=Drop audit has no kill matrix (matrix-missing)::$1 $next" + printf '## Drop audit\n\n**Failed: `matrix-missing`.** %s %s\n' "$1" "$next" >> "$GITHUB_STEP_SUMMARY" exit 1 - fi + } + runs=$(for workflow in mutation.yml kill-matrix.yml; do + gh api "repos/$GITHUB_REPOSITORY/actions/workflows/$workflow/runs?branch=main&status=completed&per_page=100" \ + --jq '.workflow_runs[] | select(.head_repository.id == .repository.id) | "\(.created_at) \(.id) \(.head_sha)"' + done | sort -r) + found='' + while read -r _ run sha; do + [ -n "$run" ] || continue + name=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$run/artifacts?per_page=100" \ + --jq '[.artifacts[] | select((.name | test("^kill-matrix-[0-9]+$")) and (.expired | not))][0].name // empty') + if [ -n "$name" ]; then + found="$name $run $sha" + break + fi + done <<< "$runs" + [ -n "$found" ] || missing 'No unexpired kill-matrix- artifact is on the newest 100 completed main runs of mutation.yml or kill-matrix.yml.' read -r name run sha <<< "$found" gh run download "$run" --name "$name" --dir "$MATRIX_DIR" projects=$(find "$MATRIX_DIR" -name stryker-incremental.json -printf '%h\n' | sed "s|^$MATRIX_DIR/||" | sort | paste -sd, -) - if [ -z "$projects" ]; then - echo "::error title=Drop audit has no kill matrix (matrix-missing)::Artifact $name of run $run holds no /stryker-incremental.json. $next" - echo "## Drop audit"$'\n\n'"**Failed: ${t}matrix-missing${t}.** Artifact ${t}$name${t} holds no ${t}/stryker-incremental.json${t}. $next" >> "$GITHUB_STEP_SUMMARY" - exit 1 - fi + [ -n "$projects" ] || missing "Artifact $name of run $run holds no /stryker-incremental.json." { echo "name=$name" echo "sha=$sha" echo "url=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$run" echo "projects=$projects" } >> "$GITHUB_OUTPUT" - - - id: scope - name: Scope the audit - env: - EVENT: ${{ github.event_name }} - MATRIX_NAME: ${{ steps.matrix.outputs.name }} - MATRIX_SHA: ${{ steps.matrix.outputs.sha }} - MATRIX_PROJECTS: ${{ steps.matrix.outputs.projects }} - run: | - node --input-type=module - <<'NODE' - import { execFileSync } from 'node:child_process' - import { appendFileSync, readFileSync } from 'node:fs' - - const { EVENT, MATRIX_DIR, MATRIX_NAME, MATRIX_SHA, MATRIX_PROJECTS, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY } = process.env - const projects = MATRIX_PROJECTS.split(',') - const output = (fields) => - appendFileSync(GITHUB_OUTPUT, Object.entries(fields).map(([key, value]) => `${key}=${value}\n`).join('')) - const corpus = (why) => output({ mode: 'corpus', projects: projects.join(','), files: '', scope: `the whole corpus, ${why}` }) - - if (EVENT !== 'pull_request') { - corpus(`on ${EVENT}`) - process.exit(0) - } - - const git = (...args) => execFileSync('git', args, { encoding: 'utf8', maxBuffer: 1 << 28 }) - git('fetch', '--quiet', '--no-tags', '--depth=1', 'origin', '+refs/heads/main:refs/remotes/origin/main') - const onMain = (path) => { - try { - return git('show', `refs/remotes/origin/main:${path}`) - } catch { - return undefined - } - } - // HEAD is the pull request's merge commit; HEAD^1 is the base branch it merges into. - const changed = git('diff', '--name-only', '--no-renames', 'HEAD^1', 'HEAD').split('\n').filter(Boolean) - const deciders = ['packages/stryker-js-instrumenter/', 'packages/toolchain/stryker-config/', ...projects.map((project) => `${project}/stryker.config.`)] - const decidingChanges = changed.filter((path) => deciders.some((prefix) => path.startsWith(prefix))) - const reports = new Map(projects.map((project) => [project, JSON.parse(readFileSync(`${MATRIX_DIR}/${project}/stryker-incremental.json`, 'utf8'))])) - const recorded = changed.flatMap((path) => { - const project = projects.find((candidate) => path.startsWith(`${candidate}/`)) - const file = project === undefined ? undefined : reports.get(project).files?.[path.slice(project.length + 1)] - return file === undefined ? [] : [{ path, project, source: file.source }] - }) - - // The audit instruments the sources the matrix recorded. When main has changed a file this pull - // request touches since the matrix was recorded, the audit would judge code main no longer has. - const stale = recorded.filter((file) => onMain(file.path) !== file.source).map((file) => file.path) - if (stale.length > 0) { - const next = `Wait for a Kill matrix run on a main commit that has those changes (or dispatch one: gh workflow run kill-matrix.yml --ref main), then re-run this job.` - const shown = stale.slice(0, 10).join(', ') + (stale.length > 10 ? ` and ${stale.length - 10} more` : '') - console.log(`::error title=Drop audit matrix is stale (matrix-stale)::Main changed ${shown} after ${MATRIX_NAME} was recorded at ${MATRIX_SHA.slice(0, 9)}. ${next}`) - appendFileSync(GITHUB_STEP_SUMMARY, [ - '## Drop audit', - '', - `**Failed: \`matrix-stale\`.** \`${MATRIX_NAME}\` was recorded at main \`${MATRIX_SHA.slice(0, 9)}\`, and main has changed ${stale.length} file(s) this pull request touches since then, so the matrix does not describe them. ${next}`, - '', - ...stale.slice(0, 20).map((path) => `- \`${path}\``), - '', - ].join('\n')) - process.exit(1) - } - - if (decidingChanges.length > 0) { - corpus(`because this pull request changes what decides drops (${decidingChanges.slice(0, 3).join(', ')}${decidingChanges.length > 3 ? ', ...' : ''})`) - } else if (recorded.length > 0) { - output({ - mode: 'files', - projects: [...new Set(recorded.map((file) => file.project))].join(','), - files: recorded.map((file) => file.path).join(','), - scope: `the ${recorded.length} file(s) this pull request changes that the matrix records`, - }) - } else { - output({ mode: 'none', projects: '', files: '', scope: '' }) - const next = 'Next: none needed; a pull request that changes a file the matrix records, or anything that decides drops, gets audited.' - console.log(`::notice title=Drop audit%3A nothing to audit (nothing-to-audit)::This pull request changes no file that ${MATRIX_NAME} (main ${MATRIX_SHA.slice(0, 9)}) records, and nothing that decides drops. ${next}`) - appendFileSync(GITHUB_STEP_SUMMARY, [ - '## Drop audit', - '', - `**Passed: \`nothing-to-audit\`.** This pull request changes no file that \`${MATRIX_NAME}\` (main \`${MATRIX_SHA.slice(0, 9)}\`) records, and nothing that decides drops (${deciders.slice(0, 2).join(', ')} or a project's stryker config). ${next}`, - '', - ].join('\n')) - } - NODE - - - if: steps.scope.outputs.mode != 'none' - uses: ./.github/actions/released-tarballs - - id: build-key - if: steps.scope.outputs.mode != 'none' - name: Content key of the build - env: - KEY: mutation-build-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'turbo.json', 'packages/**', 'test/e2e-core/**') }} - run: echo "key=$KEY" >> "$GITHUB_OUTPUT" - - if: steps.scope.outputs.mode != 'none' - run: pnpm install --frozen-lockfile - - name: Restore the build the mutation lane cached - if: steps.scope.outputs.mode != 'none' + - uses: ./.github/actions/released-tarballs + - uses: pnpm/action-setup@v6 + - uses: actions/setup-node@v7 + with: + node-version: 24 + cache: pnpm + - run: pnpm install --frozen-lockfile + - name: Restore the build the Mutation lane cached uses: actions/cache/restore@v6 with: path: .turbo/cache @@ -858,131 +772,37 @@ jobs: restore-keys: | mutation-build-${{ runner.os }}- - name: Build the CLI and the audited projects - if: steps.scope.outputs.mode != 'none' env: - SCOPE_PROJECTS: ${{ steps.scope.outputs.projects }} + PROJECTS: ${{ steps.matrix.outputs.projects }} run: | - mapfile -t filters < <(tr ',' '\n' <<< "packages/stryker-js,$SCOPE_PROJECTS" | sort -u | sed 's|^|--filter=./|') + mapfile -t filters < <(tr ',' '\n' <<< "packages/stryker-js,$PROJECTS" | sort -u | sed 's|^|--filter=./|') pnpm exec turbo run build --output-logs=errors-only --log-order=stream "${filters[@]}" - - - id: audit - name: stryker audit - if: steps.scope.outputs.mode != 'none' + - name: stryker audit + shell: bash env: - SCOPE_PROJECTS: ${{ steps.scope.outputs.projects }} - SCOPE_FILES: ${{ steps.scope.outputs.files }} + PROJECTS: ${{ steps.matrix.outputs.projects }} + NO_COLOR: "1" + run: node packages/stryker-js/dist/main.mjs audit --matrix "$MATRIX_DIR" --projects "$PROJECTS" --out "$AUDIT_OUT" | tee audit.txt + - name: Copy the audit's output to the summary + if: ${{ !cancelled() && steps.matrix.outcome == 'success' && hashFiles('audit.txt') != '' }} + env: + MATRIX_NAME: ${{ steps.matrix.outputs.name }} + MATRIX_SHA: ${{ steps.matrix.outputs.sha }} + MATRIX_URL: ${{ steps.matrix.outputs.url }} run: | - args=(audit --matrix "$MATRIX_DIR" --projects "$SCOPE_PROJECTS" --out "$AUDIT_OUT") - if [ -n "$SCOPE_FILES" ]; then args+=(--files "$SCOPE_FILES"); fi - status=0 - node packages/stryker-js/dist/main.mjs "${args[@]}" 2> audit.stderr || status=$? - cat audit.stderr >&2 - echo "status=$status" >> "$GITHUB_OUTPUT" - - - id: upload - if: steps.audit.outcome == 'success' && steps.audit.outputs.status != '2' - uses: actions/upload-artifact@v7 + { + echo '## Drop audit' + echo + echo "Against [\`$MATRIX_NAME\`]($MATRIX_URL), recorded at main \`${MATRIX_SHA:0:9}\`." + echo + echo '```text' + cat audit.txt + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + - uses: actions/upload-artifact@v7 + if: ${{ !cancelled() && hashFiles(env.AUDIT_OUT) != '' }} with: name: drop-audit-${{ github.run_id }} path: ${{ env.AUDIT_OUT }} - if-no-files-found: error retention-days: 14 overwrite: true - - - name: Report the audit - if: ${{ !cancelled() && steps.audit.outcome == 'success' }} - env: - AUDIT_STATUS: ${{ steps.audit.outputs.status }} - ARTIFACT_URL: ${{ steps.upload.outputs.artifact-url }} - SCOPE: ${{ steps.scope.outputs.scope }} - MATRIX_NAME: ${{ steps.matrix.outputs.name }} - MATRIX_SHA: ${{ steps.matrix.outputs.sha }} - MATRIX_URL: ${{ steps.matrix.outputs.url }} - run: | - node --input-type=module - <<'NODE' - import { appendFileSync, existsSync, readFileSync } from 'node:fs' - - const { AUDIT_STATUS, ARTIFACT_URL, AUDIT_OUT, SCOPE, MATRIX_DIR, MATRIX_NAME, MATRIX_SHA, MATRIX_URL, GITHUB_STEP_SUMMARY } = process.env - const status = Number(AUDIT_STATUS) - const escape = (text) => String(text).replaceAll('%', '%25').replaceAll('\r', '%0D').replaceAll('\n', '%0A') - const property = (text) => escape(text).replaceAll(':', '%3A').replaceAll(',', '%2C') - const error = (title, message, at) => - console.log(`::error ${at === undefined ? '' : `file=${property(at.file)},line=${at.line},`}title=${property(title)}::${escape(message)}`) - const summary = (lines) => appendFileSync(GITHUB_STEP_SUMMARY, [...lines, ''].join('\n')) - const header = ['## Drop audit', '', `Scope: ${SCOPE}.`, '', `Matrix: \`${MATRIX_NAME}\`, main \`${MATRIX_SHA.slice(0, 9)}\` (${MATRIX_URL}).`, ''] - - if (!existsSync(AUDIT_OUT)) { - const lines = readFileSync('audit.stderr', 'utf8').split('\n') - const refusal = lines.find((text) => text.includes('stryker audit (')) - const exitLine = lines.find((text) => /^exit \d+ \(/.test(text)) - const reason = refusal !== undefined - ? refusal.slice(refusal.indexOf('stryker audit (')) - : `stryker audit exited ${status} without writing ${AUDIT_OUT}: ${exitLine ?? 'no exit line on stderr'}. Read the "stryker audit" step log.` - error('Drop audit could not run', reason) - summary([...header, `**Failed (exit ${status}): the audit did not produce a report.** ${reason}`]) - process.exit(status === 0 ? 1 : status) - } - - const report = JSON.parse(readFileSync(AUDIT_OUT, 'utf8')) - const full = `Full report: ${ARTIFACT_URL} (${AUDIT_OUT} in artifact drop-audit-${process.env.GITHUB_RUN_ID}).` - const matrices = new Map() - const locate = (project, id) => { - if (!matrices.has(project)) { - matrices.set(project, JSON.parse(readFileSync(`${MATRIX_DIR}/${project}/stryker-incremental.json`, 'utf8'))) - } - for (const [key, file] of Object.entries(matrices.get(project).files)) { - const mutant = file.mutants.find((candidate) => candidate.id === id) - if (mutant !== undefined) return { file: `${project}/${key}`, line: mutant.location.start.line } - } - return undefined - } - - const verdictOf = (pair) => pair._tag === 'UnjoinablePair' ? 'Unjoinable' : pair.verdict._tag - const reasonOf = (pair) => pair._tag === 'UnjoinablePair' ? pair.reason : pair.verdict.reason - const tally = { Pass: 0, Vacuous: 0, AttributionUnverified: 0, Fail: 0, Unjoinable: 0 } - for (const pair of report.pairs) tally[verdictOf(pair)] += 1 - const failed = report.pairs.filter((pair) => verdictOf(pair) === 'Fail') - const unverified = report.pairs.filter((pair) => verdictOf(pair) === 'AttributionUnverified') - const unattested = report.rules.filter((rule) => rule._tag === 'Unattested') - const orphans = report.orphanedTests - - for (const pair of failed.slice(0, 20)) { - error( - `Drop audit: Fail ${pair.verdict.reason}`, - `Mutant ${pair.mutant} is dropped for ${pair.dominator} (rule ${pair.rule}). ${pair.next} ${full}`, - locate(pair.project, pair.mutant), - ) - } - for (const rule of unattested) { - error(`Drop audit: rule ${rule.rule} unattested`, `${rule.drops} drop(s), none joined the matrix (${rule.reason}). ${rule.next} ${full}`) - } - for (const orphan of orphans.slice(0, 20)) { - error('Drop audit: test left without a kill', `${orphan.test} (${orphan.project}) kills only dropped mutants: ${orphan.killed.join(', ')}. ${orphan.next} ${full}`) - } - if (status !== 0) { - error('Drop audit failed', `${tally.Fail} Fail, ${unattested.length} unattested rule(s), ${orphans.length} test(s) left without a kill. ${full}`) - } - - const detail = [ - ...failed.map((pair) => `- Fail \`${pair.verdict.reason}\`: \`${pair.mutant}\` dropped for \`${pair.dominator}\` (${pair.project}). ${pair.next}`), - ...unattested.map((rule) => `- Unattested rule \`${rule.rule}\`: ${rule.drops} drop(s). ${rule.next}`), - ...orphans.map((orphan) => `- Orphaned test \`${orphan.test}\` (${orphan.project}). ${orphan.next}`), - ...unverified.map((pair) => `- AttributionUnverified \`${reasonOf(pair)}\`: \`${pair.mutant}\` dropped for \`${pair.dominator}\` (${pair.project})`), - ] - summary([ - ...header, - `**${status === 0 ? 'Passed' : `Failed (exit ${status})`}**: ${report.pairs.length} drop pair(s) over ${report.projects.length} project(s).`, - '', - '| Verdict | Count |', - '| --- | ---: |', - ...Object.entries(tally).map(([verdict, count]) => `| ${verdict} | ${count} |`), - `| Unattested rules | ${unattested.length} |`, - `| Tests left without a kill | ${orphans.length} |`, - '', - ...detail.slice(0, 20), - ...(detail.length > 20 ? [`- ... and ${detail.length - 20} more in the full report`] : []), - '', - full, - ]) - process.exit(status) - NODE diff --git a/docs/plans/2026-10-09-2024-feat-mutant-subsumption-plan.md b/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md similarity index 90% rename from docs/plans/2026-10-09-2024-feat-mutant-subsumption-plan.md rename to docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md index 5d57e525e..4c029181b 100644 --- a/docs/plans/2026-10-09-2024-feat-mutant-subsumption-plan.md +++ b/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md @@ -1,8 +1,9 @@ --- title: Mutant Subsumption and Type-Guided Generation - Plan type: feat -date: 2026-10-09 +date: 2026-10-13 topic: mutant-subsumption +supersedes: docs/plans/2026-10-09-2024-feat-mutant-subsumption-plan.md artifact_contract: ce-unified-plan/v1 product_contract_source: ce-brainstorm execution: code @@ -15,13 +16,13 @@ execution: code - **Objective:** A stryker-js-effect run executes fewer mutants that carry no new signal. A mutant that a kept mutant dominates is not run, and its record names that dominator. A mutant that the TypeScript checker's type facts show cannot compile is never compiled or run. CI shows on the dogfood corpus that no test loses a kill and that no mutant which compiles on main is lost. - **Means:** static subsumption decided in the instrumenter (KTD1), with a check-time re-admission step in the engine (KTD13); type facts read from the TS 7 checker over the unmutated program, with a pure decision in the engine (KTD14, KTD15); a kill-matrix lane and a `stryker audit` command (KTD6, KTD7). - **Product authority:** `.omp-brief/unit-i-contract.md` (Unit I), the supervisor's and the root's 2026-10-09 rulings recorded under Outstanding Questions, then `CONSTITUTION.md`, which outranks both. Stream C (`origin/stryker/mutant-quality`, plan `docs/plans/2026-10-09-1850-feat-mutant-quality-plan.md`) owns `arid-uncovered-block` and `constant-collection-size`. Stream H owns the TS checker package. Stream F owns native generation later. -- **Execution profile:** one `gh stack` on `main` with four PRs: PR-A, PR-B, PR-W, and PR-C, in that order. Each PR is shippable and tested end to end. Local verification is targeted: typecheck and the affected package's tests, at most one build at a time, no e2e, no microVM, no mutation runs. Close every long-lived `tsc`, `--lsp`, or watch process as soon as it is done. +- **Execution profile:** one open PR per stream, stacked on `main`. PR-A (#273) and PR-B (#263, #276) are merged; PR-W (#288) and PR-C remain, in that order. Each PR is shippable and tested end to end. Local verification is targeted: typecheck and the affected package's tests, at most one build at a time, no e2e, no microVM, no mutation runs. Close every long-lived `tsc`, `--lsp`, or watch process as soon as it is done. - **Stop conditions:** - - PR-W, and PR-C's one-job change to `drop-audit`, edit `.github/workflows/`, which is Read-only for this unit. The root owns and reviews both. + - PR-W and PR-C's one-job change to `drop-audit` edit `.github/workflows/ci.yml`, a judgment surface (CONST-E9) that `AGENTS.md` makes Read-only. The root owns it: on 2026-10-13 the root conductor ruled that #288 "stays open and is driven to merge" (https://github.com/systemfsoftware/stryker-js-effect/pull/288#issuecomment-6137453174), the escalation CONST-G5 requires. The root reviews the job and decides whether `drop-audit` becomes a required check; this unit does not tune the audit's predicates to its own drops. `mutation.yml` and `kill-matrix.yml` belong to Stream K and stay untouched. - PR-C's checker-side unit (U15) needs a Stream H ruling (Q15). If that ruling is pending when U14 is done, push U14 and stop with U15 unstarted. - U11 (S3) is deferred until PR-W's `drop-audit` exists. -- **Who finishes:** `ce-work` builds PR-A, PR-B, and PR-C; the root reviews and owns PR-W; the supervisor merges every layer. -- **Open blockers:** Q14 (PR-W bootstrap, the root) and Q15 (checker-side gatherer, Stream H). Q16-Q19 go to the supervisor. +- **Who finishes:** PR-A and PR-B are merged on `main`; `ce-work` builds PR-W (#288) and PR-C; the root reviews PR-W's job; the supervisor merges every layer. +- **Open blockers:** Q15 (checker-side gatherer, Stream H). Q16-Q20 go to the supervisor. - **Applicable packs** (`.compound-engineering/config.yaml`): - cell-architecture: pure-decision-workflows, ports-separate-from-layers, sandwich-phase-order, scoped-lifecycle-boundaries - schema-laws: tagged-unions-over-state-by-presence, refusals-beside-generated-laws, arbitrary-filter-floors @@ -233,7 +234,7 @@ Retired with the SMT layer: R13, R19-R30, AE7-AE10, KTD8-KTD12, U3, U7, U8, and - from the audit (not `--counts-only`, which reads only reports): Layer 2 facts per variant (`NotAssignable`, `MissingReturn`, `Fits`, `NoAnchor` by reason, `Unavailable` by reason), so a low N can be traced to the anchors or probes that did not fire, and the time spent gathering facts; - the list of dropped ids. - The counts ship as a CI artifact that a bench lane can ingest. Publishing them to the lane is a follow-up (Q3). + The counts ship as a CI artifact that a bench lane can ingest. Publishing them to the lane is a follow-up (Q3). The step that writes them in each Mutation run belongs to Stream K, which owns `mutation.yml` (Q20). ```mermaid flowchart TB @@ -277,15 +278,15 @@ flowchart TB ### Delivery order -Four stacked PRs. Each is shippable alone and tested end to end. The ruling's PR shape (2026-10-09): +Four PRs in this order, the ruling's PR shape (2026-10-09). PR-A and PR-B are merged on `main`; PR-W and PR-C remain, stacked on `main`: -| PR | Base | Units | What ships | -| --------------------------- | ---------- | ---------------------------- | ------------------------------------------------------------------------------------------------- | -| PR-A Layer 1 | `main` | U12 (first), U2, U4, U5, U13 | the R4 repair; S1 on by default; the dominator reference on every record; check-time re-admission | -| PR-B Evidence tooling | PR-A | U1, U6 | the kill-matrix switch; `stryker audit` with the R17 and R40 predicates and the R18 counts | -| PR-W Workflow wiring | PR-B | U9 | the kill-matrix workflow, the `drop-audit` job, and the counts step. The root owns and reviews it | -| PR-C Type-guided generation | PR-W | U14, U15 | type facts, `type-invalid`, and the audit's Layer 2 drop list | -| Follow-up | after PR-W | U11 | S3 under class P, gated by its own `drop-audit` run | +| PR | Base | Units | What ships | +| --------------------------- | --------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------- | +| PR-A Layer 1 | merged (#273) | U12 (first), U2, U4, U5, U13 | the R4 repair; S1 on by default; the dominator reference on every record; check-time re-admission | +| PR-B Evidence tooling | merged (#263 for U1, #276 for U6) | U1, U6 | the kill-matrix switch; `stryker audit` with the R17 and R40 predicates and the R18 counts | +| PR-W Workflow wiring (#288) | `main` | U9 | the `drop-audit` job in `ci.yml` | +| PR-C Type-guided generation | PR-W | U14, U15 | type facts, `type-invalid`, and the audit's Layer 2 drop list | +| Follow-up | after PR-W | U11 | S3 under class P, gated by its own `drop-audit` run | ### Verification @@ -312,7 +313,7 @@ Four stacked PRs. Each is shippable alone and tested end to end. The ruling's PR **Rulings (2026-10-09)** -- Q1. Accepted by the root. This unit builds the kill-matrix lane as an evidence producer (U1) and the no-signal-loss predicate as a pure, tested workflow (U6). The root owns and reviews the `.github/workflows/` wiring that makes it a gate (PR-W, U9). +- Q1. Accepted by the root. This unit builds the kill-matrix lane as an evidence producer (U1) and the no-signal-loss predicate as a pure, tested workflow (U6). The root owns and reviews the `.github/workflows/` wiring that makes it a gate (PR-W, U9); on 2026-10-13 it ruled that #288 is driven to merge, and it decides whether `drop-audit` becomes a required check. - Q2. Moot: SMT is rejected, so neither `z3-solver` nor `async-mutex` is added. - Q3. No bench lane exists on main. R18 counts ship as a CI artifact that a bench lane can ingest. Publishing them to the lane is a follow-up through the supervisor. - Q4. The R4 repair is in scope and breaking. It lands first in PR-A, and the PR body flags it for Stream C. @@ -325,11 +326,12 @@ Four stacked PRs. Each is shippable alone and tested end to end. The ruling's PR ### Open questions for the supervisor -- Q14 (the root). **PR-W bootstrap.** The `drop-audit` job needs one kill-matrix artifact, and a new workflow file cannot be dispatched until it exists on `main`. So on PR-W's own head the job can only fail with `NoMatrix`. Option (a): the root lands `kill-matrix.yml` on `main` alone first, dispatches it once, and PR-W then carries only the `drop-audit` job and the counts step, green on its head. Option (b): PR-W merges with `drop-audit` red as `NoMatrix`, not yet required, and the first green run is cited on PR-C. Recommended: (a), because every PR in the stack then meets "green on its head". A drop audit that passes without a matrix would be vacuous, so it does not pass. +- Q14 (the root). **Resolved: option (a).** `kill-matrix.yml` is on `main` and called from the Mutation workflow; `kill-matrix-457` (run 38335732420, main `0aebc83e6`, kept 90 days) is the newest artifact. Original question: **PR-W bootstrap.** The `drop-audit` job needs one kill-matrix artifact, and a new workflow file cannot be dispatched until it exists on `main`. So on PR-W's own head the job can only fail with `NoMatrix`. Option (a): the root lands `kill-matrix.yml` on `main` alone first, dispatches it once, and PR-W then carries only the `drop-audit` job and the counts step, green on its head. Option (b): PR-W merges with `drop-audit` red as `NoMatrix`, not yet required, and the first green run is cited on PR-C. Recommended: (a), because every PR in the stack then meets "green on its head". A drop audit that passes without a matrix would be vacuous, so it does not pass. - Q15 (Stream H). **The checker-side gatherer.** U15 adds a `type-facts` module beside `ts-compiler.handle.ts`, reads `CompilerState.api`/`snapshot` (`:120-130`), and serves the `facts` capability (`CheckerRuntime.service.ts`, `CheckerWorker.service.ts`). All of these are Stream H's files. Option (a): this unit writes U15 under Stream H's review. Option (b): Stream H builds the gatherer to R34, R35, and R38's fact contract, and this unit supplies U14 and the R42 spec. U14 needs neither. - Q16. **"Never generated."** The instrumenter still produces a `type-invalid` mutant: it gets an id and a place in the mutant switch, and the report lists it as Ignored. It is never compiled, tested, or counted as CompileError. Keeping the record gives the audit an id to join (R39). Dropping it before placement would need type facts before instrumentation, so the checker would have to start before `instrumentCell` (`run/run-stages.cell.ts:18-35`), and the audit would lose its id. Is an Ignored `type-invalid` record acceptable as "not generated"? - Q17. **The audit starts the checker.** For Layer 2 drops, `stryker audit` asks the checker for facts over the unmutated corpus programs. That type-checks the original program and compiles no mutant. Please confirm that this stays within Q12. - Q18. **Alignment with Stream C U9.** Stream C U9 reads a receiver kind in `ts-compiler.handle.ts` during `check`. This plan proposes one gatherer module and one `TypeFact` union that both use: U9 calls the gatherer at check time, and Layer 2 calls it through `facts`. Whichever of U9 and U15 lands second adapts to the first. Neither depends on the other. +- Q20 (Stream K). **The R18 counts step.** It belongs in `mutation.yml`'s `report` job, which Stream K owns. This unit supplies `stryker audit --counts-only` (U6); Stream K wires it. - Q19. **Shard co-location.** U13 places each subsumed mutant in the same shard as its dominators (`plan-shards.workflow.ts`). Stream C U7 adds guard-group placement to the same workflow. Whichever lands second merges both constraints into one grouping key. ### Sources / Research @@ -384,6 +386,7 @@ Four stacked PRs. Each is shippable alone and tested end to end. The ruling's PR A shard run sees its dominators' verdicts only if they run in the same process, so the shard plan places each subsumed mutant with its first dominator (Q19). S1 and S3 each name exactly one dominator, so co-locating the first one makes every outcome the step needs local. A placement group is placed as one item and never raises the shard count above `maxShards`. Subsumed records are never served from remembered results (R9), so a remembered drop cannot outlive a dominator that has stopped running. Governs R5, R31. - KTD14. **Layer 2 facts are gathered in the checker; the decision runs in the engine.** Only the checker holds a type checker (`ts-compiler.handle.ts:120-130`), and the engine is where every plan path converges (R37). The checker exposes facts as data through a new optional `facts` capability on `CheckerService` and a `facts` RPC (`packages/stryker-js-plugin-interface/src/Checker.service.ts:8-17`, `PluginRpcs.service.ts:59-74`). The decision workflow lives in the engine. This keeps the fact contract serializable for Stream F. It also keeps checker-specific code inside the checker. Rejected: deciding inside `check`, the Stream C U9 pattern. The audit would then have to run `check`, which compiles mutants, to learn the drop list, and the plan path would price `type-invalid` mutants as compiled. Governs R32, R37, R38. - KTD15. **Each fact is read on the unmutated program, at an anchor the mutation leaves in place.** That is what separates this from Stream C's refuted KD3. A mutant replaces only its own range, so any type outside that range is identical in the mutated program, unless TS infers it from the replaced range. R34's anchor list excludes exactly those inferred types. Each probe is the narrowest type TypeScript gives that replacement: the fresh `""` literal, the empty object literal, and an arrow whose return type is `undefined`. So when the probe is not assignable at the anchor, TypeScript reports an assignability error there in the mutated program. R42 makes that argument executable against the pinned compiler. Governs R34, R35. +- KTD17. **`drop-audit` audits the whole corpus on every run; it has no scope step.** The audit instruments the source text the matrix recorded (`audit-request.cell.ts:194`), not the PR's files, so a PR changes the result only through the audit command, the plugins it loads, or a project's config. A trigger for that is 68% of main commits (110 of 162, `1f553da50`..`251201755`, as `@systemfsoftware/stryker-js` in `turbo query affectedPackages` or a corpus `stryker.config.*` change). The remaining runs would save about 15 runner-seconds each (42 s against 27 s on #288's runs) in a job that runs beside `check`, and a scope step can miss a judge change, as #288's hand list did, and would grade its own PR. Rejected: #288's files mode and `nothing-to-audit`, which audit main's recorded sources and so carry no PR signal; a `tsc --listFilesOnly` closure; turbo's `file.dependencies` trace, which stops at every workspace import because it ignores the `@systemfsoftware/source` condition (233 unresolved imports from `audit-request.cell.ts`). Governs R16. - KTD16. **The new rule id is `type-invalid`.** It joins `RULE_IDS` (`ignore-rule.schema.ts:6-19`) in U14, the layer that first emits it. The SOTA plan's `type-invalid-return` and `type-invalid-object` were never added. One id with the probe in its detail covers all four shapes. Governs R8, R36. ### High-Level Technical Design @@ -420,7 +423,7 @@ flowchart TB U1[U1 kill-matrix switch] U6[U6 stryker audit] end - subgraph W [PR-W, root-owned] + subgraph W [PR-W] U9[U9 CI wiring] end subgraph C [PR-C] @@ -437,7 +440,8 @@ flowchart TB ### Assumptions -- [INFERENCE] The merged incremental reports of a kill-matrix run carry every killer for each mutant. The vitest runner reports all killers under `disableBail` (`interpret-vitest-mutant-run.workflow.ts:94-110`), and `shard/incremental-union.ts` keeps arbitrary fields. U9's first artifact confirms this when it shows `killedBy` lengths above 1. +- The root's 2026-10-13 ruling on #288 ("stays open and is driven to merge") makes #288 PR-W, under the root's ownership of `.github/workflows/`. Whether `drop-audit` becomes a required status check stays the root's call. +- Verified 2026-10-13: the merged incremental reports of a kill-matrix run carry every killer for each mutant. In `kill-matrix-457` (run 38335732420), 1651 of 2677 Killed mutants have more than one `killedBy`, up to 59. - [INFERENCE] The TS 7 `TypeObject` and symbol responses expose what R35 needs: literal values, the optional flag on property symbols, and whether a property is declared or produced by a mapped type. If any of these is missing from `typescript` 7.0.2, R34 and R35 narrow to the facts that are available, the mutants they cannot decide are kept, and the gap goes to Stream H (Q15). ### Risks @@ -464,7 +468,7 @@ flowchart TB Units are grouped by PR. Within a PR, the order shown is the commit order. Mutant ids come from #416 (run 37960922409), or from #406 (run 37918729445) where #416 lacks diagnostics. They name the corpus mutants whose CI outcome each unit is expected to change. Tests use their own fixtures, because ids are derived from file content. -### PR-A: Layer 1 (base `main`) +### PR-A: Layer 1 (merged, #273) #### U12. Condition-position repair (lands first) @@ -578,7 +582,7 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan - The PR body lists the R4 break first and names the files shared with Stream C. - After the release that carries PR-A, the first main Mutation run shows the 76 S1 complements Ignored as `redundant-relational`, each naming its dominator, cited by run id. -### PR-B: Evidence tooling (base PR-A) +### PR-B: Evidence tooling (merged, #263 and #276) #### U1. Kill-matrix switch in the shared Stryker config @@ -586,9 +590,9 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan - **Requirements:** R14, R15. - **Dependencies:** none. - **Files:** `packages/toolchain/stryker-config/lib/base.js`. -- **Approach:** when `STRYKER_KILL_MATRIX=1`, `sharedConfig` adds `disableBail: true` and `mutator: { mutantSetPolicy: 'full' }`. `coverageAnalysis: 'perTest'` is already set (`base.js:17`). With the variable unset, the object is unchanged. A cold start (no cache restore, `plan --full`) is U9's job (KTD7). +- **Approach:** when `STRYKER_KILL_MATRIX=1`, `sharedConfig` adds `disableBail: true` and `mutator: { mutantSetPolicy: 'full' }`. `coverageAnalysis: 'perTest'` is already set (`base.js:17`). With the variable unset, the object is unchanged. A cold start (no cache restore, `plan --full`) is the job of `kill-matrix.yml`, on `main` since #263 (KTD7). - **Patterns:** the environment reads at `base.js:4-8`. -- **Test expectation:** none. This is a config switch; its proof is U9's first artifact showing Killed mutants with more than one `killedBy`. +- **Test expectation:** none. This is a config switch; its proof is `kill-matrix-457`, where 1651 of 2677 Killed mutants have more than one `killedBy`. - **Verification:** `pnpm format:check`; `pnpm --filter @systemfsoftware/stryker-js typecheck`. #### U6. `stryker audit`: drop list, predicates, and counts @@ -624,34 +628,29 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan - U6's scenarios pass; with `STRYKER_KILL_MATRIX` unset, `sharedConfig` is unchanged. - `node packages/stryker-js/dist/main.mjs audit --counts-only` over the downloaded `mutation-report-416` artifact gives 8626 planned and 4136 CompileError, output quoted in the PR body. It reads JSON only and instruments nothing. -### PR-W: Workflow wiring (base PR-B, owned by the root) +### PR-W: Workflow wiring (#288, base `main`) #### U9. CI wiring -- **Goal:** - - Main produces a kill matrix on a schedule and on dispatch. - - Every PR runs the drop audit against main's latest matrix and latest Mutation report. - - Every Mutation run publishes R18 counts. -- **Requirements:** R15, R16, R18. -- **Dependencies:** U1, U6, and Q14's bootstrap choice. The root owns and reviews this unit (`.github/workflows/` is Read-only). -- **Files:** - - New `.github/workflows/kill-matrix.yml` (under Q14 option (a), landed by the root before PR-W). - - `.github/workflows/ci.yml`: a new non-matrix `drop-audit` job. - - `.github/workflows/mutation.yml`: one counts step in the `report` job. +- **Goal:** every PR and every push to `main` runs the drop audit over the whole corpus against `main`'s newest kill matrix, and the job fails when the audit fails or no matrix exists. +- **Requirements:** R16 (KTD17). +- **Dependencies:** U6; `kill-matrix.yml` on `main` (Q14). The R18 counts step is Stream K's (Q20). +- **Files:** `.github/workflows/ci.yml`: a new non-matrix `drop-audit` job. - **Approach:** - - **Kill-matrix workflow** (schedule and dispatch, on `main`): it mirrors the `plan`, `build`, `mutation`, and `report` jobs of `mutation.yml` with `STRYKER_KILL_MATRIX=1` and `plan --full`. It restores and saves no incremental cache, and it uses its own artifact names: `kill-matrix-shard-*`, and `kill-matrix-` holding `reports/mutation/**/stryker-incremental.json`. The report job succeeds only if every merged incremental report exists, with no budget gate. A separate file keeps the Mutation workflow's contexts unchanged (`docs/solutions/workflow-issues/matrix-legs-rename-the-required-status-check.md`). - - **`drop-audit` job:** it builds the workspace `stryker-js` and downloads, through REST, the latest successful kill-matrix artifact. It then runs `node packages/stryker-js/dist/main.mjs audit --matrix --out audit.json` and uploads `audit.json`. If the artifact is missing, the job fails with `::error` and does not skip. Its context name `drop-audit` goes to the root, who decides whether to require it. - - **Counts step:** after `merge`, the workspace-built `audit --counts-only` writes `reports/mutation/mutation-counts.json` into the existing report artifact. -- **Patterns:** `mutation.yml:39-352`; `.github/actions/released-tarballs`; `docs/solutions/workflow-issues/mutation-lane-green-while-every-job-failed.md`. -- **Test expectation:** none. This is workflow wiring; its evidence is the artifacts and runs below. + - `needs: lane`, skipped for docs-only changes and bot pushes, like `check`; 60-minute timeout; `contents: read`, `actions: read`. + - Find the matrix by run, not in the repository-wide artifact list: walk the newest 100 completed `main` runs of this repository (fork runs excluded) of each workflow that publishes `kill-matrix-` (`mutation.yml` through its `kill-matrix` job, `kill-matrix.yml` on dispatch), newest first, and take the first run whose own artifacts (`actions/runs//artifacts`) hold an unexpired one; download it. The repository-wide list grows by 1200 to 2300 artifacts a day (`kill-matrix-455` was on page 6 five hours after upload), so a fixed page scan reports a false `matrix-missing`. 100 Mutation runs cover about a week, and every non-bot push to `main` publishes a matrix, so a miss means the lane is broken; the walk then costs at most 202 API calls. If no run in that window has a matrix, or the artifact holds no `/stryker-incremental.json`, fail with `::error` code `matrix-missing` and the dispatch command as next action. + - Build `packages/stryker-js` and the matrix's projects through turbo, restoring the Mutation lane's turbo cache by its existing key, hashed right after checkout as that lane hashes it (after install, `packages/**` also matches the installer's shims and the key can never match). `GH_TOKEN` is scoped to the lookup step; the lookup step runs under `bash -eo pipefail`, so a failed API read fails the step rather than reporting `matrix-missing`. + - Run `node packages/stryker-js/dist/main.mjs audit --matrix --projects --out reports/audit.json`, copy its output to the step summary, and upload `audit.json` (14 days, overwrite on re-run) whenever the audit wrote it. The audit's exit code is the job's verdict. + - No logic in YAML beyond artifact lookup: the verdict, counts, and next actions come from `stryker audit` (U6). +- **Patterns:** `.github/actions/released-tarballs`; #288's rulings (fork filter, overwrite, `matrix-missing`), but not its five-page artifact scan; `docs/solutions/workflow-issues/mutation-lane-green-while-every-job-failed.md` (the verdict comes from the command's exit, no `continue-on-error`). +- **Test expectation:** none. This is workflow wiring; its evidence is the run below. - **Verification:** on GitHub only, observed with `xd://github run_watch` (OP13), with run ids cited. **PR-W Definition of Done:** -- The root has approved the workflow diff. -- The first kill-matrix artifact is cited by run id and shows Killed mutants with more than one `killedBy`. -- `drop-audit` on PR-W's head reports the 76 S1 pairs joined, 0 failures, 0 orphaned tests, and no `Unattested` rule, cited by run id (under Q14 option (a)). -- The counts step's artifact is cited from the first Mutation run that carries it. +- `drop-audit` on PR-W's head audits the whole corpus against the newest `kill-matrix-`, reports 0 `Fail`, 0 orphaned tests and no `Unattested` rule, and exits 0, cited by run id. (Main's build against `kill-matrix-457` gives 99 `complement` pairs: 35 `Pass`, 61 `Vacuous`, 3 `AttributionUnverified`, 0 `Fail`.) +- `check`, `Changeset Check` and every other required check are green on PR-W's head. +- #288's description states the root's ownership of the job (CONST-E9, CONST-G5) and the code-review receipt. ### PR-C: Type-guided generation (base PR-W) @@ -665,7 +664,7 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan - `packages/stryker-js-plugin-interface/src/Checker.service.ts` (optional `facts`), `PluginRpcs.service.ts` (the `facts` RPC), and `ignore-rule.schema.ts` (`type-invalid`, KTD16). - New `packages/stryker-js/src/decide-type-invalid.workflow.ts` and `src/__tests__/decide-type-invalid.workflow.property.test.ts`. - `packages/stryker-js/src/audit-drops.workflow.ts` and its property test: the R40 branch and `NotCompiledOnMain`. `audit-request.cell.ts`, `Cli.schema.ts`, and `bin/cli-command.ts`: `--statuses `, which reads `//stryker-incremental.json` from main's latest Mutation artifact. - - `.github/workflows/ci.yml` (root-owned, as in U9): `drop-audit` also downloads main's latest successful Mutation artifact and passes `--statuses`. + - `.github/workflows/ci.yml`, as in U9: `drop-audit` also downloads main's latest successful Mutation artifact and passes `--statuses`. - `packages/stryker-js-plugin-interface/src/Mutant.schema.ts`, `packages/stryker-js-cli-contract/src/run-event.schema.ts`, `stream-version.schema.ts`, and `packages/stryker-js/src/IncrementalReport.schema.ts`: the `TypeInvalid` variant of the drop reference (KTD5). A new wire variant takes the next `StreamSchemaVersion` major and a `major` changeset for `@systemfsoftware/stryker-js-cli-contract` (Q13). - New `packages/stryker-js/src/run/type-facts.cell.ts`, called from `run/mutation-test.cell.ts:91`, `run/deferrable-dry-run.cell.ts:64-65`, `plan-request.cell.ts:201-214`, and `audit-request.cell.ts`. - `packages/stryker-js-instrumenter/src/Transformer.service.ts`: each candidate of an R33 shape records its shape on the mutant. @@ -715,7 +714,7 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan **PR-C Definition of Done:** -- The root has approved the `drop-audit` change. `check`, every `e2e (…)` leg, `Changeset Check`, and `drop-audit` are green on PR-C's head, cited by run id. `drop-audit` reports N > 0 `type-invalid` drops joined to main's statuses, 0 compiling mutants lost, and no `Unattested` rule. +- `check`, every `e2e (…)` leg, `Changeset Check`, and `drop-audit` are green on PR-C's head, cited by run id. `drop-audit` reports N > 0 `type-invalid` drops joined to main's statuses, 0 compiling mutants lost, and no `Unattested` rule. - The PR body states, from the audit's counts, the baseline (the R18 count of the last main Mutation run before PR-C, which includes PR-A; #416's 4136 CompileError, 47.9%, and 3921 s are the pre-PR-A reference) and the predicted after figure (baseline CompileError − N, with the baseline checker time of those N removed). It also states the fact-gathering time. - After the release that carries PR-C, the first main Mutation run's counts artifact shows CompileError and checker time falling by the predicted amounts, cited by run id. - If Q15 is pending when U14 is done, PR-C is pushed with U14 only and stops there. Its Definition of Done then covers U14's scenarios, and the corpus figures wait for U15. @@ -743,7 +742,7 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan | Single plan (REPO-D2) | `pnpm gate:repo` | the PR that carries this plan | | CI (START-4, START-5) | `check`, `e2e (…)`, and `Changeset Check` on each PR head, observed with `xd://github run_watch` | every PR | | Kill matrix | the `kill-matrix` workflow on `main`, cited by run id and artifact | from PR-W | -| No signal loss | `drop-audit` reports 0 failures, 0 orphaned tests, 0 compiling mutants lost | every PR from PR-W | +| No signal loss | `drop-audit` audits the whole corpus: 0 failures, 0 orphaned tests, 0 compiling mutants lost | every PR from PR-W | Locally: one build at a time, no e2e, no full-workspace test, no mutation run, and no long-lived `tsc`/watch process left running. A unit's tests count as passing only when the parent session ran them (VER1). From 048908579805a644aaddcb99ba6ce0bbeb28161d Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Tue, 13 Oct 2026 04:00:42 +0000 Subject: [PATCH 10/12] ci(repo): pin drop-audit's projects to the Mutation corpus The audited projects come from mutation.yml's PROJECTS, not from the downloaded artifact. A kill matrix missing any corpus project fails with matrix-missing, so a kill-matrix.yml dispatch over a subset of projects cannot narrow the audit. The plan records the root's 2026-10-13 rulings: one-layer stack on main, root ownership of drop-audit, the corpus pin, and no zero-pairs floor. --- .github/workflows/ci.yml | 9 +++++++-- .../2026-10-13-0342-feat-mutant-subsumption-plan.md | 12 +++++++++--- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ffc7ccc66..2f7dd428d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -749,8 +749,13 @@ jobs: [ -n "$found" ] || missing 'No unexpired kill-matrix- artifact is on the newest 100 completed main runs of mutation.yml or kill-matrix.yml.' read -r name run sha <<< "$found" gh run download "$run" --name "$name" --dir "$MATRIX_DIR" - projects=$(find "$MATRIX_DIR" -name stryker-incremental.json -printf '%h\n' | sed "s|^$MATRIX_DIR/||" | sort | paste -sd, -) - [ -n "$projects" ] || missing "Artifact $name of run $run holds no /stryker-incremental.json." + projects=$(awk '/^env:/ { inEnv = 1; next } inEnv && /^[^ ]/ { exit } inEnv && $1 == "PROJECTS:" { print $2; exit }' .github/workflows/mutation.yml) + if [ -z "$projects" ]; then + echo "::error title=Drop audit cannot read the corpus (corpus-unreadable)::.github/workflows/mutation.yml has no top-level env PROJECTS, so the job cannot tell which projects the kill matrix must cover." + exit 1 + fi + absent=$(tr ',' '\n' <<< "$projects" | while read -r project; do [ -f "$MATRIX_DIR/$project/stryker-incremental.json" ] || echo "$project"; done | paste -sd, -) + [ -z "$absent" ] || missing "Artifact $name of run $run has no kill matrix for $absent, so it does not cover the Mutation corpus ($projects)." { echo "name=$name" echo "sha=$sha" diff --git a/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md b/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md index 4c029181b..f2da4d7bd 100644 --- a/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md +++ b/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md @@ -324,6 +324,12 @@ Four PRs in this order, the ruling's PR shape (2026-10-09). PR-A and PR-B are me - Q12. Confirmed. `stryker audit` instruments and runs no tests, so it is not a mutation run, and it runs from the workspace build. - Q13. Confirmed. Whichever PR lands second, this unit's or Stream C U1, takes the next `StreamSchemaVersion` (`packages/stryker-js-cli-contract/src/stream-version.schema.ts:3`); merge main up when that happens. +**Rulings (2026-10-13, the root)** + +- #288 (PR-W) goes out as a one-layer stack on `main`. #319 and #286 also edit `.github/workflows/ci.yml`, in hunks that merge cleanly with it, so the OP13b overlap check's exit 1 is accepted for this PR and recorded in its body. +- The root owns `drop-audit` (CONST-E9); Stream I authors it under that ownership. +- The audited projects are pinned to the Mutation corpus (`mutation.yml`'s `PROJECTS`): an artifact missing any of them fails with `matrix-missing`. A zero-pairs floor is not added. + ### Open questions for the supervisor - Q14 (the root). **Resolved: option (a).** `kill-matrix.yml` is on `main` and called from the Mutation workflow; `kill-matrix-457` (run 38335732420, main `0aebc83e6`, kept 90 days) is the newest artifact. Original question: **PR-W bootstrap.** The `drop-audit` job needs one kill-matrix artifact, and a new workflow file cannot be dispatched until it exists on `main`. So on PR-W's own head the job can only fail with `NoMatrix`. Option (a): the root lands `kill-matrix.yml` on `main` alone first, dispatches it once, and PR-W then carries only the `drop-audit` job and the counts step, green on its head. Option (b): PR-W merges with `drop-audit` red as `NoMatrix`, not yet required, and the first green run is cited on PR-C. Recommended: (a), because every PR in the stack then meets "green on its head". A drop audit that passes without a matrix would be vacuous, so it does not pass. @@ -638,9 +644,9 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan - **Files:** `.github/workflows/ci.yml`: a new non-matrix `drop-audit` job. - **Approach:** - `needs: lane`, skipped for docs-only changes and bot pushes, like `check`; 60-minute timeout; `contents: read`, `actions: read`. - - Find the matrix by run, not in the repository-wide artifact list: walk the newest 100 completed `main` runs of this repository (fork runs excluded) of each workflow that publishes `kill-matrix-` (`mutation.yml` through its `kill-matrix` job, `kill-matrix.yml` on dispatch), newest first, and take the first run whose own artifacts (`actions/runs//artifacts`) hold an unexpired one; download it. The repository-wide list grows by 1200 to 2300 artifacts a day (`kill-matrix-455` was on page 6 five hours after upload), so a fixed page scan reports a false `matrix-missing`. 100 Mutation runs cover about a week, and every non-bot push to `main` publishes a matrix, so a miss means the lane is broken; the walk then costs at most 202 API calls. If no run in that window has a matrix, or the artifact holds no `/stryker-incremental.json`, fail with `::error` code `matrix-missing` and the dispatch command as next action. - - Build `packages/stryker-js` and the matrix's projects through turbo, restoring the Mutation lane's turbo cache by its existing key, hashed right after checkout as that lane hashes it (after install, `packages/**` also matches the installer's shims and the key can never match). `GH_TOKEN` is scoped to the lookup step; the lookup step runs under `bash -eo pipefail`, so a failed API read fails the step rather than reporting `matrix-missing`. - - Run `node packages/stryker-js/dist/main.mjs audit --matrix --projects --out reports/audit.json`, copy its output to the step summary, and upload `audit.json` (14 days, overwrite on re-run) whenever the audit wrote it. The audit's exit code is the job's verdict. + - Find the matrix by run, not in the repository-wide artifact list: walk the newest 100 completed `main` runs of this repository (fork runs excluded) of each workflow that publishes `kill-matrix-` (`mutation.yml` through its `kill-matrix` job, `kill-matrix.yml` on dispatch), newest first, and take the first run whose own artifacts (`actions/runs//artifacts`) hold an unexpired one; download it. The repository-wide list grows by 1200 to 2300 artifacts a day (`kill-matrix-455` was on page 6 five hours after upload), so a fixed page scan reports a false `matrix-missing`. 100 Mutation runs cover about a week, and every non-bot push to `main` publishes a matrix, so a miss means the lane is broken; the walk then costs at most 202 API calls. If no run in that window has a matrix, or the artifact lacks `/stryker-incremental.json` for any project in `mutation.yml`'s `PROJECTS` (the Mutation corpus), fail with `::error` code `matrix-missing` and the dispatch command as next action. A `kill-matrix.yml` dispatch over a subset of projects therefore cannot narrow the audit. + - Build `packages/stryker-js` and the corpus projects through turbo, restoring the Mutation lane's turbo cache by its existing key, hashed right after checkout as that lane hashes it (after install, `packages/**` also matches the installer's shims and the key can never match). `GH_TOKEN` is scoped to the lookup step; the lookup step runs under `bash -eo pipefail`, so a failed API read fails the step rather than reporting `matrix-missing`. + - Run `node packages/stryker-js/dist/main.mjs audit --matrix --projects --out reports/audit.json`, copy its output to the step summary, and upload `audit.json` (14 days, overwrite on re-run) whenever the audit wrote it. The audit's exit code is the job's verdict. - No logic in YAML beyond artifact lookup: the verdict, counts, and next actions come from `stryker audit` (U6). - **Patterns:** `.github/actions/released-tarballs`; #288's rulings (fork filter, overwrite, `matrix-missing`), but not its five-page artifact scan; `docs/solutions/workflow-issues/mutation-lane-green-while-every-job-failed.md` (the verdict comes from the command's exit, no `continue-on-error`). - **Test expectation:** none. This is workflow wiring; its evidence is the run below. From 3d1fecc513128daf5f244f03eb3a35196fdf2c3a Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Tue, 13 Oct 2026 07:42:54 +0000 Subject: [PATCH 11/12] ci(repo): write drop-audit's wall time to the step summary The full-path review of #288 (run 20261013-072157-4dd07ea9) confirmed a ci-wall-clock violation: every job that runs steps writes its wall time to the step summary (one-stuck-job-backstop), and drop-audit wrote none. Its first step now records the start, and a final if: always() step writes the wall time, so a failed audit still reports how long it ran. The 60-minute backstop was already in place. --- .github/workflows/ci.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2f7dd428d..4441b44f6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -712,6 +712,8 @@ jobs: MATRIX_DIR: kill-matrix AUDIT_OUT: reports/audit.json steps: + - id: start + run: echo "at=$(date +%s)" >> "$GITHUB_OUTPUT" - uses: actions/checkout@v7 - id: build-key name: Key of the Mutation lane's build cache, hashed before install as that lane hashes it @@ -811,3 +813,8 @@ jobs: path: ${{ env.AUDIT_OUT }} retention-days: 14 overwrite: true + - if: always() + env: + STARTED: ${{ steps.start.outputs.at }} + run: | + echo "wall time: $(( $(date +%s) - STARTED )) s" >> "$GITHUB_STEP_SUMMARY" From ba0a22e416af5c74906aa912a4ec2f328ba4118a Mon Sep 17 00:00:00 2001 From: systemfsoftware-maker Date: Tue, 13 Oct 2026 08:02:16 +0000 Subject: [PATCH 12/12] docs(plans): reissue the subsumption plan with S1 counts tied to their matrix Review run 20261013-072157-4dd07ea9 of #288 confirmed finding 7: the Success Criteria said S1 drops 76 mutants and the PR-W Definition of Done cited 99 pairs, with nothing saying both are snapshots. The 76 is #416's corpus; the audit's pair count follows the matrix it reads (99 against kill-matrix-457 on main, 100 against kill-matrix-459 on this PR's head, run 38344422811). The plan is under execution (PR-A and PR-B are merged), so it is reissued as 2026-10-13-0801, which supersedes 0342, instead of edited in place. --- ...n.md => 2026-10-13-0801-feat-mutant-subsumption-plan.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename docs/plans/{2026-10-13-0342-feat-mutant-subsumption-plan.md => 2026-10-13-0801-feat-mutant-subsumption-plan.md} (99%) diff --git a/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md b/docs/plans/2026-10-13-0801-feat-mutant-subsumption-plan.md similarity index 99% rename from docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md rename to docs/plans/2026-10-13-0801-feat-mutant-subsumption-plan.md index f2da4d7bd..2ea12f578 100644 --- a/docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md +++ b/docs/plans/2026-10-13-0801-feat-mutant-subsumption-plan.md @@ -3,7 +3,7 @@ title: Mutant Subsumption and Type-Guided Generation - Plan type: feat date: 2026-10-13 topic: mutant-subsumption -supersedes: docs/plans/2026-10-09-2024-feat-mutant-subsumption-plan.md +supersedes: docs/plans/2026-10-13-0342-feat-mutant-subsumption-plan.md artifact_contract: ce-unified-plan/v1 product_contract_source: ce-brainstorm execution: code @@ -272,7 +272,7 @@ flowchart TB ### Success Criteria -- On the corpus, S1 drops 76 mutants, and the R16 check on the first kill matrix (available once PR-W lands) reports 0 failures. +- On #416's corpus, S1 drops 76 mutants. The R16 check on the first kill matrix (available once PR-W lands) reports 0 failures; its drop count is the audit's measured count against that matrix, not the 76. - On the corpus, PR-C's audit reports N > 0 `type-invalid` drops with 0 compiling mutants lost. N is at most the 1170-mutant ceiling measured on #406. The baseline is the R18 count of the last main Mutation run before PR-C's release, which already includes PR-A's R4 repair and S1; #416 (4136 CompileError, 47.9%; 3921 s) is the reference before PR-A. The first main Mutation run after PR-C's release shows CompileError falling by N from that baseline, and checker milliseconds over CompileError falling by the baseline cost of those N mutants. - The R18 counts appear as a CI artifact for each before/after pair, and the dropped ids match the R16 drop list. @@ -654,7 +654,7 @@ Units are grouped by PR. Within a PR, the order shown is the commit order. Mutan **PR-W Definition of Done:** -- `drop-audit` on PR-W's head audits the whole corpus against the newest `kill-matrix-`, reports 0 `Fail`, 0 orphaned tests and no `Unattested` rule, and exits 0, cited by run id. (Main's build against `kill-matrix-457` gives 99 `complement` pairs: 35 `Pass`, 61 `Vacuous`, 3 `AttributionUnverified`, 0 `Fail`.) +- `drop-audit` on PR-W's head audits the whole corpus against the newest `kill-matrix-`, reports 0 `Fail`, 0 orphaned tests and no `Unattested` rule, and exits 0, cited by run id. (The pair count follows the matrix's corpus, not #416's 76: main's build against `kill-matrix-457` gave 99 `complement` pairs, and PR-W at `048908579` against `kill-matrix-459` gave 100: 36 `Pass`, 61 `Vacuous`, 3 `AttributionUnverified`, 0 `Fail`, run 38344422811.) - `check`, `Changeset Check` and every other required check are green on PR-W's head. - #288's description states the root's ownership of the job (CONST-E9, CONST-G5) and the code-review receipt.