diff --git a/.changeset/record-check-time-for-untested-verdicts.md b/.changeset/record-check-time-for-untested-verdicts.md new file mode 100644 index 000000000..7f840a387 --- /dev/null +++ b/.changeset/record-check-time-for-untested-verdicts.md @@ -0,0 +1,5 @@ +--- +"@systemfsoftware/stryker-js": minor +--- + +A verdict the engine decided without running a test now records the cost it measured. A mutant whose checker rejected it, or which no test covered while a checker looked at it, records its share of that check call's wall time; a mutant every configured checker ignored, or one left uncovered with no checker configured, records zero. A static mutant that a checker rejected carries that check time in the `StaticVerdict` event's `costMs` as well. A shard plan therefore balances on recorded times instead of pricing those verdicts at a whole-suite prediction, which previously dominated the plan. Killed, Survived and Timeout verdicts keep the measured test time they already recorded, and the run budget still counts only the verdicts that ran a test. diff --git a/docs/solutions/performance-issues/shard-plan-priced-untested-verdicts-at-a-whole-suite-prediction.md b/docs/solutions/performance-issues/shard-plan-priced-untested-verdicts-at-a-whole-suite-prediction.md new file mode 100644 index 000000000..353fe0370 --- /dev/null +++ b/docs/solutions/performance-issues/shard-plan-priced-untested-verdicts-at-a-whole-suite-prediction.md @@ -0,0 +1,48 @@ +--- +title: Shard plans balanced on CompileError counts because untested verdicts recorded no cost +date: 2026-10-07 +category: performance-issues +module: shard-planning +problem_type: performance_issue +component: tooling +symptoms: + - "Mutation shards finish minutes apart although the plan predicts equal shard seconds" + - "Every shard receives an almost identical count of CompileError mutants" + - "The incremental report's costs entries for CompileError, NoCoverage and Ignored mutants have actualMs null" +root_cause: logic_error +resolution_type: code_fix +severity: medium +tags: [mutation-testing, sharding, lpt, cost-record, compile-error, incremental-report] +--- + +# Shard plans balanced on CompileError counts because untested verdicts recorded no cost + +## Problem + +`stryker plan` bin-packs mutants longest-first at `costs[id].actualMs ?? costs[id].predictedMs`. A verdict reached without running a test (CompileError, NoCoverage, Ignored) recorded no cost, so `actualMs` was null. The plan then priced it at `predictedMs`, the summed time of every test covering that mutant, even though no test would run for it. + +## Symptoms + +Main Mutation run 37602076567: stryker-js had 3547 CompileError mutants priced at about 68 s each, 241,914 s in total. All its Killed and Survived mutants together measured 3,216 s. The planner therefore balanced CompileError counts (177 or 178 per shard), while the test time each shard actually ran ranged from 88 s to 237 s. Shard wall time follows the test time (99 s + 1.51 × test seconds, r = 0.91), so the Mutation step ranged from 271 s to 498 s across shards. + +## What Didn't Work + +- **Pricing untested verdicts at 0.** LPT places each zero-cost item into the bin that is currently least loaded, and that bin does not change while only zero-cost items are being placed. Every check would land in a single shard. +- **Capping CompileErrors per shard.** This is a second heuristic layered on the first, not a measurement. + +## Solution + +Record what each verdict measurably cost. The checker pool times each `check` call and splits the elapsed time evenly across that call's mutants. A mutant that several checkers see sums its shares. A CompileError, or a NoCoverage verdict that a checker looked at, carries `{ fixedOverheadMs: checkShare, testBodyMs: 0, testsExecuted: 0 }` as its verdict cost. An Ignored verdict, or NoCoverage with no checker, records 0 because nothing ran. The run budget still prices only verdicts that ran a test (`decidedWithoutATest` in `mutant-cost.ts`), so budget gates are unchanged. + +## Why This Works + +The planner's fallback assumes a mutant with no recorded cost will run its covering tests. That holds for a mutant that was never measured. It does not hold for a mutant whose verdict a checker or the coverage map has already decided. Once these verdicts carry a measured cost, the plan prices the work that will actually happen. + +## Prevention + +- A new verdict path must record a finite `costs[id].actualMs`. If it records null, the planner falls back to the whole-suite prediction. Gate: `pnpm --filter @systemfsoftware/stryker-js exec vitest run tests/check-cost-record.integration.test.ts tests/plan-recorded-costs.integration.test.ts`. + - wrong: a new early verdict is returned without a `cost`. + - right: it carries `checkOnlyCostOf(measuredMs)`, or 0 when nothing ran. +- Judge plan quality by comparing per-shard Mutation-step seconds (from the run's jobs API) with per-shard recorded test seconds (from the merged report). The plan's `predictedSeconds` is not evidence. Gate: review. The reviewer cites both numbers by run id. + - wrong: "the plan predicts equal shards." + - right: "run 37602076567: Mutation step 271–498 s, recorded test time 88–237 s per shard." diff --git a/packages/stryker-js/src/Checker/checker-pool.handle.ts b/packages/stryker-js/src/Checker/checker-pool.handle.ts index 76910b43c..ccbf4a6dd 100644 --- a/packages/stryker-js/src/Checker/checker-pool.handle.ts +++ b/packages/stryker-js/src/Checker/checker-pool.handle.ts @@ -3,6 +3,7 @@ import { SpanTaxonomy } from '@systemfsoftware/stryker-js-cli-contract' import type { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' import { Checker } from '@systemfsoftware/stryker-js-plugin-interface' import * as Array from 'effect/Array' +import * as Duration from 'effect/Duration' import * as Effect from 'effect/Effect' import * as Exit from 'effect/Exit' import { dual } from 'effect/Function' @@ -36,6 +37,7 @@ export interface CheckedPlans { readonly passedPlans: readonly Mutant.MutantRunPlan[] readonly failedChecks: readonly (readonly [Mutant.MutantRunPlan, Checker.FailedCheckResult])[] readonly ignoredChecks: readonly (readonly [Mutant.MutantRunPlan, Checker.IgnoredCheckResult])[] + readonly checkMsByMutantId: Readonly> } const CheckerPoolHandle = Handle.make, CheckerPool>()(TypeId) @@ -112,6 +114,28 @@ const noFailedChecks: readonly (readonly [Mutant.MutantRunPlan, Checker.FailedCh const noIgnoredChecks: readonly (readonly [Mutant.MutantRunPlan, Checker.IgnoredCheckResult])[] = [] +const noCheckMs: Readonly> = {} + +const checkMsOf = (checkMsByMutantId: Readonly>, mutantId: string): number => + Option.getOrElse(Option.fromUndefinedOr(checkMsByMutantId[mutantId]), () => 0) + +const chargedWith = ( + carried: Readonly>, + group: readonly Mutant.MutantRunPlan[], + elapsedMs: number, +): Readonly> => { + if (group.length === 0) { + return carried + } + const shareMs = elapsedMs / group.length + return { + ...carried, + ...Object.fromEntries( + group.map((plan) => [plan.mutant.id, checkMsOf(carried, plan.mutant.id) + shareMs] as const), + ), + } +} + export const splitCheckedPlans = Effect.fn(SpanTaxonomy.Spans.checkerPoolSplitChecked.name)(function*( checked: readonly (readonly [Mutant.MutantRunPlan, Checker.CheckResult])[], ) { @@ -146,7 +170,7 @@ export const splitCheckedPlans = Effect.fn(SpanTaxonomy.Spans.checkerPoolSplitCh ), }), ) - return { passedPlans, failedChecks, ignoredChecks } + return { passedPlans, failedChecks, ignoredChecks } satisfies Omit }) const checkerNamesOf = (pool: CheckerPool): Effect.Effect => @@ -154,20 +178,32 @@ const checkerNamesOf = (pool: CheckerPool): Effect.Effect>, ): Stream.Stream => Option.match(Array.head(failedChecks), { onNone: () => Stream.empty, onSome: () => - Stream.succeed({ passedPlans: noPassedPlans, failedChecks, ignoredChecks: noIgnoredChecks }), + Stream.succeed({ + passedPlans: noPassedPlans, + failedChecks, + ignoredChecks: noIgnoredChecks, + checkMsByMutantId, + }), }) const ignoredElement = ( ignoredChecks: readonly (readonly [Mutant.MutantRunPlan, Checker.IgnoredCheckResult])[], + checkMsByMutantId: Readonly>, ): Stream.Stream => Option.match(Array.head(ignoredChecks), { onNone: () => Stream.empty, onSome: () => - Stream.succeed({ passedPlans: noPassedPlans, failedChecks: noFailedChecks, ignoredChecks }), + Stream.succeed({ + passedPlans: noPassedPlans, + failedChecks: noFailedChecks, + ignoredChecks, + checkMsByMutantId, + }), }) const checkedGroupsFor = ( @@ -175,6 +211,7 @@ const checkedGroupsFor = ( checkerNames: readonly string[], checkerIndex: number, plans: readonly Mutant.MutantRunPlan[], + carried: Readonly>, ): Stream.Stream => Option.match(Array.get(checkerNames, checkerIndex), { onNone: () => @@ -182,6 +219,7 @@ const checkedGroupsFor = ( passedPlans: plans, failedChecks: noFailedChecks, ignoredChecks: noIgnoredChecks, + checkMsByMutantId: carried, }), onSome: (checkerName) => Stream.unwrap( @@ -190,18 +228,26 @@ const checkedGroupsFor = ( Stream.fromIterable(groups).pipe( Stream.mapEffect( (group) => - onCheckerSlot(pool, checkerIndex, (checker) => checkPlansWithChecker(checker, checkerName, group)) - .pipe( - Effect.flatMap((checked) => splitCheckedPlans(checked)), - ), + onCheckerSlot(pool, checkerIndex, (checker) => + Effect.timed(checkPlansWithChecker(checker, checkerName, group)).pipe( + Effect.flatMap(([elapsed, checked]) => + Effect.map( + splitCheckedPlans(checked), + (split) => ({ + split, + charged: chargedWith(carried, group, Duration.toMillis(elapsed)), + }), + ) + ), + )), { concurrency: 'unbounded', unordered: true }, ), - Stream.flatMap((split) => + Stream.flatMap(({ split, charged }) => Stream.concat( - ignoredElement(split.ignoredChecks), + ignoredElement(split.ignoredChecks, charged), Stream.concat( - failedElement(split.failedChecks), - checkedGroupsFor(pool, checkerNames, checkerIndex + 1, split.passedPlans), + failedElement(split.failedChecks, charged), + checkedGroupsFor(pool, checkerNames, checkerIndex + 1, split.passedPlans, charged), ), ) ), @@ -232,10 +278,13 @@ export const checkPlansStream: { passedPlans: plans, failedChecks: noFailedChecks, ignoredChecks: noIgnoredChecks, + checkMsByMutantId: noCheckMs, }), onSome: (handle) => checkerNamesOf(CheckerPoolHandle.slot(handle)).pipe( - Effect.map((checkerNames) => checkedGroupsFor(CheckerPoolHandle.slot(handle), checkerNames, 0, plans)), + Effect.map((checkerNames) => + checkedGroupsFor(CheckerPoolHandle.slot(handle), checkerNames, 0, plans, noCheckMs) + ), Stream.unwrap, ), }), @@ -251,18 +300,24 @@ export const checkPlans = Effect.fnUntraced(function*( const passedPlans: Mutant.MutantRunPlan[] = [] const failedChecks: (readonly [Mutant.MutantRunPlan, Checker.FailedCheckResult])[] = [] const ignoredChecks: (readonly [Mutant.MutantRunPlan, Checker.IgnoredCheckResult])[] = [] + const checkMsByMutantId: Record = {} for (const group of checkedGroups) { passedPlans.push(...group.passedPlans) failedChecks.push(...group.failedChecks) ignoredChecks.push(...group.ignoredChecks) + Object.assign(checkMsByMutantId, group.checkMsByMutantId) } - return { passedPlans, failedChecks, ignoredChecks } satisfies CheckedPlans + return { passedPlans, failedChecks, ignoredChecks, checkMsByMutantId } satisfies CheckedPlans }) export interface CheckedPlansExecution { - readonly settleFailure: (plan: Mutant.MutantRunPlan, result: Checker.FailedCheckResult) => Effect.Effect + readonly settleFailure: ( + plan: Mutant.MutantRunPlan, + result: Checker.FailedCheckResult, + checkMs: number, + ) => Effect.Effect readonly settleIgnored: (plan: Mutant.MutantRunPlan, result: Checker.IgnoredCheckResult) => Effect.Effect - readonly runPlan: (plan: Mutant.MutantRunPlan) => Effect.Effect + readonly runPlan: (plan: Mutant.MutantRunPlan, checkMs: number) => Effect.Effect readonly concurrency: number } @@ -283,11 +338,13 @@ export const runCheckedPlans: { execution: CheckedPlansExecution, ): Stream.Stream => self.pipe( - Stream.flatMap(({ passedPlans, failedChecks, ignoredChecks }) => + Stream.flatMap(({ passedPlans, failedChecks, ignoredChecks, checkMsByMutantId }) => Stream.fromIterable<() => Effect.Effect>([ - ...failedChecks.map(([plan, result]) => () => execution.settleFailure(plan, result)), + ...failedChecks.map(([plan, result]) => () => + execution.settleFailure(plan, result, checkMsOf(checkMsByMutantId, plan.mutant.id)) + ), ...ignoredChecks.map(([plan, result]) => () => execution.settleIgnored(plan, result)), - ...passedPlans.map((plan) => () => execution.runPlan(plan)), + ...passedPlans.map((plan) => () => execution.runPlan(plan, checkMsOf(checkMsByMutantId, plan.mutant.id))), ]) ), Stream.mapEffect((work) => work(), { concurrency: Math.max(1, execution.concurrency), unordered: true }), diff --git a/packages/stryker-js/src/budget.ts b/packages/stryker-js/src/budget.ts index 2ebd917c7..3ca72116a 100644 --- a/packages/stryker-js/src/budget.ts +++ b/packages/stryker-js/src/budget.ts @@ -1,8 +1,9 @@ import { RunEvent } from '@systemfsoftware/stryker-js-cli-contract' import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' import * as Arr from 'effect/Array' +import * as Boolean from 'effect/Boolean' -import { costTotalMsOf } from './mutant-cost.js' +import { costOrZero, decidedWithoutATest } from './mutant-cost.js' const MILLIS_PER_SECOND = 1000 @@ -13,11 +14,15 @@ export interface BudgetInput { readonly planPredictedSeconds?: number | undefined } -const costMsOf = (result: Mutant.RunMutantResult): number => costTotalMsOf(result.cost) ?? 0 +const testWorkMsOf = (result: Mutant.RunMutantResult): number => + Boolean.match(decidedWithoutATest(result.status), { + onTrue: () => 0, + onFalse: () => costOrZero(result.cost), + }) const predictedSecondsOf = (results: readonly Mutant.RunMutantResult[], concurrency: number): number => { const lanes = concurrency > 0 ? concurrency : 1 - return Arr.reduce(results, 0, (total, result) => total + costMsOf(result)) / lanes / MILLIS_PER_SECOND + return Arr.reduce(results, 0, (total, result) => total + testWorkMsOf(result)) / lanes / MILLIS_PER_SECOND } export const budgetOf = (input: BudgetInput): RunEvent.Budget => ({ diff --git a/packages/stryker-js/src/mutant-cost.ts b/packages/stryker-js/src/mutant-cost.ts index c677d8c07..1d585cd67 100644 --- a/packages/stryker-js/src/mutant-cost.ts +++ b/packages/stryker-js/src/mutant-cost.ts @@ -1,5 +1,6 @@ import type { Mutant, TestRunner } from '@systemfsoftware/stryker-js-plugin-interface' import * as Match from 'effect/Match' +import * as Option from 'effect/Option' export interface MutantCostInput { readonly elapsedMs: number @@ -23,6 +24,24 @@ const bodyTimeWithinWall = (elapsedMs: number, testBodyMs: number): number => Ma export const costTotalMsOf = (cost: Mutant.MutantCost | undefined): number | undefined => cost === undefined ? undefined : cost.fixedOverheadMs + cost.testBodyMs +export const costOrZero = (cost: Mutant.MutantCost | undefined): number => + Option.getOrElse(Option.fromUndefinedOr(costTotalMsOf(cost)), () => 0) + +const DECIDED_WITHOUT_A_TEST: Partial> = { + CompileError: true, + NoCoverage: true, + Ignored: true, +} + +export const decidedWithoutATest = (status: Mutant.MutantStatus): boolean => DECIDED_WITHOUT_A_TEST[status] === true + +export const checkOnlyCostOf = (checkMs: number): Mutant.MutantCost => ({ + fixedOverheadMs: checkMs, + testBodyMs: 0, + testsExecuted: 0, + shared: false, +}) + export const mutantCostOf = (input: MutantCostInput): Mutant.MutantCost => { const testBodyMs = bodyTimeWithinWall(input.elapsedMs, input.testBodyMs) return { diff --git a/packages/stryker-js/src/reporting/static-verdict.ts b/packages/stryker-js/src/reporting/static-verdict.ts index 95656db9d..cadc17120 100644 --- a/packages/stryker-js/src/reporting/static-verdict.ts +++ b/packages/stryker-js/src/reporting/static-verdict.ts @@ -3,10 +3,7 @@ import { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' import * as Arr from 'effect/Array' import * as Option from 'effect/Option' -import { costTotalMsOf } from '../mutant-cost.js' - -const costOrZero = (cost: Mutant.MutantCost | undefined): number => - Option.getOrElse(Option.fromUndefinedOr(costTotalMsOf(cost)), () => 0) +import { costOrZero } from '../mutant-cost.js' export const staticVerdictOf = (results: readonly Mutant.RunMutantResult[]): RunEvent.StaticVerdict | null => { const statics = Arr.filter(results, (result) => result.static === true) diff --git a/packages/stryker-js/src/run/mutation-test.cell.ts b/packages/stryker-js/src/run/mutation-test.cell.ts index 844bd38b3..2b9a17ba0 100644 --- a/packages/stryker-js/src/run/mutation-test.cell.ts +++ b/packages/stryker-js/src/run/mutation-test.cell.ts @@ -2,6 +2,7 @@ import { Sandwich } from '@systemfsoftware/effect-cell-types' import { RunEvent, SpanTaxonomy } from '@systemfsoftware/stryker-js-cli-contract' import type { Mutant } from '@systemfsoftware/stryker-js-plugin-interface' import type { Plugin } from '@systemfsoftware/stryker-js-plugin-interface' +import * as Boolean from 'effect/Boolean' import * as Clock from 'effect/Clock' import * as Duration from 'effect/Duration' import * as Effect from 'effect/Effect' @@ -24,6 +25,7 @@ import { makeCheckerPoolHandle, runCheckedPlans, } from '../Checker/checker-pool.handle.js' +import { checkOnlyCostOf, decidedWithoutATest } from '../mutant-cost.js' import { MutationReporting } from '../mutation-reporting.service.js' import { MutationTestCommand } from '../MutationTest.schema.js' import { withPhaseSpan } from '../reporter-stream.service.js' @@ -60,6 +62,12 @@ const countIgnoredByReason = ( reason: string, ): number => results.filter((result) => result.status === 'Ignored' && result.statusReason === reason).length +const withMeasuredCheckCost = (result: Mutant.RunMutantResult, checkMs: number): Mutant.RunMutantResult => + Boolean.match(Boolean.and(result.cost === undefined, decidedWithoutATest(result.status)), { + onFalse: () => result, + onTrue: () => ({ ...result, cost: checkOnlyCostOf(checkMs) }), + }) + export interface MutationTestDone { readonly results: readonly Mutant.RunMutantResult[] readonly verdict: Plugin.ExitClass | null @@ -169,7 +177,10 @@ const proceedPipeline = Effect.fnUntraced(function*(raw: MutationTestRaw) { plannedMutants: [...rememberedResults, ...reuse.mutants], pathService, } - const settledResults = [...rememberedResults, ...plan.earlyResults] + const settledResults = [ + ...rememberedResults, + ...plan.earlyResults.map((result) => withMeasuredCheckCost(result, 0)), + ] yield* Effect.forEach(settledResults, (result) => announceSettledMutant(context, result), { concurrency: 1, discard: true, @@ -187,19 +198,22 @@ const proceedPipeline = Effect.fnUntraced(function*(raw: MutationTestRaw) { { total: plan.plannedTotal, testRunners: prev.concurrency.testRunners }, () => runCheckedPlans(checkedPlans, { - settleFailure: (mutantPlan, result) => + settleFailure: (mutantPlan, result, checkMs) => reporting.reportCheckFailure(toReportedMutant(mutantPlan.mutant), result).pipe( + Effect.map((reported) => withMeasuredCheckCost(reported, checkMs)), Effect.flatMap(settleReported), ), settleIgnored: (mutantPlan, result) => reporting.reportIgnored(toReportedMutant(mutantPlan.mutant), result).pipe( + Effect.map((reported) => withMeasuredCheckCost(reported, 0)), Effect.flatMap(settleReported), ), - runPlan: (runPlan) => + runPlan: (runPlan, checkMs) => Option.match(Option.liftPredicate(runPlan, isNoCoveragePlan), { onNone: () => Effect.scoped(mutantRunCell.run({ context, testRunnerPool, checkpoint, plan: runPlan })), onSome: (noCoverageRunPlan) => reporting.reportNoCoverage(toReportedMutant(noCoverageRunPlan.mutant)).pipe( + Effect.map((reported) => withMeasuredCheckCost(reported, checkMs)), Effect.flatMap(settleReported), ), }), diff --git a/packages/stryker-js/tests/__fixtures__/check-cost-workspace.fixture.ts b/packages/stryker-js/tests/__fixtures__/check-cost-workspace.fixture.ts new file mode 100644 index 000000000..ec6b60cd0 --- /dev/null +++ b/packages/stryker-js/tests/__fixtures__/check-cost-workspace.fixture.ts @@ -0,0 +1,271 @@ +import { NodeFileSystem, NodePath } from '@effect/platform-node' +import { Engine } from '@systemfsoftware/stryker-js' +import { RunEvent } from '@systemfsoftware/stryker-js-cli-contract' +import { type Options } from '@systemfsoftware/stryker-js-plugin-interface' +import * as Arr from 'effect/Array' +import type * as Cause from 'effect/Cause' +import * as Effect from 'effect/Effect' +import * as FileSystem from 'effect/FileSystem' +import { dual } from 'effect/Function' +import * as Layer from 'effect/Layer' +import * as Option from 'effect/Option' +import * as Path from 'effect/Path' +import * as Queue from 'effect/Queue' +import * as S from 'effect/Schema' + +const PACKAGE_ROOT = decodeURIComponent(new URL('../..', import.meta.url).pathname).replace(/\/$/, '') + +const CHECKER_PLUGIN = new URL('./cost-checker/index.mjs', import.meta.url).href + +export const REPORT_FILE = 'reports/main.json' + +const COVERING_TEST_WALL_MS = 120 + +const REJECTED_SOURCE = [ + 'export const hit = (value: number): number => value + 1', + 'export const miss = (value: number): number => value * 2', + '', +].join('\n') + +const IGNORED_SOURCE = 'export const skipped = (value: number): number => value + 1\n' + +const KEPT_SOURCE = 'export const kept = (left: number, right: number): number => left * right + 1\n' + +const UNREACHED_SOURCE = 'export const unreached = (value: number): number => value * 2\n' + +const SLOW_TEST_SOURCE = [ + "import { expect, test } from 'vitest'", + "import { hit } from '../src/lib/rejected.ts'", + "import { kept } from '../src/lib/kept.ts'", + '', + "test('exercises the covered modules with wall time to spare', () => {", + ' expect(hit(1)).toBe(2)', + ' expect(kept(2, 3)).toBe(7)', + ` const until = Date.now() + ${COVERING_TEST_WALL_MS}`, + ' while (Date.now() < until) {}', + '})', + '', +].join('\n') + +const FAST_TEST_SOURCE = [ + "import { expect, test } from 'vitest'", + "import { kept } from '../src/lib/kept.ts'", + '', + "test('doubles what the kept module computes', () => {", + ' expect(kept(2, 3)).toBe(7)', + '})', + '', +].join('\n') + +export const checkedWorkspaceFiles: ReadonlyArray = [ + ['package.json', '{ "type": "commonjs" }\n'], + ['src/lib/rejected.ts', REJECTED_SOURCE], + ['src/lib/ignored.ts', IGNORED_SOURCE], + ['src/lib/kept.ts', KEPT_SOURCE], + ['src/lib/unreached.ts', UNREACHED_SOURCE], + ['test/sample.test.mjs', SLOW_TEST_SOURCE], +] + +export const uncheckedWorkspaceFiles: ReadonlyArray = [ + ['package.json', '{ "type": "commonjs" }\n'], + ['src/lib/kept.ts', KEPT_SOURCE], + ['src/lib/unreached.ts', UNREACHED_SOURCE], + ['test/sample.test.mjs', FAST_TEST_SOURCE], +] + +export const filePorts = Layer.mergeAll(NodeFileSystem.layer, NodePath.layer) + +export const PLAN_FILE = 'plan.json' + +export const planWorkspaceFiles: ReadonlyArray = [ + ['package.json', '{ "type": "commonjs" }\n'], + ['src/lib/rejected.ts', REJECTED_SOURCE], + ['src/lib/ignored.ts', IGNORED_SOURCE], + ['src/lib/kept.ts', KEPT_SOURCE], + ['src/lib/unreached.ts', UNREACHED_SOURCE], +] + +export const cliConfigTextOf = (directory: string): string => + `export default ${ + JSON.stringify( + { + testRunner: 'command', + commandRunner: { command: 'true' }, + checkers: [{ plugin: CHECKER_PLUGIN }], + testFiles: ['test/**/*.mjs'], + mutate: ['src/**/*.ts'], + coverageAnalysis: 'off', + concurrency: 1, + reporters: [], + tempDirName: `.stryker-tmp-${directory.slice(directory.lastIndexOf('/') + 1)}`, + cleanTempDir: 'always', + incremental: true, + incrementalFile: `${directory}/${REPORT_FILE}`, + }, + null, + 2, + ) + }\n` + +const environmentFor = (directory: string): Engine.RunEnvironmentShape => ({ + runId: '01ARZ3NDEKTSV4RRFFQ69G5FAX', + resolvedMode: { mode: 'machine', signal: 'flag', stdoutIsTTY: false }, + runStartedAt: 0, + basePath: directory, + builtinReporters: {}, + allowConsoleColors: false, +}) + +const workspaceNameOf = (directory: string): string => directory.slice(directory.lastIndexOf('/') + 1) + +export const checkedOptionsOf = (directory: string): Options.PartialStrykerOptions => ({ + testRunner: 'vm', + plugins: [], + reporters: [], + checkers: [{ plugin: CHECKER_PLUGIN }], + testFiles: ['test/**/*.mjs'], + mutate: ['src/**/*.ts'], + coverageAnalysis: 'perTest', + concurrency: 1, + tempDirName: `.stryker-tmp-${workspaceNameOf(directory)}`, + cleanTempDir: 'always', + incremental: true, + incrementalFile: `${directory}/${REPORT_FILE}`, +}) + +export const uncheckedOptionsOf = (directory: string): Options.PartialStrykerOptions => ({ + ...checkedOptionsOf(directory), + checkers: [], +}) + +export const writeWorkspace = ( + files: ReadonlyArray, +): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + const root = yield* fs.realPath(yield* fs.makeTempDirectory({ prefix: 'stryker-check-cost-' })) + yield* Effect.forEach( + files, + ([file, content]) => + Effect.gen(function*() { + const target = path.join(root, file) + yield* fs.makeDirectory(path.dirname(target), { recursive: true }) + yield* fs.writeFileString(target, content) + }), + { discard: true }, + ) + yield* fs.symlink(path.join(PACKAGE_ROOT, 'node_modules'), path.join(root, 'node_modules')) + return root + }).pipe(Effect.orDie) + +export const removeWorkspace = (root: string): Effect.Effect => + Effect.provide( + Effect.orDie(Effect.flatMap(FileSystem.FileSystem, (fs) => fs.remove(root, { recursive: true, force: true }))), + filePorts, + ) + +export interface MutantCostEntry { + readonly predictedMs: number + readonly actualMs: number | null + readonly coveringTests: number +} + +export interface Observation { + readonly costs: Readonly> + readonly statuses: Readonly> + readonly idsByFile: Readonly> + readonly streamCosts: Readonly> + readonly verdictBudget: RunEvent.Budget | null +} + +export type ReportObservation = Pick + +const readReport = (text: string): ReportObservation => + Option.match(S.decodeOption(S.fromJsonString(Engine.IncrementalReportSchema))(text), { + onNone: () => ({ costs: {}, statuses: {}, idsByFile: {} }), + onSome: (report) => { + const files = Object.entries(report.files) + const mutants = files.flatMap(([, file]) => file.mutants) + return { + costs: report.costs, + statuses: Object.fromEntries(mutants.map((mutant) => [mutant.id, mutant.status])), + idsByFile: Object.fromEntries(files.map(([file, entry]) => [file, entry.mutants.map((mutant) => mutant.id)])), + } + }, + }) + +export const readReportIn = ( + directory: string, +): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + const text = yield* fs.readFileString(path.join(directory, REPORT_FILE)).pipe(Effect.orElseSucceed(() => '')) + return readReport(text) + }).pipe(Effect.orDie) + +const streamCostsOf = (events: ReadonlyArray): Readonly> => + Object.fromEntries( + events.flatMap((event) => + S.is(RunEvent.RunMutantTestedEvent)(event) && event.cost !== null + ? [[event.id, event.cost.fixedOverheadMs + event.cost.testBodyMs] as const] + : [] + ), + ) + +const verdictBudgetOf = (events: ReadonlyArray): RunEvent.Budget | null => + Option.getOrNull(Option.map(Arr.findLast(events, S.is(RunEvent.VerdictReached)), (event) => event.budget)) + +const runEngineWith = ( + directory: string, + options: Options.PartialStrykerOptions, +): Effect.Effect => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + const queue = yield* Queue.bounded(RunEvent.RunEvent.QUEUE_BOUND) + const ports = Engine.nodePlatformLayer + const runLayer = Layer.merge( + Layer.provide(Engine.RunEnvironment.stage(environmentFor(directory), queue), ports), + ports, + ) + yield* Engine.mutationTestCell + .run({ cliOptions: options, targetMutatePatterns: undefined }) + .pipe(Effect.provide(runLayer), Effect.scoped, Effect.orDie) + const events = yield* Queue.end(queue).pipe( + Effect.andThen(Queue.takeAll(queue)), + Effect.orElseSucceed((): ReadonlyArray => []), + ) + const text = yield* fs.readFileString(path.join(directory, REPORT_FILE)).pipe(Effect.orElseSucceed(() => '')) + return { ...readReport(text), streamCosts: streamCostsOf(events), verdictBudget: verdictBudgetOf(events) } + }).pipe(Effect.orDie, Effect.provide(filePorts)) + +export const runEngine: { + ( + directory: string, + options: Options.PartialStrykerOptions, + ): Effect.Effect + ( + options: Options.PartialStrykerOptions, + ): (directory: string) => Effect.Effect +} = dual(2, runEngineWith) + +const runWorkspaceWith = ( + files: ReadonlyArray, + optionsOf: (directory: string) => Options.PartialStrykerOptions, +): Effect.Effect => + Effect.gen(function*() { + const root = yield* writeWorkspace(files) + return yield* Effect.ensuring(runEngineWith(root, optionsOf(root)), removeWorkspace(root)) + }).pipe(Effect.orDie, Effect.provide(filePorts)) + +export const runWorkspace: { + ( + files: ReadonlyArray, + optionsOf: (directory: string) => Options.PartialStrykerOptions, + ): Effect.Effect + ( + optionsOf: (directory: string) => Options.PartialStrykerOptions, + ): (files: ReadonlyArray) => Effect.Effect +} = dual(2, runWorkspaceWith) diff --git a/packages/stryker-js/tests/__fixtures__/cost-checker/index.mjs b/packages/stryker-js/tests/__fixtures__/cost-checker/index.mjs new file mode 100644 index 000000000..2350dd9a2 --- /dev/null +++ b/packages/stryker-js/tests/__fixtures__/cost-checker/index.mjs @@ -0,0 +1,3 @@ +export const strykerPlugins = [ + { kind: 'Checker', name: 'cost', workerEntry: new URL('./worker.mjs', import.meta.url).href }, +] diff --git a/packages/stryker-js/tests/__fixtures__/cost-checker/worker.mjs b/packages/stryker-js/tests/__fixtures__/cost-checker/worker.mjs new file mode 100644 index 000000000..dcf244d25 --- /dev/null +++ b/packages/stryker-js/tests/__fixtures__/cost-checker/worker.mjs @@ -0,0 +1,49 @@ +import { NodeFileSystem, NodeSocketServer } from '@effect/platform-node' +import * as NodeRuntime from '@effect/platform-node/NodeRuntime' +import { Plugin } from '@systemfsoftware/stryker-js-plugin-interface' +import { Worker } from '@systemfsoftware/stryker-js-plugin-runtime' +import * as Config from 'effect/Config' +import * as Effect from 'effect/Effect' +import * as Layer from 'effect/Layer' + +const REJECTED_FILE = 'src/lib/rejected.ts' +const REJECTION_REASON = 'rejected by the fixture checker' +const IGNORED_FILE = 'src/lib/ignored.ts' +const IGNORE_REASON = 'ignored by the fixture checker' + +const isFile = (mutant, file) => mutant.fileName === file || mutant.fileName.endsWith(`/${file}`) + +const answerOf = (mutant) => { + if (isFile(mutant, REJECTED_FILE)) { + return { status: 'compileError', reason: REJECTION_REASON } + } + if (isFile(mutant, IGNORED_FILE)) { + return { status: 'ignored', reason: IGNORE_REASON } + } + return { status: 'passed' } +} + +const CHECK_DELAY = '25 millis' + +const handlers = Plugin.CheckerRpcs.toLayer({ + group: ({ mutants }) => Effect.succeed([mutants.map((mutant) => mutant.id)]), + check: ({ mutants }) => + Effect.as( + Effect.sleep(CHECK_DELAY), + Object.fromEntries(mutants.map((mutant) => [mutant.id, answerOf(mutant)])), + ), +}) + +const platform = Layer.unwrap( + Effect.gen(function*() { + const socketPath = yield* Config.String('STRYKER_SOCKET') + return Layer.merge(NodeSocketServer.layer({ path: socketPath }), NodeFileSystem.layer) + }), +) + +NodeRuntime.runMain( + Worker.workerServerLayer({ rpcs: Plugin.CheckerRpcs, handlers, schemaServices: Layer.empty }).pipe( + Layer.provide(platform), + Layer.launch, + ), +) diff --git a/packages/stryker-js/tests/check-cost-record.integration.test.ts b/packages/stryker-js/tests/check-cost-record.integration.test.ts new file mode 100644 index 000000000..d9e244780 --- /dev/null +++ b/packages/stryker-js/tests/check-cost-record.integration.test.ts @@ -0,0 +1,131 @@ +import { Gherkin, Given, it, makeFeature, Then } from '@systemfsoftware/effect-gherkin-spec' +import * as Layer from 'effect/Layer' + +import { + checkedOptionsOf, + checkedWorkspaceFiles, + type Observation, + runWorkspace, + uncheckedOptionsOf, + uncheckedWorkspaceFiles, +} from './__fixtures__/check-cost-workspace.fixture.js' + +const Feature = makeFeature({ it }) + +const LANES_OF_ONE_TEST_RUNNER_AND_ONE_CHECKER = 2 + +const idsWithStatus = (statuses: Readonly>, status: string): readonly string[] => + Object.entries(statuses).filter(([, seen]) => seen === status).map(([id]) => id) + +const isMeasured = (actualMs: number | null | undefined): boolean => + typeof actualMs === 'number' && Number.isFinite(actualMs) && actualMs >= 0 + +const budgetPricesOnlyTheVerdictsThatRanATest = (observed: Observation): boolean => { + const testRunning = [ + ...idsWithStatus(observed.statuses, 'Killed'), + ...idsWithStatus(observed.statuses, 'Survived'), + ...idsWithStatus(observed.statuses, 'Timeout'), + ] + const { verdictBudget } = observed + if (verdictBudget === null || testRunning.length === 0) return false + const testWorkSeconds = testRunning.reduce((total, id) => total + (observed.streamCosts[id] ?? 0), 0) / + LANES_OF_ONE_TEST_RUNNER_AND_ONE_CHECKER / 1000 + return Math.abs(verdictBudget.predictedSeconds - testWorkSeconds) <= + Number.EPSILON * 8 * Math.max(1, verdictBudget.predictedSeconds) +} + +const idsOfFile = (observed: Observation, suffix: string): readonly string[] => + Object.entries(observed.idsByFile).flatMap(([file, ids]) => (file.endsWith(suffix) ? ids : [])) + +const recordSummaryOf = (observed: Observation) => { + const compileErrors = idsWithStatus(observed.statuses, 'CompileError') + const rejectedIds = idsOfFile(observed, 'src/lib/rejected.ts') + const noCoverage = idsWithStatus(observed.statuses, 'NoCoverage') + const ignored = idsWithStatus(observed.statuses, 'Ignored') + const finishedWithoutATest = [...compileErrors, ...noCoverage, ...ignored] + const testRunning = [ + ...idsWithStatus(observed.statuses, 'Killed'), + ...idsWithStatus(observed.statuses, 'Survived'), + ] + const coveredCompileErrors = compileErrors.filter((id) => (observed.costs[id]?.coveringTests ?? 0) > 0) + return { + compileErrorCountAboveZero: compileErrors.length > 0, + compileErrorsRecordFiniteCosts: compileErrors.length > 0 && + compileErrors.every((id) => isMeasured(observed.costs[id]?.actualMs)), + rejectedCountAboveZero: rejectedIds.length > 0, + rejectedMutantsAreCompileErrorsChargedTheCheckTime: rejectedIds.length > 0 && + rejectedIds.every((id) => observed.statuses[id] === 'CompileError' && (observed.costs[id]?.actualMs ?? 0) > 0), + noCoverageCountAboveZero: noCoverage.length > 0, + noCoverageRecordsFiniteCosts: noCoverage.length > 0 && + noCoverage.every((id) => isMeasured(observed.costs[id]?.actualMs)), + ignoredCountAboveZero: ignored.length > 0, + ignoredRecordsZeroCosts: ignored.length > 0 && ignored.every((id) => observed.costs[id]?.actualMs === 0), + noTestVerdictsPublishTheirMeasuredCostOnTheStream: finishedWithoutATest.length > 0 && + finishedWithoutATest.every((id) => observed.streamCosts[id] === observed.costs[id]?.actualMs), + testRunningCountAtLeastTwo: testRunning.length >= 2, + testRunningKeepsItsMeasuredTestTime: testRunning.length > 0 && + testRunning.every((id) => + isMeasured(observed.costs[id]?.actualMs) && + observed.costs[id]?.actualMs === observed.streamCosts[id] && + (observed.streamCosts[id] ?? 0) > 0 + ), + coveredCompileErrorsCostLessThanTheirWholeSuitePrediction: coveredCompileErrors.length > 0 && + coveredCompileErrors.every((id) => + (observed.costs[id]?.actualMs ?? Number.POSITIVE_INFINITY) < (observed.costs[id]?.predictedMs ?? 0) + ), + budgetPricesOnlyTheVerdictsThatRanATest: budgetPricesOnlyTheVerdictsThatRanATest(observed), + } +} + +Feature('The measured cost recorded for a verdict the engine decided without a test') + .withLayer(Layer.empty) + .live('the scenarios drive the real engine over the host filesystem, so the report it writes is the real one') + .body(({ scenario }) => { + scenario( + 'A checker rejects and ignores mutants, and the report records the check time it measured for them', + Gherkin.Do.pipe( + Given('a workspace whose covered modules are checked by a rejecting and ignoring checker')( + 'observed', + () => runWorkspace(checkedWorkspaceFiles, checkedOptionsOf), + ), + Then( + 'the compile errors and the uncovered mutants carry a finite measured cost, the ignored mutants carry zero, the test-running mutants keep the measured test time the stream published, and the run budget prices only that test time', + )( + (s, expect) => + expect(recordSummaryOf(s.observed)).toEqual({ + compileErrorCountAboveZero: true, + compileErrorsRecordFiniteCosts: true, + rejectedCountAboveZero: true, + rejectedMutantsAreCompileErrorsChargedTheCheckTime: true, + noCoverageCountAboveZero: true, + noCoverageRecordsFiniteCosts: true, + ignoredCountAboveZero: true, + ignoredRecordsZeroCosts: true, + noTestVerdictsPublishTheirMeasuredCostOnTheStream: true, + testRunningCountAtLeastTwo: true, + testRunningKeepsItsMeasuredTestTime: true, + coveredCompileErrorsCostLessThanTheirWholeSuitePrediction: true, + budgetPricesOnlyTheVerdictsThatRanATest: true, + }), + ), + ), + ) + + scenario( + 'Without a checker an uncovered mutant records a zero cost rather than no cost at all', + Gherkin.Do.pipe( + Given('a workspace with no checker whose tests never reach one module')( + 'observed', + () => runWorkspace(uncheckedWorkspaceFiles, uncheckedOptionsOf), + ), + Then('every uncovered mutant carries an actualMs of zero')((s, expect) => { + const noCoverage = idsWithStatus(s.observed.statuses, 'NoCoverage') + return expect({ + noCoverageCountAboveZero: noCoverage.length > 0, + everyUncoveredMutantIsPricedAtZero: noCoverage.length > 0 && + noCoverage.every((id) => s.observed.costs[id]?.actualMs === 0), + }).toEqual({ noCoverageCountAboveZero: true, everyUncoveredMutantIsPricedAtZero: true }) + }), + ), + ) + }) diff --git a/packages/stryker-js/tests/plan-recorded-costs.integration.test.ts b/packages/stryker-js/tests/plan-recorded-costs.integration.test.ts new file mode 100644 index 000000000..b18a5f5c6 --- /dev/null +++ b/packages/stryker-js/tests/plan-recorded-costs.integration.test.ts @@ -0,0 +1,195 @@ +import { Gherkin, Given, it, makeFeature, Then } from '@systemfsoftware/effect-gherkin-spec' +import { Engine } from '@systemfsoftware/stryker-js' +import { ShardPlan } from '@systemfsoftware/stryker-js-cli-contract' +import * as Effect from 'effect/Effect' +import * as Fiber from 'effect/Fiber' +import * as FileSystem from 'effect/FileSystem' +import * as Option from 'effect/Option' +import * as Path from 'effect/Path' +import * as ChildProcess from 'effect/process/ChildProcess' +import * as ChildProcessSpawner from 'effect/process/ChildProcessSpawner' +import * as S from 'effect/Schema' +import * as Scope from 'effect/Scope' +import * as Stream from 'effect/Stream' + +import { + cliConfigTextOf, + PLAN_FILE, + planWorkspaceFiles, + readReportIn, + removeWorkspace, + type ReportObservation, + writeWorkspace, +} from './__fixtures__/check-cost-workspace.fixture.js' + +const Feature = makeFeature({ it }) + +const STRYKER_BIN = decodeURIComponent(new URL('../dist/main.mjs', import.meta.url).pathname) + +const TARGET_SECONDS = '600' + +const MAX_SHARDS = '8' + +const CONFIG_FILE = 'stryker.config.mjs' + +interface ExecOutcome { + readonly exitCode: number + readonly output: string +} + +const spawnCli = ( + root: string, + args: ReadonlyArray, +): Effect.Effect => + Effect.scoped( + Effect.gen(function*() { + const spawner = yield* ChildProcessSpawner.ChildProcessSpawner + const handle = yield* spawner.spawn( + ChildProcess.make(globalThis.process.execPath, [STRYKER_BIN, ...args], { + cwd: root, + stdin: 'ignore', + stdout: 'pipe', + stderr: 'pipe', + env: { STRYKER_MODE: 'machine', NO_COLOR: '1', GITHUB_ACTIONS: '', ALLOW_LOCAL_MUTATION: '1' }, + extendEnv: true, + }), + ) + const stdout = yield* Effect.forkScoped(handle.stdout.pipe(Stream.decodeText, Stream.mkString)) + const stderr = yield* Effect.forkScoped(handle.stderr.pipe(Stream.decodeText, Stream.mkString)) + const exitCode = yield* handle.exitCode + return { + exitCode: Number(exitCode), + output: `${yield* Fiber.join(stdout)}\n${yield* Fiber.join(stderr)}`, + } + }), + ).pipe(Effect.orDie) + +const decodePlan = S.decodeUnknownOption(S.fromJsonString(ShardPlan)) + +interface ObservedPlan { + readonly runExitCode: number + readonly planExitCode: number + readonly plan: ShardPlan | null + readonly report: ReportObservation + readonly output: string +} + +const planAfterFullRun = ( + directory: string, +): Effect.Effect< + ObservedPlan, + never, + FileSystem.FileSystem | Path.Path | Scope.Scope | ChildProcessSpawner.ChildProcessSpawner +> => + Effect.gen(function*() { + const fs = yield* FileSystem.FileSystem + const path = yield* Path.Path + yield* fs.writeFileString(path.join(directory, CONFIG_FILE), cliConfigTextOf(directory)) + const run = yield* spawnCli(directory, ['run']) + const plan = yield* spawnCli(directory, [ + 'plan', + '--target-seconds', + TARGET_SECONDS, + '--max-shards', + MAX_SHARDS, + '--out', + PLAN_FILE, + '--full', + ]) + const planText = yield* fs.readFileString(path.join(directory, PLAN_FILE)).pipe(Effect.orElseSucceed(() => '')) + const report = yield* readReportIn(directory) + return { + runExitCode: run.exitCode, + planExitCode: plan.exitCode, + plan: Option.getOrNull(decodePlan(planText)), + report, + output: `${run.output}\n${plan.output}`, + } + }).pipe(Effect.orDie) + +const TEST_RUNNING_STATUSES: Readonly> = { + Killed: true, + Survived: true, + Timeout: true, +} + +const DECIDED_WITHOUT_A_TEST: Readonly> = { + CompileError: true, + NoCoverage: true, + Ignored: true, +} + +const roundMs = (seconds: number): number => Math.round(seconds * 1000) + +const planSummaryOf = (observed: ObservedPlan) => { + const plan = observed.plan + if (plan === null) { + return { planWritten: false, runExitCode: observed.runExitCode, planExitCode: observed.planExitCode } + } + const scheduledIds = plan.shards.flatMap((shard) => shard.projects.flatMap((project) => project.mutants)) + const plannedSecondsMs = plan.shards.reduce((total, shard) => total + shard.predictedSeconds * 1000, 0) + const recordedCostsMs = scheduledIds.reduce( + (total, id) => total + (observed.report.costs[id]?.actualMs ?? 0), + 0, + ) + const costsWithoutMeasuredCheckTimeMs = scheduledIds.reduce((total, id) => { + const cost = observed.report.costs[id] + const status = observed.report.statuses[id] ?? '' + return total + + (DECIDED_WITHOUT_A_TEST[status] === true + ? (cost?.predictedMs ?? cost?.actualMs ?? 0) + : (cost?.actualMs ?? 0)) + }, 0) + return { + planWritten: true, + runExitCode: observed.runExitCode, + planExitCode: observed.planExitCode, + everyScheduledMutantCarriesAMeasuredCost: scheduledIds.length > 0 && + scheduledIds.every((id) => observed.report.costs[id]?.actualMs !== null), + aCheckDecidedMutantIsScheduled: scheduledIds.some( + (id) => DECIDED_WITHOUT_A_TEST[observed.report.statuses[id] ?? ''] === true, + ), + aTestRunningMutantIsScheduled: scheduledIds.some( + (id) => TEST_RUNNING_STATUSES[observed.report.statuses[id] ?? ''] === true, + ), + thePlanPricesTheRecordedCosts: roundMs(plannedSecondsMs) === roundMs(recordedCostsMs), + thePlanPricesNoWholeSuitePrediction: roundMs(plannedSecondsMs) < roundMs(costsWithoutMeasuredCheckTimeMs), + } +} + +const observedPlanAfterFullRun: Effect.Effect< + ObservedPlan, + never, + FileSystem.FileSystem | Path.Path | Scope.Scope | ChildProcessSpawner.ChildProcessSpawner +> = Effect.gen(function*() { + const root = yield* writeWorkspace(planWorkspaceFiles) + return yield* Effect.ensuring(planAfterFullRun(root), removeWorkspace(root)) +}).pipe(Effect.orDie) + +Feature('Shard planning on the measured costs a run records') + .withLayer(Engine.nodePlatformLayer) + .live('the built stryker binary runs and then plans a workspace whose checker rejects mutants') + .body(({ scenario }) => { + scenario( + 'A full plan prices the mutants it schedules at the costs the run measured', + Gherkin.Do.pipe( + Given( + 'a workspace whose covered modules are checked by a rejecting checker and whose test runner passes, run in full and then planned in full', + )('observed', () => observedPlanAfterFullRun), + Then( + 'the plan prices its scheduled mutants at their measured costs rather than at a whole-suite prediction', + )((s, expect) => + expect(planSummaryOf(s.observed)).toEqual({ + planWritten: true, + runExitCode: 0, + planExitCode: 0, + everyScheduledMutantCarriesAMeasuredCost: true, + aCheckDecidedMutantIsScheduled: true, + aTestRunningMutantIsScheduled: true, + thePlanPricesTheRecordedCosts: true, + thePlanPricesNoWholeSuitePrediction: true, + }) + ), + ), + ) + })