Repository navigation
Mutation testing reports mutants as survived that unit tests cover #43
Description
Mutation scores are near zero across packages, with most mutants reported as survived even where a unit test asserts the behaviour they change. The latest mutation run on main (https://github.com/ExaDev/process-graph/actions/runs/38113501922) reports All files scores of 4.03, 2.92 and 31.74 for its first shards.
A local pnpm --dir packages/targets/graph-cli run _test:mutation on #41 gives the same picture (score 4.88, with 41 killed and 849 survived). Among the survivors:
- emptying the loop in
program.tsthat writesopened.warningsto stderr, althoughdata-file.unit.test.tsasserts that exact stderr line; - every mutant of
emit/text.ts(0 killed), althoughterminal-text.unit.test.tsandemit.unit.test.tsassert Mermaid and DOT label text.
That points at the runner not exercising active mutants (the Vitest runner with coverageAnalysis: "perTest" and the per-package unit project) rather than at missing tests. The dry run itself passes.
Activity
Mearman commentedon Oct 11, 2026
Root cause found by experiment on the dag package: with vitest 5.0.3 the mutation score is 4.03 (232 survived, 10 killed, with coverageAnalysis perTest and off alike); with vitest 4.1.11 and @vitest/coverage-v8 4.1.11 the same package scores 93.95 (220 killed, 13 timed out, 9 survived). @stryker-mutator/vitest-runner 10.0.0 declares a vitest >=2 peer but does not exercise active mutants under vitest 5. The vitest 5 bump landed in PR 24, so every mutation result since has been vacuous. Plan: return vitest and its coverage provider to 4.1.11 through the catalog once the single-package restructure lands (one-line change), add a Dependabot ignore for vitest majors with this reason, and re-measure each package's baseline before any break threshold is set.
Mearman commentedon Oct 12, 2026
Cause was the Stryker vitest runner under vitest 5; vitest is back on 4.1.11 and mutation runs on main now report real scores. Break thresholds are derived from the measured baselines per package, and the surviving mutants are tracked per package in the issues for each package.
Mearman commentedon Oct 12, 2026
Correction to the note above: the break thresholds are not set yet. The derivation (
breakThresholdFromBaseline) landed but no package config passes a value. Doing that next, from the artifact reports of the passing run on main.test: gate each package's mutation score on its measured baseline
test: gate each package's mutation score on its measured baseline
test: gate each package's mutation score on its measured baseline
test: gate each package's mutation score on its measured baseline
github-actions commentedon Oct 12, 2026
🎉 This issue has been resolved in version 2.0.2 🎉
The release is available on:
Your semantic-release bot 📦🚀