Skip to content

fix: the mutation report runs Stryker for real and stays short - #180

Merged
ExtraToast merged 1 commit into
mainfrom
fix/mutation-report
Oct 19, 2026
Merged

ExtraToast merged 1 commit into
mainfrom
fix/mutation-report

Conversation

@ExtraToast

Copy link
Copy Markdown
Contributor

Summary

Closes #167. The mutation report had two problems: it printed every mutant, and Stryker never really ran.

Why Stryker read as "did not finish". On identity it ran 60 of 427 tests, and almost every mutant "survived". There were two causes:

  • The sandbox loses imports from outside the web directory. identity's src/app/contract.ts imports ../../../openapi/v1/openapi.yaml?raw. Stryker copies only web/, so every spec that loads the app failed to import, and the Vitest runner dropped those files without a word.
  • Under Vitest 5 its Vitest runner switches no mutant on. In place, the dry run runs 379 tests and inject() delivers the active mutant to the setup file. Mutants still never activate: 237 of 241 "survive" on format.ts, a file its tests cover fully.

The fix.

  • Changed lines only. Stryker mutates only the lines the pull request adds or changes. A new scripts/changed_ranges.py turns git diff -U0 into file:start-end ranges, as gremlins --diff already does for Go.
  • One run per changed file, in place. Each run uses the command runner with pnpm exec vitest related --run <file>. The command runner switches a mutant on through the environment, and in place the outside imports resolve.
  • Proven locally. On identity's format.ts, the command runner killed 225 of 241 mutants over the whole file. Limited to the 22 mutants on chore(main): release 0.15.1 #96's changed lines, all were caught, in 1.5 minutes.
  • JSON only. The clear-text reporter is gone, so the job log no longer prints every mutant.

A shorter comment.

  • Survivors. Still listed, up to 30, then "…and N more".
  • Unreached mutants. Counted per file, up to 15 files, instead of one row each. chore(main): release 0.15.1 #96's comment was 30 KB, mostly regenerated client code.
  • Generated code. Left out everywhere: a file whose first lines say Code generated, auto-generated or DO NOT EDIT, read against --go-root and --web-root.
  • Several runs. --stryker takes one report per run.
  • Coverage-off runs. stryker_worked trusts a run that killed or timed out mutants even without coverage, as the command runner reports. A run that switched nothing on still reads as not finished.

A caller no longer needs @stryker-mutator/vitest-runner, only @stryker-mutator/core and Vitest.

Testing

  • changed_ranges.py and mutation_report.py are both at 100% line and branch coverage, with a new CI step for the former.
  • The new tests cover:
    • ranges per hunk, deletions, deleted files;
    • per-file counts and their order and cap, the survivor cap;
    • generated files left out, and an unreadable file kept;
    • several reports merged;
    • trusting a coverage-off run, and not trusting a run that switched nothing on;
    • the workflow's command runner, in-place mode, changed-line ranges and no clear-text reporter.
  • The whole suite (226 tests) and actionlint pass.
  • The workflow's Stryker loop ran end to end locally on identity: ranges, jq-written config, Stryker, report.

Stryker's Vitest runner switches no mutant on under Vitest 5, and its
sandbox loses imports from outside the web directory, so a project
whose code imports its contract ran a fraction of its tests and every
mutant read as survived. Stryker now mutates only the changed lines,
one changed file at a time, in place, with the command runner running
vitest related on that file. The comment lists survivors up to 30,
counts unreached mutants per file instead of listing each, and leaves
out generated code.

Closes #167
@ExtraToast ExtraToast added type: bug Something is broken or behaving incorrectly. area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. component: ci Continuous integration workflow or check behavior. labels Oct 19, 2026
@ExtraToast ExtraToast self-assigned this Oct 19, 2026
@ExtraToast
ExtraToast merged commit 7fef8b6 into main Oct 19, 2026
6 checks passed
@ExtraToast
ExtraToast deleted the fix/mutation-report branch October 19, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. component: ci Continuous integration workflow or check behavior. type: bug Something is broken or behaving incorrectly.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Web mutation testing waits on a Stryker vitest runner that supports Vitest 5

1 participant