Skip to content

fix: the mutation report runs gremlins and Stryker the way each actually works - #163

Merged
ExtraToast merged 2 commits into
mainfrom
fix/mutation-report-runs
Oct 16, 2026
Merged

ExtraToast merged 2 commits into
mainfrom
fix/mutation-report-runs

Conversation

@ExtraToast

@ExtraToast ExtraToast commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

The mutation report from #157 had not yet run against a real repository. Running both tools on identity showed four things wrong, and one that made a broken run look like a perfect one.

  • gremlins found nothing. Given ./..., gremlins v0.6.0 reports "No results to report" even when mutants exist on the changed lines. Given the module root ., --diff selects exactly the changed lines' mutants. Checked on identity with a one-line change: two mutants, both on that line.
  • gremlins timed every mutant out. At its default timeout coefficient, all six mutants in identity's Launcher service timed out and efficacy read 0%. At --timeout-coefficient 40 (as grimoire uses) all six are killed and efficacy reads 100%. gremlins first gathers coverage over the whole module (about 10 minutes for identity), which fits the job's 30-minute tier.
  • gremlins' binary path. It installs into GOBIN when mise sets one, so the workflow looks there first.
  • Stryker found no test runner. Under pnpm, Stryker does not discover its plugins ("Cannot find TestRunner plugin "vitest""), so the workflow names --plugins @stryker-mutator/vitest-runner.
  • Stryker has no flag for the JSON report's path. It always writes reports/mutation/mutation.json, which the workflow now copies.
  • A crash read as success. A tool that crashes leaves no report, and a missing report counted as "Every mutant on the changed lines was caught by a test." The workflow now passes each failed tool's step outcome as --failed. The comment then says "Stryker did not finish, so its mutants are not listed: see the job log." and no longer claims every mutant was caught.

Part of JorisJonkers-dev/workspace#128, for #155.

Checks

  • tests/test_mutation_report.py: 15 tests, 100% line and branch coverage. The new test covers a tool that did not finish.
  • Locally on identity: Stryker with the plugin flag writes the JSON report for useLauncher.ts (survivors listed). gremlins with . and --diff selects the changed line's two mutants.
  • actionlint passes.

…lly works

gremlins finds no mutants given ./..., and times every mutant out at its default
coefficient; Stryker cannot find its vitest runner under pnpm, and has no flag
for the JSON report's path. A tool that did not finish is now named in the
comment instead of reading as every mutant caught.
@ExtraToast ExtraToast added type: bug Something is broken or behaving incorrectly. area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. labels Oct 16, 2026
@ExtraToast
ExtraToast merged commit e47e025 into main Oct 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. type: bug Something is broken or behaving incorrectly.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant