Skip to content

feat: a mutation report on the lines a pull request changes - #157

Merged
ExtraToast merged 1 commit into
mainfrom
feat/mutation-report
Oct 16, 2026
Merged

ExtraToast merged 1 commit into
mainfrom
feat/mutation-report

Conversation

@ExtraToast

Copy link
Copy Markdown
Contributor

A reusable workflow that shows, on a pull request, which mutants survive on the lines it changes. A surviving mutant is a line a test runs but never checks: the code changed, and no test failed. This is the diff-scoped mutation report from the estate testing standard (ADR-0022).

  • mutation-report.yml. The caller passes go-directory, web-directory, or both.
    • Go runs gremlins unleash --diff origin/<base> (gremlins v0.6.0 by default).
    • TypeScript and Vue run Stryker with --mutate set to the changed source files under src. Specs, generated clients and *.gen.ts are left out.
    • The toolchain comes from the caller's mise.toml. Stryker runs through the caller's own devDependencies, so it uses the project's vitest.
  • One comment per pull request. It is found again by a marker line and edited on the next run, and the same text goes to the job summary. Both tools' steps are continue-on-error, so the report never fails the build. Whether it becomes a gate, and at which threshold, is decided later.
  • scripts/mutation_report.py turns gremlins' JSON (LIVED, NOT COVERED) and Stryker's mutation-testing-report JSON (Survived, NoCoverage) into that comment. Survivors are sorted by file and line, and mutants no test reaches get their own section. A missing report counts as no mutants, and an unreadable one is named in the comment. Its tests follow the testing standard: one behaviour per test, named as a sentence. CI holds the script to 100% line and branch coverage, like path_buckets.py.
  • Release. release-please keeps the self-checkout ref in step with the tag, as for the other reusable workflows.
  • Billing. One job with steps, so a pull request pays for one runner-minute block, not one per tool.

Part of JorisJonkers-dev/workspace#128. #155 closes once identity calls a released version and a deliberately weak test shows up as a survivor.

Checks

  • tests/test_mutation_report.py: 14 tests, 100% line and branch coverage.
  • All 190 tests in tests/ pass.
  • actionlint passes on both changed workflows.

gremlins and Stryker mutate the diff, and one pull request comment lists the
mutants that survived and those no test reaches. It never fails the build.
@ExtraToast ExtraToast added type: feature New user-facing or operator-facing capability. area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. labels Oct 16, 2026
@ExtraToast
ExtraToast merged commit 686eacd into main Oct 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: tooling Reusable workflows, Gradle, templates, Renovate, and API tooling. type: feature New user-facing or operator-facing capability.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant