fix(ci): derive the killed-mutant count instead of grepping a line mutmut never prints - #19
Merged
Merged
Conversation
…e mutmut never prints The mutation summary table shipped in #18 reports "Killed 0" on a perfectly healthy run. `mutmut results` lists ONLY the mutants worth looking at -- survived, no tests, timeout, suspicious -- and never lists killed ones, so `grep -c ': killed'` can only ever return 0. Caught on the very first real run of the job (30308667584), which printed `killed=0 survived=19 no-tests=355` for a run mutmut's own counter scored at 87 killed. The 87 in the docs was read off that counter and is correct; only the table's arithmetic was wrong. Derive it instead: the run's final progress line carries the total, and every non-killed mutant is exactly one line of the results file, so killed = total - listed. Validated against that run's own uploaded artifact -- 461 total, 374 listed, 87 killed, matching mutmut exactly. If the total cannot be read the table prints "?" rather than a fabricated 0, because a wrong number here is worse than an obviously missing one -- "Killed 0" reads as a catastrophic test suite, and that is precisely the misreading this job existed to avoid. Pinned by a test asserting the grep is gone and the derivation is present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #18, caught on that PR's own first run of the job it added.
The mutation summary table reports "Killed 0" on a perfectly healthy run.
mutmut resultslistsonly the mutants worth looking at — survived, no tests, timeout, suspicious — and never lists killed
ones, so
grep -c ': killed'can only ever return 0.Run
30308667584printedkilled=0 survived=19 no-tests=355for a run mutmut's own progress counterscored at 87 killed. The 87 quoted in the docs was read off that counter and is correct; only the
table's arithmetic was wrong.
The fix
Derive it: the run's final progress line carries the total, and every non-killed mutant is exactly one
line of the results file, so
killed = total − listed. Validated against that run's own uploadedartifact — 461 total − 374 listed = 87, matching mutmut exactly.
If the total can't be read, the table prints
?rather than a fabricated0. A wrong number is worsethan an obviously missing one here: "Killed 0" reads as a catastrophically broken test suite, which is
exactly the misreading this job exists to prevent.
Pinned by a test asserting the grep is gone and the derivation is present.
Verification
ruff check+format --checkclean · zizmor real exit 0 · pytest 9098 passed, 797 skipped ·the shell derivation re-run against the real CI artifact.
Advisory-only, as before: no write scope on any job, no new required check.