Repository navigation
Compare the throw site as well as the class when shrinking - #161
Merged
Merged
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A shrink candidate was accepted on the exception class alone, so a body that can fail in more than one place offered the descent failures it could not tell apart: shrinking could leave the failure that was found and minimise another one, and the reported counterexample then belonged to a bug the run never started from. Nothing in the output said so. `FailureIdentity` adds the place to the criterion. The place is the line of the property's OWN file in the failure's stack — the throwable's own line when the body threw it, otherwise the innermost frame in that file — and not the throw itself: PHPUnit raises every constraint failure from `Constraint::fail()` and Testo every comparison from `Assert`, so `getFile()`/`getLine()` alone would tell two assertions of one body apart from nothing. The file comes from the class part of the property id, which both adapters derive from the test class. The issue asks whether to walk to the first frame outside the engine instead: that does not work. Under Testo the body is reached through the framework's interceptor pipeline, so the frames next to the engine belong to Testo, not to the property. The scan is bounded the other way round — it stops at the runner's own file, because beyond it the stack is the engine calling the body, and an adapter that invokes the body from the test file (PHPUnit's `check()` call) would otherwise report that call as the site of every failure while an adapter that invokes it through reflection reported none. Nothing is guessed when the place is unreadable — an id that names no loaded class, a body written in another file, a failure raised by the engine rather than by the body: the class stays the whole criterion. A criterion that guessed, or one that compared whole traces, would refuse candidates that fail exactly as the original did (a failure reached through recursion arrives on a stack of a different height on every input) and stall the descent on the input it started from — the same silence, at a larger counterexample. The corpus stores no failure identity, and a path replay accepts any falsification, so neither is affected. Fixes #155
The site is a line of the property's own file — the file is fixed for the whole identity, so carrying it in the compared string only made the concat unkillable. Two mutants that no test could reach are gone rather than ignored: the unknown-file guard in the search (an unknown file matches neither the throwable's nor any frame's, so the search already returns nothing) and the autoload flag, now pinned by a test that registers an autoloader and asserts an id never reaches it. A frame with no file of its own — a callback invoked by an internal function — is stepped over, not taken as the end of the search.
The shrinking guide described the loop but never said what "still fails" means, which is now the interesting part. States the criterion, both fallbacks, and the one consequence a user of stateful properties should know: the rule-based facade's #[Rule] methods live in the test file, so a failure that moves from one rule to another is a different failure. "Not the throw itself" was also wrong in the README and the changelog for a body that throws directly — there the throwable's own line IS the place.
rasuvaeff
force-pushed
the
feat/shrink-failure-site
branch
from
September 28, 2026 08:10
2d5ba76 to
1280fa8
Compare
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #155
A shrink candidate was accepted on the exception class alone, so a body that can fail in more than one place — two assertions, or an assertion beside a call that throws the same type — offered the descent failures it could not tell apart. Shrinking could leave the failure that was found and minimise another one, and the reported minimal counterexample then belonged to a bug the run never started from, with nothing in the output saying so.
Internal\FailureIdentityadds the place to the criterion: the exception class and the line of the property's own file in the failure's stack. It is built once per descent and compared against the ORIGINAL failure, never against the last accepted one.The two points the issue left open
getFile()/getLine()alone is nearly a no-op for both adapters. PHPUnit raises every constraint failure fromConstraint::fail()(Constraint.php:124) and Testo every comparison fromassert/Assert.php:55, so the throw site is identical for twoassertSamecalls in one body — the headline case. Verified on live traces throughPhpUnitTrialExecutorandTestoTrialExecutor.Walking to "the first frame outside the engine" does not work either. Under Testo the body is reached through the framework's interceptor pipeline, so the frames next to the engine belong to Testo, not to the property. The place is instead read from the property's own file, which the class part of the property id names — both adapters derive the id from the test class. The scan is bounded the other way round: it stops at the runner's own file, because beyond it the stack is the engine calling the body, and an adapter that invokes the body from the test file (PHPUnit's
check()call) would otherwise report that call as the site of every failure raised outside the body while an adapter that invokes it through reflection reported none — the same property would shrink differently under the two.Nothing is guessed when the place is unreadable — an id that names no loaded class, a body written in another file (a trait, an included fixture), a failure raised by the engine rather than by the body. There the class stays the whole criterion. A criterion that guessed, or one that compared whole traces or frame counts, would refuse candidates that fail exactly as the original did (a failure reached through recursion arrives on a stack of a different height on every input) and stall the descent on the input it started from — the same silence, at a larger counterexample.
Neither the corpus nor a path replay is affected: a corpus entry stores no failure identity, and a replayed path accepts any falsification by design.
Before / after, through the real adapters
Two assertions in one body,
seed: 42,Gen::intBetween(0, 10_000), failing at the first assertion forvalue >= 100:Identical under
property-testing-testoandproperty-testing-phpunit.Behaviour note for stateful properties
With the rule-based façade the
#[Rule]methods live in the test file, so a command sequence whose failure moves from one rule to another is now a different failure and the descent stops before it. No existing test changed, but it is worth knowing.Verification
composer build: green (1680 passed, 1 skipped), Psalm 0, cs 0, Rector clean.FailureIdentityon its own: 39/39 mutants killed, no ignores; the two unkillable ones were removed by construction (a redundant unknown-file guard) and pinned by a test (an autoloader registered to assert an id never reaches it).master's runner and pass here.bc-checkreports the five pre-existing[BC] SKIPPEDentries (enum defaults in promoted readonly properties) and nothing else, which this workflow's gate treats as a pass.Docs:
README.md+README.ru.md,llms.txt,AGENTS.md, and a new "What counts as still failing" section indocs/src/guide/shrinking.md. Changelog entry under## Unreleased, a minor: the compatibility policy §2 already states that what a descent minimises to is not under SemVer.