Skip to content

Compare the throw site as well as the class when shrinking - #161

Merged
rasuvaeff merged 3 commits into
masterfrom
feat/shrink-failure-site
Sep 28, 2026
Merged

rasuvaeff merged 3 commits into
masterfrom
feat/shrink-failure-site

Conversation

@rasuvaeff

Copy link
Copy Markdown
Owner

Fixes #155

A shrink candidate was accepted on the exception class alone, so a body that can fail in more than one place — two assertions, or an assertion beside a call that throws the same type — offered the descent failures it could not tell apart. Shrinking could leave the failure that was found and minimise another one, and the reported minimal counterexample then belonged to a bug the run never started from, with nothing in the output saying so.

Internal\FailureIdentity adds the place to the criterion: the exception class and the line of the property's own file in the failure's stack. It is built once per descent and compared against the ORIGINAL failure, never against the last accepted one.

The two points the issue left open

getFile()/getLine() alone is nearly a no-op for both adapters. PHPUnit raises every constraint failure from Constraint::fail() (Constraint.php:124) and Testo every comparison from assert/Assert.php:55, so the throw site is identical for two assertSame calls in one body — the headline case. Verified on live traces through PhpUnitTrialExecutor and TestoTrialExecutor.

Walking to "the first frame outside the engine" does not work either. Under Testo the body is reached through the framework's interceptor pipeline, so the frames next to the engine belong to Testo, not to the property. The place is instead read from the property's own file, which the class part of the property id names — both adapters derive the id from the test class. The scan is bounded the other way round: it stops at the runner's own file, because beyond it the stack is the engine calling the body, and an adapter that invokes the body from the test file (PHPUnit's check() call) would otherwise report that call as the site of every failure raised outside the body while an adapter that invokes it through reflection reported none — the same property would shrink differently under the two.

Nothing is guessed when the place is unreadable — an id that names no loaded class, a body written in another file (a trait, an included fixture), a failure raised by the engine rather than by the body. There the class stays the whole criterion. A criterion that guessed, or one that compared whole traces or frame counts, would refuse candidates that fail exactly as the original did (a failure reached through recursion arrives on a stack of a different height on every input) and stall the descent on the input it started from — the same silence, at a larger counterexample.

Neither the corpus nor a path replay is affected: a corpus entry stores no failure identity, and a replayed path accepts any falsification by design.

Before / after, through the real adapters

Two assertions in one body, seed: 42, Gen::intBetween(0, 10_000), failing at the first assertion for value >= 100:

before: Shrunk: value=0    Failure: `0` is the same as `-2`     <- the other assertion
after:  Shrunk: value=100  Failure: `100` is the same as `-1`   <- the one that was found

Identical under property-testing-testo and property-testing-phpunit.

Behaviour note for stateful properties

With the rule-based façade the #[Rule] methods live in the test file, so a command sequence whose failure moves from one rule to another is now a different failure and the descent stops before it. No existing test changed, but it is worth knowing.

Verification

  • composer build: green (1680 passed, 1 skipped), Psalm 0, cs 0, Rector clean.
  • Mutation: gate 97 passes — 3880 killed, 115 escaped (all pre-existing, elsewhere). FailureIdentity on its own: 39/39 mutants killed, no ignores; the two unkillable ones were removed by construction (a redundant unknown-file guard) and pinned by a test (an autoloader registered to assert an id never reaches it).
  • Both drift tests fail on master's runner and pass here.
  • Adapter suites against this core: testo 224, phpunit 139, names 72 — all green.
  • bc-check reports the five pre-existing [BC] SKIPPED entries (enum defaults in promoted readonly properties) and nothing else, which this workflow's gate treats as a pass.

Docs: README.md + README.ru.md, llms.txt, AGENTS.md, and a new "What counts as still failing" section in docs/src/guide/shrinking.md. Changelog entry under ## Unreleased, a minor: the compatibility policy §2 already states that what a descent minimises to is not under SemVer.

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 83be7832-2a26-448d-8f4b-33cdba594690


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

A shrink candidate was accepted on the exception class alone, so a body that
can fail in more than one place offered the descent failures it could not tell
apart: shrinking could leave the failure that was found and minimise another
one, and the reported counterexample then belonged to a bug the run never
started from. Nothing in the output said so.

`FailureIdentity` adds the place to the criterion. The place is the line of the
property's OWN file in the failure's stack — the throwable's own line when the
body threw it, otherwise the innermost frame in that file — and not the throw
itself: PHPUnit raises every constraint failure from `Constraint::fail()` and
Testo every comparison from `Assert`, so `getFile()`/`getLine()` alone would
tell two assertions of one body apart from nothing. The file comes from the
class part of the property id, which both adapters derive from the test class.

The issue asks whether to walk to the first frame outside the engine instead:
that does not work. Under Testo the body is reached through the framework's
interceptor pipeline, so the frames next to the engine belong to Testo, not to
the property. The scan is bounded the other way round — it stops at the
runner's own file, because beyond it the stack is the engine calling the body,
and an adapter that invokes the body from the test file (PHPUnit's `check()`
call) would otherwise report that call as the site of every failure while an
adapter that invokes it through reflection reported none.

Nothing is guessed when the place is unreadable — an id that names no loaded
class, a body written in another file, a failure raised by the engine rather
than by the body: the class stays the whole criterion. A criterion that
guessed, or one that compared whole traces, would refuse candidates that fail
exactly as the original did (a failure reached through recursion arrives on a
stack of a different height on every input) and stall the descent on the input
it started from — the same silence, at a larger counterexample.

The corpus stores no failure identity, and a path replay accepts any
falsification, so neither is affected.

Fixes #155
The site is a line of the property's own file — the file is fixed for the
whole identity, so carrying it in the compared string only made the concat
unkillable. Two mutants that no test could reach are gone rather than
ignored: the unknown-file guard in the search (an unknown file matches
neither the throwable's nor any frame's, so the search already returns
nothing) and the autoload flag, now pinned by a test that registers an
autoloader and asserts an id never reaches it. A frame with no file of its
own — a callback invoked by an internal function — is stepped over, not
taken as the end of the search.
The shrinking guide described the loop but never said what "still fails"
means, which is now the interesting part. States the criterion, both
fallbacks, and the one consequence a user of stateful properties should know:
the rule-based facade's #[Rule] methods live in the test file, so a failure
that moves from one rule to another is a different failure. "Not the throw
itself" was also wrong in the README and the changelog for a body that
throws directly — there the throwable's own line IS the place.
@rasuvaeff
rasuvaeff force-pushed the feat/shrink-failure-site branch from 2d5ba76 to 1280fa8 Compare September 28, 2026 08:10
@rasuvaeff
rasuvaeff merged commit bedf2f1 into master Sep 28, 2026
18 checks passed
@rasuvaeff
rasuvaeff deleted the feat/shrink-failure-site branch September 28, 2026 08:20
@rasuvaeff rasuvaeff mentioned this pull request Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shrinking: compare the throw site, not only the exception class

1 participant