Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,21 @@ make release-check
- `Random` uses an object-scoped `\Random\Randomizer` (MT19937), NOT the
global `mt_srand`/`mt_rand` — same seed, same sequence, regardless of
intervening random calls.
- **A shrink candidate is accepted on failure identity, not on failure.**
`FailureIdentity` (built once per descent, compared against the ORIGINAL
failure and never against the last accepted one) is the exception class plus
the place in the property's own file: the throwable's own line when the body
threw it, otherwise the innermost frame in that file, and the scan stops at
`PropertyRunner`'s own file so frames beyond the engine — an adapter's
`check()` call sits in the test file too — name no place. Never compare the
throwable's `getFile()`/`getLine()` alone: PHPUnit raises every constraint
failure from `Constraint::fail()` and Testo every comparison from `Assert`,
so that criterion tells two assertions of one body apart from nothing. Never
compare traces or frame counts either: a failure reached through recursion
arrives on a stack of a different height on every input, and a criterion that
noticed would refuse every smaller candidate and report the input it started
from. When the place is unknown on either side, the class is the whole
identity, deliberately — the pre-1.1 behaviour, not a stall.
- In-body draw shrinking is replay-tape-based and intentionally does NOT
re-validate replayed nodes (fast-check `gen()` model). A regrown tape breaks
the finite-tree termination argument, so accepted steps are capped by
Expand Down
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
# Changelog

## Unreleased

- **Changed:** a shrink candidate is accepted only when it fails the same way
the original run did — the same exception class **and** the same place in the
property's own file. A body that can fail in more than one place (two
assertions, or an assertion beside a call that throws the same type) used to
offer the descent failures it could not tell apart, so the reported minimal
counterexample could belong to a bug the run never found, with nothing in the
output saying so. The place is the line of the property's own file in the
failure's stack, which is not where the exception was raised: an assertion
library raises every assertion of a body from one line of its own, while the
line that called in is the assertion the body wrote. Where the place cannot be read —
an id that names no loaded class, a body written in another file (a trait, an
included fixture), a failure the engine raised rather than the body — the
exception class stays the whole criterion. Minimised counterexamples may
therefore differ from 1.0.0 on such a body (compatibility policy §2: what a
descent minimises to is not under SemVer); a corpus recorded before this
release replays unchanged. Worth knowing for stateful properties: the
rule-based façade's `#[Rule]` methods live in the test file, so a command
sequence whose failure moves from one rule to another now stops the descent
rather than being minimised into the other rule's bug.

## 1.0.0 — 2026-09-20

The stability release. The `@api` surface is that of 0.12.0, unchanged: no
Expand Down
18 changes: 13 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,19 @@ The `Changed:` line diffs the original against the shrunk counterexample —
arguments the shrinker left untouched are omitted. `trial(s)` counts every
candidate the shrinker ran (accepted and rejected); `shrink step(s)` counts
only the accepted ones. A candidate is accepted only when it fails the same way
the original run did — the same exception class — so the descent minimises the
bug that was found instead of sliding into a different one (a smaller input that
trips a `TypeError` in the body's setup is not a smaller counterexample of an
assertion failure). Reproduce the exact run by pinning the reported seed in
`PropertyConfig`.
the original run did — the same exception class, raised at the same place in the
property's own file — so the descent minimises the bug that was found instead of
sliding into a different one (a smaller input that trips a `TypeError` in the
body's setup is not a smaller counterexample of an assertion failure, and neither
is one that trips the body's *other* assertion). The place is the line of the
property's own file in the failure's stack, which is not where the exception was
raised: an assertion library raises every assertion of a body from one line of
its own, so that line would tell two assertions apart from nothing, while the
line that called in is the assertion the body wrote. Where the place cannot be read —
a property id that names no loaded class, a body written in another file (a
trait, an included fixture), a failure the engine raised rather than the body —
the exception class is the whole criterion, as it was before. Reproduce the exact
run by pinning the reported seed in `PropertyConfig`.

See [`examples/standalone_runner.php`](examples/standalone_runner.php) for the
full runnable script.
Expand Down
16 changes: 12 additions & 4 deletions README.ru.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,18 @@ Property falsified after 0 successful run(s); seed=42
аргументы, которые shrinker не тронул, опущены. `trial(s)` — все кандидаты
shrinker'а (принятые и отвергнутые); `shrink step(s)` — только принятые.
Кандидат принимается, только если падает так же, как исходный прогон — с тем
же классом исключения: спуск минимизирует найденный баг, а не сползает в
другой (меньший вход, роняющий `TypeError` в подготовке тела, — не меньший
контрпример для assertion-падения). Точное воспроизведение прогона — запиньте
seed из отчёта в `PropertyConfig`.
же классом исключения и в том же месте собственного файла property: спуск
минимизирует найденный баг, а не сползает в другой (меньший вход, роняющий
`TypeError` в подготовке тела, — не меньший контрпример для assertion-падения,
и таким же не является вход, роняющий *другой* assert того же тела). Место —
это строка собственного файла property в стеке падения, а не то место, где
исключение подняли: assert-библиотека роняет любой assert тела из одной своей
строки, так что она не отличила бы два assert'а ни от чего, тогда как строка,
из которой вызвали, — это и есть написанный в теле assert. Когда место прочитать нельзя — id property не называет
загруженный класс, тело написано в другом файле (трейт, подключённая фикстура),
падение подняла не body, а движок — критерием остаётся один класс исключения,
как и раньше. Точное воспроизведение прогона — запиньте seed из отчёта в
`PropertyConfig`.

Полный исполняемый скрипт —
[`examples/standalone_runner.php`](examples/standalone_runner.php).
Expand Down
27 changes: 27 additions & 0 deletions docs/src/guide/shrinking.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,33 @@ step(s)` counts only the accepted descents. You can cap the descent with
get the original counterexample unchanged — see
[Bounding shrink work](/guide/controlling-runs/bounding-shrink).

## What counts as still failing

"Still fails" is stricter than *fails*. A candidate is accepted only when it
fails **the same way** the original run did: the same exception class, raised at
the same place in the property's own file. A body can fail in more than one place
— two assertions, or an assertion beside a call that throws the same type — and
without that the descent would leave the failure it was given and minimise
another one, reporting a minimal counterexample for a bug the run never found.

The place is the line of the property's own file in the failure's stack, which is
not the same as where the exception was raised: an assertion library raises every
assertion of a body from one line of its own (PHPUnit from `Constraint::fail()`,
Testo from `Assert`), so that line tells two assertions apart from nothing, while
the line that called in is exactly the assertion the body wrote.

Where the place cannot be read, the exception class is the whole criterion — the
engine never guesses one, because a criterion that refused candidates failing
exactly as the original did would stall the descent on the input it started from.
That is the case when the property id names no loaded class (an adapter-supplied
id string), when the body is written in another file (a trait, an included
fixture), and when the failure was raised by the engine rather than by the body.

A stateful property is worth a note here: with the rule-based façade the
`#[Rule]` methods live in the test file, so a command sequence whose failure
moves from one rule to another is now a different failure, and the descent stops
before it.

## The termination invariant

A shrink tree is only useful if walking it is guaranteed to stop. Two rules,
Expand Down
10 changes: 7 additions & 3 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,13 @@ Runtime deps: PHP `8.3 - 8.5`, `ext-mbstring`, `ext-random`, `ext-tokenizer`.
dependency is never told to narrow its generators.
- Shrinking is integrated: `generate(Random)` returns a `Shrinkable` (value +
lazy tree of smaller candidates). There is NO `shrink(mixed)` method. A shrink
candidate is accepted only when it fails with the SAME exception class as the
original run (a different failure is not a smaller counterexample). Greedy
per-parameter descent; best-effort minimal.
candidate is accepted only when it fails the SAME way as the original run:
same exception class, raised at the same place in the property's own file (the
line of that file in the stack — an assertion library throws every assertion
from one line of its own, so two assertions in one body are two failures).
Unreadable place -> class alone, as before: an id that names no loaded class,
a body in another file (trait, fixture), a failure raised by the engine.
Greedy per-parameter descent; best-effort minimal.
- Corpus format is byte-compatible with `rasuvaeff/property-testing` 2.8
(`FORMAT_VERSION` 1): one JSON file per property, max 8 values + 2 seed
entries. Seed entries are fenced off by `FilesystemCorpus::SEQUENCE_EPOCH`.
Expand Down
153 changes: 153 additions & 0 deletions src/Internal/FailureIdentity.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
<?php

declare(strict_types=1);

namespace Rasuvaeff\PropertyTesting\Internal;

/**
* What makes two failures of one property the same failure: the exception
* class, and the place in the property's own file that raised it.
*
* The class alone is not enough. A body that can fail in more than one place —
* two assertions, or an assertion next to a call that throws the same type —
* offers the shrink descent several failures that look identical, so the
* descent may leave the failure it was given and minimise a different one. The
* reported counterexample then belongs to a bug the run never found, and
* nothing in the output says so.
*
* The place is read from the property's own file rather than from the
* throwable's `getFile()`/`getLine()` alone, because an assertion library
* throws from one line whatever the assertion was: PHPUnit raises every
* constraint failure from `Constraint::fail()`, Testo every comparison from
* `Assert`. Comparing those would separate an assertion failure from a
* `TypeError` — which the class already does — and collapse two assertions
* into one. The frame in the property's file is the assertion the body wrote.
*
* Walking to "the first frame outside the engine" instead does not work: under
* Testo the body is reached through the framework's interceptor pipeline, so
* the frames next to the engine belong to Testo, not to the property.
*
* Both sides fall back to the class alone whenever the place is unknown — an
* id that names no loaded class (an adapter-supplied string, a closure-derived
* id), a body defined in another file (a trait, an included fixture), a
* failure raised by the engine rather than by the body. A criterion that
* guessed instead would reject candidates that fail the way the original did,
* and a descent that rejects everything stalls on the input it started from —
* the same silence, at a larger counterexample.
*
* @internal
*/
final readonly class FailureIdentity
{
private function __construct(
private ?string $class,
private ?string $file,
private string $boundary,
private ?int $site,
) {}

/**
* @param string $propertyId The property's id; its class part, when it names a loaded
* class, locates the file the property is written in.
* @param ?\Throwable $failure What the original run failed with. Null — or a candidate
* that fails without a throwable — leaves nothing to compare, and any failure
* counts as the same one.
* @param string $boundary The engine's own file. Frames from it outwards are the engine
* calling the body, not the body: without that bound, an adapter that invokes the
* body from the test file itself (PHPUnit's `check()` call) would report that call
* as the site of every failure raised outside the body, while an adapter that
* invokes it through reflection (Testo) would report none — the same property would
* shrink differently under the two.
*/
public static function of(string $propertyId, ?\Throwable $failure, string $boundary): self
{
$file = self::propertyFile($propertyId);

return new self(
class: $failure instanceof \Throwable ? $failure::class : null,
file: $file,
boundary: $boundary,
site: $failure instanceof \Throwable ? self::site($failure, $file, $boundary) : null,
);
}

/**
* Whether $candidate is the failure this identity was built from.
*/
public function matches(?\Throwable $candidate): bool
{
if ($this->class === null || !$candidate instanceof \Throwable) {
return true;
}

if ($candidate::class !== $this->class) {
return false;
}

if ($this->site === null) {
return true;
}

$site = self::site($candidate, $this->file, $this->boundary);

return $site === null || $site === $this->site;
}

/**
* Which line of $file raised the failure: the throwable's own line when the
* body threw it, otherwise the line of the innermost frame that is still in
* the body — an assertion helper is entered from the line the body wrote.
*
* A line is enough to tell two places apart because the file is fixed for
* the whole identity: the frame scan only ever matches $file, and a
* throwable from anywhere else has no place in the property at all.
*/
private static function site(\Throwable $failure, ?string $file, string $boundary): ?int
{
// An unknown file needs no guard of its own: it is neither the file the
// throwable names nor any frame's, so the search below finds nothing.
if ($failure->getFile() === $file) {
return $failure->getLine();
}

foreach ($failure->getTrace() as $frame) {
if (!isset($frame['file'], $frame['line'])) {
// An internal call — a property method reached through
// `ReflectionMethod::invoke()` — has no place of its own.
continue;
}

if ($frame['file'] === $boundary) {
return null;
}

if ($frame['file'] === $file) {
return $frame['line'];
}
}

return null;
}

/**
* The file of the class an id names, or null when the id does not name a
* loaded one. Loading is deliberately not attempted: an id is an arbitrary
* string an adapter chose, and autoloading it would put every failing
* property's id through the autoloader chain — a class map lookup at best,
* an autoloader with side effects at worst. A property that runs has its
* class loaded already; anything else is not a class.
*/
private static function propertyFile(string $propertyId): ?string
{
$separator = strpos($propertyId, '::');
$class = $separator === false ? $propertyId : substr($propertyId, 0, $separator);

if (!class_exists($class, autoload: false)) {
return null;
}

$file = (new \ReflectionClass($class))->getFileName();

return $file === false ? null : $file;
}
}
Loading
Loading