diff --git a/README.md b/README.md index 454ba4f..e13cb34 100644 --- a/README.md +++ b/README.md @@ -144,6 +144,11 @@ What the adapter does and does not combine with: Testo's lifecycle interceptor, so `#[BeforeTest]`/`#[AfterTest]` execute once per generated input, not once per test (PHPUnit's `setUp` runs once). A hook that throws is that run's failure and is shrunk like any other. +- **PHP diagnostics can opt into falsifying a run** with `failOn:`: + `#[Property(failOn: E_DEPRECATED | E_WARNING)]` converts matching diagnostics + to `ErrorException`s. The default `0` leaves normal handling unchanged; + `failOn` explicitly observes levels Testo masks suite-wide, and `@`-suppressed + diagnostics remain suppressed. - **Assertion and test-double expectations are checked on every property run.** Testo's assertion interceptors run inside the property loop, so a failed expectation belongs to the input that failed it and is shrunk like any other @@ -357,7 +362,8 @@ public function delayStaysWithinCap( | `path` | Replays a recorded shrink descent (`CounterExample::$path`) instead of searching for it; requires `seed` | | `edgeCases` | `EdgeCases::None` turns off the numeric boundary bias — for a property the edges only cost runs | | `auto` | Derives generators from the property's signature for every parameter the provider does not cover; the provider becomes partial overrides. Off by default, and stays off | -| `throws` | The exception class every run must throw — a run that throws it passes, one that returns normally or throws another class fails and shrinks. An adapter alternative to Testo's per-run `#[ExpectException]` | +| `throws` | The exception class every run must throw — a run that throws it passes, one that returns normally or throws another class fails and shrinks. The per-run replacement for `#[ExpectException]`, which is refused on a property | +| `failOn` | PHP diagnostic bitmask that turns matching warnings, notices or deprecations into shrinkable run failures; `0` disables it and `@`-suppressed diagnostics stay suppressed | | `exhaustive` | Walk the whole parameter domain instead of sampling it when every generator is `Enumerable` and the product fits `exhaustiveBudget`; otherwise the phase samples and a warning says why. `runs` is ignored when it walks — see the [core README](https://github.com/rasuvaeff/property-testing-core#exhaustive-mode) | | `exhaustiveBudget` | The largest domain `exhaustive` walks (default 10 000) | | `flakyReplays` | Re-executions of the minimised counterexample (default 2); one that passes marks the counterexample flaky, with a `Flaky:` line in the failure. `0` disables — see [flaky detection](https://github.com/rasuvaeff/property-testing-core#flaky-detection) | diff --git a/README.ru.md b/README.ru.md index 274986b..f719fa1 100644 --- a/README.ru.md +++ b/README.ru.md @@ -144,6 +144,11 @@ dead-code-набор Rector удалил бы приватные. исполняются на каждый сгенерированный вход, а не один раз на тест (в PHPUnit `setUp` — один раз). Хук, бросивший исключение, — падение этого прогона, оно shrink-ается как любое другое. +- **PHP-диагностики можно сделать причиной падения прогона** через `failOn:`: + `#[Property(failOn: E_DEPRECATED | E_WARNING)]` превращает выбранные + диагностики в `ErrorException`. Значение по умолчанию `0` сохраняет обычную + обработку; явный `failOn` видит уровни, которые Testo маскирует для всего + набора, а диагностики под `@` остаются подавленными. - **Ассерты и ожидания test doubles проверяются на каждом прогоне property.** Assertion-интерцепторы Testo выполняются внутри цикла property, поэтому провал ожидания относится к конкретному входу и shrink-ается как любая другая @@ -354,7 +359,8 @@ public function delayStaysWithinCap( | `path` | Воспроизводит записанный спуск shrink (`CounterExample::$path`) вместо поиска; требует `seed` | | `edgeCases` | `EdgeCases::None` выключает граничное смещение числовых генераторов — для property, которой края стоят только прогонов | | `auto` | Достраивает генераторы из сигнатуры property для параметров, не покрытых провайдером; провайдер становится частичными overrides. По умолчанию выключен и дефолтом не станет | -| `throws` | Класс исключения, который обязан бросить каждый прогон: бросивший его проходит, вернувшийся нормально или бросивший другой класс падает и shrink-ается. Альтернатива проверяемому на каждом прогоне `#[ExpectException]` Testo | +| `throws` | Класс исключения, который обязан бросить каждый прогон: бросивший его проходит, вернувшийся нормально или бросивший другой класс падает и shrink-ается. Замена `#[ExpectException]` на уровне прогона; сам `#[ExpectException]` на property отклоняется | +| `failOn` | Битовая маска PHP-диагностик: выбранные warning, notice или deprecation становятся ошибкой прогона и участвуют в shrink; `0` отключает проверку, а `@` сохраняет подавление | | `exhaustive` | Обойти весь домен параметров вместо выборки, когда каждый генератор — `Enumerable`, а произведение умещается в `exhaustiveBudget`; иначе фаза делает выборку, а предупреждение говорит почему. `runs` при обходе игнорируется — см. [README core](https://github.com/rasuvaeff/property-testing-core#exhaustive-mode) | | `exhaustiveBudget` | Наибольший домен, который обходит `exhaustive` (по умолчанию 10 000) | | `flakyReplays` | Повторные исполнения минимизированного контрпримера (по умолчанию 2); прошедший помечает контрпример flaky, со строкой `Flaky:` в сообщении. `0` выключает — см. [детекцию flaky](https://github.com/rasuvaeff/property-testing-core#flaky-detection) | diff --git a/src/Property.php b/src/Property.php index cb62500..df6764c 100644 --- a/src/Property.php +++ b/src/Property.php @@ -116,6 +116,11 @@ * body that calls `Target::maximize()`/`minimize()`: the best-scoring inputs are mutated * one parameter at a time. 0 (the default) performs no search. `PROPERTY_SEARCH_RUNS` * overrides it for the suite. + * @param int $failOn PHP diagnostic levels that should falsify a run. A matching warning, + * notice or deprecation is converted to an {@see \ErrorException}, so it carries the + * generated input and shrinks like any other failure. 0 (the default) preserves PHP's + * normal diagnostic handling. This opt-in overrides Testo's suite-level diagnostic + * mask, while `@`-suppressed diagnostics remain suppressed. */ public function __construct( public int $runs = 100, @@ -141,6 +146,7 @@ public function __construct( public int $exhaustiveBudget = 10_000, public int $flakyReplays = 2, public int $searchRuns = 0, + public int $failOn = 0, ) { // A data holder: every value is validated by the interceptor, which // can name the property. Testo instantiates the attribute long before diff --git a/src/Testo/PropertyInterceptor.php b/src/Testo/PropertyInterceptor.php index 3f2ab58..be8030d 100644 --- a/src/Testo/PropertyInterceptor.php +++ b/src/Testo/PropertyInterceptor.php @@ -200,7 +200,7 @@ public function runTest(TestInfo $info, callable $next): TestResult $listeners[] = new VerboseListener($this->messenger); } - $executor = new TestoTrialExecutor($info, \Closure::fromCallable($next), $property->throws); + $executor = new TestoTrialExecutor($info, \Closure::fromCallable($next), $property->throws, $property->failOn); $result = $this->runner->run($definition, $executor, $listeners, $corpus); if ($executor->everyRunSkipped()) { @@ -231,6 +231,7 @@ private function validate(Property $property, string $name): void $property->exhaustiveBudget < 1 => 'exhaustiveBudget must be greater than or equal to 1', $property->flakyReplays < 0 => 'flakyReplays must be greater than or equal to 0', $property->searchRuns < 0 => 'searchRuns must be greater than or equal to 0', + $property->failOn < 0 => 'failOn must be greater than or equal to 0', // The engine says the same thing, but the adapter draws a seed for // an unseeded property before the engine sees it, and an // environment seed replays a suite, not the run this path came from. diff --git a/src/Testo/TestoTrialExecutor.php b/src/Testo/TestoTrialExecutor.php index 485853c..dfb0a9d 100644 --- a/src/Testo/TestoTrialExecutor.php +++ b/src/Testo/TestoTrialExecutor.php @@ -59,11 +59,13 @@ final class TestoTrialExecutor implements TrialExecutor /** * @param \Closure(TestInfo): TestResult $next * @param ?class-string<\Throwable> $expectedExceptionClass + * @param int $failOn PHP diagnostic levels that turn a run into an ErrorException. */ public function __construct( private readonly TestInfo $info, private readonly \Closure $next, private readonly ?string $expectedExceptionClass = null, + private readonly int $failOn = 0, ) {} #[\Override] @@ -71,6 +73,35 @@ public function execute(array $arguments): TrialOutcome { ++$this->runs; + $reportingAtStart = error_reporting(); + if ($this->failOn !== 0) { + /** @var array{handler: mixed} $handlerState */ + $handlerState = ['handler' => null]; + $handler = function ( + int $severity, + string $message, + string $file, + int $line, + ) use (&$handlerState, $reportingAtStart): bool { + // An explicit failOn is allowed to see levels Testo masks for + // the suite (notably E_DEPRECATED and E_NOTICE). The error + // control operator changes error_reporting() for the handler, + // so a level that was suppressed with @ remains suppressed. + $suppressed = ($reportingAtStart !== error_reporting()) && (($severity & error_reporting()) === 0); + + if (!$suppressed && ($severity & $this->failOn) !== 0) { + throw new \ErrorException($message, 0, $severity, $file, $line); + } + + if (is_callable($handlerState['handler'])) { + return (bool) ($handlerState['handler'])($severity, $message, $file, $line); + } + + return false; + }; + $handlerState['handler'] = set_error_handler($handler); + } + try { $result = ($this->next)($this->info->with(arguments: array_values($arguments))); } catch (AssumptionSkipped) { @@ -91,6 +122,10 @@ public function execute(array $arguments): TrialOutcome // must reach the engine as one — escaping here would abort the // whole property with no counterexample. return $this->verdict($failure); + } finally { + if ($this->failOn !== 0) { + restore_error_handler(); + } } foreach (array_keys($result->attributes) as $key) { diff --git a/tests/Fixture/DiagnosticFixture.php b/tests/Fixture/DiagnosticFixture.php new file mode 100644 index 0000000..acf22ee --- /dev/null +++ b/tests/Fixture/DiagnosticFixture.php @@ -0,0 +1,27 @@ + */ + public static function ints(): array + { + return ['value' => Gen::intBetween(1, 10)]; + } +} diff --git a/tests/Testo/PropertyInterceptorTest.php b/tests/Testo/PropertyInterceptorTest.php index 277e9bf..d05cf78 100644 --- a/tests/Testo/PropertyInterceptorTest.php +++ b/tests/Testo/PropertyInterceptorTest.php @@ -282,6 +282,7 @@ public static function invalidAttributeValuesProvider(): iterable yield 'exhaustiveBudget: 0' => [ZeroExhaustiveBudgetStub::class, 'exhaustiveBudget must be greater than or equal to 1']; yield 'flakyReplays: -1' => [NegativeFlakyReplaysStub::class, 'flakyReplays must be greater than or equal to 0']; yield 'searchRuns: -1' => [NegativeSearchRunsStub::class, 'searchRuns must be greater than or equal to 0']; + yield 'failOn: -1' => [NegativeFailOnStub::class, 'failOn must be greater than or equal to 0']; yield 'path without seed' => [PathWithoutSeedStub::class, 'path replays a recorded descent and requires the seed it was recorded with']; yield 'throws: not a Throwable' => [ThrowsNonThrowableStub::class, 'throws names "stdClass", which is not a Throwable']; yield 'throws: a class a failed assertion is an instance of' => [ThrowsExceptionStub::class, 'throws names "Exception", which a failed assertion is an instance of — a falsified body would pass; name the exception the body throws']; @@ -1048,6 +1049,41 @@ static function (TestInfo $info): TestResult { Assert::true($outcome->isDiscarded()); } + public function failOnTurnsAMatchingDiagnosticIntoATrialFailure(): void + { + $executor = new TestoTrialExecutor( + $this->info(PassingStub::class, 'check'), + static function (TestInfo $info): TestResult { + trigger_error('diagnostic from input', E_USER_WARNING); + + return new TestResult(info: $info, status: Status::Passed); + }, + failOn: E_USER_WARNING, + ); + + $outcome = $executor->execute(['x' => 7]); + + Assert::true($outcome->isFailed()); + Assert::instanceOf($outcome->failure, \ErrorException::class); + Assert::same($outcome->failure?->getMessage(), 'diagnostic from input'); + Assert::same($outcome->failure?->getSeverity(), E_USER_WARNING); + } + + public function failOnLeavesSuppressedDiagnosticsSuppressed(): void + { + $executor = new TestoTrialExecutor( + $this->info(PassingStub::class, 'check'), + static function (TestInfo $info): TestResult { + @trigger_error('suppressed diagnostic', E_USER_WARNING); + + return new TestResult(info: $info, status: Status::Passed); + }, + failOn: E_USER_WARNING, + ); + + Assert::true($executor->execute(['x' => 7])->isPassed()); + } + /** * The executor's half of #61: the interceptor refuses a class the * assertion failure is an instance of, and the executor holds the same diff --git a/tests/Testo/PropertyRunnerE2ETest.php b/tests/Testo/PropertyRunnerE2ETest.php index 2a93676..4a42b37 100644 --- a/tests/Testo/PropertyRunnerE2ETest.php +++ b/tests/Testo/PropertyRunnerE2ETest.php @@ -16,6 +16,7 @@ use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\CorpusRegressionFixture; use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\CoverageFixture; use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\DeadlineFixture; +use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\DiagnosticFixture; use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\DrawPropertyFixture; use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\ExampleFailingFixture; use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\ExhaustedFixture; @@ -136,6 +137,17 @@ public function anExpectExceptionAttributeNeverTurnsAFalsifiedPropertyGreen(): v Assert::true($result->status === Status::Error || $result->status === Status::Failed); } + public function aSelectedDiagnosticFalsifiesAndShrinksItsInput(): void + { + $result = TestRunner::runTest([DiagnosticFixture::class, 'everyInputRaisesADiagnostic']); + + Assert::true($result->status->isFailure()); + Assert::instanceOf($result->failure, PropertyViolationException::class); + Assert::same($result->failure->getCounterExample()->shrunkArguments['value'], 1); + Assert::instanceOf($result->failure->getCounterExample()->failure, \ErrorException::class); + Assert::string($result->failure->getCounterExample()->failure?->getMessage())->contains('diagnostic for'); + } + public function anExpectExceptionAttributeCanBeSatisfiedByEveryRun(): void { $result = TestRunner::runTest([ExpectExceptionFixture::class, 'throwsTheExpectedException']); diff --git a/tests/Testo/PropertyStubs.php b/tests/Testo/PropertyStubs.php index 5b6a1d0..38ab307 100644 --- a/tests/Testo/PropertyStubs.php +++ b/tests/Testo/PropertyStubs.php @@ -996,6 +996,18 @@ public static function provide(): array } } +final class NegativeFailOnStub +{ + #[Property(runs: 1, seed: 1, generators: 'provide', failOn: -1)] + public function check(int $x): void {} + + /** @return array */ + public static function provide(): array + { + return ['x' => Gen::intBetween(1, 10)]; + } +} + final class ExhaustiveStub { #[Property(runs: 3, seed: 1, generators: 'provide', exhaustive: true)] diff --git a/tests/Testo/PropertyTest.php b/tests/Testo/PropertyTest.php index 4193164..230cf15 100644 --- a/tests/Testo/PropertyTest.php +++ b/tests/Testo/PropertyTest.php @@ -181,4 +181,9 @@ public function throwsDefaultsToNullAndKeepsTheClassAsWritten(): void // Not validated here either; the interceptor refuses a non-Throwable. Assert::same((new Property(throws: \stdClass::class))->throws, \stdClass::class); } + public function failOnDefaultsToDisabledAndKeepsTheDiagnosticMask(): void + { + Assert::same((new Property())->failOn, 0); + Assert::same((new Property(failOn: E_DEPRECATED | E_WARNING))->failOn, E_DEPRECATED | E_WARNING); + } }