Fail property runs on selected PHP diagnostics

README.md

@@ -144,6 +144,11 @@ What the adapter does and does not combine with:
   Testo's lifecycle interceptor, so `#[BeforeTest]`/`#[AfterTest]` execute
   once per generated input, not once per test (PHPUnit's `setUp` runs once).
   A hook that throws is that run's failure and is shrunk like any other.
+- **PHP diagnostics can opt into falsifying a run** with `failOn:`:
+  `#[Property(failOn: E_DEPRECATED | E_WARNING)]` converts matching diagnostics
+  to `ErrorException`s. The default `0` leaves normal handling unchanged;
+  `failOn` explicitly observes levels Testo masks suite-wide, and `@`-suppressed
+  diagnostics remain suppressed.
 - **Assertion and test-double expectations are checked on every property run.**
   Testo's assertion interceptors run inside the property loop, so a failed
   expectation belongs to the input that failed it and is shrunk like any other
@@ -357,7 +362,8 @@ public function delayStaysWithinCap(
 | `path` | Replays a recorded shrink descent (`CounterExample::$path`) instead of searching for it; requires `seed` |
 | `edgeCases` | `EdgeCases::None` turns off the numeric boundary bias — for a property the edges only cost runs |
 | `auto` | Derives generators from the property's signature for every parameter the provider does not cover; the provider becomes partial overrides. Off by default, and stays off |
-| `throws` | The exception class every run must throw — a run that throws it passes, one that returns normally or throws another class fails and shrinks. An adapter alternative to Testo's per-run `#[ExpectException]` |
+| `throws` | The exception class every run must throw — a run that throws it passes, one that returns normally or throws another class fails and shrinks. The per-run replacement for `#[ExpectException]`, which is refused on a property |
+| `failOn` | PHP diagnostic bitmask that turns matching warnings, notices or deprecations into shrinkable run failures; `0` disables it and `@`-suppressed diagnostics stay suppressed |
 | `exhaustive` | Walk the whole parameter domain instead of sampling it when every generator is `Enumerable` and the product fits `exhaustiveBudget`; otherwise the phase samples and a warning says why. `runs` is ignored when it walks — see the [core README](https://github.com/rasuvaeff/property-testing-core#exhaustive-mode) |
 | `exhaustiveBudget` | The largest domain `exhaustive` walks (default 10 000) |
 | `flakyReplays` | Re-executions of the minimised counterexample (default 2); one that passes marks the counterexample flaky, with a `Flaky:` line in the failure. `0` disables — see [flaky detection](https://github.com/rasuvaeff/property-testing-core#flaky-detection) |

README.ru.md

@@ -144,6 +144,11 @@ dead-code-набор Rector удалил бы приватные.
   исполняются на каждый сгенерированный вход, а не один раз на тест (в
   PHPUnit `setUp` — один раз). Хук, бросивший исключение, — падение этого
   прогона, оно shrink-ается как любое другое.
+- **PHP-диагностики можно сделать причиной падения прогона** через `failOn:`:
+  `#[Property(failOn: E_DEPRECATED | E_WARNING)]` превращает выбранные
+  диагностики в `ErrorException`. Значение по умолчанию `0` сохраняет обычную
+  обработку; явный `failOn` видит уровни, которые Testo маскирует для всего
+  набора, а диагностики под `@` остаются подавленными.
 - **Ассерты и ожидания test doubles проверяются на каждом прогоне property.**
   Assertion-интерцепторы Testo выполняются внутри цикла property, поэтому
   провал ожидания относится к конкретному входу и shrink-ается как любая другая
@@ -354,7 +359,8 @@ public function delayStaysWithinCap(
 | `path` | Воспроизводит записанный спуск shrink (`CounterExample::$path`) вместо поиска; требует `seed` |
 | `edgeCases` | `EdgeCases::None` выключает граничное смещение числовых генераторов — для property, которой края стоят только прогонов |
 | `auto` | Достраивает генераторы из сигнатуры property для параметров, не покрытых провайдером; провайдер становится частичными overrides. По умолчанию выключен и дефолтом не станет |
-| `throws` | Класс исключения, который обязан бросить каждый прогон: бросивший его проходит, вернувшийся нормально или бросивший другой класс падает и shrink-ается. Альтернатива проверяемому на каждом прогоне `#[ExpectException]` Testo |
+| `throws` | Класс исключения, который обязан бросить каждый прогон: бросивший его проходит, вернувшийся нормально или бросивший другой класс падает и shrink-ается. Замена `#[ExpectException]` на уровне прогона; сам `#[ExpectException]` на property отклоняется |
+| `failOn` | Битовая маска PHP-диагностик: выбранные warning, notice или deprecation становятся ошибкой прогона и участвуют в shrink; `0` отключает проверку, а `@` сохраняет подавление |
 | `exhaustive` | Обойти весь домен параметров вместо выборки, когда каждый генератор — `Enumerable`, а произведение умещается в `exhaustiveBudget`; иначе фаза делает выборку, а предупреждение говорит почему. `runs` при обходе игнорируется — см. [README core](https://github.com/rasuvaeff/property-testing-core#exhaustive-mode) |
 | `exhaustiveBudget` | Наибольший домен, который обходит `exhaustive` (по умолчанию 10 000) |
 | `flakyReplays` | Повторные исполнения минимизированного контрпримера (по умолчанию 2); прошедший помечает контрпример flaky, со строкой `Flaky:` в сообщении. `0` выключает — см. [детекцию flaky](https://github.com/rasuvaeff/property-testing-core#flaky-detection) |

src/Property.php

@@ -116,6 +116,11 @@
      *        body that calls `Target::maximize()`/`minimize()`: the best-scoring inputs are mutated
      *        one parameter at a time. 0 (the default) performs no search. `PROPERTY_SEARCH_RUNS`
      *        overrides it for the suite.
+     * @param int $failOn PHP diagnostic levels that should falsify a run. A matching warning,
+     *        notice or deprecation is converted to an {@see \ErrorException}, so it carries the
+     *        generated input and shrinks like any other failure. 0 (the default) preserves PHP's
+     *        normal diagnostic handling. This opt-in overrides Testo's suite-level diagnostic
+     *        mask, while `@`-suppressed diagnostics remain suppressed.
      */
     public function __construct(
         public int $runs = 100,
@@ -141,6 +146,7 @@ public function __construct(
         public int $exhaustiveBudget = 10_000,
         public int $flakyReplays = 2,
         public int $searchRuns = 0,
+        public int $failOn = 0,
     ) {
         // A data holder: every value is validated by the interceptor, which
         // can name the property. Testo instantiates the attribute long before

src/Testo/PropertyInterceptor.php

@@ -200,7 +200,7 @@ public function runTest(TestInfo $info, callable $next): TestResult
             $listeners[] = new VerboseListener($this->messenger);
         }
 
-        $executor = new TestoTrialExecutor($info, \Closure::fromCallable($next), $property->throws);
+        $executor = new TestoTrialExecutor($info, \Closure::fromCallable($next), $property->throws, $property->failOn);
         $result = $this->runner->run($definition, $executor, $listeners, $corpus);
 
         if ($executor->everyRunSkipped()) {
@@ -231,6 +231,7 @@ private function validate(Property $property, string $name): void
             $property->exhaustiveBudget < 1 => 'exhaustiveBudget must be greater than or equal to 1',
             $property->flakyReplays < 0 => 'flakyReplays must be greater than or equal to 0',
             $property->searchRuns < 0 => 'searchRuns must be greater than or equal to 0',
+            $property->failOn < 0 => 'failOn must be greater than or equal to 0',
             // The engine says the same thing, but the adapter draws a seed for
             // an unseeded property before the engine sees it, and an
             // environment seed replays a suite, not the run this path came from.

src/Testo/TestoTrialExecutor.php

@@ -59,18 +59,49 @@ final class TestoTrialExecutor implements TrialExecutor
     /**
      * @param \Closure(TestInfo): TestResult $next
      * @param ?class-string<\Throwable> $expectedExceptionClass
+     * @param int $failOn PHP diagnostic levels that turn a run into an ErrorException.
      */
     public function __construct(
         private readonly TestInfo $info,
         private readonly \Closure $next,
         private readonly ?string $expectedExceptionClass = null,
+        private readonly int $failOn = 0,
     ) {}
 
     #[\Override]
     public function execute(array $arguments): TrialOutcome
     {
         ++$this->runs;
 
+        $reportingAtStart = error_reporting();
+        if ($this->failOn !== 0) {
+            /** @var array{handler: mixed} $handlerState */
+            $handlerState = ['handler' => null];
+            $handler = function (
+                int $severity,
+                string $message,
+                string $file,
+                int $line,
+            ) use (&$handlerState, $reportingAtStart): bool {
+                // An explicit failOn is allowed to see levels Testo masks for
+                // the suite (notably E_DEPRECATED and E_NOTICE). The error
+                // control operator changes error_reporting() for the handler,
+                // so a level that was suppressed with @ remains suppressed.
+                $suppressed = ($reportingAtStart !== error_reporting()) && (($severity & error_reporting()) === 0);
+
+                if (!$suppressed && ($severity & $this->failOn) !== 0) {
+                    throw new \ErrorException($message, 0, $severity, $file, $line);
+                }
+
+                if (is_callable($handlerState['handler'])) {
+                    return (bool) ($handlerState['handler'])($severity, $message, $file, $line);
+                }
+
+                return false;
+            };
+            $handlerState['handler'] = set_error_handler($handler);
+        }
+
         try {
             $result = ($this->next)($this->info->with(arguments: array_values($arguments)));
         } catch (AssumptionSkipped) {
@@ -91,6 +122,10 @@ public function execute(array $arguments): TrialOutcome
             // must reach the engine as one — escaping here would abort the
             // whole property with no counterexample.
             return $this->verdict($failure);
+        } finally {
+            if ($this->failOn !== 0) {
+                restore_error_handler();
+            }
         }
 
         foreach (array_keys($result->attributes) as $key) {

tests/Fixture/DiagnosticFixture.php

@@ -0,0 +1,27 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Rasuvaeff\PropertyTesting\Testo\Tests\Fixture;
+
+use Rasuvaeff\PropertyTesting\ArbitraryInterface;
+use Rasuvaeff\PropertyTesting\Gen;
+use Rasuvaeff\PropertyTesting\Property;
+use Testo\Test;
+
+/** A diagnostic from any input fails and shrinks to the generator's minimum. */
+final class DiagnosticFixture
+{
+    #[Test]
+    #[Property(runs: 5, seed: 1, generators: 'ints', failOn: E_USER_WARNING)]
+    public function everyInputRaisesADiagnostic(int $value): void
+    {
+        trigger_error('diagnostic for ' . $value, E_USER_WARNING);
+    }
+
+    /** @return array<string, ArbitraryInterface> */
+    public static function ints(): array
+    {
+        return ['value' => Gen::intBetween(1, 10)];
+    }
+}

tests/Testo/PropertyInterceptorTest.php

@@ -282,6 +282,7 @@ public static function invalidAttributeValuesProvider(): iterable
         yield 'exhaustiveBudget: 0' => [ZeroExhaustiveBudgetStub::class, 'exhaustiveBudget must be greater than or equal to 1'];
         yield 'flakyReplays: -1' => [NegativeFlakyReplaysStub::class, 'flakyReplays must be greater than or equal to 0'];
         yield 'searchRuns: -1' => [NegativeSearchRunsStub::class, 'searchRuns must be greater than or equal to 0'];
+        yield 'failOn: -1' => [NegativeFailOnStub::class, 'failOn must be greater than or equal to 0'];
         yield 'path without seed' => [PathWithoutSeedStub::class, 'path replays a recorded descent and requires the seed it was recorded with'];
         yield 'throws: not a Throwable' => [ThrowsNonThrowableStub::class, 'throws names "stdClass", which is not a Throwable'];
         yield 'throws: a class a failed assertion is an instance of' => [ThrowsExceptionStub::class, 'throws names "Exception", which a failed assertion is an instance of — a falsified body would pass; name the exception the body throws'];
@@ -1048,6 +1049,41 @@ static function (TestInfo $info): TestResult {
         Assert::true($outcome->isDiscarded());
     }
 
+    public function failOnTurnsAMatchingDiagnosticIntoATrialFailure(): void
+    {
+        $executor = new TestoTrialExecutor(
+            $this->info(PassingStub::class, 'check'),
+            static function (TestInfo $info): TestResult {
+                trigger_error('diagnostic from input', E_USER_WARNING);
+
+                return new TestResult(info: $info, status: Status::Passed);
+            },
+            failOn: E_USER_WARNING,
+        );
+
+        $outcome = $executor->execute(['x' => 7]);
+
+        Assert::true($outcome->isFailed());
+        Assert::instanceOf($outcome->failure, \ErrorException::class);
+        Assert::same($outcome->failure?->getMessage(), 'diagnostic from input');
+        Assert::same($outcome->failure?->getSeverity(), E_USER_WARNING);
+    }
+
+    public function failOnLeavesSuppressedDiagnosticsSuppressed(): void
+    {
+        $executor = new TestoTrialExecutor(
+            $this->info(PassingStub::class, 'check'),
+            static function (TestInfo $info): TestResult {
+                @trigger_error('suppressed diagnostic', E_USER_WARNING);
+
+                return new TestResult(info: $info, status: Status::Passed);
+            },
+            failOn: E_USER_WARNING,
+        );
+
+        Assert::true($executor->execute(['x' => 7])->isPassed());
+    }
+
     /**
      * The executor's half of #61: the interceptor refuses a class the
      * assertion failure is an instance of, and the executor holds the same

tests/Testo/PropertyRunnerE2ETest.php

@@ -16,6 +16,7 @@
 use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\CorpusRegressionFixture;
 use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\CoverageFixture;
 use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\DeadlineFixture;
+use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\DiagnosticFixture;
 use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\DrawPropertyFixture;
 use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\ExampleFailingFixture;
 use Rasuvaeff\PropertyTesting\Testo\Tests\Fixture\ExhaustedFixture;
@@ -136,6 +137,17 @@ public function anExpectExceptionAttributeNeverTurnsAFalsifiedPropertyGreen(): v
         Assert::true($result->status === Status::Error || $result->status === Status::Failed);
     }
 
+    public function aSelectedDiagnosticFalsifiesAndShrinksItsInput(): void
+    {
+        $result = TestRunner::runTest([DiagnosticFixture::class, 'everyInputRaisesADiagnostic']);
+
+        Assert::true($result->status->isFailure());
+        Assert::instanceOf($result->failure, PropertyViolationException::class);
+        Assert::same($result->failure->getCounterExample()->shrunkArguments['value'], 1);
+        Assert::instanceOf($result->failure->getCounterExample()->failure, \ErrorException::class);
+        Assert::string($result->failure->getCounterExample()->failure?->getMessage())->contains('diagnostic for');
+    }
+
     public function anExpectExceptionAttributeCanBeSatisfiedByEveryRun(): void
     {
         $result = TestRunner::runTest([ExpectExceptionFixture::class, 'throwsTheExpectedException']);

tests/Testo/PropertyStubs.php

@@ -996,6 +996,18 @@ public static function provide(): array
     }
 }
 
+final class NegativeFailOnStub
+{
+    #[Property(runs: 1, seed: 1, generators: 'provide', failOn: -1)]
+    public function check(int $x): void {}
+
+    /** @return array<string, ArbitraryInterface> */
+    public static function provide(): array
+    {
+        return ['x' => Gen::intBetween(1, 10)];
+    }
+}
+
 final class ExhaustiveStub
 {
     #[Property(runs: 3, seed: 1, generators: 'provide', exhaustive: true)]

tests/Testo/PropertyTest.php

@@ -181,4 +181,9 @@ public function throwsDefaultsToNullAndKeepsTheClassAsWritten(): void
         // Not validated here either; the interceptor refuses a non-Throwable.
         Assert::same((new Property(throws: \stdClass::class))->throws, \stdClass::class);
     }
+    public function failOnDefaultsToDisabledAndKeepsTheDiagnosticMask(): void
+    {
+        Assert::same((new Property())->failOn, 0);
+        Assert::same((new Property(failOn: E_DEPRECATED | E_WARNING))->failOn, E_DEPRECATED | E_WARNING);
+    }
 }