Skip to content

refactor(test-writing): correct unapplicable rules and scope fix remediation - #180

Merged
Martin Bens (SpiGAndromeda) merged 3 commits into
mainfrom
refactor/test-rule-catalog-refinement
Sep 6, 2026
Merged

Martin Bens (SpiGAndromeda) merged 3 commits into
mainfrom
refactor/test-rule-catalog-refinement

Conversation

@SpiGAndromeda

@SpiGAndromeda Martin Bens (SpiGAndromeda) commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator

Rule-catalog corrections, two removals, and a remediation invariant for test-writing, released as 5.2.0. The rule corrections implement the catalog items from #174; the CONV-005 ordering item is withdrawn there with a warning note. The remediation invariant relates to #176, whose structure-churn item stays open.

Rules corrected against what they actually enforce

Each of these rules flagged code the project's own configuration permits, demanded something its cited mechanism does not check, or contradicted its own examples. Each fix quotes the deciding source.

  • UNIT-004 — the shared-mock exception is scoped to the namespaces where shopware.createMockWithoutExpectations actually enables it (Shopware\Tests\Unit\, Shopware\Tests\DevOps\, per common.neon:73), and the mock-without-expectations mechanics defer to PHPStan's NoCreateMockWithoutExpectationsRule. A never()/once() pair that pins reachability is carved out with a worked example — the issue text is explicit that no outcome assertion substitutes for it. A "Stub Conversion Gotchas" section quotes the guideline verbatim (->with() exists on mocks only; the replacement type is Foo&Stub, not MockObject), because remediations kept producing stubs that no longer compile.
  • CONV-005 — demoted to consider and given a single-category exemption. A class whose methods all belong to one category has no ordering to violate.
  • CONV-011 — suppressed for mechanical data-provider expansions, which repeat a shape the provider already documents.
  • CONV-014 — a carve-out for position-load-bearing statements, plus a repaired Fix example that previously violated the rule it illustrates.
  • CONV-017 — the exemption now requires at least two reads inside the single referencing method observing the same instance; one read is exactly the inline-it case the rule exists for.
  • DESIGN-003 — assertion-based comparison and provider scope stated with the guideline quoted verbatim.
  • DESIGN-004 — boundary-input carve-out, guideline quoted verbatim.
  • UNIT-001 — findings must name the enforcing mechanism from a fixed seven-name vocabulary instead of asserting "this is enforced elsewhere".
  • UNIT-003 — defers its stub-typing remainder to the same PHPStan rule as UNIT-004.
  • MIGRATION-008 — standalone clarifying comment.

Removed: checks a deterministic gate already runs

Removed Enforced upstream by
CONV-007 — class-element ordering php-cs-fixer ordered_class_elements over tests/
MIGRATION-007 — assertEquals discouraged PHPStan AssertEqualsIsDiscouragedRule, all paths

The unit catalog goes from 44 rules to 43. get_rules(ids=...) returns not-found for either id. Fallout is propagated: the composed class-structure claims name only CONV-005, the migration skill's should-fix example cites CONV-014, the red-team lens shorthand drops UNIT-005 (removed in 4.2.1) and cites CONV-012 where it named MIGRATION-007, and the BATS assertions that pinned CONV-007 behavior retarget onto CONV-005 and CONV-017 while keeping what each test pins.

Remediation invariant

A reviewer's suggested fix could satisfy a structural rule by weakening what the test proves — reorder methods and drop an assertion, rename and retarget one, silence a layout finding by asserting something adjacent. The three reviewing skills' output requirements and the four team-review prompt builders (reviewer, reconcile, red team, defense) now carry the same invariant: a fix never changes what an existing assertion pins, and never introduces an assertion as a means of satisfying a structural, layout, naming, or style constraint. Coverage-gap rules (MIGRATION-008, DESIGN-006, INTEGRATION-007) keep new assertions as the finding's explicit deliverable, deletions ride removed_assertions, and re-expressing the same pinned fact stays legal (CONV-012's assertTrue($a === $b) to assertSame($b, $a)). The migration skill's warnings example now carries empty deleted_methods/removed_assertions, matching CONV-014's reorder-only semantics.

Limits

  • The CONV-005 ordering problem is deliberately unfixed. A deterministic method-name classifier was built and withdrawn: against 15,682 real tests/unit method names, 61% fell into the default category, so deterministic category identification is not possible and must not be reattempted. Test-rules catalog rules produce findings that cannot be applied as written #174 documents this under the CONV-005 bullet.
  • INTEGRATION-006's relation to the invariant is unresolvable from its rule file — its Fix example elides the code that would decide it.

Several rules in the test catalog generated findings that fail when applied: remediations the target repository's static gates reject, intentional patterns the rules could not see, and re-derivations of checks that active PHPStan rules already make deterministically.

UNIT-004 gains an exception for a shared `createMock()` double whose sibling methods set expectations, and a deferral (echoed in UNIT-003) to `NoCreateMockWithoutExpectationsRule` where that rule already reports a double. Both are tied to the `createMockWithoutExpectationsEnabledNamespaces` regime in Shopware's PHPStan config. A `never()`/`once()` pairing across two test methods is recognized as the assertion that pins branch reachability, exempt from the over-coupling remedy. Remediation notes quote the Shopware unit-test guideline's two stub-conversion gotchas (`->with()` exists on mocks only, and a fixture-helper parameter takes `Foo&Stub`, not `MockObject`), because a remedy omitting them passes PHPUnit and fails PHPStan.

CONV-007 and MIGRATION-007 are removed: php-cs-fixer's `ordered_class_elements` runs over `tests/` and enforces CONV-007's member slots mechanically, and phpstan-phpunit's `AssertEqualsIsDiscouragedRule` is active on every analyzed path including `tests/migration`. MIGRATION-008's comment no longer references the removed rule.

CONV-005 drops to `consider` and gains a single-category exemption, so a class deliberately holding one half of a split is no longer told to grow the other half. CONV-014 treats an assertion between two actions as intentional when moving it past the next action changes the observed value, and its own Fix example no longer performs that defective move. DESIGN-003 compares assertions rather than bodies and requires a provider fold to respect the target provider's documented scope. DESIGN-004 keeps boundary inputs whose distinctness the implementation deliberately erases. CONV-011 suppresses annotations that mechanically expand the method name. UNIT-001 findings name the mechanism enforcing an assertion called trivial. CONV-017 sends reused state to `setUp()` instead of inlining it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reviewing skills, team-review prompts, and docs now match the reduced catalog. Composed class-structure claims name only CONV-005, the migration skill's should-fix example cites CONV-014 because CONV-005 is a consider rule now, the red-team lens shorthand drops UNIT-005 (removed in 4.2.1) and cites CONV-012 where it named MIGRATION-007, the README and AGENTS.md tables drop the removed rows, and the root README's count names what it counts ("the 43 rules of the Shopware unit-test catalog"). Assertions in `review_unit.bats` and `build_rule_package.bats` retarget their CONV-007 checks onto CONV-005 and CONV-017, preserving what each test pins (class-structure filtering, `scoped-review=exclude` handling, exact-id matching).

A remediation invariant lands in the three reviewing skills' output requirements and in the four team-review prompt builders (reviewer, reconcile, red team, defense): a suggested fix never changes what an existing assertion pins, and never introduces an assertion as a means of satisfying a structural, layout, naming, or style constraint. Coverage-gap rules (MIGRATION-008, DESIGN-006, INTEGRATION-007) keep new assertions as the finding's explicit deliverable, deletions ride `removed_assertions`, and re-expressing the same pinned fact (CONV-012's `assertTrue($a === $b)` to `assertSame($b, $a)`) stays legal. The migration skill's warnings example now carries an empty `deleted_methods`/`removed_assertions` payload, matching CONV-014's reorder-only semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

@SpiGAndromeda
Martin Bens (SpiGAndromeda) merged commit 46569de into main Sep 6, 2026
13 checks passed
@SpiGAndromeda
Martin Bens (SpiGAndromeda) deleted the refactor/test-rule-catalog-refinement branch October 11, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant