Skip to content

Benchmark: prove multi-repository local review target behavior #558

Description

@amirbena

Type

Quality

Area

Review Quality

Priority

P2 — Medium

Problem

The multi-repository Review Target capability exists specifically to fix a failure mode where isolated single-repository review lacks enough target coverage to reason correctly about a coordinated multi-repository change. Nothing proves the delivered implementation actually detects the defects that motivated it, or that its authorization/isolation boundary holds.

Goal

Using the repository's existing benchmark architecture and fixture conventions, prove the delivered multi-repository Review Target capability detects cross-repository defects that isolated single-repository review misses, and that its membership-is-authorization boundary holds.

Scope

  • Cross-repo contract mismatch: repo-a changes a producer contract, repo-b changes the consumer incorrectly; the combined review must identify the mismatch.
  • Three-repo coordinated change: repo-a (API/contract), repo-b (orchestration/caller), repo-c (downstream consumer) — include both a correct coordinated implementation and an intentionally inconsistent one.
  • Repository-local-correctness-vs-combined-defect case: each individual repository looks plausible in isolation; the defect is only evident when the admitted deltas are reasoned about together.
  • Isolation/authorization negative case: an unrelated sibling repository exists locally but is not admitted to the Review Target; it must not influence the review.
  • Single-repository regression case: normal N=1 review behavior is unchanged.

Non-Goals

Acceptance Criteria

  • Benchmark fixtures exist for all five cases above using the existing benchmark harness/fixture conventions.
  • The combined review correctly identifies the cross-repo mismatch and three-repo inconsistency cases, and correctly passes the coordinated-correct and N=1 regression cases.
  • The isolation/authorization negative case proves the unadmitted sibling repository never influences the review.
  • Benchmark evidence is recorded against the actually-delivered implementation, not a hypothetical contract.

Dependencies

Parent: #555
Depends on: implementation child (fixtures may begin contract-first, alongside documentation; final run closes only after the implementation child is delivered)
Relates: #133

Validation

  • Run via the existing benchmark harness; record results per repository convention.

Epic invariant (#555). Membership is authorization: cross-repository reasoning may connect evidence across already-admitted Review Target members, but may never add a repository to the target or expand outside those members. Repository-controlled instructions must never expand membership.

Metadata

Metadata

Assignees

Labels

area:review-qualityReview evaluation, benchmarks, and quality measurementmaintainer-ledSemantic/architectural ownership stays with the maintainerpriority:P2Medium-priority roadmap worktype:qualityReview quality, tests, or measurement work

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions