Benchmark Testing

Benchmark Testing

How this repository's own review-quality benchmark works. It is a contributor and maintainer concern, not a capability of either Skill: it measures whether the reviewer still finds the right defects, and is never consulted while a review runs. For where it sits among the repository's other checks, see Reliability-and-Benchmarks.

This page is orientation. The contracts, schemas, and exact numbers live in the repository and are linked from each section; if a figure here and a repository file ever differ, the repository file is correct. Ownership of the benchmark's files is stated in benchmark/README.md, and the cross-component design is docs/benchmark-measurement-architecture/.

What it is

A fixed, versioned set of review cases with known-correct answers, a runner that executes the unmodified Skill over them, and deterministic scoring that compares what the reviewer produced against what was expected.

Piece	What it is	Canonical source
Corpus	About 180 benchmark-case/v2 fixtures in sub-corpora (for example decision derivation, API compatibility, deployment and infrastructure, dependency tree). Each case holds a patch, its expected findings, expected severity and location, and typed variance (accepted alternatives, optional findings, permitted severity range).	benchmark/corpus/, format in fixture-format.md
Runner	Runs the reviewer over a case in a disposable workspace, keeps every source checkout protected, cleans up on success and failure, and writes a machine-readable per-case result.	runner-contract.md
Scoring	Deterministic: whether a produced finding matches an expected one (MATCH / NEAR_MISS / NO_MATCH), missed and incorrect findings, severity accuracy (exact, over, under), duplicate noise, and a citation-existence check.	match-criteria.md, missed-and-incorrect-findings.md, severity-accuracy.md, duplicate-noise.md, citation-fidelity.md
Regression report	Compares a run against a stored baseline, guarded by a corpus-identity check, and reports per-case and aggregate deltas.	regression-report.md

The metrics are reported side by side and none of them gates a merge.

Case selection for a change

Running every case for every change would be wasteful, so a deterministic selector narrows the corpus to the cases a change could affect.

change diff
  → classify into the closed benchmark taxonomy   (one bounded model call)
  → look up candidate cases in the inverted index (corpus-index.json)
  → score and pick a bounded Top-K                 (deterministic arithmetic)
  → require 60% weighted Selection Coverage
  → otherwise report an explicit `insufficient-coverage` outcome

Everything after the single classification call is reproducible, with no embeddings or second model call, and the selector explains why each case was chosen. See taxonomy.md and selection.md.

The selector and its classifier are command-line tools (classify_pr_diff.py, select_benchmark_cases.py) and are informational only: no repository CI workflow invokes them, and their output is never a merge requirement. A shadow-validation tool (shadow-validation.md) measures whether a selector's choices would have caught the cases that later regressed. The committed burn-in report is a worked example built from reconstructed samples to demonstrate the method — it is not live evidence.

Scheduled execution

Real reviews by a model run on a maintainer-controlled schedule, never from contributor pull requests. The schedule is defined in schedule-spec.md and its manifest.

Lane	Runs	Cadence	Enforced longest gap
Sentinel	the 4 permanent canonical cases	about every 3 days from 01:00 Israel local time	96 hours
Comprehensive	every fixture in the corpus tree	weekly, Friday 01:00 Asia/Jerusalem	192 hours

• Execution happens in maintainer-configured Claude Cloud Routines through run_benchmark_routine.py. The Routine holds no GitHub write credential; it seals each run as a content-hashed benchmark-result/v1 record. See cloud-routine-integration.md and benchmark-result-schema.md.
• Baselines are pinned, deliberately refreshed references, keyed independently per lane (nightly-history-and-baseline.md).
• Drift separates real regressions from noise using three closed types (missed-required-finding, decision-flip, severity-accuracy-drop), confirmed by re-running before anything is reported. Each drift has a stable fingerprint and opens at most one tracking issue, which is commented on when it recurs and closed when it resolves (drift-detection-and-regression-lifecycle.md).
• Publication is separate from execution. A deterministic publisher (sweep and watchdog commands, run by the benchmark-publish.yml workflow with no model credentials) validates sealed records, publishes results, and raises an issue when an expected run is missing (publication-cli.md). Design rationale and operating procedures: scheduled-operations/.

The benchmark is a quality-observability signal for the maintainer. It never blocks a contributor's pull request or merge, by design: execution only runs through a maintainer-controlled runtime, and no automatic, repository- triggered runtime is provisioned. The execution contract is runtime-execution-contract.md.

Whether a given lane is running right now depends on the maintainer's Routine configuration, which the repository does not record; the publisher's health-status comment is the place that shows it.

Reliability-boundary checks

Benchmark-side checks of whether review output can be trusted. They are benchmark evidence only and change nothing in a live review:

• Citation-existence check — mechanically confirms that the file, line span, symbol, and quoted evidence a finding cites exist in the reviewed tree. Each finding is verified, fabricated, or unverifiable. It proves existence, not that the code was inspected.
• Blocking-verdict proof — corpus fixtures with an unambiguous blocking defect (for example dd-blocking-p0-sql-injection) that assert the real Skill never renders a clean verdict for them.
• Verdict-consistency research — a research record that recommends a narrow pre-publication check; it implements nothing (verdict-consistency-boundary-research.md).

Citation grounding (whether a finding's reasoning actually rests on its citations) is not implemented.

Three kinds of signal — keep them apart

• Review-time evidence supports a finding while a review runs.
• Observational telemetry records what a review inspected and never affects it — see Feature-Review-Execution-Telemetry.
• Benchmark ground truth (this page) asks whether the reviewer was right against fixed cases with known-correct findings. It runs against the corpus, never against a real user's pull request.

Telemetry is not ground truth, and neither substitutes for the other. Historical analytics and repository-scoped learning are a product-layer concern outside the Skill, recorded in the architecture model.

Where the code lives

Concern	Script
Taxonomy classification and inverted-index build	classify_pr_diff.py, build_benchmark_index.py
Top-K selection	select_benchmark_cases.py
Shadow validation	shadow_validate.py
Run, history, and drift	run_benchmark.py, benchmark_history.py, benchmark_drift.py
Cloud Routine entrypoint	run_benchmark_routine.py
Publication	publish_benchmark.py and the publisher/ package

Related pages

• Reliability-and-Benchmarks — how the benchmark relates to the deterministic tests.
• Feature-Review-Execution-Telemetry — the observational record that sits beside this benchmark.
• Development-Workflow and Repository-Documentation-Map — the contributor entrypoint and where each kind of canonical content lives.
• Architecture-Overview — the Skills' own architecture, a separate concern from this measurement system.

Canonical references

• docs/benchmark-measurement-architecture/benchmark-measurement-architecture-model.md — the cross-component design, layer ownership, and runtime contract.
• runtime_platform/benchmark/README.md — the document map for every benchmark contract.
• benchmark/README.md — corpus and ownership.