[vincentkoc]

Summary

Pair analysis could report quality gains or cost savings from a row whose acceptance tests never completed, including a forged passing verdict. Require valid: true for every completed or graded row before single-run or repeated analysis. Preserve known-cost execution failures that never reached grading.

Verification

• Six new regressions failed before the fix and pass afterward, covering zero/one scores, false/missing validity, error rows and repeat reports.
• Focused pair benchmark and analysis tests: 86 passed, one native-gated skip.
• Full dependency-free Fusion test suite passed.
• Independent P0–P2 review: no actionable findings, complete source context.
• Diff whitespace and private-data checks passed.

Deployment and risk

No release, deployment, provider call, or production routing change. Analysis now rejects legacy or hand-authored completed rows without an explicit complete acceptance verdict; such data must be requalified rather than silently counted.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

Require an explicitly valid acceptance verdict before analyzing completed or graded Fusion pair rows.

Example: A completed candidate row has score 1, passed true, valid false, and costs 40 micros versus the strong control's 100 micros.

• Before: An otherwise consistent report can claim 60% cost savings despite incomplete acceptance grading.
• After: Analysis rejects the report with incomplete_pair_grade.

Review scores

Measure	Result	What it means
Overall readiness	🧂 unranked krab (1/6)	The implementation is focused and source-correct, but supplied verification remains unit-level rather than demonstrating the changed CLI behavior.
Proof confidence	🧂 unranked krab (1/6)	Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Offline reports can present quality gains or savings using acceptance grades that never completed.
Reason: This restores the analyzer's documented complete-report contract, with the compatibility tradeoff explicitly accepted by its maintainer author.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR fixes a current analyzer validation gap, and the focused diff introduces no actionable correctness findings.

Priority: P2
Reviewed head: f47e8bdefbcf39e10f33fc93babb62b7ff581a53

Before merge

None.

Findings

None.

Tests

• Missing end-to-end proof: Provide redacted terminal output showing the analyzer CLI rejecting incomplete grades in single and repeated reports while still accepting a valid pre-change native report and an ungraded execution failure.

Agent review details

How this fits together

The offline Fusion analyzer reads benchmark JSON reports, validates outcomes and settled costs, and emits single-run comparisons or repeated pilot summaries.

flowchart LR
  A[Benchmark reports] --> B[Analyzer CLI]
  B --> C[Shared row validation]
  C --> D[Single run comparisons]
  C --> E[Repeated pilot summary]
  C --> F[Reject incomplete grades]

Loading

Technical review

Best possible solution:

Keep the validity check in the existing shared analyzer while preserving settled execution failures that never reached grading.

Do we have a high-confidence way to reproduce the issue?

Current-main source accepts an otherwise consistent passing row with verdict.valid false; zero-score rows also bypass grade-completeness checks, and repeated analysis reuses that path.

Is this the best way to solve the issue?

The guard is the smallest complete fix: it enforces the existing completeness contract before aggregation and leaves ungraded execution failures intact.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against cd22662c64a9.

Provenance checked

• Pair-row acceptance before aggregation keeps the original intent (feat(fusion): analyze paired quality and cost tradeoffs #931: Analyze complete, settled pair reports without treating incomplete outcomes or inconsistent scores as evidence of gains.)
• Repeated pair analysis keeps the original intent (a1772b0: Validate matched repeated comparisons through the shared single-run validator before computing exploratory pilot results.)

Testing

Proof path: unit tests only.

Security

None.

Evidence

What I checked:

• Exact introduced change: The PR adds one three-line guard in the existing shared analyzer, updates its fixture, adds six rejection regressions, and documents the behavior. (scripts/fusion-pair-analysis.mjs:79, f47e8bdefbcf)
• Current main remains affected: Main validates passing scores and accounting without requiring verdict.valid; repeated analysis calls the same validator, so both analysis modes remain affected. (scripts/fusion-pair-analysis.mjs:79, cd22662c64a9)
• Existing report compatibility: The original native grader already emitted the valid field, and the benchmark retained its verdict. The PR changes qualification of existing reports without changing their serialized structure or rewriting stored data. (scripts/fusion-workloads.mjs:107, fd36a06a9b4c)
• Complementary producer repair: fix(fusion): stop paired campaigns after incomplete tests #1033 stops new benchmark campaigns on incomplete grading; it does not validate previously saved or supplied reports at the analyzer boundary. (scripts/fusion-pair-benchmark.mjs:205, cd22662c64a9)
• Intent and owner direction: feat(fusion): analyze paired quality and cost tradeoffs #931 established complete-report analysis. This PR's MEMBER author explicitly accepts rejecting legacy or hand-authored completed rows without valid true. (f47e8bdefbcf)
• Policy and inspection limits: Read the complete root AGENTS.md; no nested policies, maintainer notes, or pr-simplify skill were found for these paths. The read-only review instruction overrides running tests or autoreview. Some historical blobs were unavailable locally; GitHub reads recovered the relevant original producer contract. (AGENTS.md:1, f47e8bdefbcf)

Likely related people:

• Vincent Koc: Raw commit fd36a06 adds scripts/fusion-pair-analysis.mjs:45 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: fd36a06a9b4c; files: scripts/fusion-pair-analysis.mjs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Review metrics

Metric	Value	Why it matters
Focused patch size	Production +3; tests +25/-1; documentation +5	The shared guard avoids parallel validation paths, and the regression cases cover distinct acceptance conditions.

Labels

Label changes:

• add P2: Incomplete benchmark grades can produce misleading quality and cost conclusions, with impact limited to offline research analysis.
• add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P2: Incomplete benchmark grades can produce misleading quality and cost conclusions, with impact limited to offline research analysis.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Attach the redacted analyzer CLI trace described in the proof assessment.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 10:00 AM ET / 15:00 UTC.