[vincentkoc]

Summary

Fusion repeat analysis counted every zero-score visible pass as a false accept, including budget stops that had never submitted. Distinguish submitted failures, visible-pass failures, and passing unsubmitted patches in both single-block and repeated reports. Scores, accounting and pilot criteria are unchanged; older receipts without an outcome cannot establish submission.

Document the frozen two-task, two-block confirmation: Sol-to-Luna handoff and Sol each completed 2/4 repairs, Luna 0/4. Handoff cost 57.51% less and used 9.20% more agent time than Sol. All completions were the streaming task; neither arm completed lifecycle repair. The 50% completion rate fails the predeclared 75% reliability threshold. No superiority or production-promotion claim.

Correct earlier result labels that reversed the executed model order: edit-handoff runs Sol first, then Luna. The recorded calls establish the direction; costs and scores are unchanged.

Verification

• 90 focused pair-analysis/runner tests passed; one optional Docker test skipped.
• New diagnostic regression fails against the original analyzer.
• Replayed both settled blocks without new model calls: one submitted false accept per arm; one passing unsubmitted patch in the Sol arm. Original receipts and scores preserved.
• Independent P0–P2 review: scoped-clean, with scope limited to supplied diff and benchmark evidence.
• Public diff privacy checks passed.

Deployment and risk

No release, deployment or routing-default change. Analysis consumers should use visiblePassFailures for the previous broad diagnostic; falseAccepts now requires an observed submission. Benchmark results concern two historical development tasks, not public endpoint quality or statistical superiority.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs changes before merge.

What this changes

Separate submitted benchmark failures from unfinished attempts and document the repeated repair results.

Example: A budget-stopped attempt has publicPassed=true, score=0, and outcome=budget.

• Before: Repeated analysis counts one false accept.
• After: Analysis counts one visible-pass failure and zero false accepts.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	Useful, focused repair with relevant regression coverage, but a supported submission outcome is omitted.
Proof confidence	🦐 gold shrimp (3/6)	This offline benchmark-analysis and documentation change requires no shipped application proof; the body reports settled-receipt replay, while inspected coverage is unit tests and exact-head CI.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Partial
User problem: Benchmark readers cannot distinguish a submitted repair that failed independent grading from an attempt that stopped before submission.
Reason: The maintainer-authored direction improves experimental interpretation while retaining conservative scoring; supported draft submissions remain misclassified.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep open: this maintainer-authored PR addresses a real diagnostic error, but its submission check misses supported draft submissions.

Priority: P2
Reviewed head: 7392f57329e2c0d7be57628fff5c587c92855997

Before merge

• Recognize draft_submitted as a submitted repair (P2) - The supported draft-sampling, probe-selection, and contract-draft paths can return outcome='draft_submitted'. runSingleDraft emits that value, and runContractAgent explicitly treats it as submission. When such a candidate passes visible tests but fails independent acceptance, its receipt has publicPassed=true and score=0, yet this predicate reports zero false accepts. The inverse predicate also treats this submission outcome as unfinished. Recognize both existing submission outcomes consistently and add a regression using a draft_submitted receipt in single-block and repeated analysis.

Findings

• [P2] Recognize draft_submitted as a submitted repair — scripts/fusion-pair-analysis.mjs:26-29

Agent review details

How this fits together

The offline Fusion analyzer reads settled benchmark receipts and emits quality, cost, timing, failure diagnostics, and pilot-screen results.

flowchart LR
  A[Repair agents] --> B[Pair benchmark runner]
  B --> C[Settled receipts]
  C --> D[Pair analyzer]
  D --> E[Failure diagnostics]
  D --> F[Pilot screen]

Loading

Technical review

Best possible solution:

Recognize both existing submission outcomes in the analyzer’s shared classifier and cover draft failures without changing scores, accounting, or pilot gates.

Do we have a high-confidence way to reproduce the issue?

Source establishes the original broad false-accept count and the introduced draft-submission omission; execution was excluded by the read-only review instructions.

Is this the best way to solve the issue?

Shared diagnostic classification is the right small fix, provided it follows the submission contract already used by the contract and draft agents.

Full review comments:

• [P2] Recognize draft_submitted as a submitted repair — scripts/fusion-pair-analysis.mjs:26-29
  The supported draft-sampling, probe-selection, and contract-draft paths can return outcome='draft_submitted'. runSingleDraft emits that value, and runContractAgent explicitly treats it as submission. When such a candidate passes visible tests but fails independent acceptance, its receipt has publicPassed=true and score=0, yet this predicate reports zero false accepts. The inverse predicate also treats this submission outcome as unfinished. Recognize both existing submission outcomes consistently and add a regression using a draft_submitted receipt in single-block and repeated analysis.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.97

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 1f12686c1448.

Provenance checked

• Repeated false-accept diagnostics changes intended behavior with a stated reason (a1772b0: Expose failed visible-passing repairs alongside repeated quality, cost, and timing comparisons.)
• Earlier handoff result labels changes intended behavior with a stated reason (test(fusion): add quota snapshot repair workload #1039: Record a matched development repair comparison; the PR corrects its model-order label to agree with existing edit-handoff model selection.)
• Complete-adviser comparison labels changes intended behavior with a stated reason (test(fusion): add complete-adviser repair workload #1037: Record a controlled development comparison without claiming a fusion quality gain; the corrected direction preserves that interpretation.)

Testing

Proof path: unit tests only.

Security

None.

Evidence

Acceptance criteria:

• node --test test/fusion-pair-analysis.test.mjs
• node --test test/fusion-pair-benchmark.test.mjs

What I checked:

• Pinned introduced change: The PR adds shared failure diagnostics to single-block and repeated reports while preserving score aggregation and pilot criteria. (scripts/fusion-pair-analysis.mjs:22, 7392f57329e2)
• Supported submission contract: The existing contract agent explicitly recognizes both submitted and draft_submitted as submissions; the draft producer emits draft_submitted after applying and testing a patch. (scripts/fusion-contract-agent.mjs:92, 1f12686c1448)
• Reachable affected policy: Draft sampling selects a visible-passing draft_submitted candidate and preserves its outcome in the result; the pair runner copies that result into the receipt before independent grading. (scripts/fusion-draft-sampling.mjs:36, 1f12686c1448)
• Original diagnostic intent: feat(fusion): validate repeated benchmark comparisons #965 introduced repeated-analysis false-accept reporting to expose misleading repair outcomes; this PR preserves that purpose but needs all supported submission outcomes. (scripts/fusion-pair-analysis.mjs:184, a1772b0e3d76)
• Current branch and discussion: The default branch remains the pinned base, the author association is MEMBER, and discussion contains only review-status bot comments. (1f12686c1448)
• Validation scope: Exact-head script-tests passed; qualify remains in progress and cancelled dispatch checks establish no diff-caused failure; no tests or target code were run during this read-only review. (test/fusion-pair-analysis.test.mjs:230, 7392f57329e2)

Likely related people:

• Vincent Koc: Raw commit a1772b0 adds scripts/fusion-pair-analysis.mjs:150 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: a1772b0e3d76; files: scripts/fusion-pair-analysis.mjs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

• add P2: Incorrect offline benchmark diagnostics can mislead experiment interpretation, with no production routing impact.
• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦐 gold shrimp and patch quality is 🦪 silver shellfish.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

• P2: Incorrect offline benchmark diagnostics can mislead experiment interpretation, with no production routing impact.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦐 gold shrimp and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 1:08 PM ET / 18:08 UTC.