[vincentkoc]

Summary

Stop the paired Fusion benchmark campaign when an agent reports incomplete tests or acceptance grading lacks a complete verdict. Previously, later arms could spend more model calls; incomplete grading could even count as a successful result if its other fields looked successful. The incomplete checkpoint now preserves the settled call cost and failure reason, then stops.

Complete assertion failures remain ordinary scored outcomes.

Verification

• Four new regressions fail on the previous source and pass with this change. They check that only the first arm runs, its usage remains settled, and the checkpoint stays incomplete.
• Focused pair benchmark/analysis tests: 80 passed, one native-gated skip.
• All Fusion script tests: 304 passed, 43 native-gated skips.
• Independent full-context P0–P2 review: no actionable findings.

Deployment and risk

No release or deployment. This changes the experimental benchmark runner only; incomplete execution now requires investigation before a new campaign instead of continuing automatically.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

Stop paired Fusion benchmark campaigns after incomplete agent tests or acceptance grading while preserving settled usage.

Example: The first of three benchmark arms raises incomplete_test_run after one settled model call.

• Before: The runner records the arm error, runs later arms, and can mark the campaign complete.
• After: The runner saves one error record and the settled call, leaves complete=false, and stops before another arm.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Focused, correct repair with meaningful campaign regressions and passing script CI; behavioral evidence remains an in-process setup with substituted agents, grading and transport.
Proof confidence	🦐 gold shrimp (3/6)	This maintainer-authored change affects experimental benchmark tooling; focused runner regressions and exact-head CI support it, and shipped-product proof is not required.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A benchmark operator can incur more model spend after incomplete tests and receive a campaign marked complete despite that execution failure.
Reason: The maintainer-authored direction protects benchmark validity and paid-call boundaries with a focused change in the existing owner.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR fixes a current campaign-level failure path that main still permits, with no actionable introduced defects found.

Priority: P2
Reviewed head: c354f60407088881cd02d29a5d408e1753d88477

Before merge

None.

Findings

None.

Agent review details

How this fits together

The experimental Node benchmark runner executes qualified repository repairs through native model clients, grades their outputs in isolated sandboxes, and writes accounting checkpoints for offline analysis.

flowchart TD
  A[Qualified repair tasks] --> B[Paired campaign runner]
  B --> C[Repair agent]
  C --> D[Native model client]
  C --> E[Acceptance grader]
  E --> B
  B --> F[Usage checkpoint]
  F --> G[Offline analyzer]

Loading

Technical review

Best possible solution:

Use the existing campaign failure and checkpoint machinery to stop incomplete execution while retaining complete failure outcomes and settled accounting.

Do we have a high-confidence way to reproduce the issue?

Current main's agent raises incomplete_test_run, but the campaign catches it without terminating and continues its arm loop; the added regressions directly encode this path, although this review ran no tests.

Is this the best way to solve the issue?

The patch is the smallest complete repair at the campaign owner: validate the existing grading contract and propagate the existing finite error without changing cost settlement or ordinary scored failures.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 2fe4a64a085d.

Provenance checked

• Campaign continuation and scoring changes intended behavior with a stated reason (24d7379: Matched controls intentionally continue after known-cost model truncation; incomplete test execution was not a campaign stop condition.)
• Grading timing and usage finalization keeps the original intent (f0e33cf: Separate agent and acceptance latency while preserving end-to-end timing and failed-phase measurements.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

• Applicable repository policy: Read the complete root AGENTS.md; no nested policies, maintainer notes, or pr-simplify skill were found for the changed paths. Applied architecture and privacy guidance; the explicit single-reviewer instruction supersedes the bundled autoreview requirement. (AGENTS.md:1, c354f6040708)
• Verified introduced change: The pinned diff adds a verdict-validity check and propagates incomplete_test_run through the existing campaign failure path; it adds no alternate runner or accounting owner. (scripts/fusion-pair-benchmark.mjs:205, c354f6040708)
• Production grader contract: testResult supplies a boolean valid field, and verify returns that receipt after removing raw feedback; complete assertion failures remain valid results. (scripts/fusion-workloads.mjs:186, c354f6040708)
• Settlement and checkpoint compatibility: The existing finally block saves call counts, costs and timings before the outer catch records failure; the analyzer already rejects incomplete checkpoints. No persisted field or format changes. (scripts/fusion-pair-benchmark.mjs:224, c354f6040708)
• Focused regression coverage: Four parameterized cases exercise agent failure and invalid or missing grading validity through the real campaign runner, asserting termination, settled usage, incomplete checkpoint status and private-content omission; model transport and grading are substituted. (test/fusion-pair-benchmark.test.mjs:36, c354f6040708)
• Original campaign intent: feat(fusion): benchmark model delegation with matched controls #919 introduced matched controls and deliberately continued after known-cost model truncation; this PR preserves that distinct behavior and its existing regression test. (test/fusion-pair-benchmark.test.mjs:634, 24d73792c239)

Likely related people:

• Vincent Koc: Raw commit f0e33cf adds scripts/fusion-pair-benchmark.mjs:200 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: f0e33cf4200d; files: scripts/fusion-pair-benchmark.mjs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

• add P2: Incomplete experimental benchmark execution can trigger unnecessary paid arms and produce a misleadingly complete campaign; production routing is unaffected.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P2: Incomplete experimental benchmark execution can trigger unnecessary paid arms and produce a misleadingly complete campaign; production routing is unaffected.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 9:42 AM ET / 14:42 UTC.