[vincentkoc]

Summary

Stop one-shot and refined Fusion drafts when visible-test execution is incomplete. A watchdog-terminated candidate now throws incomplete_test_run before another draft, a probe, or fallback inference. Complete assertion failures retain their correction and fallback paths.

Verification

• Full dependency-free Fusion suite: 300 passed, 43 native-gated skips, zero failures.
• Focused contract/sampling suite with native Docker watchdog: 30 passed, one unrelated native-gated skip.
• Removing the validity guards makes both new rejection regressions fail.
• Independent full-context P0–P2 review: no findings.

Deployment and risk

Benchmark-only change; no release, deployment, production routing, or default changes. Incomplete runs now terminate instead of consuming more model calls. No model-performance gain is claimed.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

Stop one-shot and refined Fusion drafts when visible tests return an incomplete result.

Example: A generated patch contains while (true) {} and the visible-test watchdog terminates it.

• Before: Sampling can request a second draft, and contract execution can continue verification or fallback.
• After: The draft throws incomplete_test_run and closes its sandbox before further inference.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Focused, source-correct repair with meaningful regression coverage; execution evidence remains contributor-reported and harness-based.
Proof confidence	🦐 gold shrimp (3/6)	This experimental benchmark-script repair requires no shipped-entry-point proof; inspected coverage includes injected incomplete verdicts and a native Docker watchdog regression, with execution results reported in the PR body.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Benchmark trials can spend more model calls or continue verification after visible tests terminate without a complete result.
Reason: The feature owner explicitly chooses terminal handling for incomplete execution while retaining correction and fallback for complete failures.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR fixes draft paths that still ignore incomplete visible tests on main; no actionable introduced defect was found.

Priority: P2
Reviewed head: cba135c4d631e22c120683bfa21563755b35ea71

Before merge

None.

Findings

None.

Agent review details

How this fits together

The experimental Fusion benchmark sends model-generated edits into isolated sandboxes, checks visible tests, and passes completed candidates to contract verification and independent acceptance grading.

flowchart TD
  A[Benchmark runner] --> B[Draft model request]
  B --> C[Apply edits in sandbox]
  C --> D[Visible tests]
  D --> E[Complete result continues verification]
  D --> F[Incomplete result throws]
  F --> G[Sandbox cleanup]

Loading

Technical review

Best possible solution:

Land the two validity guards in the existing draft owners, preserving cleanup and recovery from complete assertion failures.

Do we have a high-confidence way to reproduce the issue?

Main visibly ignores the checker's validity verdict at both draft boundaries; the supplied native regression exercises a watchdog-terminated infinite loop, although its execution was not independently repeated.

Is this the best way to solve the issue?

The guards are the smallest complete fix for the stated draft behavior, using the existing validity verdict, finite error code, propagation, and sandbox cleanup.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 1d05a345bbea.

Provenance checked

• One-shot validation and sampling after timeout changes intended behavior with a stated reason (cd4a014: fix(fusion): bound draft test execution inside sandbox #1020 bounded test execution and allowed sampling to recover with a second candidate after a hanging first candidate.)
• Refinement verification and correction changes intended behavior with a stated reason (e6113d7: feat(fusion): bound cheap draft repair by executable feedback #949 introduced bounded revisions from executable feedback; this repair preserves that mechanism for complete results and explicitly terminates incomplete execution.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

• Current main still contains the gap: The default branch remains at the supplied base; runSingleDraft consumes passed without checking valid, and runRefinedDraft proceeds directly from visible tests to the reproducer. (scripts/fusion-contract-agent.mjs:152, 1d05a345bbea)
• Focused repair: Both visible-test boundaries reject valid !== true; existing finally blocks close the sandboxes, while valid assertion failures retain their existing paths. (scripts/fusion-contract-agent.mjs:153, cba135c4d631)
• Existing checker contract: The shared checker already reports validity from complete TAP totals and runs tests under an in-container watchdog; this PR consumes that verdict without introducing another execution mechanism. (scripts/fusion-workloads.mjs:185, cba135c4d631)
• Intentional change from timeout recovery: fix(fusion): bound draft test execution inside sandbox #1020 deliberately allowed a working second sample after a hanging first sample; this maintainer-authored PR explicitly replaces that behavior with terminal incomplete_test_run, consistent with fix(fusion): stop agents after incomplete visible tests #1029. (test/fusion-draft-sampling.test.mjs:119, cd4a014daa42)
• Regression coverage: New rejection cases cover both draft boundaries and sandbox closure; the updated native sampling case uses the real Docker checker with a hanging patch. The body reports successful focused execution and failures when the guards are removed; this read-only review ran no target code. (test/fusion-contract-agent.test.mjs:248, cba135c4d631)
• Policy and history scope: The complete root AGENTS.md is the only applicable policy file; no maintainer-notes directory or pr-simplify skill exists. Earlier follow-history stops at an unavailable historical object, but the single-draft and refinement introduction patches were independently readable. (AGENTS.md:1, 1d05a345bbea)

Likely related people:

• Vincent Koc: Raw commit e6113d7 adds scripts/fusion-contract-agent.mjs:177 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: e6113d79ba26; files: scripts/fusion-contract-agent.mjs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

• add P2: The defect permits unnecessary inference and verification after incomplete tests in an experimental benchmark, without affecting production routing.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P2: The defect permits unnecessary inference and verification after incomplete tests in an experimental benchmark, without affecting production routing.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 8:59 AM ET / 13:59 UTC.