Skip to content

fix(fusion): stop grounded agents on incomplete tests - #1030

Merged
vincentkoc merged 2 commits into
mainfrom
fix/fusion-grounded-incomplete-tests-20261113
Nov 13, 2026
Merged

vincentkoc merged 2 commits into
mainfrom
fix/fusion-grounded-incomplete-tests-20261113

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

Stop grounded and checked repair workers when visible tests do not produce a complete result. Previously an incomplete check after an edit or submission could enter another model turn or final grading. Both paths now raise the existing finite incomplete_test_run error and close the sandbox.

Complete assertion failures remain repairable.

Verification

  • 32 focused grounded-agent and local-runner tests passed, with no skips, including a real Docker watchdog-termination case.
  • Removing the validity guard made both new rejection regressions fail.
  • Independent full-context P0–P2 review: clean.

Deployment and risk

No deployment or default promotion. Incomplete execution now stops the run instead of being treated as model feedback; no retry or model fallback is added.

@clawsweeper

clawsweeper Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vincentkoc
vincentkoc marked this pull request as ready for review November 13, 2026 13:19
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Nov 13, 2026
@clawsweeper

clawsweeper Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

Stop grounded and checked Fusion workers when visible tests return an incomplete result.

Example: An edit introduces an infinite loop and the visible-test watchdog terminates execution.

  • Before: The grounded worker could request another model turn or return files for acceptance grading.
  • After: The worker throws incomplete_test_run and closes its sandbox.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) Focused, source-correct repair with useful regression coverage; inspected execution evidence remains harness-based and contributor-reported.
Proof confidence 🦐 gold shrimp (3/6) This maintainer-authored experimental benchmark repair does not require shipped-entry-point proof; supplied coverage includes a native Docker watchdog regression and mocked edit/finish cases.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Benchmark runs can continue inference or grading after visible tests terminate without a complete result.
Reason: The feature owner's focused repair enforces the existing execution contract without changing production routing, profile defaults, or complete-failure recovery.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR fixes a grounded-worker gap still present on main; no actionable introduced defect was found.

Priority: P2
Reviewed head: a93d6f2275c14629b0721f274b1fa768cb0e4a50

Before merge

None.

Findings

None.

Agent review details

How this fits together

The experimental local-repair runner sends model actions to a sandboxed grounded worker, which edits source and runs visible checks before returning candidates for independent grading.

flowchart TD
  A[Local benchmark runner] --> B[Grounded worker]
  B --> C[Model action]
  C --> D[Sandbox visible checks]
  D --> E[Complete result continues]
  D --> F[Incomplete result throws]
  E --> G[Candidate grading]
  F --> H[Sandbox cleanup]
Loading

Technical review

Best possible solution:

Land the shared visible-check guard in the grounded worker while preserving repairable complete failures and existing cleanup.

Do we have a high-confidence way to reproduce the issue?

Main consumes passed without checking valid at both grounded visible-test call sites; the supplied watchdog regression exercises the resulting incomplete-run path, but no target code was executed during this read-only review.

Is this the best way to solve the issue?

The two-call-site helper is the smallest complete fix: it uses the checker's existing validity verdict and bypasses recoverable-action handling before existing cleanup.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 1e4104fb47f7.

Provenance checked

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

  • add P2: The defect affects experimental benchmark execution and unnecessary continuation after incomplete tests, with no production routing impact.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

  • P2: The defect affects experimental benchmark execution and unnecessary continuation after incomplete tests, with no production routing impact.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 8:34 AM ET / 13:34 UTC.

@vincentkoc
vincentkoc merged commit 9050206 into main Nov 13, 2026
24 checks passed
@vincentkoc
vincentkoc deleted the fix/fusion-grounded-incomplete-tests-20261113 branch November 13, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant