[vincentkoc]

Local Fusion benchmark responses previously collapsed model, completion, usage and stop-reason failures into invalid_local_response, discarding the cause from the checkpoint. Record a fixed, content-free responseFailure code on the failed call, including malformed or unreadable response bodies.

The campaign still stops after that call, leaves rejected usage unknown, and does not grade or try another model. No raw response, provider-supplied reason, or parser exception is retained.

Verification: all 13 focused local-repository tests pass, including nine rejection cases and sentinel checks. Related local/agent/execution tests: 42 passed, 33 gated skips. The new regression failed before the fix.

Deployment: none; benchmark tooling only. Remaining risk: this identifies a rejection category, not the missing metadata of older runs, and does not establish any Fusion quality gain.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

This PR records fixed, content-free rejection categories in failed local Fusion benchmark calls.

Example: A local response reports eval_count: -1.

• Before: The checkpoint records invalid_local_response without identifying the rejected field category.
• After: The failed call additionally records responseFailure: invalid_usage, with token usage still unknown.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Focused, source-consistent repair with meaningful regression coverage; validation evidence is harness-based and was not independently executed.
Proof confidence	🦐 gold shrimp (3/6)	This offline benchmark diagnostic change is outside the shipped-entry-point proof requirement; supplied validation consists of reported focused tests and an inspected campaign harness.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Benchmark operators cannot distinguish why a rejected local response stopped a campaign.
Reason: Maintainer-authored direction delivers useful diagnostics with fixed categories and no raw response retention.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR fixes a diagnostic gap still present on main; source review found no actionable defects.

Priority: P2
Reviewed head: 51978f593339f800438bb85013e26ef99b81c096

Before merge

None.

Findings

None.

Agent review details

How this fits together

The offline repository benchmark sends bounded repair requests to pinned local models and writes content-free campaign checkpoints; it does not change Worker routing.

flowchart LR
  A[Benchmark CLI] --> B[Repository campaign]
  B --> C[Local response adapter]
  C --> D[Local model endpoint]
  D --> C
  C --> E[Validated result or fixed rejection]
  E --> B
  B --> F[Content free checkpoint]

Loading

Technical review

Best possible solution:

Retain the bounded rejection categories in the existing adapter and checkpoint owner while preserving stop behavior and unknown rejected usage.

Do we have a high-confidence way to reproduce the issue?

Main source deterministically discards the rejection category for a response with eval_count: -1; the added test exercises that campaign checkpoint path with injected responses.

Is this the best way to solve the issue?

The change is focused and preserves the existing acceptance checks, error identity, safety handling and campaign control flow.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 318a04d9a47a.

Provenance checked

• Local response validation and failure checkpointing keeps the original intent (00ed689: The original local benchmark preserved content-free usage records, pinned attribution and explicit campaign failure; grounded profiles later retained those constraints.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

• Pinned change: The introduced diff separates existing validation failures into fixed categories and copies only trusted error-associated metadata into the failed call. (scripts/fusion-local-repository.mjs:71, 51978f593339)
• Main still lacks the diagnostic: Main combines model, completion, content, usage and stop-reason validation into invalid_local_response and does not persist a responseFailure category. (scripts/fusion-local-repository.mjs:66, 318a04d9a47a)
• Focused regression coverage: The added campaign test covers nine rejection categories, unknown usage, no grading or subsequent call, and exclusion of private sentinels. The body reports a before-fix failure and passing focused tests; no tests were executed during this read-only review. (test/fusion-local-repository.test.mjs:139, 51978f593339)
• Original reporting contract: The original local benchmark work requires content-free records, exact attribution and campaign stops on infrastructure failures; this PR preserves that intent. (docs/fusion-small-models.md:260, 00ed68970662)
• Discussion and ownership: The full PR body and both comments were inspected; comments contain review-status notices only. GitHub confirms MEMBER authorship, which independently prevents cleanup closure. (51978f593339)
• Feature history: Scoped history identifies Vincent Koc across the original local runner and subsequent grounded, reproducer and qualification changes. Follow history reaches the creation boundary and then encounters an unavailable parent object; supplied provenance and local commit metadata cover the changed behavior. (scripts/fusion-local-repository.mjs:120, 318a04d9a47a)

Likely related people:

• Vincent Koc: Raw commit 00ed689 adds scripts/fusion-local-repository.mjs:98 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 00ed68970662; files: scripts/fusion-local-repository.mjs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

• add P2: Useful diagnostic repair with impact limited to offline local benchmark failures.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P2: Useful diagnostic repair with impact limited to offline local benchmark failures.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 10:52 AM ET / 15:52 UTC.