Skip to content

fix(fusion): reject incomplete adviser drafts - #1036

Merged
vincentkoc merged 2 commits into
mainfrom
fix/fusion-complete-adviser-responses-20261114
Nov 13, 2026
Merged

vincentkoc merged 2 commits into
mainfrom
fix/fusion-complete-adviser-responses-20261114

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Fusion accepted text from truncated, filtered, or tool-calling adviser responses as draft evidence. Require the first Chat choice to finish with stop and contain no refusal or tool call before including its text in synthesis. Fully consume each response so rejected advice retains normal usage accounting; completed advisers and aggregator-only fallback still work.

Validation:

  • Worker TypeScript check and 54 focused collector, routing, readiness, and accounting tests pass in an isolated image with the exact frozen dependency inputs.
  • All 11 rejection regressions fail on the original runtime and pass after this fix.
  • Independent P0–P2 review: no findings.

Deployment: not deployed. Nonconforming upstreams that omit finish_reason now count as failed advisers. This is a correctness fix, not a benchmark claim of better answer quality.

Follow-up validation: corrected the local Worker smoke fixture to return finish_reason: "stop". The end-to-end local Worker smoke passes with Docker-backed containers disabled in the disposable validation environment; hosted CI checks the full configuration.

@clawsweeper

clawsweeper Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vincentkoc
vincentkoc marked this pull request as ready for review November 13, 2026 16:18
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Nov 13, 2026
@clawsweeper

clawsweeper Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

Require completed, non-refused, non-tool-calling adviser responses before including their text in Fusion synthesis.

Example: local/rejected returns "discarded draft" with finish_reason="length" and usage totaling 10 tokens.

  • Before: Fusion includes the truncated draft in the synthesizer request.
  • After: Fusion omits the draft, records the failed adviser, and retains its normal usage accounting.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The patch is focused and source-correct, but the required real-behavior proof gate remains unmet.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A final answer can be influenced by adviser output that was truncated, filtered, refused, or attempting a tool call.
Reason: The maintainer-authored direction repairs a concrete response-contract gap with a small change and explicitly accepts omission of nonconforming advisers.

Merge readiness

✅ Ready for maintainer review

Keep open: current main still accepts incomplete adviser text, and this maintainer-authored PR provides a focused correction with no introduced code defect found.

Priority: P2
Reviewed head: 4fd1d16c15e14680e3279831ae351fa4a4a2d823

Before merge

None.

Findings

None.

Tests

  • Missing end-to-end proof: Provide redacted Worker HTTP traces showing rejected adviser text excluded, correct fallback or surviving advice, and retained usage, including an upgrade from v0.5.0 with existing Fusion settings intact.
Agent review details

How this fits together

Fusion receives an explicitly selected chat request, invokes bounded advisers through normal proxy controls, and passes accepted drafts to the configured synthesizer.

flowchart LR
  A[Chat request] --> B[Fusion dispatch]
  B --> C[Adviser proxy calls]
  C --> D[Bounded response reader]
  D --> E[Completion validation]
  E --> F[Synthesizer]
  C --> G[Usage accounting]
  F --> G
  F --> H[Client response]
Loading

Technical review

Best possible solution:

Keep completion validation in the existing parser after full bounded response consumption, preserving fail-open synthesis and normal accounting.

Do we have a high-confidence way to reproduce the issue?

Current-main source directly accepts nonempty first-choice text with finish_reason="length"; the submitted regressions exercise that input, but this review executed no target code.

Is this the best way to solve the issue?

This is the smallest complete correction for the stated completion-metadata defect and preserves the existing response reader, proposal assembly, and accounting owners.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against ee1626510f59.

Provenance checked

  • Adviser text extraction changes intended behavior with a stated reason (720c458: The original Fusion implementation accepted bounded OpenAI-compatible adviser text as untrusted evidence and continued when advisers failed.)
  • Discarded-response accounting keeps the original intent (3175974: Discarded responses must finish consumption or cancellation so accounting can complete without hanging reservations.)
  • Existing positive-response fixtures keeps the original intent (e3be150: The fixtures exercise request-parameter qualification; adding valid completion metadata preserves their original purpose under the stricter parser.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Likely related people:

  • Vincent Koc: Raw commit e3be150 adds worker/fusion.ts:91 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: e3be15000846; files: worker/fusion.ts)
  • Peter Steinberger: Raw commit 720c458 adds worker/fusion.ts:220 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 720c45881b6d; files: worker/fusion.ts)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Review metrics

Metric Value Why it matters
Runtime scope Production +8/-2 lines The correction fits the existing parser without adding modules, state, or dispatch paths.

Root-cause cluster

Relationship: canonical
Canonical: #1036
Summary: This PR owns production adviser completion validation; its dependent benchmark changes do not replace that repair.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Labels

Label changes:

  • add P2: Incomplete optional adviser evidence can reach synthesis, while the final-model workflow remains available.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

  • P2: Incomplete optional adviser evidence can reach synthesis, while the final-model workflow remains available.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 11:45 AM ET / 16:45 UTC.

@vincentkoc
vincentkoc merged commit 4f806f6 into main Nov 13, 2026
57 of 58 checks passed
@vincentkoc
vincentkoc deleted the fix/fusion-complete-adviser-responses-20261114 branch November 13, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant