Skip to content

[PR] fix(responses-wire): tool_search pair guard + hosted web_search_call run integrity (#5670, #7296) - #7372

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-11-20_responses-tool-ordering-root-cause
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-11-20_responses-tool-ordering-root-cause

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Root-cause fix for the responses-tool-ordering batch (aggregation #7354): the two Responses-wire ordering defects in the same failure domain — the wire-boundary tool-pair integrity pass not knowing a tool-item family, and the assistant-run repair misclassifying flattened hosted calls — fixed in one PR per the root-cause aggregation contract.

Both fixes are cherry-picked with -x from the member PRs (authorship preserved):

#5670 — tool_search_call / tool_search_output unknown to the tool-pair pass → upstream 400

Cherry-picked from #5675 (commits 742d992, d61520e).

Codex 0.166 emits tool_search_call / tool_search_output pairs that src/tool-pair-order.ts did not know, so the #766 hoist guarantee never applied to them; a compression round that dropped a call while keeping its output produced an unshippable request that strict backends (DeepSeek) reject with No tool call found for tool output <id> on every later turn, and the session is stuck on that model until restart.

  1. Pair registration — tool_search_call/tool_search_output added to CALL_TYPES/OUTPUT_TYPES (src/tool-pair-order.ts:30-31).
  2. Fail-closed orphan drop (dropOrphanToolOutputs) — an output whose call_id matches no call in the outbound array is dropped and logged; an output with a missing/empty call_id is kept (unjudgeable — dropping would lose data we cannot prove orphaned).
  3. Fail-closed orphan completion (completeOrphanToolCalls) — the mirror direction: a call before the trailing run whose output was dropped was abandoned mid-history by compression; its pairing is completed with an honest family-matched synthesized result (function_call_output/custom_tool_call_output with tool output lost during context compression, or tool_search_output {status: "completed", tools: [], execution: "client"} — field set matches real Codex traffic, upstream-verified in the tool_search_call / tool_search_output missing from tool-pair ordering -> upstream 400 "No tool call found for tool output" (Codex 0.166 + DeepSeek) #5670 thread), so the model sees the loss instead of re-issuing the call blind. The trailing run (calls + post-call reasoning only) is exempt: it is where an in-flight batch legitimately sits.
  4. Wire-constraint ledger ([CI/testing] wire-contract anti-regression system: validation-parity fake upstreams + golden schema snapshots + constraint ledger + staged 4-protocol x 3-mode client smoke matrix #1304) — WC-026 (orphan output) + WC-027 (orphan call) with validator clauses in the responses fake upstream and self-check probes.

All three guarantees are no-ops (same array reference, byte-stable) on healthy input; all four wire call sites (src/server/prepare-responses.ts:466/:855, src/compress-loop-responses.ts:219, src/loop/adapter-responses.ts:332) inherit them with zero changes.

#7296 — ordering repair split a reasoning item from its hosted web_search_call

Cherry-picked from #7302 (commit 2a2a9e9).

responsesToCore (kernel) flattens every hosted-call item — web_search_call, file_search_call, mcp_call, computer_call/_output, image_generation_call, code_interpreter_call — into a core with contentType: "reasoning" (the raw item rides the rawResponsesItem sidecar). repairResponsesAssistantOrdering (src/server/responses-ordering.ts) classified that core as a second reasoning and its run-boundary rule then inserted a synthetic user separator between the real reasoning and the web_search_call; the Responses server rejects a web_search_call separated from its reasoning item ("provided without its required 'reasoning' item"). Fix: a contentType: "reasoning" core whose sidecar carries a hosted-call wire type is now classified as tool-call (phase 2), keeping reasoning → hosted call in one run; only the sidecar's wire type decides — never the core text or id, which the codec re-derives and a client could shape. Separators still land only between distinct runs.

Behavior changes (old → new)

Scenario Old New
tool_search pair with non-tool items wedged between call and output passed through unchanged hoisted before the earliest call (same as the other pair families)
Orphan tool output (any family) — call absent from the outbound array shipped → strict backend 400s every later turn dropped + warn log
Mid-history tool call whose output was dropped shipped → strict backend 400s every later turn synthesized honest lost-output result + warn log
user → reasoning → web_search_call after a hosted search synthetic user separator inserted between reasoning and the call → server rejects the call no separator; both stay in the same assistant run
Healthy requests (all pairs intact; no hosted call inside a reasoning run) byte-stable unchanged (same array reference returned)

Model-facing text (new, recovery-only)

The synthesized lost-output result text tool output lost during context compression (and the empty tool_search_output shape) is new text the model can read — but only in requests that were previously rejected wholesale (400), i.e. a state with no prior model-visible behavior. No existing model-facing string was changed.

Overlap note (for merge sequencing)

Verification

  • Targeted suites green locally on this head: tool-pair-order + wire-contract + issue564-responses-ordering + responses-tool-images (90/90); issue2627-turn-sep-coverage + issue5139-turn-separator-ranges + invariant-judge + fold-round2-shape + codex-responses-ws (73/73); typecheck clean.
  • [Bug]: Codex Responses ordering inserts a user separator before a hosted web search call #7296 minimal repro (from the issue) verified against this head: user → reasoning → web_search_call now renders with 0 separators (was 1).
  • Full regression: remote CI on the pushed head.

Fixes #5670
Fixes #7296

ci-check added 3 commits November 20, 2026 15:07
Rebase the #5670 branch onto master (6453fee, v0.1.216) and re-land the full fix as one coherent change:

- src/tool-pair-order.ts: register tool_search_call/tool_search_output in the pair sets; dropOrphanToolOutputs drops outputs whose call_id matches no call in the array; completeOrphanToolCalls synthesizes an honest lost-output result for mid-history calls whose output was dropped (trailing-run exemption keeps in-flight batches untouched); both run inside hoistTrappedToolItems, so all four wire-boundary call sites are covered without call-site changes.
- tests/tool-pair-order.test.ts: 26 cases covering both directions, trailing-run negatives, family shapes (tool_search synth carries status:"completed" per real Codex traffic), and same-reference identity on healthy input.
- tests/wire-contract-fakes.ts / tests/wire-contract.test.ts: ledger entries + validator clauses + probes for both directions. The rules landed as WC-023/WC-024 on the feature branch; master's #5432, #5998 and #6636 took WC-023/WC-024/WC-025 first, so they renumber to WC-026 (orphan output) / WC-027 (orphan call) on this rebase.
- src/server.ts: drop the unused hoistTrappedToolItems import.
- tests/responses-tool-images.test.ts: rebuild the adapter assertion on a well-formed tool pair (master's c31fd99 overlap-sweep rewrite of hoistTrappedToolItems is kept; the two new stages run before it on pair-clean input).

Supersedes the pre-rebase commits (7772d05, e224118, c12aca9, 93fe27c) which are no longer ancestors of this branch.

Closes #5670

(cherry picked from commit 742d992)
2026-11-19, Codex Desktop behind a Responses gateway (provider unstated): HTTP 400 'No tool call found for tool search output with call_id <redacted>' — the same constraint with the item type name interpolated into the message. Corroborates the shape; the historical 400 was not captured at its exact failing boundary, so this does not verify this branch or locate the historical loss.

(cherry picked from commit d61520e)
)

responsesToCore flattens hosted calls (web_search_call, file_search_call, mcp_call, ...) into contentType:"reasoning" cores. The Responses ordering repair classified them as a second reasoning item and inserted a user separator between the real reasoning and the hosted call, splitting the assistant run — the server then rejects the web_search_call "without its required 'reasoning' item".

Classify a flattened hosted-call core (whose rawResponsesItem sidecar carries a hosted-call wire type) as tool-call (phase 2) so the reasoning -> hosted call run stays intact. The separator still lands between distinct runs.

Closes #7296

(cherry picked from commit 2a2a9e9)
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Reviewed PR #7372 (root-cause aggregation for the responses-tool-ordering batch, aggregation #7354).

Overlap/duplicate screening. This PR cherry-picks (with -x, authorship preserved) from #5675 (member for #5670) and #7302 (member for #7296) — both become redundant once this merges (identical commits → empty diffs). #7322 (codex-lane root-cause, #4594 + #7296) carries the SAME #7296 commit, so whichever of #7372 and #7322 merges first makes the other's #7296 hunks no-ops. All four are open and mergeable. Merge sequencing is an owner decision — I flag it, don't act on it.

Verification (local, targeted).

  • Fresh base: 0 commits behind master (merge-base = origin/master @ 1e357bf).
  • Targeted suites green: tool-pair-order + wire-contract + issue564-responses-ordering + responses-tool-images (90/90); issue2627-turn-sep-coverage + issue5139-turn-separator-ranges + invariant-judge + fold-round2-shape + codex-responses-ws (73/73).
  • Typecheck clean (tsc --noEmit, exit 0).
  • [Bug]: Codex Responses ordering inserts a user separator before a hosted web search call #7296 minimal repro: user → reasoning → web_search_call renders with 0 separators (was 1) — confirmed by the passing test.
  • Full regression: remote CI still running (30 checks, 0 failed, 24 pending) — I'll follow up when it lands.

Code review findings.

  • tool_search_call / tool_search_output missing from tool-pair ordering -> upstream 400 "No tool call found for tool output" (Codex 0.166 + DeepSeek) #5670 registers tool_search_call/tool_search_output in the pair sets; dropOrphanToolOutputs drops outputs whose call_id matches no call (fail-closed); completeOrphanToolCalls synthesizes an honest lost-output result for a mid-history call whose output was dropped (trailing-run exemption keeps in-flight batches untouched). Both run inside hoistTrappedToolItems, so all four wire-boundary call sites inherit them with zero changes; all three guarantees are no-ops (same array reference) on healthy input. I traced the drop→complete ordering: the two cases are disjoint (a dropped output's call_id matches no call, so it can't create a new orphan call), so the composition is sound.
  • [Bug]: Codex Responses ordering inserts a user separator before a hosted web search call #7296 — a contentType:"reasoning" core whose rawResponsesItem sidecar carries a hosted-call wire type is now classified as tool-call (phase 2). I verified against the kernel source (kernel/src/wire/responses.ts:545-563) that the 7 types in RESPONSES_HOSTED_CALL_TYPES match the kernel's flattening exactly, and that rawResponsesItem is the verbatim wire item — so the sidecar's type is the wire type, not re-derived. Real reasoning cores (rawResponsesItem.type === "reasoning") and reasoning cores with no sidecar are unaffected.
  • Wire-contract ledger — WC-026 (orphan output) + WC-027 (orphan call) with validator clauses that mirror the repair logic exactly; the feature-branch renumbering (WC-023/024 → WC-026/027) is correctly documented and the ledger never shrinks.
  • src/server.ts — the removed hoistTrappedToolItems import was already dead in the base (no usage) — a clean-up, not a regression.
  • Minor (non-blocker): the synthesized text tool output lost during context compression is new model-facing text (recovery-only, no existing string changed — disclosed in the PR body's "Model-facing text" section), but the ⚠️ MODEL-FACING TEXT CHANGE top-of-body/commit marker required by AGENTS.md §7.3 is absent. Substance is disclosed; the marker is a process formality.

结构化八问

  1. 是什么问题: Responses 线同域两个排序缺陷——(a) tool_search_call/output 对 wire 边界的 tool-pair 完整性通道不可见,压缩丢 call 留 output 时严格后端(DeepSeek)每轮 400 No tool call found for tool output <id>,会话卡到重启;(b) 排序修复把 kernel 摊平的 hosted web_search_call 误判成第二个 reasoning,在 reasoning 与 hosted call 之间插 user 分隔符,服务端拒收该 call(缺它的 reasoning item)。
  2. 什么原因导致的: 根因在 wire 边界/分类逻辑,不是表面现象。(a) CALL_TYPES/OUTPUT_TYPES 集合缺 tool_search 家族,且无孤立项 fail-closed;(b) repairResponsesAssistantOrdering 只看 contentType、不看 sidecar 的 wire type,把摊平 hosted call 当 reasoning。
  3. 影响范围与严重程度: 触发条件明确(Codex 0.166 tool_search 对 + 压缩丢 call/output,或 hosted search 后的 reasoning → hosted call)。严重度:特定条件下会话在该模型上卡死(400 每轮重放),重启可恢复——条件性可用性缺陷,p2 合理。
  4. 怎么修复的: 在正确的层修根因,不做 payload 过滤。(a) 注册 tool_search 家族 + 双向 fail-closed(孤立 output 丢弃、mid-history 孤立 call 合成诚实 lost-output,trailing run 豁免 in-flight);(b) 用 sidecar 的 wire type(非 core text/id)把摊平 hosted call 归到 tool-call。
  5. 是否有回归问题: 无。健康输入三保证均 no-op(同数组引用、字节稳定);responses-ordering 改动只影响摊平 hosted call(真 reasoning 不受影响);server.ts 删的是死 import;四个调用点零改动继承。定向 90/90 + 73/73 绿。
  6. 是否破坏缓存: 不破坏。健康输入返回同数组引用,前缀缓存字节稳定。
  7. 是否改了设计方向: 没有。在既有 wire-boundary 完整性通道框架内加家族注册 + fail-closed,分类逻辑加 sidecar type 判定,不转向。
  8. 是否引入了新的配置: 没有。无新 env/config/默认值。

判定: 代码审核正面——修在正确的层(wire 边界/分类逻辑,非 payload 掩码)、无回归、测试有牙、diff 干净。等 CI 绿给全量回归判定;合并顺序(#7372 vs #7322 的 #7296 重叠)请 owner 拍板。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI status update: still fully queued after ~15 min (24 queued, 0 in_progress, 0 failed) — no runners are picking up the test shards yet. This is a fleet runner-availability issue, not a code issue; the PR head is unchanged. My code-review verdict (positive, see prior comment) stands; the full-regression verdict is blocked on CI landing. I'll keep monitoring and report the moment the test/gate shards finish.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI landed — full regression green: 30/30 checks complete, 14 success + 16 skipped, 0 failed. All test shards (ubuntu/windows × node 22/24 × shards 1-5) and both static-gates lanes pass; the skipped ones are the gated zero-token e2e lanes.

Final verdict: mergeable (code + CI). Recap of the review:

Remaining owner decision (not a code issue): merge sequencing — #7372 and #7322 both carry the #7296 commit; whichever merges first makes the other's #7296 hunks no-ops, and #5675/#7302 go empty once #7372 lands. I can't merge PRs (human-only) — please sequence and merge when ready.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant