Skip to content

[PR] fix: execute proxy-owned tools on the non-stream Anthropic path (#5957) - #5966

Merged
ranxianglei merged 2 commits into
masterfrom
2026-11-08_nonstream-anthropic-proxy-loop
Nov 9, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-11-08_nonstream-anthropic-proxy-loop

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Problem

A stream:false Anthropic response that calls a proxy tool (compress, acp_status, search_context, decompress) was rewritten once by rewriteJsonSteps: each proxy tool_use became its result text and stop_reason was forced to end_turn. A compress round therefore ended the client turn with the fold receipt as the turn's only output — the model never saw the fold result, and any follow-up in that turn was lost. The streaming lane has no such gap: its loop re-requests until the model yields text or a real client tool call (core.ts:1531).

This is the Anthropic-lane analogue of #4019 / #4037 (OpenAI chat-completions).

Fix

New src/compress-loop-anthropic.ts (compressLoopAnthropicJson), wired into the relay's non-stream section for prepared.protocol === "anthropic":

  • Per round, extract proxy tool_use blocks from the current upstream response (same isProxyToolFor classification as every other lane).
  • Proxy-only round → execute each call through the SAME executors the one-shot rewriter uses (applyConfiguredCompression for compress, executeAnthropicProxyTool otherwise; raw result text, matching the stream lane's carrier), append the original assistant content plus a tool_result user turn to the forwarded body, and re-request. Retrieval injections ride the re-request after their ack (Feature: content-addressed message store — ID-referenced tool results + acp_retrieve (built-in CCR) #1097 parity).
  • Mixed or tool-free round → return unrewritten; the relay's existing one-shot rewriter chain is untouched and handles these cases exactly as before. Each upstream response is executed by exactly one executor, so no call runs twice.
  • Bounded by MAX_LOOP_ROUNDS; at the limit the last response falls through to the one-shot rewriter (bounded degradation to the pre-fix shape — never an undeclared proxy tool_use handed to the client).

The continuation always ends on the tool_result USER turn (never an assistant prefill, #5605), and the replayed assistant turn carries its content verbatim — signed thinking blocks included (KDD #10). Like #4037 the re-request resends the UNFOLDED body; the fold materializes on the next client request (prefix-cache friendly).

Behavior change (disclosed)

Non-stream Anthropic proxy-only rounds now make one extra upstream request per proxy-call round and deliver the model's real post-compress answer instead of ending the client turn with the receipt + end_turn. Client-visible turn output changes from the receipt text to the final answer. tests/ccr-v2.test.ts ("e2e CCR v2 non-stream") previously pinned the buggy shape and now pins the fixed one.

Tests

  • New tests/issue5957-nonstream-anthropic-proxy-loop.test.ts (8 tests): execution + re-request continuation shape for acp_status / search_context / decompress / compress; negative gates (real client tool_use, plain text, mixed rounds → zero fetches, byte-identical passthrough); two proxy calls in one round → single re-request with both tool_results in order; signed thinking block rides the replayed assistant turn verbatim (KDD release v0.1.3 #10); rejected-compress receipt rides the tool_result while final text still reaches the client.
  • Updated tests/ccr-v2.test.ts non-stream e2e: turn 1 now asserts the client gets the model's answer (no tool_use block), the fold receipt rides the re-request's tool_result, unfolded history is resent on the re-request, the folded view materializes on turn 2 outbound, and the range restore rides the turn-2 re-request.
  • Targeted local: typecheck clean; issue5957 suite 8/8; ccr-v2 16/16; decompress-image-ref-dispatch 8/8; visibility-markers + marker-lane-coverage 17/17. Full-suite verdict comes from CI.

Note: open PR #4037 touches the same relay.ts non-stream region (OpenAI lane) — no textual conflict at this base, but review them together if they merge close in time.

Closes #5957

A stream:false Anthropic response that calls a proxy tool (compress,
acp_status, search_context, decompress) was rewritten once: each
tool_use became its result text and stop_reason was forced to
end_turn, so a compress round ended the client turn with the fold
receipt and the model never saw the result. Port the streaming
decision (core.ts reRequest gate) to this lane: proxy-only rounds
execute through the same executors, append the original assistant turn
plus a tool_result user turn to the forwarded body, and re-request;
mixed or tool-free rounds fall through to the untouched one-shot
rewriter. Bounded by MAX_LOOP_ROUNDS with degradation to the pre-fix
shape at the limit.
@ranxianglei ranxianglei changed the title fix: execute proxy-owned tools on the non-stream Anthropic path (#5957) [PR] fix: execute proxy-owned tools on the non-stream Anthropic path (#5957) Nov 8, 2026
@ranxianglei ranxianglei added bug Something isn't working p1 labels Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-08_nonstream-anthropic-proxy-loop (5386b61)

Latest build — install this exact version

npm install -g billion-context@0.0.8687-pr.5966.ef5a901

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
5386b61 billion-context@0.0.8687-pr.5966.ef5a901 build
e6cb37e billion-context@0.0.8670-pr.5966.08c471a build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5966.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 同家族但机制域不同——本 PR 是 Anthropic wire(stream:false JSON path)的 proxy-tool loop;OpenAI chat-completions 的对位修复在已开的 PR #4037(对应 issue #4019),两条车道的协议形状、rewriter、executors 都不同,按 AGENTS.md「同症状不同机制域」各自成单成立。镜像内 open [PR] 行中指向本 thread 的只有本 PR 一条,无并行第二 PR。

开始审核:分支相对当前 master 新鲜(1 commit ahead / 0 behind,base = master tip 82e2d53c)。CI 镜像缓存暂无 check 记录(套件可能仍在跑),稍后按规则读缓存确认。正在通读 diff(新模块 src/compress-loop-anthropic.ts + relay 接线 + 测试),随后给出八问结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review of PR #5966 — verdict: mergeable, human tier (client-visible message-shape change on the Anthropic non-stream lane → auto-merge gate §7.4 "must stay human"). Dedup conclusion was posted in floor 2: not a duplicate; the OpenAI-lane analogue is open PR #4037 (issue #4019), different wire protocol → separate thread justified.

What I verified independently (head e6cb37e, now 5386b61)

  • Full code read of src/compress-loop-anthropic.ts + the relay wiring (src/server/relay.ts:2397-2411) + the executor chain in src/stream.ts. The three-way gate mirrors the streaming lane exactly (core.ts:1531: re-request iff proxy results && no real tool calls); the executors are the SAME functions the one-shot rewriter uses (applyConfiguredCompression / executeAnthropicProxyTool), all under withSessionLock; each upstream response is executed by exactly one executor (at the round limit the response breaks out UNEXECUTED for the one-shot rewriter — bounded degradation to the pre-fix shape, never double execution).
  • KDD release v0.1.3 #10: replayed assistant turn is structuredClone(raw content) — signed thinking blocks ride verbatim (pinned by deepEqual in the suite). [PR] Failed-compress recovery: end the re-request on a user turn, not an assistant prefill (fixes upstream 400) #5605: continuation always ends on the tool_result USER turn, never an assistant prefill. Prefix cache: re-request resends the unfolded egress body + appended tail turns; cc anchors survive (ccr-v2 e2e asserts across all 4 captured requests).
  • Local targeted runs at head: issue5957 suite 8/8, ccr-v2 e2e 16/16 (real HTTP relay: turn 1 delivers the model's answer with no tool_use leak, receipt rides the re-request's tool_result, unfolded history resent, folded view materializes turn 2, range restore rides the turn-2 re-request without duplicating the head anchor).
  • CI: mirror cache shows 30 checks on e6cb37e = 14 success / 16 skipped / 0 failed. Master advanced 6 commits during review (CI workflow + storage-test pins + a docs comment only — zero file overlap with this PR), so the "behind" state is cosmetic; no rebase required.

Findings

  1. Limit-gate test was missing — fixed directly (pushed). Gate Test Discipline (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617) requires both directions of every gate; the MAX_LOOP_ROUNDS degradation path had no row. Added test(#5957): pin the MAX_LOOP_ROUNDS degradation gate… → commit 5386b61 on the PR branch: scripts N consecutive proxy-only responses, asserts exactly N−1 re-requests and that the limit-round response stays proxy-only/unexecuted (one-shot rewriter performs the single execution). Verified locally: 9/9 + typecheck clean. CI re-runs on the new head.
  2. Error path parity (non-blocking, noted): a loop throw (executor exception or exhausted re-request retries) falls into the relay catch → raw passthrough of the upstream response, which can contain an undeclared proxy tool_use. This exposure is IDENTICAL in the merged Responses-lane loop and the pre-existing one-shot rewriter; the only new surface is re-request network failure. Consistent with house precedent — not blocking.
  3. Multi-round usage settlement (non-blocking, cross-lane observation): non-stream JSON loops settle ONLY the final round's usage (relay reads json.usage once after the loop); intermediate rounds — round 1 carries the largest unfolded input — are undercounted in stats/cache ledger (cache ledger: non-streaming rewriter path never records usage samples (ledger blind for stream:false sessions) #1547). The merged Responses lane and open sister [PR] fix: execute proxy-owned tools on the non-stream OpenAI path (#4019) #4037 have the identical gap; the streaming lane settles per SSE round. One mechanism spanning three lanes → recommend ONE follow-up item covering all three when you decide to track it (parked here per policy rather than filed).

Existing-behavior inventory (regression pass)

  • Terminal rounds (no proxy tools): byte-identical passthrough — pinned by negative gates (deepEqual(out, initial), zero fetches).
  • Mixed rounds (proxy + real tool): unchanged — deferred to the one-shot rewriter as before.
  • Non-Anthropic protocols: untouched (protocol guard before the loop). Streaming path: untouched.
  • Single-round usage settlement: unchanged. Capture/save flow (captureHandoffJson → reply.save() → res.end): unchanged.
  • High-risk patterns walked: no fail-fast→swallow change (errors still throw into the same catch as before); no threshold/timing drift; output-format change = the disclosed fix itself; error receipts unchanged ([Compression FAILED: …] still produced by applyRanges).

Behavior-change disclosure in the PR body is complete (old→new + reason + the ccr-v2 pin flip). Coordination note: if #4037 lands close in time, review them together — same mechanism family, no textual conflict at this base.

Recommendation: MERGE (after your approval; my test commit 5386b61 is already on the branch, CI will confirm on the new head).


结构化八问(中文,供 reviewer)

  1. 是什么问题:Anthropic stream:false 车道上,模型调用代理工具(compress/acp_status/search_context/decompress)时,一次性改写器把 tool_use 替换成回执文本并强制 stop_reason=end_turn——客户端回合以回执结束,模型永远看不到压缩结果,该回合内的后续内容全部丢失。
  2. 什么原因导致的:非流式 JSON 车道缺少流式车道的重请求循环(core.ts:1531:有代理结果且无真实工具调用就重请求);一次性改写器只为终局回合设计,遇到纯代理回合只能把回合终结掉。
  3. 影响范围与严重程度:影响走 Anthropic 非流式(stream:false)且使用代理工具的客户端——compress 调用以回执+end_turn 收场,模型丢失折叠结果与回合延续。属特定条件下的功能缺陷,非全面不可用。
  4. 怎么修复的:新增 src/compress-loop-anthropic.ts,与流式车道同一三分支门:纯代理回合→同一执行器执行(withSessionLock)+ 追加 assistant/tool_result 续轮 + 重请求(上限 MAX_LOOP_ROUNDS=10);混合/无工具回合→原样交回既有的一次性改写器(与修复前字节一致);到达上限→不执行、降级回修复前形态。每个上游响应只被一个执行器执行一次。
  5. 是否有回归问题:无。既有行为逐项清点:终局回合字节一致(负向门 deepEqual 钉死)、混合回合不变、非 Anthropic 协议与流式路径未触碰、单回合用量结算不变、错误路径仍抛入同一个 catch(与已合并的 Responses 车道同构);KDD#10 签名 thinking 块逐字回放(测试钉死)、[PR] Failed-compress recovery: end the re-request on a user turn, not an assistant prefill (fixes upstream 400) #5605 续轮以 USER 收尾、前缀缓存稳定(ccr-v2 e2e 四个请求全程断言 cc anchor)。
  6. 是否破坏缓存:否。重请求重发未折叠正文(前缀稳定),折叠在下一个客户端请求物化——与流式车道及 [PR] fix: execute proxy-owned tools on the non-stream OpenAI path (#4019) #4037 同策略。观察项(非回归):多回合只结算最后一轮 usage,中间轮次(第 1 轮输入最大)在统计/缓存账本中漏计;已合并的 Responses 车道与在途 [PR] fix: execute proxy-owned tools on the non-stream OpenAI path (#4019) #4037 存在同样缺口,建议日后按一个跨车道事项统一处理。
  7. 是否改了设计方向:否。本 PR 是把非流式 Anthropic 车道对齐到既有的流式车道设计(同一门、同一执行器、同一载体纪律),并镜像已合并的 Responses 车道先例。
  8. 是否引入了新的配置:无新 env/config/默认值。

判定:可合入(需人工批准——客户端可见消息形状变更属自动合并门禁的人工档);reviewer 已在 PR 分支补上上限门测试提交 5386b61(本地 9/9 + typecheck 干净,CI 将在新 head 复验)。

@luan-cestari-ppro

Copy link
Copy Markdown
Contributor

Field check of #5966 (head e6cb37e) on Bedrock with Claude Code. I used it in place of our local loop for #5957.

Setup

Results (6 runs each)

Build Upstream 400 / req stream:false req Model folds followed Preflight folds Runs with a non-empty diff, no receipt-only end Extra upstream calls per proxy-only round
master + 3 PRs, no fix 0/2 1 0/1 0 0/1 (turn ended on the receipt) -
our local loop 0/166 108 5/5 33 6/6 1.0
this PR 0/181 125 6/6 35 6/6 1.0
  • 5 of 6 model folds came on a stream:false request and went through compressLoopAnthropicJson (proxy-tool round 1 log line). One came on a streaming request and the stream lane handled it.
  • No compress-loop-limit, no loop upstream error, no receipt text sent to the client, no non-200 status.
  • The re-request ends on a user tool_result turn in every case. It resends the unfolded body (51-55K tokens on the 55K window), with about 93% cache read and 4.8-5.1 s latency. The fold takes effect on the next client request. This is the same as our loop.
  • Cache hit and tokens saved are in the same range as our loop (smoke runs: mean 52K vs 46K tokens saved, cache hit 0.735 vs 0.725).
  • Regression cells (Claude Code e2e, with and without Headroom, and body replay): Haiku 0/427 and Opus 0/63 upstream 400s, no client-visible errors. Folds, savings and cache hit stay in the range of our previous build.

Problems found: none in this PR. The runs had no thinking blocks in the replayed assistant turn, so the live check does not cover the signed-thinking replay. Your test covers it.

One note for others who reproduce this with Headroom: if the Headroom process has no network and no tiktoken cache, it skips compression and adds no CCR markers. Then it keeps the streaming path and the non-stream lane never runs. Set TIKTOKEN_CACHE_DIR to a seeded cache.

Thanks for the port. From our side it can merge.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status update (the full review with the 8-question checklist is in my previous comment):

@ranxianglei
ranxianglei merged commit ba0ae45 into master Nov 9, 2026
52 checks passed
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1112-master.ba0ae45 (rolling master tag).

npm install -g billion-context@0.0.1112-master.ba0ae45  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Anthropic stream:false: a model compress call ends the client turn (receipt returned with end_turn, no re-request)

2 participants