Skip to content

[PR] fix(anthropic): never send an outbound body that ends on an assistant turn (#5604, preflight-fold tail) - #5978

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
luan-cestari-ppro:fix/anthropic-egress-no-trailing-assistant
Nov 9, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
luan-cestari-ppro:fix/anthropic-egress-no-trailing-assistant

Conversation

@luan-cestari-ppro

Copy link
Copy Markdown
Contributor

Summary

A preflight fold can leave the outbound Anthropic messages array ending on an assistant turn. Anthropic and Bedrock reject that body as assistant prefill:

  • This model does not support assistant message prefill. The conversation must end with a user message.
  • final assistant content cannot end with trailing whitespace (same shape, when the surviving assistant text ends in whitespace)

This PR is the smaller, targeted variant named in the #5891 / #5605 discussion: one guard local to prepareAnthropic. It is the second mechanism of the #5604 symptom family (the first one is the failed-compress re-request that #5605 fixes). It closes nothing by itself. It does not touch #5605.

Root cause

  1. The client body ends on a user tool_result (for example a large Read result).
  2. The payload is over the window. Preflight relaxes the soft-protected recent zone to 0 (relaxedConfig, src/preflight.ts:1554, used at src/preflight.ts:2318, :2359 and :3125). The same happens without the relax when the tail tool is in neverPreserveRecentTools.
  3. The fold takes the trailing tool_use / tool_result pair. The text part of the last assistant message survives, so the rebuilt array ends on that assistant message.
  4. prepareAnthropic (src/server/prepare-anthropic.ts) forwards the rebuilt array as is. The upstream rejects it, and the client retries the same body until it gives up.

Line numbers are on master 848f59b99.

The change

src/server/prepare-anthropic.ts, after the rebuild and before dedupeAdjacentOutputConfig:

  • New exported helper ensureUserTail(outbound, clientMessages).
  • Condition: the last non-system outbound message is assistant, and the last non-system client message is not assistant.
  • Action: insert one thin ephemeral user message, [bili] Earlier context was compressed above. Continue the task., with insertBeforeTrailingSystemRun. The nudge goes before the trailing system run and after all cache-stable content, so no cached prefix moves.
  • The nudge is never stored in the session and never shows up in the client's re-sent history.

Notes on the shape:

Tests

New tests/issue5604-anthropic-egress-user-tail.test.ts:

  1. e2e: a fresh session with a trailing large Read pair over a 24K window. Preflight folds the trailing pair. The test checks that the forwarded body ends on a user turn and that the last message is the nudge. Without the guard it fails: the forwarded body ends on the surviving assistant text, which is the exact upstream-rejected shape.
  2. e2e, same session, next client turn: every message before the surviving assistant turn is re-sent unchanged, and that turn keeps its blocks (the client reply only extends it). The nudge is not replayed.
  3. Unit: an assistant tail gets one nudge appended. Every earlier message is the same object.
  4. Unit: the nudge goes before a trailing system run.
  5. Unit: a user tail, a user tail with a trailing system run, and an all-system array return the same array reference (no change, no prefix churn).
  6. Unit: a client-authored assistant tail passes through unchanged.

Results on this branch (848f59b99 + this commit):

Bedrock measurements

Chain: client -> bili -> LiteLLM -> Bedrock (eu-central-1). Builds: "before" = master 848f59b99 + #4898 + #5441 + #5605 + #5966. "after" = "before" + this PR. #5966 is in both builds so that agentic sessions continue after a model fold.

Direct reproducer (the e2e test shape against real Bedrock): a Claude Code-like body with two small Read pairs and a 240K-char trailing Read result, 55K-token window, fresh session per request, 3 requests per model.

Build Model Client result Outbound bodies ending on assistant
before Haiku 3/3 HTTP 400 does not support assistant message prefill 3
before Opus 3/3 HTTP 400 does not support assistant message prefill 3
after Haiku 3/3 HTTP 200 0 (nudge on 3)
after Opus 3/3 HTTP 200 0 (nudge on 3)

Agentic runs (an agentic Java coding task, Claude Code -> Headroom -> bili -> LiteLLM -> Bedrock, Opus, 55K window, 3 runs per build): 0 upstream 400s in both builds (98 requests before, 90 after). No outbound body ended on an assistant turn, and the guard never fired. Every run finished with a non-empty diff. Earlier, the same task on master + #4898 + #5441 + #5605 + a bug-4 loop hit this error in 1 of 3 runs (4 of 21 requests, and the session failed). So the residual is rare after #4898, but it is real and fatal when it happens.

Haiku regression cells (Claude Code -> bili -> LiteLLM -> Bedrock, 4 sessions each, before / after): small 0/84 and 0/108 upstream 400s, g2b 0/29 and 0/33, Headroom in front g2b 0/27 and 0/30, replay 0/35 and 0/32. In all cells, no outbound body ends on an assistant turn, and the guard never fires on bodies that already end on a user turn. Cache hit and saved tokens stay in the same range (for example small 0.764 and 0.771).

Spend for all measurements: about USD 20.6.

Links

This PR closes no issue by itself. It is the targeted variant. The owner can still choose the shared egress guard (option (b) in #5605).

… turn (ranxianglei#5604)

A preflight fold of the tail (soft protection relaxed to 0, or a tail tool
pair in neverPreserveRecentTools) can leave the rebuilt Anthropic messages
ending on the surviving assistant turn. Anthropic and Bedrock reject that as
assistant prefill ("This model does not support assistant message prefill",
"final assistant content cannot end with trailing whitespace").

prepareAnthropic now closes such a tail with one thin ephemeral user nudge,
inserted before the trailing system run and after all cache-stable content,
via the kernel's insertBeforeTrailingSystemRun. A client-authored assistant
tail (deliberate prefill) passes through unchanged.
@ranxianglei ranxianglei changed the title fix(anthropic): never send an outbound body that ends on an assistant turn (#5604, preflight-fold tail) [PR] fix(anthropic): never send an outbound body that ends on an assistant turn (#5604, preflight-fold tail) Nov 8, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。与 #5605 同症状家族(#5604)但机制不同:#5605 修 failed-compress 的 loop re-request(第一机制),本 PR 修 preflight-fold 的出站尾巴(第二机制)——两者都是 #5891 讨论里点名的 #5604 子缺陷,互补不重叠。检索了 open issue/PR(assistant-tail / prefill / 5604 关键词),没有其他条目动这个 egress-guard 机制。

标签: bug / claude-code / scope/wiring / p1 —— anthropic-wire 出站层的缺陷(prepare-anthropic.ts);Claude Code 是复现载体;p1 因为一旦触发会话直接死掉(fatal 400 循环),虽然 #4898 之后触发概率已很低。

Review verdict: mergeable. Verified live just now: PR open, base = current master 848f59b99 (fresh, not stale), single commit 8c98898ed8, CI run 42505341992 completed success on that head. mergeable_state=blocked is the required-review gate, not a check failure.

What I independently verified (beyond reading the description):

  • Test teeth (red/green): in a scratch checkout I disabled the guard (ensureUserTail made a no-op) and re-ran the new e2e — it goes red at exactly the invariant: expected: 'user', actual: 'assistant' on the forwarded tail (tests/issue5604-anthropic-egress-user-tail.test.ts:119). Restored the guard → 5/5 green. The red run reproduces the exact upstream-rejected shape, so the regression test genuinely catches the bug.
  • cache-proof (the suite the author flagged for the rejected strict condition): 13/13 green locally on the PR head.
  • Sandbox note: cache-friendly-plugin fails identically on plain master in this environment (ACP-tag well-formedness assertion, truncated ref) — inherited sandbox noise, not this PR; the authoritative full-suite verdict is CI, which is green.
  • The kernel helper exists on master (kernel/src/wire/anthropic.ts, exported via the acp-kernel/wire barrel) and its contract matches the usage (empty additions → same reference; insertion before trailing system run). No kernel/, package.json, or version touched — no Kernel Boundary or release concerns.
  • The three early-return passthroughs (auto-mode classifier, threads lane, global-scope system) bypass the guard and still forward client bytes untouched — deliberate-prefill semantics preserved there too. On the kernel-transform-failure fallback path, rebuiltMessages === parsed.messages, so the guard structurally cannot fire on an unprocessed body.
  • dedupeAdjacentOutputConfig ([Bug] fold reconstruction can emit two adjacent output_config effort directives → deterministic 400 on Anthropic lane (pi thinkingLevel max + OpenRouter) #5688) runs after the guard and only collapses adjacent system directives — it cannot undo the user-tail closure. Billing/estimate consumers read parsedBody read-only; nothing persists the nudge (e2e turn-2 pins non-replay + byte-identical prefix).

One coordination item (non-blocking): #5441 merge order. #5441 adds import { insertBeforeTrailingSystemRun } from "../util.js" to the same file; if it lands first, this PR's kernel import is a TS duplicate identifier until someone rebases. The kernel doc comment marks the host copy as interim (deleted in a follow-up after both merge), so the convergent end-state is the kernel import used here. Suggest merging this PR first, or rebasing #5441 onto the kernel import.

Minor observation (accepted, no action needed): unit test 6 pins the client-prefill pass-through using the same array for both arguments; the distinct-array variant exercises the identical code path (two independent findLast calls), so a separate row would be a tautology — leaving it out is correct. Code quality otherwise meets the external-contributor bar (named constant for the nudge text, why-comments matching file style, no dead code, no new deps) — no refactor round.

Behavior change disclosure (old → new): previously an outbound body ending on an assistant turn — produced by bili's own fold — was forwarded as-is → upstream HTTP 400 ("This model does not support assistant message prefill") → client retries the same body until it gives up → session death. Now: one ephemeral user nudge ([bili] Earlier context was compressed above. Continue the task.) is appended after all cache-stable content and before any trailing system run; the request succeeds. That is the intended fix itself; a warn log line (trailing-assistant: closed outbound tail with a user nudge (#5604)) marks every firing for observability.


结构化八问(review 结论)

  1. 是什么问题:preflight 在窗口溢出时折掉尾部工具对后,重建出的出站 Anthropic body 以 assistant 回合结尾,上游按 assistant prefill 拒绝(HTTP 400),客户端拿同一 body 重试到放弃,会话死亡。属 [Bug] failed-compress recovery receipt (#5220/#5231) leaves the re-request ending on an assistant turn → upstream 400 (trailing whitespace / prefill not supported), subagent dies #5604 症状家族的第二机制(第一机制由 [PR] Failed-compress recovery: end the re-request on a user turn, not an assistant prefill (fixes upstream 400) #5605 修)。
  2. 什么原因导致的:fold 取走尾部 tool_use/tool_result 对,但最后一条 assistant 消息的文本部分幸存,重建数组因此以该 assistant 消息结尾;prepareAnthropic 对重建数组没有任何尾部形状校验就原样转发。根因是「fold 重建可把 user 尾翻成 assistant 尾」,修复放在稳态出站 body 定稿的唯一决策点。
  3. 影响范围与严重程度:任何 plain-proxy anthropic-wire 会话,payload 超窗且尾部工具对被折(soft protection relax 到 0,或尾工具 ∈ neverPreserveRecentTools)。触发即完全不可用(400 循环、会话死);[PR] kernel: keep signed thinking with its tool_use across folds (#4885) #4898 之后为残留低概率(实测 agentic 3 跑中 1 跑),但发生即致命 → p1。
  4. 怎么修复的:出站守卫 ensureUserTail——仅当「出站最后非 system 是 assistant 且客户端原 body 最后非 system 不是 assistant」(即 bili 自己把 user 尾翻成了 assistant 尾)才触发;用 kernel 现成的 insertBeforeTrailingSystemRun 在 trailing system run 之前、所有 cache-stable 内容之后插入一条 ephemeral user nudge。客户端自撰的 assistant 尾(故意 prefill)原样放行;nudge 不落盘、不进客户端重发历史。
  5. 是否有回归问题:逐项审过既有行为——user 尾 body 同引用零分配返回(单测钉死);客户端 prefill 放行不变(cache-proof 本地 13/13 绿 + CI 全量绿);正常 nudge/statusLine/retrieval-note 注入路径结尾已是 user,守卫不触发、无干扰;kernel transform 失败回退路径结构上不可能触发;三个早退 passthrough 不过守卫、保持字节级转发;[Bug] fold reconstruction can emit two adjacent output_config effort directives → deterministic 400 on Anthropic lane (pi thinkingLevel max + OpenRouter) #5688 dedupe 只收拢相邻 system 指令,无法撤销 user 尾闭合;计费估算只读消费 parsedBody,无持久化副作用。未发现回归。
  6. 是否破坏缓存:否。nudge 插在所有 cache-stable 消息之后、trailing system run 之前,不带 cache_control,e2e 钉死下一轮前缀逐字节重发且 nudge 不重放;作者实测 cache hit 同区间(small 0.764→0.771)。
  7. 是否改了设计方向:否。单模块出站守卫,不动架构/边界/数据流。共享 egress guard([PR] Failed-compress recovery: end the re-request on a user turn, not an assistant prefill (fixes upstream 400) #5605 的 option b)明确留给 owner 决策,本 PR 不扩面;loop re-request 路径已由 [PR] Failed-compress recovery: end the re-request on a user turn, not an assistant prefill (fixes upstream 400) #5605+[PR] fix: execute proxy-owned tools on the non-stream Anthropic path (#5957) #5966 覆盖。
  8. 是否引入了新的配置:无。没有新 env/config/CLI 字段。

判定:可合并——建议 owner 在方便时 merge(合并顺序注意上面 #5441 那条)。

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1103-master.c4bae93 (rolling master tag).

npm install -g billion-context@0.0.1103-master.c4bae93  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Nov 9, 2026
…anxianglei#5970 retained folds, ranxianglei#5978 anthropic egress tail, ranxianglei#5981 truncated compress receipt

Advances the base from 33d4951 to 7d18b7a (15 commits, 35 files). Four
files overlap with this lane: src/plugin.ts, src/server/relay.ts and the two
generated docs. All four auto-merged; upstream added `noteTruncatedUpstream`
calls inside the plugin pipelines it already touches, while the lane's strip
gates live elsewhere in the same files.

Notable upstream commits adopted:
 - 0f93793 feat(pi): pi.commands — configurable slash-command registration surface
 - 4f80702 fix: recover retained compression blocks against fork replay
 - 8c98898 fix(anthropic): never send an outbound body that ends on an assistant turn
 - adc863a fix: tell the model to re-issue a compress whose args were cut in transit
 - 38e6b07 fix: bound the lost-in-transit re-issue advice with the session breaker
 - d1b2d8f fix: 会话历史按真实维度拦死 — count gate + 32MiB ingest cost band
 - fa447f9 fix: exempt image-bearing bodies from the ingest cost band
 - 33bec44 fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port
 - 2826c51 fix(launcher): field-tune wedged-takeover timings + quiet win32 ESRCH alarm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants