Skip to content

[PR] Responses/chat plugin pipes: stop emitting a truncation error after the terminal frame already reached the client - #4062

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-21_post-terminal-read-error
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-21_post-terminal-read-error

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Problem

In the plugin lane, when an upstream read failure happens after the turn's terminal frame was already forwarded to the client, the catch path unconditionally emits its truncation signal:

  • responses wire (pipePluginResponsesWithStrip): after response.completed, the client still receives event: error {code: "upstream_stream_truncated"} — a failure following a completion. The diagnostic blob self-contradicts too (responseStatus: "completed" next to classification: "no-terminal-seen"). Clients treat the turn as failed → retry or stop.
  • openai / anthropic chat wire (pipePluginChatWithStrip): after [DONE] / message_stop, the same path re-sends a duplicate terminal byte (strict lifecycle validators reject it).

Root cause: the catch path conflates "generation cut" with "connection ended". It knows sawTerminal (set at parse time, also feeding the empty-turn retry) but not whether the terminal byte actually reached the client.

Fix

Track terminalForwarded per pipe — set only after the awaited write of the terminal frame succeeds:

  • responses: the response.completed / response.failed / response.incomplete frame, or [DONE];
  • openai: [DONE]; anthropic: message_stop; google: the finish-reason chunk (its terminator).

In both catch paths, when the flag is set: run the existing usage/witness cleanup, log one warn line, and end clean without appending any frame. All pre-terminal behavior is unchanged: genuine cuts still emit the in-band error (or their finished=true synthesis for openai/anthropic). A terminal consumed by the empty-turn retry (never forwarded) does NOT set the flag, so a cut retry stream still signals as before.

response.failed / response.incomplete keep their own terminal state — nothing is re-voiced into a success.

Behavior change (old → new)

Situation Old New
Read error after forwarded completed-family frame (responses) appends upstream_stream_truncated error frame ends clean right after the terminal frame; warn log terminal already forwarded
Read error after forwarded [DONE] / message_stop (chat wire) re-sends a duplicate terminal byte ends clean; warn log terminal byte already forwarded
Read error after google finish-reason chunk no frame appended, but logged the misleading — truncation signal path line + built a diag same clean end, now logged via the guarded branch
Pre-terminal cut, malformed terminal frame, failure while forwarding the terminal in-band error / synthesis unchanged

No new config surface (no env var / file key / CLI flag). No kernel/ changes. Wire shape: no new frames are added — only spurious post-terminal frames are suppressed; every genuine failure path keeps its exact previous frames.

Tests

New tests/issue4059-post-terminal-read-error.test.ts (13 cases driving the real pipes with synthetic SSE), both directions of the gate per wire:

  • post-terminal read errors end clean: responses × {completed, failed, incomplete} and [DONE]-terminated; exactly one [DONE] / message_stop on the chat wires; stream closed exactly once;
  • pre-terminal cuts keep the in-band signal, including the exact openai error-frame shape (top-level error object + trailing [DONE]) and the no-terminal-seen classification;
  • malformed terminal JSON is still treated as non-terminal;
  • normal completion + clean EOF unchanged (guarded branch never fires);
  • terminal consumed by the empty-turn retry, then retry stream cut → signal still fires (pins that the flag tracks DELIVERY, not parsing).

Local targeted run: 13 new + 167 tests across 14 adjacent suites (truncation diagnostics, zero-byte retry, empty-turn retry, draft-tail, CRLF cross-chunk, usage settlement, backpressure) all green. Full-suite verdict = CI on this head.

Closes #4059

@ranxianglei ranxianglei changed the title Responses/chat plugin pipes: stop emitting a truncation error after the terminal frame already reached the client [PR] Responses/chat plugin pipes: stop emitting a truncation error after the terminal frame already reached the client Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-21_post-terminal-read-error (720f338)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4916-pr.4062.c6feabe

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4062.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 对应 #4059,是修复该机制的唯一在途项。同错误码条目逐条比对:#3576(DSH/Windows 传输频繁中断——真断流,检测本身正确,问题在传输侧)、#2993/#2996(accept-encoding 解码)、#2323/#2329(CRLF 跨 chunk 拆坏)——与本例症状和机制都不同(它们都是「真断流」,本例是「完成后误报失败」),不并入。

标签:bug / dsh / scope/wiring / p2(+既有 pr)。理由:缺陷类;证据车道 dsh Desktop;领域为协议接线(plugin-lane SSE 管道);能用但有明显问题,不到 p1(无数据丢失、无缓存泄露、会话可恢复)。

审核方式(独立于 issue 侧结论重做):全量读 diff,两条 pipe 完整路径逐行走查(write helper / 终局分支 / catch / finally),对照 src/stream-error.ts 确认各 wire 在终局之后原本会发什么字节;本地跑新套件 + 9 个相邻套件(10 文件 97 测试全绿);拉旧 head 的 CI job 日志核实红因。

方向:bug 属实。根因 = catch 路径只知道「解析时见过终局」(sawTerminal),不知道「终局字节是否真的到了客户端」(投递状态),把「生成被截断」与「连接结束」混为一谈。修法是加一个只在 awaited write 成功后置位的投递跟踪标志,在 catch 决策点分叉——修的是产生错误帧的决策本身,不是遮症状。

回归清点(触碰路径的既有行为逐项判定):

  • 四条 wire 的前终局 cut 路径(in-band error 帧 / finished=true 合成 / pi 透明直通):字节级不变,有对比测试钉住;
  • 正常 clean EOF、客户端 abort、empty-turn retry 交互:不变(被 retry 消费的终局不置标志,cut 掉的 retry 流仍发信号——测试已钉);
  • usage 结算 / witness / markDirty:settleUsage/settleWitnesses 在 catch 内先于 guard 执行,顺序与旧路径一致,计费/持久化无影响;
  • 流关闭:finally{res.end()} 保证任何出口恰好 end 一次(endedCount==1 断言钉住);
  • response.failed/incomplete 保留自身状态,不被改写成成功(代码核实 + 断言钉住);
  • [Pi/Windows] 偶发 Responses 流截断:长间隔/TTL 重读标记后几轮内报错的两次观察(因果待确认) #2563 pi 透明直通只覆盖真·前终局 cut(!finished)——post-terminal 从来不在其范围:修复前所有宿主(含 pi)在 post-terminal 读错时都会拿到重复终局/误报信号,修复后统一变 clean end;前终局的透明行为本身不变。
  • 唯一行为变化 = 下表三个 post-terminal 情形(有意变更,已在 PR 披露,我独立复核了每一行的旧行为)。

代码:标志在 4 处统一于 awaited write 成功后置位;写前 alive 检查与 write() helper 自身门一致(write 对死流是 no-op,故「wrote」==「写前活着」);若 drain 期间 res 死亡导致标志 stale-true,循环会在 destroyed 检查处 break 或 abort 检查先行命中,guard 分支不会误触发。无 as any/@ts-ignore,命名达意,注释带 #NNNN 来源符合仓库惯例。

测试:13 例驱动真实 pipe 入口(合成 SSE + 带 end 计数的 mock ServerResponse),每条 wire 双向断言门控;断言有牙(终局出现次数 == 1、error 帧缺席、openai error 帧精确形状 + [DONE] 先后序、classification 字符串、guard 分支日志行)。发现一处缺口:google wire 一例都没有(行为表声称 google 有变化但 Tests 未覆盖)→ 已直接补上:新增 2 例(finish-reason chunk 后的读错 clean end + finish 前的 cut 保留 UNAVAILABLE 帧),本地 15/15 绿,作为独立提交 8bfebe0 推入 PR 分支。

审核中采取的行动:

  1. 把分支 rebase 到当前 master 56a0f5a(含 [PR] fix(ci): Storage mode isolation — #3769 pin both to full, plugin-fork CCR cold recovery removes sidecar, same assertion | fix(ci): 存储模式隔离 — #3769 双件钉 full,plugin-fork CCR 冷恢复删 sidecar,两模式同断言 #4077 存储 hermeticity 修复)。旧 head 的 5 个红 lane:拉 job 日志抽查了 ubuntu-22-shard1(3319 个测试仅 1 失败,即已知 [Bug] MCP failed compression record rotation prematurely rewrites the history prefix, triggering expensive cache recompu | [Bug] MCP 失败压缩记录轮换提前改写历史前缀,触发大额缓存重算 #3769 存储测试 not ok 1108 …(#3769)),与本 PR 无关(diff 完全不碰存储);其余 4 个红 lane 未逐一核日志,但同一分片体系且 master 已合入该族的修复,以 rebase 后重跑为准。另:旧 head 上全部真实客户端 e2e lane(dsh/opencode/pi/codex-fake/registry)均 SUCCESS。
  2. 推入上述 google 测试补充提交。

行为变更(old → new,有意):

情形 旧 新
responses completed-family 帧已转发后的读错 追加自相矛盾的 upstream_stream_truncated error 帧 紧跟终局帧 clean end;warn 日志 "terminal already forwarded"
chat wire [DONE]/message_stop 已转发后的读错 重发重复终局字节(严格生命周期校验器会拒) clean end;warn 日志 "terminal byte already forwarded"
google finish-reason chunk 之后的读错 无帧追加但记误导性 truncation signal path 日志 + 构建 diag 同样 clean end,经 guard 分支记录
前终局 cut / 畸形终局帧 / 终局转发中途失败 in-band error / 合成 不变

理由:终局一到客户端,turn 从客户端视角已完成;其后追加帧要么是重复终局(生命周期违规),要么是虚假失败报告(客户端徒劳重试或停摆)——即 #4059 的两个症状。无新配置面、无 kernel/ 改动、不新增任何帧。

结构化八问:

  1. 是什么问题:turn 的终局字节已到客户端后发生上游读错误,代理仍追加「流被截断」失败信号——把已完成的 turn 报成失败(responses wire),或重发终局(chat wire)。
  2. 什么原因导致的:catch 路径只有解析期状态「见过终局」,没有投递状态「终局是否到达客户端」,把生成截断与连接结束混为一谈。
  3. 影响范围与严重程度:任意 plugin-lane 宿主、四条 wire 上的 post-terminal 读错都会触发(实测证据:dsh Desktop/Windows);已完成的 turn 被误报失败 → 客户端重试(浪费 token)或停摆(用户被迫重发)。p2:可用、无数据丢失、无缓存泄露。
  4. 怎么修复的:每条 pipe 加投递跟踪标志,仅在终局帧的 awaited write 成功后置位;catch 中标志已置则保留 usage/witness 清理、记一行日志、clean end,不追加任何帧。
  5. 是否有回归问题:无。触碰路径既有行为逐项判定见回归清点:前终局信号形状字节级不变、关闭语义不变、计费/持久化顺序不变、pi 透明路径前终局行为不变。
  6. 是否破坏缓存:不涉及——未触碰会话状态/缓存键/增量游标;usage 结算逻辑与调用顺序不变。
  7. 是否改了设计方向:不涉及——现有 pipe 内纯增量标志,无架构/边界/数据流转向。
  8. 是否引入新配置:无——未加 env/file key/CLI flag。

CI 状态:最终 head 8bfebe0(rebase + google 补充)上目前 32 项绿、7 项进行中(test 分片)、0 红;旧 head 的红均为存量 #3769 存储 flake(master 已随 #4077 修复)。

判定:方向正确、修复打到根因、回归面封闭、测试有牙(含我补的 google 两例)。可合入——待进行中的 CI 项转绿即可。合并留给你:#4062

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已跑完 — 最终 head 8bfebe0 的状态:33 项绿、5 个 test lane 红。 逐条拉了 5 个红 lane 的 job 日志核实,全部是存量/环境问题,没有一项由本 PR 引起:

为何断定不是本 PR:diff 只碰 src/plugin.ts + tests/issue4059-post-terminal-read-error.test.ts(git diff --name-only 实证);没有任何一个失败的测试会走到 pipePluginChatWithStrip/pipePluginResponsesWithStrip;失败签名与 PR 改动面零交集。

本 PR 相关套件全绿:新增 15 例 + truncation/zero-byte/empty-turn/draft-tail/CRLF/usage/backpressure 各套件在各自分片里都过。

判定不变:可合入。 这 5 个红是环境性 CI 噪音,对任何 PR 一视同仁——现在合入没问题,或等 verif-abs-path-fix 落地后重跑拿一块干净的板子。合并留给你:#4062

awork added 3 commits October 21, 2026 14:38
… (CI triage #4062)

Found while triaging the #4062 CI red: the async test body kept executing
after t.skip() because the gate line had no return. On every solverless CI
shard it then threw deeper (no-solver error on Linux, doubled-drive-letter
ENOENT on Windows), producing a 'not ok ... # SKIP' TAP line that is NOT
counted in the failure total but reads like a failure and misled this
triage. The return makes the skip honest per the KDD #14 auto-skip
contract; behavior with a solver present is unchanged (branch never taken).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已修到可合状态:原 5 条红 lane 的根因全部是 PR base(56a0f5a)过期——verif 门控/绝对路径缺陷已在 #4084、#3769 Windows EBUSY 已在 #4087 修入 master。我把分支 rebase 到最新 master(5f3b1e6)并推了新 head 720f338,这 5 条 lane 现已全绿;另附一个标注过的顺手小修(verif 测试 skip 门控缺 return,一行,零行为变化)。

剩余唯一红:test (windows-latest, 22, 2),两个现象均判定与本 PR 代码路径无关:

  1. e2e Context Window Overflow Issue Summary: Raw Requests Sent Directly to Upstream Rejected + Local Intercept Cooldown Lockin | 上下文超窗问题汇总:请求裸冲上游被拒 + 本地拦截冷却锁死会话(统一分析入口) #4079 两例 drain(cancelledByParent,「Promise resolution is still pending but the event loop has already resolved」,fail=0、零断言失败):两次运行(不同 runner)都复现,但 Context Window Overflow Issue Summary: Raw Requests Sent Directly to Upstream Rejected + Local Intercept Cooldown Lockin | 上下文超窗问题汇总:请求裸冲上游被拒 + 本地拦截冷却锁死会话(统一分析入口) #4079 测试文件是纯 proxy 模式(无 x-bili-plugin、无 register),从不进入本 PR 改动的 plugin pipe;且 master 5f3b1e6 约一小时前在同一 lane 跑过同一文件并全绿。符合已知 Flaky Windows CI: issue1993-preflight-400 suite cancelled by early event-loop drain | Windows CI 偶发假红:preflight-400 测试进程提前退出致 5 例取消 #3665 event-loop-drain 家族(Windows 负载),非本 PR 回归。
  2. 客户端取消 = 无主 AbortError ×260/天 + 已计费 usage 丢弃(lastInputTokens 冻结)+ 透传路径漏 10min timer(无 finally) #411-abort-usage 一例 3 秒 waitFor 超时(等 async info 日志行):同场景下前两个异步断言(usage 落账、timer 清零)都过了,只有第三个日志等待超时;本 PR 在 catch 块新增的 terminalForwarded 分支位于 client-abort 分支之后,客户端已 abort 的场景走的分支与改前逐字相同。该测试文件自身注释就记载过 windows-latest 负载 flake(release v0.1.187 #2305),属同类。

两轮(run + rerun)其余 5 个 test 分片(ubuntu×4 + win22-s1)及全部 gate/e2e 辅助 lane 全绿。本 lane 按「最多 2 次」到此为止——剩余红的根治需要动无关测试文件的等待预算/重试语义,属测试基建范畴,超出本 PR 范围。


结构化八问

  1. 是什么问题:PR 头 8bfebe0 的 CI 五条 test lane 全红,阻塞合并。
  2. 什么原因导致的:PR base 落后于 master——Verif expansion: extend solver verification from the kernel decision core to latch discipline, all wire adapters, and th | verif 推广:把求解器验证从 kernel 决策核扩到 latch 纪律、全部 wire 适配器和 src 纯核 #3989/Messages with duplicate IDs overwrite the rendered bytes of already-sent messages: protected messages are re-tagged by l | 重复 id 的消息会改写已发送消息的渲染字节:受保护消息被后来的同名 id 消息重新打标 #3945 系列新加的 verif 求解器测试缺 solver-availability 门控(view-latch)、硬编码开发机绝对路径(anthropic round-trip),加上 [Bug] MCP failed compression record rotation prematurely rewrites the history prefix, triggering expensive cache recompu | [Bug] MCP 失败压缩记录轮换提前改写历史前缀,触发大额缓存重算 #3769 Windows EBUSY 存储 flake;三者随后已被 [PR] fix(verif): drop dev-machine absolute paths in wire harnesses + gate solver-dependent anthropic round-trip tests #4084/[PR] fix(ci): close store1 before dropping index.db in #3769 pooled-bare-load — Windows EBUSY #4087 修入 master,与 PR 本身改动无关。
  3. 影响范围与严重程度:仅影响 CI 判绿(所有 PR 的红 lane,非用户面);对最终产物零影响。剩余 win22-s2 红为已知 Windows 负载 flake 家族(Flaky Windows CI: issue1993-preflight-400 suite cancelled by early event-loop drain | Windows CI 偶发假红:preflight-400 测试进程提前退出致 5 例取消 #3665/release v0.1.187 #2305 口径),特定条件下出现(高负载 Windows 分片),体验瑕疵级。
  4. 怎么修复的:rebase 到含修复的最新 master(吸收 [PR] fix(verif): drop dev-machine absolute paths in wire harnesses + gate solver-dependent anthropic round-trip tests #4084/[PR] fix(ci): close store1 before dropping index.db in #3769 pooled-bare-load — Windows EBUSY #4087),不向 PR 塞任何重复提交;顺手小修 = tests/verif-generic-append-stability.test.ts:50 的 t.skip() 后补 return(此前 skip 后函数体继续执行、在无求解器机器上抛错,产生误导性的 not ok … # SKIP TAP 行,虽不计入失败数但误导本次排查;有求解器环境不受影响)。
  5. 是否有回归问题:无。PR 自身 diff 未变(仍为 src/plugin.ts + issue4059 测试);小修只让 skip 诚实化,断言语义零变化;rebase 引入的 master 变更均已各自过 CI。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:PR 本体 CI-clean、可合;建议人工对该 PR 再 rerun 一次 test (windows-latest, 22, 2)(或知悉该 lane 为已知 flake 家族后直接合并)。若重跑仍 drain,请按 #3665 家族处理 e2e #4079 的 drain 韧性,与本 PR 解耦。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

# [Bug] Responses: a read failure after a forwarded terminal event emits contradictory upstream_stream_truncated

1 participant