Skip to content

[PR] fix: answer WS-lane failures with object error frames and true 1009 sizes (#4601) - #4624

Closed
ranxianglei wants to merge 9 commits into
masterfrom
2026-10-25_4601-ws-error-frame
Closed

ranxianglei wants to merge 9 commits into
masterfrom
2026-10-25_4601-ws-error-frame

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What this fixes

Field incident (bili 0.1.206, codex-tui 0.164.0): an un-intercepted compaction was forwarded at 17,490,755 bytes, the upstream closed with 1009, bili logged WS upstream rejected the 0.0 MiB request with close 1009, and codex failed the turn only after ~15 minutes of 5-minute idle timeouts x retries — because it never saw the failure.

Two defects on that failure path:

  1. Invisible failure frames. dispatch answers pipeline failures with buildProxyFailureResponse, whose body carries error as a string. On the WebSocket lane ResponsesWsResponse.end() spread that body into { ...details, type: "error", status } — so the frame kept a string error. Codex parses WS error events as { type, status, error: { code?, message?, headers? } } (parse_wrapped_websocket_error_event in codex-rs/codex-api/src/endpoint/responses_websocket.rs); a string fails deserialization, the frame is not treated as an error, and the stream waits out its idle timeout. Every pipeline-level failure on the lane (upstream timeout/reset, circuit fail-fast, 1009 oversize, anything) hit this.
  2. Lying 1009 diagnostic. relay.ts computed sentBytes only for string bodies — a verbatim-passthrough (Buffer) body reported 0.0 MiB together with the default "no sheddable images outside the protected recent messages" reason, although no shedding was ever attempted (shedding needs a JSON wire body).

Changes

  • src/responses-ws.ts — ResponsesWsResponse.end() normalizes the non-terminal JSON body into the WS-lane error contract at this single conversion point: object error passes through unchanged; string error becomes { code: "proxy_failure", message: "<error>: <detail>" }; a missing error falls back to the incomplete_response shape. Status keeps its semantics (statusCode >= 400, default 502).
  • src/server/relay.ts — sentBytes measures Buffer bodies (wireBody.byteLength) instead of hardcoding 0; wsOversizeDiagnosis gains an optional why override used by the no-shed guard branch to state the truth ("the body is not a JSON wire body (verbatim passthrough) — image shedding cannot apply") instead of claiming sheddability was checked.
  • tests/issue4601-ws-failure-frame.test.ts — new regression suite: pipeline 1009 → object-form frame codex can parse (fail-fast within 15 s vs the 300 s idle timeout), passthrough 1009 → real size + honest reason, plus negative gates pinning the HTTP string-error contract and the admission errorFrame shape.

Behavior changes (old → new)

  • WS-lane failure frames (any client on the Responses-WebSocket lane, e.g. codex / opencode-v2): error was a string, now an object { code, message }. Old frames were silently ignored by codex (the bug); new frames make it fail fast. Other fields (detail, upstreamFailure) remain additive as before.
  • WS-lane unparseable-buffer fallback: status was hardcoded 502, now honors an explicit 4xx/5xx statusCode (default still 502). Only reachable when the server set a 4xx/5xx status but wrote a non-JSON body; previously unreachable in practice.
  • 1009 diagnostic text for verbatim-passthrough bodies: "0.0 MiB … no sheddable images" → actual size + "not a JSON wire body (verbatim passthrough) — image shedding cannot apply".
  • HTTP lanes: unchanged. buildProxyFailureResponse keeps its string-error contract (pinned by the new gate test).

Verification

  • New suite 3/3 green; negative control confirmed both defect tests FAIL on the unfixed tree (gate test passes on both, by design).
  • Adjacent lanes green locally: tests/codex-responses-ws.test.ts (9), tests/issue3701-ws-byte-budget.test.ts (11), tests/responses-ws-http-fallback.test.ts (9) — 29/29.
  • tsc --noEmit clean. Full-suite verdict = CI on this head.
  • Hot-file note: open PR [PR] fix: route codex context-only prewarms to the probe relay via the upstream proxy #4595 also touches src/responses-ws.ts (probe-gate region in onMessage, ~line 655) — disjoint from this change (end(), ~line 569); rebase if they land together.

Closes #4601

…izes (#4601)

Two defects on the codex Responses-WebSocket failure path (field: bili 0.1.206,
codex-tui 0.164.0, 17.5 MB un-intercepted compaction, close 1009, turn failed
after ~15 minutes):

1. Pipeline failures were answered with the HTTP proxy-failure body whose
   'error' is a string. The WS-lane error contract is { type, status,
   error: { code?, message?, headers? } } (codex parse_wrapped_websocket_error_event);
   a string fails that deserialization, so codex dropped the frame and waited
   out its 5-minute idle timeout x retries. ResponsesWsResponse.end() now
   normalizes at this single conversion point: object errors pass through,
   string errors become { code: proxy_failure, message: '<error>: <detail>' },
   missing errors fall back to incomplete_response. Status semantics keep
   statusCode >= 400 (default 502); the unparseable-buffer fallback now honors
   an explicit 4xx/5xx status instead of always 502. HTTP lanes keep their
   string-error contract untouched.

2. The 1009 oversize diagnostic measured sentBytes only for string bodies —
   a verbatim-passthrough (Buffer) body reported '0.0 MiB' plus the false
   'no sheddable images' reason although no shedding was attempted. Buffer
   bodies are now measured, and the no-shed branch names the real reason
   (body is not a JSON wire body — image shedding cannot apply).

Regression suite: tests/issue4601-ws-failure-frame.test.ts (both defects +
negative gates: HTTP string contract, admission errorFrame shape).
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_4601-ws-error-frame (5089237)

Latest build — install this exact version

npm install -g billion-context@0.0.6167-pr.4624.c56bffd

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
5089237 billion-context@0.0.6167-pr.4624.c56bffd build
461a5b3 billion-context@0.0.6080-pr.4624.8dd7112 build
d133e41 billion-context@0.0.5982-pr.4624.dec82e7 build
3a5a656 billion-context@0.0.5878-pr.4624.d69e0dc build
1fac542 billion-context@0.0.5811-pr.4624.d386d2b build
bc0573e billion-context@0.0.5652-pr.4624.69f43c1 build
ff903c0 billion-context@0.0.5619-pr.4624.128a493 build
ce00312 billion-context@0.0.5604-pr.4624.807a2af build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4624.tgz
npm install -g package

Install instructions are refreshed on each push.

…master-red)

The unused-exports gate fails on master itself (verified on origin/master tip
146b34f): src/agent/client-build-note.ts exported ClientBuildCheck while every
reference sat inside the declaring file (INTERNAL_ONLY). This PR touched neither
that file nor its importers, so the failure is inherited from master, not
introduced by #4601.

Drop the leading export per the gate's sanctioned fix for INTERNAL_ONLY. The
type stays defined and used internally (maxTestedVersion / DSH_BUILD_CHECK and
the emit call sites); no other module references it by name, so nothing breaks.
Isolated drive-by commit so the #4601 diff stays scoped.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI auto-fix (attempt 1/2): fixed the red unused-exports check directly on the PR branch — pushed ff903c00a → 2026-10-25_4601-ws-error-frame.

  1. 是什么问题:PR 的 unused-exports 检查红。门禁把 src/agent/client-build-note.ts 的 ClientBuildCheck 判为「导出了但没人用」(INTERNAL_ONLY)。关键点:这不是本 PR 引入的问题——master 本身就挂同一检查。

  2. 什么原因导致的:根因在 master 侧。[PR] dsh lane: note in-session when running an unverified @deepseek-ai/dsh client build (soft, exact-match) #4545 引入 client-build-note.ts 时导出了 ClientBuildCheck,但该类型的所有引用都在声明文件内部(maxTestedVersion / DSH_BUILD_CHECK / 调用点),没有任何其他模块按名字 import 它,于是文本引用计数门禁判为 INTERNAL_ONLY。本 PR(Codex WS lane: proxy-failure error frames carry error as a string, so codex ignores them and waits for its idle timeout; 1009 diagnostic reports 0.0 MiB #4601)只改 responses-ws.ts / relay.ts 并加测试,没碰那个文件及其 importers。已验证:对 origin/master tip 146b34f 跑同一门禁,复现完全相同的那一条 finding → 继承性红。

  3. 影响范围与严重程度:零运行时影响。纯导出卫生门禁违规,无任何行为变更。严重度 = 仅 CI lane 红(合并不绿前被挡);无用户面影响。波及当前 master 上任何合并的 PR(继承性),不止本 PR。

  4. 怎么修复的:去掉类型声明前的 export,让它变回模块内部符号(门禁对 INTERNAL_ONLY 的标准修法)。类型仍在文件内定义并被使用;外部无按名引用,故不破坏任何东西。独立成一个标注 commit,Codex WS lane: proxy-failure error frames carry error as a string, so codex ignores them and waits for its idle timeout; 1009 diagnostic reports 0.0 MiB #4601 的 diff 保持聚焦。

  5. 是否有回归问题:无。行为中性——对一个无外部消费者的类型去掉 export 不改任何运行时行为/wire/config/持久化。改后本地跑门禁通过;grep 确认无其他文件按名字引用 ClientBuildCheck。高危模式(语义漂移 / fail-fast 变吞错 / 阈值·时序·输出格式漂移):均未触碰。

  6. 是否破坏缓存:不涉及(不涉缓存/重试/游标/去重)。

  7. 是否改了设计方向:不涉及(无架构/边界/数据流转向,仅收窄一个符号的可见性)。

  8. 是否引入了新的配置:不涉及(无新 env/config/默认值)。

判定:已推送,新 head ff903c00a 上 unused-exports 应转绿;其余 lane(typecheck/test)由 CI 把关。注:此继承性 master 红现随本 PR 带上了修复——合入后即落到 master;其他撞同一失败在途的 PR 可 rebase 到本分支,或同样做这一行去导出。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

这是 [PR] 镜像单,未重做工作——已拉取分支(ce00312,基于当前 master 146b34f,领先 1 个提交)逐行核对代码、核查了 CI 状态。结论如下。

查重: 不重复。 #4624 是 #4601 的唯一在途修复;最近邻 #4595(codex prewarm probe relay)也动 src/responses-ws.ts,但区域不同(onMessage probe-gate ~L655 vs 本 PR 的 end() ~L569)、机制不同(探针路由 vs 错误帧契约),PR 正文已作 hot-file 标注。

标签: bug / codex / opencodev2 / scope/wiring / p2(缺陷只在管道级失败时显形,正常成功回合不受影响,故 p2)。

八问

  1. 是什么问题:Responses-WebSocket 车道上,代理层一切失败都用 HTTP 风格的字符串 error 应答;codex 的 WS 错误解析器反序列化不了字符串,直接忽略该帧——客户端只能等到 5 分钟空闲超时 × 重试(~15 分钟)才失败这一轮。同时 1009 诊断对 verbatim-passthrough(Buffer)体谎报「0.0 MiB + 无可剥离图片」,而剥离流程根本没跑过。
  2. 什么原因导致的:WS 车道直接复用了 HTTP 失败体(buildProxyFailureResponse,error 为字符串),而 ResponsesWsResponse.end() 把该体 spread 进帧时没有按 WS 车道契约 { type, status, error: { code?, message?, headers? } } 做转换——两条车道的契约在唯一转换点没对齐。1009 侧:sentBytes 只对 string 体计量,Buffer 体记 0,诊断模板又无条件套用「无图可剥」措辞。
  3. 影响范围与严重程度:所有走 WS 车道的客户端(codex-tui、opencode-v2);仅在管道级失败时触发(上游超时/reset、熔断 fail-fast、1009 超尺寸等),正常回合不受影响。触发后果:该轮必然失败且慢失败(~15 分钟而非秒级)——能用但有明显问题,p2。
  4. 怎么修复的:单一转换点归一化——end() 里对象 error 原样透传、字符串变 { code: "proxy_failure", message: "<error>: <detail>" }、缺失回退 incomplete_response 形状;HTTP 车道契约不动。1009 侧:Buffer 体按 byteLength 实测,no-shed 分支明说「body 不是 JSON wire body,图片剥离不适用」。
  5. 是否有回归问题:逐项审过既有行为——terminal 成功路径不经过新逻辑(end() 归一化仅 !this.terminal 生效);admission errorFrame(invalid_request 形状)未动且有负向测试钉住;HTTP 字符串契约未动且有负向测试钉住;wsOversizeDiagnosis 新参数可选,另两个调用点(shed 失败、retry 错误)行为不变;高危模式逐项过:无语义漂移、无 fail-fast 变吞错、无阈值/时序漂移。两处有意的行为变更均已披露:① 无 error 字段的对象体现在会补一个 incomplete_response 形状的错误字段(codex 契约全字段可选,语义上也确属不完整响应);② 不可解析 buffer 兜底从硬编码 502 改为尊重显式 4xx/5xx(默认仍 502,该路径此前实际不可达)。
  6. 是否破坏缓存:不涉及(不触碰前缀缓存/ref 账本/压缩状态)。
  7. 是否改了设计方向:没有——修的就是「转换点对齐契约」本身,双模式架构与 WS 错误契约归属不变。
  8. 是否引入新配置:没有。

测试有效性

3 个用例都有牙:核心回归断言 typeof error.error === "object" + code/message 内容 + 尺寸不得为 0.0 MiB + 诚实理由;两个负向门双向钉住(HTTP 字符串契约、admission 形状);作者报告负向控制确认两个缺陷测试在未修树上 FAIL。我未在本地重跑(本地资源稀缺,静态门禁已在 head 上验证),全套件以 CI 为准。

CI 状态与合并判定

ci-check added 7 commits October 25, 2026 10:09
master window commits 293ae65/de5e03659 bumped kernel/package.json to
0.0.165 without regenerating the lockfiles; lock-sync re-resolution then
rewrites the stale recorded versions (root 0.0.163, kernel 0.0.164) and
fails. Pure recorded-version sync via npm install --package-lock-only;
no dependency or semantic change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b) 已合入 master: 本 PR 的改动经「codex 车道小修合集第三辑」#5971 送审合入。本 PR 关闭以免重复。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex WS lane: proxy-failure error frames carry error as a string, so codex ignores them and waits for its idle timeout; 1009 diagnostic reports 0.0 MiB

1 participant