Skip to content

[PR] fix: abort the HTTP fallback egress when the Codex WS client disconnects - #6008

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
juunghyun:2026-11-09_ws-fallback-client-abort
Nov 10, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
juunghyun:2026-11-09_ws-fallback-client-abort

Conversation

@juunghyun

Copy link
Copy Markdown
Contributor

Fixes #6007

Problem

ResponsesWsResponse is a socketless http.ServerResponse. OutgoingMessage.destroy() emits 'close' only through its own socket slot, so when the Codex WS client disconnects and onClose() destroys the active sink, the relay never sees 'close' and clientAbort never fires. On the #3342 HTTP/SSE fallback that signal is the only thing that cancels egressHttp(), so the supplier request runs on until the 12-minute idle watchdog. The test that pins this was failing on master, hidden by a force-exit after() hook.

Change

File Change
src/responses-ws.ts ResponsesWsResponse.destroy() emits 'close' once when the exchange has not ended. end() after a destroy no longer emits a second 'close'. This is the same socket-death behaviour InlineRes.destroy() already has in src/inline-engine.ts.
tests/responses-ws-http-fallback.test.ts Removes the after(() => process.exit(...)) hook. The fixture's close() now calls upstream.closeAllConnections(), so a regression fails within seconds instead of hanging the shard. Adds a mid-stream variant (the existing test only covers the phase before headers) and a negative row: closing the client after a finished turn is not treated as a disconnect.

Points worth a look:

  • Every caller of destroy() on this class abandons the exchange: onClose, shutdown, the [Bug] Codex Responses WebSocket blocks long sessions at 100 MiB before ACP compression, including small incremental continuations #4804 SnapshotAnchorFallbackError path and the output-buffer-limit path in write(). Aborting the upstream is what each of them needs. super.destroy() behaves as before: it does not go through the socket getter, so req.socket is not destroyed.
  • If the client closes after it has received the terminal frame but before the relay calls end(), this now counts as a client disconnect: the upstream is aborted and the relay logs client disconnected mid-stream. That matches the HTTP lanes and InlineRes. The window is small: a supplier that sends the terminal event and only then delays EOF. If you would rather not count it, the guard could also require !this.terminal.
  • src/server/relay.ts is not touched. The existing 'close' contract now fires on this lane too.

Testing

AI-assisted: I investigated and prepared this change with Claude Code, then reproduced and verified the results myself.

@ranxianglei ranxianglei changed the title fix: abort the HTTP fallback egress when the Codex WS client disconnects [PR] fix: abort the HTTP fallback egress when the Codex WS client disconnects Nov 9, 2026
@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring ws labels Nov 9, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 结论:可合并(先 rebase 到最新 master——分支落后 3 个提交,零文件重叠,GitHub mergeable=true / state=behind)。外部贡献质量达标(命名、注释、测试有效性、风格逐项过审),无需对齐重构。

独立验证(未复用 issue 侧结论,全部自己重做):

  • 根因:读 Node v22 _http_outgoing.js 源码——OutgoingMessage.prototype.destroy() 走内部 symbol kSocket,不走 JS 层 socket getter;本类构造函数把 getter 覆写成返回 req.socket,因此 destroy 既不 emit 'close' 也不会销毁客户端 socket(探针实测三场景:socketless / getter 挂真 socket / 真实 server response,与 issue 描述一致)。relay 只在 'close' 且 !writableEnded 时 abort 上游(src/server/relay.ts:838-849)→ 该车道断连信号丢失。
  • 测试双向有效性:PR head 本地跑 12/12 pass(~2.7 s,exit 0);master 源码 + 新测试文件 → 恰好两个 disconnect 测试快速失败(until 5 s deadline),10/12,~11 s 自然退出。有牙;masking hook 移除后 teardown(upstream.closeAllConnections())防 shard 挂起。
  • destroy() 四个调用点全审计(onClose / shutdown / [Bug] Codex Responses WebSocket blocks long sessions at 100 MiB before ACP compression, including small incremental continuations #4804 anchor-fallback / write 输出缓冲上限):均属「放弃本轮 exchange」,abort 上游在每种场景都是正确语义;[Bug] Codex Responses WebSocket blocks long sessions at 100 MiB before ACP compression, including small incremental continuations #4804 路径原本存在同类泄漏(egress 放任跑到 watchdog),本次一并堵上。正常完成仍走 end() → writableEnded=true → 不误 abort。
  • 一致性:与 InlineRes.destroy()(src/inline-engine.ts:217-223)同一「socket-death semantics」契约,实现模式相同。
  • CI:head ed60bb9 全绿(30 checks,0 failed)。

既有行为回归清点(diff 触碰路径逐项):

既有行为 结论
WS 车道正常完成(end→finish+close) 保持;'close' 仍恰好一次(destroy 后再 end 不再二次 emit)
轮中客户端断连(HTTP/SSE fallback) 有意变更(修复本身):supplier 请求从「跑到 12 min watchdog 或 supplier 完成」→「毫秒级 abort」
输出缓冲上限(write 里 peer close 1013) 有意变更:上游立即 abort(原放任运行),同机制族
#4804 anchor-fallback 有意变更:在途 egress 立即 abort(原放任运行)
shutdown 有意变更:在途 exchange 立即 abort
terminal frame 后、end() 前客户端关闭 行为变化:计为 mid-stream disconnect(log + abort)。窗口极小(supplier 已发 terminal event 仅延迟 EOF);代价 = 跳过本次 checkpoint,codex 下轮全量重发(标准恢复);与 HTTP 车道同窗口语义一致。评审判定可接受,保留 !this.ended guard(作者备选 !this.terminal 不采纳,理由见上)
测试文件退出行为(hook 移除) 已验证进程自然退出(带 fix 2.7 s / 不带 11 s),不影响其他 shard

高危模式逐项过:无 fail-fast 变吞错;无阈值/时序/默认值漂移;无输出格式变化;wire 形状未动(relay 零改动)。

八问:

  1. 是什么问题:Codex WS 车道回落到 HTTP/SSE 的 supplier 时,客户端断开不会中止 supplier 请求,用户取消后仍持续为看不到的输出付费。
  2. 什么原因导致的:ResponsesWsResponse 覆写 socket getter,但 Node 原生 destroy() 只读内部 socket slot,destroy 不 emit 'close';relay 仅凭 'close'(且 !writableEnded)abort 上游,断连信号在该车道丢失。
  3. 影响范围与严重程度:仅「WS 客户端 + HTTP/SSE-only supplier」配置(codex-responses-ws 与 opencode WS 两个 codec 共用此 sink,均受影响),用户取消时触发;触发期间 token 持续消耗直到 idle watchdog(默认 12 min)或 supplier 完成。p2:可用,特定条件下成本泄漏,有界。
  4. 怎么修复的:让 destroy() 在 exchange 未结束时补发 'close',对齐仓库既有 socket-death 契约(InlineRes 同款);relay 不动;测试文件移除 masking hook 并强化 teardown。
  5. 是否有回归问题:见上表;有意变更均已披露,未发现无意回归。
  6. 是否破坏缓存:缓存/重试/增量游标/去重键均不受影响;noteClientAbort 在该车道变得可达(记录 abort 关联时间戳供 cache-ledger 使用),与其他车道一致,属增量。
  7. 是否改了设计方向:无。把该 sink 对齐到既有契约,数据流不变。
  8. 是否引入了新的配置:无。

收尾事项:① rebase 最新 master 后合并;② 标签已打 bug/p2/ws/scope/wiring;③ 作者提出的 guard 取舍我已给推荐(保持现状),无需额外拍板。

@juunghyun
juunghyun force-pushed the 2026-11-09_ws-fallback-client-abort branch from ed60bb9 to 3307bf4 Compare November 9, 2026 06:36
@juunghyun

Copy link
Copy Markdown
Contributor Author

(ai: anthropic/claude-opus-5-5)

Rebased onto current master (5c3c2688); no conflicts. tests/responses-ws-http-fallback.test.ts still passes 12/12 locally. Keeping the !this.ended guard as you suggested.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Rebase verified and independently re-reviewed on the new head (3307bf43 on 5c3c2688).

查重: 不重复。相关但机制不同:#5601 改的是「已触发的断连 abort 如何归因/展示」(分类层,不是触发层);HTTP/SSE fallback 车道本身源自 #3342/#3447。

标签: 已补打 bug + codex + scope/wiring + p2(缺陷在 Codex WS 客户端接入层;产品可用,特定条件下每次断连有界成本泄漏 → p2,无数据/会话/缓存丢失,不升 p1)。

State verified live this round — everything below re-run by me on the rebased head, not carried over from the earlier review:

One factual refinement for the record: on master the pre-existing pinning test did not visibly fail — it passed green vacuously. I instrumented it: the abort assertion is unreachable (the supplier never aborts, so the wait never resolves), the test hangs in teardown, hits the 30 s timeout, and the force-exit after() hook converts the whole run into exit 0. So CI was green all along while the bug was live; "was failing on master" reads more precisely as "was masked-green on master". I also reproduced the leak standalone on master: client cut at T+906 ms → supplier request still open at T+10929 ms (never aborted), no client disconnected mid-stream log line. The PR's net effect is unchanged and verified: with the fix removed, the new test file fails fast.

Verdict: ready to merge once the in-flight CI shards go green; no code changes requested. (Merge stays human-only.)


八问:

  1. 是什么问题:Codex WS 车道回落到 HTTP/SSE-only supplier 时,客户端轮中断开不会中止 supplier 请求——用户已取消,代理仍持续为看不到的输出付费,直到 supplier 自然完成或 12 分钟 idle watchdog。
  2. 什么原因导致的:ResponsesWsResponse 覆写了 socket getter,但 Node 原生 OutgoingMessage.destroy() 只读内部 socket slot,不 emit 'close';relay 仅在 'close' 且 !writableEnded 时 abort 上游,断连信号在该车道结构性丢失。
  3. 影响范围与严重程度:仅「WS 客户端 + HTTP/SSE-only supplier」组合(codex-responses-ws 与 opencode WS 两个 codec 共用此 sink);用户轮中取消/断连时触发;触发期间 token 持续消耗至 watchdog(默认 12 min)。p2:可用、特定条件成本泄漏、有界。
  4. 怎么修复的:让 destroy() 在 exchange 未结束时补发一次 'close',对齐仓库既有 socket-death 契约(InlineRes.destroy() 同款);destroy 后的 end() 不再二次 emit;relay 零改动;测试文件移除 masking hook 并强化 teardown(closeAllConnections),回归从「挂起 shard」变为「数秒内失败」。
  5. 是否有回归问题:逐项审计——正常完成路径保持('close' 仍恰好一次);destroy() 四个调用点(onClose / shutdown / [Bug] Codex Responses WebSocket blocks long sessions at 100 MiB before ACP compression, including small incremental continuations #4804 anchor-fallback / 输出缓冲上限)语义均为「放弃本轮 exchange」,abort 上游在每种场景都正确([Bug] Codex Responses WebSocket blocks long sessions at 100 MiB before ACP compression, including small incremental continuations #4804 路径原有同类泄漏,顺带堵上);terminal frame 后、end() 前客户端关闭现在计为 mid-stream disconnect(见下方行为变更披露);无 fail-fast 变吞错、无阈值/时序/默认值漂移、wire 形状未动。
  6. 是否破坏缓存:缓存/重试/增量游标/去重键均不受影响;noteClientAbort 在该车道变得可达(供 cache-ledger 记录 abort 关联时间戳),与其他车道一致,属增量非破坏。
  7. 是否改了设计方向:无。把该 sink 对齐到既有契约,数据流不变。
  8. 是否引入了新的配置:无。

行为变更披露(old→new):

  • 轮中客户端断连(HTTP/SSE fallback):supplier 请求「跑到完成或 12 min watchdog」→「毫秒级 abort」(即本修复本身);
  • 输出缓冲上限 / [Bug] Codex Responses WebSocket blocks long sessions at 100 MiB before ACP compression, including small incremental continuations #4804 anchor-fallback / shutdown 路径:在途 egress「放任运行」→「立即 abort」(同机制族);
  • 收到 terminal frame 后、end() 前关闭:原先不计断连 → 现计为 mid-stream disconnect(log + abort)。窗口极小(supplier 已发 terminal event 仅延迟 EOF);代价 = 跳过本次 checkpoint,下轮全量重发(标准恢复);与 HTTP 车道同窗口语义一致。保留 !this.ended guard 的取舍维持上一轮结论。

判定:CI 绿后可合并。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status update: CI on 3307bf43 is now fully green — 30/30 checks passed, 0 failed. All local verification from my previous comment stands. The PR is ready for human merge.

@ranxianglei
ranxianglei merged commit 8def027 into ranxianglei:master Nov 10, 2026
84 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1154-master.8def027 (rolling master tag).

npm install -g billion-context@0.0.1154-master.8def027  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex WS lane over an HTTP/SSE-only supplier: a client disconnect never aborts the supplier request, and the test that pins it is masked

2 participants