Skip to content

[PR] fix: close pending WebSocket upgrades during server shutdown - #7018

Closed
Afloat16 wants to merge 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-17_close-pending-ws-upgrades
Closed

Afloat16 wants to merge 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-17_close-pending-ws-upgrades

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The bridge tracks sessions only after destination admission. A claimed HTTP upgrade socket waiting on the asynchronous check is absent from both the session set and wss.clients. Shutdown can leave that socket open, and a successful late check creates a new session after the teardown pass.

Change

Keep a set of pending upgrade sockets from claim through admission. Shutdown closes these sockets immediately and marks the bridge as closing. Late checks cannot admit a new session. Pending entries and their temporary close listeners are removed on settlement or client close.

Existing destination policy, frame handling, codec behavior, and admitted-session shutdown remain intact.

Current-master verification

Updated in c74bf9b8e6ddaa1f4f65090c838259cd72571ed7 by a normal merge of current master 6a641f378537e06e7430ec90c9abf035ffa12377 (0.1.216), retaining the original PR commit 6b08f615ddd3f50c1a632277bb9e24373615309f as the first parent. The original two-file patch is byte-for-byte unchanged relative to the new base: src/ws-bridge.ts and tests/ws-bridge-pending-close.test.ts remain the complete PR scope.

This includes the upstream fixes discussed in the maintainer review: the architecture module-map entry from 8ed56be997c292584844ab8b482c04f8fe77256f and the schedule-independent fold probe from 7247410976508f4dedf42afb89450c2041141c0b, both already in current master.

Validated tree: ffe9d21b1e31ff7f48d550fdbe33f2df9640360a. Linux, Node 24.22.1; dependencies installed using npm ci --ignore-scripts --no-fund --no-audit.

  • Both new pending-close regressions fail against current master's unchanged bridge implementation: late admission reports opened rather than closed, and shutdown waits for DNS. Both pass with the PR implementation. These use real loopback WebSocket clients.
  • The eight affected WebSocket suites pass: 81 tests, 0 failures, 0 skips. This includes the two new regressions, generic bridge behavior, the complete responses-websocket.test.ts suite, codex continuation, cache diagnostics, history linearization, HTTP fallback, and XAI error envelopes.
  • tests/issue2929-codex-fold-double-miss.test.ts passes with the CI preload chain: 1 test passed; the nudge is observed at request 7 and the fold at request 8. This is the scenario that failed in the prior CI run.
  • node tools/check-architecture-docs.mjs passes. The original PR head was checked separately and reproduces the missing module-map entry.
  • npm run typecheck and npm run build pass.
  • The PR diff against current master passes git diff --check. No production or test edits were made beyond bringing in the already-merged upstream changes.

Targeted tests use the repository's storage, temporary-directory, and zone-slot preloads. The published head passed the core CI run: all 27 validation jobs succeeded, including the Linux Node 22/24 and Windows Node 22 test shards, static gates, architecture documentation and the MySQL contract. The conditional red-signal job was skipped.

The branch protection also requires test (windows-latest, 24, 1), test (windows-latest, 24, 2) and test (windows-latest, 24, 3). Those checks are absent from the core matrix. The workflow's existing ci:full label enables the full matrix; these remaining checks still need to run before merge.

No kernel, wire format, configuration-schema, release, or dependency changes are introduced by the PR's diff against current master.

Fixes #7017

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 镜像 open [PR] 行与 GitHub 全仓搜索均无其他 PR 处理同一主题(WS upgrade 关闭期 pending 跟踪);ladder 相关发现另见下方 #7047,亦无既有 issue。

标签: bug, codex, opencode, scope/wiring, p3 — 缺陷修复,位置在 WS 客户端连接边界(协议接线);受影响车道为 opencode/codex 的 Responses WS;严重度为边缘时序(仅关服与 in-flight 目的地检查重叠时触发)。

Review

Bug verified real on base. I checked out base 301e1a5459 (0.1.215, current master tip — not stale), added your two new tests, and both fail on base exactly as claimed:

  • server close does not admit a WebSocket whose destination check has not settled → 'opened' !== 'closed' (the late check admitted a session after the teardown pass);
  • server close releases an upgrade socket without waiting for DNS admission → 'still waiting for DNS' (the server.close callback is pinned while the socket waits on DNS).

On your head (6b08f615dd): 2 new + 3 generic bridge (tests/ws-bridge.test.ts, incl. the existing "server close shuts every session down and terminates peers") + 45 Responses-WS tests = 50 passed, 0 fail. npm run typecheck clean, npm run build OK, git diff --check clean.

Code review (src/ws-bridge.ts, +21/−1): the pending set + one-way closing latch is the right shape. Notes from walking the touched paths:

  • The latch is set in both the server.close override and the close backstop — correct, since the post-bind port ladder (src/server.ts:1539) closes the same server object and re-listens it.
  • Re-checking closing || socket.destroyed after the async verdict, before admission, closes the late-success race; the check and wss.handleUpgrade are synchronous after the await, so no new race is introduced.
  • The 403/502 rejection ordering is user-behavior-identical when not closing (when closing, the socket is already destroyed, so the old write was a no-op anyway).
  • Entry cleanup on both settlement and client close is complete; no listener leak.
  • Tests use real loopback clients with a mocked dns.lookup — deterministic, no port-range assumption, good negative direction (a settled check still admits normally).

One pre-existing bug found while reviewing (not introduced by this PR): after the post-bind port ladder closes and re-listens the same server (src/server.ts:1539, reachable when postBindCanLadder), the WS compression lane is permanently dead — on base the close backstop runs wss.close() and every later upgrade gets a 503 from ws (websocket-server.js:385, state > RUNNING); on your head the closing latch destroys the sockets earlier. Same dead-lane outcome either way, verified by a repro test that fails identically on base and head. This is a different symptom + mechanism from #7017, so per policy it is filed separately: mirror issue #7047 (p2). Your PR does not materially worsen it.

CI status (verified live, 07:24 UTC): 1 failure, 12 in progress, 6 success, 11 skipped. The one failure — architecture-docs — is inherited, not yours: master 301e1a5459 itself fails the same check locally (reference/architecture.md missing: src/responses-system-sections.ts), and the fix already sits on origin/ci-fixes (8ed56be997). I merged ci-fixes into your branch locally (merge commit 1195645629; the docs check and all bridge tests are green on that merge) but cannot push it: the fork rejects my push (403) and maintainer_can_modify can only be changed by you. → Please enable "Allow edits by maintainers" on the PR (or rebase onto master yourself); once the branch carries the ci-fixes merge, CI will go green. Logged on #4599.

Disclosure (my incident, not yours): while probing the mirror's issue-create API shape I accidentally created a garbage mirror-local issue #7046 (empty body, never synced to GitHub). Disclosed and left for you to close there per policy; the real finding is #7047.

结构化八问

  1. 是什么问题:WS 桥接在关服时,已认领但目的地异步检查(DNS 等)未落定的升级套接字不在任何跟踪集合里——关服可能把它留开(拖住 drain 回调),且检查迟到成功会在 teardown 之后新建会话。
  2. 什么原因导致的:桥接只在异步准入完成后才跟踪会话;claim 与准入之间的空窗没有跟踪集合,异步检查落定后也没有复查关服状态。
  3. 影响范围与严重程度:仅关服与 in-flight 目的地检查重叠的窄窗口触发。后果:drain 回调被 pending 套接字拖住(上限 = DNS 重试预算,有界);迟到会话逃过 teardown pass(在途交换不被失败化、上游传输不释放),直到客户端断开/进程退出。p3 边缘时序。
  4. 怎么修复的:pending 套接字从 claim 到准入全程入集合;server.close(及 close 兜底)置单向 closing latch 并销毁全部 pending 套接字;异步检查落定后先复查 latch 与 socket.destroyed 再准入,迟到成功不再建会话;条目与临时 close 监听在落定或客户端关闭时清理。
  5. 是否有回归问题:逐路径过——正常升级准入不变(通用桥接测试 + 45 个 Responses WS 测试全过);已准入会话的关服行为不变(既有 "server close shuts every session down" 测试过);403/502 拒绝的用户可见行为在非关服时逐字节不变;drain 由「可能被 pending 拖住」变「立即返回」= 本修复本身。另发现既有 ladder 缺陷(关服重监听后 WS 车道永久死,base 上已坏,见 [PR] fix(ws-bridge): re-arm wss lazily after a post-bind ladder re-listen #7047)——非本 diff 引入,不恶化。本 diff 无回归。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标/去重键)。
  7. 是否改了设计方向:否——桥接架构(claim → 异步准入 → handleUpgrade → session)不变,仅补关服跟踪集合。
  8. 是否引入了新的配置:否。

结论:代码可合并(mergeable)——修复正确、测试有牙、无回归;CI 当前唯一红为继承性 architecture-docs(master 同红,修复已在 ci-fixes),待你开 maintainer edits(或自行 rebase)后 CI 重跑即绿。

Merge current master, including the architecture module map and schedule-independent fold probe fixes. Preserve the original WebSocket bridge fix and regression patch unchanged.
@Afloat16

Copy link
Copy Markdown
Contributor Author

(ai: OpenAI/GPT-6 Astra Pro)

The branch now includes current master in c74bf9b8, and CI 44047355573 completed successfully with 27 jobs passed. Afloat16's original fix and regression coverage are unchanged.

Three required checks are still absent because that run used the core matrix:

  • test (windows-latest, 24, 1)
  • test (windows-latest, 24, 2)
  • test (windows-latest, 24, 3)

The current workflow enables these jobs when the PR receives the ci:full label. Could a maintainer add ci:full to run the full matrix on this head, then complete the merge review once the required checks pass?

@ranxianglei

Copy link
Copy Markdown
Owner

This fix is included verbatim (original commits and authorship preserved) in the merged batch PR #7074 — the changes are already on master. Closing to avoid a duplicate merge. 该修复已随批量 PR #7074 原样合入 master,为避免重复合并,关闭此 PR。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server shutdown leaves WebSocket upgrades pending during destination admission

2 participants