Repository navigation
[PR] fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) - #6301
ranxianglei wants to merge 6 commits into
Conversation
…d configurable WS idle watchdog (#6171) - Failure log lines now carry phase, elapsed_ms since the last create frame, and a sanitized cause: error_class/error_message for transport errors, or close_code+was_clean for closes. The upstream-set close reason text is never logged (same untrusted-content contract as the stream-phase leak guard). The synchronous follow-up close event is no longer dropped: it is observed for one informational second line, then listeners are cleaned. Each exchange still ends in exactly one terminal state. - New server-side idle watchdog on the WS leg (handshake + inter-event, any upstream frame re-arms): a fully stalled exchange previously hung until the client gave up; now it terminates with one synthesized client error frame, an 'event=timeout reason=idle-timeout' line, and the connection drop so the next turn reconnects fresh. Default 600000 ms (10 min, owner-set on #6171), knob network.wsIdleTimeoutMs / BILI_WS_IDLE_TIMEOUT_MS. - ResponsesWsResponse logs its synthesized client terminal frame; a session logs recovery=ok when the next exchange completes after a failed one, so one conn/session stamp correlates failure -> client presentation -> recovery. - Handshake timeout rejects with a distinct message (not a refusal, so the #3342 HTTP fallback path and its exact refusal string are untouched).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10152-pr.6301.0d7ab01This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr6301.tgz
npm install -g packageInstall instructions are refreshed on each push. |
lifecycle-ledger-guard fails because src/knobs.ts reads the new BILI_WS_IDLE_TIMEOUT_MS env var but docs/lifecycle.md has no row for it. Add the permanent-registry entry next to its WS sibling so the guard passes again. Doc-only; no behavior change.
[bot] 🏷 Review of PR #6171 — codex-responses-ws pre-first-event diagnostics + configurable WS idle watchdog. 查重: 不重复 — 本线程即 #6171 的唯一修复 PR,未发现并行同题 PR(我的修复也直接叠进本分支,未另开)。 What I didPulled the head branch and read the full diff against master, reasoned the changed control flow across every failure scenario (transport loss / clean close / client abort / full stall), assessed the new + updated tests, and pulled CI status live from GitHub (not the stale mirror cache).
CI (verified live, single GitHub fetch of the head SHA)Before my fix: all test shards green; 35 success / 13 skipped / 1 failure = Root cause of the one failure (deterministic, confirmed locally): the PR adds literal Fixed on the spot (小问题直接修 — no decision needed): added one permanent-registry row under §2.3 网络调优 next to its WS sibling. Pushed to the existing branch (fast-forward 结构化八问
行为变更披露(old → new,Deliverables #4)
遗留观察(非阻断)
判定代码质量高、测试扎实(含隐私契约与 conn 关联戳断言)、CI 除已修的 |
Resolve the process.yaml conflict: master added desc_ja to the network entries while the PR added network.wsIdleTimeoutMs (BILI_WS_IDLE_TIMEOUT_MS). Kept master's desc_ja lines AND the PR's new entry, adding the missing desc_ja for the new entry (gen-config-docs requires all three desc langs). Regenerated docs/configuration.ja.md from the merged seed.
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Merge conflict resolved and pushed. PR #6301 (closes #6171) is now What I did this run:
Note: I did not run the full local test suite (it needs a kernel build + 结构化八问(中文,供人工复核)
判定:冲突已解决并推送,PR #6301 现为 |
(ai: vllm-qwen/qwen3.8-27b) CI red-fix run — PR #6301 (head 8a0348f) What I did this run:
PR functional changes verified intact in the merged tree: Expected outcome: the notice-registry failures are fixed by the ci-fixes merge. The Windows EBUSY flake (store-zstd + metadata-fold) will remain red until the sentinel lands its fix on ci-fixes; a re-run after that convergence should go green. No new red was introduced by this PR. 八问
结论:本 PR 侧 CI 红已全部归因并处置——notice-registry 红由 ci-fixes 合并修复,Windows EBUSY flake 为继承性失败、由哨兵在 ci-fixes 收敛。本 PR 无自身回归,等 Windows EBUSY 面落地后重触发即绿。 |
(ai: vllm-qwen/qwen3.8-27b) CI auto-fix lane, attempt 1/3 — verdict: inherited flake, not a PR regression. No code change made this run. I re-verified the live check state first (mirror cache, head Root cause (from the logs): all 10 failing assertions are the identical Why this is NOT this PR's fault:
Action taken (per the CI-failure-coordination policy, 2026-10-24):
This PR goes green once the sentinel lands the Windows EBUSY fix in ci-fixes (the affected suites pin 结构化八问
判定:继承性 flake,非本 PR 回归;等哨兵把 Windows EBUSY 修复落到 ci-fixes 后本 PR 转绿。本轮无代码改动。 |
…lict Union resolution: keep #6171 watchdog vars (failureEvents, startedAt) AND master's send-sequence var (send). All three are required in the exchange block (startedAt/failureEvents for the idle watchdog + failure diagnostics; send for the send-sequence cache-diagnostic logging). Verified: typecheck clean, build clean, targeted WS tests 45/45 pass.
(ai: vllm-qwen/qwen3.8-27b) CI auto-fix lane, attempt 1/6 (conflict resolution) — DONE: merge conflict resolved and pushed. What I did this run:
Live state after push (mirror cache, fresh): sha 结构化八问
判定:合并冲突已解决并推送( |
|
(ai: vllm-qwen/qwen3.8-27b) Review verdict: APPROVE — the PR does exactly what it claims, and the tests have real teeth. (Cannot submit a formal APPROVE review via the API because the PR is authored by the same account; posting the review content as a comment.) Correctness (all four claimed changes verified):
Tests: 4 new tests in Behavior changes (disclosed in PR body):
Config surface: new knob No kernel/ touched — no kernel-change declaration needed. 八问(中文结构化结论)
判定:可合(APPROVE)。 |
✅ Review done: PR #6301 (GitHub) — APPROVE. All 4 claimed changes verified against the full diff (diagnosable two-line failures, privacy contract, client correlation, configurable idle watchdog); 54/54 targeted tests green locally (issue6171 4 + responses-websocket 41 + codex-responses-ws 9), typecheck + build clean. No kernel/ touched (no kernel-change needed), no small findings to fix (minor observations are nits below the fix-effort bar), no 🔶 需要决策. Full review comment posted on the PR. |
What & why
Issue #6171: six log lines
upstream failed phase=await-first-event event=error close_code=none received_events=0with no way to tell WHICH upstream condition fired, what the client saw, or whether the session recovered. Root cause: undici fireserrorand then SYNCHRONOUSLYclose(1006)when no close frame arrived — the old handler removed both listeners on the first event, so the close code was never observed (close_code=none) and clean closes dropped their reason entirely.This PR:
upstream failedline now carriesphase,elapsed_ms(since the lastresponse.createframe), and a sanitized cause:error_class=<Name> error_message=<msg>for transport errors, orclose_code=<n|none> was_clean=<t|f>for closes. The synchronous follow-upcloseevent is no longer dropped — it becomes ONE informational second line under the same conn/session stamp, then listeners are cleaned. Each exchange still ends in exactly one terminal state (audited: fail/end/destroy/settled all guarded).ResponsesWsResponselogs its synthesized client terminal frame (client terminal=synthesized code=<c> status=<n>), and a session logsrecovery=ok after_ms=<n>when the next exchange completes after a failed one — so one conn/session stamp correlates failure -> client presentation -> recovery.event=timeout reason=idle-timeoutline, one synthesized client error frame, and the connection drop so the next turn reconnects fresh. Any upstream frame re-arms it, so steady streaming never trips it.Behavior changes (disclosed)
upstream failed phase=X event=Y close_code=none received_events=Nupstream failed phase=X event=Y <detail> elapsed_ms=N [received_events=M], max 2 lines/exchange (internal log; old-format greps break)wsIdleTimeoutMs(default 600000 ms) with one synthesized client error frame + connection dropResponses WebSocket handshake timed out— deliberately NOT a refusal, so the #3342 HTTP fallback path and its exact refusal stringResponses WebSocket handshake failedare untouchedConfig surface
New knob (owner sign-off in-thread on #6171):
network.wsIdleTimeoutMs/BILI_WS_IDLE_TIMEOUT_MS, default600000(10 min), env > file > default resolution viasrc/knobs.ts; non-integer or non-positive values fall back to the default. The existing surface cannot express it:BILI_UPSTREAM_TIMEOUT_MSgoverns the HTTP leg only — the WS leg had no server-side stall budget at all (the HTTP lane's two-phase watchdog has no WS equivalent). Docs updated: seed entry + generated tables (en/zh) + env-detail rows (en/zh).Tests
tests/issue6171-ws-failure-diagnostics.test.ts: (A) peer terminate before first event -> exactly two correlatedupstream failedlines (error + follow-up close 1006/was_clean=false), exactly ONE client error frame, next turn completes withrecovery=ok; (B) clean upstream close keeps its code on one line AND the free-text reason never reaches the log; (C) idle-timeout trigger -> one timeout line + recovery; (D) knob parsing (default / valid / garbage / negative).tests/responses-websocket.test.tsto the new format; the Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 refusal-fallback regression test still passes unchanged.Closes #6171