Skip to content

[PR] fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) - #6301

Open
ranxianglei wants to merge 6 commits into
masterfrom
2026-11-11_ws-failure-diagnostics
Open

ranxianglei wants to merge 6 commits into
masterfrom
2026-11-11_ws-failure-diagnostics

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

Issue #6171: six log lines upstream failed phase=await-first-event event=error close_code=none received_events=0 with no way to tell WHICH upstream condition fired, what the client saw, or whether the session recovered. Root cause: undici fires error and then SYNCHRONOUSLY close(1006) when no close frame arrived — the old handler removed both listeners on the first event, so the close code was never observed (close_code=none) and clean closes dropped their reason entirely.

This PR:

  1. Diagnosable failures — every upstream failed line now carries phase, elapsed_ms (since the last response.create frame), and a sanitized cause: error_class=<Name> error_message=<msg> for transport errors, or close_code=<n|none> was_clean=<t|f> for closes. The synchronous follow-up close event is no longer dropped — it becomes ONE informational second line under the same conn/session stamp, then listeners are cleaned. Each exchange still ends in exactly one terminal state (audited: fail/end/destroy/settled all guarded).
  2. Privacy contract — the upstream-set close REASON text is arbitrary untrusted content and is NEVER logged (same contract as the stream-phase leak guard in tests/responses-websocket.test.ts); standard close codes carry the intent. Error messages are sanitized (control chars stripped, whitespace collapsed, 160-char cap). No frame bodies, no credentials.
  3. Client-side correlation — ResponsesWsResponse logs its synthesized client terminal frame (client terminal=synthesized code=<c> status=<n>), and a session logs recovery=ok after_ms=<n> when the next exchange completes after a failed one — so one conn/session stamp correlates failure -> client presentation -> recovery.
  4. Server-side idle watchdog (new capability, owner-approved in-thread: default 10 min, configurable) — previously a fully stalled WS exchange or handshake hung until the client gave up. Now any stall beyond the budget terminates the exchange with its single terminal state, an event=timeout reason=idle-timeout line, one synthesized client error frame, and the connection drop so the next turn reconnects fresh. Any upstream frame re-arms it, so steady streaming never trips it.

Behavior changes (disclosed)

Change Old New
Failure log line format upstream failed phase=X event=Y close_code=none received_events=N upstream failed phase=X event=Y <detail> elapsed_ms=N [received_events=M], max 2 lines/exchange (internal log; old-format greps break)
Stalled WS exchange/handshake hung until the client aborted terminates after wsIdleTimeoutMs (default 600000 ms) with one synthesized client error frame + connection drop
Handshake timeout rejection n/a (no timeout existed) distinct Responses WebSocket handshake timed out — deliberately NOT a refusal, so the #3342 HTTP fallback path and its exact refusal string Responses WebSocket handshake failed are untouched
Spurious post-abort handshake warn lines could appear after a client abort suppressed (log-only)

Config surface

New knob (owner sign-off in-thread on #6171): network.wsIdleTimeoutMs / BILI_WS_IDLE_TIMEOUT_MS, default 600000 (10 min), env > file > default resolution via src/knobs.ts; non-integer or non-positive values fall back to the default. The existing surface cannot express it: BILI_UPSTREAM_TIMEOUT_MS governs the HTTP leg only — the WS leg had no server-side stall budget at all (the HTTP lane's two-phase watchdog has no WS equivalent). Docs updated: seed entry + generated tables (en/zh) + env-detail rows (en/zh).

Tests

  • New tests/issue6171-ws-failure-diagnostics.test.ts: (A) peer terminate before first event -> exactly two correlated upstream failed lines (error + follow-up close 1006/was_clean=false), exactly ONE client error frame, next turn completes with recovery=ok; (B) clean upstream close keeps its code on one line AND the free-text reason never reaches the log; (C) idle-timeout trigger -> one timeout line + recovery; (D) knob parsing (default / valid / garbage / negative).
  • Updated tests/responses-websocket.test.ts to the new format; the Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 refusal-fallback regression test still passes unchanged.
  • Local targeted runs green: issue6171 file + responses-websocket (45/45) + codex-responses-ws (9/9); typecheck + build clean. Full-suite verdict = remote CI on this head.

Closes #6171

…d configurable WS idle watchdog (#6171)

- Failure log lines now carry phase, elapsed_ms since the last create frame,
  and a sanitized cause: error_class/error_message for transport errors, or
  close_code+was_clean for closes. The upstream-set close reason text is
  never logged (same untrusted-content contract as the stream-phase leak
  guard). The synchronous follow-up close event is no longer dropped: it is
  observed for one informational second line, then listeners are cleaned.
  Each exchange still ends in exactly one terminal state.
- New server-side idle watchdog on the WS leg (handshake + inter-event, any
  upstream frame re-arms): a fully stalled exchange previously hung until
  the client gave up; now it terminates with one synthesized client error
  frame, an 'event=timeout reason=idle-timeout' line, and the connection
  drop so the next turn reconnects fresh. Default 600000 ms (10 min,
  owner-set on #6171), knob network.wsIdleTimeoutMs / BILI_WS_IDLE_TIMEOUT_MS.
- ResponsesWsResponse logs its synthesized client terminal frame; a session
  logs recovery=ok when the next exchange completes after a failed one, so
  one conn/session stamp correlates failure -> client presentation -> recovery.
- Handshake timeout rejects with a distinct message (not a refusal, so the
  #3342 HTTP fallback path and its exact refusal string are untouched).
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-11_ws-failure-diagnostics (ae0df9d)

Latest build — install this exact version

npm install -g billion-context@0.0.10152-pr.6301.0d7ab01

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
ae0df9d billion-context@0.0.10152-pr.6301.0d7ab01 build
8a0348f billion-context@0.0.9692-pr.6301.9d753e8 build
7f50cc2 billion-context@0.0.9648-pr.6301.ca59ce2 build
fdab912 billion-context@0.0.9210-pr.6301.09c01e5 build
40b480f billion-context@0.0.9192-pr.6301.e75076a build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6301.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) [PR] fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) Nov 11, 2026
lifecycle-ledger-guard fails because src/knobs.ts reads the new
BILI_WS_IDLE_TIMEOUT_MS env var but docs/lifecycle.md has no row for it.
Add the permanent-registry entry next to its WS sibling so the guard
passes again. Doc-only; no behavior change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

Review of PR #6171 — codex-responses-ws pre-first-event diagnostics + configurable WS idle watchdog.

查重: 不重复 — 本线程即 #6171 的唯一修复 PR,未发现并行同题 PR(我的修复也直接叠进本分支,未另开)。

What I did

Pulled the head branch and read the full diff against master, reasoned the changed control flow across every failure scenario (transport loss / clean close / client abort / full stall), assessed the new + updated tests, and pulled CI status live from GitHub (not the stale mirror cache).

  • Code (src/responses-ws.ts, src/knobs.ts, src/config.ts): high quality. sanitizeDetail/describeFailureEvent convert undici error/close events into a redacted single line (close_code=/was_clean=/error_class=/error_message=) and never log the upstream close reason text (privacy contract). idleWatchdog is a re-armable, unref()'d timer re-timed on every upstream frame, so only a full stall trips it. The handshake/exchange state machines cap follow-up logging at 2 lines, keep the close listener alive on error so undici's synchronous close(1006) is observed, and use a distinct "handshake timed out" reject string deliberately different from the refusal string so the Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 HTTP/SSE fallback does NOT fire on a stalled upgrade. Terminal-state invariants hold (each exchange ends in exactly one terminal state; end() synthesizes a client error frame only when no real terminal arrived).
  • Tests (tests/issue6171-ws-failure-diagnostics.test.ts, real http server + WebSocketServer mock): genuinely sharp, not tautological. Pins (A) raw-TCP drop → exactly 2 correlated failure lines sharing one conn stamp + 1 synthesized client error frame + next-turn recovery=ok; (B) clean close → exactly 1 line AND doesNotMatch(/private-upstream-close-reason-6171/) (the reason-leak contract); (C) idle timeout fires + recovers; (D) knob parse (default/valid/garbage/negative). The #3342 refusal regression in responses-websocket.test.ts is preserved (still asserts reason=ws-handshake-refused + checkpoint reset). These would catch a dropped follow-up close, a leaked reason, a dead watchdog, or a knob regression.
  • Two compression modes: N/A — this is the codex Responses-WS transport lane, not a plugin/proxy compression path; no wire-rebuild/system/nudge surface touched.

CI (verified live, single GitHub fetch of the head SHA)

Before my fix: all test shards green; 35 success / 13 skipped / 1 failure = lifecycle-ledger-guard / 1 gate job queued.

Root cause of the one failure (deterministic, confirmed locally): the PR adds literal BILI_WS_IDLE_TIMEOUT_MS to src/knobs.ts but never registers it in docs/lifecycle.md. scripts/lifecycle-ledger-guard.mjs requires every BILI_* var read from src/ to have a ledger row, so it exits 1 with missing: [BILI_WS_IDLE_TIMEOUT_MS]. This is purely this PR's omission (master lacks the var too).

Fixed on the spot (小问题直接修 — no decision needed): added one permanent-registry row under §2.3 网络调优 next to its WS sibling. Pushed to the existing branch (fast-forward 40b480f..fdab912), commit docs(lifecycle): register BILI_WS_IDLE_TIMEOUT_MS in §2 ledger (#6171). Guard now passes locally on the pushed tree (OK: … every BILI_* env var read from src/ is registered, exit 0). It should turn green on the CI re-run; I could not yet observe the re-run complete.

结构化八问

  1. 是什么问题: codex-responses-ws 车道在收到上游第一个事件之前的失败(握手被拒 / 传输断链 / 干净关闭 / 完全 stall)此前零日志、且无界挂起直到客户端自行 abort;本 PR 让其可诊断并加空闲看门狗。
  2. 什么原因导致的: 旧控制流在 pre-first-event 窗口既没有失败事件采集(undici 的 error/close 未进诊断日志)也没有空闲超时兜底,故失败静默且挂起。根因在 src/responses-ws.ts 的 handshake/exchange 缺少失败观测与超时终止机制。
  3. 影响范围与严重程度: 仅 codex-responses-ws(Responses WS 传输)车道;触发条件 = 上游在首事件前拒绝升级/断链/干净关闭/完全 stall。p2 —— 非不可用(有客户端 abort 兜底),但「失败不可见 + 无界挂起」是明显的排障/体验缺陷。
  4. 怎么修复的: ①sanitizeDetail/descriptionFailureEvent 把 error/close 事件转成脱敏单行(绝不含 reason 文本);②可重入 unref() 看门狗,每帧重新计时、仅整段全停才触发;③handshake/exchange 状态机把 follow-up 日志限两行、error 时保留 close 监听以捕获同步 close(1006)、超时用独立 reject 串刻意区别于 refusal 串避免误触 Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 回退;④end() 在无真实终态时合成 client error frame(透传 error.code)。
  5. 是否有回归问题: 逐项核过既有行为 —— Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 的 isHandshakeRefusal(精确匹配 "Responses WebSocket handshake failed")未动,超时串刻意不同故不触发回退;clean-close 仍只记一行;每个 exchange 恰好一个终态不变量成立;post-abort 假 warn 属日志噪声抑制而非语义变更。唯一实际回归:本 PR 曾令 lifecycle-ledger-guard 变红(新增 env 未登记),已当场补一行修绿。有意行为变更(见下披露):默认 600000ms 把原先「无界挂起」变为「10 min 封顶后主动终止」。
  6. 是否破坏缓存: 不涉及 —— 改动全在 WS 传输观测/超时层,不触碰前缀缓存、cache ref、增量游标、去重键。
  7. 是否改了设计方向: 不涉及架构转向;是既有车道内的诊断增强 + 一个新网络调优旋钮,数据流不变。
  8. 是否引入新配置: 是 —— network.wsIdleTimeoutMs / BILI_WS_IDLE_TIMEOUT_MS,默认 600000ms(10 min),env>file>default(src/knobs.ts),非正整数回退默认。已在 configuration.md(en/zh)+ website 登记,owner 已在 [Codex CLI][v0.1.213] 6 WebSocket failures before the first event, with no correlation between error cause and recovery | [Codex CLI][v0.1.213] WebSocket 首事件前失败 6 条,缺少错误原因与恢复结果关联 #6171 签核;CI 侧缺失的 lifecycle.md 登记由我补齐。

行为变更披露(old → new,Deliverables #4)

  • 失败日志格式: 由旧格式 → 结构化 upstream failed phase=… event=… elapsed_ms=… received_events=… + close_code=/error_class= 等(内部日志;依赖旧格式的 grep 会失效)。
  • stalled exchange/handshake: 旧 = 挂起直到客户端 abort → 新 = wsIdleTimeoutMs(默认 600000ms)后主动终止 + 合成 client error frame + 断连。
  • 新增独立 reject: Responses WebSocket handshake timed out,刻意不等于 refusal 串,故 Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 回退与既有精确匹配串均不受影响。
  • post-abort 假 warn: 抑制(纯日志)。
  • caveat: 任意两帧之间静默 >10 min 也会触发看门狗(stream:true 的 codex 车道极不可能出现;可调小/调大,escape hatch 存在)。

遗留观察(非阻断)

  • describeFailureEvent 里 event.error.name 未经 sanitizeDetail(低风险:undici 错误名是固定串);如想更严可对 name 也走一遍 sanitize。
  • sendCreate 引用了在其后声明的 watchdog(TDZ-safe,所有调用点都在初始化之后),略脆弱但正确。
  • finally 块把「非终态」一律记为潜在 upstream 失败做 recovery 计时,实际也可能包含客户端主动断开 —— 纯诊断启发式,低 stakes。

判定

代码质量高、测试扎实(含隐私契约与 conn 关联戳断言)、CI 除已修的 lifecycle-ledger-guard 外全绿、所有行为变更均已披露。建议合并 —— 待该 guard 在新 head(fdab912)上重跑转绿即可合。我没有合并(系统规则禁止 agent 合并 PR),请 owner 自行合并: #6171

ci-check added 2 commits November 14, 2026 02:01
Resolve the process.yaml conflict: master added desc_ja to the network
entries while the PR added network.wsIdleTimeoutMs (BILI_WS_IDLE_TIMEOUT_MS).
Kept master's desc_ja lines AND the PR's new entry, adding the missing
desc_ja for the new entry (gen-config-docs requires all three desc langs).
Regenerated docs/configuration.ja.md from the merged seed.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Merge conflict resolved and pushed. PR #6301 (closes #6171) is now mergeable: true — the dirty state is cleared; it's only blocked on human approval to merge.

What I did this run:

  1. Merged origin/master into the PR branch (2026-11-11_ws-failure-diagnostics) — a merge, not a rebase, so no force-push was needed.
  2. Resolved the single conflict in website/config-reference/process.yaml (the config-docs seed). Root cause: the PR branched before master added a desc_ja (Japanese) field to every config entry. Master added desc_ja to the existing network entries; the PR added a brand-new network.wsIdleTimeoutMs entry (en/zh only). I kept both — master's desc_ja lines and the PR's new entry — and wrote the missing desc_ja for the new entry (gen-config-docs hard-fails if any entry lacks all three desc langs; tools/gen-config-docs.mjs:114).
  3. Master advanced mid-run (99c22d283f → e1f31ad9d5, the pure-SQLite storage default + bounded-boot work). I merged that delta on top. It touched only the storage region (process.yaml lines ~392+, knobs.ts ~981+), which is disjoint from the PR's network changes, so it merged clean with no new conflicts.
  4. Regenerated docs/configuration.ja.md from the merged seed; check-coverage passes (205 env names + 63 top-level keys documented, i18n ok, all refs resolve).
  5. Verified no conflict markers anywhere in the tree, and the PR's functional changes are intact (wsIdleTimeoutMs knob in src/config.ts:2305, src/knobs.ts:107-119, src/responses-ws.ts:439/572/578).
  6. Pushed fdab91268c → 7f50cc21cc.

Note: I did not run the full local test suite (it needs a kernel build + npm install and is the remote CI's job per §7.2); the conflict was docs-only and all code files 3-way-merged cleanly. Remote CI on the pushed head is the authoritative gate.


结构化八问(中文,供人工复核)

  1. 是什么问题:PR [PR] fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) #6301(修 [Codex CLI][v0.1.213] 6 WebSocket failures before the first event, with no correlation between error cause and recovery | [Codex CLI][v0.1.213] WebSocket 首事件前失败 6 条,缺少错误原因与恢复结果关联 #6171:让 codex-responses-ws 首事件前失败可诊断 + 可配置 WS 空闲看门狗)与 master 冲突(merge state: dirty),无法进入合并。
  2. 什么原因导致的:PR 在 master 给所有配置项加 desc_ja(日语)字段之前分叉;master 侧给既有 network 条目加 desc_ja,PR 侧新增 network.wsIdleTimeoutMs 条目(只有 en/zh),两者在 process.yaml 的 network 段文本重叠 → 冲突。
  3. 影响范围与严重程度:仅阻塞该 PR 的合并(CI 自动车道),不影响任何运行中的实例。严重程度:阻塞性(PR 进不了人工合并),非功能性回归。
  4. 怎么修复的:把 master 合进 PR 分支(merge,非 rebase,故无需 force-push)。唯一冲突在 website/config-reference/process.yaml:保留 master 的 desc_ja 行 + PR 的新条目,并为新条目补上缺失的 desc_ja(gen-config-docs 强制三语齐全)。master 运行中又前进(纯 SQLite 存储默认 + bounded-boot),其 delta 只动 storage 段(与 PR 的 network 段不重叠),干净合入。重新生成 docs/configuration.ja.md,check-coverage 通过。
  5. 是否有回归问题:无。冲突只在文档 seed 文件;代码文件(responses-ws.ts/knobs.ts/config.ts)三方合并无冲突,PR 的 wsIdleTimeoutMs 功能改动完整保留;全树无冲突标记。代码语义回归由远程 CI 全量套件把关(本地未跑全量,按规范交给 CI)。
  6. 是否破坏缓存:不涉及。纯冲突解决 + 文档再生成,不触碰缓存/重试/增量游标逻辑。
  7. 是否改了设计方向:不涉及。无架构/边界/数据流转向。
  8. 是否引入了新的配置:不涉及。本次未新增配置;PR 本身的 BILI_WS_IDLE_TIMEOUT_MS 是 PR 原有内容,非本次冲突解决所加。

判定:冲突已解决并推送,PR #6301 现为 mergeable: true(等待人工批准合并)。本次 CI 自动车道任务完成。

…ure-sqlite pin (sentinel fix c6369fb; inherited-red from master #6480)
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI red-fix run — PR #6301 (head 8a0348f)

What I did this run:

  1. Re-verified live check state. The mirror cache (upstream-checks?number=6301) is stale (fetched 22:33Z, before the test shards finished on the prior head) — its 30-check list does not include the test-matrix shard jobs, so it cannot verify shard state. I fetched the 4 failing shard job logs via the GitHub API (allowed: "fetch check logs via the API") and confirmed the per-shard failures.

  2. Confirmed all 3 failure families are INHERITED from master (master is red on the identical tests — verified by fetching master's own failing shard logs, run 43469650189):

  3. Merged origin/ci-fixes into the PR branch (merge commit 8a0348f, clean — only touched tests/notice-registry.test.ts, +5 lines). This converges the sentinel's notice-registry fix (c6369fb: pin BILI_STORAGE_UNIFIED=sqlite in the notice suite). Pushed 7f50cc2 → 8a0348f (fast-forward, no force-push; remote head verified = 8a0348f).

  4. Did NOT fix the Windows EBUSY flake in-branch — it is inherited (master red on the same tests) and the sentinel is tracking it as a convergence item on ci-fixes (flake-vs-real confirmation pending, per CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 floor 1333). Per CI-coordination policy (owner 2026-10-24): inherited red → merge ci-fixes + push, do not re-fix master-level failures in the PR branch.

  5. Posted status to CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 (CI control center): PR [PR] fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) #6301 red, merged ci-fixes, sha=8a0348f321; the Windows EBUSY face is inherited and tracked by the sentinel.

PR functional changes verified intact in the merged tree: src/config.ts:2305 (wsIdleTimeoutMs?: number), src/knobs.ts:107 (WS_IDLE_TIMEOUT_MS_DEFAULT=600_000), src/responses-ws.ts (4 usage sites). The PR touches only the WS idle-watchdog surface + docs — none of the notice-gate or storage/metadata dependency chains.

Expected outcome: the notice-registry failures are fixed by the ci-fixes merge. The Windows EBUSY flake (store-zstd + metadata-fold) will remain red until the sentinel lands its fix on ci-fixes; a re-run after that convergence should go green. No new red was introduced by this PR.


八问

  1. 是什么问题:PR [PR] fix: make codex-responses-ws pre-first-event failures diagnosable; add configurable WS idle watchdog (#6171) #6301 的 4 个 test 分片 CI 红(ubuntu 22/24 ×2、windows 22 1/22 4),共 3 类失败:notice-registry ×2、metadata fold Scheduled full test failed: ee0dcc7c | 定时全量测试失败: ee0dcc7c #5506 ×3、store BILI_PERSIST_ZSTD ×10。
  2. 什么原因导致的:全部为继承性失败——master(e1f31ad,run 43469650189)自身在同一批测试上同样红。① notice-registry:CI 预载把 BILI_STORAGE_UNIFIED 钉成 legacy,使 sqliteDefaultNoticeActive 门为 false,storage-sqlite-default 通知缺失;②/③ Windows EBUSY unlink index.db 文件锁 flake(windows-22-1 lane randomly red: during test cleanup, unlink index.db hit an EBUSY external lock, couldn't survive tmp-rm | windows-22-1 车道随机红:测试清理期 unlink index.db 遇 EBUSY 外部锁,熬不过 tmp-rm 的 9.6 秒重试梯子(#2799/#2908 族复发) #4347 家族),store-zstd 与 metadata-fold 各命中。本 PR 只改 wsIdleTimeoutMs + docs,不触及任何一条失败路径。
  3. 影响范围与严重程度:仅 CI 检查矩阵(信号级),非用户侧功能问题;本 PR 无新增红。
  4. 怎么修复的:按 CI 协调政策(2026-10-24)——master 红 = 继承 → git merge origin/ci-fixes(收敛哨兵的 notice-registry 修复 c6369fb)+ push;Windows EBUSY 面不在 ci-fixes 落地,由哨兵统一跟踪,不在本分支重复修。
  5. 是否有回归问题:无。merge 仅追加 tests/notice-registry.test.ts 5 行(pin 纯 sqlite 选择),PR 功能改动逐处核对完整,无冲突标记。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有(本 PR 的 BILI_WS_IDLE_TIMEOUT_MS 是 PR 自身功能,非本次 CI 修复引入)。

结论:本 PR 侧 CI 红已全部归因并处置——notice-registry 红由 ci-fixes 合并修复,Windows EBUSY flake 为继承性失败、由哨兵在 ci-fixes 收敛。本 PR 无自身回归,等 Windows EBUSY 面落地后重触发即绿。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI auto-fix lane, attempt 1/3 — verdict: inherited flake, not a PR regression. No code change made this run.

I re-verified the live check state first (mirror cache, head 8a0348f321, run 43511831726): 2 failures, both windows-latest, node 22 — shards 1 and 4. Not green, so I pulled both job logs.

Root cause (from the logs): all 10 failing assertions are the identical EBUSY: resource busy or locked, unlink '…\index.db', thrown in the test teardown (tests/tmp-rm.ts:30 → rmrf). The SQLite handle is not released before the temp dir is deleted, which Windows refuses. The 10 are the persist-zstd.test.ts store suite (BILIZSTD1 / plain-JSON / BILIENC1 round-trips). The CI harness already retried on the same runner and reproduced all 10.

Why this is NOT this PR's fault:

Action taken (per the CI-failure-coordination policy, 2026-10-24):

This PR goes green once the sentinel lands the Windows EBUSY fix in ci-fixes (the affected suites pin BILI_STORAGE_UNIFIED — the established #4011-lesson remedy).


结构化八问

  1. 是什么问题:win22 两个 shard(1/4)的 persist-zstd.test.ts 存储套件 10 个用例全挂,同一报错 EBUSY unlink index.db——teardown 删临时目录时 SQLite 句柄未释放,Windows 拒绝删除被占用文件。
  2. 什么原因导致的:继承性 master 级失败——5a80e2b8ea(pure-SQLite 默认翻转,[PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480)之后,存储模式相关测试未自行钉 BILI_STORAGE_UNIFIED,在 Windows 上确定性触发文件锁;属 windows-22-1 lane randomly red: during test cleanup, unlink index.db hit an EBUSY external lock, couldn't survive tmp-rm | windows-22-1 车道随机红:测试清理期 unlink index.db 遇 EBUSY 外部锁,熬不过 tmp-rm 的 9.6 秒重试梯子(#2799/#2908 族复发) #4347 文件锁家族,与 [PR] fix: normalize inline requests with Fetch semantics #6563/[PR] fix: preserve Request fields during inline fallback #6566/[PR] fix: calibrate the dsh ledger floor against the output clamp (#5596) #6551 同签名,master e1f31ad9d5 同红。
  3. 影响范围与严重程度:仅 CI 的 windows-latest/node22 shard 1/4 受影响;不影响生产行为、不影响本 PR 功能;属 CI 环境/测试卫生问题,非用户可见缺陷。
  4. 怎么修复的:本轮未改代码——按 CI 协调纪律(2026-10-24),master 级失败必须收敛到 ci-fixes(哨兵维护),不在 feature PR 分支重复修;已确认 ci-fixes 合入 head(8a0348f321),Windows EBUSY 面哨兵修复尚未落地,已在 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 留一行协调记录。
  5. 是否有回归问题:无——本 PR 未触碰 persist-zstd/storage 任何路径,红项与 PR 改动无因果。
  6. 是否破坏缓存:不涉及(未改任何代码)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及(未改代码)。

判定:继承性 flake,非本 PR 回归;等哨兵把 Windows EBUSY 修复落到 ci-fixes 后本 PR 转绿。本轮无代码改动。

…lict

Union resolution: keep #6171 watchdog vars (failureEvents, startedAt) AND
master's send-sequence var (send). All three are required in the exchange
block (startedAt/failureEvents for the idle watchdog + failure diagnostics;
send for the send-sequence cache-diagnostic logging).

Verified: typecheck clean, build clean, targeted WS tests 45/45 pass.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI auto-fix lane, attempt 1/6 (conflict resolution) — DONE: merge conflict resolved and pushed.

What I did this run:

  1. Verified the live state first. The PR branch (2026-11-11_ws-failure-diagnostics, head 8a0348f3) was 219 commits behind master (26d9bcc6); merge-base c6369fb6. The PR branch already carried 3 merge commits (non-linear history), so I used a merge (not rebase) — rebase would mangle the existing merge commits.
  2. Merged origin/master into a fresh branch from the PR head. Exactly one conflict: src/responses-ws.ts (lines 554–561). Both sides had added different local variables at the same spot in the exchange block:
  3. Verified the resolution locally: typecheck clean (exit 0), build clean (exit 0), targeted WS tests tests/responses-websocket.test.ts + tests/issue6171-ws-failure-diagnostics.test.ts → 45/45 pass.
  4. Committed the merge (ae0df9d5, parents 8a0348f3 + 26d9bcc6) and pushed to the PR branch (fast-forward, no force-push; confirmed the remote head hadn't moved before pushing, per push discipline).

Live state after push (mirror cache, fresh): sha ae0df9d5, mergeable_state: blocked — no longer dirty. The conflict is resolved; blocked = branch-protection gating while the required checks re-run on the new head (all queued now). The PR becomes mergeable once the re-run passes — note the inherited Windows EBUSY flake (#4347 family, reported last run) will still fail win22 shards 1/4 until the sentinel lands that fix in ci-fixes.


结构化八问

  1. 是什么问题:PR 分支落后 master 219 个提交,与 base 分支产生合并冲突(merge state dirty),需要把 master 合入 PR 分支并解决冲突。
  2. 什么原因导致的:master 在 PR 分支创建后前进了 219 个提交,其中 send-sequence 诊断功能在 src/responses-ws.ts 的 exchange 块新增了 let send = 0;,与本 PR([Codex CLI][v0.1.213] 6 WebSocket failures before the first event, with no correlation between error cause and recovery | [Codex CLI][v0.1.213] WebSocket 首事件前失败 6 条,缺少错误原因与恢复结果关联 #6171)在同一位置新增的 failureEvents/startedAt(空闲看门狗 + 失败诊断)发生文本冲突。
  3. 影响范围与严重程度:仅 src/responses-ws.ts 一处冲突;不解决则 PR 无法合并(阻塞);解决后无行为变化(纯变量声明合并,两侧变量都被后续代码使用)。
  4. 怎么修复的:用 merge(而非 rebase,因分支已有 3 个 merge 提交、非线性)把 master 合入 PR 头;冲突按「并集」解决——保留本 PR 的 failureEvents/startedAt 和 master 的 send 三个变量(逐一验证了各自使用点,缺一不可);本地 typecheck/build/定向 WS 测试(45/45)全绿后提交并推送。
  5. 是否有回归问题:无——冲突解决是纯变量声明的并集,未改动任何逻辑;typecheck + build + 45 个定向 WS 测试全过,确认合并后代码可编译且 WS 行为不变。
  6. 是否破坏缓存:不涉及(未改任何缓存/重试/游标逻辑)。
  7. 是否改了设计方向:不涉及(仅解决合并冲突,未引入新设计)。
  8. 是否引入了新的配置:不涉及(未新增配置项;master 带入的变更是 master 既有内容,非本 PR 引入)。

判定:合并冲突已解决并推送(ae0df9d5),merge state 从 dirty 转为 blocked(仅剩 CI 重跑 + 分支保护);等 CI 重跑通过即可合并——其中 win22 shard 1/4 的继承性 EBUSY flake 仍需哨兵在 ci-fixes 落地修复(上一轮已报告)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Review verdict: APPROVE — the PR does exactly what it claims, and the tests have real teeth. (Cannot submit a formal APPROVE review via the API because the PR is authored by the same account; posting the review content as a comment.)

Correctness (all four claimed changes verified):

  1. Diagnosable failures — every upstream failed line now carries phase, elapsed_ms, and a sanitized cause (error_class=<Name> error_message=<msg> for error events; close_code=<n> was_clean=<t|f> for close events). The synchronous follow-up close (undici fires error then close(1006)) is no longer dropped — it is a second correlated line under the same conn/session stamp. Each exchange settles to exactly one terminal state (ended flag + failureEvents counter prevent double-settlement).
  2. Privacy — the free-text close REASON (untrusted upstream text) is never logged; error messages are sanitized (control chars stripped, whitespace collapsed, 160-char cap); no frame bodies or credentials.
  3. Client correlation — client terminal=synthesized code=<c> status=<n> (failure-only, gated on !this.terminal) + recovery=ok after_ms=<n> on the next successful turn.
  4. Idle watchdog — server-side, default 10 min, configurable via network.wsIdleTimeoutMs/BILI_WS_IDLE_TIMEOUT_MS; a stalled exchange now terminates (single terminal + event=timeout reason=idle-timeout line + one synthesized client error frame + conn drop); any upstream frame re-arms.

Tests: 4 new tests in tests/issue6171-ws-failure-diagnostics.test.ts (transport loss → 2 lines + 1 client frame + recovery; clean close → 1 line + reason never logged; idle watchdog → 1 timeout line + recovery; knob parsing). The refusal-fallback test in tests/responses-websocket.test.ts is updated to the new two-line format. All 54 pass locally (issue6171 4/4 + responses-websocket 41/41 + codex-responses-ws 9/9); typecheck + build clean. The tests count lines, assert conn-stamp equality, and doesNotMatch the reason text — they would catch a regression.

Behavior changes (disclosed in PR body):

Config surface: new knob network.wsIdleTimeoutMs/BILI_WS_IDLE_TIMEOUT_MS, default 600000, env>file>default via src/knobs.ts (non-integer/non-positive → default). Owner sign-off confirmed in #6171 (2026-11-11T07:15). Docs (en/zh/ja) + website + seed + lifecycle all updated.

No kernel/ touched — no kernel-change declaration needed.


八问(中文结构化结论)

  1. 是什么问题:codex-responses-ws 车道首事件前失败时,日志只有一行 upstream failed ... event=error close_code=none received_events=0,无法判断具体上游错误、客户端最终呈现、是否恢复;且 stall 的 exchange 无超时、永久挂起。
  2. 什么原因导致的:undici 在 transport 错误时先 fire error 再同步 fire close(1006);旧 handler 在第一个事件就移除全部 listener(once-listener),follow-up close 的 1006/wasClean 事实被丢弃;且无 idle watchdog。
  3. 影响范围与严重程度:所有 codex-responses-ws 车道(Codex CLI 用户);触发条件为上游 transport/协议错误、首事件前 stall。严重程度:体验瑕疵(日志不可诊断 + stall 永久挂起),非数据丢失/安全。
  4. 怎么修复的:每条失败行带 phase/elapsed_ms/脱敏 cause;follow-up close 保留(两行关联、同 stamp);每 exchange 恰好一个终态(ended + failureEvents 防双结算);close 的 free-text reason 永不入日志、error message 脱敏(控制字符剥离/空白折叠/160 上限);新增 server-side idle watchdog(默认 10 分钟、可配置、任何上游帧 re-arm)。
  5. 是否有回归问题:无。日志格式变更(已披露、测试已更新);stall 现在终止(有意变更、已披露);handshake timeout 独立于 refusal(Support Codex Responses WS clients with HTTP/SSE upstreams, preserving local delta checkpoints #3342 fallback 与精确字符串 Responses WebSocket handshake failed 未动);无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(idle watchdog 是新增 server-side 保护,不改变既有数据流/边界)。
  8. 是否引入了新的配置:是。network.wsIdleTimeoutMs(文件)/ BILI_WS_IDLE_TIMEOUT_MS(env),默认 600000,env>file>default(src/knobs.ts),非整数/非正数→默认。owner 已在 [Codex CLI][v0.1.213] 6 WebSocket failures before the first event, with no correlation between error cause and recovery | [Codex CLI][v0.1.213] WebSocket 首事件前失败 6 条,缺少错误原因与恢复结果关联 #6171 拍板。

判定:可合(APPROVE)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

✅ Review done: PR #6301 (GitHub) — APPROVE. All 4 claimed changes verified against the full diff (diagnosable two-line failures, privacy contract, client correlation, configurable idle watchdog); 54/54 targeted tests green locally (issue6171 4 + responses-websocket 41 + codex-responses-ws 9), typecheck + build clean. No kernel/ touched (no kernel-change needed), no small findings to fix (minor observations are nits below the fix-effort bar), no 🔶 需要决策. Full review comment posted on the PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant