diff --git a/docs/configuration.ja.md b/docs/configuration.ja.md index 215a6ffc3e..9cc748e0e7 100644 --- a/docs/configuration.ja.md +++ b/docs/configuration.ja.md @@ -161,6 +161,7 @@ | `network` | object | {} (all defaults) | — | 上流呼び出しのタイムアウト、ウォッチドッグ、リプレイポリシー。 | | `network.upstreamTimeoutMs` | number | 720000 | BILI_UPSTREAM_TIMEOUT_MS | 単一の上流呼び出しに対する絶対上限(12分は、遅い深層思考モデルをカバーする)。 | | `network.upstreamFirstByteMs` | number | = upstreamTimeoutMs (resolved; 720000 when unset) | BILI_UPSTREAM_FIRST_BYTE_MS | フェーズ1の予算:リクエスト送信 → 最初のレスポンスバイト(undiciは最初のボディデータ付近でfetchを解決する)。未設定の場合、解決済みの上流タイムアウト(upstreamTimeoutMs)に従う — BILI_UPSTREAM_TIMEOUT_MS を12分以上に引き上げるデプロイメントを含む(#5917 レビュー:定数 720000 のフォールバックはそれらを静かに再制限してしまう)— したがって動作は変更されない;より厳格なデプロイメントはオプトインする(#5904:リクエストを受け入れるが以前は応答しなかったスタックした上流は、クライアントが中断するまで静かにハングしていた)。最初のバイト後、チャンク間の沈黙は引き続き完全な upstreamTimeoutMs によって管理される(#1706/#1714 の保証)。この値が upstreamTimeoutMs 未満に設定されると、最初のバイトのタイムアウトは独自の失敗クラス upstream-headers-timeout となる — レスポンスバイトがゼロであるため、bili はリクエスト全体を一度で失敗させるのではなく、既存のリトライ予算(BILI_REPLAY_RETRY_MAX、デフォルト 3)の下でプロキシ内でリプレイする;ストリーム中(ボディ)のアイドルタイムアウトは upstream-timeout のままであり、リトライされない。未設定の場合、動作は以前とバイト単位で同一である(#5807/#5831)。リリースゲートは 120000 を設定する。 | +| `network.wsIdleTimeoutMs` | number | 600000 | BILI_WS_IDLE_TIMEOUT_MS | Responses WebSocket レグのイベント間アイドル予算(#6171):上流フレームのたびに再アームされるため、安定したストリーミングでは決してトリガーされず、完全なスタック時のみ発火する。ハンドシェイクも対象(アップグレードがスタックしても永遠に待たず即失敗する)。発火時、その交換は単一の終端状態を取得し、同一 conn/session スタンプの下で相関可能な `upstream failed … event=timeout reason=idle-timeout` ログ行を出力し、接続を切断して次のターンで新たに再接続する。以前は完全にスタックした WS 交換はクライアントが諦めるまでハングしていた — HTTP レグの対応物は BILI_UPSTREAM_TIMEOUT_MS。 | | `network.requestWatchdogMs` | number | 2× upstreamTimeoutMs | BILI_REQUEST_WATCHDOG_MS | 呼び出しごとのタイマーが設定されていなくても、スタックした上流ソケットを強制中断する絶対的なウォッチドッグ。 | | `network.keepAliveTimeoutMs` | number | 5000 | BILI_KEEP_ALIVE_TIMEOUT_MS | プールされた上流ソケットのアイドルキープアライブタイムアウト。 | | `network.clientErrorBackstopMs` | number | 30000 | BILI_CLIENT_ERROR_BACKSTOP_MS | コミットされたレスポンスの読み取りを停止したクライアント接続を中断するバックストップ。 | @@ -2057,6 +2058,7 @@ ACP ネイティブエージェント(現在は `pi` 拡張機能)は、プ | `BILI_UPSTREAM_PROXY` | `proxy` | unset (direct) | | `BILI_UPSTREAM_PROXY_MODE` | `upstreamProxyMode` | auto (unset behaves as direct) | | `BILI_UPSTREAM_TIMEOUT_MS` | `network.upstreamTimeoutMs` | 720000 | +| `BILI_WS_IDLE_TIMEOUT_MS` | `network.wsIdleTimeoutMs` | 600000 | | `BILI_WS_OUTBOUND_BYTES` | `network.wsOutboundBytes` | 10485760 | | `PI_ACP_DELEGATE_ASYNC_TIMEOUT_MINUTES` | `pi.subagents` | {} (acp_delegate enabled with package defaults) | | `PI_ACP_DELEGATE_FORCE_ENABLE` | `pi.subagents` | {} (acp_delegate enabled with package defaults) | diff --git a/docs/configuration.md b/docs/configuration.md index 6eaf046056..939101c09f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -164,6 +164,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `network` | object | {} (all defaults) | — | Transport timeouts, watchdogs and replay policy for upstream calls. | | `network.upstreamTimeoutMs` | number | 720000 | BILI_UPSTREAM_TIMEOUT_MS | Hard ceiling on a single upstream call (12 min covers slow deep-thinking models). | | `network.upstreamFirstByteMs` | number | = upstreamTimeoutMs (resolved; 720000 when unset) | BILI_UPSTREAM_FIRST_BYTE_MS | Phase-1 budget: request sent → first response byte (undici resolves the fetch around the first body data). Unset follows the RESOLVED upstreamTimeoutMs — including deployments that raise BILI_UPSTREAM_TIMEOUT_MS above 12 min (#5917 review: a constant 720000 fallback would silently re-cap them) — so behavior is unchanged; tighter deployments opt in (#5904: a wedged upstream that accepts requests but never answered previously hung silently until the client aborted). After the first byte, chunk-gap silence stays governed by the full upstreamTimeoutMs (#1706/#1714 guarantee). When this value is set BELOW upstreamTimeoutMs, a first-byte timeout becomes its own failure class `upstream-headers-timeout` — zero response bytes were received, so bili replays it in-proxy under the existing retry budget (BILI_REPLAY_RETRY_MAX, default 3) instead of failing the whole request once; the mid-stream (body) idle timeout stays `upstream-timeout` and non-retried. Unset, behavior is byte-identical to before (#5807/#5831). The release gate sets 120000. | +| `network.wsIdleTimeoutMs` | number | 600000 | BILI_WS_IDLE_TIMEOUT_MS | Inter-event idle budget for the Responses WebSocket leg (#6171): any upstream frame re-arms it, so steady streaming never trips it — only a full stall does. The handshake is covered too (a wedged upgrade fails fast instead of waiting forever). On fire the exchange gets its single terminal state, a correlatable `upstream failed … event=timeout reason=idle-timeout` log line under the same conn/session stamp, and the connection drops so the next turn reconnects fresh. Previously a fully stalled WS exchange hung until the client gave up — the HTTP lane's equivalent is BILI_UPSTREAM_TIMEOUT_MS. | | `network.requestWatchdogMs` | number | 2× upstreamTimeoutMs | BILI_REQUEST_WATCHDOG_MS | Absolute watchdog that force-aborts a stuck upstream socket even when the per-call timer was never armed. | | `network.keepAliveTimeoutMs` | number | 5000 | BILI_KEEP_ALIVE_TIMEOUT_MS | Idle keep-alive timeout for pooled upstream sockets. | | `network.clientErrorBackstopMs` | number | 30000 | BILI_CLIENT_ERROR_BACKSTOP_MS | Backstop that aborts a client connection that stops reading a committed response. | @@ -2070,6 +2071,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_UPSTREAM_PROXY` | `proxy` | unset (direct) | | `BILI_UPSTREAM_PROXY_MODE` | `upstreamProxyMode` | auto (unset behaves as direct) | | `BILI_UPSTREAM_TIMEOUT_MS` | `network.upstreamTimeoutMs` | 720000 | +| `BILI_WS_IDLE_TIMEOUT_MS` | `network.wsIdleTimeoutMs` | 600000 | | `BILI_WS_OUTBOUND_BYTES` | `network.wsOutboundBytes` | 10485760 | | `PI_ACP_DELEGATE_ASYNC_TIMEOUT_MINUTES` | `pi.subagents` | {} (acp_delegate enabled with package defaults) | | `PI_ACP_DELEGATE_FORCE_ENABLE` | `pi.subagents` | {} (acp_delegate enabled with package defaults) | @@ -2128,6 +2130,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_REPLAY_RETRY_MAX` | Total attempts for replay retries (default `3`; set `1` to disable retries entirely — legacy single-attempt behavior). See #189. On the acp-loop/preflight/compress loops it applies to both transient HTTP failures and #1263 fail-fast network failures (pre-response reset/refused — proxy-recycle class), never to timeout/abort kinds. On the main model-request path (#1688) only fail-fast pre-response network failures are replayed; HTTP verdicts (4xx/5xx) always pass through verbatim, untried again. | | `BILI_PROXY_KEEPALIVE_MAX_MS` | Keep-alive reuse ceiling (ms) for connections through an upstream proxy (default `55000`). Common proxies recycle idle tunnels on a ~60s cadade; capping our reuse window below that trades a few reconnects for the classic "first request after idle dies with ECONNRESET" failure (#1263). `0` = uncapped (undici defaults). Direct (non-proxied) connections are unaffected. | | `BILI_UPSTREAM_TIMEOUT_MS` | Idle budget (ms) for upstream requests: time-to-first-byte and time between body chunks (default `720000` = 12 min). A healthy stream that keeps producing chunks is never cut mid-flight; a silent one is. The same value drives the underlying HTTP client's transport timeouts, so this single knob bounds long local-model prefills end-to-end (#551). | +| `BILI_WS_IDLE_TIMEOUT_MS` | Inter-event idle budget (ms) for the Responses WebSocket leg (#6171): any upstream frame re-arms it, so steady streaming never trips it — only a full stall does. It covers the handshake too (a wedged upgrade fails fast instead of waiting forever). On fire the exchange gets its single terminal state — one synthesized client error frame plus the connection drop, so the next turn reconnects fresh — and a correlatable `upstream failed … event=timeout reason=idle-timeout` log line under the same conn/session stamp. Previously a fully stalled WS exchange hung until the client gave up; this is the WS leg's equivalent of `BILI_UPSTREAM_TIMEOUT_MS`. Default `600000` = 10 min (owner-set on #6171); non-integer or non-positive values fall back to the default. File-config key: `network.wsIdleTimeoutMs`. | | `BILI_CA_FILE` | Path to a PEM file with extra root CAs for bili's own upstream TLS trust (#3776). Security software doing HTTPS/encrypted-connection scanning (Kaspersky, enterprise DLP) or a corporate MITM proxy re-signs the upstream with its own CA: browsers trust it via the OS certificate store, but Node's fetch reads only its built-in roots plus `NODE_EXTRA_CA_CERTS` — so bili fails with `SELF_SIGNED_CERT_IN_CHAIN` while the rest of the machine connects fine. Export the scanner/vendor root to PEM and set this (or `NODE_EXTRA_CA_CERTS`); alternatively exclude node.exe from the software's HTTPS inspection. The file is MERGED with Node's built-in roots and any `NODE_EXTRA_CA_CERTS` entries — never replaces them. Unset = today's default trust, byte-for-byte. File-config key: `network.caFile`. | | ~~`BILI_STREAM_STALL_MS`~~ | **Removed in #1714** (the #1452 opt-in streaming-phase stall guard is retired). In #1706 a stale `400` export turned every thinking-phase silence into a false truncation; local-model deployments legitimately go silent for minutes mid-stream (thinking phases, long prefills), so no finite sub-budget was safe for everyone. Upstream silence — prefill and mid-stream alike — is now bounded solely by the always-on `BILI_UPSTREAM_TIMEOUT_MS` idle budget (default 12 min, tunable). Old values are ignored; bili names a stale export once at startup (`no longer read`) — remove it from your shell profile. | | `BILI_KEEP_ALIVE_TIMEOUT_MS` | Keep-alive timeout (ms) for client-facing sockets (default `5000`, matching Node's implicit default; #1452). Idle client connections are reaped with a clean FIN by Node's built-in reaper; this makes the previously implicit value explicit and configurable, and the reap is classified as `reason=idle-timeout` in the connection lifecycle ledger (debug log). Non-numeric or non-positive values fall back to `5000`. | diff --git a/docs/configuration.zh-CN.md b/docs/configuration.zh-CN.md index 6209c06540..210948681d 100644 --- a/docs/configuration.zh-CN.md +++ b/docs/configuration.zh-CN.md @@ -164,6 +164,7 @@ | `network` | object | {} (all defaults) | — | 上游调用的传输超时、看门狗与重放策略。 | | `network.upstreamTimeoutMs` | number | 720000 | BILI_UPSTREAM_TIMEOUT_MS | 单次上游调用的硬上限(12 分钟覆盖慢速深度思考模型)。 | | `network.upstreamFirstByteMs` | number | = upstreamTimeoutMs (resolved; 720000 when unset) | BILI_UPSTREAM_FIRST_BYTE_MS | 阶段一预算:请求发出 → 首个响应字节(undici 在首个 body 数据到达时才 resolve fetch)。未设时跟随解析后的 upstreamTimeoutMs——包括把 BILI_UPSTREAM_TIMEOUT_MS 提到 12 分钟以上的部署(#5917 评审:常量 720000 回落会悄悄把它们压回 12 分钟)——行为不变;需要更快约束首字节时延的部署可显式收紧(#5904:上游接收请求后永不应答的卡死,之前会静默拖到客户端中止)。首字节到达后,chunk 间隙静默仍由完整 upstreamTimeoutMs 管辖(#1706/#1714 保证)。当该值被收紧到低于 upstreamTimeoutMs 时,首字节超时成为独立失败类 `upstream-headers-timeout`——因为一个响应字节都未收到,bili 会在代理内按既有重试预算(BILI_REPLAY_RETRY_MAX,默认 3 次)重放,而不是让整步只失败一次;流中途(body)空闲超时仍是 `upstream-timeout` 且不重试。未设时行为与之前逐字节一致(#5807/#5831)。发布 gate 设 120000。 | +| `network.wsIdleTimeoutMs` | number | 600000 | BILI_WS_IDLE_TIMEOUT_MS | Responses WebSocket 车道的逐事件空闲预算(#6171):任何上游帧都会重新上弦,所以稳定流式永不触发——只有完全卡死才会。握手阶段同样受管(升级卡死会快速失败而不是永远等待)。触发时该交换获得唯一终态、同一 conn/session 标识下可关联的 `upstream failed … event=timeout reason=idle-timeout` 日志行,并断开连接让下一轮重新连接。此前完全卡死的 WS 交换会一直挂到客户端放弃——HTTP 车道的对应物是 BILI_UPSTREAM_TIMEOUT_MS。 | | `network.requestWatchdogMs` | number | 2× upstreamTimeoutMs | BILI_REQUEST_WATCHDOG_MS | 绝对看门狗:即使按调用计时器从未挂上,也强制中止卡死的上游套接字。 | | `network.keepAliveTimeoutMs` | number | 5000 | BILI_KEEP_ALIVE_TIMEOUT_MS | 连接池内上游套接字的空闲保活超时。 | | `network.clientErrorBackstopMs` | number | 30000 | BILI_CLIENT_ERROR_BACKSTOP_MS | 中止停止读取已提交响应的客户端连接的兜底超时。 | @@ -2047,6 +2048,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_UPSTREAM_PROXY` | `proxy` | unset (direct) | | `BILI_UPSTREAM_PROXY_MODE` | `upstreamProxyMode` | auto (unset behaves as direct) | | `BILI_UPSTREAM_TIMEOUT_MS` | `network.upstreamTimeoutMs` | 720000 | +| `BILI_WS_IDLE_TIMEOUT_MS` | `network.wsIdleTimeoutMs` | 600000 | | `BILI_WS_OUTBOUND_BYTES` | `network.wsOutboundBytes` | 10485760 | | `PI_ACP_DELEGATE_ASYNC_TIMEOUT_MINUTES` | `pi.subagents` | {} (acp_delegate enabled with package defaults) | | `PI_ACP_DELEGATE_FORCE_ENABLE` | `pi.subagents` | {} (acp_delegate enabled with package defaults) | @@ -2138,6 +2140,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_UPSTREAM_PROXY` | 代理自身出站连接的上游代理 —— 在*全局*来源中优先级最高(高于 Web UI 手动代理与配置文件 `proxy`)。per-URL 的 `providers..proxy` 对其匹配的 provider URL 仍然优先。完整解析顺序、防环规则与示例见 [服务端设置 → `proxy`](#服务端设置)。 | | `BILI_INHERITED_HTTP_PROXY` / `BILI_INHERITED_HTTPS_PROXY` / `BILI_INHERITED_ALL_PROXY` / `BILI_INHERITED_NO_PROXY` | 非用户直接使用 —— launcher 起代理子进程时自动设置(#1012)。launcher 会从客户端和代理子进程两侧剥掉 shell 的代理变量(客户端必须把流量发给 bili;代理的模型出网也不能被 shell 代理劫持),但会把用户剥离前的代理转发到这些变量里,让代理的**辅助出网**(MITM 盲隧道 —— 客户端侧的 MCP/web 流量)仍能走用户的 VPN。它们只作用于盲隧道的 fallback 层:显式路由 / 全局 `proxy` / `BILI_UPSTREAM_PROXY` / 显式 `"upstreamProxyMode": "direct"` 仍然优先,指向 bili 自身端口的值会被丢弃。模型出网不受影响(未显式配置则保持直连)。 | | `BILI_UPSTREAM_TIMEOUT_MS` | 上游请求的空闲预算(毫秒):首字节时间(TTFB)与响应体块之间的间隔(默认 `720000` = 12 分钟)。持续产出数据块的健康流永远不会被中途切断;静默的流才会。同一个值同时驱动底层 HTTP 客户端的传输层超时,因此这一个旋钮即可端到端约束本地大模型的超长 prefill(#551)。 | +| `BILI_WS_IDLE_TIMEOUT_MS` | Responses WebSocket 车道的逐事件空闲预算(毫秒)(#6171):任何上游帧都会重新上弦,所以稳定流式永不触发——只有完全卡死才会。握手阶段同样受管(升级卡死会快速失败而不是永远等待)。触发时该交换获得唯一终态——一帧合成的客户端错误加断开连接(下一轮重新连接)——并在同一 conn/session 标识下记一条可关联的 `upstream failed … event=timeout reason=idle-timeout` 日志行。此前完全卡死的 WS 交换会一直挂到客户端放弃;本旋钮是 WS 车道上 `BILI_UPSTREAM_TIMEOUT_MS` 的对应物。默认 `600000` = 10 分钟(owner 在 #6171 拍板);非整数或非正值回落到默认值。文件配置键:`network.wsIdleTimeoutMs`。 | | `BILI_CA_FILE` | 指向一个 PEM 文件的路径,包含要额外信任的根 CA(#3776),仅作用于 bili 自己的上游 TLS。做 HTTPS/加密连接扫描的安全软件(卡巴斯基、企业 DLP)或企业 MITM 代理会用自家 CA 重签上游:浏览器通过系统证书库信任它,但 Node 的 fetch 只读内置根证书加 `NODE_EXTRA_CA_CERTS`——于是 bili 报 `SELF_SIGNED_CERT_IN_CHAIN`,而机器上其他一切连接正常。把扫描软件/厂商根证书导出为 PEM 后设置此项(或 `NODE_EXTRA_CA_CERTS`);或在安全软件的 HTTPS 检查中排除 node.exe。该文件与 Node 内置根证书及 `NODE_EXTRA_CA_CERTS` 条目合并——从不替换。未设置 = 当前默认信任,逐字节不变。配置键:`network.caFile`。 | | `BILI_ATTACH_HEALTH_DEADLINE_MS` | dsh/opencode attach 校验中,attach 目标已挂但本进程模型通道**钉死**在其上(观察到指向它的 `/bili/…` 路由流量)时的健康等待上限(毫秒):bili 等待目标恢复而不是 spawn 第二实例——spawn 会把会话劈成两半(模型流量保持钉死,bili 工具在另一实例上 404)。超时后大声报错,并在每次模型请求时持续重查直到目标恢复(默认 `15000`)。见 #1365。 | | `BILI_ATTACH_EVIDENCE_GRACE_MS` | dsh/opencode attach 校验探测到目标已挂时,等待路由通道证据出现的宽限窗口(毫秒),超时才回退到旧的 spawn 路径(覆盖「判定早于首个请求」的竞态:t≈0 时探测失败、t≈1s 时首个模型请求才落地)(默认 `5000`)。见 #1365。 | diff --git a/docs/lifecycle.md b/docs/lifecycle.md index 97c4400148..ad8a4a08af 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -110,6 +110,7 @@ | `BILI_RECEIVE_BODY_BYTES` / `BILI_DECODED_BODY_BYTES` | network.receive/decodedBodyBytes,5 GiB / 10 GiB——单请求入站/解码后内存上限(#3128:曾硬编码 100 MiB,多 GB 单请求被 reset) | | `BILI_OUTBOUND_BODY_BYTES` | network.outboundBodyBytes,5 GiB——重建后转发上游的硬上限(#3128) | | `BILI_WS_OUTBOUND_BYTES` | network.wsOutboundBytes,10 MiB——WS 车道出站软预算:超限先把旧的非保护图归档/降采样再发送(#3701:网关对大帧 close 1009;0 关闭预发送削减;HTTP 车道不受影响) | +| `BILI_WS_IDLE_TIMEOUT_MS` | network.wsIdleTimeoutMs,600000(10 min)——WS 车道握手+帧间空闲看门狗:上游任一帧重新计时,仅整段全停才触发;超时合成 client error frame 并断连(#6171:pre-first-event 失败此前零日志且无界挂起;默认值不改变既有行为,调小可加速诊断) | | `BILI_HTTP_OUTBOUND_BYTES` | network.httpOutboundBytes,10 MiB——普通/代理/pi 原生 HTTP 中继车道出站软预算:超限先把旧的非保护图归档/降采样再发送(#3909:图片密集会话按字节/像素解码 CPU 被上游拒收、token 折叠不触发;仅在无受管传输时生效,受管 HTTP/dsh-inline 从不裁减 #3701 Finding 1;0 关闭) | | `BILI_INGEST_BODY_BYTES` | network.ingestBodyBytes,32 MiB——解码后进入 JSON.parse+归一化清扫的成本带(#5794:51.2 MiB 体每轮吞 5.5–7.2s 事件循环 → 747s FATAL;超带的**文本为主**体即 413,因为任何可解析的 32 MiB+ 文本体 ≥ ~8M token、超过一切真实窗口;带图片的体豁免——像素按设计不进 token 计量,由槽位预算/收件上限/车道 shed 治理(#496/#3909/#3701);0 关闭) | | `BILI_MAX_MESSAGES` | network.maxMessages,默认 0(关闭)——单上游 messages[] 数组长度硬限(COUNT 维度,任何字节/token 估算都表达不了;#5000:26,903 条消息体清掉所有尺寸闸仍在上游死掉;超限 413 提示 restart/rewind,不做折叠;按上游各自设置) | diff --git a/src/config.ts b/src/config.ts index e8c42325c5..b8c515d2c8 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2297,6 +2297,12 @@ type FileConfig = { * answers fails fast instead of silently eating the full 12-minute watchdog * while the client times out first (#5904). */ upstreamFirstByteMs?: number; + /** Inter-event idle budget for the Responses WebSocket leg (was env-only + * BILI_WS_IDLE_TIMEOUT_MS; default 600000 = 10 min). Any upstream frame + * re-arms it — steady streaming never trips it, only a full stall does. + * Covers the handshake too; on fire the exchange gets its single terminal + * state and the connection drops for a fresh reconnect next turn (#6171). */ + wsIdleTimeoutMs?: number; /** Request watchdog budget (was BILI_REQUEST_WATCHDOG_MS; default 2× upstream timeout). */ requestWatchdogMs?: number; /** Server keep-alive timeout (was BILI_KEEP_ALIVE_TIMEOUT_MS; default 5000). */ diff --git a/src/knobs.ts b/src/knobs.ts index 32e49ad6a0..9bbccd357d 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -104,6 +104,21 @@ export function upstreamFirstByteMs(): number { return Number.isInteger(v) && v > 0 ? v : upstreamTimeoutMs(); } +const WS_IDLE_TIMEOUT_MS_DEFAULT = 600_000; // owner-set on #6171: 10 minutes + +/** BILI_WS_IDLE_TIMEOUT_MS > network.wsIdleTimeoutMs > 10 minutes. Inter-event idle + * budget for the Responses WebSocket leg (#6171): any upstream frame re-arms it, so + * steady streaming never trips it — only a full stall does. The HTTP lane's + * equivalent is the fetchWithTimeout watchdog; the WS leg owns its own socket and had + * no server-side bound, so a stalled supplier hung the exchange until the client gave + * up. Covers the handshake too (a wedged upgrade request fails fast instead of + * waiting forever). On fire the exchange gets its single terminal state and the + * connection drops for a fresh reconnect next turn. */ +export function wsIdleTimeoutMs(): number { + const v = tNum("BILI_WS_IDLE_TIMEOUT_MS", fileNetwork().wsIdleTimeoutMs); + return Number.isInteger(v) && v > 0 ? v : WS_IDLE_TIMEOUT_MS_DEFAULT; +} + /** BILI_CA_FILE > network.caFile > unset. Path to a PEM bundle of extra root CAs * trusted by bili's OWN outbound upstream fetch — for security software doing * HTTPS/encrypted-connection scanning (Kaspersky, enterprise DLP) or a corporate diff --git a/src/responses-ws.ts b/src/responses-ws.ts index d1c5abddd1..064fb91e9f 100644 --- a/src/responses-ws.ts +++ b/src/responses-ws.ts @@ -4,7 +4,7 @@ import type { ResponsesRequestBody } from "acp-kernel/wire"; import { WebSocket as UpstreamWebSocket, type Dispatcher } from "undici"; import WebSocket from "ws"; import { fetchWithTimeout, type FetchOptions } from "./fetch-util.js"; -import { limitsReceiveBytes } from "./knobs.js"; +import { limitsReceiveBytes, wsIdleTimeoutMs } from "./knobs.js"; import { featureEnabled } from "./rollout.js"; import { getStore } from "./persist.js"; import { responsesToCoreWithToolImages } from "./responses-tool-output.js"; @@ -23,6 +23,54 @@ function closeCode(event: Event): number | "none" { return "code" in event && typeof event.code === "number" ? event.code : "none"; } +const FAILURE_DETAIL_MAX_CHARS = 160; + +// #6171: failure-event text is untrusted upstream input — strip control chars, +// collapse whitespace and cap the length before it touches the log. Frame bodies +// never pass through this helper. +function sanitizeDetail(value: unknown): string { + if (typeof value !== "string") return "none"; + const cleaned = value.replace(/[\u0000-\u001f\u007f]/g, " ").replace(/\s+/g, " ").trim(); + if (cleaned === "") return "none"; + return cleaned.length <= FAILURE_DETAIL_MAX_CHARS ? cleaned : `${cleaned.slice(0, FAILURE_DETAIL_MAX_CHARS - 1)}…`; +} + +// #6171: sanitized one-line description of a WebSocket failure event. The old log +// line only carried event.type + close_code, so a raw-TCP drop (undici fires +// error THEN a synchronous close(1006)) read as `event=error close_code=none` and +// every failure looked identical. Only STRUCTURED facts land here: the close +// REASON is arbitrary upstream-set text (untrusted content) and must never reach +// the log — the leak guard in tests/responses-websocket.test.ts pins that; the +// standard codes (1006/1009/1011/1013 …) already carry the intent. +function describeFailureEvent(event: Event): string { + if (event.type === "close") { + const wasClean = "wasClean" in event && typeof event.wasClean === "boolean" ? event.wasClean : false; + return `close_code=${closeCode(event)} was_clean=${wasClean}`; + } + const cause = "error" in event ? event.error : undefined; + if (cause instanceof Error) return `error_class=${cause.name} error_message=${sanitizeDetail(cause.message)}`; + const message = "message" in event && typeof event.message === "string" ? sanitizeDetail(event.message) : "none"; + return `error_class=unknown error_message=${message}`; +} + +// #6171: inter-event idle watchdog for the WS leg. The HTTP lane gets this from +// fetchWithTimeout; the upstream WebSocket owns its own socket and had no +// server-side bound, so a stalled supplier (no events, no close) hung the +// exchange until the client gave up. Re-armed on every liveness signal; unref'd +// so it never holds the process open. +function idleWatchdog(onFire: () => void, timeoutMs: number): { arm: () => void; disarm: () => void } { + let timer: ReturnType | undefined; + const disarm = (): void => { if (timer !== undefined) { clearTimeout(timer); timer = undefined; } }; + return { + arm: () => { + disarm(); + timer = setTimeout(onFire, timeoutMs); + timer.unref(); + }, + disarm, + }; +} + // #3342: the ONLY failure that may trigger the HTTP/SSE egress fallback is a // genuine WS-upgrade refusal in the handshake phase (before any generation // request is sent). An abort (client cancelled) or any other error propagates @@ -382,23 +430,49 @@ export class ResponsesWsUpstream { this.socket = socket; this.connectionGeneration++; this.key = key; + const startedAt = Date.now(); await new Promise((resolve, reject) => { - const clean = (): void => { + let failureEvents = 0; + let done = false; + // #6171: a wedged upgrade request (TCP up, no response) used to wait + // forever — the handshake now shares the exchange's idle budget. A + // timeout is NOT a refusal (#3342): it rejects with its own message so + // the HTTP/SSE fallback does not fire for a supplier that merely stalled. + const watchdog = idleWatchdog(() => { + if (done) return; + done = true; + clean(); + this.log("warn", `upstream failed phase=handshake event=timeout reason=idle-timeout elapsed_ms=${Math.max(0, Date.now() - startedAt)}`); + reject(new Error("Responses WebSocket handshake timed out")); + }, wsIdleTimeoutMs()); + const clean = (keepCloseListener = false): void => { + watchdog.disarm(); socket.removeEventListener("open", opened); + if (!keepCloseListener) socket.removeEventListener("close", failed); socket.removeEventListener("error", failed); - socket.removeEventListener("close", failed); options.signal?.removeEventListener("abort", aborted); }; - const opened = (): void => { clean(); this.log("debug", "upstream connected phase=handshake"); resolve(); }; + const opened = (): void => { done = true; clean(); this.log("debug", "upstream connected phase=handshake"); resolve(); }; const failed = (event: Event): void => { - clean(); - this.log("warn", `upstream failed phase=handshake event=${event.type} close_code=${closeCode(event)}`); + // #6171: undici fires 'error' and, on transport loss, a synchronous + // follow-up 'close'(1006). The first event decides the terminal state + // (and keeps the close listener alive to observe the follow-up); the + // second is logged informationally. Max two lines per handshake. + if (failureEvents >= 2) return; + const isFollowUp = failureEvents === 1; + if (!isFollowUp && done) return; + failureEvents++; + this.log("warn", `upstream failed phase=handshake event=${event.type} ${describeFailureEvent(event)} elapsed_ms=${Math.max(0, Date.now() - startedAt)}`); + if (isFollowUp) { clean(); return; } + done = true; + clean(event.type === "error"); reject(new Error("Responses WebSocket handshake failed")); }; - const aborted = (): void => { clean(); this.log("debug", "upstream aborted phase=handshake"); this.close("handshake-abort"); reject(new DOMException("Aborted", "AbortError")); }; + const aborted = (): void => { done = true; clean(); this.log("debug", "upstream aborted phase=handshake"); this.close("handshake-abort"); reject(new DOMException("Aborted", "AbortError")); }; socket.addEventListener("open", opened, { once: true }); socket.addEventListener("error", failed, { once: true }); socket.addEventListener("close", failed, { once: true }); + watchdog.arm(); if (options.signal?.aborted) aborted(); else options.signal?.addEventListener("abort", aborted, { once: true }); }); @@ -477,6 +551,10 @@ export class ResponsesWsUpstream { let ended = false; let retryFull = request.previous_response_id !== undefined; let receivedEvents = 0; + // #6171: elapsed is measured from the LAST create frame, so a resend-full + // retry reports its own wait instead of the whole exchange's age. + let failureEvents = 0; + let startedAt = Date.now(); let send = 0; const encoder = new TextEncoder(); // #3701: log the EXACT outbound frame bytes — the upstream gateway judges by @@ -493,6 +571,8 @@ export class ResponsesWsUpstream { previous_response_id_sha256: typeof payload.previous_response_id === "string" ? cacheDiagnosticHash(payload.previous_response_id) : null, })}`); socket.send(frameText); + startedAt = Date.now(); + watchdog.arm(); }; // Backpressure (#2076): the WS socket cannot be paused, so frames arriving // while the consumer lags wait in the stream's own queue — delivery is @@ -500,30 +580,57 @@ export class ResponsesWsUpstream { // the queue is under the highWaterMark, conservative beyond) so the guard // below still bounds memory for a consumer that never catches up. let retained = 0; - const clean = (): void => { + // #6171: inter-event idle budget for the exchange — any upstream frame re-arms + // it, so steady streaming never trips it; only a full stall does. On fire the + // exchange gets its single terminal state and the connection drops so the next + // turn reconnects fresh. The HTTP lane's equivalent is fetchWithTimeout. + const watchdog = idleWatchdog(() => { + if (ended) return; + const phase = receivedEvents > 0 ? "stream" : "await-first-event"; + this.log("warn", `upstream failed phase=${phase} event=timeout reason=idle-timeout elapsed_ms=${Math.max(0, Date.now() - startedAt)} received_events=${receivedEvents}`); + fail(new ResponsesWsUpstreamError( + `Responses WebSocket upstream timed out before completion (idle_ms=${wsIdleTimeoutMs()}, received_events=${receivedEvents})`, + "none", + phase, + receivedEvents, + ), "upstream-idle-timeout"); + this.close("upstream-idle-timeout"); + }, wsIdleTimeoutMs()); + const clean = (keepCloseListener = false): void => { ended = true; + watchdog.disarm(); socket.removeEventListener("message", message); socket.removeEventListener("error", failed); - socket.removeEventListener("close", failed); + if (!keepCloseListener) socket.removeEventListener("close", failed); options.signal?.removeEventListener("abort", aborted); this.active = undefined; }; - const fail = (error: Error, reason = "transport-close"): void => { + const fail = (error: Error, reason = "transport-close", opts?: { keepCloseListener?: boolean }): void => { if (ended) return; this.resetHistory(reason); - clean(); + clean(opts?.keepCloseListener === true); if (settled) controller?.error(error); else reject(error); }; const failed = (event: Event): void => { + // #6171: undici fires 'error' and, on transport loss, a synchronous + // follow-up 'close'(1006). The first event decides the terminal state + // (and keeps the close listener alive to observe the follow-up); the + // second is logged informationally so the close code/reason are never + // lost again. Max two lines per exchange. + if (failureEvents >= 2) return; + const isFollowUp = failureEvents === 1; + if (!isFollowUp && ended) return; + failureEvents++; const phase = receivedEvents > 0 ? "stream" : "await-first-event"; - this.log("warn", `upstream failed phase=${phase} event=${event.type} close_code=${closeCode(event)} received_events=${receivedEvents}`); + this.log("warn", `upstream failed phase=${phase} event=${event.type} ${describeFailureEvent(event)} elapsed_ms=${Math.max(0, Date.now() - startedAt)} received_events=${receivedEvents}`); + if (isFollowUp) { clean(); return; } fail(new ResponsesWsUpstreamError( - `Responses WebSocket upstream disconnected before completion (close_code=${closeCode(event)}, received_events=${receivedEvents})`, + `Responses WebSocket upstream disconnected before completion (${describeFailureEvent(event)}, received_events=${receivedEvents})`, closeCode(event), phase, receivedEvents, - ), "upstream-disconnect"); + ), "upstream-disconnect", { keepCloseListener: event.type === "error" }); }; const aborted = (): void => { this.log("debug", `upstream aborted phase=${receivedEvents > 0 ? "stream" : "await-first-event"}`); @@ -532,6 +639,7 @@ export class ResponsesWsUpstream { }; const message = (event: { data: unknown }): void => { if (ended) return; + watchdog.arm(); // #6171: any upstream frame is a liveness signal try { if (typeof event.data !== "string" || Buffer.byteLength(event.data) > this.requestBytes) throw new Error("Invalid or oversized Responses WebSocket frame"); const frame: unknown = JSON.parse(event.data); @@ -656,7 +764,7 @@ class ResponsesWsResponse extends http.ServerResponse { restoredOutputItems = 0; terminal = false; - constructor(req: http.IncomingMessage, private readonly peer: WebSocket) { + constructor(req: http.IncomingMessage, private readonly peer: WebSocket, private readonly log: DiagnosticLog = () => {}) { super(req); Object.defineProperty(this, "headersSent", { get: () => this.sent }); Object.defineProperty(this, "writableEnded", { get: () => this.ended }); @@ -725,6 +833,7 @@ class ResponsesWsResponse extends http.ServerResponse { let details: unknown; try { details = JSON.parse(this.buffer); } catch { details = undefined; } const status = this.statusCode >= 400 ? this.statusCode : 502; + let code = "incomplete_response"; if (!object(details)) { this.peer.send(errorFrame("incomplete_response", "Responses exchange ended without a terminal event", status)); } else { @@ -738,8 +847,13 @@ class ResponsesWsResponse extends http.ServerResponse { : typeof rawError === "string" ? { code: "proxy_failure", message: detailText !== "" && detailText !== rawError ? `${rawError}: ${detailText}` : rawError } : { code: "incomplete_response", message: "Responses exchange ended without a terminal event" }; + if (typeof error.code === "string") code = error.code; this.peer.send(JSON.stringify({ ...details, error, type: "error", status })); } + // #6171: the client-visible terminal state was synthesized by bili, not + // received from upstream — name it so the log line pairs with the + // upstream-side failure line under the same conn/session stamp. + this.log("warn", `client terminal=synthesized code=${code} status=${status}`); } this.ended = true; this.emit("finish"); @@ -763,6 +877,10 @@ export class ResponsesWsSession implements WsBridgeSession { private readonly budget = limitsReceiveBytes(); private active?: ResponsesWsResponse; private busy = false; + // #6171: timestamp of a turn that ended without an upstream terminal event — + // the next successful turn logs how long the recovery took, pairing the + // failure line with its outcome under the same conn/session stamp. + private lastFailedExchangeAt?: number; constructor(private readonly context: WsBridgeContext) { this.transport = new ResponsesWsUpstream(context.log, undefined, context.instanceId, context.codec.allowStreamFlag === true); @@ -933,7 +1051,7 @@ export class ResponsesWsSession implements WsBridgeSession { } req.push(Buffer.from(wire)); req.push(null); - const res = new ResponsesWsResponse(req, peer); + const res = new ResponsesWsResponse(req, peer, this.context.log); res.on("error", () => {}); this.active = res; // markWsLaneTransport (#3701 Finding 1): the egress-shed byte budget keys on @@ -966,6 +1084,16 @@ export class ResponsesWsSession implements WsBridgeSession { } finally { this.active = undefined; this.busy = false; + // #6171: pair each failed exchange with its recovery outcome — the next + // terminal turn reports how long the client had to wait for a usable lane. + if (res.terminal) { + if (this.lastFailedExchangeAt !== undefined) { + this.context.log("info", `recovery=ok after_ms=${Math.max(0, Date.now() - this.lastFailedExchangeAt)}`); + this.lastFailedExchangeAt = undefined; + } + } else { + this.lastFailedExchangeAt = Date.now(); + } } }); } diff --git a/tests/issue6171-ws-failure-diagnostics.test.ts b/tests/issue6171-ws-failure-diagnostics.test.ts new file mode 100644 index 0000000000..807b09f3a4 --- /dev/null +++ b/tests/issue6171-ws-failure-diagnostics.test.ts @@ -0,0 +1,235 @@ +// #6171: pre-first-event WS failures were undiagnosable — the log line carried only +// event.type + close_code, so a raw TCP drop (undici fires 'error', then a synchronous +// 'close'(1006)) read as `event=error close_code=none`, a clean server close lost its +// reason text, nothing correlated the client-visible terminal state with the failure or +// with later recovery, and a fully stalled upstream hung the exchange until the client +// gave up (no server-side idle budget on the WS leg). This suite pins: +// (a) transport loss before the first event → two correlated lines under the SAME +// conn stamp (error + follow-up close 1006), each with elapsed_ms/received_events; +// the client gets EXACTLY ONE object-form terminal frame; the next turn succeeds +// and logs recovery=ok; +// (b) a clean server close keeps its code + cleanness on a single line, and its +// free-text reason never reaches the log (untrusted upstream content); +// (c) BILI_WS_IDLE_TIMEOUT_MS terminates a stalled exchange in one terminal state +// and the next turn recovers; +// (d) the knob resolves env > file > 10-minute default, garbage falls back. +// The refused-handshake fallback (#3342) keeps its own regression test in +// tests/responses-websocket.test.ts, updated in place for the enriched handshake lines. +import assert from "node:assert/strict"; +import http from "node:http"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { once } from "node:events"; +import { randomUUID } from "node:crypto"; +import test from "node:test"; +import WebSocket, { WebSocketServer } from "ws"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import { wsIdleTimeoutMs } from "../src/knobs.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import type { ProxyOptions } from "../src/config.ts"; + +type Item = Record; + +const testHome = fs.mkdtempSync(path.join(os.tmpdir(), "bili-6171-home-")); +process.env.XDG_STATE_HOME = testHome; +process.env.XDG_DATA_HOME = testHome; +process.env.XDG_CACHE_HOME = testHome; + +const user = (text: string): Item => ({ type: "message", role: "user", content: [{ type: "input_text", text }] }); + +interface F6171 { + sid: string; + peer: WebSocket; + proxy: http.Server; + upstream: http.Server; + wss: WebSocketServer; + logPath: string; + frame: (input: Item[], extra?: Item) => string; + exchange: (payload: string) => Promise; + close: () => Promise; +} + +/** Full codex-lane fixture; the mock upstream decides per-turn behavior via callback. */ +async function fixture(turnBehavior: (peer: WebSocket, turnIndex: number) => void): Promise { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "bili-6171-")); + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + let turns = 0; + const upstream = http.createServer((req, res) => { res.writeHead(404).end(); }); + const wss = new WebSocketServer({ server: upstream }); + wss.on("connection", peer => { + peer.on("message", raw => { + const request = JSON.parse(raw.toString()) as Item; + if (Array.isArray(request.input) && request.input.length === 0 && request.generate === false) { + peer.send(JSON.stringify({ type: "response.created", response: { id: "resp_probe", status: "in_progress", output: [] } })); + peer.send(JSON.stringify({ type: "response.completed", response: { id: "resp_probe", status: "completed", output: [], usage: { input_tokens: 0, output_tokens: 1, total_tokens: 1 } } })); + return; + } + turnBehavior(peer, turns++); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamOrigin = `http://127.0.0.1:${(upstream.address() as { port: number }).port}`; + const config = defaultConfig(200000); + config.preserveRecentTokens = 0; + const opts: ProxyOptions = { host: "127.0.0.1", port: 0, upstream: upstreamOrigin, routes: {}, modelContextLimit: 200000, kernelConfig: config, compress: { injectTool: true, injectNudge: false }, promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", log: true, logFile: path.join(tmp, "bili.log"), debug: false, passthrough: false, autoUpdate: false, mitm: { enabled: false, domains: [] } }; + const proxy = await startServer(opts); + await once(proxy, "listening"); + const proxyOrigin = `http://127.0.0.1:${(proxy.address() as { port: number }).port}`; + const sid = `codex-thread-6171-${randomUUID()}`; + const peer = new WebSocket(`${proxyOrigin.replace(/^http/, "ws")}/bili/${upstreamOrigin}/v1/responses`, { headers: { authorization: "Bearer fake-credential", "session-id": sid, "thread-id": sid, "openai-beta": "responses_websockets=2026-02-06", "user-agent": "codex_exec/0.166.0" } }); + await once(peer, "open"); + peer.on("error", () => {}); + const frame = (input: Item[], extra: Item = {}): string => JSON.stringify({ type: "response.create", model: "gpt-5.2", instructions: "You are Codex", store: false, stream: true, prompt_cache_key: sid, tools: [], input, client_metadata: { session_id: sid, thread_id: sid }, ...extra }); + const exchange = (payload: string): Promise => new Promise((resolve, reject) => { + peer.send(payload); + const events: Item[] = []; + const timer = setTimeout(() => { cleanup(); reject(new Error("WS turn timed out")); }, 20000); + const cleanup = (): void => { clearTimeout(timer); peer.off("message", onMessage); peer.off("close", onClose); }; + const onClose = (): void => { cleanup(); reject(new Error("WS client closed")); }; + const onMessage = (raw: WebSocket.RawData): void => { + const event = JSON.parse(raw.toString()) as Item; + events.push(event); + if (["response.completed", "response.failed", "response.incomplete", "error"].includes(String(event.type))) { + cleanup(); + resolve(events); + } + }; + peer.on("message", onMessage); + peer.once("close", onClose); + }); + return { + sid, peer, proxy, upstream, wss, logPath: path.join(tmp, "bili.log"), frame, exchange, + close: async () => { + peer.terminate(); + for (const client of wss.clients) client.terminate(); + await Promise.all([new Promise(resolve => proxy.close(() => resolve())), new Promise(resolve => upstream.close(() => resolve()))]); + wss.close(); + }, + }; +} + +/** The logger flushes asynchronously (#1971 class) — poll until the pattern lands. */ +async function waitForLog(logPath: string, pattern: RegExp, timeoutMs = 15000): Promise { + const deadline = Date.now() + timeoutMs; + let text = ""; + for (;;) { + text = fs.readFileSync(logPath, "utf8"); + if (pattern.test(text)) return text; + if (Date.now() > deadline) break; + await new Promise(resolve => setTimeout(resolve, 20)); + } + throw new Error(`log did not match ${pattern} within ${timeoutMs}ms; tail:\n${text.slice(-2000)}`); +} + +const failedLines = (log: string): string[] => log.split("\n").filter(line => line.includes("upstream failed")); + +test("#6171 transport loss before the first event: correlated lines, one client terminal, recovery logged", { timeout: 60000 }, async () => { + const f = await fixture((peer, turnIndex) => { + if (turnIndex === 0) peer.terminate(); // raw TCP drop: undici fires 'error' THEN a synchronous close(1006) + else answerTurn(peer); + }); + try { + const events = await f.exchange(f.frame([user("hello")])); + const errors = events.filter(e => e.type === "error"); + assert.equal(errors.length, 1, `the client must see exactly ONE terminal frame, got: ${JSON.stringify(events.map(e => String(e.type)))}`); + const error = errors[0]; + assert.equal(typeof error.error, "object", "WS-lane error contract: object form (#4601)"); + assert.equal((error.error as Item).code, "proxy_failure"); + assert.equal(error.status, 502); + assert.match(String((error.error as Item).message), /disconnected before completion/); + // Both failure lines land under the SAME conn/session stamp, each carrying + // elapsed_ms + received_events — the correlation the old line could not give. + const log = await waitForLog(f.logPath, /upstream failed phase=await-first-event event=error error_class=\w+ error_message=\S+ elapsed_ms=\d+ received_events=0/); + const log2 = await waitForLog(f.logPath, /upstream failed phase=await-first-event event=close close_code=1006 was_clean=false elapsed_ms=\d+ received_events=0/); + const lines = failedLines(log2); + assert.equal(lines.length, 2, `expected exactly two failure lines per exchange, got ${lines.length}:\n${lines.join("\n")}`); + assert.match(lines[0], /event=error error_class=\w+ error_message=\S+/); + assert.match(lines[1], /event=close close_code=1006 was_clean=false/); + const connOf = (line: string): string => line.match(/conn=(\d+)/)?.[1] ?? ""; + assert.equal(connOf(lines[0]), connOf(lines[1]), "both failure lines must share the conn stamp"); + // The client-visible terminal state is named in the log, paired with the upstream line. + await waitForLog(f.logPath, /client terminal=synthesized code=proxy_failure status=502/); + // Recovery: the next turn rides a fresh upstream connection and succeeds. + const recovered = await f.exchange(f.frame([user("again")])); + assert.ok(recovered.some(e => e.type === "response.completed"), JSON.stringify(recovered.map(e => String(e.type)))); + const log3 = await waitForLog(f.logPath, /recovery=ok after_ms=\d+/); + assert.equal(failedLines(log3).length, 2, "the recovered turn must not add failure lines"); + } finally { await f.close(); } +}); + +test("#6171 clean upstream close keeps its code on one line; the free-text reason never reaches the log", { timeout: 60000 }, async () => { + const f = await fixture((peer) => peer.close(1013, "private-upstream-close-reason-6171")); + try { + const events = await f.exchange(f.frame([user("hi")])); + const errors = events.filter(e => e.type === "error"); + assert.equal(errors.length, 1, JSON.stringify(events.map(e => String(e.type)))); + assert.equal(typeof errors[0].error, "object"); + const log = await waitForLog(f.logPath, /upstream failed phase=await-first-event event=close close_code=1013 was_clean=true elapsed_ms=\d+ received_events=0/); + const lines = failedLines(log); + assert.equal(lines.length, 1, `a clean close is ONE line (no phantom follow-up), got ${lines.length}:\n${lines.join("\n")}`); + // The close reason is arbitrary upstream-set text — untrusted content that must + // stay out of the log entirely (same contract the leak guard in + // tests/responses-websocket.test.ts pins for the stream-phase close). + assert.doesNotMatch(log, /private-upstream-close-reason-6171/, "the upstream close reason must never be logged"); + } finally { await f.close(); } +}); + +test("#6171 idle watchdog: a stalled upstream fails the exchange once, next turn recovers", { timeout: 60000 }, async () => { + const prev = process.env.BILI_WS_IDLE_TIMEOUT_MS; + process.env.BILI_WS_IDLE_TIMEOUT_MS = "400"; + let stalled = false; + try { + const f = await fixture((peer) => { + if (!stalled) { stalled = true; return; } // accept the create frame, never send anything + answerTurn(peer); + }); + try { + const events = await f.exchange(f.frame([user("stall me")])); + const errors = events.filter(e => e.type === "error"); + assert.equal(errors.length, 1, JSON.stringify(events.map(e => String(e.type)))); + assert.equal(typeof errors[0].error, "object"); + assert.equal((errors[0].error as Item).code, "proxy_failure"); + assert.match(String((errors[0].error as Item).message), /timed out before completion/); + const log = await waitForLog(f.logPath, /upstream failed phase=await-first-event event=timeout reason=idle-timeout elapsed_ms=\d+ received_events=0/); + const recovered = await f.exchange(f.frame([user("again")])); + assert.ok(recovered.some(e => e.type === "response.completed"), JSON.stringify(recovered.map(e => String(e.type)))); + const log2 = await waitForLog(f.logPath, /recovery=ok after_ms=\d+/); + assert.equal(failedLines(log2).length, 1, "one timeout line per stalled exchange"); + } finally { await f.close(); } + } finally { + if (prev === undefined) delete process.env.BILI_WS_IDLE_TIMEOUT_MS; + else process.env.BILI_WS_IDLE_TIMEOUT_MS = prev; + } +}); + +test("#6171 wsIdleTimeoutMs resolves env > file > 10-minute default; garbage falls back", () => { + const prev = process.env.BILI_WS_IDLE_TIMEOUT_MS; + try { + assert.equal(wsIdleTimeoutMs(), 600000, "owner-set default: 10 minutes"); + process.env.BILI_WS_IDLE_TIMEOUT_MS = "4321"; + assert.equal(wsIdleTimeoutMs(), 4321); + process.env.BILI_WS_IDLE_TIMEOUT_MS = "abc"; + assert.equal(wsIdleTimeoutMs(), 600000, "garbage env falls back to the default, never NaN"); + process.env.BILI_WS_IDLE_TIMEOUT_MS = "-5"; + assert.equal(wsIdleTimeoutMs(), 600000, "non-positive values fall back to the default"); + } finally { + if (prev === undefined) delete process.env.BILI_WS_IDLE_TIMEOUT_MS; + else process.env.BILI_WS_IDLE_TIMEOUT_MS = prev; + } +}); + +function answerTurn(peer: WebSocket): void { + const id = `resp_${randomUUID()}`; + const item: Item = { type: "message", id: `msg_${id}`, role: "assistant", status: "completed", content: [{ type: "output_text", text: "ok" }] }; + peer.send(JSON.stringify({ type: "response.created", response: { id, status: "in_progress", output: [] } })); + peer.send(JSON.stringify({ type: "response.output_item.added", output_index: 0, item: { ...item, status: "in_progress", content: [] } })); + peer.send(JSON.stringify({ type: "response.output_text.delta", item_id: item.id, output_index: 0, content_index: 0, delta: "ok" })); + peer.send(JSON.stringify({ type: "response.output_text.done", item_id: item.id, output_index: 0, content_index: 0, text: "ok" })); + peer.send(JSON.stringify({ type: "response.output_item.done", output_index: 0, item })); + peer.send(JSON.stringify({ type: "response.completed", response: { id, object: "response", status: "completed", output: [item], usage: { input_tokens: 5, output_tokens: 1, total_tokens: 6 } } })); +} diff --git a/tests/responses-websocket.test.ts b/tests/responses-websocket.test.ts index 79c1ee666c..c33766af32 100644 --- a/tests/responses-websocket.test.ts +++ b/tests/responses-websocket.test.ts @@ -719,7 +719,12 @@ test("Responses WS faults: refused handshake falls back to HTTP egress without l assert.equal((await response.json() as Item).status, "completed"); assert.equal(f.rows.length, 1); assert.equal(f.rows[0].request.previous_response_id, undefined); - assert.match(log.join("\n"), /failed phase=handshake event=error close_code=none/); + // #6171: the opaque `event=error close_code=none` line is now two correlated + // lines — the error event names its class/message, and the synchronous + // follow-up close (undici fires it for every non-101 upgrade refusal) keeps + // its 1006/wasClean facts instead of being dropped by the old once-listeners. + assert.match(log.join("\n"), /failed phase=handshake event=error error_class=\w+ error_message=\S+ elapsed_ms=\d+/); + assert.match(log.join("\n"), /failed phase=handshake event=close close_code=1006 was_clean=false elapsed_ms=\d+/); assert.match(log.join("\n"), /egress mode=http reason=ws-handshake-refused/); assert.match(log.join("\n"), /checkpoint reset reason=ws-refused-http-fallback/); assert.doesNotMatch(log.join("\n"), /private-|secret|Bearer/); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 6a0632a4e6..6c1e3e757e 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -11,8 +11,8 @@ entries: desc_en: "Transport timeouts, watchdogs and replay policy for upstream calls." desc_zh: "上游调用的传输超时、看门狗与重放策略。" desc_ja: "上流呼び出しのタイムアウト、ウォッチドッグ、リプレイポリシー。" - site_en: "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential." - site_zh: "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential。" + site_en: "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · wsIdleTimeoutMs 600000 · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential." + site_zh: "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · wsIdleTimeoutMs 600000 · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential。" - key: network.upstreamTimeoutMs type: number default: "720000" @@ -29,6 +29,14 @@ entries: desc_en: "Phase-1 budget: request sent → first response byte (undici resolves the fetch around the first body data). Unset follows the RESOLVED upstreamTimeoutMs — including deployments that raise BILI_UPSTREAM_TIMEOUT_MS above 12 min (#5917 review: a constant 720000 fallback would silently re-cap them) — so behavior is unchanged; tighter deployments opt in (#5904: a wedged upstream that accepts requests but never answered previously hung silently until the client aborted). After the first byte, chunk-gap silence stays governed by the full upstreamTimeoutMs (#1706/#1714 guarantee). When this value is set BELOW upstreamTimeoutMs, a first-byte timeout becomes its own failure class `upstream-headers-timeout` — zero response bytes were received, so bili replays it in-proxy under the existing retry budget (BILI_REPLAY_RETRY_MAX, default 3) instead of failing the whole request once; the mid-stream (body) idle timeout stays `upstream-timeout` and non-retried. Unset, behavior is byte-identical to before (#5807/#5831). The release gate sets 120000." desc_zh: "阶段一预算:请求发出 → 首个响应字节(undici 在首个 body 数据到达时才 resolve fetch)。未设时跟随解析后的 upstreamTimeoutMs——包括把 BILI_UPSTREAM_TIMEOUT_MS 提到 12 分钟以上的部署(#5917 评审:常量 720000 回落会悄悄把它们压回 12 分钟)——行为不变;需要更快约束首字节时延的部署可显式收紧(#5904:上游接收请求后永不应答的卡死,之前会静默拖到客户端中止)。首字节到达后,chunk 间隙静默仍由完整 upstreamTimeoutMs 管辖(#1706/#1714 保证)。当该值被收紧到低于 upstreamTimeoutMs 时,首字节超时成为独立失败类 `upstream-headers-timeout`——因为一个响应字节都未收到,bili 会在代理内按既有重试预算(BILI_REPLAY_RETRY_MAX,默认 3 次)重放,而不是让整步只失败一次;流中途(body)空闲超时仍是 `upstream-timeout` 且不重试。未设时行为与之前逐字节一致(#5807/#5831)。发布 gate 设 120000。" desc_ja: "フェーズ1の予算:リクエスト送信 → 最初のレスポンスバイト(undiciは最初のボディデータ付近でfetchを解決する)。未設定の場合、解決済みの上流タイムアウト(upstreamTimeoutMs)に従う — BILI_UPSTREAM_TIMEOUT_MS を12分以上に引き上げるデプロイメントを含む(#5917 レビュー:定数 720000 のフォールバックはそれらを静かに再制限してしまう)— したがって動作は変更されない;より厳格なデプロイメントはオプトインする(#5904:リクエストを受け入れるが以前は応答しなかったスタックした上流は、クライアントが中断するまで静かにハングしていた)。最初のバイト後、チャンク間の沈黙は引き続き完全な upstreamTimeoutMs によって管理される(#1706/#1714 の保証)。この値が upstreamTimeoutMs 未満に設定されると、最初のバイトのタイムアウトは独自の失敗クラス upstream-headers-timeout となる — レスポンスバイトがゼロであるため、bili はリクエスト全体を一度で失敗させるのではなく、既存のリトライ予算(BILI_REPLAY_RETRY_MAX、デフォルト 3)の下でプロキシ内でリプレイする;ストリーム中(ボディ)のアイドルタイムアウトは upstream-timeout のままであり、リトライされない。未設定の場合、動作は以前とバイト単位で同一である(#5807/#5831)。リリースゲートは 120000 を設定する。" + - key: network.wsIdleTimeoutMs + type: number + default: "600000" + env: BILI_WS_IDLE_TIMEOUT_MS + group: process + desc_en: "Inter-event idle budget for the Responses WebSocket leg (#6171): any upstream frame re-arms it, so steady streaming never trips it — only a full stall does. The handshake is covered too (a wedged upgrade fails fast instead of waiting forever). On fire the exchange gets its single terminal state, a correlatable `upstream failed … event=timeout reason=idle-timeout` log line under the same conn/session stamp, and the connection drops so the next turn reconnects fresh. Previously a fully stalled WS exchange hung until the client gave up — the HTTP lane's equivalent is BILI_UPSTREAM_TIMEOUT_MS." + desc_zh: "Responses WebSocket 车道的逐事件空闲预算(#6171):任何上游帧都会重新上弦,所以稳定流式永不触发——只有完全卡死才会。握手阶段同样受管(升级卡死会快速失败而不是永远等待)。触发时该交换获得唯一终态、同一 conn/session 标识下可关联的 `upstream failed … event=timeout reason=idle-timeout` 日志行,并断开连接让下一轮重新连接。此前完全卡死的 WS 交换会一直挂到客户端放弃——HTTP 车道的对应物是 BILI_UPSTREAM_TIMEOUT_MS。" + desc_ja: "Responses WebSocket レグのイベント間アイドル予算(#6171):上流フレームのたびに再アームされるため、安定したストリーミングでは決してトリガーされず、完全なスタック時のみ発火する。ハンドシェイクも対象(アップグレードがスタックしても永遠に待たず即失敗する)。発火時、その交換は単一の終端状態を取得し、同一 conn/session スタンプの下で相関可能な `upstream failed … event=timeout reason=idle-timeout` ログ行を出力し、接続を切断して次のターンで新たに再接続する。以前は完全にスタックした WS 交換はクライアントが諦めるまでハングしていた — HTTP レグの対応物は BILI_UPSTREAM_TIMEOUT_MS。" - key: network.requestWatchdogMs type: number default: "2× upstreamTimeoutMs" diff --git a/website/docs/index.html b/website/docs/index.html index 4b1172324e..0ed446b4c5 100644 --- a/website/docs/index.html +++ b/website/docs/index.html @@ -297,7 +297,7 @@

Other top-level fields

  • mitm.enabled / mitm.domains — HTTPS interception switch and whitelist domains for login clients.
  • mitm.handshakeTimeoutMs — TLS handshake timeout (default 10000).
  • Process blocks
  • -
  • network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential.
  • +
  • network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · wsIdleTimeoutMs 600000 · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential.
  • guardrails — pi/omp plugin tool-output + stream text-run guardrails: bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200.
  • persist — session persistence to disk: enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0.
  • storage — unified storage sidecar: unified legacy — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.
  • diff --git a/website/main.js b/website/main.js index 19461eee64..a377175b7d 100644 --- a/website/main.js +++ b/website/main.js @@ -132,7 +132,7 @@ "tt.ccx": "codexCompact — intercept | pass (default intercept): handle Codex's compaction requests in bili or forward upstream — explicit forms plus the inline auto-compact transiting as an ordinary turn (prompt-form detection, #3008; overridden compact_prompt transits verbatim).", "ttl.ttl": "ccrRetrievalTtlMs — lifetime of CCR retrieval pointers (default 600000; 0 disables retrieval).", "ttl.tmp": "decompressTmpCap — size cap for temporary decompressed blocks (default 50).", - "tt.network": "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential.", + "tt.network": "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · wsIdleTimeoutMs 600000 · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential.", "tt.guardrails": "guardrails — pi/omp plugin tool-output + stream text-run guardrails: bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200.", "tt.persist": "persist — session persistence to disk: enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0.", "tt.storage": "storage — unified storage sidecar: unified legacy — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.", @@ -390,7 +390,7 @@ "tt.ccx": "codexCompact —— intercept | pass(默认 intercept):在 bili 内处理 Codex 压缩请求或转给上游 —— 显式形态 + 以普通请求形态过线的内联自动压缩(prompt 文本检测,#3008;自定义 compact_prompt 按原样转发)。", "ttl.ttl": "ccrRetrievalTtlMs —— CCR 回取指针有效期(默认 600000;0 禁用回取)。", "ttl.tmp": "decompressTmpCap —— 临时展开块的大小上限(默认 50)。", - "tt.network": "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential。", + "tt.network": "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · upstreamFirstByteMs 720000 · requestWatchdogMs 2× · wsIdleTimeoutMs 600000 · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential。", "tt.guardrails": "guardrails —— pi/omp 插件工具输出与流式 text-run 守卫:bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200。", "tt.persist": "persist —— 会话落盘持久化:enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0。", "tt.storage": "storage —— 统一存储边车:unified legacy —— 文件层选定全 lane 存储模式;设置了 BILI_STORAGE_UNIFIED 时环境变量优先。",