diff --git a/CONFIGURATION.md b/CONFIGURATION.md index f73bced02d..3ab18b7e1d 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -822,6 +822,7 @@ Environment variables take precedence over the config file. They are useful for | `BILI_EXPOSURE_LOG_INTERVAL_MS` | Interval (ms) for the periodic long-lived-exposure telemetry line `[exposure] uptime=… liveConns=… tcpHandles=… handles=… sessions=… blindTunnels=… inFlight=…` (default `3600000` = hourly; #1452). `0` disables. Exists so socket-handle leaks and zombie connections surface in long-running logs instead of post-mortem forensics. | | `BILI_CLIENT_ERROR_BACKSTOP_MS` | Terminal backstop (ms) for the clientError drain path (#1529, follow-up to #1452 item 1): after the 300ms drain bail has `end()`ed a connection whose peer never sends FIN, the socket would otherwise sit half-open on our side indefinitely — the keep-alive reaper keys off completed responses and Node enables no SO_KEEPALIVE by default. After this much post-bail silence the proxy destroys the socket instead, classified as `reason=clienterror-backstop` in the connection lifecycle ledger with its own warn marker. Safe against the #1452 RST signature: the socket has been `resume()`-drained for the whole window, so the destroy carries no unread residual bytes. Default `30000`; `0` restores the hold-until-peer-death status quo. Non-numeric or negative values fall back to `30000`. | | `BILI_POST_RESPONSE_LINGER_MS` | Graceful-close budget (ms) after a completed response (#1982): when the proxy initiates closing a client connection right after the last response (e.g. `Connection: close`), it holds the socket open for up to this budget waiting for the peer's close signal — a TCP FIN or TLS close_notify, which can only arrive after our final byte was received+ACKed — then closes cleanly; if no signal arrives within the budget the socket is destroyed anyway and classified as `reason=linger-backstop` (warn line). This removes the abortive destroy that could race unACKed bytes into an RST on the client side (nginx `lingering_time` parity). Default `5000`; non-numeric or non-positive values fall back to `5000`. Error-driven closes, pre-handshake teardown, the clientError drain (#1529), and the idle reaper are intentionally unaffected. | +| `BILI_WS_IDLE_WARN_SECONDS` | Idle-retention warn threshold (seconds) for the WebSocket bridge (#1926 option (c)): an idle client peer pins one live upstream connection plus both checkpoints with no reclamation until it closes; once a connection's client-side silence crosses this threshold, ONE `warn` line per idle episode is logged (tagged `(#1926)`, reset by any client activity) and per-connection `idleMs` / `retainedBytes` / `inFlight` surface under `wsBridge` in `GET /__bili/stats`. Pure observability — nothing is closed. Default `1800` (30 min); `0` disables the warn scan (stats stay live); non-numeric or negative values fall back to `1800`. Scan cadence is fixed at 60 s. | | `ACP_SESSION_HEADER` | Conversation-id header name (default `x-acp-session`). | | `ACP_REASONING_KEEP` | Responses API only: set `none` to drop all reasoning items. Default routes reasoning through the compression pipeline so it is hidden automatically once its turn is summarized (prevents the unbounded accumulation that broke Codex's prompt-cache prefix). | | `ACP_RENDER_NONE` | Set to `1` to stop injecting per-message render tags (the `` `` `` markers carrying `mNNNNN` refs) into outgoing request history — applies to every wire format (OpenAI chat, Anthropic, Responses) and compact rebuilds (#933). Default is `text-only`: the model reads these refs to cite messages in `compress` calls, so only disable them once you've confirmed your workflow doesn't need ref-based compression (e.g. tag echoes leaking into client-visible output). Previously this variable was honored only on the Responses path and compact; #933 extended it to all paths. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 1390d1673e..9df559b502 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -819,6 +819,7 @@ | `BILI_EXPOSURE_LOG_INTERVAL_MS` | 长驻暴露遥测行 `[exposure] uptime=… liveConns=… tcpHandles=… handles=… sessions=… blindTunnels=… inFlight=…` 的周期(毫秒,默认 `3600000` 即每小时;#1452)。`0` 关闭。目的是让套接字句柄泄漏与僵尸连接在长期运行日志中现形,而不是靠事后取证。 | | `BILI_CLIENT_ERROR_BACKSTOP_MS` | clientError 排空路径的终局兜底(毫秒)(#1529,#1452 第 1 项后续):排空 bail(300ms)对连接调用 `end()` 后,若对端始终不发 FIN,该套接字否则会在我方无限期半开滞留——keep-alive 回收器以已完成响应为键,且 Node 默认不开 SO_KEEPALIVE。bail 后静默超过此值时,代理改为销毁该套接字,在连接生命周期台账中分类为 `reason=clienterror-backstop` 并带独立的 warn 标记。对 #1452 的 RST 签名安全:整个窗口内套接字一直处于 `resume()` 排空状态,销毁时不携带未读残留字节。默认 `30000`;`0` 恢复「持有直到对端死亡」的旧行为。非数字或负值回退到 `30000`。 | | `BILI_POST_RESPONSE_LINGER_MS` | 响应完成后的优雅关闭预算(毫秒)(#1982):代理在最后一个响应结束后主动关闭客户端连接(如 `Connection: close`)时,最多为此预算时长持有该套接字,等待对端的关闭信号——TCP FIN 或 TLS close_notify(它只能在我方最后字节被接收并 ACK 之后到达)——然后干净地关闭;预算内无信号则照常销毁套接字,分类为 `reason=linger-backstop`(warn 行)。此举消除了可能在客户端侧把未 ACK 字节竞态成 RST 的激进销毁(与 nginx `lingering_time` 对齐)。默认 `5000`;非数字或非正值回退到 `5000`。错误驱动的关闭、握手前拆除、clientError 排空(#1529)与空闲回收器均刻意不受影响。 | +| `BILI_WS_IDLE_WARN_SECONDS` | WebSocket 桥接的空闲保留告警阈值(秒)(#1926 选项 (c)):一个空闲的客户端对端会固定占用一条活跃的上游连接及两个检查点,且在其关闭前不会被回收;当某条连接的客户端侧静默时长超过该阈值时,每个空闲事件仅记录一行 `warn` 日志(标记 `(#1926)`,任何客户端活动即重置),同时每连接的 `idleMs` / `retainedBytes` / `inFlight` 指标会在 `GET /__bili/stats` 的 `wsBridge` 下暴露。纯可观测性——不会关闭任何连接。默认值 `1800`(30 分钟);设为 `0` 则禁用告警扫描(stats 仍保持实时);非数字或负值回退为 `1800`。扫描周期固定为 60 秒。 | | `ACP_SESSION_HEADER` | 会话 id 请求头名称(默认 `x-acp-session`)。 | | `ACP_REASONING_KEEP` | 仅 Responses API:设 `none` 丢弃全部 reasoning 项。默认让 reasoning 走压缩管道,其轮次被摘要后自动隐藏(避免无限累积破坏 Codex 的 prompt-cache 前缀)。 | | `ACP_RENDER_NONE` | 设为 `1` 停止向出站请求历史注入逐消息渲染标签(承载 `mNNNNN` ref 的 `` `` `` 标记)——适用于所有线格式(OpenAI chat、Anthropic、Responses)及 compact 重建(#933)。默认 `text-only`:模型靠这些 ref 在 `compress` 调用中引用消息,只有确认自己的工作流不需要基于 ref 的压缩(例如标签回声泄漏到客户端可见输出)后才应禁用。此前该变量仅在 Responses 路径与 compact 上生效;#933 扩展到了所有路径。 | diff --git a/docs/opencode-responses-websocket.md b/docs/opencode-responses-websocket.md index 9fbba09bf0..67ad166829 100644 --- a/docs/opencode-responses-websocket.md +++ b/docs/opencode-responses-websocket.md @@ -84,6 +84,21 @@ Unknown protocols cannot be compressed (folding requires knowing where history lives in the wire format); they stay on the #1472 transparent passthrough lane and remain untouched by this bridge. +## Idle-socket observability (#1926, option (c)) + +Idle client peers each pin one live upstream connection plus both checkpoints, +with no reclamation until the client closes (idle-close (a)/(b) awaits +verified retry-full host recovery — see #1926). Until then the lane surfaces +the retention: + +- `GET /__bili/stats` gains a `wsBridge` object: per live connection — codec, + connection id, `idleMs`, `retainedBytes` (client + upstream checkpoints), + `inFlight`. +- A scan (default every 60s) logs one `warn` per idle episode per connection + once idle age crosses the threshold (default 30min; + `BILI_WS_IDLE_WARN_SECONDS` overrides, `0` disables), tagged `(#1926)` and + reset by client activity. Pure observability — nothing is closed. + ## Verification Use random loopback ports and fake Responses WebSocket upstreams, with no real diff --git a/src/responses-ws.ts b/src/responses-ws.ts index c1b903e972..f9f2053b83 100644 --- a/src/responses-ws.ts +++ b/src/responses-ws.ts @@ -109,6 +109,13 @@ export class ResponsesWsHistory { clear(): void { this.checkpoint = undefined; } + + /** #1926 observability: bytes retained by the live checkpoint. */ + bytes(): number { + const checkpoint = this.checkpoint; + if (!checkpoint) return 0; + return Buffer.byteLength(JSON.stringify(checkpoint.request)) + Buffer.byteLength(JSON.stringify(checkpoint.output)); + } } // The pipeline emits JSON bodies as strings (prepare* re-serialization) or @@ -135,6 +142,11 @@ export class ResponsesWsUpstream { this.log("debug", `upstream checkpoint reset reason=${reason}`); } + /** #1926 observability: bytes retained by the upstream-leg checkpoint. */ + retainedBytes(): number { + return this.history.bytes(); + } + close(reason = "transport-close"): void { const socket = this.socket; this.socket = undefined; @@ -393,6 +405,10 @@ class ResponsesWsSession implements WsBridgeSession { context.log("info", "Responses socket connected (ACP request pipeline)"); } + stats(): { retainedBytes: number; inFlight: boolean } { + return { retainedBytes: this.history.bytes() + this.transport.retainedBytes(), inFlight: this.busy }; + } + onMessage(data: Buffer, binary: boolean): void { const peer = this.context.peer; if (binary) { peer.close(1003, "Responses requires text frames"); return; } diff --git a/src/server.ts b/src/server.ts index 1a3d8d25bc..e280bddc17 100644 --- a/src/server.ts +++ b/src/server.ts @@ -130,7 +130,12 @@ import { applyOutputSteering, applyOutputSteeringJson } from "./output-steering. import { bodyDumpEnabled, getUnrecognizedPathStats, isModelDiscoveryPath, logDumpFailure, logUnrecognizedPath } from "./server/observability.js"; import { BILI_HOP_HEADER, anthropicBetaContextWindow, capRegistryWindowByStandard, expandedContextSuffixWindow, LAUNCHER_MODEL_WINDOWS, LAUNCHER_MODEL_MAX_OUTPUTS, launcherContextWindow, launcherMaxOutput, parseLauncherModelWindows, windowSourceLogged } from "./server/context-window.js"; import { buildForwardHeaders, connectionNamedHeaders, NO_IDENTITY_MESSAGE, RESPONSE_ONLY_STRIP_HEADERS, safeSessionId, UPSTREAM_HOP_HEADERS } from "./server/headers.js"; -import { installWebSocketBridge } from "./ws-bridge.js"; +import { installWebSocketBridge, type WsBridgeStats } from "./ws-bridge.js"; + +// #1926 observability: the WS bridge handle's stats accessor, published once +// the server installs the bridge so `GET /__bili/stats` can report live WS +// connections, idle age, and retained checkpoint bytes per connection. +let wsBridgeStats: (() => WsBridgeStats) | undefined; import { codexResponsesCodec, responsesCodec } from "./responses-ws.js"; import { currentFetchTransport } from "./fetch-transport.js"; import { isSideRequest, outputBudgetField, restoreOutputBudget, SIDE_REQUEST_MAX_TOKENS, sideRequestGuard, stripLeakedBiliTools } from "./server/side-request.js"; @@ -468,9 +473,11 @@ export async function startServer(opts: ProxyOptions): Promise { }; const server = http.createServer(dispatch); // Generic WebSocket bridge: protocol codecs claim upgrades here (#1467 - // phase-2 shell); the Responses codec is the first (and currently only) - // entry. Unclaimed upgrades still fall through to the 426 contract below. + // phase-2 shell); the Responses codecs (opencode plugin lane + codex + // prefix lane) are the current entries. Unclaimed upgrades still fall + // through to the 426 contract below. const wsUpgrade = installWebSocketBridge(server, dispatch, log, [responsesCodec, codexResponsesCodec]); + wsBridgeStats = wsUpgrade.stats; // Unclaimed upgrades retain the immediate HTTP fallback contract. // An explicit 'upgrade' listener is // required: without one Node's behavior is version-dependent (some @@ -7302,7 +7309,7 @@ function sendStats(res: http.ServerResponse): void { }; }); res.writeHead(200, { "content-type": "application/json" }); - res.end(JSON.stringify({ sessions, blindTunnels: getBlindTunnelStats(), unrecognizedPaths: getUnrecognizedPathStats(), conflicts: summarizeConflicts(all) }, null, 2)); + res.end(JSON.stringify({ sessions, blindTunnels: getBlindTunnelStats(), unrecognizedPaths: getUnrecognizedPathStats(), conflicts: summarizeConflicts(all), wsBridge: wsBridgeStats?.() ?? { connections: [] } }, null, 2)); } /** Stale-install state for the web UI badge (#811): whether the on-disk diff --git a/src/ws-bridge.ts b/src/ws-bridge.ts index 355fdc9324..29aa9cb588 100644 --- a/src/ws-bridge.ts +++ b/src/ws-bridge.ts @@ -30,6 +30,12 @@ export interface WsBridgeSession { onClose(code: number): void; /** The server is shutting down; fail in-flight exchanges and drop upstream transports. */ shutdown(reason: string): void; + /** + * Optional resource accounting for the #1926 observability surface + * (`/__bili/stats` + the idle-retention warn line). Implementations report + * the bytes their checkpoints retain and whether an exchange is in flight. + */ + stats?(): { retainedBytes?: number; inFlight?: boolean }; } export interface WsBridgeContext { @@ -76,29 +82,82 @@ export interface WsBridgeCodec { createSession(context: WsBridgeContext): WsBridgeSession; } +export interface WsBridgeStats { + connections: { codec: string; connection: number; idleMs: number; retainedBytes?: number; inFlight?: boolean }[]; +} + +export interface WsBridgeHandle { + /** The upgrade handler; wire into the http server's "upgrade" event. */ + (req: http.IncomingMessage, socket: Duplex, head: Buffer): boolean; + /** #1926 observability: live connections, idle age, retained bytes. */ + stats(): WsBridgeStats; +} + +export interface WsBridgeOptions { + /** Clock injection for tests. */ + now?: () => number; + /** Idle-scan cadence (ms); default 60_000. */ + scanMs?: number; + /** Idle threshold (ms) before the retention warn line fires; default 30 min, 0 disables. */ + idleWarnMs?: number; +} + +interface BridgeEntry { + session: WsBridgeSession; + peer: WebSocket; + codec: string; + connection: number; + label: string; + lastActivityAt: number; + idleWarned: boolean; +} + export function installWebSocketBridge( server: http.Server, dispatch: (req: http.IncomingMessage, res: http.ServerResponse) => Promise, log: (level: string, message: string) => void, codecs: readonly WsBridgeCodec[], -): (req: http.IncomingMessage, socket: Duplex, head: Buffer) => boolean { + options: WsBridgeOptions = {}, +): WsBridgeHandle { const wss = new WebSocketServer({ noServer: true, maxPayload: MAX_REQUEST_BYTES, perMessageDeflate: false }); - const sessions = new Set<{ session: WsBridgeSession; peer: WebSocket }>(); + const sessions = new Set(); let connectionId = 0; + const now = options.now ?? Date.now; + const scanMs = options.scanMs ?? 60_000; + const idleWarnSeconds = Number(process.env.BILI_WS_IDLE_WARN_SECONDS ?? 1800); + // Non-numeric or negative env values fall back to the 30-minute default instead of silently disabling the scan (sibling-knob convention, e.g. BILI_MODEL_INFO_RETRY_MS). + const idleWarnMs = options.idleWarnMs ?? (Number.isFinite(idleWarnSeconds) && idleWarnSeconds >= 0 ? idleWarnSeconds : 1800) * 1000; + // #1926: idle client peers pin one live upstream connection plus both + // checkpoints each, with no reclamation until the client closes. Until + // idle-close (a)/(b) is verified against the retry-full contract, surface + // the retention: one warn line per idle episode per connection (reset on + // activity), and per-connection stats via `stats()` below. + const scanner = idleWarnMs > 0 && scanMs > 0 ? setInterval(() => { + for (const entry of sessions) { + const idleMs = now() - entry.lastActivityAt; + if (entry.idleWarned || idleMs < idleWarnMs) continue; + entry.idleWarned = true; + const retained = entry.session.stats?.().retainedBytes; + log("warn", `${entry.label} idle ${Math.round(idleMs / 1000)}s: socket pins a live upstream connection and checkpoints${retained === undefined ? "" : ` (retained≈${retained} bytes)`} — no reclamation until client close (#1926)`); + } + }, scanMs) : undefined; + scanner?.unref(); const close = server.close.bind(server); server.close = callback => { const live = [...sessions]; sessions.clear(); // exactly-once: the 'close' event backstop below must not re-run + scanner?.close(); for (const { session } of live) session.shutdown("server-close"); for (const peer of wss.clients) peer.terminate(); return close(callback); }; server.on("close", () => { + scanner?.close(); for (const { session } of sessions) session.shutdown("server-close"); for (const peer of wss.clients) peer.terminate(); wss.close(); }); - return (source, socket, head) => { + const upgrade = (source: http.IncomingMessage, socket: Duplex, head: Buffer): boolean => { const admitted = isLoopbackAddress(source.socket.remoteAddress); const claim = admitted ? codecs.flatMap(candidate => { const conversation = source.headers[candidate.conversationHeader ?? "x-bili-plugin-conversation"]; @@ -125,17 +184,22 @@ export function installWebSocketBridge( } if (socket.destroyed) return; wss.handleUpgrade(source, socket, head, peer => { - const label = `[${codec.name}] [conn=${++connectionId}] [session=${JSON.stringify(conversationId.slice(0, 128))}]`; + const connection = ++connectionId; + const label = `[${codec.name}] [conn=${connection}] [session=${JSON.stringify(conversationId.slice(0, 128))}]`; const trace: WsBridgeLog = (level, message) => log(level, `${label} ${message}`); const session = codec.createSession({ log: trace, peer, source, upstreamUrl: upstream, dispatch, codec }); - const entry = { session, peer }; + const entry: BridgeEntry = { session, peer, codec: codec.name, connection, label, lastActivityAt: now(), idleWarned: false }; sessions.add(entry); peer.on("error", () => {}); peer.on("close", (code: number) => { session.onClose(code); sessions.delete(entry); }); - peer.on("message", (data: Buffer, binary: boolean) => session.onMessage(data, binary)); + peer.on("message", (data: Buffer, binary: boolean) => { + entry.lastActivityAt = now(); + entry.idleWarned = false; + session.onMessage(data, binary); + }); }); })().catch(() => { log("warn", `[${codec.name}] upgrade failed`); @@ -143,4 +207,11 @@ export function installWebSocketBridge( }); return true; }; + const stats = (): WsBridgeStats => ({ + connections: [...sessions].map(entry => { + const entryStats = entry.session.stats?.(); + return { codec: entry.codec, connection: entry.connection, idleMs: Math.max(0, now() - entry.lastActivityAt), retainedBytes: entryStats?.retainedBytes, inFlight: entryStats?.inFlight }; + }), + }); + return Object.assign(upgrade, { stats }); } diff --git a/tests/codex-responses-ws.test.ts b/tests/codex-responses-ws.test.ts index e689a2f7e0..69fb36e125 100644 --- a/tests/codex-responses-ws.test.ts +++ b/tests/codex-responses-ws.test.ts @@ -141,7 +141,7 @@ test("codex WS: prewarm probe plus full-replay turns ride the prefix lane throug assert.match(String(f.rows[0].headers["user-agent"]), /^codex_exec\//); // The real turn is a full replay the pipeline processed: bili's ACP // tools ride alongside codex's own (empty) tool list. - assert.ok(Array.isArray(f.rows[1].request.tools) && (f.rows[1].request.tools as Item[]).some(t => t.name === "acp_status" || t.name === "compress"), JSON.stringify((f.rows[1].request.tools as Item[] | undefined)?.map?.((t: Item) => t.name))); + assert.ok(Array.isArray(f.rows[1].request.tools) && (f.rows[1].request.tools as Item[]).some(t => t.name === "acp_status" || t.name === "compress"), JSON.stringify((f.rows[1].request.tools as Item[] | undefined)?.map(t => t.name))); assert.ok(JSON.stringify(f.rows[1].request.input).includes("codex-first-turn")); const session = peekSession(f.sid); assert.ok(session); @@ -184,12 +184,12 @@ test("codex WS: folding covered history shrinks the replayed turn", { timeout: 3 const play = async (text: string): Promise => { const reply = completed(await f.turn([...replay, user(text)])); replay.push(user(text)); - const message = Array.isArray(reply.output) ? (reply.output.find(item => item && typeof item === "object" && item.type === "message") as Item | undefined) : undefined; + const message = Array.isArray(reply.output) ? (reply.output.find((item): item is Item => !!item && typeof item === "object" && item.type === "message")) : undefined; if (message) replay.push(message); }; await play("session purpose: codex prefix-lane ACP integration"); await play("OLD-BULKY-SENTINEL " + "unique filler content ".repeat(1200)); - const firstText = JSON.stringify((f.rows.at(-1)!.request.input as Item[]).find((item: Item) => JSON.stringify(item).includes("OLD-BULKY-SENTINEL"))); + const firstText = JSON.stringify((f.rows.at(-1)!.request.input as Item[]).find(item => JSON.stringify(item).includes("OLD-BULKY-SENTINEL"))); const ref = firstText.match(/\x3cacp[^\x3e]*\x3e(m\d+)\x3c\/acp\x3e/)?.[1]; assert.ok(ref, firstText.slice(-300)); for (let i = 0; i < 4; i++) await play(`push ${i}`); diff --git a/tests/issue1987-preflight-baseline-anchor.test.ts b/tests/issue1987-preflight-baseline-anchor.test.ts index f73ffabf8b..1f60bdf68c 100644 --- a/tests/issue1987-preflight-baseline-anchor.test.ts +++ b/tests/issue1987-preflight-baseline-anchor.test.ts @@ -1,5 +1,6 @@ import assert from "node:assert/strict"; import http from "node:http"; +import type { AddressInfo } from "node:net"; import { once } from "node:events"; import test from "node:test"; @@ -10,7 +11,6 @@ process.env.BILI_REPLAY_RETRY_MAX = "1"; import { defaultConfig, type Config } from "acp-kernel"; import { startServer, type ProxyOptions } from "../src/server.ts"; import { type CompressSettings } from "../src/config.ts"; -import type { AddressInfo } from "node:net"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; import { getSession } from "../src/session.ts"; diff --git a/tests/issue1993-preflight-400.test.ts b/tests/issue1993-preflight-400.test.ts index c4d5398b24..fbe4edb4ef 100644 --- a/tests/issue1993-preflight-400.test.ts +++ b/tests/issue1993-preflight-400.test.ts @@ -14,7 +14,8 @@ process.env.BILI_REPLAY_RETRY_BASE_MS = "0"; process.env.BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS = "400"; import { defaultConfig } from "acp-kernel"; -import { startServer, type ProxyOptions } from "../src/server.ts"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; import { listSessions } from "../src/session.ts"; diff --git a/tests/responses-websocket.test.ts b/tests/responses-websocket.test.ts index ca11303f81..908a86ca04 100644 --- a/tests/responses-websocket.test.ts +++ b/tests/responses-websocket.test.ts @@ -668,9 +668,9 @@ test("Responses WS: a title-persona side request rides the lane transport instea const events = await f.turn([user("title over the WS lane")], undefined, {}, titlePeer); const done = events.find(e => e.type === "response.completed"); assert.ok(done, JSON.stringify(events)); - const log = fs.readFileSync(f.logPath, "utf8"); - assert.match(log, /side request \(agent=title\)/); - assert.doesNotMatch(log, /Unsupported request in Responses WebSocket transport/); + // Poll the log: on slow runners the side-request line can lag the + // completed frame (same race family the other sites fixed). + await readLogUntil(f.logPath, [/side request \(agent=title\)/], /Unsupported request in Responses WebSocket transport/); } finally { await f.close(); } diff --git a/tests/ws-bridge.test.ts b/tests/ws-bridge.test.ts index 6d2a20fa7e..4f13254c26 100644 --- a/tests/ws-bridge.test.ts +++ b/tests/ws-bridge.test.ts @@ -15,6 +15,7 @@ class RecordingSession implements WsBridgeSession { static readonly live = new Set(); readonly events: string[] = []; shutdowns: string[] = []; + retainedBytes = 4096; constructor(readonly context: Parameters[0]) { RecordingSession.live.add(this); context.log("info", `fake socket connected upstream=${context.upstreamUrl}`); @@ -30,6 +31,9 @@ class RecordingSession implements WsBridgeSession { shutdown(reason: string): void { this.shutdowns.push(reason); } + stats(): { retainedBytes: number; inFlight: boolean } { + return { retainedBytes: this.retainedBytes, inFlight: this.events.length === 1 }; + } } const fakeCodec: WsBridgeCodec = { @@ -39,10 +43,10 @@ const fakeCodec: WsBridgeCodec = { createSession: context => new RecordingSession(context), }; -async function harness(codecs: readonly WsBridgeCodec[]) { +async function harness(codecs: readonly WsBridgeCodec[], options: Parameters[4] = {}) { const server = http.createServer((_req, res) => res.writeHead(404).end()); const logs: string[] = []; - const handler = installWebSocketBridge(server, async (_req, res) => { res.writeHead(200).end(); }, (level, message) => logs.push(`${level} ${message}`), codecs); + const handler = installWebSocketBridge(server, async (_req, res) => { res.writeHead(200).end(); }, (level, message) => logs.push(`${level} ${message}`), codecs, options); server.on("upgrade", (req, socket, head) => { if (!handler(req, socket, head)) socket.destroy(); }); await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); return { server, handler, logs, port: (server.address() as AddressInfo).port }; @@ -75,6 +79,57 @@ test("ws bridge: second codec claims its own path+marker and exchanges frames", } }); +test("ws bridge: stats() reports live connections, idle age and retained bytes; idle warn fires once per episode (#1926)", async () => { + const upstream = http.createServer(); + await new Promise(resolve => upstream.listen(0, "127.0.0.1", resolve)); + const upstreamPort = (upstream.address() as AddressInfo).port; + let clock = 1_000_000; + const { server, handler, logs, port } = await harness([fakeCodec], { now: () => clock, scanMs: 5, idleWarnMs: 1_800_000 }); + try { + const socket = new WebSocket(`ws://127.0.0.1:${port}/bili/fake/http://127.0.0.1:${upstreamPort}/echo`, { + headers: { "x-bili-plugin": "fakehost", "x-bili-plugin-conversation": "ses_stats" }, + }); + await new Promise((resolve, reject) => { socket.on("open", resolve); socket.on("error", reject); }); + socket.send("ping"); + await new Promise(resolve => setTimeout(resolve, 25)); // let the frame land + one scan pass at idle ~0 + // Idle age grows with the injected clock. + clock += 60_000; + const early = handler.stats(); + assert.equal(early.connections.length, 1); + assert.equal(early.connections[0].codec, "fake-ws"); + assert.equal(early.connections[0].retainedBytes, 4096); + assert.equal(early.connections[0].inFlight, true); + assert.ok(Math.abs(early.connections[0].idleMs - 60_000) < 1, `idleMs ${early.connections[0].idleMs}`); + assert.ok(!logs.some(line => line.includes("(#1926)")), "no warn before the threshold"); + // Cross the idle threshold → exactly one warn line (seconds may lag the + // threshold slightly — the scan runs on a wall-clock interval). + clock += 1_800_000; + const deadline = Date.now() + 5000; + while (!logs.some(line => /idle 18\d\ds/.test(line) && line.includes("(#1926)")) && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 25)); + const warns = logs.filter(line => line.includes("(#1926)")); + assert.equal(warns.length, 1, `expected one idle warn, got: ${logs.join(" | ")}`); + assert.ok(warns[0].includes("[fake-ws] [conn=1]")); + assert.ok(warns[0].includes("retained≈4096 bytes")); + // Activity resets the episode — a later idle crossing is a NEW warn, + // but only one per episode. + socket.send("pong"); + await new Promise(resolve => setTimeout(resolve, 25)); + clock += 2_400_000; + const deadline2 = Date.now() + 5000; + while (logs.filter(line => line.includes("(#1926)")).length < 2 && Date.now() < deadline2) await new Promise(resolve => setTimeout(resolve, 25)); + assert.equal(logs.filter(line => line.includes("(#1926)")).length, 2, `second episode must warn exactly once, got: ${logs.join(" | ")}`); + // Close removes the connection from stats(). + socket.close(1000); + await new Promise(resolve => socket.on("close", resolve)); + const deadline3 = Date.now() + 5000; + while (handler.stats().connections.length > 0 && Date.now() < deadline3) await new Promise(resolve => setTimeout(resolve, 25)); + assert.equal(handler.stats().connections.length, 0); + } finally { + await new Promise(resolve => server.close(resolve)); + await new Promise(resolve => upstream.close(resolve)); + } +}); + test("ws bridge: marker mismatch or foreign path falls through unclaimed", async () => { const { server, handler } = await harness([fakeCodec]); try {