Repository navigation
fix: client-facing socket lifecycle hardening + stall guard (#1452) - #1528
Conversation
Opt-in guard for upstream streams that go silent AFTER the first body byte: BILI_STREAM_STALL_MS=<ms> arms a per-request timer that resets on every chunk; expiry aborts the fetch with a DOMException AbortError whose message names the budget and #1452. The abort flows through the existing #721 in-band truncation chain (upstream_stream_truncated frame + [DONE] + clean FIN) — same wire outcome as an upstream that FINs mid-stream, just detected after N ms of silence instead of waiting for the client to give up. Default 0 = off: behavior is byte-identical when unset.
…at knob, exposure telemetry (#1452) clientError: Node's default disposition (no listener) destroys the socket; with residual bytes in the kernel recv buffer the OS then answers RST (the #1452 incident signature) or strands the peer's pooled connection with neither FIN nor RST. The handler drains then end()s — clean FIN in every case — with a warn marker. One handler covers plain TCP and MITM TLS legs alike (same server instance). Lifecycle ledger: every accepted socket gets a ConnRecord (kind, request count, last-response-end); on close it logs one debug line classifying the initiator — peer-fin / idle-timeout / server-end / error(code) / destroyed — so the next RST incident can be adjudicated from logs alone. End markers use performance.now() (µs): same-millisecond closes are the norm (the FIN receiver reacts immediately), only sub-ms resolution keeps causal order. idle-timeout keys off the response budget because Node's keep-alive reaper destroy()s idle sockets without prefinish/end. keepAliveTimeout: was an implicit Node default (5s); now explicit 5000ms, configurable via BILI_KEEP_ALIVE_TIMEOUT_MS (default unchanged), logged at startup. Exposure telemetry: one info line per BILI_EXPOSURE_LOG_INTERVAL_MS (default hourly, 0 disables) with liveConns/tcpHandles/handles/sessions/ blindTunnels/inFlight — ground truth for long-process fd/connection drift (both #1452 incidents died inside one 36.7h process). blindTunnels comes from a new liveBlindTunnels() counter in mitm.ts (incremented on tunnel establish, decremented once on first leg close). Shutdown force-exit warning gains (liveConns=N). All new lines are debug/info/warn-level diagnostics or default-invariant config.
Eight tests against a real proxy + real upstream fixture: - BILI_STREAM_STALL_MS env parsing (default off / positive / garbage) - clientError drains then FINs with the warn marker (no RST path) - peer-fin classification when the client ends first - idle-timeout classification at kat reap — raw net.Socket client on purpose: Node's http clients drop pooled sockets on Keep-Alive timeout=0 (sub-second kat truncation) and a socket with unread buffered data never counts as idle to the reaper (both traps documented in-file); asserts clean FIN (no error event) and the ledger line - [exposure] line shape and interval knob - stall guard: silent upstream after first byte -> in-band upstream_stream_truncated frame + [DONE] + warn naming the budget + no leaked live timers; healthy stream and default-off stay open
BILI_STREAM_STALL_MS (opt-in streaming-phase stall guard, default off), BILI_KEEP_ALIVE_TIMEOUT_MS (explicit keep-alive reap budget, default 5000ms unchanged), BILI_EXPOSURE_LOG_INTERVAL_MS (hourly exposure telemetry, 0 disables).
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.1477-pr.1528.f012b85This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr1528.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 收到。这是 PR #1528( |
… (review fix: stray leading spaces, revert whitespace churn on 2 pre-existing rows)
… fast instead of hanging CI (#1452)
[bot] 🏷 PR #1528 review complete — verdict: mergeable (CI 13/13 green on final head ① 原 issue 方向 — pass. #1452 的三轮取证已确立 bili 未主动发 RST;PR 实现的正是 owner floor-14 批准的 5 项,无 scope creep。floor-17 曾声称修复的 "loop swallows errors" 缺陷已在 floor-20 撤回——我核实最终 diff 完全不触碰 ② 回归审查 — 无意回归零。 触碰路径的既有行为逐项清点:
无 wire protocol / config schema / persistence / acp-kernel pin / version 变更——与披露清单一致。 ③ 代码审查。 5 项逐行对照改动前代码读过:ledger 分类逻辑(peer-fin/idle-timeout/server-end/error/destroyed)正确,idle-timeout 以 response budget 为键是对的(Node reaper destroy 空闲 socket 不触发 prefinish/end);µs 时间戳、close 时删记录(无泄漏)、bail timer unref(不拖进程退出);stall guard 仅在 ④ 测试有效性 — 有牙。 8 个测试全是行为级断言。我用独立 probe matrix(node v22.23.2 linux,64KiB NUL burst)实证了三态:A PR 行为 ≈310ms 干净 FIN(paused/flowing client 皆然);B destroy 在 Linux 上也是干净 FIN(RST 分支是平台/时序相关,即 Windows 现场环境);C 无 listener(Node 默认)= peer 永久挂起(无 FIN 无 RST)。所以 test 2 同时咬得住 hang 与 RST 两种回归形态。kat 测试用 raw ⑤ diff 整洁度 — 修一处后 pass。 5 文件每行都在目的内。CONFIGURATION.md 新增表格行带了前导空格(还顺带 churn 了 2 行既有行的空白)——已修回与 master 字节对齐。 Review 中直接修的 2 处(已推分支,各自独立 commit):
另发现 1 处残余边界(非阻塞,已立 issue): bail 证据汇总: 本地 typecheck clean;目标套件 8/8 ×3(含我改后复跑);probe matrix 三态实证;CI 旧 head 中文摘要:审查了 #1528(clientError 排空后关闭、连接生命周期台账、keep-alive 显式化、暴露面遥测、流式停顿守卫五项加固),方向与批准范围一致、既有行为逐项核对无意外回归,顺手修了文档表格错位和一处会让 CI 挂死的测试缺陷并立了 #1529 跟进残余边界,双 head CI 全绿,可以合并。 |
|
Deep review — verdict: FIX-REQUIRED (one mechanical docs item), no functional blockers. Everything behavioral is mergeable; mutation-tested end to end. Verified locally (branch head
|
…de default (review fix) Owner deep review measured Linux/Node 25: no-listener clientError writes 'HTTP/1.1 400 Bad Request' + close before destroying. The comment's destroy-only narrative was our Node 22/Linux strand-case measurement. Both now recorded, plus the intentional wire delta (malformed request gets FIN, not 400).
[bot] 🏷 FIX-REQUIRED 已处理 + NOTE 1 已落实,新 head ① zh/en 同步(必改项)— 已在本 PR 落地(选项 A)。 commit ② NOTE 1(400→FIN wire delta)— 两处都已做。
③ NOTE 2(kat 作用域)— 查证后无需改文档。 ④ NOTE 4(kat vs headersTimeout clamp)— 按你的定性保持 follow-up,本 PR 不动。 验证: 本地 typecheck clean、目标套件 8/8;CI 新 head 堆叠顺序不变:先合 #1528,#1530 再 rebase(rebase 时跳过空的 中文摘要:按深审意见补上了 zh-CN 三个环境变量行(逐字取自 #1530 以便其 rebase 时干净跳过)、把 clientError 注释修正为实测的 Node 默认行为并在 PR body 披露「malformed 请求得 FIN 而非 400」的有意 wire delta、核实 kat 为 server-local 无需改文档,新 head CI 13/13 全绿,可以合并。 |
Closes #1452
Hardening of the client-facing socket lifecycle after the #1452 forensics (client-side RST on the MITM chain + unguarded upstream stalls). Owner-approved scope: five items, each minimal; all are default-invariant or opt-in except two new log families, disclosed below.
What changed
1.
clientError: drain-then-close instead of destroy (src/server.ts)Node's default disposition (no listener) destroys the socket; with residual bytes in the kernel recv buffer the OS then answers RST (the #1452 incident signature) or strands the peer's pooled connection with neither FIN nor RST (behavior matrix verified on Node v22). The handler drains whatever is left, then
end()s — clean FIN in every case — with a warn marker ([conn] clientError: … — draining then closing). One handler covers both plain TCP and MITM TLS legs (same server instance).2. Connection lifecycle ledger (
src/server.ts)Every accepted socket gets a record (kind, request count, last-response-end); on close it logs one line classifying who initiated:
peer-fin/idle-timeout/server-end/error(code)/destroyed. End markers useperformance.now()(µs) — same-millisecond closes are the norm, only sub-ms resolution keeps causal order.idle-timeoutkeys off the response budget because Node's reaperdestroy()s idle sockets without prefinish/end (measured). The next RST incident can now be adjudicated from logs alone.3.
keepAliveTimeoutmade explicit +BILI_KEEP_ALIVE_TIMEOUT_MS(src/server.ts)Was an implicit Node default (5 s); now set explicitly to 5000 ms and configurable, logged at startup. Default behavior unchanged.
4. Exposure telemetry (
src/server.ts,src/mitm.ts)One info line per
BILI_EXPOSURE_LOG_INTERVAL_MS(default hourly,0disables):liveConns/tcpHandles/handles/sessions/blindTunnels/inFlight. Ground truth for long-process fd/connection drift — both #1452 incidents died inside one 36.7 h process while fresh processes stayed clean under higher load.blindTunnelscomes from a newliveBlindTunnels()counter inmitm.ts(increment on tunnel establish, decrement once on first leg close).5.
BILI_STREAM_STALL_MSstreaming-phase stall guard (src/fetch-util.ts, opt-in)Arms a per-request timer reset on every chunk after the first body byte; expiry aborts with a named AbortError. The abort flows through the existing #721 in-band truncation chain (
upstream_stream_truncatedframe +[DONE]+ clean FIN) — same wire outcome as an upstream that FINs mid-stream, detected after N ms of silence instead of waiting for the client to give up. Default0= off: byte-identical behavior when unset.Tests:
tests/issue1452-socket-hardening.test.ts— 8 tests against a real proxy + real upstream fixture (env parsing; clientError drain+FIN+marker; peer-fin classification; idle-timeout classification via raw net.Socket client — http clients drop pooled sockets onKeep-Alive: timeout=0sub-second kat and unread-buffered sockets never count as idle to the reaper, both traps documented in-file;[exposure]shape; stall guard end-to-end incl. no leaked timers; healthy-stream survival; default-off stays open). Stable 8/8 × 3 consecutive runs.Docs: CONFIGURATION.md rows for the three new env vars.
Behavior changes (old → new)
clientError)HTTP/1.1 400 Bad Request+Connection: close, then tears down (measured Linux/Node 25)debug-labeled line per closed connection (always written when logging is on, like all proxy logs)[exposure]info line (opt-out:BILI_EXPOSURE_LOG_INTERVAL_MS=0)keepAliveTimeoutBILI_KEEP_ALIVE_TIMEOUT_MS(liveConns=N)suffixBILI_STREAM_STALL_MS; wire outcome identical to existing #721 truncation frameNo wire-protocol, config-schema, persistence-format, or acp-kernel-pin changes.
versionuntouched.Pre-flight (rebased head, base = origin/master @ v0.1.163)
npm run typecheck— cleannpm test— 3021 tests, 0 fail, 12 skippednpm run build— success (dist-import-annotations OK)Note: ci-e2e auto-triggers (diff touches
src/server.ts).