Skip to content

fix: client-facing socket lifecycle hardening + stall guard (#1452) - #1528

Merged
ranxianglei merged 8 commits into
masterfrom
2026-09-27_mitm-rst-idle-hardening
Sep 28, 2026
Merged

ranxianglei merged 8 commits into
masterfrom
2026-09-27_mitm-rst-idle-hardening

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Closes #1452

Hardening of the client-facing socket lifecycle after the #1452 forensics (client-side RST on the MITM chain + unguarded upstream stalls). Owner-approved scope: five items, each minimal; all are default-invariant or opt-in except two new log families, disclosed below.

What changed

1. clientError: drain-then-close instead of destroy (src/server.ts)
Node's default disposition (no listener) destroys the socket; with residual bytes in the kernel recv buffer the OS then answers RST (the #1452 incident signature) or strands the peer's pooled connection with neither FIN nor RST (behavior matrix verified on Node v22). The handler drains whatever is left, then end()s — clean FIN in every case — with a warn marker ([conn] clientError: … — draining then closing). One handler covers both plain TCP and MITM TLS legs (same server instance).

2. Connection lifecycle ledger (src/server.ts)
Every accepted socket gets a record (kind, request count, last-response-end); on close it logs one line classifying who initiated: peer-fin / idle-timeout / server-end / error(code) / destroyed. End markers use performance.now() (µs) — same-millisecond closes are the norm, only sub-ms resolution keeps causal order. idle-timeout keys off the response budget because Node's reaper destroy()s idle sockets without prefinish/end (measured). The next RST incident can now be adjudicated from logs alone.

3. keepAliveTimeout made explicit + BILI_KEEP_ALIVE_TIMEOUT_MS (src/server.ts)
Was an implicit Node default (5 s); now set explicitly to 5000 ms and configurable, logged at startup. Default behavior unchanged.

4. Exposure telemetry (src/server.ts, src/mitm.ts)
One info line per BILI_EXPOSURE_LOG_INTERVAL_MS (default hourly, 0 disables): liveConns/tcpHandles/handles/sessions/blindTunnels/inFlight. Ground truth for long-process fd/connection drift — both #1452 incidents died inside one 36.7 h process while fresh processes stayed clean under higher load. blindTunnels comes from a new liveBlindTunnels() counter in mitm.ts (increment on tunnel establish, decrement once on first leg close).

5. BILI_STREAM_STALL_MS streaming-phase stall guard (src/fetch-util.ts, opt-in)
Arms a per-request timer reset on every chunk after the first body byte; expiry aborts with a named AbortError. The abort flows through the existing #721 in-band truncation chain (upstream_stream_truncated frame + [DONE] + clean FIN) — same wire outcome as an upstream that FINs mid-stream, detected after N ms of silence instead of waiting for the client to give up. Default 0 = off: byte-identical behavior when unset.

Tests: tests/issue1452-socket-hardening.test.ts — 8 tests against a real proxy + real upstream fixture (env parsing; clientError drain+FIN+marker; peer-fin classification; idle-timeout classification via raw net.Socket client — http clients drop pooled sockets on Keep-Alive: timeout=0 sub-second kat and unread-buffered sockets never count as idle to the reaper, both traps documented in-file; [exposure] shape; stall guard end-to-end incl. no leaked timers; healthy-stream survival; default-off stays open). Stable 8/8 × 3 consecutive runs.

Docs: CONFIGURATION.md rows for the three new env vars.

Behavior changes (old → new)

Area Old New Why
Malformed client input (clientError) silent destroy → RST-with-unread-bytes or peer hang drain-then-end + warn marker removes the only unlogged, non-deterministic client-facing teardown path (#1452)
Malformed-request response Node default writes HTTP/1.1 400 Bad Request + Connection: close, then tears down (measured Linux/Node 25) immediate bare FIN, no 400 (measured) wire-visible delta of the item-1 disposition: for input we cannot parse, a 400 adds nothing, and the old teardown could surface as RST or a stranded pool slot (#1452)
Log volume — one debug-labeled line per closed connection (always written when logging is on, like all proxy logs) adjudicating future RST incidents needs the initiator fact
Telemetry none hourly [exposure] info line (opt-out: BILI_EXPOSURE_LOG_INTERVAL_MS=0) long-process drift was unfalsifiable without periodic ground truth
keepAliveTimeout implicit Node default (5 s) explicit 5000 ms + BILI_KEEP_ALIVE_TIMEOUT_MS default unchanged; knob for operators
Shutdown force-exit warning no connection count gains (liveConns=N) suffix log-format only
Mid-stream upstream silence no guard (client-dependent) opt-in BILI_STREAM_STALL_MS; wire outcome identical to existing #721 truncation frame #1452 question 2; off by default

No wire-protocol, config-schema, persistence-format, or acp-kernel-pin changes. version untouched.

Pre-flight (rebased head, base = origin/master @ v0.1.163)

  • npm run typecheck — clean
  • npm test — 3021 tests, 0 fail, 12 skipped
  • npm run build — success (dist-import-annotations OK)
  • target file × 3 — 8/8 each

Note: ci-e2e auto-triggers (diff touches src/server.ts).

ework-agent added 4 commits September 28, 2026 01:57
Opt-in guard for upstream streams that go silent AFTER the first body
byte: BILI_STREAM_STALL_MS=<ms> arms a per-request timer that resets on
every chunk; expiry aborts the fetch with a DOMException AbortError whose
message names the budget and #1452. The abort flows through the existing
#721 in-band truncation chain (upstream_stream_truncated frame + [DONE]
+ clean FIN) — same wire outcome as an upstream that FINs mid-stream,
just detected after N ms of silence instead of waiting for the client to
give up. Default 0 = off: behavior is byte-identical when unset.
…at knob, exposure telemetry (#1452)

clientError: Node's default disposition (no listener) destroys the socket;
with residual bytes in the kernel recv buffer the OS then answers RST
(the #1452 incident signature) or strands the peer's pooled connection
with neither FIN nor RST. The handler drains then end()s — clean FIN in
every case — with a warn marker. One handler covers plain TCP and MITM
TLS legs alike (same server instance).

Lifecycle ledger: every accepted socket gets a ConnRecord (kind, request
count, last-response-end); on close it logs one debug line classifying
the initiator — peer-fin / idle-timeout / server-end / error(code) /
destroyed — so the next RST incident can be adjudicated from logs alone.
End markers use performance.now() (µs): same-millisecond closes are the
norm (the FIN receiver reacts immediately), only sub-ms resolution keeps
causal order. idle-timeout keys off the response budget because Node's
keep-alive reaper destroy()s idle sockets without prefinish/end.

keepAliveTimeout: was an implicit Node default (5s); now explicit 5000ms,
configurable via BILI_KEEP_ALIVE_TIMEOUT_MS (default unchanged), logged
at startup.

Exposure telemetry: one info line per BILI_EXPOSURE_LOG_INTERVAL_MS
(default hourly, 0 disables) with liveConns/tcpHandles/handles/sessions/
blindTunnels/inFlight — ground truth for long-process fd/connection drift
(both #1452 incidents died inside one 36.7h process). blindTunnels comes
from a new liveBlindTunnels() counter in mitm.ts (incremented on tunnel
establish, decremented once on first leg close).

Shutdown force-exit warning gains (liveConns=N). All new lines are
debug/info/warn-level diagnostics or default-invariant config.
Eight tests against a real proxy + real upstream fixture:
- BILI_STREAM_STALL_MS env parsing (default off / positive / garbage)
- clientError drains then FINs with the warn marker (no RST path)
- peer-fin classification when the client ends first
- idle-timeout classification at kat reap — raw net.Socket client on
  purpose: Node's http clients drop pooled sockets on Keep-Alive
  timeout=0 (sub-second kat truncation) and a socket with unread
  buffered data never counts as idle to the reaper (both traps
  documented in-file); asserts clean FIN (no error event) and the
  ledger line
- [exposure] line shape and interval knob
- stall guard: silent upstream after first byte -> in-band
  upstream_stream_truncated frame + [DONE] + warn naming the budget +
  no leaked live timers; healthy stream and default-off stay open
BILI_STREAM_STALL_MS (opt-in streaming-phase stall guard, default off),
BILI_KEEP_ALIVE_TIMEOUT_MS (explicit keep-alive reap budget, default
5000ms unchanged), BILI_EXPOSURE_LOG_INTERVAL_MS (hourly exposure
telemetry, 0 disables).
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-27_mitm-rst-idle-hardening (3331a62)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1477-pr.1528.f012b85

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1528.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到。这是 PR #1528(fix: client-facing socket lifecycle hardening + stall guard,Closes #1452)的审核唤醒——我不会重做工作,按 PR Review 规范审:① 原 issue(#1452)方向核对 → ② 回归审查(既有行为清点)→ ③ 代码审查 → ④ 测试有效性 → ⑤ diff 整洁度,外加本地 typecheck/test/build 与 CI 状态核对。初步状态:open、mergeable_state=blocked(待查 CI)、5 files +595/−5、4 commits(base master)。审查完成后在本线程给结论。

ework-agent added 2 commits September 28, 2026 02:27
… (review fix: stray leading spaces, revert whitespace churn on 2 pre-existing rows)
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #1528 review complete — verdict: mergeable (CI 13/13 green on final head 76552a22)

① 原 issue 方向 — pass. #1452 的三轮取证已确立 bili 未主动发 RST;PR 实现的正是 owner floor-14 批准的 5 项,无 scope creep。floor-17 曾声称修复的 "loop swallows errors" 缺陷已在 floor-20 撤回——我核实最终 diff 完全不触碰 src/loop/* 与 src/plugin.ts,#721 in-band truncation 链本就存在,stall guard 正确地复用它而非重造。开发期自报的 mitm.ts 事故(误删 blind-tunnel 的 HTTP/1.1 200 Connection Established)确认已还原:mitm.ts diff 纯增量。

② 回归审查 — 无意回归零。 触碰路径的既有行为逐项清点:

路径 既有行为 结论
clientError teardown Node 默认 destroy socket 有意变更(item 1,已披露):drain → end() → 干净 FIN
keepAliveTimeout Node 隐式 5s 显式 5000ms,值不变,env 可覆盖(已披露)
连接关闭 静默 +1 debug 行/连接(仅日志开启时),纯增量观测
shutdown force-exit warn 原文本 追加 (liveConns=N) 后缀,prefix 保留(grep-safe)
MITM tunnel 计数 established/closed + live 计数,双 leg .once("close") + guard flag ⇒ exactly-once 递减,纯增量
upstream stream abort idle timer → AbortError → #721 链 stall guard 共用同一 controller/cleanup 路径;default-off 字节级无影响

无 wire protocol / config schema / persistence / acp-kernel pin / version 变更——与披露清单一致。

③ 代码审查。 5 项逐行对照改动前代码读过:ledger 分类逻辑(peer-fin/idle-timeout/server-end/error/destroyed)正确,idle-timeout 以 response budget 为键是对的(Node reaper destroy 空闲 socket 不触发 prefinish/end);µs 时间戳、close 时删记录(无泄漏)、bail timer unref(不拖进程退出);stall guard 仅在 onChunk(首字节后)arm、每 chunk 重装 ⇒ TTFB 语义不变,clearStallTimer 并入 stopIdleTimer ⇒ 继承 #411 的无泄漏保证,abort 走具名 AbortError 进既有链;exposure telemetry NaN→hourly、≤0 禁用、unref'd interval。均无问题。

④ 测试有效性 — 有牙。 8 个测试全是行为级断言。我用独立 probe matrix(node v22.23.2 linux,64KiB NUL burst)实证了三态:A PR 行为 ≈310ms 干净 FIN(paused/flowing client 皆然);B destroy 在 Linux 上也是干净 FIN(RST 分支是平台/时序相关,即 Windows 现场环境);C 无 listener(Node 默认)= peer 永久挂起(无 FIN 无 RST)。所以 test 2 同时咬得住 hang 与 RST 两种回归形态。kat 测试用 raw net.Socket 规避 http client 池化对 Keep-Alive: timeout=0 的非确定行为——正确的取舍。stall E2E 断言 truncated frame + [DONE] + warn 归因 + elapsed<10s;healthy-stream survival 证明是 per-chunk 重装而非总时限;default-off 测试证明零影响;_liveUpstreamTimersForTest()===0 钉死无 timer 泄漏。

⑤ diff 整洁度 — 修一处后 pass。 5 文件每行都在目的内。CONFIGURATION.md 新增表格行带了前导空格(还顺带 churn 了 2 行既有行的空白)——已修回与 master 字节对齐。

Review 中直接修的 2 处(已推分支,各自独立 commit):

  • 6b177010 docs: CONFIGURATION.md env-table 行对齐(去前导空格,还原 2 行既有行的 whitespace churn)
  • 76552a22 test: test 2 的 close await 原本无界——若 clientError disposition 回归到 Node 默认,CI 会挂数小时而不是快速失败(probe C 证实)。加 5s bound,回归时 ≤5s fail-fast。

另发现 1 处残余边界(非阻塞,已立 issue): bail end() 之后若 peer 永不 FIN(对抗性慢速输入),socket 读侧半开状态无限存续(HTTP reaper 以完成响应为键不会 arm,Node 默认不开 TCP SO_KEEPALIVE)。影响有界(每个 held connection 一个,新 [exposure] liveConns 可见),且补 terminal backstop 会重新引入 item 1 刻意消除的 RST 风险、grace 时长属产品权衡——按纪律立项给 owner 拍板:#1529。另有 1 条覆盖观察:MITM-TLS-leg 的 clientError 靠架构论证(同一 server 实例)覆盖而无直接测试,可接受。

证据汇总: 本地 typecheck clean;目标套件 8/8 ×3(含我改后复跑);probe matrix 三态实证;CI 旧 head efaddeb8 13/13 success,新 head 76552a22 13/13 success(含 windows 22/24 全矩阵)。

中文摘要:审查了 #1528(clientError 排空后关闭、连接生命周期台账、keep-alive 显式化、暴露面遥测、流式停顿守卫五项加固),方向与批准范围一致、既有行为逐项核对无意外回归,顺手修了文档表格错位和一处会让 CI 挂死的测试缺陷并立了 #1529 跟进残余边界,双 head CI 全绿,可以合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Deep review — verdict: FIX-REQUIRED (one mechanical docs item), no functional blockers. Everything behavioral is mergeable; mutation-tested end to end.

Verified locally (branch head 76552a2, base = current master)

  • 29/29 in the two touched suites; 20/20 adjacent fetch-util suites (fix-551 / fix-15-abort-lifetime / upstream-fail / wire-exit-gap-cells); typecheck clean.
  • All 3 mutations CAUGHT: ① clientError → destroy() kills the disposition test (ledger pins reason=destroyed); ② stall guard disarmed → fails in 15s (bounded, no CI hang); ③ per-chunk re-arm removed → healthy-stream test fails in 553ms.
  • Empirical leak probe: after the bail end(), Node peers auto-complete the four-way close (allowHalfOpen=false default) → socket + ledger entry fully reclaimed; the ledger Map is bounded by live connections. Residual: a deliberately half-open raw peer can pin the readable half — that tail is exactly feat: terminal backstop for the clientError drain path (#1529) #1530's terminal backstop, correct to leave stacked.

Stall guard — safe for thinking models

Armed only after the first body byte, re-armed per chunk (idle budget, not total); pre-first-byte prefill stays under the untouched #551 12-min budget. Default BILI_STREAM_STALL_MS=0 = off → zero default behavior change. A post-first-byte stall surfaces via the pre-existing in-band truncation path (upstream_stream_truncated + FIN); client socket never destroyed (test-pinned, incl. the warn attribution line). No interplay with BILI_PROXY_KEEPALIVE_MAX_MS (dispatcher-level, disjoint).

FIX-REQUIRED (mechanical)

NOTEs (non-blocking)

  1. Wire-visible change: on malformed requests Node's default is to write HTTP/1.1 400 Bad Request\r\nConnection: close then close — measured on Linux/Node 25, the probe peer got a bare FIN from the new disposition (no 400). Defensible for [0.1.158] MITM 链路上客户端侧疑似闲置连接拆除 RST+上游短周期停滞无看门狗——附一次客户端进程崩溃的完整取证,请评估是否需要加固 #1452, but the PR body should state it, and the src/server.ts:531-538 comment ("no listener → destroys") should be corrected to the measured default; the RST-on-close narrative is Windows-matrix behavior.
  2. BILI_KEEP_ALIVE_TIMEOUT_MS is process-global across all fetchWithTimeout users (registry self-update, plugin fetches too) — fine while default 5000 ≈ Node's implicit value; a doc sentence or scoping follow-up if anyone wants short budgets.
  3. MITM legs: TLS sockets flow through the same clientError handler (mitm.ts:302 re-emit) → end() sends close_notify, strictly more graceful; blindTunnelLive counter is telemetry-only and post-CONNECT-200 gated. ✓
  4. Nit: no clamp of kat against headersTimeout (>60s silently capped). Follow-up only.

Stack order with #1530 stands (backstop closes the adversarial tail). Merge this first, then #1530.

ework-agent added 2 commits September 28, 2026 09:34
…md (review fix: en/zh sync)

Owner review of #1528 flagged the zh table at zero diff while CONFIGURATION.md gained three rows (§7.3 en/zh sync). Rows taken verbatim from stacked #1530's 847b134 so that commit becomes empty on its post-merge rebase.
…de default (review fix)

Owner deep review measured Linux/Node 25: no-listener clientError writes 'HTTP/1.1 400 Bad Request' + close before destroying. The comment's destroy-only narrative was our Node 22/Linux strand-case measurement. Both now recorded, plus the intentional wire delta (malformed request gets FIN, not 400).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 FIX-REQUIRED 已处理 + NOTE 1 已落实,新 head 3331a623,CI 13/13 全绿。逐项回报:

① zh/en 同步(必改项)— 已在本 PR 落地(选项 A)。 commit b77a742a:三行(BILI_STREAM_STALL_MS / BILI_KEEP_ALIVE_TIMEOUT_MS / BILI_EXPOSURE_LOG_INTERVAL_MS)已补进 CONFIGURATION.zh-CN.md,插入位置与 en 表对应(ACP_SESSION_HEADER 之前)。译文逐字取自堆叠 #1530 的 847b1342(fetch 该分支后直接 checkout 文件版本,非重新翻译)——这样选 A 的代价最小:#1528 单独合入即自洽(master 上不会出现「en 有、zh 无」的窗口期),而 #1530 合后 rebase 到 master 时其 847b1342 变成空补丁、可干净跳过;它的 bdad624a(ACP_RENDER_NONE 回填)与本 PR 无关、原样保留在 #1530。

② NOTE 1(400→FIN wire delta)— 两处都已做。

③ NOTE 2(kat 作用域)— 查证后无需改文档。 BILI_KEEP_ALIVE_TIMEOUT_MS 是 server-local 的:仅 src/server.ts:465-468 读取(server.keepAliveTimeout + 台账 idle-timeout 分类),registry self-update / plugin fetch 等 fetchWithTimeout 用户不读这个变量(src/upstream-proxy.ts 里的 keepAliveTimeout 是 ProxyAgent 自己的,由别的旋钮驱动)。文档行现有措辞「客户端侧 socket」已是准确作用域,故只在此更正记录,不动文档。

④ NOTE 4(kat vs headersTimeout clamp)— 按你的定性保持 follow-up,本 PR 不动。

验证: 本地 typecheck clean、目标套件 8/8;CI 新 head 3331a623 13/13 success(含 windows 22/24)。本轮新增两个 commit 均为 docs/comment,零行为变更:b77a742a(zh 三行)、3331a623(注释修正)。

堆叠顺序不变:先合 #1528,#1530 再 rebase(rebase 时跳过空的 847b1342 即可,其余照落)。合并是人工操作;#1528 落地后 #1530 那条线的 agent 已表示可代做 rebase + 全套重跑。

中文摘要:按深审意见补上了 zh-CN 三个环境变量行(逐字取自 #1530 以便其 rebase 时干净跳过)、把 clientError 注释修正为实测的 Node 默认行为并在 PR body 披露「malformed 请求得 FIN 而非 400」的有意 wire delta、核实 kat 为 server-local 无需改文档,新 head CI 13/13 全绿,可以合并。

@ranxianglei
ranxianglei merged commit cb622b6 into master Sep 28, 2026
13 checks passed
@ranxianglei ranxianglei mentioned this pull request Sep 28, 2026
@ranxianglei
ranxianglei deleted the 2026-09-27_mitm-rst-idle-hardening branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant