Skip to content

fix(server): graceful post-response client close — linger for peer close signal (#1982) - #1985

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-03_mitm-graceful-close
Oct 3, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-03_mitm-graceful-close

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1982

What & why

Downstream forensics (#1982) showed that when bili initiates the client-side close right after a completed response (Connection: close), the downstream socket receives RST ~60–190 ms later instead of clean close_notify+FIN. Root cause verified against Node internals: for a last response _http_server.js resOnFinish → socket.destroySoon() runs end() and destroy() in the same tick — the final bytes (incl. TLS close_notify) written by end() can still be unACKed when close(fd) executes, so the kernel sends RST instead of FIN. On MITM legs it is worse by construction: TLSWrap.close() explicitly destroy()s the raw wrapped socket, which is why every ledger line showed tls reason=server-end paired with tcp reason=destroyed.

Fix

Post-response linger on client-facing sockets (plain-TCP and MITM TLS legs alike), installed next to the connection ledger in src/server.ts:

  • intercept end() to stamp rec.ended (prefinish fires async — after the same-tick destroy — so it cannot gate);
  • intercept destroy(): only when the close follows a completed response and no error/drain state applies (error-driven closes, pre-handshake teardown, the clientError drain clientError drain path lacks terminal backstop: silent peer holds half-open socket indefinitely (#1528 item-1 follow-up) #1529, and bare kat-reaper destroys all pass through unchanged) — resume() the recv side and hold the fd until the peer's close signal (TCP FIN or TLS close_notify, which can only arrive after our last byte was received+ACKed — the only delivery proof Node exposes; there is no SO_LINGER/ACK-state API), then close cleanly;
  • bounded budget caps silent-peer exposure: backstop destroy classified reason=linger-backstop (warn line names the RST risk);
  • ledger observability (request 2 of the issue): structurally-destroyed raw legs of normally-closed MITM connections classify as paired-clean (new additive reason); genuinely abortive closes are elevated from debug to warn with an [ABORTIVE — peer may see RST/ECONNRESET] marker, so downstream "I got RST at T" reports reconcile against bili.log directly.

Config surface

New env var BILI_POST_RESPONSE_LINGER_MS (default 5000; non-numeric/non-positive → default). Maps onto the existing BILI_* timeout-seam species (BILI_KEEP_ALIVE_TIMEOUT_MS, BILI_CLIENT_ERROR_BACKSTOP_MS, BILI_MITM_HANDSHAKE_TIMEOUT_MS) rather than inventing a new shape; env-only like its siblings (no file-config key); no migration needed. Documented in CONFIGURATION.md (en/zh) + new README section (also covers request 3: the two pre-existing knobs were source-only, now documented).

Behavior changes (disclosed)

Path Old New
Proxy-initiated close right after a completed response immediate abortive destroy — final bytes could be unACKed → client-side RST linger up to BILI_POST_RESPONSE_LINGER_MS (default 5 s, nginx lingering_time parity) for the peer's FIN/close_notify, then clean close; silent peer → bounded backstop destroy (warn)
Idle reaper (kat), error-driven closes, pre-handshake teardown, clientError drain (#1529) destroy unchanged
Abortive close ledger lines debug warn + [ABORTIVE …] marker
Close reasons — additive: linger-backstop, paired-clean
Incidental, separate commit: MITM upstream key dropped the CONNECT port → every non-443 tunnel misrouted to :443 misroute-to-443 port preserved; 443 path byte-identical

Error paths deliberately stay abortive: an errored/vanished peer has nothing left to protect, and draining cert-reject floods or slowloris timeouts would multiply exposure; the handshake timeout remains the resource-protection fallback the issue itself allows. Blind tunnels (opaque pipe, no response-completion semantics) are out of scope — separate follow-up issue filed.

Regression review (existing behaviors walked)

  • request pipeline / wire shape: untouched (close-path only); both compression modes unaffected (no message content change).
  • keep-alive reuse cadence: kat pin + Node buffer unchanged; idle-timeout classification unchanged (test step 3 pins it).
  • clientError drain ownership (clientError drain path lacks terminal backstop: silent peer holds half-open socket indefinitely (#1528 item-1 follow-up) #1529): the drainArmed gate keeps that path exclusive; its backstop still classifies clienterror-backstop.
  • persistence / prompt cache / nudge / preflight: untouched.
  • new test pins: happy MITM close (clean EOF, no ECONNRESET, paired-clean), silent-peer backstop, idle reaper unchanged, pre-handshake abortive warning, plain-TCP direct leg. Real bili server in a child process (CA isolation via XDG dirs + NODE_EXTRA_CA_CERTS).

Tests

tests/graceful-close.test.ts (+ fixture tests/fixtures/graceful-close-bili.mts): full bili server, real CONNECT/TLS handshake against a mock upstream signed by bili's own test CA; 5 sequential steps, ~4 s total. Local evidence: green ×4 consecutive runs; tests/mitm.test.ts 21/21; typecheck + build clean. Full suite left to CI.

ework-agent added 2 commits October 3, 2026 18:33
…ses (#1982)

When the proxy initiates the client-side close right after a completed response (Connection: close), Node's destroySoon() destroys the socket in the same tick as end(), so the final bytes (incl. TLS close_notify) can still be unACKed when close(fd) runs and the kernel sends RST instead of FIN — observed downstream in #1982.

Intercept destroy() on client-facing sockets (plain-TCP and MITM TLS legs alike): when the close follows a completed response and no error/drain state applies, resume() the recv side and hold the fd until the peer's close signal — TCP FIN or TLS close_notify, which can only arrive after our last byte was received+ACKed — then close cleanly. A bounded budget (BILI_POST_RESPONSE_LINGER_MS, default 5000, nginx lingering_time parity) caps silent-peer exposure; backstop closes are classified reason=linger-backstop (warn line). Structurally-destroyed raw legs of normally-closed MITM connections are classified paired-clean; genuinely abortive closes are elevated from debug to warn with an ABORTIVE marker so downstream RST reports reconcile against bili.log directly.

Error-driven closes, pre-handshake teardown, the clientError drain (#1529) and the idle reaper are deliberately unchanged.
doMitm stamped the upstream key as https://<host>, dropping the CONNECT port, so every non-443 MITM tunnel was misrouted to :443 (connect-refused against local gateways). Discovered while building the #1982 graceful-close test suite, which exercises a non-default-port mock upstream through the MITM leg. The 443 path stays byte-identical.
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-03_mitm-graceful-close (86b8fe7)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2002-pr.1985.f6db9f0

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1985.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #1985 审查结论:无拦下项,可合入——但需先 rebase 到当前 master(ffd949b)并取一次新 CI。 方向、机制、测试、diff 均已逐项核验;结构化七问如下。

审前取证(不是只读 diff)

  • 全量 diff(8 文件,+578/−13)+ src/server.ts 连接台账区全文 + src/mitm.ts doMitm 配对时序 + resolveUpstream 消费点逐一读过。
  • MITM 配对时序安全:MITM_RAW_SOCKET_KEY 在 server.emit("connection", tlsSocket) 之前同步盖章(mitm.ts:285 → :311),raw 腿 ledger 记录必然先于 TLS 腿存在,不会漏配。
  • 核心机制声明在本机 Node v22.23.2 上实证成立,而非纸面推演:新测试 step1/step5 断言的 linger-arm 行出现,证明 destroySoon() 路径确实先走被拦截的 end() 再走 destroy();step3 钉住 kat reaper 仍是裸 destroy(v22 实测不经过 end()),空闲回收行为未漂移。
  • 本地定向验证(scratch cherry-pick 到当前 master,零冲突):tests/graceful-close.test.ts + tests/mitm.test.ts —— 首轮在负载较高的共享机上 22 中挂 1(瞬态),随后连续 4 轮全绿(graceful-close 1/1 ×4,mitm 21/21)。CI 侧:head 86b8fe7 全部 10 条 lane(CI 主套件 + real/fake codex + native dsh/pi/opencode + local-registry + Windows overlay + image + artifact)全绿——注意是对旧 base 跑的。
  • follow-up 合规:blind tunnel 边界已按交付规则立项 建议:blind tunnel(tunnelThrough)连接拆除同为 abortive 关闭,属 #1982 同一 RST 家族 #1986(带来源标记,另一会话已给出 A/B 建议待 owner 拍板)。

① 是什么问题:响应完成后代理主动关闭客户端连接(Connection: close)时,Node 的 destroySoon() 同 tick 执行 end()+destroy(),最后一个字节/close_notify 可能尚未被对端 ACK,内核带着未确认发送字节关 fd → 回 RST 而非 FIN,池化下游表现为 ECONNRESET(#1982 两周 196 次、两次击穿下游进程)。附带两个观测缺陷:裸 TCP 腿被 TLSWrap.close() 结构性 destroy,台账把完全优雅的 MITM 正常关闭误报为 abortive destroyed;且该行仅 debug 级,下游报 RST 时无法直接对账。

② 什么原因导致的:根因是「fd 关闭时刻与字节 ACK 完成之间的竞态」——产生者是 destroySoon() 的同 tick 时序 + TLSWrap.close() 对裸腿的结构性 destroy(公共 API 无法改为 end())。issue 原文建议的「给裸 clientSocket 补 end()」打不到点上:TLSWrap 拥有裸腿命运,唯一 chokepoint 是 destroy() 本身。PR 的机制选择因此比 issue 字面建议更正确。

③ 怎么修复的:拦截 destroy() 本身——当且仅当关闭发生在已完成响应之后且 socket 健康(无 error/drain/pre-handshake/bare-destroy 状态)时:resume() 排空 recv 侧,持有 fd 等待对端关闭信号(TCP FIN / TLS close_notify——协议上只能在我方最后字节被接收+ACK 之后到达,即交付证明),收到后干净销毁;静默对端最多占用一个 fd 至预算(默认 5s,nginx lingering_time 对齐)→ warn reason=linger-backstop。裸 MITM 腿按配对 TLS 腿分类为 paired-clean;真 abortive 关闭 debug→warn + [ABORTIVE] 标记(issue 请求二);env 旋钮 en/zh 文档化(请求三)。附带独立 commit:MITM upstream key 保留非 443 CONNECT 端口(此前静默误路由到 :443;443 路径逐字节不变,已核对 resolveUpstream 全部消费点——与 /bili/ 模式 u.host 含端口的既有约定一致)。

④ 是否有回归问题:触碰路径的既有行为逐项清点过——happy-path post-response close(有意变更,现 linger;客户端见干净 EOF,静默对端 ≤5s fd 占用,有界);error-driven destroy(不变,gate 放行);pre-handshake teardown(不变仍 abortive,现 warn 升格,已披露);clientError drain #1529(不变,drainArmed gate);idle kat reaper(不变立即销毁,v22 实测不经过 end(),test step3 钉死);blind tunnel(不触碰,ended 恒 false 永不 arm,#1986 跟进);ledger 词表(新增 paired-clean/linger-backstop,原本误报的 reason=destroyed 行数会减少——grep 该词的工具需注意,已在 PR 表格里披露)。无 fail-fast 变吞错、无阈值/时序漂移超出披露范围。本地首轮 1 个瞬态失败(共享机负载下 waitFor 时序),后续 4 连绿,判定为环境负载而非代码缺陷。

⑤ 是否破坏缓存:不涉及——纯连接生命周期层,请求内容/压缩/prefix-cache/持久化零接触。

⑥ 是否改了设计方向:不改。是 #1452/#1529 台账+drain-then-end 纪律向剩余正常关闭段的自然延伸,机制与 nginx lingering_time 同构。blind-tunnel 边界显式划出(等 owner 在 #1986 拍 A/B)。

⑦ 是否引入了新的配置:是——BILI_POST_RESPONSE_LINGER_MS,默认 5000ms,env-only,CONFIGURATION.md en/zh + README 新小节,非数字/非正→默认。两点提请 owner 注意:(a) 与兄弟旋钮不同,0 不能禁用(BILI_CLIENT_ERROR_BACKSTOP_MS 的 0 恢复旧行为、BILI_EXPOSURE_LOG_INTERVAL_MS 的 0 关闭)——若将来 linger 出问题没有 kill switch,只能缩到 1ms;建议考虑 0 = 立即销毁(旧行为)以对齐。(b) 小 nit:arm 行用 info 级(每次 post-response close 一条),稳态下我倾向 debug——结果行已有 debug/warn 分级;若是为 rollout 期可见性刻意选的,保持即可。另:self-update drain 窗口内旧进程退出最多被 linger fd 拖长 5s(有界,监听端口独立释放,nginx 同款性质),非缺陷,记录在案。

合并前置动作

  1. rebase 到当前 master(ffd949b,落后 ~40 commit,master 同期改过 src/server.ts 的 prepare* 区与 import 行——与本 PR 区域不相交,我已本地验证 cherry-pick 零冲突 + rebased head 定向测试全绿);rebase 后取一次新 CI(现有 CI 绿是对旧 base 的)。
  2. 与 refactor(#1714): retire all BILI_*/ACP_* behavior env vars to config.json #1747(env-surface retirement P1)排序协调:若 refactor(#1714): retire all BILI_*/ACP_* behavior env vars to config.json #1747 先合,新的 BILI_POST_RESPONSE_LINGER_MS 需按其 config.json 化方向对账(owner 决策)。
  3. 非自动合并资格(新增 config surface + hot file + wire 邻接行为)——人工合并。

判定:代码质量高、机制打在根因上、三条 issue 诉求全覆盖、测试有牙且诚实披露 loopback 局限,rebase + 新 CI 后可合入。

@ranxianglei
ranxianglei merged commit 70cc1ba into master Oct 3, 2026
15 checks passed
@ranxianglei ranxianglei mentioned this pull request Oct 4, 2026
@ranxianglei
ranxianglei deleted the 2026-10-03_mitm-graceful-close branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

建议:客户端侧连接请优雅关闭(close_notify+FIN),避免向下游客户端发出 RST

1 participant