Repository navigation
fix(server): graceful post-response client close — linger for peer close signal (#1982) - #1985
Conversation
…ses (#1982) When the proxy initiates the client-side close right after a completed response (Connection: close), Node's destroySoon() destroys the socket in the same tick as end(), so the final bytes (incl. TLS close_notify) can still be unACKed when close(fd) runs and the kernel sends RST instead of FIN — observed downstream in #1982. Intercept destroy() on client-facing sockets (plain-TCP and MITM TLS legs alike): when the close follows a completed response and no error/drain state applies, resume() the recv side and hold the fd until the peer's close signal — TCP FIN or TLS close_notify, which can only arrive after our last byte was received+ACKed — then close cleanly. A bounded budget (BILI_POST_RESPONSE_LINGER_MS, default 5000, nginx lingering_time parity) caps silent-peer exposure; backstop closes are classified reason=linger-backstop (warn line). Structurally-destroyed raw legs of normally-closed MITM connections are classified paired-clean; genuinely abortive closes are elevated from debug to warn with an ABORTIVE marker so downstream RST reports reconcile against bili.log directly. Error-driven closes, pre-handshake teardown, the clientError drain (#1529) and the idle reaper are deliberately unchanged.
doMitm stamped the upstream key as https://<host>, dropping the CONNECT port, so every non-443 MITM tunnel was misrouted to :443 (connect-refused against local gateways). Discovered while building the #1982 graceful-close test suite, which exercises a non-default-port mock upstream through the MITM leg. The 443 path stays byte-identical.
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.2002-pr.1985.f6db9f0This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr1985.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 PR #1985 审查结论:无拦下项,可合入——但需先 rebase 到当前 master(ffd949b)并取一次新 CI。 方向、机制、测试、diff 均已逐项核验;结构化七问如下。 审前取证(不是只读 diff)
① 是什么问题:响应完成后代理主动关闭客户端连接( ② 什么原因导致的:根因是「fd 关闭时刻与字节 ACK 完成之间的竞态」——产生者是 ③ 怎么修复的:拦截 destroy() 本身——当且仅当关闭发生在已完成响应之后且 socket 健康(无 error/drain/pre-handshake/bare-destroy 状态)时:resume() 排空 recv 侧,持有 fd 等待对端关闭信号(TCP FIN / TLS close_notify——协议上只能在我方最后字节被接收+ACK 之后到达,即交付证明),收到后干净销毁;静默对端最多占用一个 fd 至预算(默认 5s,nginx lingering_time 对齐)→ warn ④ 是否有回归问题:触碰路径的既有行为逐项清点过——happy-path post-response close(有意变更,现 linger;客户端见干净 EOF,静默对端 ≤5s fd 占用,有界);error-driven destroy(不变,gate 放行);pre-handshake teardown(不变仍 abortive,现 warn 升格,已披露);clientError drain #1529(不变,drainArmed gate);idle kat reaper(不变立即销毁,v22 实测不经过 end(),test step3 钉死);blind tunnel(不触碰,ended 恒 false 永不 arm,#1986 跟进);ledger 词表(新增 paired-clean/linger-backstop,原本误报的 ⑤ 是否破坏缓存:不涉及——纯连接生命周期层,请求内容/压缩/prefix-cache/持久化零接触。 ⑥ 是否改了设计方向:不改。是 #1452/#1529 台账+drain-then-end 纪律向剩余正常关闭段的自然延伸,机制与 nginx lingering_time 同构。blind-tunnel 边界显式划出(等 owner 在 #1986 拍 A/B)。 ⑦ 是否引入了新的配置:是—— 合并前置动作
判定:代码质量高、机制打在根因上、三条 issue 诉求全覆盖、测试有牙且诚实披露 loopback 局限,rebase + 新 CI 后可合入。 |
Closes #1982
What & why
Downstream forensics (#1982) showed that when bili initiates the client-side close right after a completed response (
Connection: close), the downstream socket receives RST ~60–190 ms later instead of cleanclose_notify+FIN. Root cause verified against Node internals: for a last response_http_server.js resOnFinish → socket.destroySoon()runsend()anddestroy()in the same tick — the final bytes (incl. TLS close_notify) written byend()can still be unACKed whenclose(fd)executes, so the kernel sends RST instead of FIN. On MITM legs it is worse by construction:TLSWrap.close()explicitlydestroy()s the raw wrapped socket, which is why every ledger line showedtls reason=server-endpaired withtcp reason=destroyed.Fix
Post-response linger on client-facing sockets (plain-TCP and MITM TLS legs alike), installed next to the connection ledger in
src/server.ts:end()to stamprec.ended(prefinish fires async — after the same-tick destroy — so it cannot gate);destroy(): only when the close follows a completed response and no error/drain state applies (error-driven closes, pre-handshake teardown, the clientError drain clientError drain path lacks terminal backstop: silent peer holds half-open socket indefinitely (#1528 item-1 follow-up) #1529, and bare kat-reaper destroys all pass through unchanged) —resume()the recv side and hold the fd until the peer's close signal (TCP FIN or TLS close_notify, which can only arrive after our last byte was received+ACKed — the only delivery proof Node exposes; there is no SO_LINGER/ACK-state API), then close cleanly;reason=linger-backstop(warn line names the RST risk);paired-clean(new additive reason); genuinely abortive closes are elevated from debug to warn with an[ABORTIVE — peer may see RST/ECONNRESET]marker, so downstream "I got RST at T" reports reconcile against bili.log directly.Config surface
New env var
BILI_POST_RESPONSE_LINGER_MS(default5000; non-numeric/non-positive → default). Maps onto the existingBILI_*timeout-seam species (BILI_KEEP_ALIVE_TIMEOUT_MS,BILI_CLIENT_ERROR_BACKSTOP_MS,BILI_MITM_HANDSHAKE_TIMEOUT_MS) rather than inventing a new shape; env-only like its siblings (no file-config key); no migration needed. Documented in CONFIGURATION.md (en/zh) + new README section (also covers request 3: the two pre-existing knobs were source-only, now documented).Behavior changes (disclosed)
BILI_POST_RESPONSE_LINGER_MS(default 5 s, nginxlingering_timeparity) for the peer's FIN/close_notify, then clean close; silent peer → bounded backstop destroy (warn)[ABORTIVE …]markerlinger-backstop,paired-cleanError paths deliberately stay abortive: an errored/vanished peer has nothing left to protect, and draining cert-reject floods or slowloris timeouts would multiply exposure; the handshake timeout remains the resource-protection fallback the issue itself allows. Blind tunnels (opaque pipe, no response-completion semantics) are out of scope — separate follow-up issue filed.
Regression review (existing behaviors walked)
idle-timeoutclassification unchanged (test step 3 pins it).drainArmedgate keeps that path exclusive; its backstop still classifiesclienterror-backstop.paired-clean), silent-peer backstop, idle reaper unchanged, pre-handshake abortive warning, plain-TCP direct leg. Real bili server in a child process (CA isolation via XDG dirs + NODE_EXTRA_CA_CERTS).Tests
tests/graceful-close.test.ts(+ fixturetests/fixtures/graceful-close-bili.mts): full bili server, real CONNECT/TLS handshake against a mock upstream signed by bili's own test CA; 5 sequential steps, ~4 s total. Local evidence: green ×4 consecutive runs;tests/mitm.test.ts21/21; typecheck + build clean. Full suite left to CI.