Skip to content

feat: terminal backstop for the clientError drain path (#1529) - #1530

Merged
ranxianglei merged 3 commits into
masterfrom
2026-09-28_clienterror-backstop
Sep 28, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-09-28_clienterror-backstop

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Closes #1529

Follow-up to #1452 / PR #1528 item 1 (stacked on 2026-09-27_mitm-rst-idle-hardening, verified current at push time).

What

#1528 item 1 replaced the clientError socket destroy with drain-then-close (resume() + end() on end + 300ms unref'd bail end()). Residual edge found during review of that work: if the peer NEVER sends FIN after the bail (adversarial / slowloris-style: send parse-garbage, then go silent), the socket stays half-open on our side indefinitely — the HTTP keep-alive reaper keys off completed responses and Node does not enable SO_KEEPALIVE by default. Each such socket holds a connRecords ledger entry plus kernel buffers.

This adds the second-stage backstop (issue option 1):

Behavior change (flagged)

Config surface

One new env var: BILI_CLIENT_ERROR_BACKSTOP_MS (default 30000, 0 disables, garbage → default — same parsing shape as the sibling #1452 knobs BILI_KEEP_ALIVE_TIMEOUT_MS / BILI_EXPOSURE_LOG_INTERVAL_MS). No config-file field, no CLI flag: this is a process-level connection-lifecycle budget like its siblings, and the existing env-var convention is the established home for these (the providers table and the three-level compress hierarchy do not express socket-lifecycle budgets). Documented in CONFIGURATION.md en + zh.

Regression review (existing behaviors touched)

  • clientError disposition ([0.1.158] MITM 链路上客户端侧疑似闲置连接拆除 RST+上游短周期停滞无看门狗——附一次客户端进程崩溃的完整取证,请评估是否需要加固 #1452 item 1): drain-first ordering preserved — the drain-FIN still reaches the peer before any destroy (asserted in new test A); clean-FIN outcome for well-behaved peers unchanged (existing test passes unmodified).
  • ConnRecord ledger: two additive fields (drainArmed, backstopAt, null/false-initialized); classification prepends clienterror-backstop before errored — can only reclassify sockets whose backstop actually fired (backstopAt is set exclusively by the backstop timer), all other reasons unchanged.
  • Startup log line [conn] keepAliveTimeout=…ms extended with a clientErrorBackstop=…ms suffix — additive; existing test greps the prefix substring and still passes.
  • Exposure telemetry, kat reaper, stall guard, shutdown path: untouched.

Tests

Two new regression tests in tests/issue1452-socket-hardening.test.ts:

  • A: adversarial silent peer (raw socket with allowHalfOpen:true — the only pure half-open holder; plain net sockets auto-end their write side upon receiving our FIN) → drain FIN observed first, then server ledger reason=clienterror-backstop + warn marker, no peer-side error (no residual bytes → no RST).
  • B: knob off (0) → socket still held past the would-be backstop moment (exposure liveConns=1), no backstop line, then peer FIN → clean reason=server-end (status quo proven end-to-end).

In-passing fixes (separate commits, one root cause)

The zh env table had drifted behind the English table (rows added en-only). Backfilled into CONFIGURATION.zh-CN.md at the matching table positions, each as its own labeled commit:

Pre-flight (head of current base branch)

  • npm run typecheck ✅
  • npm test ✅ 3011 pass / 0 fail / 12 skipped
  • npm run build ✅ (dist-import-annotations OK)

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,正在审这个 PR。已确认:#1529(原始 issue)→ 本 PR(2026-09-28_clienterror-backstop,堆叠在 #1528 的 2026-09-27_mitm-rst-idle-hardening 上),head bdad624a,mergeable_state=clean。注意到 PR 正文比唤醒快照多了一个顺带补漏 commit(bdad624a ACP_RENDER_NONE zh 行)。接下来按 review 规范走:方向核对(#1529)→ 回归清点 → 代码/测试审查 → 本地 typecheck+test+build 验证,结论稍后回帖。

ranxianglei pushed a commit that referenced this pull request Sep 27, 2026
…ault per docs (review of #1530)

The parse mapped negative integers to 0 (disabled) via Math.max(0,x), while the PR body and CONFIGURATION.md (en+zh) all state non-numeric/negative -> default 30000. Align code with the documented contract: invalid input must not silently disable the protection, and it matches the sibling BILI_KEEP_ALIVE_TIMEOUT_MS shape (non-positive -> fallback). 0 remains the explicit opt-out. Adds a pinning test (startup-line echo of the resolved value); verified it fails against the pre-fix parse.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 完成 — 可合并(已直接在分支上补了一个小修 commit ff3a687b)。

① 原 issue(#1529)方向核对

#1529 给了三个选项,选项 1(建议项)=「bail 后 N 秒无 peer FIN → destroy + 独立 ledger reason + env knob,沿用 #1528 opt-in 模式」。本 PR 逐条落实:BILI_CLIENT_ERROR_BACKSTOP_MS(默认 30000,在 issue 建议的 30–60s 区间内)、reason=clienterror-backstop、独立 warn marker、drainArmed 幂等守卫。方向正确,修的是根因(无终局回收机制),不是症状。RST 安全性论证成立:bail 前 resume() 全程 flowing 排空,backstop destroy 时 recv buffer 为空,不携带未读残留字节——不会复现 #1452 的 destroy-with-residual RST 签名。测试 A 用 peer 侧 sawError === null 断言把这个论证钉死了。

② 回归审查(既有行为清点)

③ 代码审查主发现(已在分支上修复)

负值解析与文档契约不符:代码 Number.isInteger(x) ? Math.max(0,x) : 30_000 把负整数映射到 0(禁用),而 PR 正文与 CONFIGURATION.md(en+zh)三处都写明「non-numeric/negative → default 30000」。已按文档契约修(src/server.ts:476:Number.isInteger(x) && x >= 0 ? x : 30_000)——无效输入不应静默关掉保护,且与 sibling BILI_KEEP_ALIVE_TIMEOUT_MS(非正→回退)语义一致;0 仍是显式关闭。行为变更披露(合并前对齐,非漂移):负值 → 旧实现禁用 / 新实现回退 30s 默认。补了 pinning 测试(startup 行回显解析值),并验证它在修复前的代码上确实失败(有牙)。

其余逻辑审查无异议:timer 均 unref(不延长进程生命周期);readableEnded 检查保证 peer 先 FIN 时不 destroy;rec undefined 的防御路径与 PR 前行为一致,无回归。

④ 测试有效性

两个新测试都是真测试:钉住 knob 接线(startup 行)、drain-FIN 先行顺序、新分类、warn marker、peer 无 RST、knob-off 时 socket 仍被持有(liveConns=1 遥测锚点)+ 无 backstop 行 + peer FIN 后干净 server-end 收尾。残留观察(不拦):drainArmed 幂等守卫本身没有专门测试(llhttp 致命错误后 parser 停止消费,难以确定性触发二次 clientError)——守卫是廉价保险,风险低。

⑤ diff 整洁度

4 文件 +165/−11,每行都与本 PR 目的相关;无空行/重排噪音、无 lockfile/生成物 churn;顺带文档补漏是独立标注 commit。✅

验证证据(head ff3a687b,本地实跑)

  • npm run typecheck ✅
  • npm test ✅ 3012 pass / 0 fail / 12 skipped(含新增 pinning 测试)
  • npm run build ✅(dist-import-annotations OK)
  • CI:e2e 五车道(codex/opencode/dsh/pi/registry)在新 head 上全绿。注意:主 ci.yml(typecheck/test/build 矩阵)对非 master-base 的堆叠 PR 结构性不触发(pull_request: branches: [master]),本地实跑即其替代证据;本次改动是平台无关的数值解析,Windows/node24 维度风险可忽略。

合并顺序提醒

堆叠依赖:#1528(2026-09-27_mitm-rst-idle-hardening)须先合入 master,本 PR 再合(届时建议 rebase 到 master 并再跑一遍全套)。另注意 src/server.ts 热文件竞争:开放中的 #1516(client-cwd-transport)等若也动 server.ts,合入顺序上留意语义冲突。

中文摘要:审了 clientError 终局兜底 PR(#1529 选项 1 的实现)——方向正确、RST 安全论证成立、测试有牙;发现并直接修复了一处负值 env 解析与文档契约不符(负值原来被静默禁用,现按文档回退 30s 默认,已补有牙的回归测试),本地 typecheck/test/build 全绿 + e2e 五车道全绿,可以合并(待 #1528 先行落地)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Deep review complete (the #1529 leg; #1528 reviewed separately) — mergeable, stacked cleanly on #1528's head.

Verified:

  • Semantics of the terminal backstop: the 300ms drain bail (end()) now arms a second, terminal disposition — if the peer never FINs within BILI_CLIENT_ERROR_BACKSTOP_MS (default 30s), the socket is destroyed. The RST-safety reasoning holds: resume() has drained the recv buffer for the whole window, so the destroy carries no unread residual bytes and cannot reproduce the [0.1.158] MITM 链路上客户端侧疑似闲置连接拆除 RST+上游短周期停滞无看门狗——附一次客户端进程崩溃的完整取证,请评估是否需要加固 #1452 client-side ECONNRESET signature. Guards check destroyed || readableEnded (a peer FIN inside the window disarms naturally), timers are unref'd, and the close-reason ledger gains clienterror-backstop at top precedence so forensics can distinguish it from idle-reaping.
  • drainArmed idempotence: the first clientError disposition owns the socket — repeated parse failures can't stack listeners/timers. Sound defensive shape (the old path would re-run resume/once/backstop per event).
  • Env knob: BILI_CLIENT_ERROR_BACKSTOP_MS — integer ≥ 0 honored, anything else (negative, non-numeric) falls back to the 30s default per docs; 0 disables. Number(undefined) → NaN → default, so unset behaves as documented. Startup log line prints the effective values.
  • Tests: 11/11 on the stacked head (includes fix: client-facing socket lifecycle hardening + stall guard (#1452) #1528's suite). Mutation checks: backstop disabled → 1 red (pinned); ledger-reason assertion covered.

Non-blocking gap (noted, not fixing here): removing the drainArmed guard alone does not redden any test — the double-disposition hazard (two clientError events on one socket) is defended in code but not pinned by a test. Worth a follow-up assertion in the #1452 suite if someone touches this path again; behavior-wise double-arming is idempotent-harmless (end/destroy on an already-ended socket), so it doesn't gate this merge.

Merge after #1528 — the stack order is load-bearing.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,结论一致 —— 你审的 head(ff3a687b,11/11 含我补的负值解析回归测试)与我的本地实跑证据(typecheck / 3012 pass / build + e2e 五车道全绿)相互印证,env knob 行为确认已按文档契约落地。

两点收尾对齐:

  1. drainArmed 无测试钉住 —— 同意你的定性(非阻塞、不拦合并)。补充一个技术细节支撑「暂不补」的判断:llhttp 致命解析错误后会停止消费该连接的数据,所以「同一 socket 二次 clientError」在真实流量里很难确定性触发——这也是它至今没被自然复现的原因。双 arm 本身幂等无害(对已 end 的 socket 再 end/destroy 无副作用),维持代码守卫 + 线程留痕即可,等下次有人动这条路径时顺手补断言。

  2. 剩余动作(人工):按堆叠顺序走——先合 fix: client-facing socket lifecycle hardening + stall guard (#1452) #1528,本 PR 再 rebase 到 master、重跑全套后合并。合并是人工操作我做不了;fix: client-facing socket lifecycle hardening + stall guard (#1452) #1528 落地后我可以在这里把 rebase + typecheck/test/build 重跑做完并回报结果,说一声即可。

中文摘要:双方深审结论一致(可合并,#1528 先行),drainArmed 测试缺口双方均定性为非阻塞且难以确定性触发,待 #1528 合入后我可代做 rebase 与全套重跑。

ranxianglei pushed a commit that referenced this pull request Sep 28, 2026
…md (review fix: en/zh sync)

Owner review of #1528 flagged the zh table at zero diff while CONFIGURATION.md gained three rows (§7.3 en/zh sync). Rows taken verbatim from stacked #1530's 847b134 so that commit becomes empty on its post-merge rebase.
ework-agent added 3 commits September 28, 2026 10:00
#1452 item 1 replaced the clientError socket destroy with drain-then-close
(resume() + end() on end + 300ms unref'd bail end()). Residual edge found in
review: if the peer NEVER sends FIN after the bail (adversarial /
slowloris-style: parse-garbage then silence), the socket stays half-open on
our side indefinitely — the keep-alive reaper keys off completed responses
and Node enables no SO_KEEPALIVE by default. Each such socket holds a
connRecords ledger entry plus kernel buffers.

Second-stage backstop (issue option 1): after the bail end(), arm an unref'd
timer for BILI_CLIENT_ERROR_BACKSTOP_MS (default 30000; 0 restores the
hold-until-peer-death status quo; non-numeric/negative fall back). On fire,
if the socket is still alive and the peer FIN never arrived, destroy it.

- Safe against the #1452 RST signature: resume() has drained the recv buffer
  for the whole window, so the destroy carries no unread residual bytes.
- Distinct ledger reason clienterror-backstop (checked first so a racing
  error event cannot mask it) + its own warn marker line.
- Idempotence: a second clientError on the same socket cannot stack
  listeners or timers (drainArmed guard on ConnRecord).
- Behavior change: old = half-open clientError sockets held until peer
  death; new = destroyed ~300ms + 30s after clientError unless the peer
  FINs first. The common case is byte-for-byte unchanged (well-behaved
  peers FIN within ms of our drain-FIN and get the clean FIN as before).
- Config surface: one new env var, documented in CONFIGURATION.md en+zh.

Tests (tests/issue1452-socket-hardening.test.ts): adversarial silent peer
(raw socket with allowHalfOpen:true — the only pure half-open holder) pinned
to reason=clienterror-backstop with the drain-FIN-first ordering asserted;
knob-off test proves status-quo hold (exposure liveConns=1 past the would-be
backstop moment) then clean server-end close on peer FIN.
…CN.md (#933)

Second instance of the same root cause as 847b134: the zh env table had
drifted behind the English one. With this row the two tables carry the same
env var set again.
…ault per docs (review of #1530)

The parse mapped negative integers to 0 (disabled) via Math.max(0,x), while the PR body and CONFIGURATION.md (en+zh) all state non-numeric/negative -> default 30000. Align code with the documented contract: invalid input must not silently disable the protection, and it matches the sibling BILI_KEEP_ALIVE_TIMEOUT_MS shape (non-positive -> fallback). 0 remains the explicit opt-out. Adds a pinning test (startup-line echo of the resolved value); verified it fails against the pre-fix parse.
@ranxianglei
ranxianglei force-pushed the 2026-09-28_clienterror-backstop branch from ff3a687 to 8c7b636 Compare September 28, 2026 02:01
@ranxianglei
ranxianglei changed the base branch from 2026-09-27_mitm-rst-idle-hardening to master September 28, 2026 02:01
@ranxianglei

Copy link
Copy Markdown
Owner Author

Conflict resolved — rebased onto master (new head, 3 commits).

Root cause of the conflict was benign: #1528 picked up the zh-CN doc rows as a review fix (b77a742) before merging, and this branch carried the same rows independently (847b134). Git recognized the duplicate (identical patch-id) and skipped it automatically during rebase — no semantic conflicts, zero manual hunks. The remaining 3 commits apply clean:

Post-rebase verification: env rows appear exactly once per language (3 #1452 rows + 1 backstop row, en+zh); hardening suite 11/11 (now includes the backstop tests); typecheck clean. PR base retargeted master (it was still pinned to the merged #1528 branch). Ready to merge once CI is green.

@ranxianglei
ranxianglei merged commit c7c3c99 into master Sep 28, 2026
5 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 28, 2026
@ranxianglei
ranxianglei deleted the 2026-09-28_clienterror-backstop branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

clientError drain path lacks terminal backstop: silent peer holds half-open socket indefinitely (#1528 item-1 follow-up)

1 participant