Skip to content

[PR] fix: release process handlers when an embedded server closes - #7026

Closed
Afloat16 wants to merge 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-17_server-process-listeners
Closed

Afloat16 wants to merge 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-17_server-process-listeners

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

Each startServer() attaches process-level SIGINT/SIGTERM, uncaught-exception and rejection handlers. Closing the returned HTTP server leaves those closures registered. Repeated embedded start/close cycles therefore retain closed server instances, multiply logging, and leave their process-exiting shutdown callbacks subscribed to future signals.

Change

Give this server's handlers stable identities and remove only those handlers on close. Restore exactly one set when that same server listens again, including the port-claim retry path. Preserve foreign process listeners and handlers belonging to another live server. Keep the existing per-listen callback reset unchanged.

Standalone proxy lifecycle only: plugin injection uses createEngineContext and never installs these process handlers. Request payloads, compression modes, shutdown drain timing, and error classifications are unchanged.

Verification

  • Base: 301e1a5459a77d5578206927c8062a58381af642.
  • All three new real-server lifecycle regressions fail on the base with retained uncaughtException listeners.
  • Final lifecycle, HTTP close/linger, and port-owner suites: 34 tests passed. Coverage includes three consecutive starts/closes, simultaneous instances plus an unrelated listener, and re-listening on the same server.
  • Full npm run typecheck, npm run build, and git diff --check: passed.
  • During implementation, preserving Node's previously removed built-in listening callbacks changed the existing linger test's behavior. That approach was removed; the final patch keeps the original reset and reinstalls only its own lifecycle hook. The unchanged close/linger suite passes on both the base and final patch.
  • Codex's zero-token e2e preflight finds the binary and built artifact, but the configured upstream at 127.0.0.1:8199 is unreachable. The real-model suite was not run, and its gate is unchanged.

This is separate from #1240's logger EPIPE handling and #4380's socket-linger force-close behavior. Author history and live all-state searches for startServer/process-listener cleanup found no equivalent fix.

Fixes #7025

@ranxianglei ranxianglei changed the title fix: release process handlers when an embedded server closes [PR] fix: release process handlers when an embedded server closes Nov 17, 2026
Merge upstream master 6a641f3, including the architecture module-map and schedule-independent ranxianglei#2929 probe fixes.

Preserve the existing PR head and exact contribution patch. Validation of this refreshed head runs in upstream CI; no local test rerun is claimed.
@ranxianglei

Copy link
Copy Markdown
Owner

Review 结论:方向对,但 close 时机有一个回归,补一个 guard 后即可收。

修复本身没问题:install/remove 幂等 + close 时摘除,解决了内嵌 server 反复起停重复挂处理器的问题。但摘除时机有个洞:

server.on("close", removeProcessHandlers) 里的 close 在 shutdown() 刚开始(listener 停止接受新连接)就触发,而 session flush(finishDrain → flushConversations/flushAllSessions)发生在之后、可能持续数秒。这个窗口里 SIGINT/SIGTERM 处理器已被摘除——用户在 flush 期间觉得"卡住了"再按一次 Ctrl+C(非常常见的操作),这次信号没有任何人接 → 进程直接死 → 未 flush 的压缩状态丢失。恰好是这次修复想保护的"宿主进程干净退出"的反面。

修法:removeProcessHandlers 里加一个 shuttingDown guard——正在 graceful shutdown 就跳过摘除(进程本来就要走 process.exit(0) 退出,处理器留着无副作用);"close 但没 shutdown"的嵌入式路径(端口重试)照常清理。

已验证:新回归测试在未修代码上红、修后绿(第二次 Ctrl+C 窗口内处理器必须在位);相邻 server 生命周期测试(server-close-all-connections / issue4982 / issue5167)13/13 通过;typecheck 通过。

我对该分支没有 push 权限,请应用以下补丁(或给我写权限,我来推):

From 6b7865d847e7da14817b6bc5e563c9fd1f630fde Mon Sep 17 00:00:00 2001
From: ranxianglei <ranxianglei@users.noreply.github.com>
Date: Wed, 18 Nov 2026 09:18:58 +0800
Subject: [PATCH] fix: keep signal handlers alive during the shutdown flush
 window
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

close fires when the listener stops accepting, i.e. before the session
flush in finishDrain. Removing the handlers there left a second Ctrl+C
during the flush window unhandled — the process died with unflushed
compression state. Skip removal while shuttingDown; a closed embedded
server that never shut down (port-ladder retry) still cleans up.

(cherry-pick pending: proposed fix for PR #7026)
---
 src/server.ts                          |  7 +++++
 tests/server-process-listeners.test.ts | 38 ++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/src/server.ts b/src/server.ts
index 9bdf2dc91..3e7ecb5f4 100644
--- a/src/server.ts
+++ b/src/server.ts
@@ -1832,6 +1832,13 @@ export async function startServer(opts: ProxyOptions): Promise<http.Server> {
     }
     const removeProcessHandlers = (): void => {
         if (!processHandlersInstalled) return;
+        // #7026 review: the close event fires when the listener stops accepting,
+        // i.e. BEFORE the session flush in finishDrain. Stripping the handlers
+        // here would leave a second Ctrl+C during the flush window unhandled —
+        // the process would die with unflushed compression state. Keep them
+        // until the process actually exits; a closed embedded server that never
+        // shut down (port-ladder retry path) still cleans up normally.
+        if (shuttingDown) return;
         processHandlersInstalled = false;
         process.removeListener("uncaughtException", onUncaughtException);
         process.removeListener("unhandledRejection", onUnhandledRejection);
diff --git a/tests/server-process-listeners.test.ts b/tests/server-process-listeners.test.ts
index cad3f910e..a736ffd7e 100644
--- a/tests/server-process-listeners.test.ts
+++ b/tests/server-process-listeners.test.ts
@@ -104,3 +104,41 @@ test("re-listening restores exactly one handler set and closing removes it again
         restoreListeners(before);
     }
 });
+
+// #7026 review: the "close" event fires when the listener stops accepting, BEFORE
+// the session flush in finishDrain. Handlers stripped there left the flush window
+// unprotected — a second Ctrl+C would kill the process mid-flush. The handlers
+// must survive until the process actually exits. (Placed last: shutdown() latches
+// the module-level flag, so the clean-remove tests above must run first.)
+// #7026 review: the "close" event fires when the listener stops accepting,
+// BEFORE the session flush in finishDrain. Handlers stripped there left the
+// flush window unprotected — a second Ctrl+C would kill the process mid-flush.
+// The handlers must survive until the process actually exits. (Placed last:
+// shutdown() latches the module-level flag, so the clean-remove tests above
+// must run first.)
+test("graceful shutdown keeps signal handlers until the process exits", async () => {
+    const before = snapshot();
+    const realExit = process.exit;
+    const exits: Array<number | string | undefined> = [];
+    process.exit = ((code?: number | string) => { exits.push(code); }) as typeof process.exit;
+    const server = await start();
+    try {
+        const serving = snapshot();
+        process.kill(process.pid, "SIGINT");
+        await once(server, "close");
+        // Let the shutdown chain reach (the stubbed) process.exit first: the
+        // test process must not drain out from under the assertions while the
+        // flush is still running.
+        await new Promise((resolve) => setTimeout(resolve, 500));
+        assertSameListeners(serving);
+        // Second Ctrl+C during the flush window: must be absorbed by the
+        // idempotent shutdown(), not delivered to Node's default handler.
+        process.kill(process.pid, "SIGTERM");
+        await new Promise((resolve) => setImmediate(resolve));
+        assertSameListeners(serving);
+        assert.ok(exits.length <= 1);
+    } finally {
+        restoreListeners(before);
+        process.exit = realExit;
+    }
+});
-- 
2.43.0

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复 —— 本条是 PR #7026 的 review 跟进(guard 补丁),无其他同机制修复条目;PR 描述已声明与 #1240(logger EPIPE)、#4380(socket-linger)不同。

标签:bug / scope/wiring / p1(缺陷类;进程信号接线属 wiring;review 发现的 flush 窗口二次信号丢状态属数据丢失类,按 2026-10-19 口径 p1 不降 p2;无具体客户端,不打客户端标)

已独立复核诊断并应用补丁,但推 fork 分支被拒,需要你或 Afloat16 走一步(见下)。

诊断复核(代码路径已逐行核对)

  • shutdown() 先 latch shuttingDown 再 server.close(cb),cb 即 finishDrain(src/server.ts:1777-1788);removeProcessHandlers 在 startServer 时就挂在 close 上(src/server.ts:1845),早于 close 回调注册,所以 close 事件触发时先摘处理器、后开始 flush。精确时序:server.close() 立即停止接受新连接,close 事件在连接排空后发出(代码注释 1783-1785 与 drain backstop 1794-1814 都印证这一点),flush(flushConversations/flushAllSessions)在事件之后跑、可达数秒——窗口内 SIGINT/SIGTERM 无人接,二次 Ctrl+C 直接杀死进程,未 flush 的压缩状态丢失。结论与你的判断一致。
  • 顺带确认:die / busy-port / crash 三条路径也是先 shuttingDown = true 再 server.close()(1545/1686/1700),guard 让它们的 flush 窗口同样受保护,且进程秒级退出,无长期滞留。

分支上已做(commit e6e5b6d45f,独立提交)

  1. 先把 master 合入 PR 分支(落后 41 个提交,clean merge;master 侧 src/server.ts 仅有无关的 AUTO_INJECTED_NOTIFICATION_PREFIXES 一条漂移)。
  2. 应用你的 guard:removeProcessHandlers 加 if (shuttingDown) return; + 注释;回归测试照收。两处小修(已标注):补丁里测试头注释块重复了两遍,去重保留一份;注释原文「latches the module-level flag」与代码事实不符——shuttingDown 是 per-server 实例局部变量(src/server.ts:1749),测试放最后的真实原因是其 shutdown 链会在测试进程里跑完整退出路径(flush、closeLogger()、实例清理),已改准。

验证(独立重跑)

  • 红/绿:只加测试不加 fix 时新测试红(uncaughtException 监听器里 onUncaughtException 缺失,正是 close 时被摘除);加 guard 后绿;原有 3 个生命周期测试前后均绿。
  • 相邻生命周期套件(server-close-all-connections / issue4982 ×2 / issue5167)31/31 通过;npm run typecheck、npm run build 通过。

卡点:fork 无 push 权限

  • git push Afloat16/billion-context → remote rejected (permission denied)。推送账号就是 ranxianglei(与你的账号相同),已核对该账号在 fork 上只有 pull 权限(permissions.push=false)——所以你的账号同样推不上去,不是凭据问题。
  • 交接分支已推到主仓:ranxianglei/billion-context 分支 2026-11-18_pr7026-guard-handoff(= PR head + master 同步 + guard 提交 e6e5b6d45f,全部验证在该树上完成)。

解法二选一

  • A:Afloat16 在 fork 的 Settings → Collaborators 加上 ranxianglei,我立刻推到 Afloat16:2026-11-17_server-process-listeners 并删掉交接分支;
  • B:Afloat16 自己 git fetch https://github.com/ranxianglei/billion-context.git 2026-11-18_pr7026-guard-handoff && git cherry-pick e6e5b6d45f 到 PR 分支(master 同步部分已在交接分支里,cherry-pick 前需先 merge master 或整体取交接分支)。

八问

  1. 是什么问题:PR 在 server close 时摘进程信号处理器,但 close 早于 session flush 完成——flush 窗口内第二次 Ctrl+C 无人接,进程直接死,未 flush 的压缩状态丢失。
  2. 什么原因导致的:摘除时机绑在 close 事件上,而 graceful shutdown 是「先 close、后 flush」;原修复的「close 即清理」与该顺序冲突,窗口内无处理器。
  3. 影响范围与严重程度:standalone proxy 的 graceful shutdown(SIGINT/SIGTERM,flush 窗口数秒级)且用户在窗口内二次发信号时触发;后果 = 进程死亡 + 未落盘会话状态丢失(数据丢失类 → p1);嵌入式端口重试路径不受影响。
  4. 怎么修复的:removeProcessHandlers 加 shuttingDown guard——graceful shutdown 中跳过摘除(进程本就走 process.exit(0),处理器留着无副作用,二次信号被 shutdown() 幂等 latch 吸收);「close 但没 shutdown」的嵌入式路径照常清理。
  5. 是否有回归问题:既有行为逐项核过——① Closed embedded servers retain signal and exception handlers #7025 原修复(嵌入式 close 清理):不变,shuttingDown=false 时照常摘除(前 3 个测试钉住);② 端口阶梯 re-listen 恢复:不变(close→摘、listening→重装);③ die/busy-port/crash 路径:处理器从「close 时摘」变为「保留到退出」,进程秒级退出无滞留,属附带收益非语义漂移;④ 无阈值/时序/输出格式/fail-fast 变化。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(仅加一个时机 guard)。
  8. 是否引入了新的配置:不涉及。

判定:代码侧已收口(master 同步 + guard,红→绿、31/31、typecheck+build 全绿),分支处于可合并状态;唯一剩余动作是 fork push 权限(A/B 二选一),解阻后我立即推 PR 分支。

@ranxianglei ranxianglei added bug Something isn't working p1 scope/wiring labels Nov 18, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #7101 (batch-2) — this PR's content (and its review-patch fix where applicable) is now merged via #7101. Closing to avoid a duplicate merge. 本 PR 的内容(含评审补丁,如适用)已通过 #7101 合并,关闭以免重复合并。

Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Nov 18, 2026
close fires when the listener stops accepting, i.e. before the session
flush in finishDrain. Removing the handlers there left a second Ctrl+C
during the flush window unhandled — the process died with unflushed
compression state. Skip removal while shuttingDown; a closed embedded
server that never shut down (port-ladder retry) still cleans up.

(cherry-pick pending: proposed fix for PR ranxianglei#7026)

(cherry picked from commit 6b7865d847e7da14817b6bc5e563c9fd1f630fde)
ranxianglei added a commit that referenced this pull request Nov 18, 2026
…led its own child on win32)

test 4 (added by 894835a, #7026 review) assumed POSIX signal
semantics: process.kill(self, SIGINT) was expected to dispatch to the
installed handler and drive the graceful-shutdown chain. On WINDOWS the
same call terminates the process (TerminateProcess) WITHOUT running the
handlers - Node docs: 'On Windows, the signal being used to terminate a
process cannot be changed. SIGTERM is sent.' Verified on a win10 box:
SIGINT and SIGTERM self-kills die the process with the handler never
invoked. The windows-latest CI child therefore died at the first kill
line: file-level failure, zero TAP output, deterministic.

Invoke the installed handler directly on win32 instead (the identical
shutdown() path a console Ctrl+C would deliver). Also: bounded wait for
the stubbed exit instead of a fixed 500ms sleep (each flush stage is
deadline-capped at 5s, so a fixed sleep races slow volumes on every
platform), and keep the exit stub in finally while the chain is still
pending (last test in the file; a late REAL exit(0) would kill the
runner before the TAP summary).

Verified: 4/4 pass on a real win10 box (node v22.23.2), 4/4 on Linux.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Closed embedded servers retain signal and exception handlers

2 participants