Skip to content

[PR] fix: event-loop watchdog kills the whole process, not just its worker - #6158

Closed
PatelVro wants to merge 2 commits into
ranxianglei:masterfrom
PatelVro:2026-11-10_watchdog-kill-whole-process
Closed

PatelVro wants to merge 2 commits into
ranxianglei:masterfrom
PatelVro:2026-11-10_watchdog-kill-whole-process

Conversation

@PatelVro

Copy link
Copy Markdown
Contributor

Fixes #6157

Problem

The #4762 watchdog's worker called process.exit(1) on a hard trip. In a worker thread that ends only the worker, so the wedged proxy survived its own watchdog, kept the port, and after unblocking logged could not arm (Error: watchdog worker exited with code 1) - continuing unwatched. Seen in the field in the linked issue and in #5823.

Fix

src/event-loop-watchdog.ts: on a trip in "exit" mode the worker now calls process.kill(process.pid, "SIGKILL"), with process.exit(1) kept as a last resort if the kill throws. SIGKILL is delivered by the OS and needs nothing from the blocked main thread (a JS SIGTERM listener would never run); on Windows process.kill maps to TerminateProcess. The forensic FATAL line is still written synchronously (appendFileSync) before the kill. "report" mode is unchanged.

Behavior note for spawners: on POSIX the child now ends by signal (signal: "SIGKILL", code: null) instead of exit code 1; on Windows it still exits with status 1. A respawn loop that only restarts on a non-zero code would need to treat a signal exit the same way.

Test

New test in tests/event-loop-watchdog.test.ts runs the real "exit" path in a child process (thresholds 60/150 ms), wedges the child's main thread for 5 s, and asserts that the child dies before printing SURVIVED, exits abnormally, and wrote the FATAL line to its log file.

  • Before the fix: fails (wedged process outlived its watchdog (status=0 signal=null)).
  • After the fix: passes, 3 of 3 local runs; the other 7 watchdog tests still pass.
  • npm run typecheck: clean.
  • Local runs on Windows 11 / Node 24.19.0 only; the full-suite verdict is the CI test lanes.

The child is written as a module file instead of -e --input-type=module: an eval worker inherits execArgv, and an ESM-typed eval worker has no require, so the watchdog would never start in that harness.

Scope

Single module, no config, wire, persistence or kernel change.

🤖 Generated with Claude Code

process.exit() inside a worker thread ends only the worker, so a wedged
proxy survived its own watchdog, kept the port and ran on unwatched.
SIGKILL the process from the worker instead, and cover the real "exit"
path with a child-process test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ranxianglei ranxianglei changed the title fix: event-loop watchdog kills the whole process, not just its worker [PR] fix: event-loop watchdog kills the whole process, not just its worker Nov 10, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

🔶 需要决策

查重结论: duplicate of PR #5450 — same symptom, same mechanism, same root cause. #5450 ("fix(watchdog): terminate the serving process from fatal worker", @Maziar123) was opened 2026-11-01, nine days before this PR (#6158, opened 2026-11-10). Both fix GitHub issue #6157 by replacing the fatal-mode worker's process.exit(1) with process.kill(process.pid, "SIGKILL") after the synchronous FATAL append. Per first-come-first-served the earlier lane (#5450) should survive; I can't close either PR — that's yours.

背景一句话: two external PRs independently fixed the same watchdog bug with the same mechanism; only one should merge.

待决问题: which PR survives — keep #5450 / close #6158, or keep #6158 / close #5450?

选项与代价:

推荐: A. 不决定的默认行为: both stay open, nothing merges, and I push nothing further to either branch until the lane is picked.


Independent review of #6158 (I was woken on this thread; stands regardless of the duplicate decision):

  • Direction gate: issue [Bug] event-loop watchdog never kills the proxy: process.exit() in a worker thread ends only the worker, so the wedged proxy keeps the port and runs on unwatched #6157 is valid — field evidence is solid (bili 0.1.212 on Windows 11: FATAL line, then 18 min later "could not arm … continuing unwatched", same pid serving ~24 h more until manual kill), and the root-cause analysis is correct: process.exit() in a WorkerThread ends only the worker, so the wedged proxy outlives its own watchdog. The fix targets the producer (wrong termination primitive), not a symptom mask. ✓
  • Regression pass: sole production call site is src/server.ts:701 (armEventLoopWatchdog, onSpawnFailure → warn log). No in-repo respawn logic keys off non-zero exit code — src/launcher.ts reports signal exits correctly, src/restart.ts:336 handles both code and signal. The parent-side worker 'exit' handler (src/event-loop-watchdog.ts:217) sees code=null under SIGKILL, so the misleading "continuing unwatched" warn can no longer fire (the whole process dies anyway). "report" mode, thresholds, heartbeat, stop semantics untouched. No regression found. ✓
  • Test validity: the new test runs the real "exit" path in a child process (not report mode, not mocked), wedges the child's main thread synchronously for 5 s, and asserts death before SURVIVED + abnormal termination + FATAL line present — exactly the regression the old suite missed ("every watchdog test runs fatalMode: report"). Deterministic: fixed thresholds with wide margins, 60 s timeout cap, no free-port assumption. CI green on ubuntu+windows × node 22/24. ✓
  • Diff cleanliness: 2 files, single module, every line on-topic. ✓
  • Behavior disclosure: author disclosed that on POSIX the child now ends by signal (code:null, signal:"SIGKILL") instead of exit code 1; Windows unchanged (status 1). Honest and complete. ✓
  • Gap found (minor, belongs to whichever PR survives): docs/technical-notes.md:566 and that section's "Behavior change (disclosed)" paragraph still say the fatal trip does process.exit(1) — factually wrong once either PR lands; the Tests bullet also omits the new exit-mode child test. I will fix these as a labeled commit on the surviving lane after you pick it — pushing now would risk wasted/conflicting commits.

Live status (verified today): #6158 CI green at head 5c650a8 (36 success / 12 skipped / 0 failed; skips are gated e2e lanes this PR doesn't touch). master has moved 5 commits past the PR base, none touching src/event-loop-watchdog.ts, tests/event-loop-watchdog.test.ts or docs/technical-notes.md — no rebase needed. #5450 is likewise open, mergeable, CI green at c5cacea; its mirror thread has no review posted yet.

标签: bug, scope/other, p1, duplicate, needs-decision(watchdog 自杀路径失效 = 触发即可用性缺陷;重复车道待 owner 选定)

八问:

  1. 是什么问题: watchdog 硬超时触发时 worker 调 process.exit(1),但 worker 线程里这只会杀掉 worker 自己——卡死的代理进程继续活着、继续占端口,永久失去自动自愈能力(现场案例:同一 pid 又服务了约 24 小时直到人工 kill)。
  2. 什么原因导致的: WorkerThread 内 process.exit() 只终止 worker 线程而非宿主进程;主线程同步卡死时没有任何其他机制能结束整个进程。[bug] After event loop wedge, zombie still holds port 18789 — parent respawn loop spins on EADDRINUSE indefinitely; manu | [bug] 事件循环卡死后僵尸仍握 18789 端口,parent 重拉全部 EADDRINUSE 死循环,只能手工杀进程恢复(event loop wedge holds listen socket, respawn loop fails with EADDR… #4762 原设计隐含了「worker 退出 == 进程退出」的错误假设。
  3. 影响范围与严重程度: 所有启用 event-loop watchdog 的部署,且需主事件循环被完全阻塞超过 hardTimeoutMs 才触发(触发面窄);一旦触发,核心承诺「卡死自动退出」完全失效、必须人工重启——可用性缺陷,p1 级。
  4. 怎么修复的: 把致命模式的终止原语从 process.exit(1) 换成 process.kill(process.pid, "SIGKILL")——kill 由 OS 投递给整个进程(worker 与主线程同 pid),不依赖被卡死的主线程配合;Windows 上映射为 TerminateProcess。FATAL 取证行仍在 kill 前同步写入。
  5. 是否有回归问题: 逐项审过——唯一生产调用点 src/server.ts:701;仓内无 respawn 逻辑以非零 exit code 为键(launcher/restart 均正确处理 signal 退出);worker 'exit' handler 在 SIGKILL 下 code=null,误导性的 "continuing unwatched" warn 反而不会再出现(进程已死);"report" 模式、阈值、心跳、stop 语义均未动。未发现回归。一项已知披露变更:POSIX 上子进程改为以信号结束而非 exit code 1——外部 spawner 若只认非零 code 需同等对待信号退出(作者已在 PR 中说明)。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及——架构不变,仅更换终止原语。
  8. 是否引入了新的配置: 不涉及。

收工判定: #6158 本身质量达标(单模块修复 + 真回归测试 + CI 绿 + diff 干净),可合;但它与更早车道的 #5450 是 100% 重复(同症状+同机制+同根因),按先来后到建议关闭本 PR、由 #5450 承载修复(选项 A)。决策前我不向任何分支推送提交。

@ranxianglei ranxianglei added duplicate This issue or pull request already exists needs-decision scope/other labels Nov 11, 2026
@PatelVro

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I had not seen #5450 when I opened this; it found the same bug first, so I'm fine with whichever lane you pick.

One factual update for the decision: as of now GitHub reports #5450 as CONFLICTING (merge state DIRTY), while #6158 is MERGEABLE with CI green at 5c650a8.

Just say which and I'll follow it.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Thanks for the update — I re-verified everything live before replying, and the picture moved overnight (both PRs got new commits):

#5450 is no longer conflicting. Its author merged origin/master into the branch (new head eeda190, resolving the CHANGELOG conflict by keeping both sides' Unreleased entries). Live state just now: mergeable: True, mergeable_state: "blocked" — that is a branch-protection/required-checks state, not a merge conflict — and CI green at that head (36 success + 12 skipped, 0 failed). So the "needs conflict resolution" cost on Option A is gone.

#6158 picked up your docs commit. The current head is now 431aea9 ("docs: describe the watchdog fatal path as a whole-process SIGKILL…"). I read the diff: it fixes exactly the three lines I flagged — the Watchdog bullet, the "Behavior change (disclosed)" paragraph, and the Tests bullet — and the wording is accurate. CI at the new head: main CI lane green; one e2e lane (e2e-codex-persona) shows cancelled in the new batch (at the old head 5c650a8 every lane was green) — it needs a re-run before the head can be called fully green. Merge state is behind (master advanced past the base); no file conflicts.

So the corrected record vs your comment: #5450 is not DIRTY anymore (mergeable, blocked only by branch protection), and #6158 is green on main CI at its new head with one cancelled e2e lane to re-run — "CI green at 5c650a8" was true for that head, but the head has since moved.

The lane pick is @ranxianglei's call — I can't close or merge either PR, so "just say which" can only be answered by the owner. My recommendation is unchanged and now reinforced: Option A — merge #5450, close #6158 as duplicate (earlier lane; now conflict-free, current with master, green; superset coverage incl. the port-rebind test and the CHANGELOG entry). If the owner picks A: your docs commit (431aea9) should be ported onto #5450's lane as well, since #5450 leaves the same doc drift. If the owner picks B: your offer to port the port-rebind assertion + CHANGELOG entry (with credit to @Maziar123) is the right completion path, plus a rebase onto master and the e2e-codex-persona re-run.

八问(结论与上一条回复一致,仅状态事实更新):

  1. 是什么问题: watchdog 硬超时触发时 worker 调 process.exit(1),只杀掉 worker 自己,卡死的代理进程继续活着占端口、永久失去自愈能力。
  2. 什么原因导致的: WorkerThread 内 process.exit() 只终止 worker 线程而非宿主进程;[bug] After event loop wedge, zombie still holds port 18789 — parent respawn loop spins on EADDRINUSE indefinitely; manu | [bug] 事件循环卡死后僵尸仍握 18789 端口,parent 重拉全部 EADDRINUSE 死循环,只能手工杀进程恢复(event loop wedge holds listen socket, respawn loop fails with EADDR… #4762 原设计隐含了「worker 退出 == 进程退出」的错误假设。
  3. 影响范围与严重程度: 所有启用 watchdog 的部署且主循环完全阻塞超过 hardTimeoutMs 才触发(面窄);触发后「卡死自动退出」承诺完全失效、需人工重启——可用性缺陷,p1。
  4. 怎么修复的: 致命模式终止原语换成 process.kill(process.pid, "SIGKILL")(OS 投递给整个进程,不依赖被卡死的主线程;Windows 映射 TerminateProcess),FATAL 取证行仍先同步写入。
  5. 是否有回归问题: 无——唯一生产调用点 src/server.ts:701 已审;仓内无 respawn 逻辑以非零 exit code 为键;worker 'exit' handler 在 SIGKILL 下 code=null,误导性 warn 反而消失;report 模式/阈值/心跳/stop 未动。已知披露变更:POSIX 子进程改以信号结束(code:null)而非 exit code 1。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及。
  8. 是否引入了新的配置: 不涉及。

状态更新(本轮新增事实): #5450 已合入 master 最新代码、无冲突、CI 绿;#6158 新增 docs 提交(修复了我上条指出的三行文档漂移,措辞准确)、主 CI 绿、e2e-codex-persona 车道被取消需重跑、merge state 为 behind。

收工判定: 两个 PR 现在都技术可合,但仍是同一修复的两条车道——等 owner 拍板选 A(推荐)或 B;拍板前我不向任何分支推送提交。

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6464 (merged) — closing as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working duplicate This issue or pull request already exists needs-decision p1 scope/other

Projects

None yet

2 participants