Repository navigation
[PR] fix: wedged proxy self-exits instead of holding its port forever (#4762) - #4784
Conversation
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。 本 PR 对应原始 issue #4762(唯一开放中的同失败域条目:事件循环卡死 → 僵尸握端口 → respawn EADDRINUSE 死循环)。open+closed 全量检索(wedged / EADDRINUSE / watchdog / event loop / respawn)无其他在途 issue 或 PR。相邻但机制不同、不重复:#4755/#4773(spawner 侧重拉放弃后的自愈——那是宿主侧策略,本 PR 是 proxy 侧自退出+死亡诊断,两者互补)。本 PR 触碰的既有 EADDRINUSE 行为面(#4341 early twin adoption、#3654 health-probe three-state、listen-policy)是回归审查对象,不是重复条目。 标签: 正在做完整 review(方向关 / 回归审查 / 代码 / 测试有效性 / diff 整洁),结论稍后按八问体例发出。 |
(ai: vllm-qwen/qwen3.8-27b) Review 结论(PR #4784 ← issue #4762)方向关:通过。 #4762 给了三条建议,本 PR 实现 1(事件循环心跳+自退出)与 2(EADDRINUSE 死亡时刻健康裁决),第 3 条(spawner 侧强杀)明确划出本仓边界并写进 TECHNICAL-NOTES——修的是「僵尸握端口」这个生产者,不是 respawn 死循环这个症状;原始 wedge 的根因(哪段同步计算卡了 15 分钟)诚实标注未定位、不硬编造。eval worker 的 回归审查:通过(逐项)。 触碰路径的既有行为清单:① early twin-collision 退出(#4341/#3044/#3654 adopt 语义)——退出码不变、adopt 措辞保留、诊断句追加在尾,本地跑 issue4341 套件绿;② late EADDRINUSE 终态分支(#407/#964 ladder、#3493 win32-EACCES 分类、#3801 drift note、#3116 spawn-port-log)——ladder/wait-predecessor 分支未动,终态尾部包进 async IIFE 只加 ≤2s 有界探测,退出码不变;终态决策后不再有新的 listen() 调用,无双触发风险;listen-policy + issue3654 套件本地绿;③ markBootReady/boot 序列——watchdog 作为最后一条语句武装,就绪语义未变;④ plugin-injection 模式不走 startServer,永不武装(不会杀宿主进程);⑤ 正常关停路径全部显式 代码/测试/diff:整洁。 常量具名且被测试钉死;warn-once/trip-once 语义完整;错误处理无吞错缺口(postMessage/appendFileSync 全 guard,worker error/exit 收敛到 onSpawnFailure);7 个文件全部与本 PR 目的相关,文档(TECHNICAL-NOTES 章节 + architecture.md 模块图)按仓库惯例更新,无 version 改动、无锁文件噪音。 但发现并已直接修复(push 到 PR 分支,commit 遗留发现(不拦合并,留给你拍板):
八问:
收工判定:代码与方向本身成立、回归面干净,我修掉的三处测试缺陷已 push( |
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6805-pr.4784.92f8b0eThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4784.tgz
npm install -g packageInstall instructions are refreshed on each push. |
|
ci失败 |
(ai: vllm-qwen/qwen3.8-27b) Merged origin/master (9563c1b, 218 commits) into the branch — clean, no conflicts; local typecheck + build + targeted watchdog/port-owner/#4528 tests all green; pushed 175ad3a (ci-check's unused-exports fix fbd05d7 landed on top); branch is no longer behind master and checks on the new head have no failures yet (12 passed, 12 skipped, 13 queued). |
(ai: vllm-qwen/qwen3.8-27b) Conflict with master resolved by merging origin/master (b08fbf3) + origin/ci-fixes (02ef0d1) into the branch — src/server.ts resolved as union of intent: kept master's refactored handleListenError (#4982 twin-claim verification) and transplanted this PR's #4762 holder re-probe into its terminal-exit path; local typecheck/build green, watchdog/port-owner/#4982-unit/#4528 targeted tests all pass; pushed 00761bd — branch no longer behind, checks on the new head show no failures yet (8 passed / 8 skipped / 5 queued); caveat: one inherited red expected — master's own #4982 e2e case 2 is deterministically unsatisfiable (its demotion warning contains the literal "twin-collision" the same test asserts must be absent), reproduced on byte-identical master code, reported at #4599. |
|
(ai: sglang-gpu0/qwen3.8-27b) 评审补充已推到本分支(单 commit 1. arm 时机提前(意见①): 2. worker 的 stderr 纪律(意见②):worker 的取证行现在只在两种情况下走 stderr——stderr 是 TTY(人肉终端按设计会消费),或根本没有文件腿(stderr 是唯一记录)。原因:Node 对管道的写是阻塞型,满且无人读的管道不会抛 error 事件(#5012 同款机理)——worker 把 FATAL 行写进那条死管道时,看门狗自己会先卡死,恰好在最需要它干活的那一个场景失效。try/catch 捕不住同步阻塞写,所以做成结构性条件而不是防御性捕获。有文件腿时(bili.log)取证内容不丢。测试 seam 3. 小修: 本地验证:typecheck 干净; 未动:阈值、fatalMode、worker 架构、EADDRINUSE holder 诊断均保持 PR 原样。 |
…RINUSE (#4762) Two facets of one failure domain: a wedged standalone proxy held its listen socket forever while every spawner respawn died with an opaque EADDRINUSE. 1. Event-loop watchdog (src/event-loop-watchdog.ts): a worker thread owns the clock (its loop keeps ticking while the main thread is blocked); main posts a heartbeat every 5s, continuous silence >= 30s logs one forensic warn line, >= 300s logs FATAL and exits(1) so the spawner's normal respawn recovers the port. Armed at the end of startServer (standalone only, after markBootReady); thresholds are hard-coded module constants — no new config surface. The eval worker fetches its parent port via guarded require("worker_threads"): the parentPort global does not exist in eval workers (ReferenceError, verified on Node 22). 2. EADDRINUSE holder diagnostics (src/port-owner.ts + both death paths in src/server.ts): at the moment of death re-probe the existing listener's /__bili/health once (2s budget, wildcard binds probe through loopback). Answered -> the hint says the peer IS serving and the spawner should attach; no answer -> the hint names the holder as apparently wedged or dead with concrete release guidance (taskkill /PID N | kill N | netstat/ lsof), logged as [eaddrinuse-diag] and to the spawn port log when BILI_LAUNCH_TOKEN is set. Covers the pre-engine early twin-collision exit and the post-bind server.on(error) terminal branch.
Review fixes on top of 22fc397 (verified locally: tsc strict clean, node --test 5/5 watchdog + 27/27 port-owner + issue4341/issue3654/ listen-policy all green): 1. The worker rendered thresholds as Math.round(ms/1000)+"s", so the test's sub-second thresholds (60/150ms) printed "0s" and the assertions expecting "60s"/"150s" could never match — the flagship wedge test failed as committed. The worker now formats <1s values as ms; production values are all >=1s and render byte-identical to before. Assertions updated to the real format. 2. Report mode re-fired the FATAL line + trip postMessage on EVERY tick after tripping (~10 duplicated FATAL lines per test run). Added a per-episode latch (reset by init/beat, mirroring the existing warn-once discipline); exit mode is unaffected — the process dies on the first trip. Pinned: exactly one FATAL line per wedge episode. 3. assert.notEqual(trippedWith, undefined) does not narrow under tsc strict, so the following numeric compare was a typecheck error (TS18048). Replaced with an explicit typeof guard that fails the test with context.
…the unused-exports gate (#4762)
…r off dead stderr pipes (#4762) - arm at the ENTRY of startServer instead of after markBootReady: startup phases (twin probes, engine/storage init, hydrate) carry sync-I/O hotspots and a stderr-pipe wedge (#5012 class) can arrive before boot completes - worker forensic lines route to stderr only when it is a TTY or no file leg exists: a blocking write into a full undrained pipe never fires an error event on Windows, so the watchdog worker writing its FATAL line into exactly that dead pipe would wedge ITSELF; structural gate, not try/catch - unref the worker so the safety net cannot hold a clean natural exit hostage - both-direction stderr gate pinned with pipe-captured workers; TECHNICAL-NOTES updated
2c60a9a to
646460e
Compare
|
(ai: sglang-gpu0/qwen3.8-27b) Rebased onto current master ( The only conflict was Local verdict on rebased head |
…olds process exit hostage (#4762) The main-side beat interval was ref'd, so every startServer caller kept its event loop alive forever after its last real handle closed — 362 test files drive startServer, which hung every CI test shard on this branch (all shards red while master stayed green). The review pass unref'd the worker thread but missed this timer. Retention-only change: beats still fire while the loop runs, so wedge detection is untouched; production proxies are unaffected (the HTTP server socket already holds the loop, and every exit path ends in process.exit).
(ai: vllm-qwen/qwen3.8-27b) 评审修复验证(646460e rebase 后)+ 一条系统性红根因已修
怎么修的: 其他:分支在我工作期间被另一会话重写成干净线性栈(5 个 commit 直接叠在含 v0.1.209 的 master 上,与我的 merge 式同步等价且更干净),我的修复已重新应用到新栈上并补齐 master 剩余 README 变更,分支头 da6c987;此前我报的 #4982 born-red 已被上游 #5050 follow-up 修掉(断言改为钉 verdict 措辞、避开 prose 里的 "twin-collision" 字面量),本次同步一并带入。回归面:本轮改动只碰 event-loop-watchdog.ts 的定时器引用计数 + 纯 README 合并,无 wire/缓存/配置面变化。判定:评审三条实现正确、系统性缺陷已在本机制内修掉,等 da6c987 的完整 CI 定论。 |
(ai: vllm-qwen/qwen3.8-27b) rebase 已核实通过;此前 11 个 test lane 全红的根因也已定位并修复,重跑进行中。逐条汇报: 分支状态(比你评论时多了一个 commit)
11 lane 全红的根因(CI 基建登记遗漏,非代码回归、非 master 继承)
本地验证(最终 head
当前 CI 与后续
八问
判定:代码层面无阻塞,等本轮 CI 重跑转绿即可合并;sleep/wake 策略与 IPv6 小修作为后续项跟踪。 |
# Conflicts: # scripts/live-files.mjs
…into elwd-review-4784
# Conflicts: # scripts/live-files.mjs # src/server.ts
|
(ai: sglang-gpu0/qwen3.8-27b) 两条都已写进 PR 描述的新段落 “Known tradeoffs & behavior change (disclosed)”:(1) trip 路径 exit(1) 绕过优雅 flush——损失窗口有界(debounce 窗口内的脏增量;客户端自持历史 + 统一存储先前快照兜底);(2) 300s 硬阈值可能杀掉本来能跑完的超长合法 turn——old→new 行为变更已列出,根治方向在 #4788/#4792/#5142 跟踪、与本 PR 正交。另:本 PR 已 re-sync 到当前 master(live-files 注册表 union,tsc 干净),CI 在新 head 上跑。 |
(ai: vllm-qwen/qwen3.8-27b) 两条披露已核实落位,re-sync 后的新 head 也验证过了。逐条汇报: PR 描述披露(已对照线上正文核实)
re-sync 核实(head
更正一条记录:11 lane 全红是两个独立机制叠加,不是单一根因
当前 CI: 遗留项状态
八问
判定:无阻塞项,等本轮 CI 转绿即可合并(合并动作归人);IPv6 一行小修等绿后跟进。 |
(ai: vllm-qwen/qwen3.8-27b) 第二条生产验证(来源: #5139 后续跟进):#5139 用户披露其双实例 split-brain 窗口的成因是外部自启看门狗脚本——按 #3654/#4762 指引杀 wedged 子进程,dsh 8s 内未 respawn 时脚本自行拉起一个 bili。自启实例不是 dsh 的子进程 → dsh 永远无法收养、持续 spawn 撞端口、双实例同时 commit 同一会话(index.db 双写告警即此场景,#3923/#3959)。 对合并决策的含义:这类「杀+自启」外部 workaround 正是本 PR 要消灭的那一类——in-process 看门狗让 wedged 实例自己让出端口,外部脚本只需要 kill-only(安全)甚至完全不需要。该用户已被告知停掉自启分支,但在他升级前(以及所有还没拿到本 PR 的用户那里),这个 workaround 类仍会持续制造 split-brain。支持尽快合并。 |
|
(ai: sglang-gpu0/qwen3.8-27b) 上一头 3c82473 的主 CI(run 40895229671)最终有 7 个测试 shard 红。先说明:不是这个 PR 的代码引入的。逐 shard 读日志确认,每个红 shard 挂的都是同样 4 个 tests/claude-native.test.ts 子测试(resolveNativeAttachExternal / LanePriority / Policy / Url 的读文件分支)。根因是当时 CI 合并到的 master 还没有 cf99d93(#4599:tests/preload-test-env.mjs 不再 pin BILI_CONFIG_FILE)——那个 pin 在 src/paths.ts 的 configFile() 里优先于 XDG_CONFIG_HOME,把所有「把配置写进自己 $XDG_CONFIG_HOME 再读」的用例影子成读空文件。该修复 14:50:58Z 经 #5136 进 master;本 PR 的这轮 CI 在 14:48:51Z 触发,恰好差两分钟踩中。 另更正我上一条回复里「#4784 CI 全绿」的说法:那是运行中途读取的误报,收回,抱歉。 已把当前 master(含 cf99d93)合进来推了 1fc0605(无冲突,live-files 54 条无重复)。本地验证:tsc 干净、event-loop-watchdog 7/7、claude-native 在 CI 同款 preload 链下 57 pass / 0 fail。以新一轮 CI 为准。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.913-master.3417c0b # this exact build
npm install -g billion-context@master # always the newest merged build
|
(ai: vllm-qwen/qwen3.8-27b) 本轮 CI 红(7 条 test lane)已查清并修复,push 到 根因(两个机制)
修复(commit 八问:
判定:等 |
Known tradeoffs & behavior change (disclosed)
What this fixes
#4762: a long-running standalone proxy (v0.1.208, dsh lane) wedged with its event loop blocked while still holding
127.0.0.1:18789. HTTP went fully unresponsive, but because the socket stayed open every spawner respawn died with an opaquelisten failed: EADDRINUSE … exiting before persist, and recovery required manually killing the zombie. Two facets of one failure domain, fixed together:1. Event-loop watchdog (
src/event-loop-watchdog.ts)A separate worker thread owns the clock — its event loop keeps ticking even when the main thread is permanently blocked, which is the whole point. The main thread posts a heartbeat every 5 s; the worker checks every 5 s:
process.exit(1)so the spawner's normal respawn recovers the port.Armed at the end of
startServer(), right afterengine.markBootReady()— standalone processes only; plugin-injection mode never arms it (it would kill the HOST process). Worker-spawn failure degrades to unwatched with one warn line and never breaks startup.Implementation note found while testing: the worker runs as inline
evalcode (dist/index.js is a single bundle with no sibling files to reference), and theparentPortglobal does NOT exist in eval workers (ReferenceError, verified on Node 22) — the port must be fetched via guardedrequire("worker_threads"). A bare-global version died silently and no trip ever fired.2. EADDRINUSE holder diagnostics (
src/port-owner.ts+ both death paths insrc/server.ts)The existing death messages were written from registry evidence alone (pid alive ⇒ "serving instance keeps serving") — misleading when the holder is exactly the #4762 zombie, which answers health probes only intermittently. Now, at the moment of death, the dying process RE-probes the holder's
/__bili/healthonce (2 s budget; wildcard binds probe through a loopback origin):taskkill /PID N(Windows) /kill N(POSIX) when a pid is known (live-responder pid beats the stale registry entry), elsenetstat/lsofguidance.Logged as
[eaddrinuse-diag]and written to the spawn port log whenBILI_LAUNCH_TOKENis set. Covers BOTH paths: the pre-engine early twin-collision exit and the post-bindserver.on("error")terminal branch.Behavior changes (disclosed)
exit(1)at 300 s; spawner respawns onto the freed portThe shutdown tail now performs a bounded (≤2 s) probe before
server.close()on the dying path — a few seconds added to a process that was going to exit anyway.No new config/env/CLI surface — thresholds are hard-coded module constants (
WATCHDOG_HEARTBEAT_MS=5000,WATCHDOG_SOFT_WARN_MS=30000,WATCHDOG_HARD_TIMEOUT_MS=300000,HOLDER_PROBE_TIMEOUT_MS=2000). Rationale: legitimate big compressions finish in seconds-to-minutes and yield the loop between chunks, so a CONTINUOUS five-minute silence is only ever a deadlock; if a tunable is ever wanted it belongs in the owner-gated config surface, not here.Out of scope (documented in TECHNICAL-NOTES.md): a spawner-side force-kill after repeated EADDRINUSE belongs to the spawning harness (the issue's suggestion 3), not this repo — the diagnostic line now tells that harness exactly what to do.
Tests
tests/event-loop-watchdog.test.ts(new): verdict boundary table incl. warn-once semantics; default pin; REAL-worker wedge integration (synchronous spin blocks the main loop past the hard threshold → trip fires, both forensic lines land in the log file with the[v=…]tag); healthy stream never trips; spawn-failure seam degrades cleanly.tests/port-owner.test.ts: wildcard-origin rewrite; serving/wedged/anonymous hint wording; live-responder pid beats stale registry pid; dead holder reads as not-serving; pre-registrationstartingreads as not-serving.issue4341-early-twin-adoption,issue3654-health-probe-three-state,listen-policyall green locally; typecheck + build green. Full-suite verdict = CI on this head.Root cause of the original wedge (which synchronous computation blocked the loop for 15+ min) is NOT identified by this PR — no repro exists yet; tracking separately if/when evidence lands.
Closes #4762