Skip to content

[PR] fix: wedged proxy self-exits instead of holding its port forever (#4762) - #4784

Merged
ranxianglei merged 11 commits into
masterfrom
2026-10-26_event-loop-watchdog
Oct 29, 2026
Merged

ranxianglei merged 11 commits into
masterfrom
2026-10-26_event-loop-watchdog

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Known tradeoffs & behavior change (disclosed)

  1. Trip path bypasses graceful flush. At the hard verdict the watchdog exits the process while the main thread is still blocked inside synchronous work — shutdown hooks and pending session flushes cannot run. Bounded loss window: only dirty increments inside the debounced save window can be lost; the client keeps its own history copy and unified storage holds prior snapshots as backstop. Known tradeoff, documented here so nobody later hunts it down as a regression.
  2. 300 s hard threshold can kill a long legitimate turn. The design rationale assumed legitimate work never holds the loop for five straight minutes. Mega-sessions have since approached/broken that assumption ([Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788 measured 14–22 s per turn plus ~6 s persistence handoff; [Bug][Codex] Temporary separator selected as compression endpoint, causing acp_status to silently fail to report ~180K t | [Bug][Codex] 临时分隔符被选作压缩端点,导致 acp_status 静默漏报约 18 万 token 可压缩历史 #5139/[Performance][dsh] Mega-sessions pay an ever-growing O(full-history) fold-reconcile tax every turn: expected host churn (one-shot carrier retirement, image/offload retroactive rewrites) misclassified as drift | 大会话每轮付不断增长的 O(全史) fold-reconcile 税:合法宿主变更(o… #5142 show per-turn cost growth 1.8 s → 25 s and one 19-minute continuous block that eventually completed). So occasionally a turn that would have finished gets killed at 300 s. Old → new: previously such turns could take several minutes to complete; now they die at 300 s and the host retries. This is the right availability-first trade (kill + respawn beats holding the port indefinitely), but it is a behavior change. Root-cause direction (moving heavy legs off the main loop) is tracked in [Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788/decision request: #4788 Fix direction for memory growth and request latency in long sessions (retention cap / envelope w | decision request: #4788 长会话内存增长与请求延迟的修复方向(保留上限 / envelope 写入默认 / 超预算驱逐策略) #4792/[Performance][dsh] Mega-sessions pay an ever-growing O(full-history) fold-reconcile tax every turn: expected host churn (one-shot carrier retirement, image/offload retroactive rewrites) misclassified as drift | 大会话每轮付不断增长的 O(全史) fold-reconcile 税:合法宿主变更(o… #5142 and is orthogonal to this PR.

What this fixes

#4762: a long-running standalone proxy (v0.1.208, dsh lane) wedged with its event loop blocked while still holding 127.0.0.1:18789. HTTP went fully unresponsive, but because the socket stayed open every spawner respawn died with an opaque listen failed: EADDRINUSE … exiting before persist, and recovery required manually killing the zombie. Two facets of one failure domain, fixed together:

1. Event-loop watchdog (src/event-loop-watchdog.ts)

A separate worker thread owns the clock — its event loop keeps ticking even when the main thread is permanently blocked, which is the whole point. The main thread posts a heartbeat every 5 s; the worker checks every 5 s:

  • continuous silence ≥ 30 s → ONE forensic warn line (uptime / RSS);
  • continuous silence ≥ 300 s → FATAL line to bili.log + stderr, then process.exit(1) so the spawner's normal respawn recovers the port.

Armed at the end of startServer(), right after engine.markBootReady() — standalone processes only; plugin-injection mode never arms it (it would kill the HOST process). Worker-spawn failure degrades to unwatched with one warn line and never breaks startup.

Implementation note found while testing: the worker runs as inline eval code (dist/index.js is a single bundle with no sibling files to reference), and the parentPort global does NOT exist in eval workers (ReferenceError, verified on Node 22) — the port must be fetched via guarded require("worker_threads"). A bare-global version died silently and no trip ever fired.

2. EADDRINUSE holder diagnostics (src/port-owner.ts + both death paths in src/server.ts)

The existing death messages were written from registry evidence alone (pid alive ⇒ "serving instance keeps serving") — misleading when the holder is exactly the #4762 zombie, which answers health probes only intermittently. Now, at the moment of death, the dying process RE-probes the holder's /__bili/health once (2 s budget; wildcard binds probe through a loopback origin):

  • answered → hint states the peer IS serving; the spawner should attach to it;
  • no answer → hint states the holder appears wedged or dead and names the concrete release: taskkill /PID N (Windows) / kill N (POSIX) when a pid is known (live-responder pid beats the stale registry entry), else netstat/lsof guidance.

Logged as [eaddrinuse-diag] and written to the spawn port log when BILI_LAUNCH_TOKEN is set. Covers BOTH paths: the pre-engine early twin-collision exit and the post-bind server.on("error") terminal branch.

Behavior changes (disclosed)

old new
wedged standalone proxy lived forever holding its port; every respawn died with an opaque EADDRINUSE; manual kill required one warn at 30 s of continuous main-loop silence, FATAL log line + exit(1) at 300 s; spawner respawns onto the freed port
terminal EADDRINUSE message verdict from registry alone ("serving" if the pid was alive) re-probed at death: serving/wedged adjudication + kill guidance appended

The shutdown tail now performs a bounded (≤2 s) probe before server.close() on the dying path — a few seconds added to a process that was going to exit anyway.

No new config/env/CLI surface — thresholds are hard-coded module constants (WATCHDOG_HEARTBEAT_MS=5000, WATCHDOG_SOFT_WARN_MS=30000, WATCHDOG_HARD_TIMEOUT_MS=300000, HOLDER_PROBE_TIMEOUT_MS=2000). Rationale: legitimate big compressions finish in seconds-to-minutes and yield the loop between chunks, so a CONTINUOUS five-minute silence is only ever a deadlock; if a tunable is ever wanted it belongs in the owner-gated config surface, not here.

Out of scope (documented in TECHNICAL-NOTES.md): a spawner-side force-kill after repeated EADDRINUSE belongs to the spawning harness (the issue's suggestion 3), not this repo — the diagnostic line now tells that harness exactly what to do.

Tests

  • tests/event-loop-watchdog.test.ts (new): verdict boundary table incl. warn-once semantics; default pin; REAL-worker wedge integration (synchronous spin blocks the main loop past the hard threshold → trip fires, both forensic lines land in the log file with the [v=…] tag); healthy stream never trips; spawn-failure seam degrades cleanly.
  • tests/port-owner.test.ts: wildcard-origin rewrite; serving/wedged/anonymous hint wording; live-responder pid beats stale registry pid; dead holder reads as not-serving; pre-registration starting reads as not-serving.
  • Regression-checked the touched EADDRINUSE paths: issue4341-early-twin-adoption, issue3654-health-probe-three-state, listen-policy all green locally; typecheck + build green. Full-suite verdict = CI on this head.

Root cause of the original wedge (which synchronous computation blocked the loop for 15+ min) is NOT identified by this PR — no repro exists yet; tracking separately if/when evidence lands.

Closes #4762

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 对应原始 issue #4762(唯一开放中的同失败域条目:事件循环卡死 → 僵尸握端口 → respawn EADDRINUSE 死循环)。open+closed 全量检索(wedged / EADDRINUSE / watchdog / event loop / respawn)无其他在途 issue 或 PR。相邻但机制不同、不重复:#4755/#4773(spawner 侧重拉放弃后的自愈——那是宿主侧策略,本 PR 是 proxy 侧自退出+死亡诊断,两者互补)。本 PR 触碰的既有 EADDRINUSE 行为面(#4341 early twin adoption、#3654 health-probe three-state、listen-policy)是回归审查对象,不是重复条目。

标签: bug dsh scope/wiring p1 —— 修的可用性缺陷(proxy 整体无响应、压缩链路全断、只能手工杀进程恢复,按 owner 修订属「核心能力停摆」类 p1;事故车道 dsh;改动全在 src 侧 server/进程生命周期,非 kernel)。

正在做完整 review(方向关 / 回归审查 / 代码 / 测试有效性 / diff 整洁),结论稍后按八问体例发出。

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 25, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 结论(PR #4784 ← issue #4762)

方向关:通过。 #4762 给了三条建议,本 PR 实现 1(事件循环心跳+自退出)与 2(EADDRINUSE 死亡时刻健康裁决),第 3 条(spawner 侧强杀)明确划出本仓边界并写进 TECHNICAL-NOTES——修的是「僵尸握端口」这个生产者,不是 respawn 死循环这个症状;原始 wedge 的根因(哪段同步计算卡了 15 分钟)诚实标注未定位、不硬编造。eval worker 的 parentPort 全局不存在问题用 guarded require("worker_threads") 解决,集成测试用真 worker 真同步自旋端到端验证过该机制成立。

回归审查:通过(逐项)。 触碰路径的既有行为清单:① early twin-collision 退出(#4341/#3044/#3654 adopt 语义)——退出码不变、adopt 措辞保留、诊断句追加在尾,本地跑 issue4341 套件绿;② late EADDRINUSE 终态分支(#407/#964 ladder、#3493 win32-EACCES 分类、#3801 drift note、#3116 spawn-port-log)——ladder/wait-predecessor 分支未动,终态尾部包进 async IIFE 只加 ≤2s 有界探测,退出码不变;终态决策后不再有新的 listen() 调用,无双触发风险;listen-policy + issue3654 套件本地绿;③ markBootReady/boot 序列——watchdog 作为最后一条语句武装,就绪语义未变;④ plugin-injection 模式不走 startServer,永不武装(不会杀宿主进程);⑤ 正常关停路径全部显式 process.exit(finishShutdown/EADDRINUSE 尾/early 尾均核实),watchdog 的 interval+worker 无人 stop() 也不会拖住进程退出——无生命周期回归。

代码/测试/diff:整洁。 常量具名且被测试钉死;warn-once/trip-once 语义完整;错误处理无吞错缺口(postMessage/appendFileSync 全 guard,worker error/exit 收敛到 onSpawnFailure);7 个文件全部与本 PR 目的相关,文档(TECHNICAL-NOTES 章节 + architecture.md 模块图)按仓库惯例更新,无 version 改动、无锁文件噪音。

但发现并已直接修复(push 到 PR 分支,commit 9a1ae796e): 新写的旗舰集成测试在提交状态是红的——worker 用 Math.round(ms/1000)+"s" 渲染阈值,测试用的亚秒阈值 60/150ms 被渲染成 "0s",断言期望的 "60s"/"150s" 永远匹配不上(本地实测 fail 1/5);同文件还有 tsc strict 报错 TS18048(assert.notEqual(x, undefined) 不收窄类型,后续数值比较编译不过);另外 report 模式 trip 后每个 tick 重复打 FATAL 行(每次测试运行刷 ~10 行重复 FATAL)。三处均已修:worker 对 <1s 值渲染为 ms(生产值全部 ≥1s,日志措辞逐字节不变)、trip 加每 episode 闩锁(对齐既有 warn-once 纪律,exit 模式不受影响)、类型收窄改显式 typeof guard。验证:tsc strict 干净;watchdog 5/5、port-owner 27/27、issue4341/issue3654/listen-policy 14/14 本地全绿。 提醒一点:PR 描述声称「typecheck + build green」,但这两个缺陷说明新测试文件当时没有真正跑过——后续 PR 的本地验证请包含自己新增的文件。

遗留发现(不拦合并,留给你拍板):

  1. 休眠误杀风险:gap 基于墙钟(Date.now)。笔记本合盖休眠 ≥5 分钟后唤醒,worker 会看到虚假 gap≥300s,把完全健康的 proxy 杀掉。影响面:spawner 车道(dsh/opencode 等)靠重拉自愈无感;手动 bili start 守护进程会在唤醒后静默死亡需重启(此前能扛过休眠)。选项:A(推荐)trip 判定要求连续两个 tick 确认——真实 wedge 检测最多多 5s,健康进程唤醒后一个心跳内恢复即免疫,~4 行改动;B 接受风险不动;C 做挂起感知(hrtime 对比,复杂度高收益低)。你拍板选 A 的话我立即补进本分支。
  2. IPv6 字面量边角:holderProbeOrigin("--host ::1") 生成 http://::1:port(缺方括号,非法 URL)→ 探测必失败 → 误判 wedged。默认绑定全是 127.0.0.1,仅手动 IPv6 loopback 绑定的死亡路径可达,一行级小修可随手做,先记录不阻塞。

八问:

  1. 是什么问题:长跑 proxy 事件循环被同步计算卡死后仍握着监听端口,HTTP 全断,宿主每次重拉都 EADDRINUSE 死掉,只能手工杀僵尸恢复。
  2. 什么原因导致的:两层机制缺失——①独立进程没有任何「主循环永久失声」的检测/自退路径,socket 开着端口就永远不让;②死亡诊断只信注册表证据(pid 活着 ⇒ 「对方在服务」),而僵尸恰好间歇性应答探测,把假服务当真服务报给 spawner。原始 wedge 的具体根因未定位(PR 已如实声明,另案追踪)。
  3. 影响范围与严重程度:所有独立 proxy 形态(手动 daemon、各车道子进程、docker)在发生 ≥5 分钟连续主循环阻塞时中招;严重度 p1——压缩链路整体停摆且无自动恢复,用户必须人工介入。触发条件苛刻(需要真的出现长同步阻塞,目前只有一例现场证据),但一旦发生就是完全不可用。
  4. 怎么修复的:worker 线程拥有时钟(主线程卡死它照样走表):主线程 5s 心跳,连续静默 30s 记一次取证 warn、300s 记 FATAL 并 process.exit(1) 让 spawner 正常重拉回收端口;同时两条 EADDRINUSE 死亡路径在退出前对端口持有者做一次 2s 有界健康复探,把「在服务/疑似卡死」裁决和具体释放手段(pid + kill/taskkill/netstat 指引)写进死亡消息与 spawn port log。只武装独立进程,plugin 注入模式永不武装。
  5. 是否有回归问题:审过的既有行为逐项保持(twin-collision adopt 语义、EADDRINUSE ladder 契约、boot 就绪序列、plugin 模式、正常关停生命周期——见上回归审查节)。有意的行为变更两处均为作者已披露:卡死 proxy 从永生占端口变为 300s 自退;死亡消息从纯注册表裁决变为复探裁决(死亡进程上多 ≤2s 延迟)。我的修复 commit 对生产行为零改动(日志措辞逐字节不变,闩锁只影响测试专用 report 模式)。新增风险一处:休眠唤醒误杀(见遗留发现 1,待拍板)。
  6. 是否破坏缓存:不涉及。未触碰任何缓存/重试/增量游标/去重键;死亡路径多出的探测是 connection: close 的一次性 GET,不进连接账本、不碰会话状态与前缀缓存。
  7. 是否改了设计方向:不涉及。纯增量护栏+诊断,架构/边界/数据流未转向;spawner 侧强杀策略明确留在宿主侧(边界已文档化)。
  8. 是否引入了新的配置:无。阈值全部为硬编码模块常量(5s/30s/300s/2s),理由已写明(合法大压缩秒到分钟级完成且分片让出循环,连续 5 分钟静默只可能是死锁);将来若要可调应走 owner-gated 配置面,此处不私开。

收工判定:代码与方向本身成立、回归面干净,我修掉的三处测试缺陷已 push(9a1ae796e,fast-forward,无冲突,mergeable=true)。等 CI 在新 head 上全绿即可合并(合并动作归人);休眠误杀一项等你拍板 A/B/C,选 A 我当场补进分支。

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-26_event-loop-watchdog (1fc0605)

Latest build — install this exact version

npm install -g billion-context@0.0.6805-pr.4784.92f8b0e

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
1fc0605 billion-context@0.0.6805-pr.4784.92f8b0e build
3c82473 billion-context@0.0.6798-pr.4784.976e898 build
cdcaff0 billion-context@0.0.6752-pr.4784.ea2c1a3 build
da6c987 billion-context@0.0.6712-pr.4784.f652f15 build
8ff5fe2 billion-context@0.0.6707-pr.4784.e6622bd build
646460e billion-context@0.0.6698-pr.4784.0eac985 build
2c60a9a billion-context@0.0.6690-pr.4784.b029292 build
00761bd billion-context@0.0.6609-pr.4784.4b091b9 build
fbd05d7 billion-context@0.0.6589-pr.4784.730af9e build
58700fb billion-context@0.0.6283-pr.4784.eefae64 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4784.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

ci失败

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merged origin/master (9563c1b, 218 commits) into the branch — clean, no conflicts; local typecheck + build + targeted watchdog/port-owner/#4528 tests all green; pushed 175ad3a (ci-check's unused-exports fix fbd05d7 landed on top); branch is no longer behind master and checks on the new head have no failures yet (12 passed, 12 skipped, 13 queued).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Conflict with master resolved by merging origin/master (b08fbf3) + origin/ci-fixes (02ef0d1) into the branch — src/server.ts resolved as union of intent: kept master's refactored handleListenError (#4982 twin-claim verification) and transplanted this PR's #4762 holder re-probe into its terminal-exit path; local typecheck/build green, watchdog/port-owner/#4982-unit/#4528 targeted tests all pass; pushed 00761bd — branch no longer behind, checks on the new head show no failures yet (8 passed / 8 skipped / 5 queued); caveat: one inherited red expected — master's own #4982 e2e case 2 is deterministically unsatisfiable (its demotion warning contains the literal "twin-collision" the same test asserts must be absent), reproduced on byte-identical master code, reported at #4599.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

评审补充已推到本分支(单 commit fix: arm the event-loop watchdog at startServer entry; keep its worker off dead stderr pipes (#4762)),对应两条评审意见 + 一处实施时发现的小修:

1. arm 时机提前(意见①):armEventLoopWatchdog 从 markBootReady() 之后移到 startServer 入口(twin 探测之前)。理由:#4762 自己的病灶类就是同步 I/O 热点(storage init/hydrate/fold-reconcile),stderr 管道楔死也随时可至——只看 boot-ready 之后的窗口,启动期的楔死会完全漏掉。plugin 注入模式不走 startServer,宿主进程仍不可能被自杀(原约束保持)。configureLogger 此处幂等(与 early-twin 路径同写法),engine 上下文随后用同一路径再配一次。

2. worker 的 stderr 纪律(意见②):worker 的取证行现在只在两种情况下走 stderr——stderr 是 TTY(人肉终端按设计会消费),或根本没有文件腿(stderr 是唯一记录)。原因:Node 对管道的写是阻塞型,满且无人读的管道不会抛 error 事件(#5012 同款机理)——worker 把 FATAL 行写进那条死管道时,看门狗自己会先卡死,恰好在最需要它干活的那一个场景失效。try/catch 捕不住同步阻塞写,所以做成结构性条件而不是防御性捕获。有文件腿时(bili.log)取证内容不丢。测试 seam stderrTty 让两个方向都能在任意终端环境下确定性跑。

3. 小修:worker.unref()——安全网不得扣押进程自然退出(un-unref'd worker 会让一个正在关闭的 proxy 卡住退出;现有出口都走 process.exit,但可用性大于护栏,看门狗应当优雅消失而不是滞留)。

本地验证:typecheck 干净;tests/event-loop-watchdog.test.ts 7/7 通过(含新增的 stderr 双向门:文件腿在且非 TTY → stderr 断言为空、文件有 FATAL;无文件腿 → FATAL 必须到 stderr);early-twin(listen-policy / issue4341)套件本地绿。完整判定以本次 push 触发的 CI 为准(此前 test lane 是被 cancel 的旧记录)。

未动:阈值、fatalMode、worker 架构、EADDRINUSE holder 诊断均保持 PR 原样。

ci-check and others added 4 commits October 29, 2026 10:13
…RINUSE (#4762)

Two facets of one failure domain: a wedged standalone proxy held its listen
socket forever while every spawner respawn died with an opaque EADDRINUSE.

1. Event-loop watchdog (src/event-loop-watchdog.ts): a worker thread owns
   the clock (its loop keeps ticking while the main thread is blocked);
   main posts a heartbeat every 5s, continuous silence >= 30s logs one
   forensic warn line, >= 300s logs FATAL and exits(1) so the spawner's
   normal respawn recovers the port. Armed at the end of startServer
   (standalone only, after markBootReady); thresholds are hard-coded
   module constants — no new config surface. The eval worker fetches its
   parent port via guarded require("worker_threads"): the parentPort global
   does not exist in eval workers (ReferenceError, verified on Node 22).

2. EADDRINUSE holder diagnostics (src/port-owner.ts + both death paths in
   src/server.ts): at the moment of death re-probe the existing listener's
   /__bili/health once (2s budget, wildcard binds probe through loopback).
   Answered -> the hint says the peer IS serving and the spawner should
   attach; no answer -> the hint names the holder as apparently wedged or
   dead with concrete release guidance (taskkill /PID N | kill N | netstat/
   lsof), logged as [eaddrinuse-diag] and to the spawn port log when
   BILI_LAUNCH_TOKEN is set. Covers the pre-engine early twin-collision
   exit and the post-bind server.on(error) terminal branch.
Review fixes on top of 22fc397 (verified locally: tsc strict clean,
node --test 5/5 watchdog + 27/27 port-owner + issue4341/issue3654/
listen-policy all green):

1. The worker rendered thresholds as Math.round(ms/1000)+"s", so the
   test's sub-second thresholds (60/150ms) printed "0s" and the
   assertions expecting "60s"/"150s" could never match — the flagship
   wedge test failed as committed. The worker now formats <1s values as
   ms; production values are all >=1s and render byte-identical to
   before. Assertions updated to the real format.
2. Report mode re-fired the FATAL line + trip postMessage on EVERY tick
   after tripping (~10 duplicated FATAL lines per test run). Added a
   per-episode latch (reset by init/beat, mirroring the existing
   warn-once discipline); exit mode is unaffected — the process dies on
   the first trip. Pinned: exactly one FATAL line per wedge episode.
3. assert.notEqual(trippedWith, undefined) does not narrow under tsc
   strict, so the following numeric compare was a typecheck error
   (TS18048). Replaced with an explicit typeof guard that fails the
   test with context.
…r off dead stderr pipes (#4762)

- arm at the ENTRY of startServer instead of after markBootReady: startup phases (twin probes, engine/storage init, hydrate) carry sync-I/O hotspots and a stderr-pipe wedge (#5012 class) can arrive before boot completes
- worker forensic lines route to stderr only when it is a TTY or no file leg exists: a blocking write into a full undrained pipe never fires an error event on Windows, so the watchdog worker writing its FATAL line into exactly that dead pipe would wedge ITSELF; structural gate, not try/catch
- unref the worker so the safety net cannot hold a clean natural exit hostage
- both-direction stderr gate pinned with pipe-captured workers; TECHNICAL-NOTES updated
@ranxianglei
ranxianglei force-pushed the 2026-10-26_event-loop-watchdog branch from 2c60a9a to 646460e Compare October 29, 2026 02:14
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

Rebased onto current master (5594c131); the branch is now four clean commits on top.

The only conflict was src/server.ts: master's #4982/#4994 work rewrote the inline listen-error body into handleListenError (async verifyEntryOwnsPort twin check + promise handler). Resolution: keep master's structure; ported the #4762 [eaddrinuse-diag] holder re-probe into handleListenError's terminal exit path so wedged zombies are still named before we die. Import union covers both sides' port-owner.js exports.

Local verdict on rebased head 646460e2: typecheck clean; event-loop-watchdog 7/7, port-owner 24/24. Full-suite verdict = CI lanes on this push.

ranxianglei and others added 3 commits October 29, 2026 10:36
…olds process exit hostage (#4762)

The main-side beat interval was ref'd, so every startServer caller kept its event loop alive forever after its last real handle closed — 362 test files drive startServer, which hung every CI test shard on this branch (all shards red while master stayed green). The review pass unref'd the worker thread but missed this timer. Retention-only change: beats still fire while the loop runs, so wedge detection is untouched; production proxies are unaffected (the HTTP server socket already holds the loop, and every exit path ends in process.exit).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

评审修复验证(646460e rebase 后)+ 一条系统性红根因已修

  1. arm 提前到 startServer 入口 — 成立:configureLogger 幂等(getStream 复用既有 live stream);createEngineContext 随后用同一路径表达式(opts.logFile ?? defaultLogFile())再配一次;startServer 全仓唯一调用方(cli.ts:725),无重复 arm;plugin 注入模式不走 startServer,宿主进程自杀约束保持。
  2. worker stderr 纪律 — 机理正确:Node 对管道的写是阻塞且永不抛 error,看门狗把 FATAL 写进死管道会先卡死自己;emitStderr = (logFile === null) || stderrTty 四象限覆盖正确,双向测试 seam 可确定性跑。
  3. worker.unref() — 正确,但漏了同一机制的另一半:主侧心跳 setInterval 仍是 ref'd——arm 之后进程永远无法自然退出(本地探针实测:不 stop 则挂死,stop 即退)。362 个测试文件调 startServer → 每个 node --test 子进程跑完挂住 → 全部 11 个 CI test shard 红而 master 全绿(master 6b298c3 实测 25 绿/5 skip)。这是本轮红的根因,不是评审三条的问题。

怎么修的:beatTimer.unref()(commit 93c9675,一行+注释)。只改 loop 保留语义:loop 在跑时心跳照常发,wedge 检测不受影响;生产 proxy 行为不变(HTTP server socket 本就持有 loop,所有出口走 process.exit)。行为变更披露:① 观测窗口从 boot-ready 后扩到整个启动期(评审意见①要求);② armed 进程不再被看门狗 timer 扣押退出(测试进程现在能正常收尾;proxy 无感)。修后探针自然退出,typecheck/build 干净,watchdog/port-owner/#4341/listen-policy/route-passthrough 定向 54/54 绿。

其他:分支在我工作期间被另一会话重写成干净线性栈(5 个 commit 直接叠在含 v0.1.209 的 master 上,与我的 merge 式同步等价且更干净),我的修复已重新应用到新栈上并补齐 master 剩余 README 变更,分支头 da6c987;此前我报的 #4982 born-red 已被上游 #5050 follow-up 修掉(断言改为钉 verdict 措辞、避开 prose 里的 "twin-collision" 字面量),本次同步一并带入。回归面:本轮改动只碰 event-loop-watchdog.ts 的定时器引用计数 + 纯 README 合并,无 wire/缓存/配置面变化。判定:评审三条实现正确、系统性缺陷已在本机制内修掉,等 da6c987 的完整 CI 定论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

rebase 已核实通过;此前 11 个 test lane 全红的根因也已定位并修复,重跑进行中。逐条汇报:

分支状态(比你评论时多了一个 commit)

11 lane 全红的根因(CI 基建登记遗漏,非代码回归、非 master 继承)

本地验证(最终 head 8ff5fe22)

  • build ✓、typecheck ✓、定向 5 套件(event-loop-watchdog / port-owner / issue4341 / issue3654 / listen-policy)48/48 全绿。

当前 CI 与后续

  • 镜像缓存显示 8ff5fe22 上 0 failed / 11 in progress(test lane 运行中,gate 已绿,e2e 按惯例 skip)。
  • 分支又 behind master(6b298c37,纯 README 改动,零代码 churn)——behind 不是 blocker,无需 rebase。
  • 两个待你拍板项不阻塞合并:① sleep/wake 误杀(wall-clock gap ≥300s 会把休眠唤醒的健康进程判死)A/B/C 选项仍等你定;② IPv6 literal bind 边界(holderProbeOrigin 对 ::1 产出无括号的非法 URL,探测必败 → 诊断行误判 wedged)一行可修,计划等本轮重跑绿后作为 follow-up 小修补上,避免现在推 commit 杀掉进行中的 run。

八问

  1. 是什么问题:PR 自身 CI 的 11 个 test lane 全红——新测试套件未登记 spawn tripwire 清单导致,外加对 rebase 结果的核实需求。
  2. 什么原因导致的:run-ci-tests.mjs 的全局 SPAWN_RE 门在分片前扫描所有测试文件,未登记的 spawn 类文件直接 die;watchdog 套件用了 new Worker( 却没进 LIVE_FILES。与 rebase 本身和 master 均无关。
  3. 影响范围与严重程度:仅本 PR 的 CI(全部 shard fail-fast);不影响其他 PR、不影响任何运行时行为。CI 阻塞级,运行时零影响。
  4. 怎么修复的:把该套件按 append-only 规则追加到 LIVE_FILES 末尾,不动既有端口 band;rebase 侧保持 master 结构、把 [bug] After event loop wedge, zombie still holds port 18789 — parent respawn loop spins on EADDRINUSE indefinitely; manu | [bug] 事件循环卡死后僵尸仍握 18789 端口,parent 重拉全部 EADDRINUSE 死循环,只能手工杀进程恢复(event loop wedge holds listen socket, respawn loop fails with EADDR… #4762 诊断移植进 terminal exit 路径。
  5. 是否有回归问题:无。既有行为逐项过:twin 健康探测语义([PR] release v0.1.209 #4982)、win32 EACCES 归类([server] Windows IPv6 wildcard port collision surfaces as EACCES and disables the listen ladder #3493)、ladder/wait-predecessor 分支、退出码与时序均不变;EADDRINUSE 回归三件套(issue4341/issue3654/listen-policy)本地全绿。
  6. 是否破坏缓存:不涉及(不触碰压缩/缓存/存储路径)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:不涉及(无新 env/config key)。

判定:代码层面无阻塞,等本轮 CI 重跑转绿即可合并;sleep/wake 策略与 IPv6 小修作为后续项跟踪。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

不是完整 review(那是本 PR 自己的审核流程),补一份事故侧证据 + 两点观察——我这边正在处理同家族的 #5139(0.1.208 dsh/Windows,wedged 子进程握端口 18788,respawn 14 次全 code 3,直连降级 22 分钟)。

对本事故的覆盖确认:覆盖。

  1. dsh 代理子进程是 standalone start(走 startServer)→ 在守卫范围内 ✓;
  2. [Bug][Codex] Temporary separator selected as compression endpoint, causing acp_status to silently fail to report ~180K t | [Bug][Codex] 临时分隔符被选作压缩端点,导致 acp_status 静默漏报约 18 万 token 可压缩历史 #5139 的 wedge 被证明是 CPU-bound 而非死锁——卡了 19 分钟后进程自行恢复并继续服务。worker 线程独立事件循环的设计恰好吃这种形态:主线程静默 ≥300s → exit(1) → 端口释放 → dsh 下一次 respawn 即接管。本事故停摆从「19 分钟靠运气 / 通常无限」缩到 ≤5 分钟 + 一次 respawn;
  3. 一个 v0.1.208 代码级事实支持本 PR 的必要性:父看门狗机制结构上覆盖不了无 owner-heartbeat 的 wedged 实例——watcherIsDead 的心跳分支要求 watcherLastSeen 有值(安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 opt-in),legacy owner 只有 kill-0 轴(父活着 = 永不判死);而且即便有心跳,watcher 的 setInterval 与 wedged 主循环共享同一事件循环,阻塞期间 tick 根本跑不到,只能在恢复后的第一个 tick 判死(那正是 dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently #4994 facet⑤ 的「恢复后自杀」形态)。[Bug][Codex] Temporary separator selected as compression endpoint, causing acp_status to silently fail to report ~180K t | [Bug][Codex] 临时分隔符被选作压缩端点,导致 acp_status 静默漏报约 18 万 token 可压缩历史 #5139 的子进程 19 分钟不退出、恢复后也不自杀,与「无心跳注册的 legacy owner」完全吻合。in-process worker 看门狗是唯一不依赖宿主协作、不依赖主循环存活的检测面——这个事故就是它的生产用例。

两点观察(非阻断):

  1. trip 路径绕过优雅 flush。worker 直接 process.exit(1),主线程正卡在同步计算里,shutdown hook/session flush 都跑不到。数据损失窗口 = trip 时刻未落盘的脏状态(debounce 保存窗口内的增量,有界;客户端自身持有历史副本、统一存储有先前快照兜底)。建议在 PR 描述里把这个有界损失写成已知取舍,免得日后有人当回归查。
  2. 「legit work never blocks the loop for five straight minutes」假设在本代 mega 会话上已被实测逼近/突破:[Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788 实测 local= 14–22s/turn 且转交后持久化再占 ~6s;[Bug][Codex] Temporary separator selected as compression endpoint, causing acp_status to silently fail to report ~180K t | [Bug][Codex] 临时分隔符被选作压缩端点,导致 acp_status 静默漏报约 18 万 token 可压缩历史 #5139/[Performance][dsh] Mega-sessions pay an ever-growing O(full-history) fold-reconcile tax every turn: expected host churn (one-shot carrier retirement, image/offload retroactive rewrites) misclassified as drift | 大会话每轮付不断增长的 O(全史) fold-reconcile 税:合法宿主变更(o… #5142 给出单轮成本逐日爬升(1.8s→25s)与本次 19 分钟连续阻塞(最终完成)。也就是说 300s 硬阈值偶尔会杀掉一个「本来能跑完」的超长合法 turn——这是可用性优先的正确取舍(杀掉换 respawn 重试,优于无限握端口),但属于行为变更(旧 = 超长 turn 可能耗时数分钟后完成;新 = 300s 处被杀、由宿主重试),按披露义务建议写进 PR 描述的 old→new 清单。根治方向(让重腿不再同步霸占主循环)在 [Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788/decision request: #4788 Fix direction for memory growth and request latency in long sessions (retention cap / envelope w | decision request: #4788 长会话内存增长与请求延迟的修复方向(保留上限 / envelope 写入默认 / 超预算驱逐策略) #4792/[Performance][dsh] Mega-sessions pay an ever-growing O(full-history) fold-reconcile tax every turn: expected host churn (one-shot carrier retirement, image/offload retroactive rewrites) misclassified as drift | 大会话每轮付不断增长的 O(全史) fold-reconcile 税:合法宿主变更(o… #5142 跟踪,与本 PR 正交。

关联:#5139(本证据来源)、#4762(原始 wedge 报告)、#4994 facet⑤ + PR #5085/决策 #5086(父看门狗心跳轴误杀,与本 PR 不同轴、可共存:#5086 选 A 后忙碌但存活 <300s 的 turn 两个机制都不会误伤)、#5012(worker stderr 路由已按该类缺陷设计,赞)。

# Conflicts:
#	scripts/live-files.mjs
#	src/server.ts
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

两条都已写进 PR 描述的新段落 “Known tradeoffs & behavior change (disclosed)”:(1) trip 路径 exit(1) 绕过优雅 flush——损失窗口有界(debounce 窗口内的脏增量;客户端自持历史 + 统一存储先前快照兜底);(2) 300s 硬阈值可能杀掉本来能跑完的超长合法 turn——old→new 行为变更已列出,根治方向在 #4788/#4792/#5142 跟踪、与本 PR 正交。另:本 PR 已 re-sync 到当前 master(live-files 注册表 union,tsc 干净),CI 在新 head 上跑。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

两条披露已核实落位,re-sync 后的新 head 也验证过了。逐条汇报:

PR 描述披露(已对照线上正文核实)

re-sync 核实(head 3c824734)

更正一条记录:11 lane 全红是两个独立机制叠加,不是单一根因

  • 我上一轮只报了 ②,补全如下:
    • ① ref'd 心跳 timer(系统性,commit 93c9675):主侧 setInterval 未 unref,arm 之后每个 startServer 调用方的事件循环永不自然退出——362 个测试文件都驱动 startServer,每个 node --test 子进程跑完挂住 → 全部 shard 超时红而 master 全绿。前一轮验证已本地实测复现(不 stop 则挂死,stop 即退;master 对照 25 绿/5 skip)。
    • ② SPAWN_RE tripwire 登记(commit 8ff5fe22,我上轮已报):tests/event-loop-watchdog.test.ts:120 的 new Worker(code, {eval:true…}) 命中全局扫描模式,未登记时每个 shard 在分片前直接 die。
  • 两者互相独立、缺一不可:只修 timer 仍会被 tripwire 秒杀,只登记仍会挂死超时。现在都在分支里。

当前 CI:3c824734 上 0 failed / 12 in progress(test lane 运行中,3 success,15 skipped,取数 14:55Z)。

遗留项状态

八问

  1. 是什么问题:确认两条行为变更披露已进 PR 描述,并核实 re-sync 后新 head 的正确性;顺带把此前「11 lane 红单根因」的报告更正为完整图景。
  2. 什么原因导致的:CI 红 = 两个独立机制叠加(ref'd 心跳 timer 让所有测试进程跑完不退 + 新增 spawn 类测试文件未过全局 tripwire 登记);两条披露本身是设计取舍而非缺陷(有界损失换可用性、300s 硬阈值换端口必回收)。
  3. 影响范围与严重程度:CI 红仅本 PR 全部 test shard,运行时零影响;披露的两条运行时变更面=所有 standalone proxy(wedge 自退路径的损失边界、超长 turn 被杀重试),均已按 old→new 列明。
  4. 怎么修复的:timer unref 为纯引用计数变更(loop 在跑时心跳照常,wedge 检测语义不变,生产 proxy 由 HTTP socket 持有 loop 不受影响);tripwire 走 append-only 登记不动既有端口 band;取舍以文档形式钉进 PR 描述,根治另行正交跟踪。
  5. 是否有回归问题:未发现。本地 48/48(EADDRINUSE/watchdog 五套件);unref 不改检测语义;对 master 净 diff 与此前审核过的完全一致;CI 全量定论看本轮 run。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(披露不转向;重腿下主循环的根治在 [Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788/decision request: #4788 Fix direction for memory growth and request latency in long sessions (retention cap / envelope w | decision request: #4788 长会话内存增长与请求延迟的修复方向(保留上限 / envelope 写入默认 / 超预算驱逐策略) #4792/[Performance][dsh] Mega-sessions pay an ever-growing O(full-history) fold-reconcile tax every turn: expected host churn (one-shot carrier retirement, image/offload retroactive rewrites) misclassified as drift | 大会话每轮付不断增长的 O(全史) fold-reconcile 税:合法宿主变更(o… #5142 正交推进)。
  8. 是否引入新配置:无。

判定:无阻塞项,等本轮 CI 转绿即可合并(合并动作归人);IPv6 一行小修等绿后跟进。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

第二条生产验证(来源: #5139 后续跟进):#5139 用户披露其双实例 split-brain 窗口的成因是外部自启看门狗脚本——按 #3654/#4762 指引杀 wedged 子进程,dsh 8s 内未 respawn 时脚本自行拉起一个 bili。自启实例不是 dsh 的子进程 → dsh 永远无法收养、持续 spawn 撞端口、双实例同时 commit 同一会话(index.db 双写告警即此场景,#3923/#3959)。

对合并决策的含义:这类「杀+自启」外部 workaround 正是本 PR 要消灭的那一类——in-process 看门狗让 wedged 实例自己让出端口,外部脚本只需要 kill-only(安全)甚至完全不需要。该用户已被告知停掉自启分支,但在他升级前(以及所有还没拿到本 PR 的用户那里),这个 workaround 类仍会持续制造 split-brain。支持尽快合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

上一头 3c82473 的主 CI(run 40895229671)最终有 7 个测试 shard 红。先说明:不是这个 PR 的代码引入的。逐 shard 读日志确认,每个红 shard 挂的都是同样 4 个 tests/claude-native.test.ts 子测试(resolveNativeAttachExternal / LanePriority / Policy / Url 的读文件分支)。根因是当时 CI 合并到的 master 还没有 cf99d93(#4599:tests/preload-test-env.mjs 不再 pin BILI_CONFIG_FILE)——那个 pin 在 src/paths.ts 的 configFile() 里优先于 XDG_CONFIG_HOME,把所有「把配置写进自己 $XDG_CONFIG_HOME 再读」的用例影子成读空文件。该修复 14:50:58Z 经 #5136 进 master;本 PR 的这轮 CI 在 14:48:51Z 触发,恰好差两分钟踩中。

另更正我上一条回复里「#4784 CI 全绿」的说法:那是运行中途读取的误报,收回,抱歉。

已把当前 master(含 cf99d93)合进来推了 1fc0605(无冲突,live-files 54 条无重复)。本地验证:tsc 干净、event-loop-watchdog 7/7、claude-native 在 CI 同款 preload 链下 57 pass / 0 fail。以新一轮 CI 为准。

@ranxianglei
ranxianglei merged commit 3417c0b into master Oct 29, 2026
55 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.913-master.3417c0b (rolling master tag).

npm install -g billion-context@0.0.913-master.3417c0b  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard "behind" notice was stale: this PR was already merged into master at 2026-10-29T15:54:52Z (merge commit 3417c0b, head 1fc0605 verified as ancestor of master) — no branch sync/push needed or performed.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

本轮 CI 红(7 条 test lane)已查清并修复,push 到 a7874c185。结论先行:7 条全部是今天 master #5107 提交(测试环境 hermetic preload)引入的继承性缺陷,与本 PR 自身改动无关——所有失败文件与 master 逐字节相同,本 PR diff 不触碰其中任何一个。

根因(两个机制)

  1. Running the test suite locally writes test sessions into the production sessions directory. | 本地跑测试套件会把测试会话写进生产 sessions 目录 #5107 新增的 tests/preload-test-env.mjs 给每个测试进程无条件设置 BILI_CONFIG_FILE 指向一个不存在的路径,而 configFile() 的解析顺序是它优先于 XDG_CONFIG_HOME。于是所有「换 XDG_CONFIG_HOME 再写配置文件」的测试在 CI 里实际读的是那个空文件:
    • claude-native attach 四件套:4 个 file 层断言失败(u22/u24/w22 各一 lane);
    • issue3436 mitm.domains:子进程记录到的 domains 为空,健康等待耗尽预算报「60s 未 healthy」——报错里的 pid 42470 其实是测试里写死的假子进程(makeFakeChild),并非真实 spawn 挂死(u22/u24/w22 各一 lane)。
      最小复现已做:本地按 CI 完整 preload 链跑 claude-native,恰好挂同样 4 个断言,与线上一致。
  2. Running the test suite locally writes test sessions into the production sessions directory. | 本地跑测试套件会把测试会话写进生产 sessions 目录 #5107 leak guard 的 snapshotSessionsTree 用 path.relative 做键,Windows 上是反斜杠,与断言的正斜杠不符(w22_2 lane,3 个断言)。此项哨兵已修好并合入分支(1fc06052d,与我起草的一行修法相同),我丢弃了自己的重复提交、保留已收敛的那份。

修复(commit 689c963e6,只动两个测试文件,零生产代码):在两个测试文件的 XDG-swap 处同步 save/delete/restore BILI_CONFIG_FILE,让它们对新的 ambient 约定保持 hermetic,#5107 的隔离语义本身不动。本地按与 CI 完全相同的 preload 链验证:三套相关 suite 104 例 0 失败,typecheck 干净。全量判定 = 本次 push 的 CI。

八问:

  1. 是什么问题:本 PR CI 红 7 条 lane,实际全是 master 侧 Running the test suite locally writes test sessions into the production sessions directory. | 本地跑测试套件会把测试会话写进生产 sessions 目录 #5107 引入的测试隔离缺陷,不是本 PR 代码问题。
  2. 什么原因:ambient BILI_CONFIG_FILE 在 configFile() 里压过 XDG_CONFIG_HOME,让一切靠 XDG 换目录写配置的测试读到空文件;另有 leak guard 快照键未做 Windows 分隔符归一。
  3. 影响范围与严重程度:master 级——凡分片含这些文件的每个在途 PR 都会同样红(ci-fixes 目前为空);仅 CI 红,无生产行为影响。
  4. 怎么修复的:受影响测试对新 ambient 变量做 save/delete/restore 保持 hermetic,不改 Running the test suite locally writes test sessions into the production sessions directory. | 本地跑测试套件会把测试会话写进生产 sessions 目录 #5107 隔离语义;Windows 分隔符项复用已合入的哨兵修复。
  5. 是否有回归:无——仅两个测试文件的环境保存/恢复,未触碰任何生产代码、配置面、wire、持久化;高危模式逐项过无语义漂移。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:不涉及。

判定:等 a7874c185 的 CI 转绿即可进入最终评审。#5107 的 ambient 变量问题会连带打红其他在途 PR,已在 #4599 留结论请哨兵向 master/ci-fixes 收敛。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant