diff --git a/docs/technical-notes.md b/docs/technical-notes.md index c3ce68259d..7ef9082f52 100644 --- a/docs/technical-notes.md +++ b/docs/technical-notes.md @@ -603,3 +603,15 @@ Declared trust boundary (§7.4 — human review required, no auto-merge): a clie Honest limits: (1) once a multimodal tail (content array containing `image_url`) lands in the stash, the lane degrades to full legacy turns for the remainder of that history — the kernel renders user content-array images differently when the message is the LAST element versus mid-history, so a frozen stash cannot reproduce legacy bytes position-independently; text workloads (the #5403 reporter's shape) are unaffected. (2) The head fragment is re-parsed (8 elements) each hit solely for the system-run check. (3) Both `x-bili-pf-*` headers are stripped upstream (`UPSTREAM_HOP_HEADERS`) — protocol headers never leak into model traffic. Measured (provenance: `tests/issue5403-watermark-fastpath.test.ts`, hermetic dual-proxy A/B on the same corpus): hit turns skip the full parse/conversion/pin entirely; resident `lastMessages` 8684 B → 1210 B (7.2×) on an 8-turn corpus including a multipart image landing turn; outbound bytes byte-identical to legacy modulo kernel `time="…"` attributes across all turns; `x-bili-pf-*` absent from every captured upstream request, both phases. Unit coverage: `tests/fastpath.test.ts` (full verdict ladder incl. breaker TTL re-arm), `tests/fastpath-scanner.test.ts` (adversarial JSON table, depth cap, surrogate pairs, 50k-element scale), `tests/lane-decisions-matrix.test.ts` (gate row matrix, both directions). + +## Off-loop owner-death reap — the second watchdog axis (#5449) + +The #4762 watchdog covers continuous wedges ≥ 5 min, but the #5449 incident (dsh/Windows) exposed the middle regime: the DSH host died while its proxy's event loop was wedged with sparse yields (1–7 log lines/min). The in-loop parent watcher (2 s ticks) can only detect and reap when the loop yields, and the ONLY path that removes a dead spawner from `proxyWatchers` is that same starved reaper — so the orphan held the `index.db` write lock and the preferred port for 33 min at ~51% of one core, the replacement drifted 18789→18790, and callers pinned to 18789 stalled 826 s with 0 bytes. + +Design: the #4762 worker thread gains a second, independent axis. The worker probes the spawner's owner pid (`BILI_PARENT_PID`, passed as `parentPid`) off-loop every tick via `process.kill(pid, 0)` — EPERM counts as alive (#3890 discipline), and a reused pid reads alive, so the probe can miss a reap but never false-trips. It trips only when ALL THREE hold: dead streak ≥ `WATCHDOG_OWNER_DEAD_CONFIRM_TICKS` (3), main loop silent ≥ `WATCHDOG_SOFT_WARN_MS` (30 s), and no other owners attached (the count is sampled per beat). The loop-silence gate guarantees no preemption: a still-yielding loop reaches the in-loop reaper's graceful path (~11 s) first. The count is `otherOwnerCount()` — the in-loop watcher set minus the probed pid, whose removal depends on the starved reaper (without the exclusion the guard freezes at ≥1 and the axis degenerates back into the in-loop path). A wedged loop freezes the last known count — sound, because attaching a new owner during a wedge needs an HTTP answer. + +Net effect: orphan hold under a sparse-yield wedge converges from minutes to ~15–45 s after owner death. The FATAL line carries the last probe's errno (on Windows the errno is the only clue left to attribute false-dead episodes, #3890). + +Behavior change (disclosed): OLD = a wedged proxy after owner death was reaped only by in-loop ticks (bounded by yield frequency — minutes in the incident) or #4762's full-wedge exit (≥ 5 min). NEW = additionally, the worker exits the process ~15–45 s after confirming owner death ×3 ticks + loop silence ≥ 30 s + zero other watchers. Unchanged: manual `bili start` (no `BILI_PARENT_PID` → axis disabled, byte-identical); healthy/slow-yielding loops (graceful shutdown still wins); multi-owner shared proxies (last-owner guard); all wire/config/persistence surfaces (none touched). Out of scope: #5449 Part 2 (callers stranded on the drifted port) is tracked separately (#5834). + +Tests: `tests/issue5449-offloop-owner-reap.test.ts` (pure-predicate boundary tables for `ownerReapVerdict` and `otherOwnerCount` + real-worker trip/no-trip matrix: dead owner + wedge + empty set trips with reason `parent-gone`; dead spawner still in the set trips through the production counter; another live owner attached never trips; live owner never trips; a healthy beating loop never trips (no preemption); the beat-carried count update, not the init snapshot, unblocks the guard; axis disabled without `parentPid`). diff --git a/docs/technical-notes.zh-CN.md b/docs/technical-notes.zh-CN.md index f051d78c66..b07464643d 100644 --- a/docs/technical-notes.zh-CN.md +++ b/docs/technical-notes.zh-CN.md @@ -180,3 +180,15 @@ chat wire 上 bili 自己 vs 客户端自带:插件车道上,请求体里等于 诚实边界:(1) 一旦多模态尾部(含 `image_url` 的 content array)进入缓存,该段历史此后整段降级回 legacy 全量轮——kernel 按位置区分 user 图片消息在尾部与在历史中部的渲染,冻结缓存无法做到位置无关地复现 legacy 字节;纯文本工作负载(#5403 报告者的形态)不受影响。(2) 每次命中会现解析头部 8 个元素,只为做 system-run 校验。(3) 两个 `x-bili-pf-*` 头都在转发上游前剥掉(`UPSTREAM_HOP_HEADERS`)——协议头绝不漏进模型流量。 实测(出处:`tests/issue5403-watermark-fastpath.test.ts`,同语料 hermetic 双代理 A/B):命中轮完全跳过全量解析/转换/钉存;含一个 multipart 图片落位回合的 8 回合语料上,常驻 `lastMessages` 8684 B → 1210 B(7.2×);全程出站字节与 legacy 逐字一致(仅 kernel `time="…"` 属性归一);两个阶段捕获到的上游请求里均无 `x-bili-pf-*`。单元覆盖:`tests/fastpath.test.ts`(verdict 阶梯全表,含熔断 TTL 解除后重播种)、`tests/fastpath-scanner.test.ts`(对抗 JSON 表、深度上限、surrogate 对、5 万元素规模)、`tests/lane-decisions-matrix.test.ts`(门矩阵,双向)。 + +## 离环 owner 死亡回收 —— 看门狗第二轴(#5449) + +#4762 看门狗覆盖 ≥5 分钟的连续卡死,但 #5449 事故(dsh/Windows)暴露了中间地带:DSH 宿主在代理事件循环卡死且让出稀疏(日志密度 1–7 行/分钟)时死亡。环内父进程监视器(2 秒 tick)只有循环让出时才能检测/回收,而把死掉的 spawner 从 `proxyWatchers` 里移除的唯一路径正是这个被饿死的回收器——于是孤儿进程以 ~51% 单核 CPU 占住 `index.db` 写锁与首选端口长达 33 分钟,接替实例漂移到 18790,钉在 18789 上的调用方 826 秒零字节。 + +设计:#4762 的 worker 线程增加第二条独立轴。worker 在自己的事件循环上每 tick 用 `process.kill(pid, 0)` 探测 spawner 的 owner pid(`BILI_PARENT_PID`,以 `parentPid` 传入)——EPERM 记为存活(#3890 纪律),被复用的 pid 读作存活,所以探测可能漏回收但绝不会误杀。仅当三者同时成立才触发:死亡连续 ≥ `WATCHDOG_OWNER_DEAD_CONFIRM_TICKS`(3 次)、主循环静默 ≥ `WATCHDOG_SOFT_WARN_MS`(30 秒)、无其他 owner 附着(计数随每次心跳采样)。循环静默门保证不越权:仍在让出的循环会先走到环内回收器的优雅路径(~11 秒)。计数为 `otherOwnerCount()`——环内监视集减去被探测 pid(它的移除依赖被饿死的回收器;不排除则守卫冻结在 ≥1,该轴退化回环内路径)。卡死的循环冻结最后已知计数——合理,因为卡死期间附着新 owner 需要 HTTP 应答。 + +净效果:稀疏让出卡死下的孤儿占用,从数分钟收敛到 owner 死亡后 ~15–45 秒。FATAL 行携带最后一次探测的 errno(Windows 上 errno 是归因假死 episode 的唯一线索,#3890)。 + +行为变更(已披露):旧 = owner 死亡后卡死的代理只由环内 tick 回收(受让出频率限制——事故中为分钟级)或 #4762 全卡死退出(≥5 分钟)。新 = 额外地,worker 在确认 owner 死亡 ×3 tick + 循环静默 ≥30 秒 + 零其他 watcher 后 ~15–45 秒退出进程。不变:手动 `bili start`(无 `BILI_PARENT_PID` → 轴禁用,逐字节一致);健康/慢让出循环(优雅关闭仍获胜);多 owner 共享代理(最后 owner 守卫);全部 wire/config/持久化面(未触碰)。范围外:#5449 Part 2(漂移端口上的滞留调用方)另行跟踪(#5834)。 + +测试:`tests/issue5449-offloop-owner-reap.test.ts`(`ownerReapVerdict` 与 `otherOwnerCount` 纯谓词边界表 + 真实 worker 触发/不触发矩阵:死 owner+卡死+空集以 reason `parent-gone` 触发;死 spawner 仍在集合中经生产计数器触发;另有活 owner 附着绝不触发;活 owner 绝不触发;健康心跳循环绝不触发(不越权);计数随心跳更新而非 init 快照;无 `parentPid` 时轴禁用)。 diff --git a/reference/architecture.md b/reference/architecture.md index 728248834d..1436f48f72 100644 --- a/reference/architecture.md +++ b/reference/architecture.md @@ -50,7 +50,7 @@ billion-context/ │ ├── model-endpoints.ts # Declarable custom-wire model endpoints: pattern parse + longest-prefix match, one source feeds proxy classifier + client claim (#1295) │ ├── instance.ts # Live-instance registry record in the proxy-origin file (#394/#403/#417) │ ├── port-owner.ts # Post-bind port-ownership verification + pointer-clobber guard (#2941) + EADDRINUSE holder diagnostics (#4762) -│ ├── event-loop-watchdog.ts # Worker-thread event-loop wedge detector: self-exit after 5 min of continuous main-loop silence (#4762) +│ ├── event-loop-watchdog.ts # Worker-thread event-loop wedge detector: self-exit after 5 min of continuous main-loop silence (#4762) + off-loop owner-death reap when the loop is wedged (#5449) │ ├── paths.ts # XDG paths (config/cache/state) + relocation env vars │ ├── logger.ts # Tee logger: state-dir file + stderr │ ├── log-retention.ts # Private-log generations: 48h target under a hard byte budget, 0600, honest coverage diff --git a/scripts/live-files.mjs b/scripts/live-files.mjs index 9252275e85..d1bda95715 100644 --- a/scripts/live-files.mjs +++ b/scripts/live-files.mjs @@ -161,6 +161,10 @@ export const LIVE_FILES = [ // mkdtemp-isolated per test, so the zone band is irrelevant; the entry is // for the tripwire, like #5046/#4612. "tests/storage-backup-repair.test.ts", + // #5449: spawns one bare `node -e` child purely as a live/dead PID fixture + // (like #5046/B's) plus in-process eval workers (like #4762's); binds no + // ports — the entry is for the SPAWN_RE tripwire alone + "tests/issue5449-offloop-owner-reap.test.ts", ]; /** Band width per live file: the zone ladder drifts +1 per EADDRINUSE retry; diff --git a/src/event-loop-watchdog.ts b/src/event-loop-watchdog.ts index fc0aa1f289..1f3f3de495 100644 --- a/src/event-loop-watchdog.ts +++ b/src/event-loop-watchdog.ts @@ -24,6 +24,18 @@ * seconds to minutes and yield between requests — so the hard threshold is * deadlock detection, not load detection. * + * Second axis (#5449): the same worker probes the SPAWNER'S owner pid off-loop + * (`process.kill(pid, 0)` — EPERM counts as alive, #3890 discipline). When the + * owner has failed its probe N ticks running, the main loop has been silent ≥ + * SOFT_WARN_MS (a still-yielding loop reaches its OWN graceful shutdown path + * first — this axis only acts where the in-loop reaper cannot), and no other + * owner is attached (#7 shared-proxy semantics: die when the LAST owner + * exits), the worker exits the process so the spawner respawns onto the freed + * port. This closes the orphan window the in-loop watcher cannot close: under + * sparse event-loop yields the in-loop reaper's detect→decide chain takes + * minutes, while the incident orphan held its port and DB write lock for 33. + * A reused pid reads alive, so the probe can miss a reap but never false-trips. + * * Scope discipline: plugin-injection mode (dist/agent/*) never calls * startServer, so a wedged HOST process is never at risk of being killed by * this guardrail. Thresholds are deliberately hardcoded — a new config/env @@ -41,6 +53,12 @@ * wrote its FATAL line into exactly that dead pipe would wedge ITSELF instead * of tripping the exit. try/catch cannot rescue a synchronous blocking write, * so the condition has to be structural, not defensive. + * + * CI rerun trigger (2026-11-07, no behavior change): the only red on this + * branch was the documented msgid-lane-flow settle-race load flake (#4599 + * floor-903 family) — a single stale read right after the #5155 ack/flush + * decoupling under CI load; not caused by this diff. This note re-triggers + * the matrix. */ import { Worker, type WorkerOptions } from "node:worker_threads"; @@ -48,8 +66,13 @@ export const WATCHDOG_HEARTBEAT_MS = 5_000; const WATCHDOG_CHECK_INTERVAL_MS = 5_000; export const WATCHDOG_SOFT_WARN_MS = 30_000; export const WATCHDOG_HARD_TIMEOUT_MS = 5 * 60_000; +/** Consecutive failed kill-0 probes of the owner before the off-loop axis may + * act (#5449). Mirrors the in-loop watcher's confirm-tick posture (#3890): + * Windows probe errors are transient, one bad read must not kill the proxy. */ +export const WATCHDOG_OWNER_DEAD_CONFIRM_TICKS = 3; type WatchdogVerdict = "ok" | "warn" | "trip"; +type WatchdogTripReason = "loop-wedge" | "parent-gone"; /** Pure gap→verdict mapping — the worker's tick logic, unit-testable without * spawning anything. `warned` prevents re-warning on every tick. */ @@ -59,6 +82,29 @@ export function watchdogVerdict(gapMs: number, warned: boolean, softWarnMs: numb return "ok"; } +/** Pure predicate for the off-loop owner-reap axis (#5449): true only when ALL + * three hold — the owner failed its kill-0 probe `confirmTicks` times running, + * the main loop has been silent ≥ softWarnMs (a still-yielding loop reaches + * its own graceful shutdown first, so this axis never preempts it), and no + * other owner is attached (#7: die when the LAST owner exits). */ +export function ownerReapVerdict(deadStreak: number, confirmTicks: number, gapMs: number, softWarnMs: number, watcherCount: number): boolean { + return deadStreak >= confirmTicks && gapMs >= softWarnMs && watcherCount <= 0; +} + +/** The production count of OTHER attached owners for the off-loop reap guard + * (#5449/#7): the in-loop watcher set minus the probed parentPid, whose + * removal from the set depends on the in-loop reaper this axis backstops — + * without the exclusion the guard freezes at ≥1 and the off-loop reap + * degenerates back into the starved in-loop path (the incident shape). + * undefined = pre-context window, where no socket exists yet so no attach + * could have landed: 0 is truth there, not a fallback. */ +export function otherOwnerCount(watchers: ReadonlySet | undefined, probedPid: number | null): number { + if (!watchers) return 0; + let n = watchers.size; + if (probedPid !== null && watchers.has(probedPid)) n -= 1; + return n; +} + interface EventLoopWatchdogHandle { /** Stop heartbeating and terminate the worker. Idempotent. */ stop(): void; @@ -81,8 +127,16 @@ interface ArmEventLoopWatchdogOptions { fatalMode?: "exit" | "report"; /** Test seam: swap the Worker constructor. */ spawnWorker?: (code: string, opts: WorkerOptions) => Worker; - /** Called (report mode) when the worker trips. */ - onTrip?: (gapMs: number) => void; + /** Owner pid to probe off-loop (#5449): BILI_PARENT_PID on spawned proxies. + * undefined/0 = axis disabled (manual `bili start`) — zero behavior change. */ + parentPid?: number | null; + /** Number of OTHER attached owners — excluding the probed parentPid, whose + * removal from the in-loop set depends on the reaper this axis backstops + * (#5449) — sampled per beat so a wedged loop freezes the last known value + * (#7: reap only when the LAST owner is gone). */ + getWatcherCount?: () => number; + /** Called when the worker trips (report mode). */ + onTrip?: (gapMs: number, reason?: WatchdogTripReason) => void; /** Called once when the worker fails to spawn or dies early. */ onSpawnFailure?: (err: unknown) => void; /** Test seam: pin the TTY verdict so both directions of the stderr gate @@ -99,14 +153,19 @@ interface WatchdogInitMsg { checkIntervalMs: number; fatalMode: "exit" | "report"; stderrTty: boolean; + parentPid: number; + parentConfirmTicks: number; + watcherCount: number; } interface WatchdogBeatMsg { type: "beat"; + count: number; } type WatchdogToWorker = WatchdogInitMsg | WatchdogBeatMsg; interface WatchdogTripMsg { type: "trip"; gapMs: number; + reason?: WatchdogTripReason; } // The worker runs as inline code (eval): dist/index.js is a SINGLE bundled @@ -125,6 +184,12 @@ const WORKER_CODE = [ "let lastBeat = Date.now();", "let warned = false;", "let tripped = false;", + "let parentPid = 0;", + "let parentConfirmTicks = 3;", + "let parentDeadStreak = 0;", + "let parentTripped = false;", + "let watcherCount = 0;", + "let lastOwnerProbeErr = \"\";", "let timer = null;", "let emitStderr = false;", "", @@ -138,6 +203,16 @@ const WORKER_CODE = [ " return ms >= 1000 ? Math.round(ms / 1000) + \"s\" : ms + \"ms\";", "}", "", + // kill-0 liveness probe, #3890 discipline: EPERM means the pid EXISTS but + // belongs to someone else — alive. Only ESRCH (and friends) mean gone. + // lastOwnerProbeErr keeps the errno of the latest probe for the FATAL line: + // on Windows every non-ACCESS_DENIED OpenProcess failure collapses to + // ESRCH, so the code is the only clue left to attribute false-dead + // episodes (#3890 field instrumentation, mirrored from the in-loop reaper). + "function ownerAlive() {", + " try { process.kill(parentPid, 0); lastOwnerProbeErr = \"\"; return true; } catch (e) { lastOwnerProbeErr = e != null && e.code ? e.code : String(e); return e !== null && e.code === \"EPERM\"; }", + "}", + "", "function line(level, msg) {", " const row = new Date().toISOString() + \" [\" + level + \"][v=\" + version + \"] \" + msg + \"\\n\";", " try { if (logFile !== null) require(\"fs\").appendFileSync(logFile, row); } catch (e) {}", @@ -158,9 +233,21 @@ const WORKER_CODE = [ " } else if (v === \"trip\" && !tripped) {", " tripped = true;", " line(\"error\", \"[event-loop-watchdog] FATAL: main event loop silent for \" + fmt(gap) + \" (hard threshold \" + fmt(hardTimeoutMs) + \") — exiting so the spawner can respawn onto the freed port (#4762)\");", - " try { parentPort.postMessage({ type: \"trip\", gapMs: gap }); } catch (e) {}", + " try { parentPort.postMessage({ type: \"trip\", gapMs: gap, reason: \"loop-wedge\" }); } catch (e) {}", " if (fatalMode === \"exit\") killWholeProcess();", " }", + // Owner axis (#5449), independent of the loop-gap verdict above: fires only + // when ALL of dead-streak / loop-silence / no-other-watchers hold, so a + // still-yielding loop always reaches its own graceful shutdown first. + " if (!parentTripped && parentPid > 0) {", + " if (ownerAlive()) { parentDeadStreak = 0; } else { parentDeadStreak++; }", + " if (parentDeadStreak >= parentConfirmTicks && gap >= softWarnMs && watcherCount <= 0) {", + " parentTripped = true;", + " line(\"error\", \"[event-loop-watchdog] FATAL: owner pid \" + parentPid + \" gone (kill-0 x\" + parentConfirmTicks + \" off-loop, last probe code=\" + (lastOwnerProbeErr || \"?\") + \") with loop silent for \" + fmt(gap) + \" and no other watchers — exiting so the spawner can respawn onto the freed port (#5449)\");", + " try { parentPort.postMessage({ type: \"trip\", gapMs: gap, reason: \"parent-gone\" }); } catch (e) {}", + " if (fatalMode === \"exit\") killWholeProcess();", + " }", + " }", "}", "", "if (parentPort !== null) parentPort.on(\"message\", (m) => {", @@ -175,11 +262,19 @@ const WORKER_CODE = [ " lastBeat = Date.now();", " warned = false;", " tripped = false;", + " parentPid = Number.isInteger(m.parentPid) && m.parentPid > 0 ? m.parentPid : 0;", + " parentConfirmTicks = Number.isInteger(m.parentConfirmTicks) && m.parentConfirmTicks > 0 ? m.parentConfirmTicks : 3;", + " parentDeadStreak = 0;", + " parentTripped = false;", + " watcherCount = Number.isInteger(m.watcherCount) && m.watcherCount >= 0 ? m.watcherCount : 0;", " if (timer === null) timer = setInterval(tick, Math.max(10, Number(m.checkIntervalMs) || 5000));", " } else if (m.type === \"beat\") {", " lastBeat = Date.now();", " warned = false;", " tripped = false;", + // count rides every beat; a wedged loop freezes the last known value — + // sound, because attaching a new owner during a wedge needs an HTTP answer. + " watcherCount = Number.isInteger(m.count) && m.count >= 0 ? m.count : watcherCount;", " }", "});", ].join("\n"); @@ -190,6 +285,17 @@ export function armEventLoopWatchdog(opts: ArmEventLoopWatchdogOptions = {}): Ev const hardTimeoutMs = opts.hardTimeoutMs ?? WATCHDOG_HARD_TIMEOUT_MS; const checkIntervalMs = opts.checkIntervalMs ?? WATCHDOG_CHECK_INTERVAL_MS; const fatalMode: "exit" | "report" = opts.fatalMode ?? "exit"; + // #5449 axis off-switch: non-positive / non-integer pids disable the owner + // probe entirely (manual `bili start` passes nothing — zero behavior change). + const parentPid = typeof opts.parentPid === "number" && Number.isInteger(opts.parentPid) && opts.parentPid > 0 ? opts.parentPid : 0; + const readWatcherCount = (): number => { + try { + const n = opts.getWatcherCount?.(); + return typeof n === "number" && Number.isFinite(n) && n >= 0 ? n : 0; + } catch { + return 0; + } + }; let worker: Worker; try { @@ -204,7 +310,9 @@ export function armEventLoopWatchdog(opts: ArmEventLoopWatchdogOptions = {}): Ev const beatTimer = setInterval(() => { if (stopped) return; try { - worker.postMessage({ type: "beat" } satisfies WatchdogToWorker); + // count rides every beat so the worker's last-owner guard sees live + // watcher-set changes even between trips (#7 semantics, #5449). + worker.postMessage({ type: "beat", count: readWatcherCount() } satisfies WatchdogToWorker); } catch { // worker already gone — nothing left to heartbeat to. } @@ -217,7 +325,7 @@ export function armEventLoopWatchdog(opts: ArmEventLoopWatchdogOptions = {}): Ev worker.on("message", (msg: unknown) => { const m = msg as Partial; - if (m && m.type === "trip" && typeof m.gapMs === "number") opts.onTrip?.(m.gapMs); + if (m && m.type === "trip" && typeof m.gapMs === "number") opts.onTrip?.(m.gapMs, m.reason); }); // An unhandled 'error' on a Worker THROWS — swallow it: a dead watchdog // worker degrades to "unwatched", never crashes the proxy. @@ -237,6 +345,9 @@ export function armEventLoopWatchdog(opts: ArmEventLoopWatchdogOptions = {}): Ev hardTimeoutMs, checkIntervalMs, fatalMode, + parentPid, + parentConfirmTicks: WATCHDOG_OWNER_DEAD_CONFIRM_TICKS, + watcherCount: readWatcherCount(), stderrTty: opts.stderrTty ?? process.stderr.isTTY === true, } satisfies WatchdogInitMsg); } catch (err) { diff --git a/src/server.ts b/src/server.ts index 3fb954b5ff..1837d29292 100644 --- a/src/server.ts +++ b/src/server.ts @@ -103,7 +103,7 @@ import { atomicWriteInstanceFile, clearProxyInstanceFile, classifySameLanePeer, import { createServedMarkerWriter, peerStillContests, processStartMs, reapStaleWriterMarkers, writerMarkerDir, type ServedMarkerWriter } from "./served-markers.js"; import { recordSplitBrainContest, registerSplitBrainEvidenceCheck } from "./storage/runtime-status.js"; import { holderProbeOrigin, mayClobberPointer, probeHolderForHint, probeOwnerInfo, resolvePortClaim, verifyEntryOwnsPort } from "./port-owner.js"; -import { armEventLoopWatchdog } from "./event-loop-watchdog.js"; +import { armEventLoopWatchdog, otherOwnerCount } from "./event-loop-watchdog.js"; import { tcpConnectable } from "./liveness.js"; import { hoistTrappedToolItems } from "./tool-pair-order.js"; import { computeAnthropicMessageMarks, stampAnthropicSystemCacheControl, anthropicToolsCarryCacheControl } from "./loop/cache-control.js"; @@ -509,6 +509,14 @@ export interface EngineContext { adminCtx: Parameters[2]; } +// #7: BILI_PARENT_PID is seeded per-process by the spawning hook. Resolve it +// once so both creation sites (the engine context and the early watchdog +// arming below it) agree on what "the spawning owner" means for this process. +function resolveSpawnOwnerPid(): number | null { + const p = Number.parseInt(process.env.BILI_PARENT_PID ?? "", 10); + return Number.isInteger(p) && p > 0 && p !== process.pid ? p : null; +} + export async function createEngineContext(opts: ProxyOptions): Promise { // Resolve the private-log retention policy BEFORE the logger opens a file: // this is the acyclic startup boundary (config.ts + knobs.ts import logger.ts, @@ -563,8 +571,7 @@ export async function createEngineContext(opts: ProxyOptions): Promise 0 && parentWatchPid !== process.pid ? parentWatchPid : null; + const initialWatcherPid = resolveSpawnOwnerPid(); const proxyWatchers = new Set(); if (initialWatcherPid !== null) proxyWatchers.add(initialWatcherPid); // #3544: last-heartbeat timestamps for watchers that opted in (hb:true on @@ -703,9 +710,32 @@ export async function startServer(opts: ProxyOptions): Promise { // plugin-injection mode never calls startServer, so the host process can // still never trip the self-exit. const watchLogFile = configureLogger(opts.logFile ?? defaultLogFile()); + // #5449: arming precedes engine-context creation (#4762 early-arm window), + // where proxyWatchers does not exist yet — resolve the owner pid locally + // and hand the worker a ref populated as soon as the context lands. + // Pre-listen attaches are impossible (no socket yet), so reading 0 during + // that window is sound: a dead spawner mid-startup IS an orphan. + const offloopOwnerPid = resolveSpawnOwnerPid(); + let liveWatchers: Set | undefined; armEventLoopWatchdog({ logFile: watchLogFile, version: VERSION, + // #5449 off-loop owner-reap axis: the worker probes this pid from its own + // event loop, so an orphaned proxy whose loop no longer yields regularly + // is reaped ~15-45s after the owner dies instead of holding port + DB + // write lock for minutes. Manual `bili start` (no BILI_PARENT_PID) gets + // parentPid 0 — axis disabled, zero behavior change. + parentPid: offloopOwnerPid, + // #7 shared-proxy semantics ride on every beat: the worker only reaps + // when NO OTHER owner is attached, so a multi-host proxy dies when the + // LAST host exits. The probed spawner itself is excluded from the count: + // its death is exactly what the probe measures, and removing it from the + // set needs the in-loop reaper — starved by the very wedge this axis + // exists for (#5449). Without the exclusion the guard freezes at ≥1 and + // the off-loop reap degenerates back into the in-loop path. A wedged + // loop freezes the last known value — sound, because attaching during a + // wedge needs an HTTP answer. + getWatcherCount: () => otherOwnerCount(liveWatchers, offloopOwnerPid), onSpawnFailure: (err) => logMsg(opts, "warn", `[event-loop-watchdog] could not arm (${String(err)}) — continuing unwatched (#4762)`), }); @@ -765,6 +795,7 @@ export async function startServer(opts: ProxyOptions): Promise { const engine = await createEngineContext(opts); const { core, config, log, instanceId, instanceStartedAt, proxyWatchers, watcherLastSeen, initialWatcherPid, adminCtx } = engine; + liveWatchers = proxyWatchers; const dispatch = async (req: http.IncomingMessage, res: http.ServerResponse): Promise => { armRequestWatchdog(req, res, log); diff --git a/tests/issue5449-offloop-owner-reap.test.ts b/tests/issue5449-offloop-owner-reap.test.ts new file mode 100644 index 0000000000..af5c3cb4a3 --- /dev/null +++ b/tests/issue5449-offloop-owner-reap.test.ts @@ -0,0 +1,267 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { + WATCHDOG_OWNER_DEAD_CONFIRM_TICKS, + armEventLoopWatchdog, + otherOwnerCount, + ownerReapVerdict, +} from "../src/event-loop-watchdog.ts"; +import { isPidAlive } from "../src/instance.ts"; + +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); + +/** Synchronous spin — the exact shape of a wedged event loop (#4762): no + * heartbeats leave the main thread while the worker's clock keeps running. */ +function blockMainLoop(ms: number): void { + const end = Date.now() + ms; + while (Date.now() < end) { /* spin */ } +} + +async function waitUntil(fn: () => boolean, ms: number): Promise { + const end = Date.now() + ms; + while (!fn() && Date.now() < end) await sleep(10); +} + +/** Spawn a short-lived child and wait until its pid genuinely exits, so + * kill-0 on it reads ESRCH before the watchdog arms (#5449 precondition). */ +async function spawnEphemeralOwner(): Promise { + const child = spawn(process.execPath, ["-e", "setTimeout(() => {}, 20)"], { stdio: "ignore" }); + if (child.pid === undefined) throw new Error("spawn did not yield a pid"); + const pid = child.pid; + const deadline = Date.now() + 2_000; + while (isPidAlive(pid) && Date.now() < deadline) await sleep(10); + assert.ok(!isPidAlive(pid), "precondition: ephemeral owner must be dead before arming"); + return pid; +} + +test("ownerReapVerdict: all three gates must hold together (#5449)", () => { + assert.equal(WATCHDOG_OWNER_DEAD_CONFIRM_TICKS, 3); + // streak below confirm threshold — even after a long wedge + assert.equal(ownerReapVerdict(2, 3, 60_000, 30_000, 0), false); + // gap below the soft-wedge gate — owner dead, but the loop still yields + assert.equal(ownerReapVerdict(3, 3, 29_999, 30_000, 0), false); + // other watcher attached — #7 shared-proxy semantics die with the LAST owner + assert.equal(ownerReapVerdict(3, 3, 60_000, 30_000, 1), false); + // all three hold at the boundaries + assert.equal(ownerReapVerdict(3, 3, 30_000, 30_000, 0), true); + // over-streak also trips + assert.equal(ownerReapVerdict(7, 3, 60_000, 30_000, 0), true); +}); + +test("otherOwnerCount: the production counter excludes only the probed pid (#5449/#7)", () => { + const spawner = 4242; + // pre-context window: no set yet, no socket, no attach possible — 0 is truth + assert.equal(otherOwnerCount(undefined, spawner), 0); + // incident shape: the dead spawner is still in the set — it must not count + assert.equal(otherOwnerCount(new Set([spawner]), spawner), 0); + // one other live host attached alongside the spawner + assert.equal(otherOwnerCount(new Set([spawner, process.pid]), spawner), 1); + // probed pid absent from the set — raw size stands + assert.equal(otherOwnerCount(new Set([process.pid]), spawner), 1); + // axis disabled (manual bili start): nothing to exclude + assert.equal(otherOwnerCount(new Set([process.pid]), null), 1); + assert.equal(otherOwnerCount(new Set(), null), 0); +}); + +test("dead owner + wedged loop + no other watchers trips off-loop reap (#5449)", async () => { + const ownerPid = await spawnEphemeralOwner(); + let trippedWith: { gapMs: number, reason?: string } | undefined; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + parentPid: ownerPid, + getWatcherCount: () => 0, + fatalMode: "report", + onTrip: (gapMs, reason) => { trippedWith = { gapMs, reason }; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(100); // ≥3 tick intervals: dead streak accumulates while beats flow + blockMainLoop(400); // wedge: no more heartbeats reach the worker + const deadline = Date.now() + 3_000; + while (!trippedWith && Date.now() < deadline) await sleep(20); + assert.ok(trippedWith, "onTrip must fire once the loop wedges with a confirmed-dead owner"); + assert.equal(trippedWith.reason, "parent-gone"); + assert.ok(trippedWith.gapMs >= 60, `gap ${trippedWith.gapMs} must clear the soft-wedge gate`); + } finally { + handle?.stop(); + await sleep(20); + } +}); + +test("dead spawner still in the watcher SET trips — the guard counts OTHER owners only (#5449/#7)", async () => { + // Production wiring shape (src/server.ts): proxyWatchers is seeded with the + // spawner pid at creation and only the in-loop reaper removes it — under a + // wedge the set still holds the dead owner. The count handler is the + // production otherOwnerCount itself, or this axis could never fire for a + // single-spawner orphan (the incident shape). + const ownerPid = await spawnEphemeralOwner(); + const watchers = new Set([ownerPid]); + let trippedWith: { gapMs: number, reason?: string } | undefined; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + parentPid: ownerPid, + getWatcherCount: () => otherOwnerCount(watchers, ownerPid), + fatalMode: "report", + onTrip: (gapMs, reason) => { trippedWith = { gapMs, reason }; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(100); + blockMainLoop(400); + const deadline = Date.now() + 3_000; + while (!trippedWith && Date.now() < deadline) await sleep(20); + assert.ok(trippedWith, "trip must not wait for the in-loop reaper to remove the dead spawner from the set"); + assert.equal(trippedWith.reason, "parent-gone"); + } finally { + handle?.stop(); + await sleep(20); + } +}); + +test("dead owner + wedge but ANOTHER watcher attached never trips (#5449/#7 last-owner guard)", async () => { + const ownerPid = await spawnEphemeralOwner(); + // Production shape: the set holds the dead spawner plus one other live host. + const watchers = new Set([ownerPid, process.pid]); + let trips = 0; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + parentPid: ownerPid, + getWatcherCount: () => otherOwnerCount(watchers, ownerPid), + fatalMode: "report", + onTrip: () => { trips += 1; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(100); + blockMainLoop(400); + await sleep(80); + assert.equal(trips, 0, "a multi-host proxy must survive the loss of one host"); + } finally { + handle?.stop(); + await sleep(20); + } +}); + +test("live owner + wedge never trips the owner axis (#5449)", async () => { + let trips = 0; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + parentPid: process.pid, + getWatcherCount: () => 0, + fatalMode: "report", + onTrip: () => { trips += 1; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(100); + blockMainLoop(400); + await sleep(80); + assert.equal(trips, 0, "kill-0 on our own pid always reads alive"); + } finally { + handle?.stop(); + await sleep(20); + } +}); + +test("dead owner but a HEALTHY loop never trips — the in-loop reaper keeps its graceful path (#5449 no-regression)", async () => { + const ownerPid = await spawnEphemeralOwner(); + let trips = 0; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + parentPid: ownerPid, + getWatcherCount: () => 0, + fatalMode: "report", + onTrip: () => { trips += 1; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(400); // beats keep flowing: gap stays tiny, gate cannot open + assert.equal(trips, 0, "worker must not preempt the in-loop reaper's detect→decide→drain path"); + } finally { + handle?.stop(); + await sleep(20); + } +}); + +test("watcher count rides every beat — init-time count must not freeze the guard (#5449/#7)", async () => { + const ownerPid = await spawnEphemeralOwner(); + let watchers = 1; + let trippedWith: { gapMs: number, reason?: string } | undefined; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + parentPid: ownerPid, + getWatcherCount: () => watchers, + fatalMode: "report", + onTrip: (gapMs, reason) => { trippedWith = { gapMs, reason }; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(100); // init + several beats carry count=1 + watchers = 0; // the last owner exits + await sleep(45); // ≥2 beat intervals — the worker's frozen count MUST now be 0 + blockMainLoop(400); + const deadline = Date.now() + 3_000; + while (!trippedWith && Date.now() < deadline) await sleep(20); + assert.ok(trippedWith, "trip requires the beat-carried count update, not only the init snapshot"); + assert.equal(trippedWith.reason, "parent-gone"); + } finally { + handle?.stop(); + await sleep(20); + } +}); + +test("no parentPid — axis disabled, zero behavior change for manual bili start (#5449)", async () => { + let trips = 0; + const handle = armEventLoopWatchdog({ + logFile: null, + version: "0.0.0-test", + heartbeatMs: 20, + checkIntervalMs: 20, + softWarnMs: 60, + hardTimeoutMs: 30_000, + getWatcherCount: () => 0, + fatalMode: "report", + onTrip: () => { trips += 1; }, + }); + try { + assert.ok(handle, "handle expected"); + await sleep(100); + blockMainLoop(400); + await sleep(80); + assert.equal(trips, 0, "without BILI_PARENT_PID the worker has nothing to probe"); + } finally { + handle?.stop(); + await sleep(20); + } +});