Repository navigation
[PR] fix(#5152): widen the watchdog's liveness window by its own event-loop lag, and beat from a worker - #5153
Conversation
… its own blindness The stale-heartbeat reading is taken on the PROXY's own event loop: the beat is recorded by POST /__bili/watcher being HANDLED, and the ticker that consumes it runs on that same loop. One long synchronous turn therefore starves both, the recorded beat's age grows through no fault of the owner, and the first tick after recovery removes a LIVE pid. With the spawner giving up after repeated drops, model traffic then goes direct and uncompressed. Two layers, both required: * server: measure the ticker's OWN scheduling drift (tickNow - prevTick - tick) and widen the heartbeat window by it. Uncapped on purpose (ranxianglei#5138 records a ~19 minute pause; any cap below the true length re-creates the false death), and not sticky — every tick re-measures, so once the loop runs again the window snaps back and ranxianglei#3544's anti-reuse guarantee only lags by the proxy's own blindness. * launcher: schedule the beat from a worker thread, whose loop is independent of the host's, so a pinned host still attests liveness. One refcounted worker per origin; the injected `post` seam keeps its in-thread shape so ranxianglei#3544's tests are untouched. Falls back in-thread if the worker cannot start. Verified both directions: pre-fix, a 6s host stall posts 0 beats while a standalone receiver sees 20/20 from the worker; the widened window turns the same aged reading from `removed: true` into `removed: false` while a genuinely silent owner still dies. Closes ranxianglei#5152.
Independent reproduction of the root cause, run locally against this PR's head ( The reading and the consumer share one loop. The beat is recorded only when The second row is the part that matters: an in-process receiver cannot serve while the thread is pinned, so it cannot distinguish "no beats sent" from "no beats handled" — the receiver has to be a separate process (or write to a file) for the measurement to mean anything. That is likely why this survived #5085's eight unit tests: they all exercise the pure function, and none of them can see the timer-before-poll ordering. Why uncapped. #5138 records a ~19 minute full-loop block. Any cap below the true stall length re-creates exactly the false death this removes, so the widening has no ceiling. Safety comes from freshness instead: every tick re-measures, so once the loop runs again the window snaps back to the constant and #3544's anti-reuse guarantee is only delayed by the proxy's own blindness — never cancelled. One thing worth a second opinion. The worker is refcounted per 36 tests green: |
…ranxianglei#1248 tripwire) The new suite spawns the standalone beat-receiver fixture, so the ci-tests tripwire requires a LIVE_FILES entry before the shard will run. No zone band is involved: the fixture is listen(0) and publishes its ephemeral port through a file, per the existing ranxianglei#3890/ranxianglei#4811 precedent. Concurrency assumptions reviewed as the tripwire asks — one receiver and one origin per test, all state under a per-run tmpdir, so the refcounted worker table never collides.
# Conflicts: # scripts/live-files.mjs
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 —— 同症状家族、机制互补于在途 PR #5085。 症状同属「看门狗把活着的 owner 判死 → 代理 parent-gone 自杀 → 车道降级直连」家族(#4994 伞下),但机制不同:#5085 是代理侧「连续 5 tick 确认才判死」(等 #5086 拍板,且对 #5138 记录的 ~19 分钟停顿仍然不够);本 PR 修的是发送侧根因(宿主主线程被钉死时 worker 线程继续发心跳)+ 代理侧自身盲区补偿(ticker 用自己的调度漂移加宽 grace)。两层互补,不是同一修法。注意存在合入顺序耦合(若 #5085 先合,本 PR 的回归钉需要 rebase 调和),终评里单列。 标签: bug / dsh / scope/wiring / p1(已打)—— 误杀活 owner 直到宿主重启前压缩能力整体停摆 = 会话不可用类 p1。 初审发现(CI): 本 PR head(4a0fd7e)的 11 个 test 分片全红,签名已核实(job 日志逐字): 处置中(无决策依赖,先修): merge ci-fixes(分支落后 master 8 个提交,全是 #5107/#5115 test-infra)+ 登记 LIVE_FILES(并发假设已审:随机端口 listen(0)、唯一 tmpdir、pgid SIGKILL 清理,无固定资源)推到 PR 分支,随后给完整八问终审。 |
(ai: vllm-qwen/qwen3.8-27b) 冲突已解,head 处理过程:master 比 merge-base 前进了 8 个提交(#5120 hermetic data roots、#5107 path-env scrub 等), 另:按哨兵在档指令(见 #4599 floor 462/470),master@1fad21d 当前带继承红(fixture-shadowing 家族,修复在 ci-fixes@cf99d935f、载体 PR #5133 等人工合并),故预防性再 merge 本地验证:typecheck 干净;定向套件 issue5152×3 + watchdog-health×2 + issue3890 25/25 绿;launcher.test.ts 343 过 / 1 skip。全量判定以远程 CI 对新 head 结构化八问
判定:冲突车道收工,等 CI 全矩阵绿即可合;若新 head 再现继承签名,按政策走 #4599/ci-fixes 收敛,不在本分支重复修。 |
…d-flight A crashed worker left its refcount slot occupied while attesting nothing, so every later attach for the same origin rode the dead entry and the proxy starved of beats — silently reintroducing the false death (ranxianglei#5152) this heartbeat exists to prevent, worse than the old in-thread loop whose fetch failures kept looping. The error handler now starts the in-thread fallback for that origin; stop() tears down worker and fallback together. The spawner is injectable so the crash path is pinned without a real thread death (both directions asserted: no premature fallback, beats continue after the crash, stop() silences the fallback).
…hdog suites ranxianglei#4999 is the closed pi-inline streaming issue — unrelated. The source issue for this work is ranxianglei#5152 (watchdog reads its own event loop). Fix the header, four test names, and the stale cross-file reference.
Completes the two-direction coverage of the fallback gate: construction itself failing (locked-down runtime) must fall back to the in-thread loop before anything is armed and still attest.
(ai: vllm-qwen/qwen3.8-27b) 审核完成。代码层结论:通过;我已向 PR 分支直接推了 3 个独立标注的小修 commit(新 head 结构化八问
测试有效性
我推的 3 个小修(PR 分支上独立标注 commit,均可单独 revert)
本地定向验证(head CI 状态与协调
|
ranxianglei#5085 (ranxianglei#4994 heartbeat-confirm) landed on master while this PR was in flight: both sides extended tickWatchedPid independently. Union taken: the ranxianglei#5152 lag-widened grace feeds the same stale-heartbeat gate that ranxianglei#4994 now confirms over WATCHER_HEARTBEAT_CONFIRM_TICKS consecutive ticks — complementary defenses of the same false-death class, verified composable in the pure function. The ranxianglei#5152 unit suites are adapted to the merged 6-arg seam (death assertions now complete over the confirmation run instead of a single strike; grace-axis isolation preserved).
…-blind' into pr5153-conflict-fix
(ai: vllm-qwen/qwen3.8-27b) 第二轮冲突已解,head 语义并集(核心判定):本 PR 的层 1 是「心跳宽限窗按 watchdog 自己观测到的事件循环滞后加宽」( 配套适配:两个 #5152 单测按合并后的 6 参 seam 改写——死亡断言从「单次 strike」改为「确认跑完(5 tick)后完成」,存活断言额外钉住 widened 窗口内 hbStreak 不 arm(grace 轴隔离保持)。贡献者在我工作期间向分支推了 3 个新提交(worker 中途崩溃时回退 in-thread 继续 beat 的修复 + issue 引用勘误 + fallback 方向钉),已一并合入、历史原样保留。 本地验证:typecheck 干净;watchdog 全族 35/35(issue5152×3、issue4994、issue3890、health×2);launcher.test.ts 343+1 过 / 1 skip。推送后 master 又进了 #5110/#5042,但 GitHub 实时计算无冲突(state=behind),不再追——车道 2/2 次用尽,若后续真出冲突请再触发。无需决策请求 issue:并集是机械且已验证的,没有设计取舍。 结构化八问
判定:冲突车道收工(2/2 用完),等远程 CI 对新 head |
(ai: vllm-qwen/qwen3.8-27b) CI 自动修复车道(attempt 1/2)结论:两处根因都已在分支上消除并本地验证通过,本车道无需再推任何提交。 先说清编号:镜像 #5154 对应 GitHub PR #5153(head 分支在 Louie0806 fork)。红的是 run 40886295229(head 4a0fd7e)的 11 个 test 分片。 八问
处置说明:诊断完成后发现修复已由并行车道的提交先行落在分支上(LIVE_FILES 登记 + ci-fixes 合并),我本地独立复现了同样修法并确认与远端 head tree 逐字节相同,故不再推送重复提交;#4599 的协调行已由冲突车道在档(floor 480),不重复留痕。当前 head |
…U budget The ranxianglei#5152 contract is "the beat does not stop while the SENDER's own loop is pinned". Pre-fix that is exactly ZERO additional beats across the block, so any sustained trickle is already decisive; the old bar demanded ~90% of the theoretical cadence, which measures the runner instead of the property. Under `--test-concurrency 4` the beat worker competes with other suites, and a healthy fix can miss that bar on a loaded shard — observed once locally while a second node process (a timed-out suite moved to the background) was still running, turning 20/20 delivered beats into 17/20 and failing the suite. Poll for a floor instead: at least half the theoretical beats, with a bounded 10s window, so a slow-but-working worker passes and a genuinely silent one still fails. The pre-fix direction is unchanged (0 beats < floor).
|
(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync: merged origin/master into |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.914-master.b2d69f5 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…ng cannot read 0 Inherited CI flake (the fixture is on master, from ranxianglei#5153): test (ubuntu-latest, 22, 1) not ok - ranxianglei#5152 a host where the worker cannot start keeps beating from the in-thread loop error: 'receiver port must be a number, got 0' at startReceiver (tests/issue5152-owner-beat-survives-host-stall.test.ts:62) appendFileSync() opens (and creates) the file as one syscall and writes the bytes as another, so startReceiver()'s existsSync() poll can lose the race against the content. Proven with the syscall order split out: existsSync=true, readFileSync="" , Number("") === 0 — the exact assertion message on CI. Publish through a temp file + rename() instead: rename is atomic within a filesystem, so every observer sees either no file or the complete port. The natural window is narrow enough that 180 stress spawns did not hit it locally; this removes the window rather than a reproduced failure.
Problem
The proxy kills itself with
parent-gone (pid N)while its owner is alive, then the client gives up respawning and model traffic goes direct and uncompressed — which on a large session hits the model's hard context limit (1,048,576 tokens → HTTP 400).WATCHER_HEARTBEAT_GRACE_MSis a wall-clock window, and its reading is produced on the proxy's own event loop: the beat is recorded only whenPOST /__bili/watcheris handled, and the ticker that consumes that record runs on the same loop. One long synchronous turn therefore starves both at once. The recorded beat's age keeps growing regardless of what the owner is doing, and the first tick after the loop recovers removes a live pid.#5085 confirmed the reading over consecutive ticks, but ticks are not time: a stall longer than the whole confirmation budget reproduces the same false death. This PR removes the wall-clock dependency itself rather than widening a constant.
Fix — two layers, both required
src/server.ts— measure the ticker's own blindness. The watchdog now times its own scheduling drift (tickNow - prevTickAt - WATCHER_TICK_MS) and widens the heartbeat window by that amount viaeffectiveHeartbeatGrace.src/launcher.ts— beat from a worker thread. Aworker_threadsworker has its own event loop, so a pinned host still attests liveness. One refcounted worker per origin (every attach path callsstartOwnerHeartbeat). The injectedpostseam keeps its historical in-thread shape, so #3544's two tests are untouched. Falls back to the in-thread loop if the worker cannot start.worker_threadsis a builtin, so no new external and the dist gates are unchanged.Verified both directions
removed: trueremoved: falseTests: 36 green across
tests/issue5152-*(8 + 4 + 4),watchdog-health,fix-806-request-watchdog,issue3434-first-byte-watchdog,issue3890-parent-gone-confirm-drain.tsc --noEmitclean,tsup+ both dist gates OK.The pre-fix row is a real red baseline, not a modelled one: with
src/launcher.tsstashed, the stall posts zero beats; with it, a receiver on a standalone thread records 20/20 while the host thread is pinned.Related
masterso it does not stack on it.