Skip to content

[PR] fix(#5152): widen the watchdog's liveness window by its own event-loop lag, and beat from a worker - #5153

Merged
ranxianglei merged 11 commits into
ranxianglei:masterfrom
Louie0806:2026-10-29_watchdog-own-loop-blind
Oct 29, 2026
Merged

ranxianglei merged 11 commits into
ranxianglei:masterfrom
Louie0806:2026-10-29_watchdog-own-loop-blind

Conversation

@Louie0806

Copy link
Copy Markdown
Contributor

Problem

The proxy kills itself with parent-gone (pid N) while its owner is alive, then the client gives up respawning and model traffic goes direct and uncompressed — which on a large session hits the model's hard context limit (1,048,576 tokens → HTTP 400).

WATCHER_HEARTBEAT_GRACE_MS is a wall-clock window, and its reading is produced on the proxy's own event loop: the beat is recorded only when POST /__bili/watcher is handled, and the ticker that consumes that record runs on the same loop. One long synchronous turn therefore starves both at once. The recorded beat's age keeps growing regardless of what the owner is doing, and the first tick after the loop recovers removes a live pid.

#5085 confirmed the reading over consecutive ticks, but ticks are not time: a stall longer than the whole confirmation budget reproduces the same false death. This PR removes the wall-clock dependency itself rather than widening a constant.

Fix — two layers, both required

src/server.ts — measure the ticker's own blindness. The watchdog now times its own scheduling drift (tickNow - prevTickAt - WATCHER_TICK_MS) and widens the heartbeat window by that amount via effectiveHeartbeatGrace.

src/launcher.ts — beat from a worker thread. A worker_threads worker has its own event loop, so a pinned host still attests liveness. One refcounted worker per origin (every attach path calls startOwnerHeartbeat). The injected post seam keeps its historical in-thread shape, so #3544's two tests are untouched. Falls back to the in-thread loop if the worker cannot start. worker_threads is a builtin, so no new external and the dist gates are unchanged.

Verified both directions

check pre-fix post-fix
beats posted during a 6s host stall 0 20 / 20
aged reading with widened window removed: true removed: false
genuinely silent owner past the window removed still removed
kill-0 axis (#3890, 3 consecutive) works unchanged
two attach paths ⇒ workers two one, refcounted

Tests: 36 green across tests/issue5152-* (8 + 4 + 4), watchdog-health, fix-806-request-watchdog, issue3434-first-byte-watchdog, issue3890-parent-gone-confirm-drain. tsc --noEmit clean, tsup + both dist gates OK.

The pre-fix row is a real red baseline, not a modelled one: with src/launcher.ts stashed, the stall posts zero beats; with it, a receiver on a standalone thread records 20/20 while the host thread is pinned.

Related

🤖 Powered by ework · deepseek-v41-flash
(ai: lab-v41/deepseek-v41-flash)

… its own blindness

The stale-heartbeat reading is taken on the PROXY's own event loop: the beat
is recorded by POST /__bili/watcher being HANDLED, and the ticker that
consumes it runs on that same loop. One long synchronous turn therefore
starves both, the recorded beat's age grows through no fault of the owner,
and the first tick after recovery removes a LIVE pid. With the spawner giving
up after repeated drops, model traffic then goes direct and uncompressed.

Two layers, both required:

* server: measure the ticker's OWN scheduling drift (tickNow - prevTick - tick)
  and widen the heartbeat window by it. Uncapped on purpose (ranxianglei#5138 records a
  ~19 minute pause; any cap below the true length re-creates the false death),
  and not sticky — every tick re-measures, so once the loop runs again the
  window snaps back and ranxianglei#3544's anti-reuse guarantee only lags by the proxy's
  own blindness.

* launcher: schedule the beat from a worker thread, whose loop is independent
  of the host's, so a pinned host still attests liveness. One refcounted
  worker per origin; the injected `post` seam keeps its in-thread shape so
  ranxianglei#3544's tests are untouched. Falls back in-thread if the worker cannot start.

Verified both directions: pre-fix, a 6s host stall posts 0 beats while a
standalone receiver sees 20/20 from the worker; the widened window turns the
same aged reading from `removed: true` into `removed: false` while a genuinely
silent owner still dies.

Closes ranxianglei#5152.
@Louie0806

Copy link
Copy Markdown
Contributor Author

🤖 Powered by ework · deepseek-v41-flash
(ai: lab-v41/deepseek-v41-flash)

Independent reproduction of the root cause, run locally against this PR's head (4a0fd7e3) on Node 22.23.4 / Linux.

The reading and the consumer share one loop. The beat is recorded only when POST /__bili/watcher is handled; the ticker that reads it runs on the same loop. Instrumenting both axes on master:

pre-fix  (src/launcher.ts stashed), host pins its own loop 6000ms:
    beats posted during the stall = 0
    watchdog on recovery          = REMOVED, staleHeartbeat=true, age=24011ms

post-fix, receiver in a STANDALONE process, same 6000ms host pin:
    beats received during the stall = 20 / 20

The second row is the part that matters: an in-process receiver cannot serve while the thread is pinned, so it cannot distinguish "no beats sent" from "no beats handled" — the receiver has to be a separate process (or write to a file) for the measurement to mean anything. That is likely why this survived #5085's eight unit tests: they all exercise the pure function, and none of them can see the timer-before-poll ordering.

Why uncapped. #5138 records a ~19 minute full-loop block. Any cap below the true stall length re-creates exactly the false death this removes, so the widening has no ceiling. Safety comes from freshness instead: every tick re-measures, so once the loop runs again the window snaps back to the constant and #3544's anti-reuse guarantee is only delayed by the proxy's own blindness — never cancelled. effectiveHeartbeatGrace returns the historical constant for undefined, NaN, Infinity and negative lag, so garbage can only ever make the watchdog less strict.

One thing worth a second opinion. The worker is refcounted per origin, one worker for every attach path (spawn, attach, preferred fallback) — verified at 10.0 beats/s for two concurrent startOwnerHeartbeat calls (two workers would double it) and the surviving caller keeps beating after the first stop(). If a lane can legitimately want a different intervalMs for the same origin, the cache key needs to become the tuple; today every caller uses the default, so I left it keyed by origin alone rather than speculatively complicating it.

36 tests green: tests/issue5152-* (8 + 4 + 4), watchdog-health, fix-806-request-watchdog, issue3434-first-byte-watchdog, issue3890-parent-gone-confirm-drain. tsc --noEmit clean, tsup + both dist gates OK.

@ranxianglei ranxianglei changed the title fix(#5152): widen the watchdog's liveness window by its own event-loop lag, and beat from a worker [PR] fix(#5152): widen the watchdog's liveness window by its own event-loop lag, and beat from a worker Oct 29, 2026
Louie0806 and others added 2 commits October 29, 2026 22:05
…ranxianglei#1248 tripwire)

The new suite spawns the standalone beat-receiver fixture, so the ci-tests
tripwire requires a LIVE_FILES entry before the shard will run. No zone band
is involved: the fixture is listen(0) and publishes its ephemeral port through
a file, per the existing ranxianglei#3890/ranxianglei#4811 precedent. Concurrency assumptions
reviewed as the tripwire asks — one receiver and one origin per test, all
state under a per-run tmpdir, so the refcounted worker table never collides.
@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 同症状家族、机制互补于在途 PR #5085。 症状同属「看门狗把活着的 owner 判死 → 代理 parent-gone 自杀 → 车道降级直连」家族(#4994 伞下),但机制不同:#5085 是代理侧「连续 5 tick 确认才判死」(等 #5086 拍板,且对 #5138 记录的 ~19 分钟停顿仍然不够);本 PR 修的是发送侧根因(宿主主线程被钉死时 worker 线程继续发心跳)+ 代理侧自身盲区补偿(ticker 用自己的调度漂移加宽 grace)。两层互补,不是同一修法。注意存在合入顺序耦合(若 #5085 先合,本 PR 的回归钉需要 rebase 调和),终评里单列。

标签: bug / dsh / scope/wiring / p1(已打)—— 误杀活 owner 直到宿主重启前压缩能力整体停摆 = 会话不可用类 p1。

初审发现(CI): 本 PR head(4a0fd7e)的 11 个 test 分片全红,签名已核实(job 日志逐字):process-spawning test files missing from LIVE_FILES in scripts/run-ci-tests.mjs: tests/issue5152-owner-beat-survives-host-stall.test.ts —— 这是本 PR 自己的问题,不是继承红:新测试文件会 spawn 接收端子进程,但没在 run-ci-tests.mjs 的 LIVE_FILES 登记(#1248 的 pretest tripwire),全矩阵一行测试都没跑就退出。静态门禁/guards 全绿。

处置中(无决策依赖,先修): merge ci-fixes(分支落后 master 8 个提交,全是 #5107/#5115 test-infra)+ 登记 LIVE_FILES(并发假设已审:随机端口 listen(0)、唯一 tmpdir、pgid SIGKILL 清理,无固定资源)推到 PR 分支,随后给完整八问终审。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解,head abfb19675 已推送,PR 恢复可合并。

处理过程:master 比 merge-base 前进了 8 个提交(#5120 hermetic data roots、#5107 path-env scrub 等),git merge origin/master 只产生一处冲突:scripts/live-files.mjs——双方同时往 LIVE_FILES 数组尾部追加条目(本 PR 登记 tests/issue5152-owner-beat-survives-host-stall.test.ts,master 侧 #5114 登记 tests/preload-sandbox-dirs.test.ts)。解法:两条都保留,按 append-only 规则追加在数组尾(数组不变量 slot=index 驱动端口 band,尾部追加不挪动任何既有条目;两个套件都不占真实端口,条目仅为 SPAWN_RE tripwire 存在)。采用 merge 而非 rebase,外部贡献者的提交历史与作者归属原样保留。

另:按哨兵在档指令(见 #4599 floor 462/470),master@1fad21d 当前带继承红(fixture-shadowing 家族,修复在 ci-fixes@cf99d935f、载体 PR #5133 等人工合并),故预防性再 merge origin/ci-fixes 进分支——纯 test-infra 7 文件,与本 PR diff 零交集,避免白跑一轮继承红。已在 #4599 留一行协调记录。

本地验证:typecheck 干净;定向套件 issue5152×3 + watchdog-health×2 + issue3890 25/25 绿;launcher.test.ts 343 过 / 1 skip。全量判定以远程 CI 对新 head abfb19675 的结论为准(推送时上一轮 run 仍在跑,新 head 触发全矩阵重跑)。

结构化八问

  1. 是什么问题:PR 分支落后 master 8 个提交,scripts/live-files.mjs 一处文本冲突,merge state dirty。
  2. 什么原因导致的:双方各自往 LIVE_FILES 数组同一位置(尾部)追加新测试登记,git 无法自动合并同址双增。
  3. 影响范围与严重程度:仅本 PR 可合并性受阻,产品行为零变化;严重度=PR 阻塞,解完即通。
  4. 怎么修复的:merge(保贡献者历史)+ 唯一冲突按 append-only 取并集;预防性并入 ci-fixes 挡继承红。
  5. 是否有回归问题:审过既有行为——live-files.mjs 纯登记表零逻辑改动;合并后与 master 的 diff 仍恰为本 PR 自己的 7 文件;ci-fixes 带入的 7 个 test-infra 文件与本 PR diff 零交集且为哨兵已收敛的在档修复;本地 typecheck + 定向 25/25 + launcher 343 绿佐证。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无。
  8. 是否引入了新的配置:无。

判定:冲突车道收工,等 CI 全矩阵绿即可合;若新 head 再现继承签名,按政策走 #4599/ci-fixes 收敛,不在本分支重复修。

ci-check added 3 commits October 29, 2026 22:38
…d-flight

A crashed worker left its refcount slot occupied while attesting nothing,
so every later attach for the same origin rode the dead entry and the
proxy starved of beats — silently reintroducing the false death (ranxianglei#5152)
this heartbeat exists to prevent, worse than the old in-thread loop whose
fetch failures kept looping. The error handler now starts the in-thread
fallback for that origin; stop() tears down worker and fallback together.
The spawner is injectable so the crash path is pinned without a real
thread death (both directions asserted: no premature fallback, beats
continue after the crash, stop() silences the fallback).
…hdog suites

ranxianglei#4999 is the closed pi-inline streaming issue — unrelated. The source
issue for this work is ranxianglei#5152 (watchdog reads its own event loop). Fix
the header, four test names, and the stale cross-file reference.
Completes the two-direction coverage of the fallback gate: construction
itself failing (locked-down runtime) must fall back to the in-thread
loop before anything is armed and still attest.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核完成。代码层结论:通过;我已向 PR 分支直接推了 3 个独立标注的小修 commit(新 head 6d5fe6ee7),等这头 CI 全绿即可合入。

结构化八问

  1. 是什么问题:代理自己的事件循环被大回合同步卡住时(实测最坏 ~19 分钟,[Bug] 0.1.208 dsh/Windows: event loop fully blocked ~19 min after cold-load/eviction storm on 300MB+ sessions; wedged child holds port so all respawns fail (twin-collision code 3) #5138),「owner 心跳」的读取和消费都发生在这同一个循环上——卡住期间既记不到心跳到达、也跑不了判定 tick;恢复后第一个 tick 按墙钟判最后心跳已过期,把活着的父进程杀掉(parent-gone),车道降级直连,大会话再撞模型硬窗口(HTTP 400)。
  2. 什么原因导致的:存活窗口 WATCHER_HEARTBEAT_GRACE_MS=15s 是墙钟的,但它的测量点和消费点都在会卡死的循环上——「裁判睡着时不计时」这条没满足,盲区却算到被告头上。发送侧同理:心跳原本调度在宿主主线程,宿主长回合把自己也饿死(dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently #4994 facet⑤ / [Bug] Watchdog takes its stale-heartbeat reading on its own event loop, so a synchronous stall kills a live proxy (parent-gone → uncompressed 400) #5152)。tick 不是时间:[PR] fix: watchdog heartbeat branch requires consecutive confirmation before killing the proxy (#4994) #5085 的连续 tick 确认只推迟判决、不改变「读数在盲中取」的事实,任何超过确认预算的停顿复现同一误杀。
  3. 影响范围与严重程度:所有注册 owner 心跳的车道(dsh/opencode/pi/codex 等 spawn 车道);触发条件 = 事件循环停顿 >15s(大压缩/大传输场景,生产已有实证)。p1:宿主重启前能力整体不可用(会话不可用类),叠加 spawner respawn churn 放弃后的完整故障链(dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently #4994)。
  4. 怎么修复的(两层各治一边):
  5. 是否有回归问题(触碰路径既有行为逐项审过,见下表):无无意回归;两处有意变更均已披露(见表中 ⚠ 行)。高危模式逐项过:语义漂移(无——健康循环下 grace 逐位不变)、fail-fast 变吞错(无——kill-0 轴 3-strike 未动)、阈值漂移(仅 lag>0 时动态加宽,方向为放宽而非收紧)、时序(beat 节奏变化见 ⚠ 行,安全方向)、输出格式(无)。
    既有行为 结论
    tick 间隔 2s 保持(WATCHER_TICK_MS 仅提取为常量,值不变)
    健康循环下的 grace 逐位一致(lag≤0/NaN/Inf → 常数,T3 钉 no-op 方向)
    心跳轴死亡判定规则 规则未变(超窗即除,不加连续确认);⚠ 有意变更(已披露):窗口参数从固定 15s → 15s+实测自循环漂移
    kill-0 轴([Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890,3 连确认) 未动(T3 双向钉)
    安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 pid 复用防护 保持但延迟:回收 pid 恰好晚「一次代理自身盲区」死掉,永不取消(T3 钉 uncapped-not-sticky)
    idle-grace/shutdown-drain/多 owner 集合逻辑 未动
    post seam / 外部 pid(claude ppid)legacy kill-0 保持(T1 钉)
    wire 形状 {pid,hb:true} + 2s 超时 字节一致
    进程生命周期 worker unref'd,不持住进程(与旧 unref'd timer 同类);新增成本 = 每个 attach origin 一条 V8 worker 线程(小内存增量)
    服务端看到的 beat 频率(多 attach) ⚠ 有意变更(已披露):旧 N 条 attach 路径 ≈ N×beats/s,现 1 个 worker ≈ 1×beats/s;安全方向(请求减少),仍 ≪15s grace
  6. 是否破坏缓存:不涉及(纯进程监护面,不碰 wire/账本/缓存键)。
  7. 是否改了设计方向:没有转向——仍是「kill-0 证存在 + heartbeat 证身份」双轴模型;变的是发送侧作证的位置(主线程→worker)和窗口的自校准方式。
  8. 是否引入新配置:无(无 env/config/CLI/默认值;调参走源码常量 + code review,与现有 WATCHER_* 常量同类)。

测试有效性

我推的 3 个小修(PR 分支上独立标注 commit,均可单独 revert)

  1. fix(#5152) 44ca013:worker 运行中崩溃被静默吞掉(on("error") 空 handler)——死掉的 worker 还占着 refcount 槽位,之后同 origin 的 attach 全部挂在死条目上,该 origin 永久失去作证,静默重新引入本 PR 要消除的那类误杀(且比旧 in-thread loop 更差:旧代码 fetch 出错会继续循环)。现在 error 时对该 origin 启动 in-thread fallback,stop() 一并拆除;spawner 可注入,双向钉(崩前不多余回退/崩后继续作证/stop 静默 fallback)。这是同失败域的健壮性缺陷,按「小问题直接修」处置。
  2. test(#5152) 1cf0c59:T2 头部 + 4 个测试名标成了「[PR] Fix pi-inline output arriving in large batches instead of token-by-token #4999」——那是已关闭的 pi-inline streaming issue,与本 PR 无关;T3 跨文件引用指向不存在的文件名。统一改为 [Bug] Watchdog takes its stale-heartbeat reading on its own event loop, so a synchronous stall kills a live proxy (parent-gone → uncompressed 400) #5152。
  3. test(#5152) 6d5fe6e:补构造失败 fallback 门禁缺失的另一方向(spawn 抛错 ⇒ in-thread fallback 立即开始作证)。

本地定向验证(head 6d5fe6ee7 树):watchdog 相关 9 套件 58/58 + T1 扩到 6/6;typecheck clean;unused-export gate clean(新导出均被测试引用,无需 waiver);build + 两个 dist gate OK。全量套件 = 远程 CI 裁决。

CI 状态与协调

⚠️ 合入顺序耦合(需要你知晓,不阻塞审核结论)

PR #5085(proxy 侧 5-tick 连续确认,正在 #5086 等 A/B/C 拍板)与本 PR 机制互补,但若 #5085 先合,本 PR T3 的「残留洞」钉(首个 stale tick 即移除)会对 #5085 的 5-tick 语义变红,本 PR 需 rebase 对齐。建议:先合本 PR(基于 master、自包含、修发送侧根因;#5085 单独覆盖不了 #5138 那种 19 分钟级停顿),#5085/#5086 随后对齐——两者仍互补(网络丢 beat 轴上 #5085 在 lag=0 时额外提供 ~10s 容忍)。若你裁决 #5085 方案覆盖本 PR 代理侧层,告诉我结果,我按裁决对齐。

判定:代码层通过(根因打在决策点上、两层各治一边、无无意回归、测试有牙);head 6d5fe6ee7 CI 全绿后可合入,合并本身留给你。

ci-check added 2 commits October 29, 2026 22:54
ranxianglei#5085 (ranxianglei#4994 heartbeat-confirm) landed on master while this PR was in flight: both sides extended tickWatchedPid independently. Union taken: the ranxianglei#5152 lag-widened grace feeds the same stale-heartbeat gate that ranxianglei#4994 now confirms over WATCHER_HEARTBEAT_CONFIRM_TICKS consecutive ticks — complementary defenses of the same false-death class, verified composable in the pure function. The ranxianglei#5152 unit suites are adapted to the merged 6-arg seam (death assertions now complete over the confirmation run instead of a single strike; grace-axis isolation preserved).
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

第二轮冲突已解,head 807da6c8a 已推送,PR mergeable=True(behind,无冲突)。 本轮比第一轮实质:master 在两轮之间合入了 #5085(#4994 watchdog 心跳连续确认)——与本 PR 是同一机制域,两侧各自独立扩展了 tickWatchedPid,不是 append-order 那种机械冲突。

语义并集(核心判定):本 PR 的层 1 是「心跳宽限窗按 watchdog 自己观测到的事件循环滞后加宽」(effectiveHeartbeatGrace,测量侧);master 的 #4994 是「心跳分支的 stale 判读须连续 5 tick 确认才除名」(确认侧)。两者是同一假死类别(#5152/#4994 同根因:proxy 自己的 event loop 饥饿被记到 owner 头上)的两道互补防线——加宽窗减少假 stale 读数的产生,连续确认兜住漏网的。纯函数并集后签名 (probe, lastSeen, now, deadStreak, hbStreak, { heartbeatGraceMs, confirmTicks, heartbeatConfirmTicks }):计算出的 widened grace 喂进 #4994 现在守门的同一个 staleness gate,已验证可组合(union 严格比任一侧更保守,无行为回归)。#3890 kill-0 轴与 legacy pid 语义未触碰。

配套适配:两个 #5152 单测按合并后的 6 参 seam 改写——死亡断言从「单次 strike」改为「确认跑完(5 tick)后完成」,存活断言额外钉住 widened 窗口内 hbStreak 不 arm(grace 轴隔离保持)。贡献者在我工作期间向分支推了 3 个新提交(worker 中途崩溃时回退 in-thread 继续 beat 的修复 + issue 引用勘误 + fallback 方向钉),已一并合入、历史原样保留。

本地验证:typecheck 干净;watchdog 全族 35/35(issue5152×3、issue4994、issue3890、health×2);launcher.test.ts 343+1 过 / 1 skip。推送后 master 又进了 #5110/#5042,但 GitHub 实时计算无冲突(state=behind),不再追——车道 2/2 次用尽,若后续真出冲突请再触发。无需决策请求 issue:并集是机械且已验证的,没有设计取舍。

结构化八问

  1. 是什么问题:首轮解冲突后 master 又合入 [PR] fix: watchdog heartbeat branch requires consecutive confirmation before killing the proxy (#4994) #5085(watchdog 心跳连续确认)与 [PR] fix(#5012): trust the log file over dead pipes — mute the stderr tee once the file leg proves live #5089,二次 dirty。
  2. 什么原因导致的:两侧对同一函数 tickWatchedPid 做了互不知情的独立扩展(声明处 + 调用处文本冲突),外加 LIVE_FILES 同址双追加。
  3. 影响范围与严重程度:仅本 PR 可合并性;语义上两侧修复互补,并集只更保守不更激进,产品行为零回归;严重度=PR 阻塞,解完即通。
  4. 怎么修复的:取语义并集而非任选一侧——widened grace 流入 dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently #4994 的连续确认门;单测按合并 seam 适配(死亡断言走完整确认跑);顺带干净吸收贡献者的 3 个在途提交。
  5. 是否有回归问题:逐项审过——kill-0 轴([Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890)不变、legacy pid 语义不变、anti-pid-reuse(安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544)保留(真实死亡仍会终止,只是按 dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently #4994 自身分析慢 ~8s)、worker-beat 面在贡献者新修复下 crash-fallback 双向断言过;本地 35/35 + launcher 344 绿佐证。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无——watchdog 仍是 kill-0 / heartbeat 双轴连续确认设计,本 PR 给 heartbeat 轴增加一个测量输入,不新增轴。
  8. 是否引入了新的配置:无。

判定:冲突车道收工(2/2 用完),等远程 CI 对新 head 807da6c8a 的全矩阵结论;若再现继承红,按政策走 #4599/ci-fixes 收敛(该修复已在树内)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 自动修复车道(attempt 1/2)结论:两处根因都已在分支上消除并本地验证通过,本车道无需再推任何提交。

先说清编号:镜像 #5154 对应 GitHub PR #5153(head 分支在 Louie0806 fork)。红的是 run 40886295229(head 4a0fd7e)的 11 个 test 分片。

八问

  1. 是什么问题:两个叠加的 CI 门禁红——① 本 PR 新增的进程派生测试套件没登记进 scripts/live-files.mjs 的 LIVE_FILES,ci-tests 预检 tripwire(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248)在每个分片跑测试之前就 exit 1,11 片全红、零真实测试执行(三个平台 job 日志逐字同一报错行);② master 自身在 1fad21de8([PR] fix: keep test suite off the production sessions dir (#5107) #5120 hermetic data-roots 集群)上是红的(fixture 配置影子遮蔽 + Windows leak-guard relpath),PR 的 merge ref 会继承。
  2. 什么原因导致的:① tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248 纪律要求每个 spawn 型套件登记 LIVE_FILES,新套件漏登记 → SPAWN_RE 扫描命中未登记文件 → 所有分片 pretest 即死(与 fix(ci): every test shard is red — tests/issue3890-parent-gone-confirm-drain.test.ts spawns processes but is missing from LIVE_FILES #4604/[PR] fix: [instances] warning judges by sessions directory, not lane alone (#5046) #5075 同模式);② 继承红是 master 面既有缺陷,哨兵修复收敛在常驻分支 ci-fixes,PR 侧按协议合入即可。
  3. 影响范围与严重程度:仅本 PR 的 CI 门禁(全分片死于预检,不是真实测试失败);用户功能零影响。属流程级红门,非功能缺陷。
  4. 怎么修复的:① LIVE_FILES append-only 登记 tests/issue5152-owner-beat-survives-host-stall.test.ts(commit b218d03c9,已做 tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248 要求的并发审查:fixture 是 listen(0) + port 文件上报,不占 zone band,每测试独立 receiver/tmpdir);② 把 origin/ci-fixes@cf99d935f 合入 PR 分支吸收哨兵修复(commit abfb19675)。两者都是独立小提交,无语义变更。
  5. 是否有回归问题:无。登记为纯追加——合并后的 live-files.mjs 与 master 逐字节相同外加一个追加块,现有条目 slot(index→zone 端口带)零漂移;ci-fixes 合并只动测试基建 7 文件,不碰 src 语义。本地验证:tripwire 复扫干净(997 文件)、tsc --noEmit 干净、定向 6 套件 27/27 绿(issue5152×3 / watchdog-health×2 / [Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。另注:作者车道在我验证之后又自行推进了三个提交(worker 中途死亡时回退到线程内 beat 循环等改进,44ca013c9 起)+ 一次 master 重同步(807da6c8a,带入刚合入 master 的 [PR] fix: watchdog heartbeat branch requires consecutive confirmation before killing the proxy (#4994) #5085/[PR] fix(#5012): trust the log file over dead pipes — mute the stderr tee once the file leg proves live #5089)——那是作者的正常迭代,不属于本次 CI 修复范围;我在 6d5fe6ee7(含作者改进的一代)上也做了同样的本地验证,全绿。
  8. 是否引入新配置:没有。

处置说明:诊断完成后发现修复已由并行车道的提交先行落在分支上(LIVE_FILES 登记 + ci-fixes 合并),我本地独立复现了同样修法并确认与远端 head tree 逐字节相同,故不再推送重复提交;#4599 的协调行已由冲突车道在档(floor 480),不重复留痕。当前 head 807da6c8a 的 CI 在跑,截至 15:02Z 零失败;若仍现继承签名,由下一轮 auto-fix(attempt 2/2)接手。本车道收工。

Louie0806 and others added 2 commits October 29, 2026 23:16
…U budget

The ranxianglei#5152 contract is "the beat does not stop while the SENDER's own loop is
pinned". Pre-fix that is exactly ZERO additional beats across the block, so any
sustained trickle is already decisive; the old bar demanded ~90% of the
theoretical cadence, which measures the runner instead of the property. Under
`--test-concurrency 4` the beat worker competes with other suites, and a healthy
fix can miss that bar on a loaded shard — observed once locally while a second
node process (a timed-out suite moved to the background) was still running,
turning 20/20 delivered beats into 17/20 and failing the suite.

Poll for a floor instead: at least half the theoretical beats, with a bounded
10s window, so a slow-but-working worker passes and a genuinely silent one still
fails. The pre-fix direction is unchanged (0 beats < floor).
@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Merge-guard sync: merged origin/master into 2026-10-29_watchdog-own-loop-blind (clean, no conflicts) and pushed — head is now af84b3c, 0 commits behind master; CI re-running on the new head (static gates / mysql-contract / docs-coverage green, test shards in progress, 0 failures).

@ranxianglei
ranxianglei merged commit b2d69f5 into ranxianglei:master Oct 29, 2026
55 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.914-master.b2d69f5 (rolling master tag).

npm install -g billion-context@0.0.914-master.b2d69f5  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Oct 30, 2026
…ng cannot read 0

Inherited CI flake (the fixture is on master, from ranxianglei#5153):
  test (ubuntu-latest, 22, 1)
  not ok - ranxianglei#5152 a host where the worker cannot start keeps beating from the
  in-thread loop
  error: 'receiver port must be a number, got 0'
  at startReceiver (tests/issue5152-owner-beat-survives-host-stall.test.ts:62)

appendFileSync() opens (and creates) the file as one syscall and writes the
bytes as another, so startReceiver()'s existsSync() poll can lose the race
against the content. Proven with the syscall order split out: existsSync=true,
readFileSync="" , Number("") === 0 — the exact assertion message on CI.

Publish through a temp file + rename() instead: rename is atomic within a
filesystem, so every observer sees either no file or the complete port. The
natural window is narrow enough that 180 stress spawns did not hit it locally;
this removes the window rather than a reproduced failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants