Skip to content

[PR] fix: watchdog heartbeat branch requires consecutive confirmation before killing the proxy (#4994) - #5085

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-28_watchdog-heartbeat-confirm
Oct 29, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-28_watchdog-heartbeat-confirm

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Part of #4994 (the Linux heartbeat-branch parent-gone suicides reported in floor 6). Closes stays with #5010, which owns the launcher-side facets of the same issue.

What & why

#3890 gave the kill-0 axis consecutive confirmation (3 failed probes), but the heartbeat branch of tickWatchedPid stayed single-strike: ONE stale-while-alive reading removed the watched pid. That reading is taken on the proxy's OWN event loop — a long synchronous turn (big compression / big forward) starves the /__bili/watcher handler while the parent lives on, watcherLastSeen ages past the 15s grace, and the first post-stall tick removed the LIVE parent, so the proxy shut itself down mid-turn. Field evidence (#4994 floor 6, Linux where kill-0 never lies): 16 parent-gone events in 24h against a live dsh-web-main pid; measured event-loop pause 2m39s.

Fix

  • The stale-while-alive branch now requires WATCHER_HEARTBEAT_CONFIRM_TICKS consecutive ticks (new constant = 5) before removing. 5 rather than the kill-0 shape's 3 because the window must span one full 5s beat interval plus 2s tick alignment: worst case, the first stale tick is at t0 and the next live beat lands t0+5s, processed t0+6s — a 3-tick confirmation would still false-kill at t0+4s. A fresh re-beat disarms the streak immediately.
  • Axes stay independent: a dead-probe tick breaks the heartbeat run and vice versa (episodes never accumulate across interruptions); legacy registrations with no heartbeat record keep kill-0-only semantics.
  • Both removal branches now log a warn with attribution (which branch killed, beat age / last probe errno) — previously the [stale-heartbeat] suffix only survived to shutdown when the death tick also emptied the set, so most heart-death episodes were invisible.

Behavior change (old → new)

old new
heartbeat-branch removal (pid alive, beats stopped) immediate at 15s grace 15s + ~8s (5 x 2s ticks) ≈ 23s from last beat
total exit after owner loss via heartbeat ≈20s (15s grace + 5s idle grace) ≈28s
recycled-pid anti-reuse guarantee (#3544) holds holds, delayed ~8s
kill-0 axis (#3890) 3-tick confirm unchanged
legacy registrations (no heartbeat record) kill-0 only unchanged

The delay only costs time in the genuinely-dead case; it removes the false death in the busy-but-alive case. No wire/storage/config-surface changes.

Tests

  • New tests/issue4994-watchdog-heartbeat-confirm.test.ts (8 rows): single stale reading does NOT remove (fails pre-fix = repro of the bug), <5 ticks never remove, 5th removes tagged stale-heartbeat, re-beat mid-episode disarms, 5s-interval worst case fits 5 ticks not 3, legacy kill-0-only semantics, independent axes, recycled pid still dies within bounded ticks.
  • Verified against pre-fix code: 8/8 fail on origin/master's tickWatchedPid, 8/8 pass on this branch.
  • tests/issue3890-parent-gone-confirm-drain.test.ts: call sites updated to the new signature; the "heartbeat expiry stays immediate" row rewritten to pin the NEW contract (that row encoded exactly the behavior this PR changes).
  • Targeted local run: 35/35 pass across the pure-function tests of the three affected suites (child-process lanes left to CI); typecheck clean.

Decision needed

This changes documented watchdog semantics ("heartbeat expiry stays immediate — #3544"). Implemented option A (consecutive confirmation, N=5); options B (raise base grace 15s→60s) and C (logs only) are laid out in decision-request issue #5086 (p1), with per-option impact/regression analysis.

…4994)

The stale-while-alive branch of tickWatchedPid removed the watched pid on a
single reading taken across the proxy's OWN event loop: a long synchronous
turn starves the /__bili/watcher handler while the parent lives on,
watcherLastSeen ages past the 15s grace, and the first post-stall tick shut
the proxy down mid-turn (16 parent-gone events in 24h on Linux, #4994 floor
6). The branch now requires WATCHER_HEARTBEAT_CONFIRM_TICKS (5) consecutive
stale ticks — enough to span one full 5s beat interval plus tick alignment so
a live parent always gets one delivery after its own stall ends; a fresh
re-beat disarms the streak. The kill-0 axis (#3890, 3 ticks) and legacy
kill-0-only semantics are unchanged; a truly recycled pid (#3544) still
exits, ~8s later. Both removal branches now log their attribution.
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-28_watchdog-heartbeat-confirm (b0ac4d5)

Latest build — install this exact version

npm install -g billion-context@0.0.6717-pr.5085.dc92b3f

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
b0ac4d5 billion-context@0.0.6717-pr.5085.dc92b3f build
f0214e3 billion-context@0.0.6692-pr.5085.726d416 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5085.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: watchdog heartbeat branch requires consecutive confirmation before killing the proxy (#4994) [PR] fix: watchdog heartbeat branch requires consecutive confirmation before killing the proxy (#4994) Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 是 #4994 的 facet ⑤(heartbeat 分支连续确认),与同 issue 的其他 facet 症状同源(parent-gone 自杀)但机制不同:#3890 修的是 kill-0 轴误报(Windows)、#5010 管 spawner 超时路径、本 PR 修 Linux 上「事件循环饿死 → 心跳超时」这条独立的单发移除路径。按机制关键词检索 open+closed issue/PR,无其他在途修复同一分支的条目。关联:#5086(本 PR 语义变更 A/B/C 的 p1 决策请求)。

标签: bug / dsh / scope/wiring / p1(代理回合中自杀 = 可用性缺陷;现场证据来自 dsh 车道;watchdog 属服务端进程监督面;与 #4994 既有标签一致)

审核开始,初步结论(完整八问待本地验证后另帖):

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 29, 2026
The watcher loop wrote decision.hbStreak back into hbStreaks only inside the
probe.alive branch, so a kill-0-failing tick left the map untouched and the
next alive-stale tick resumed the run from where it had been — letting a
flapping probe (Windows OpenProcess) accumulate stale ticks across the gap
and confirm death without WATCHER_HEARTBEAT_CONFIRM_TICKS CONSECUTIVE
alive-stale readings. The pure function already reports hbStreak 0 for a
disagreeing tick; clear the map here too so the live loop honors the same
"episodes never accumulate across interruptions" contract the unit test pins
(mirrors how the alive branch clears deadStreak). Linux is unaffected
(kill-0 never flaps); this tightens the Windows-only cross-axis path.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

进度汇报(进程曾被强制重启一次,现已恢复,无阻塞):

  • 方向关/基线/测试已核完:新增 8 行 pre-fix 全红、post-fix 全绿(本地定向跑 8/8);issue3890 套件 7/7 绿(含 15s 子进程 drain 行)。
  • 审核独立发现一处 caller 侧记账缺口:watcher 循环的 dead-probe 分支不清 hbStreaks,纯函数返回的 hbStreak=0 没被持久化 —— kill-0 抖动穿插在 stale ticks 之间时,心跳连续确认会退化成「累计计数」(模拟验证:4 stale + 1 dead + 1 stale 在第 6 tick 就移除,而契约要求 5 个连续)。
  • 我本地已备好修复 commit,推送时发现你已独立推出 b0ac4d5 修了同一个缺口(同机制、同位置,另多一行 info 日志)——收敛发现。我的重复 commit 弃掉,改为验证你的版本:正在跑定向套件 + 查 CI。
  • 完整八问结论紧随其后。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR 审核结论(独立审核,非复读 issue/PR 描述)

八问:

  1. 是什么问题:Linux 上代理自己的事件循环在长回合(大压缩/大转发)里停顿时,看门狗的心跳分支把「忙但活着」的父进程误判成死了,代理回合中途自杀。现场证据:dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently #4994 floor 6,24h 内 16 次 parent-gone 打在活着的 dsh-web-main pid 上,实测事件循环停顿 2m39s。
  2. 什么原因导致的:心跳分支的死亡判决是单发的——一次「pid 还活着但心跳超过 15s」的读数就立即移除该 pid。而这个读数取自代理自己的事件循环:长同步回合饿死 /__bili/watcher handler,父进程活着、watcherLastSeen 却老过宽限,停顿后的第一个 tick 就误杀。[Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890 给 kill-0 轴加了连续确认,这条分支当时没加——本 PR 补的就是这个缺口(修的是判决点本身的确认纪律,不是遮症状)。
  3. 影响范围与严重程度:p1 可用性缺陷。任何「心跳停 >15s 但进程还活着」的宿主形态(长同步回合、GC 风暴、宿主侧卡顿)都会丢代理,会话压缩能力整体中断直到宿主重启。现场为 Linux dsh 车道;Windows 心跳轴同样暴露(kill-0 抖动是另一条轴的事)。
  4. 怎么修复的:心跳轴加连续确认(WATCHER_HEARTBEAT_CONFIRM_TICKS=5 × 2s tick ≈ 8s,叠加 15s 基础宽限);新心跳一拍即解除连续计数;两轴互相独立、互不累积;两个移除分支都补了带归因的 warn 日志(此前 [stale-heartbeat] 后缀只在「致死 tick 恰好清空集合」时可见)。N=5 的算术我核过:最坏情形首个 stale tick 在 t0、下一拍落在 t0+5s 于 t0+6s 被处理——3 tick 会在 t0+4s 误杀,5 tick 在 t0+8s 才移除,晚于解除时刻。✓
    审核中发现并闭环的一处缺口:纯函数对「disagreeing tick」返回 hbStreak=0,但 watcher 循环的 dead-probe 分支没把这个复位写回 map——kill-0 抖动穿插在 stale ticks 之间时,连续确认退化成累计计数。我用模拟钉死:4 stale + 1 dead + 1 stale 在第 6 tick 就移除(只有 4 个连续),契约要求 5 个连续。你已独立推出 b0ac4d5 修了同一个缺口(同位置同机制,多一行 info 日志)——收敛发现;我本地备好的重复 commit 已弃掉,改验你的版本。
  5. 是否有回归问题:无。既有行为逐项清点(读改动前代码,不只读 diff):① kill-0 轴 3-tick 确认([Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890)不变;② legacy 注册(无心跳记录)保持 kill-0-only;③ 回收 pid 防复用(安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544)仍会死,延迟 8s(PR 已披露);④ 空集 idle grace 5s 与 shutdown reason 字符串(含 [stale-heartbeat] 后缀逻辑)未动;⑤ 多 owner watchers-gone 语义未动;⑥ SIGTERM/SIGINT/SIGBREAK 路径未动;⑦ 2s tick/unref 未动;⑧ 高危模式过一遍:无 fail-fast 变吞错(有界性由第 15 行测试钉死「真死必死」)、无阈值漂移(唯一语义变更即已披露的 15s→23s)、日志为新增行不破坏既有格式。
  6. 是否破坏缓存:不涉及——wire/存储/配置面零改动,无缓存、重试、游标、去重键触碰。
  7. 是否改了设计方向:没有——同一看门狗架构、同一双轴模型,只是把 [Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890 的确认纪律对称地延到第二轴。
  8. 是否引入新配置:没有——WATCHER_HEARTBEAT_CONFIRM_TICKS 是与 WATCHER_DEAD_CONFIRM_TICKS 同级的内部常量,不进 config/env/CLI 面。

测试有效性:8 行新测试是真测试——第 1 行对 pre-fix 代码失败(复现钉住),负向(单次读数不移除)与正向(第 5 次移除且带 stale-heartbeat 标记)双向都有,「5s 间隔最坏情形装得进 5 tick 装不进 3 tick」一行专门防未来常量被随手调小;issue3890 旧「heartbeat expiry stays immediate」行改写为钉新契约而非静默删除。已知边界:caller 级循环记账无法单测(闭包在 startServer 内),b0ac4d504 那处靠套件 + 我的模拟验证,集成层交给 CI 子进程车道。可接受。

diff 整洁度:3 个文件(src/server.ts + 两个测试),每一行都在主题内,无空白/锁文件/生成物噪音。代码整洁清单(命名/死代码/魔法数/错误处理/风格一致)全过。

验证记录:本地定向跑(b0ac4d5):issue4994 8/8 + issue3890 7/7 = 15/15 绿(含 15s 子进程 drain 行);typecheck 由作者声明干净,CI typecheck 车道为准。镜像缓存 CI 状态(head b0ac4d5,fetched 03:38Z):0 failed / 7 pending,已完成项全绿;mergeable_state=behind(master 又前进了,合并前需轻 rebase,预期无冲突——只动 3 个文件)。

判定:代码侧就绪,可合入条件 = ① #5086 owner 拍板(本 PR 实现选项 A;按 floor 30 约定拍板前不合入)+ ② CI 全绿 + ③ 轻 rebase 到当前 master。若 A 获批,以上三点齐了即可合。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

更正上一条的一处出处表述:b0ac4d504 不是 owner 推的,而是处理决策请求 #5086 的另一个 daemon 会话在其审核中发现同一缺口后推到本分支的(见 #5086 floor 7)。本线程审核对该缺口属独立收敛发现——两个会话各自从 diff 出发定位到同一处 caller 记账缺口(dead-probe tick 未持久化 hbStreak 复位),互为佐证。其余审核结论不变。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.906-master.486f9c6 (rolling master tag).

npm install -g billion-context@0.0.906-master.486f9c6  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit to Louie0806/billion-context that referenced this pull request Oct 29, 2026
ranxianglei#5085 (ranxianglei#4994 heartbeat-confirm) landed on master while this PR was in flight: both sides extended tickWatchedPid independently. Union taken: the ranxianglei#5152 lag-widened grace feeds the same stale-heartbeat gate that ranxianglei#4994 now confirms over WATCHER_HEARTBEAT_CONFIRM_TICKS consecutive ticks — complementary defenses of the same false-death class, verified composable in the pure function. The ranxianglei#5152 unit suites are adapted to the merged 6-arg seam (death assertions now complete over the confirmation run instead of a single strike; grace-axis isolation preserved).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant