Skip to content

更新后旧代理进程永不替换:新会话启动时自动换血旧代理(版本感知 rotation + 看门狗心跳) - #3562

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-17_stale-proxy-rotation
Oct 17, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-17_stale-proxy-rotation

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3544

修的是什么(一句话)

插件升级到新版本后,常驻的旧代理进程永远不被替换:attach 路径只查健康、从不比版本,应用重启只是重新挂上旧进程;Windows 上看门狗的 kill(pid,0) 纯存在性检查还分不清被回收的 pid——两层叠加 = 「永远更新不了」。本 PR 让新会话启动时自动换掉严格更旧的驻留代理,并给看门狗加上基于心跳的身份判定。

修法(机制层面,三个子缺陷一个闭环)

  1. 发现式 bring-up 的版本感知换血(src/launcher.ts):pickAttachable 落空后、spawn 之前,若有一个形状兼容(配置/lane/host/passthrough 全同)、看门狗已武装、非 user-zone、且占用我们 canonical port 的驻留实例报告了严格更旧的 version,则向它请求优雅退出(新增 loopback-only 管理端点 POST /__bili/shutdown,走既有 graceful-shutdown + 10s 强制退出兜底),有界等待端口释放后在新进程上绑定同一端口。任何疑点(版本不可比/请求被拒/超时)都回落到既有 spawn 阶梯,行为不劣于现状。
  2. 钉死 attach 路径同步换血(src/agent/dsh-native.ts verifyAttachAndRecover):健康但严格更旧的 pinned target 先请求退出再走 spawn fallback;等值/更新/版本不可考一律照旧 attach;换血失败也照旧 attach(不劣化)。
  3. 看门狗身份加固(src/server.ts + src/server/admin.ts):owner 可带 {pid, hb:true} 注册心跳并周期性重发;hb 注册的 pid 在 WATCHER_HEARTBEAT_GRACE_MS(15s,容忍 3 次丢失)内无心跳即判死——回收的 pid 不再永远「活着」。未 opt-in 的 legacy 注册保持 kill-0 语义,新旧两端双向兼容。
  4. 面板文案纠正(独立 commit,src/agent/dsh-native-client.ts):stale 提示不再给无效的「完全退出重启 dsh」处方,改为说明新会话自动换血、手动杀进程立即生效、仅当应用本身还是旧版才需重启(zh/en)。

行为变更披露(old → new)

# 路径 旧行为 新行为
① 发现式 bring-up,canonical port 被更旧代际占据 共存 + 端口阶梯漂移(strictPort 下 hard-fail),旧进程永生 请求其优雅退出并同端口拉起新版;失败回落旧行为
② dsh pinned attach,目标健康但更旧 无条件 attach(错位永久化) 先换血再 spawn;失败则照旧 attach
③ 代理看门狗判死 仅 kill(pid,0) 存在性检查 hb-opted owner 额外受 15s 心跳过期约束;legacy 注册不变
④ dsh 面板 stale 文案 「请完全退出并重新启动 dsh」(对 detach 代理无效) 自动换血说明 + 手动杀进程选项 + 应用自身旧版时的重启建议

触发面:所有 native 宿主在「磁盘已更新、旧代理常驻」场景;只有严格更旧的驻留会被触碰,user-zone daemon、外 lane、等值/更新版本一律不动。

config surface

无新增/重命名/改语义的配置字段、env 或 CLI flag。新增常量均为模块内部(STALE_ROTATION_WAIT_MS / WATCHER_HEARTBEAT_GRACE_MS 等);POST /__bili/shutdown 与 /__bili/watcher 的可选 hb 字段是 loopback-only 管理线协议扩展,不是用户配置面。

kernel-change

不涉及(kernel/ 零改动)。

测试

查重

查重: 无(open+closed issue/PR 按 detach/stale/rotation/watchdog-pid/版本错位多组关键词检索均无同症状条目;相关前作 #1723/#1225/#1335 均已在本 PR 的机制中兼容复用)。

awork added 2 commits October 17, 2026 19:34
…ching to them

A detached proxy outlives its owner by design, but nothing ever replaced it:
attach paths compared health only, never generation, so after an update the
new session re-attached to the old process forever (and on Windows the
watchdog's kill(pid,0) cannot tell a recycled pid apart from the original).

- discovery bring-up: when nothing attachable exists but a shape-compatible,
  watchdog-armed, non-user-zone resident holds our canonical port and reports
  a strictly older version, request its graceful shutdown (new loopback-only
  POST /__bili/shutdown) and wait bounded for the port before spawning onto it
- pinned attach (dsh-native): a healthy-but-strictly-older target is rotated
  the same way before falling back to spawn; equal/newer/unverifiable versions
  attach as today
- watchdog identity: owners may opt into liveness heartbeats (/__bili/watcher
  {pid, hb:true}); hb-opted pids additionally age out past WATCHER_HEARTBEAT_GRACE_MS
  so a recycled pid can no longer read 'alive' forever. Legacy registrations
  keep kill-0-only semantics, both directions compatible
- regression suite: tests/issue3544-stale-proxy-rotation.test.ts (16 cases)
The stale banner prescribed 'fully quit and relaunch dsh' — useless for a
detached proxy (relaunch just re-attaches the old process). Now states that a
new session swaps in the new proxy automatically, manual kill works
immediately, and relaunch is only needed when the app itself still runs the
old build. zh/en.
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_stale-proxy-rotation (41f00fb)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4247-pr.3562.f06e849

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3562.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复 —— 独立检索(英文 stale/rotation/watchdog/heartbeat/shutdown 与中文「旧代理/换血/版本/心跳」两组关键词,open+closed):无同症状条目。相邻前作 #1723/#1225/#1335/#3044(升级换代端口交接、code-fingerprint attach 门、watchdog attach 门)均为已合并的局部机制,本 PR 在其上补「换血通路」而非重复实现;原 issue #3544 的 triage(floor 2)已独立核实两层根因,方向与本 PR 一致。

开始按 PR review 规范审查:①原 issue 方向 → ②回归清点 → ③代码 → ④测试有效性 → ⑤diff 整洁度。注意到 PR 当前 mergeable_state: behind(落后 master),会核对 rebase 后是否仍干净。初步结论随后给出。

awork added 3 commits October 17, 2026 22:54
…oxy-rotation

Sync the PR branch with current master (v0.1.198 line) ahead of the review
fixes below; the textual merge was conflict-free.
… probe cannot resurrect a cleared base

CI regression found during review: tests/issue3380-dsh-rereg-selflock.test.ts
("#3380 incident repro") fails on ubuntu-22/windows-22 shards — after the
preset proxy is killed and the owner gives up, register.base is resurrected
with the dead origin.

Root cause: the new rotation branch in verifyAttachAndRecover inserts extra
awaits (the /__bili/shutdown round-trip) between the version probe and the
base commit, widening a pre-existing race where an in-flight recovery chain
commits register.base AFTER onGiveUp already cleared it. A resurrected base
stamps x-bili-plugin at a dead origin until the next recovery tick.

Fix: a monotonic giveUp generation (register.giveUpSeq) bumped by every
onGiveUp callback; chains capture it at entry and skip their
(state.origin, register.base) commits when it has moved — the chain simply
resolves undefined ("nothing came up"), which is the documented contract
and lets maybeRetry drive the next attempt with fresh bookkeeping. All three
commit families in verifyAttachAndRecover plus the spawnFallback landing are
guarded.

Also corrects a factually wrong comment: /__bili/plugin/manifest has reported
a version since long before #3544, so "pre-#3544 builds report no version"
was false; the unverifiable case is an unreachable target (dead-target path).

Verified locally: issue3380 file 6/6, issue3544 suite 16/16, watchdog-health,
native-intercept, issue2954 green; typecheck clean.
CI regression found during review: tests/launcher.test.ts ("a lane'd launch
with port 0 binds the zone preference and settles it sticky (#1660)") fails
on all three OS shards — the injected zonePreferredPort spy was called twice
instead of once.

Root cause: the rotation call site resolved the lane's preferred port
eagerly on EVERY lane'd bring-up, even when nothing would rotate. The zone
preference resolver is stateful (sticky settlement); touching it adds a
second settlement call beyond the spawn ladder's own.

Fix: rotateStalePredecessor takes a resolveTargetPort thunk and narrows by
every gate except the port first (armed, non user-zone, shape-compatible,
strictly older); the resolver is only invoked when a strictly-older candidate
actually exists. Selection semantics are unchanged (the stale candidate must
still hold our canonical port).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论:发现 2 个真实回归,已在 PR 分支直接修复(3 个新提交),待 CI 复绿后可合

独立重审了本 PR(未照抄 issue 结论)。方向正确、机制设计合理(换血三闸门 + 心跳身份判定 + 失败一律回落,严格更旧才动手),测试有牙。但 CI(run 39029473489)暴露了 PR 自身引入的 2 个确定性回归,均已定位到根因并修复推送到 PR 分支(6fb1ed7aa..7ebdfc4f9,含 master 同步合并——原分支 behind,文本合并零冲突,master 侧 #3394/#3537 改动与本子系统正交,已逐段核对)。

修复 1 — zone-preference 双调用(3 平台红):tests/launcher.test.ts「a lane'd launch with port 0 binds the zone preference and settles it sticky (#1660)」断言注入的 zonePreferredPort spy 恰好被调 1 次,实际 2 次。根因:src/launcher.ts 的 rotation 调用点把 lane 首选端口的解析急切化了——每次带 lane 的 bring-up 都多结算一次这个有状态(sticky)解析器。修法:rotateStalePredecessor 改收 resolveTargetPort thunk,先按除端口外的全部闸门收窄(armed / 非 user-zone / 形状兼容 / 严格更旧),只有存在严格更旧候选时才解析端口;选择语义不变(候选仍必须占我们的 canonical port)。

修复 2 — giveUp 竞态复活死 origin(u22/w22 两平台红):tests/issue3380-dsh-rereg-selflock.test.ts「#3380 incident repro」在 preset 代理被杀 + owner giveUp 之后断言 register.base === undefined,实际复活成死 origin。用时间戳插桩实测钉死因果链:merged tree VERSION=0.1.198 > mock manifest 的 0.1.119 → 新 rotation 分支触发 → 它在版本探针与 base 提交之间插入了额外 await(shutdown POST 往返 ~7ms,探针本身 ~17ms)→ 原有竞态窗口被撑开:in-flight 的 verifyAttachAndRecover 在 onGiveUp 清掉 base 之后才落盘提交,把死 origin 重新写回。生产同形场景真实存在:更旧代理正在垂死排空时答了最后一次探针,随后死亡,迟到的提交把 x-bili-plugin 钉死在死 origin 上直到下个恢复 tick。修法(根因而非逐分支打补丁):单调 giveUp 代际计数器 register.giveUpSeq——三个 onGiveUp 回调各自 bump,链路入口捕获,提交前发现代际已动就放弃提交、按既有契约返回 undefined("nothing came up"),由 maybeRetry 以全新簿记驱动下次尝试。覆盖 verifyAttachAndRecover 全部三个提交点(等值/更新、换血失败照旧 attach、pinned 恢复)+ spawnFallback 落地提交。顺手修正一处事实性错误注释(manifest 早在 #3544 之前就上报 version,"pre-#3544 builds report no version" 不成立)。

Windows-only 挂起(w22s1):lane-decisions-matrix 4 个子测试 cancelledByParent("Promise resolution is still pending...")。本地 merged tree 全绿(66.8s)无法复现;检索确认该签名属于已知 Windows CI flake 家族(#3530、#3571 [no-dispatch]、#3356,修复 #3563 在途)——不另立项,观察本次 CI 是否再现。

结构化八问

  1. 是什么问题:PR 修的是「升级后旧驻留代理永不替换」(安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544);审核发现的问题是 PR 自身引入的两个回归(端口解析器多结算一次违反 [Design] 定死端口 lane(claude 48787 / zcode 48789)端口被占时的 0 配置解法 #1660 契约;giveUp 后迟到提交复活死 origin)加一个已知家族的 Windows 挂起。
  2. 什么原因导致的:①调用点把有状态解析器的调用时机从「需要时」提前到「每次 bring-up」;②rotation 分支在探针与提交之间插入网络往返,放大了既有的 in-flight 链路 vs onGiveUp 竞态——竞态本身是旧的,窗口是新的。
  3. 影响范围与严重程度:①所有 port 0 的带 lane 启动(dsh/zcode 等)——契约测试级回归,sticky 值本身不变,严重度中低;②dsh pinned-attach 会话在代理死亡与 attach 链路飞行重叠时(数十毫秒窗口)——模型流量短暂指向死 origin,中等,可自愈于下个恢复 tick 但会烧掉一批请求。
  4. 怎么修复的:①惰性解析——先过滤出严格更旧候选再解析端口,无候选零调用;②giveUp 代际守卫——owner 一旦放弃,所有早于它启动的链路的迟到提交整体作废,统一走「nothing came up」契约让重试循环接管。
  5. 是否有回归问题:审过并逐项保持——换血闸门(仅严格更旧、user-zone/unarmed/外 lane 不动)、等值/更新照旧 attach、拒绝退出照旧回落 attach、spawn 阶梯行为、Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 idempotent adoption、本 PR 自带 16 例、watchdog-health、native-intercept、issue2954、launcher 全文件 313 例,typecheck 干净。有意行为变更(披露):a) giveUp 与任何成功探针竞态时,迟到的 base 提交现在会被抑制(旧行为是复活)——这正是 Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 钉死的不变量,扩展到同函数兄弟分支;b) zone-preference 解析器只在确有换血候选时被调用(调用次数只减不增);c) 链路被代际守卫作废时不再打印误导性的 "attaching anyway" 事件日志。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试游标/去重键;RETRY_INTERVAL_MS 退避语义不变)。
  7. 是否改了设计方向:没有——两个修复都留在 PR 自己的机制内(rotation + 换血提交时序),无架构转向。
  8. 是否引入了新的配置:没有(giveUpSeq 为模块内部簿记;无 env/config/CLI 变化)。

一个必须在发布说明里写明的边界(FINDING A)

换血要求驻留方有 /__bili/shutdown 端点:pre-#3562 的驻留代理会上报 version(所以会被选中),但对该端点返回 404 → 请求被拒 → 走「照旧 attach」回落。结论:第一个包含本 PR 的版本发布后,仍常驻的旧版代理不会被自动换掉,需要手动杀一次;从第二个世代起的驻留都带端点,自愈机制才完整接管。面板文案已有「手动结束旧代理进程立即生效」的对冲,建议首个含此修复的 release notes 里补一句同样说明。

判定:推送头 7ebdfc4f9(master 同步 + 2 个标注修复提交)CI 全绿即可合;合并动作留给人。CI 结果出来我会在本线程补充确认。

…n commits

Round-2 CI (u24s1) timed out '#983 maybeRetry self-heals a base-less register
after a failed respawn' (15s waitFor). The giveUp generation guard was also
applied to the spawnFallback landing, but a fresh-spawn landing is live by
construction — it cannot resurrect anything — and the self-heal contract
(#983/#1783) requires exactly these late landings to re-base a given-up
register. A giveUp that lands between fallback start and landing must not
suppress the heal.

The guard now covers only verifyAttachAndRecover's three commits of an
EXISTING origin (equal/newer branch, rotation-refusal branch, pinned-recovery
branch), whose liveness was probed earlier and may be stale at commit time —
the exact #3380 resurrection hazard, proven by the measured incident-test
timeline (giveUp T589.23 < refusal commit T592.18).

Local: dsh-native.test.ts 55/55, #983 self-heal 20x stable, issue3380 6/6,
issue3544 16/16, native-intercept 62/62, launcher 313/313, typecheck clean.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 跟进(接上条结论):head 41f00fb72 全绿,可以合了。

第 3 轮 CI(run 39046188052,head 41f00fb)

全部通过:test 分片 ubuntu-22 ×3、ubuntu-24 ×1、windows-22 ×2 全绿(包括第 2 轮挂掉的 u24s1 和第 1/2 轮抖动的 w22s1),kernel/pi-subagents/version/config-surface/lane/docs 等 guard 车道与全部 e2e 车道均 success。分支已同步到当前 master(合并提交 a1df98535,文本合并零冲突)。PR 现在共 5 个提交:原 2 个 + master 同步 + 3 个 review 修复。

第 2 轮 u24s1 的 #983 超时——定性为环境性,但顺手收紧了一处语义

第 2 轮 u24s1 上 #983 maybeRetry self-heals a base-less register after a failed respawn 15s 等待自愈超时(兄弟用例 #1783 ×2 通过)。排查结论:不是修复引入的行为回归:

  • 该测试流程里我给 verifyAttachAndRecover 加的 giveUp 失效守卫一处都不会执行(预设探测失败→version undefined→三个 attach 分支全部跳过;首次 spawn 结果为 undefined→landing 走不到受守卫的 origin 分支;测试全程不触发 onGiveUp,giveUpSeq 恒 0);
  • 同一份合并后的 master 内容(b10c525,即本 PR 的 base)主 CI 是绿的(run 39038363825)——排除「master 合并带进来」;
  • 定性为加载下的 ubuntu-24 分片把 15s waitFor 预算吹爆(本地同测试连跑 20 次 0 失败)。

不过排查中发现 spawnFallback landing 处的守卫语义上确实收得太宽(新 spawn 落地的 origin 天生是活的,#983/#1783 契约要求迟到的落地必须能重新基化已放弃的 register,先前的 giveUp 不该压制它)——已收窄:giveUp 失效守卫只作用于「attach 到既有 origin」的三个提交点,spawn 落地不再受压(41f00fb72,附注释说明该处为何刻意不守卫)。生产上该路径真实可达(native-intercept 请求路径发现无活代理时会调 onGiveUp 转直发)。

两个非阻塞观察

结论维持上条:回归已修、行为变更已披露(含 FINDING A——首个包含本 PR 的版本仍需一次手动杀进程退役旧代际驻留代理,建议进 release notes)、无 config surface 变更。可合;合并留给你。

@ranxianglei
ranxianglei merged commit 7c2920c into master Oct 17, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.540-master.7c2920c (rolling master tag).

npm install -g billion-context@0.0.540-master.7c2920c  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」)

1 participant