Skip to content

[PR] fix: dsh lane self-heals after a respawn give-up (#4755) - #4773

Closed
ranxianglei wants to merge 5 commits into
masterfrom
2026-10-26_dsh-giveup-selfheal
Closed

ranxianglei wants to merge 5 commits into
masterfrom
2026-10-26_dsh-giveup-selfheal

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4755

What this fixes

After a runtime give-up ("proxy respawn gave up — model traffic goes direct"), the dsh lane stayed permanently direct/uncompressed even when a healthy replacement proxy was listening on the lane port — in the field case of #4755 every turn re-sent the full history, over-window bodies were refused locally (422), and the session became unrecoverable.

A give-up lands the register in the shape {base: undefined, toolsReady: true} (#3380: host-context tool registrations survive a proxy death). Two defects left exactly that shape with NO self-heal driver:

  1. The recovery timer's tick cleared itself on register.toolsReady alone — exactly the post-give-up shape — so an armed timer died on its first tick without ever calling maybeRetry. maybeRetry's own gate had been fixed for Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 ("the base-less state heals through the RESPAWN branch even when toolsReady is true"), but the timer callback still carried the pre-Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 gate.
  2. fireGiveUp never armed the timer, and degraded model traffic never reaches headersFor (origin resolution fails first), so nothing else re-drove the respawn loop until an /acp command or a host restart.

Fix

  • armRecoveryTimer: the tick now treats base-less as unhealthy even when tools are ready (dead || (toolsReady && base !== undefined)).
  • fireGiveUp: arms the recovery driver at the moment of give-up, so healing starts within one interval instead of never.

Once a base lands, the same gate clears the timer as before — no background churn in the healthy state (pinned by churn assertions in the new test).

Behavior change (old → new)

  • Old: after a respawn give-up the dsh lane had zero self-heal drivers; it stayed direct/uncompressed until an /acp command or a host restart — permanently if neither happened.
  • New: the recovery timer runs at its EXISTING cadence (default 10s between attempts, BILI_DSH_RECOVERY_INTERVAL_MS) until a base lands again. No other defaults, thresholds, or wire behavior changed.

Config surface

None — no new config keys or env vars; BILI_DSH_RECOVERY_INTERVAL_MS already exists and is unchanged.

Dedup check

查重: 不重复。Closest family: #2082 (timer drives only base-less+tools-DOWN states), #3380 (registrations survive give-up — the premise here), #4125/#4389 (give-up log reason suffix), #4516 (lane table). This mechanism (tick gate + missing arm-on-give-up) is distinct from all of them; full comparison in the issue thread.

Tests

  • New tests/issue4755-dsh-giveup-selfheal.test.ts: drives the production give-up hook into the {base:undefined, toolsReady:true} shape, then proves the recovery timer ALONE (zero model traffic) re-bases onto a replacement proxy and clears itself once healthy. Verified to FAIL on the pre-fix code (stash-and-rerun) and pass with the fix.
  • Adjacent suites green locally: issue2082-dsh-recovery, issue3380-dsh-rereg-selflock, issue4125-giveup-reason (15/15). Typecheck clean. Full-suite verdict = CI on this head.

Out of scope (tracked in the issue thread): whether spawn-mode instance adoption should accept a strictly-newer resident build (cross-version adopt, #1225/#3544 territory) — needs owner decision.

A runtime give-up lands the register base-less while toolsReady stays true (#3380). The recovery timer's tick cleared itself on toolsReady alone — exactly that shape — so an armed timer died on its first tick without calling maybeRetry, and fireGiveUp never armed one; degraded model traffic never reaches headersFor either. The lane stayed permanently direct/uncompressed until an /acp command or a host restart, so with a healthy replacement proxy already listening the client never re-attached, over-window bodies were refused locally (422), and the session became unrecoverable.

- armRecoveryTimer: treat base-less as unhealthy even when tools are ready
- fireGiveUp: arm the recovery driver at the moment of give-up
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。独立检索 + 代码比对确认:本 PR 的机制(恢复定时器 tick 门在「无 base + toolsReady」形状上自杀 + fireGiveUp 不装定时器)与同家族条目均不同——#2082 是定时器的引入者(只驱 tools-down 态)、#3380 是前提(toolsReady 跨代理死亡存活)、#4125/#4389 只改 give-up 日志措辞、#4516 只看板展示。无一触碰 tick 门或 arm-on-give-up。

标签: bug / dsh / scope/wiring / p1(保留 pr)— 缺陷类;dsh 宿主接线层(src/agent/dsh-native.ts 插件侧恢复驱动);give-up 后会话永久不可用属 owner 2026-10-19 修订的 p1 类别。已打。

结构化八问

  1. 是什么问题:dsh 原生车道在运行时放弃重生成("proxy respawn gave up")后,即使车道端口上有健康代理在听,压缩也永久失效——流量直接走模型、超窗请求被本地 422、/compact 也救不回来,会话报废直到重启 dsh。
  2. 什么原因导致的:give-up 把注册表打成 {base: undefined, toolsReady: true}(Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 起工具注册跨代理死亡存活)。这个形状恰好落在两处防御的缝里:恢复定时器 tick 的门条件 dead || toolsReady 在该形状下为真,定时器第一次跳就自清、从未调过一次 maybeRetry;而 fireGiveUp 又不装定时器,退化的模型流量又在到达 headersFor 之前就被源解析挡掉——于是没有任何东西再驱动重生成循环。两个缺陷叠加 = 零自愈驱动。根因定位准确,修的是驱动链的缺口本身,不是遮症状。
  3. 影响范围与严重程度:所有 dsh 原生安装(bili dsh 启动器 + dsh plugin profile)一旦走到运行时 give-up(代理死 + 重生成失败)即中招;修复前永久退化至宿主重启,属「会话无法恢复」级 → p1。修复后:只要 bootstrap 能落一个源(同版本 attach,或 spawn 落到空闲/漂移端口),一个重试间隔(默认 10s)内自愈。注意:dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755 现场那个「客户端捆绑包比驻留代理旧」的场景,修复后靠 spawn 回落到漂移端口恢复可用性;「是否应该直接采纳更新的驻留构建」是另一个决策项(见文末)。
  4. 怎么修复的:补驱动链的两个洞——① tick 门改为「健康 = toolsReady 且有活 base」(dead || (toolsReady && base !== undefined)),无 base 时即使工具就绪也继续驱 maybeRetry;② fireGiveUp 当场装恢复定时器,自愈从放弃那一刻起按既有节奏跑,不再依赖外部事件。base 一落地,tick 门自清定时器,健康态零后台动作。
  5. 是否有回归问题:对触碰路径的既有行为逐项清点过(tick 门四态真值表):dead 态自清 ✓ 保持;健康态(toolsReady+有 base)自清 ✓ 保持;dsh中工具会突然消失 #2082 的 tools-down 驱动(toolsReady=false,有无 base 均)✓ 保持;give-up 日志([Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 reason 后缀)✓ 未动;NODE_TEST_CONTEXT 静默守卫 ✓ 未动;单飞/代际守卫(更新后旧代理进程永不替换:新会话启动时自动换血旧代理(版本感知 rotation + 看门狗心跳) #3562 giveUpSeq、dsh attach mode test intermittently fails on Windows CI: the async recovery flow reverts the registered origin back to i | dsh attach 模式测试在 Windows CI 偶发红:异步恢复流程把注册 origin 改回旧值(测试隔离竞态) #3616 originGen)✓ 未动,晚到的链仍不能复活被清的 base。唯一行为变化(PR 已披露,我独立核实):give-up 态从「零后台活动」变为「每 10s 一次重生成尝试直到 base 落地」——频率低于修复前流量驱动的逐请求尝试(有 RESPAWN_COOLDOWN_MS 节流),attach 模式下对被钉死的不可达目标明确不二次起实例,可接受。未发现无意回归。
  6. 是否破坏缓存:不涉及。会话/前缀缓存、重试墙(retryAt)、探测 TTL、去重键全部未动;改的只是「谁触发既有的重试循环」。
  7. 是否改了设计方向:不涉及。完全在既有自愈架构([dsh] native proxyEnvOrigin() 把临时 origin 冻结进 BILLION_CONTEXT_PROXY,绕过 ensureProxyRunning() 的 instance 验活;工具调用全部 fetch failed,需重启宿主 #983/release v0.1.176 #1783/dsh中工具会突然消失 #2082/Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 谱系)内补洞,无新数据流、无新边界。
  8. 是否引入了新的配置:不涉及。BILI_DSH_RECOVERY_INTERVAL_MS 沿用,默认值/阈值/线上行为零变化。

验证证据(独立跑过,非转述)

  • 新测试 tests/issue4755-dsh-giveup-selfheal.test.ts 本地:修复后 PASS;把 src/agent/dsh-native.ts 临时还原到 origin/master 再跑:FAIL,错误正是症状本身(timeout waiting for timer-driven re-base onto the replacement)——独立复现了 PR 声称的 stash-and-rerun 结论。测试走生产 give-up 钩子、零模型流量、带双向 churn 钉(恢复后 attempts===1、manifest 请求数不变),是真测试不是假测试。
  • 相邻套件本地全绿:issue2082-dsh-recovery + issue3380-dsh-rereg-selflock + issue4125-giveup-reason = 15/15。
  • typecheck(tsc --noEmit)干净。
  • CI:head 497afe3 上 23 个 check 全部 queued,最终判定以 CI 绿为准。

两点非阻塞备注

  1. PR 描述有一处措辞偏乐观:原文说旧出路是「/acp 命令或宿主重启」。我核了代码:statusOutcome(/acp 处理器)在无 base 时在 maybeRetry 之前就早退了(src/agent/dsh-native.ts:1431 一带),所以修复前 /acp 单独并不能救这个形状,真正的出路只有宿主重启/重载(apply 重跑计划期 maybeRetry)。实际比描述更糟,不影响修法成立。顺带的正面效果:修复后 /acp 面板里「recovery is running in the background」这句(Plugin tool registration self-lock: after a failed proxy respawn, re-registration is blocked by its own prior registrations — identity headers lost forever (wire mode) → 413 preflight_compress_failed on large sessions | [Bug] 插件工具注册状态自锁:proxy respawn 失败后重注册被自己挡死,永久失去身份头(wire 模式)→ 大会话 413 preflight_compress_failed #3380 P3 写的)终于与事实相符——以前在这个形状下它说的是一场并不存在的后台恢复。
  2. 范围外的 owner 决策项(PR 已声明,我在 dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755 线程里也见过):spawn 模式实例采纳是否应接受严格更新的驻留构建(跨版本采纳,Launcher proxy reuse (ensureProxyRunning) breaks per-run isolation: shared-fate lifecycle + no code/client identity in compatibility check #1225/安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 地带)。本 PR 不动它是对的;该决策已在 dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755 线程挂着等拍板,不在这里另开条目。

判定

可合并。方向正确、diff 最小(13 行源码 + 1 个新测试文件)、测试双向钉住、相邻行为清点无回归、披露完整。等 head 上 CI 绿即可由人合并(我不合并 PR)。

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-26_dsh-giveup-selfheal (99b23e3)

Latest build — install this exact version

npm install -g billion-context@0.0.6145-pr.4773.bf0dfb5

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
99b23e3 billion-context@0.0.6145-pr.4773.bf0dfb5 build
285b645 billion-context@0.0.6060-pr.4773.815ee7c build
b922437 billion-context@0.0.5944-pr.4773.65aa8d6 build
497afe3 billion-context@0.0.5851-pr.4773.1a2c2b7 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4773.tgz
npm install -g package

Install instructions are refreshed on each push.

ci-check added 3 commits October 26, 2026 10:14
master window commits 293ae65/de5e03659 bumped kernel/package.json to
0.0.165 without regenerating the lockfiles; lock-sync re-resolution then
rewrites the stale recorded versions (root 0.0.163, kernel 0.0.164) and
fails. Pure recorded-version sync via npm install --package-lock-only;
no dependency or semantic change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

Closing: the fix landed on master as 6483bbf "fix: dsh lane self-heals after a respawn give-up (#4755)" — all commits on this branch are patch-equivalent to master (git cherry clean).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

1 participant