Skip to content

[PR] fix: dsh attach-to-exiting-agent race — self-heal dead base, skip draining instances (#6881) - #6921

Open
ranxianglei wants to merge 1 commit into
masterfrom
2026-11-16_dsh-attach-draining-self-heal
Open

ranxianglei wants to merge 1 commit into
masterfrom
2026-11-16_dsh-attach-draining-self-heal

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

When a dsh host restarts within seconds of the old host stopping, the new host can attach to the old proxy's draining tail (listener already closed, but established keep-alive connections still answering). register.base then points at a dead address; the recovery timer only re-fetches the dead manifest (backoff 10→20→40s) and never respawns; and the first model request's plugin headers were computed pre-send to the dead address → the proxy treats it as an identity-less wire request → a new anonymous session (no compression, no identity).

This is the root-cause fix for the attach-to-exiting-agent race.

Fixes

  1. probeLiveInstances (launcher.ts) — probe on a fresh connection (Connection: close) so a draining instance (listener closed) is detected dead. A BUSY instance (accept queue still drains while its loop is inside a large fold) still TCP-connects on a fresh socket, so the [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 BUSY keep is unaffected.
  2. discoverLiveInstances (instance.ts) — skip registry entries stamped portReleasedAt (draining) — mirrors classifySameLanePeer's skip.
  3. registerTools catch (dsh-native.ts) — a manifest failure against a base that no longer accepts connections (refused connect) clears the dead base so the respawn self-heal takes over; a BUSY base (connectable) stays put.
  4. Fresh-spawn landings (dsh-native.ts) — bootstrap / spawnFallback / maybeRetry clear the old base's retryAt/failStreak so the new base's first registration is not gated behind the dead base's wall.
  5. attachTo (launcher.ts) — log the watcher registration result (ok/failed/refused) on the host-safe channel — it was invisible in bili.log before.

Regression test

tests/issue6881-dsh-attach-draining-selfheal.test.ts — 4 tests:

Scope note

This PR fully fixes #6881 (the attach-to-exiting-agent race). It does NOT fully fix #5507 (event-loop-watchdog self-exit + split-brain under 30–45 MB sessions):

So this PR uses Fixes #6881 and references #5507 (not Fixes #5507 — it is not fully fixable here, and using Fixes would wrongly auto-close it).

Fixes #6881
Related: #5507

@ranxianglei ranxianglei added bug Something isn't working dsh p2 root-cause labels Nov 16, 2026
@ranxianglei ranxianglei changed the title fix: dsh attach-to-exiting-agent race — self-heal dead base, skip draining instances (#6881) [PR] fix: dsh attach-to-exiting-agent race — self-heal dead base, skip draining instances (#6881) Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-16_dsh-attach-draining-self-heal (21b7822)

Latest build — install this exact version

npm install -g billion-context@0.0.10376-pr.6921.100b361

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
21b7822 billion-context@0.0.10376-pr.6921.100b361 build
e8d65d2 billion-context@0.0.10359-pr.6921.4343f29 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6921.tgz
npm install -g package

Install instructions are refreshed on each push.

…ining instances (#6881)

When a dsh host restarts within seconds of the old host stopping, the new host
can attach to the old proxy's draining tail (listener closed, established
keep-alive conns still answering). register.base then points at a dead
address; the recovery timer only re-fetches the dead manifest (backoff
10→20→40s), never respawns; and the first model request's plugin headers were
computed pre-send to the dead addr → proxy treats it as identity-less → new
ANONYMOUS session.

Fixes:
- probeLiveInstances: probe on a FRESH connection (Connection: close) so a
  draining instance (listener closed) is detected dead; a BUSY instance
  (accept queue still drains) still TCP-connects, so the #3654/#4811 BUSY
  keep is unaffected.
- discoverLiveInstances: skip registry entries stamped portReleasedAt
  (draining) — mirrors classifySameLanePeer's skip.
- registerTools catch: a manifest failure against a base that no longer
  accepts connections (refused connect) clears the dead base so the respawn
  self-heal takes over; a BUSY base (connectable) stays put.
- Fresh-spawn landings (bootstrap/spawnFallback/maybeRetry) clear the old
  base's retryAt/failStreak so the new base's first registration is not
  gated behind the dead base's wall.
- attachTo: log the watcher registration result (ok/failed/refused) on the
  host-safe channel — it was invisible in bili.log before.

Regression test: tests/issue6881-dsh-attach-draining-selfheal.test.ts
@ranxianglei
ranxianglei force-pushed the 2026-11-16_dsh-attach-draining-self-heal branch from e8d65d2 to 21b7822 Compare November 16, 2026 10:09
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR #6921 review — fix: dsh attach-to-exiting-agent race — self-heal dead base, skip draining instances (#6881)

Reviewed the full diff (4 files, +397/-3) against the checked-out pr-6921 branch, verified every mechanism against the source, and ran the targeted regression suites. No parallel PR targets #6881.

What I verified

  1. All 5 fixes are real and correctly placed:

    • probeLiveInstances (src/launcher.ts:4230) passes {noKeepAlive:true} to fetchHealthInfo. Verified fetchHealthInfoDefault (src/launcher.ts:4046) already supports noKeepAlive (sends Connection: close) from Auto-update on dsh leaves two proxy processes writing one session store: rotation waits mistake a draining old proxy for one that never exited #5167 — so the call-site change is real, not a no-op.
    • discoverLiveInstances (src/instance.ts:802) skips portReleasedAt entries — complements the fresh-conn probe (registry path vs single-origin-file path, both covered).
    • registerTools catch (src/agent/dsh-native.ts:1088) is now async; clears the dead base via a tcpConnectable probe.
    • All 3 fresh-spawn landings (bootstrap:717, spawnFallback:879, maybeRetry:1219) clear retryAt/failStreak.
    • ensureProxyRunning (src/launcher.ts:5630) logs the watcher result (ok/failed/refused) via attachDiag.
  2. The BUSY/dead distinction is sound. tcpConnectable (src/liveness.ts:71) is a pure TCP-connect probe: a draining instance (listener closed) refuses fresh connects → false → base cleared; a BUSY instance (accept queue still drains) accepts → true → base kept. The [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 BUSY keep is preserved.

  3. The test has teeth. Restored the pre-fix source files (kept the new test) → A/B/C all FAIL (A fails with exactly the incident symptom: "timeout waiting for timer-driven re-base onto the replacement after the dead base is cleared"). Restored the PR → all 4 PASS. A-neg is a guard (passes both pre/post).

  4. No regressions. Ran 10 related suites (dsh-native 76, instance-governance 18, issue4982 7, issue5167 4, issue3654 1, issue4811 1, issue2082 4, dsh-tool-registration-lifetime 4, issue2187 2, dsh-llm-stream-attribution 7) — all 0 fail. Typecheck clean.

  5. CI: 26 passed, 4 pending, 0 failed (mirror cache, sha 21b7822). The 4 pending are the long-running e2e lanes; no failures.

Minor observations (non-blockers)

8问

  1. 是什么问题: dsh 宿主快速重启时,新宿主会 attach 到旧代理的 draining 尾巴(listener 已关但 keep-alive 连接还在应答),register.base 指向死地址,恢复定时器只重取死 manifest 不重生,第一份模型请求的插件头算在死地址上 → 无身份 wire 请求 → 匿名会话(无压缩)→ 大会话 preflight 413。
  2. 什么原因导致的: 四个机制叠加——(1) 健康探测用 keep-alive 连接,draining 实例的已建立连接仍应答 → 误判存活;discoverLiveInstances 忽略 portReleasedAt;(2) registerTools 失败分支只退避不清死 base,maybeRetry 的重生分支只在 base===undefined 时跑;(3) 重生落地不清 retryAt/failStreak,拦截器复用发往死地址前算好的 allExtra;(4) watcher 注册结果只 console.error,bili.log 不可见。
  3. 影响范围与严重程度: 触发条件:dsh 宿主在旧宿主停止后数秒内重启(5s 空闲宽限窗口内)。严重程度 p2——能用,但大会话(30–45MB)会落到零折叠匿名会话,preflight 按原始历史跑(32 轮 412s)最后 413;匿名会话留在账本。小会话影响较小。
  4. 怎么修复的: 五处——(1) 健康探测改 fresh connection(Connection: close):draining 实例 fresh 连接被拒 → 判死;BUSY 实例(accept 队列仍 drain)fresh 连接仍被接受 → 保留([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 BUSY keep 不受影响);(2) discoverLiveInstances 跳过 portReleasedAt 条目(registry 路径便宜守卫,与 fresh-conn 探测互补);(3) registerTools 失败分支用 tcpConnectable 探 base,死 base 清掉让重生自愈接管,BUSY base(可连)不动;(4) 三处 fresh-spawn 落地清旧 base 的 retryAt/failStreak,新 base 首次登记不被死 base 的墙挡住;(5) ensureProxyRunning 把 watcher 注册结果记到 host-safe 通道。
  5. 是否有回归问题: 无。跑过 10 个相关套件全 0 失败,typecheck 干净。BUSY keep([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811)有专门测试钉住(A-neg)。
  6. 是否破坏缓存: 不涉及。改动在 dsh attach/重生/登记路径,不碰压缩缓存、重试游标、去重键。
  7. 是否改了设计方向: 不涉及。修复在既有 attach/重生/登记机制内,不引入新架构/边界/数据流。
  8. 是否引入了新的配置: 不涉及。无新 env/config/默认值。

Verdict: 修复 sound 且对 #6881 完整——正常竞态窗口 discovery 直接 drop P1(portReleasedAt skip + fresh-conn 探测被拒)→ H2 自己 spawn P2;更紧的竞态窗口(listener 在探测和 attach 之间关)由 self-heal(清死 base)在 ~10s 内清掉 → 重生 → P2。测试有牙(A/B/C 无修复时全失败、有修复全通过;A-neg 是 guard)。CI 26 过 / 4 待 / 0 败。可合(建议 rebase 到最新 master 求整洁,非阻塞)。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment