Repository navigation
[PR] fix: dsh attach-to-exiting-agent race — self-heal dead base, skip draining instances (#6881) - #6921
[PR] fix: dsh attach-to-exiting-agent race — self-heal dead base, skip draining instances (#6881)#6921ranxianglei wants to merge 1 commit into
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10376-pr.6921.100b361This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr6921.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…ining instances (#6881) When a dsh host restarts within seconds of the old host stopping, the new host can attach to the old proxy's draining tail (listener closed, established keep-alive conns still answering). register.base then points at a dead address; the recovery timer only re-fetches the dead manifest (backoff 10→20→40s), never respawns; and the first model request's plugin headers were computed pre-send to the dead addr → proxy treats it as identity-less → new ANONYMOUS session. Fixes: - probeLiveInstances: probe on a FRESH connection (Connection: close) so a draining instance (listener closed) is detected dead; a BUSY instance (accept queue still drains) still TCP-connects, so the #3654/#4811 BUSY keep is unaffected. - discoverLiveInstances: skip registry entries stamped portReleasedAt (draining) — mirrors classifySameLanePeer's skip. - registerTools catch: a manifest failure against a base that no longer accepts connections (refused connect) clears the dead base so the respawn self-heal takes over; a BUSY base (connectable) stays put. - Fresh-spawn landings (bootstrap/spawnFallback/maybeRetry) clear the old base's retryAt/failStreak so the new base's first registration is not gated behind the dead base's wall. - attachTo: log the watcher registration result (ok/failed/refused) on the host-safe channel — it was invisible in bili.log before. Regression test: tests/issue6881-dsh-attach-draining-selfheal.test.ts
e8d65d2 to
21b7822
Compare
(ai: vllm-qwen/qwen3.8-27b) PR #6921 review — Reviewed the full diff (4 files, +397/-3) against the checked-out What I verified
Minor observations (non-blockers)
8问
Verdict: 修复 sound 且对 #6881 完整——正常竞态窗口 discovery 直接 drop P1(portReleasedAt skip + fresh-conn 探测被拒)→ H2 自己 spawn P2;更紧的竞态窗口(listener 在探测和 attach 之间关)由 self-heal(清死 base)在 ~10s 内清掉 → 重生 → P2。测试有牙(A/B/C 无修复时全失败、有修复全通过;A-neg 是 guard)。CI 26 过 / 4 待 / 0 败。可合(建议 rebase 到最新 master 求整洁,非阻塞)。 |
What
When a dsh host restarts within seconds of the old host stopping, the new host can attach to the old proxy's draining tail (listener already closed, but established keep-alive connections still answering).
register.basethen points at a dead address; the recovery timer only re-fetches the dead manifest (backoff 10→20→40s) and never respawns; and the first model request's plugin headers were computed pre-send to the dead address → the proxy treats it as an identity-less wire request → a new anonymous session (no compression, no identity).This is the root-cause fix for the attach-to-exiting-agent race.
Fixes
probeLiveInstances(launcher.ts) — probe on a fresh connection (Connection: close) so a draining instance (listener closed) is detected dead. A BUSY instance (accept queue still drains while its loop is inside a large fold) still TCP-connects on a fresh socket, so the [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 BUSY keep is unaffected.discoverLiveInstances(instance.ts) — skip registry entries stampedportReleasedAt(draining) — mirrorsclassifySameLanePeer's skip.registerToolscatch (dsh-native.ts) — a manifest failure against a base that no longer accepts connections (refused connect) clears the dead base so the respawn self-heal takes over; a BUSY base (connectable) stays put.bootstrap/spawnFallback/maybeRetryclear the old base'sretryAt/failStreakso the new base's first registration is not gated behind the dead base's wall.attachTo(launcher.ts) — log the watcher registration result (ok/failed/refused) on the host-safe channel — it was invisible in bili.log before.Regression test
tests/issue6881-dsh-attach-draining-selfheal.test.ts— 4 tests:portReleasedAt(draining) is skipped by discovery.Scope note
This PR fully fixes #6881 (the attach-to-exiting-agent race). It does NOT fully fix #5507 (event-loop-watchdog self-exit + split-brain under 30–45 MB sessions):
So this PR uses
Fixes #6881and references #5507 (notFixes #5507— it is not fully fixable here, and usingFixeswould wrongly auto-close it).Fixes #6881
Related: #5507