Repository navigation
[PR] fix(server): confirm parent death over consecutive probes; drain before force-exit (#3890) - #3902
[PR] fix(server): confirm parent death over consecutive probes; drain before force-exit (#3890)#3902ranxianglei wants to merge 3 commits into
Conversation
…re force-exit (#3890) Two independent defects made the proxy self-terminate on Windows while its parent lived on (12 events in 2.5h against one live pid): 1. The watchdog removed a watched pid from the set on a SINGLE failed kill-0 probe; the empty-set grace window that followed never re-probed, so one transient OpenProcess failure (Windows maps every non-ACCESS_DENIED failure to ESRCH — indistinguishable from a real death) force-exited the proxy ~6s later. Death now requires WATCHER_DEAD_CONFIRM_TICKS (=3, ~4-6s at the 2s tick) CONSECUTIVE failed probes; any live probe resets the streak. Heartbeat-expiry removal (#3544) keeps its immediate semantics. First failure of an episode logs the errno ([watchdog] kill-0 probe failed for pid N (code=...)) and recovery from a false alarm logs one line, so field episodes can be attributed. probePid() in instance.ts carries the errno; isPidAlive() is unchanged. 2. shutdown()'s hard backstop force-exited 10s after the signal regardless of live connections, cutting in-flight upstream streams mid-response. While connRecords are still alive the window now RE-ARMS, bounded by SHUTDOWN_DRAIN_MAX_PASSES (=10, worst-case total drain ~110s); then it force-exits exactly as before (the zombie backstop is preserved). Idle keep-alives don't hold the proxy — the keep-alive reaper clears them within one window, so a healthy drain still exits promptly. Behavior changes (old -> new): - owner-death -> exit latency: +up to ~6s (confirmation window) - shutdown backstop: unconditional 10s force-exit -> re-arms up to 10 extra passes while liveConns > 0, worst-case ~110s total, then force-exit as before - new log lines: [watchdog] kill-0 probe failed / pid N alive again / extending drain k/10 Tests: new suite pins tickWatchedPid (blip <3 survives, 3rd consecutive removes, recovery resets, heartbeat path unchanged, dead pids never take the heartbeat fast path) plus a LIVE child-process test proving the backstop no longer force-exits at 10s with a connection held open (verified to FAIL on pre-fix master: 'pre-#3890 behavior force-exited at 10s'). claude-native live-lane deadlines re-sized 15s->30s and the watcher-route grace phase re-timed for the wider confirmation window (ACP_TEST_CLAUDE_NATIVE-gated).
…3890) Labeled in-passing fix found while working #3890: the oversizeDirectRefusal message told clients 'Recovery is automatic (#3308); retry shortly' — but the proxy respawn does NOT re-send this request, so 'retry shortly' implied an automatic retry that never happens (the issue's own damage trail shows these refusals going unanswered during each ~51s gap). Wording now states plainly that the request must be resubmitted once the proxy is back.
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 按失败签名(parent-gone 误判 / Windows probe / drain force-exit)检索 open+closed issue 与 PR:唯一同源条目是 #3890(本 PR 的原始 issue);无并列修复 PR。最接近的 #3079(短命 CLI 客户端让 session-owned proxy 提前退出)机制不同——那是 watcher 生命周期归属问题,不是 Windows kill-0 探测误判,不构成重复。 标签: 本 PR 由本 daemon 早前会话(ses_7fa1f7e0…)产出,现在做独立复核:代码与描述逐项核对、定向测试、CI 状态核查。另注意:分支与当前 master 有真实内容冲突( |
c11bdfd to
ec1f0b6
Compare
(ai: vllm-qwen/qwen3.8-27b) 上游冲突已解决,rebase 完成并推回分支。结构化结论(八问):
判定:冲突已解,GitHub 侧 |
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4635-pr.3902.8f738d2This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3902.tgz
npm install -g packageInstall instructions are refreshed on each push. |
… gate) The rebased #3902 head trips the unused-exports static gate: WatcherTickDecision is referenced only inside src/server.ts (as tickWatchedPid's return type; the unit suite consumes decisions structurally). Drop the leading export per the gate's INTERNAL_ONLY remedy. Type-only change: gate, typecheck, build and the issue3890 suite are green.
What is broken (#3890)
On Windows the proxy self-terminated 12 times in 2.5h while its parent pid was alive — each kill cut in-flight streams and left clients staring at a refusal that said "retry shortly" with nothing actually retrying. Two independent defects, both fixed here; plus one labeled in-passing wording fix.
Mechanism 1 — single-probe death verdict (the false positive)
src/server.tswatchdog: every 2s tick ranisPidAlive(pid)per watched pid and removed the pid on ONE failed probe; the empty-set grace window that followed never re-probed, so a single transientOpenProcessfailure (Windows maps every non-ACCESS_DENIED failure to ESRCH — indistinguishable from real death) force-exited the proxy ~6s later.Fix:
probePid()(src/instance.ts) carries the errno; the per-pid decision is extracted into puretickWatchedPid()and requiresWATCHER_DEAD_CONFIRM_TICKS(=3) CONSECUTIVE failed probes (~4-6s at the 2s tick) before removal; any live probe resets the streak. Heartbeat-expiry removal (#3544) keeps its immediate semantics (unit-pinned). First failure of an episode logs[watchdog] kill-0 probe failed for pid N (code=…)and recovery from a false alarm logs one line — field instrumentation so the next Windows episode can be attributed by errno.Mechanism 2 — unconditional 10s force-exit cuts live streams
shutdown()'s hard backstop firedprocess.exit(0)10s after the signal regardless of live connections. Fix: whileconnRecords.size > 0the window RE-ARMS, bounded bySHUTDOWN_DRAIN_MAX_PASSES(=10 → worst-case total drain ~110s); then it force-exits exactly as before (zombie backstop preserved). Idle keep-alives don't hold the proxy — the keep-alive reaper clears them within one window, so a healthy drain still exits promptly.In-passing fix (separate commit)
src/agent/opencode-native.tsoversizeDirectRefusal: "Recovery is automatic (#3308); retry shortly" implied an automatic retry that never happens — the respawn does not re-send this request. Wording now says plainly: resubmit once it's back.Behavior changes (old → new)
Companion test re-timing (intentional, documented in comments): claude-native live-lane deadlines 15s→30s; watcher-route grace phase 3s/7s waits → 8s/8s (late registration now lands after the ~6s removal but inside the following 5s idle grace — still exercises pending-shutdown cancellation). These lanes are
ACP_TEST_CLAUDE_NATIVE=1-gated, off in default CI.Verification
tests/issue3890-parent-gone-confirm-drain.test.ts: 7/7 pass locally (6 puretickWatchedPidunit tests + 1 LIVE child-process drain test).issue3544-stale-proxy-rotation,watchdog-health,watchdog-health-daemon(22/22),opencode-native.No config surface change (hardcoded constants in the existing WATCHER_* style — tunability can be discussed later if wanted). No kernel/ change. No wire/persistence change.
Closes #3890