Skip to content

[PR] fix(server): confirm parent death over consecutive probes; drain before force-exit (#3890) - #3902

Closed
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-20_win32-parent-gone-confirm-drain
Closed

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-20_win32-parent-gone-confirm-drain

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What is broken (#3890)

On Windows the proxy self-terminated 12 times in 2.5h while its parent pid was alive — each kill cut in-flight streams and left clients staring at a refusal that said "retry shortly" with nothing actually retrying. Two independent defects, both fixed here; plus one labeled in-passing wording fix.

Mechanism 1 — single-probe death verdict (the false positive)

src/server.ts watchdog: every 2s tick ran isPidAlive(pid) per watched pid and removed the pid on ONE failed probe; the empty-set grace window that followed never re-probed, so a single transient OpenProcess failure (Windows maps every non-ACCESS_DENIED failure to ESRCH — indistinguishable from real death) force-exited the proxy ~6s later.

Fix: probePid() (src/instance.ts) carries the errno; the per-pid decision is extracted into pure tickWatchedPid() and requires WATCHER_DEAD_CONFIRM_TICKS (=3) CONSECUTIVE failed probes (~4-6s at the 2s tick) before removal; any live probe resets the streak. Heartbeat-expiry removal (#3544) keeps its immediate semantics (unit-pinned). First failure of an episode logs [watchdog] kill-0 probe failed for pid N (code=…) and recovery from a false alarm logs one line — field instrumentation so the next Windows episode can be attributed by errno.

Mechanism 2 — unconditional 10s force-exit cuts live streams

shutdown()'s hard backstop fired process.exit(0) 10s after the signal regardless of live connections. Fix: while connRecords.size > 0 the window RE-ARMS, bounded by SHUTDOWN_DRAIN_MAX_PASSES (=10 → worst-case total drain ~110s); then it force-exits exactly as before (zombie backstop preserved). Idle keep-alives don't hold the proxy — the keep-alive reaper clears them within one window, so a healthy drain still exits promptly.

In-passing fix (separate commit)

src/agent/opencode-native.ts oversizeDirectRefusal: "Recovery is automatic (#3308); retry shortly" implied an automatic retry that never happens — the respawn does not re-send this request. Wording now says plainly: resubmit once it's back.

Behavior changes (old → new)

Surface Old New Why
owner-death → exit latency 1 failed probe ⇒ remove ⇒ exit ~6-7s 3 consecutive failed probes (~4-6s) ⇒ remove ⇒ exit ride out transient OpenProcess failures
shutdown backstop unconditional force-exit at 10s re-arms up to 10 extra passes while liveConns > 0 (worst-case ~110s), then force-exit as before stop cutting in-flight upstream streams mid-response
oversize refusal text "retry shortly" (implies auto-retry) explicit resubmit instruction honesty — nothing auto-retries
log lines — new info/warn lines listed above field attribution of Win32 errno

Companion test re-timing (intentional, documented in comments): claude-native live-lane deadlines 15s→30s; watcher-route grace phase 3s/7s waits → 8s/8s (late registration now lands after the ~6s removal but inside the following 5s idle grace — still exercises pending-shutdown cancellation). These lanes are ACP_TEST_CLAUDE_NATIVE=1-gated, off in default CI.

Verification

No config surface change (hardcoded constants in the existing WATCHER_* style — tunability can be discussed later if wanted). No kernel/ change. No wire/persistence change.

Closes #3890

@ranxianglei ranxianglei changed the title fix(server): confirm parent death over consecutive probes; drain before force-exit (#3890) [PR] fix(server): confirm parent death over consecutive probes; drain before force-exit (#3890) Oct 20, 2026
awork added 2 commits October 20, 2026 11:40
…re force-exit (#3890)

Two independent defects made the proxy self-terminate on Windows while its
parent lived on (12 events in 2.5h against one live pid):

1. The watchdog removed a watched pid from the set on a SINGLE failed
   kill-0 probe; the empty-set grace window that followed never re-probed,
   so one transient OpenProcess failure (Windows maps every non-ACCESS_DENIED
   failure to ESRCH — indistinguishable from a real death) force-exited the
   proxy ~6s later. Death now requires WATCHER_DEAD_CONFIRM_TICKS (=3,
   ~4-6s at the 2s tick) CONSECUTIVE failed probes; any live probe resets the
   streak. Heartbeat-expiry removal (#3544) keeps its immediate semantics.
   First failure of an episode logs the errno ([watchdog] kill-0 probe failed
   for pid N (code=...)) and recovery from a false alarm logs one line, so
   field episodes can be attributed. probePid() in instance.ts carries the
   errno; isPidAlive() is unchanged.

2. shutdown()'s hard backstop force-exited 10s after the signal regardless of
   live connections, cutting in-flight upstream streams mid-response. While
   connRecords are still alive the window now RE-ARMS, bounded by
   SHUTDOWN_DRAIN_MAX_PASSES (=10, worst-case total drain ~110s); then it
   force-exits exactly as before (the zombie backstop is preserved). Idle
   keep-alives don't hold the proxy — the keep-alive reaper clears them within
   one window, so a healthy drain still exits promptly.

Behavior changes (old -> new):
- owner-death -> exit latency: +up to ~6s (confirmation window)
- shutdown backstop: unconditional 10s force-exit -> re-arms up to 10 extra
  passes while liveConns > 0, worst-case ~110s total, then force-exit as before
- new log lines: [watchdog] kill-0 probe failed / pid N alive again /
  extending drain k/10

Tests: new suite pins tickWatchedPid (blip <3 survives, 3rd consecutive
removes, recovery resets, heartbeat path unchanged, dead pids never take the
heartbeat fast path) plus a LIVE child-process test proving the backstop no
longer force-exits at 10s with a connection held open (verified to FAIL on
pre-fix master: 'pre-#3890 behavior force-exited at 10s'). claude-native
live-lane deadlines re-sized 15s->30s and the watcher-route grace phase
re-timed for the wider confirmation window (ACP_TEST_CLAUDE_NATIVE-gated).
…3890)

Labeled in-passing fix found while working #3890: the oversizeDirectRefusal
message told clients 'Recovery is automatic (#3308); retry shortly' — but the
proxy respawn does NOT re-send this request, so 'retry shortly' implied an
automatic retry that never happens (the issue's own damage trail shows these
refusals going unanswered during each ~51s gap). Wording now states plainly
that the request must be resubmitted once the proxy is back.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 按失败签名(parent-gone 误判 / Windows probe / drain force-exit)检索 open+closed issue 与 PR:唯一同源条目是 #3890(本 PR 的原始 issue);无并列修复 PR。最接近的 #3079(短命 CLI 客户端让 session-owned proxy 提前退出)机制不同——那是 watcher 生命周期归属问题,不是 Windows kill-0 探测误判,不构成重复。

标签: bug opencode scope/wiring p1(沿用 #3890 判定:父进程存活时 proxy 反复自杀 = p1 级核心承诺失灵;opencode v2 desktop 客户端;watchdog/生命周期接线层)。

本 PR 由本 daemon 早前会话(ses_7fa1f7e0…)产出,现在做独立复核:代码与描述逐项核对、定向测试、CI 状态核查。另注意:分支与当前 master 有真实内容冲突(src/server.ts,master 已前进 3 个提交 #3858/#3865/#3892),系统曾转发 conflict-worker 但截至我拉取时 head 未变——我会自行 rebase 解决并推送,随后给出完整八问审核结论。

@ranxianglei
ranxianglei force-pushed the 2026-10-20_win32-parent-gone-confirm-drain branch from c11bdfd to ec1f0b6 Compare October 20, 2026 03:44
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

上游冲突已解决,rebase 完成并推回分支。结构化结论(八问):

  1. 是什么问题:PR 分支与 master 冲突(merge state dirty)。master 前进期间重写了 src/server.ts 对 ./instance.js 的导入行——把 findSameLaneTwin 换成了 classifySameLanePeer/sameInstanceShape/entryShape 这套同车道对等实例分类 API——而本 PR 在同一行新增了 probePid/PidProbe,两边撞在同一行上。
  2. 什么原因导致的:纯文本级并发冲突,不是语义分歧。两条工作线各自改了同一份导入列表(master 侧是 [PR] fix: restore public registry URLs for two lockfile entries #3892 合入前后那次 registry/peer-classification 重构,本 PR 侧是 kill-0 连续探针确认的新导出)。
  3. 影响范围与严重程度:只影响本 PR 的可合并性,不影响任何运行时行为。冲突解决未改动任何代码逻辑。
  4. 怎么修复的:rebase 到当前 master(838e9758a)。唯一冲突行按「两侧并集」解决:保留 master 引入的 classifySameLanePeer、entryShape、sameInstanceShape、writeSpawnPortLog、InstanceShape、SameLanePeer,保留本 PR 新增的 probePid、PidProbe、RegistryEntry,删掉 master 已从 instance.ts 移除的 findSameLaneTwin(已用符号使用计数逐一核对:保留的每个符号在文件体内都有真实使用,无未用导入)。force-with-lease 推回原分支。
  5. 是否有回归问题:rebase 自身零回归。合并后 git diff origin/master..HEAD 与 rebase 前完全同形(6 文件,+315/-20),说明除 master 自己的重构外没有任何额外改动混入。本地验证:typecheck 干净、build 成功(head ec1f0b6fe)、定向测试全绿(tests/issue3890-parent-gone-confirm-drain.test.ts + tests/claude-native.test.ts:64 pass / 0 fail / 8 个门控跳过)。全量回归按纪律交给远程 CI。唯一需要点名的语义变化是 findSameLaneTwin → classifySameLanePeer 这一处替换——它来自 master 自身的重构,不是本次 rebase 引入的。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有。

判定:冲突已解,GitHub 侧 mergeable: true(mergeable_state 当前 blocked = 等 rebase 后的 CI 重跑)。CI 绿后请人工合并。

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_win32-parent-gone-confirm-drain (864f149)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4635-pr.3902.8f738d2

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3902.tgz
npm install -g package

Install instructions are refreshed on each push.

… gate)

The rebased #3902 head trips the unused-exports static gate: WatcherTickDecision is referenced only inside src/server.ts (as tickWatchedPid's return type; the unit suite consumes decisions structurally). Drop the leading export per the gate's INTERNAL_ONLY remedy. Type-only change: gate, typecheck, build and the issue3890 suite are green.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4572(master 合并,merge commit 保留原作者署名)。本 PR 关单收尾;回滚点为 #4572 单提交 revert。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant