Skip to content

[PR] fix(launcher): refuse attach to orphaned armed proxies and reap them before spawn (#6257) - #6427

Open
ranxianglei wants to merge 1 commit into
masterfrom
2026-11-12_orphan-attach-gate
Open

ranxianglei wants to merge 1 commit into
masterfrom
2026-11-12_orphan-attach-gate

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What this fixes

A resident proxy whose declared spawner (BILI_PARENT_PID) has died and which has no other live watcher is a zombie. On macOS a detached proxy has lost Local Network authorization, so every LAN upstream fails with EHOSTUNREACH (cloud upstreams are fine) while the process keeps running — and a host restart re-attaches to it, so the condition never heals. The only user-side recovery is killing the proxy pid by hand. Reported three times on one machine (0.1.213, DSH desktop lane).

Root cause: the #1335 attach gate (attachGateAllows) checks only watchdog.armed, which is a boot-time constant (armed: initialWatcherPid !== null in src/server/admin.ts) — it stays true forever after the parent dies. The launcher's HealthInfo even dropped the parentPid field the server already reports, so the liveness check was impossible at the gate.

Closes #6257. Related: #4367 (attach gate owner-liveness design), #4705 (re-attach reuses stale helper), #5663 (macOS Local Network authorization, root cause 2).

Changes

  1. HealthInfo.watchdog carries parentPid — the server already reports it; the launcher now parses it (typeof wd.parentPid === "number"; absent on pre-[Bug][dsh lane] 0.1.213 macOS: a detached proxy that outlives the host keeps running but loses Local Network authorization (all LAN upstreams EHOSTUNREACH, cloud fine), and host restart re-attaches to it so it never heals | 守护进程跨宿主存活后失去本地网络授权,attach 复用旧进… #6257 builds = unverifiable).
  2. isOrphanedWatchdog(health, selfPid, isAlive) (exported, pure): armed === true AND parentPid present AND parentPid dead AND every other watcher dead. Self is excluded (a previous bring-up may have registered us); a live co-owner or live parent keeps the candidate attachable (legitimate multi-host sharing, kimi native 共享代理存在与 #1183 同类的「首 owner 退出杀共享代理」暴露面(无 watcher 注册) #1190).
  3. pickAttachable refuses orphaned candidates after the claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335 gate, logged once per origin with the reason and the escape hatch. native.attachExternal / BILI_NATIVE_ATTACH_EXTERNAL=1 stays fully permissive (explicit opt-out, unchanged).
  4. reapOrphanedPredecessor (new): before the spawn, a same-lane orphan holding the target port gets a supervised graceful shutdown — re-probe right before acting (abort if a live owner appeared in the meantime), POST /__bili/shutdown, then poll a fresh (no-keep-alive) connection until it is refused or STALE_ROTATION_WAIT_MS (12s) elapses. No raw kill (the pid could be reused). Any doubt falls through to the untouched spawn ladder. This exists because a bare gate refusal would leave the pinned port held: the spawn would hit EADDRINUSE and ladder to a sticky +1 port drift ([Design] 定死端口 lane(claude 48787 / zcode 48789)端口被占时的 0 配置解法 #1660).
  5. strictPort squatter check no longer throws "kill that process" for an orphaned armed holder — it is not lifecycle-less; the reap (or the CI flake: claude hook e2e "manual bili start on the pinned port is attached, not refused" — attach-vs-spawn race when the squatter's health endpoint lags TCP accept (#1322/#1660) #1903 rediscovery, if the reap aborts because a live owner appears) converges it.
  6. Server watchdog warn: one-time log when the declared parent dies but adopted watchers keep the proxy serving — names the detached-proxy consequence on macOS.

Behavior changes (old → new)

Area Old New Why
Attach gate watchdog.armed === true → attach + declared owner alive or another live watcher; otherwise refuse (logged, escape hatch attachExternal: true) An armed resident with a dead owner is a zombie that lost macOS Local Network authorization; re-attaching never heals it (#6257, 3 occurrences)
Spawn on pinned port with an orphaned same-lane holder EADDRINUSE → port ladder +1 drift (#1660) Supervised graceful shutdown of the orphan first (re-probe + 12s poll, no raw kill), then spawn on the canonical port The gate refusal must not manufacture port drift; the orphan's port is exactly the one the session wants
strictPort squatter error Threw "kill that process" for any compatible holder the gate would refuse Excludes orphaned armed holders (reap/rediscovery converges them) Advising kill on a proxy that may serve other sessions is wrong; the new reap is the sanctioned path
Proxy log Silent after parent death while adopted watchers keep it serving One-time warn naming the detached-proxy consequence The user's only signal today is the 502s; the warn points at the mechanism

Unchanged: attachExternal semantics (fully permissive); the #1335 gate for non-orphaned residents; the #3544 version-strictly-newer rotation (orthogonal, still runs before the reap); the #5955 wedged-holder takeover; the watchdog's own shutdown logic (a proxy that loses ALL watchers still shuts down as before).

Windows pid-reuse caveat: kill-0 is existence-blind — a recycled pid can read as "alive", making the orphan undetectable in that window (miss, not false-positive). Same limitation as the existing watchdog probes; the #4367 heartbeat design is the root fix and this gate is its strict subset.

Tests

Review tier

Touches session-binding / lifecycle default behavior — human review required (§7.4), no auto-merge.

@ranxianglei ranxianglei changed the title fix(launcher): refuse attach to orphaned armed proxies and reap them before spawn (#6257) [PR] fix(launcher): refuse attach to orphaned armed proxies and reap them before spawn (#6257) Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-12_orphan-attach-gate (21e3423)

Latest build — install this exact version

npm install -g billion-context@0.0.9328-pr.6427.94fce7c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
21e3423 billion-context@0.0.9328-pr.6427.94fce7c build
e616266 billion-context@0.0.9323-pr.6427.15f5d4e build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6427.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — open [PR] rows checked (115 total); this PR is the only one targeting #6257. Related-but-distinct: #6358 (stale helper config on re-attach, #4705) and #5835 (reap wedged orphaned proxies off-loop, #5449) touch adjacent lifecycle surfaces but different mechanisms/issues — no overlap with this thread's scope.

Status as I pick this up:

Now doing the full review (code + regression audit against the touched paths). Will follow up with the structured conclusion.

…before spawn (#6257)

An armed resident whose declared spawner (BILI_PARENT_PID) is dead and which
has no other live watcher is a zombie: on macOS a detached proxy has lost
Local Network authorization, so every LAN upstream fails with EHOSTUNREACH
while it keeps running -- and a host restart re-attaches to it, so it never
heals. The #1335 attach gate only checked watchdog.armed, which stays true
forever for such a proxy.

- HealthInfo.watchdog now carries parentPid (the server already reports it;
  the launcher dropped it) and parses it.
- New isOrphanedWatchdog(health, selfPid, isAlive): armed + parent dead + no
  other live watcher. Self is excluded (a previous bring-up may have
  registered us); a live co-owner or live parent keeps the candidate
  attachable; a missing parentPid on older builds is unverifiable and not
  orphaned.
- pickAttachable refuses orphaned candidates after the #1335 gate (logged
  once per origin with the #6257 reason and the attachExternal escape
  hatch); attachExternal stays fully permissive.
- New reapOrphanedPredecessor: before the spawn, a same-lane orphan holding
  the target port gets a supervised graceful shutdown (/__bili/shutdown,
  re-probe first, poll until a fresh connection is refused, no raw kill) so
  the spawn binds the canonical port instead of laddering to a sticky port
  drift (#1660). Any doubt falls through to the untouched spawn ladder.
- The strictPort squatter check no longer throws 'kill that process' for an
  orphaned armed holder (it is not lifecycle-less; the reap or the #1903
  rediscovery converges it).
- The server watchdog logs one warning when the declared parent dies but
  adopted watchers keep the proxy serving (the detached-proxy hint).

Related: #4367 (attach gate owner-liveness design), #4705 (re-attach reuses
stale helper), #5663 (macOS Local Network authorization).
@ranxianglei
ranxianglei force-pushed the 2026-11-12_orphan-attach-gate branch from e616266 to 21e3423 Compare November 12, 2026 13:28
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review complete. Pulled the branch, read the full diff against master, verified every claimed pre-existing symbol, and audited the touched paths for regressions.

What I did to the branch: it was 43 commits behind master, so the full test/e2e CI lanes were skipped (only the fast guards ran — all green). I rebased the single commit onto current master (clean, no conflicts) and pushed it back to the PR branch (e616266 → 21e3423). CI is now running the full suite on the rebased head (12 pending / 0 failed so far).

Code review (the main gate):

Regression audit (existing behaviors on the touched paths):

Tests: 7 new tests cover the truth table (7 rows), gate refusal + diag assertion, live co-owner, live parent, attachExternal override, reap happy path, and reap abort. Real assertions (not tautologies) that pin the behavior. The full-suite verdict is CI's (running on the rebased head now).

8问 (structured checklist):

  1. 是什么问题: 宿主重启后 attach 到跨宿主存活的孤儿代理,该代理在 macOS 上已丢失 Local Network 授权(所有 LAN 上游 EHOSTUNREACH、云端正常),反复 attach 永远无法自愈,用户只能手动 kill pid。
  2. 什么原因导致的: claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335 attach 门只查 watchdog.armed,而 armed 是启动时常量(initialWatcherPid !== null),父进程死后永远为 true;launcher 的 HealthInfo 还丢掉了服务端已上报的 parentPid 字段,导致门处无法做存活判定。
  3. 影响范围与严重程度: 仅 macOS + DSH desktop lane + 宿主重启后 attach 到孤儿代理的场景;严重——LAN 上游完全不可用(502),云端上游正常,可手动 kill 恢复,属"特定条件下核心功能失灵"。
  4. 怎么修复的: 门处增加孤儿判定(armed 且声明 owner 已死且无其他存活 watcher,自身 pid 排除)→ 拒绝 attach;spawn 前对占用规范端口的同 lane 孤儿做受监督优雅关闭(重新探测 + POST /__bili/shutdown + 12s 轮询,不裸 kill),释放端口后再 spawn;strictPort squatter 提示排除孤儿;服务端加一次性 warn。
  5. 是否有回归问题: 逐项审了 claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335 门(非孤儿不变)、attachExternal(不变)、[Design] 定死端口 lane(claude 48787 / zcode 48789)端口被占时的 0 配置解法 #1660 user-zone(手动守护无 BILI_PARENT_PID → 不受影响)、安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 版本轮换 / [Bug][dsh] 0.1.212: Zombie proxy holds port + fingerprint branch only skips without terminating → oversized sessions stu | [Bug][dsh] 0.1.212: 僵尸代理占端口 + fingerprint 分支只 skip 不 terminate -> 超大会话持续 422(BUSY 判定 × EADDRINUSE 死锁) #5955 wedged 接管 / CI flake: claude hook e2e "manual bili start on the pinned port is attached, not refused" — attach-vs-spawn race when the squatter's health endpoint lags TCP accept (#1322/#1660) #1903 再发现 / watchdog 自身全 watcher 死亡关闭(均不变)。一处窄行为变更:长生命周期宿主(同 pid)自身是孤儿代理唯一存活 watcher 时,下次 bring-up 会拒绝并收割该代理,断掉其自身活跃会话——多进程场景、旧会话本已降级(LAN 失效),可接受,已记录。
  6. 是否破坏缓存: 不涉及——纯 launcher 生命周期 + 服务端日志,不碰缓存/重试/增量游标/去重键。
  7. 是否改了设计方向: 不涉及——在既有 attach 门/轮换/接管机制上增量加一道孤儿判定,未改架构/边界/数据流。
  8. 是否引入了新的配置: 无新配置。复用既有 native.attachExternal / BILI_NATIVE_ATTACH_EXTERNAL=1 作逃生舱(语义不变)。

判定: 代码正确、测试扎实、回归面已逐项清点,唯一窄行为变更可接受且已披露。可合——待 rebase 后 CI 全量 test lane 落绿(当前 12 pending / 0 failed)。分支已 rebase 到当前 master 并推送(21e3423),mergeable 状态等 CI 定。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI landed green on the rebased head (21e3423, base = current master 9674cfa): full test matrix (11 shards, ubuntu/windows × node 22/24) + static-gates + mysql-contract + all fast guards = 14 success / 16 skipped / 0 failed. The e2e lanes are skipped by their gates (this PR's only src/server.ts change is a log-only watchdog warn — no wire/pipeline logic; the launcher attach/reap logic is covered by the unit suite that just passed).

PR state: mergeable: true (no conflicts), mergeable_state: blocked — the block is the human-review requirement (§7.4, per the PR's own "no auto-merge" note), not a CI failure. Nothing left on the agent side; ready for you to approve + merge: #6427

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/wiring

Projects

None yet

1 participant