Skip to content

[PR] fix(launcher): adopt proven-healthy child at spawn timeout instead of killing it (#4994) - #5010

Merged
ranxianglei merged 13 commits into
masterfrom
2026-10-28_spawn-timeout-adopt-healthy-child
Oct 29, 2026
Merged

ranxianglei merged 13 commits into
masterfrom
2026-10-28_spawn-timeout-adopt-healthy-child

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4994

What this fixes

Facet ④ of #4994 (the untracked one — facets ①②③ are covered by #3890/#4572 and the merged PR #4825): when the 60s bring-up budget expires, the spawner killed its own spawned child unconditionally. In the reported incident the health probe on the preferred port was already answering with pid == our child's pid ("port 18790 finally answers healthy from our child (pid 50948) but its instance handshake never matched") — i.e. the child was bound and serving, only the launch-token handshake never matched. The spawner then taskkill /F /T'd that healthy proxy, nothing retried the bootstrap in that window, and the lane degraded to direct uncompressed traffic until a host restart.

Behavior change (old → new)

  • Old: at budget expiry the timeout path always ran killAbandonedChild(child) + threw proxy did not become healthy within 60000ms.
  • New: if the SAME evidence diagnosePort already collected proves the victim is our own healthy child (health ok AND pid === child.pid on the preferred port), the spawner settles the zone sticky port, starts the owner heartbeat, attaches to it and returns it as the handle — no kill. A second rung covers the EADDRINUSE-ladder case (child re-bound on another port per src/server.ts:1381-1420): the child's own pid-tagged listen line in its per-port log (bytes written since spawn) locates the real origin, adopted under the same strict proof.
  • Unchanged: every unproven case keeps today's behavior byte-for-byte — same kill, same error strings (diagnosePort now returns { text, selfHealthy }; all user-facing prose is identical). No config surface changes. No kernel changes.

Regression tests

New tests/issue4994-spawn-timeout-adopt.test.ts (fake-clock driven, modeled on the #2954 suite): rung-1 adopt, rung-2 ladder adopt, unproven-still-killed, foreign-responder-never-adopted. Verified the two adopt rows FAIL against pre-fix code (child SIGKILL'd) and pass with the fix; the two negative rows pass on both. Adjacent suites green locally: issue2954 / issue2187 / issue3116 / issue4850 (20/20), typecheck + build clean. Full-suite verdict = remote CI.

…tead of killing it (#4994)

When the 60s spawn budget expires, the spawner killed its own spawned
child unconditionally. If the health probe on the preferred port already
answered with pid == our child's pid (the instance handshake merely never
matched), that kill destroyed a proxy that was bound and serving — and
the lane then degraded to direct uncompressed traffic until a host
restart.

- diagnosePort now returns { text, selfHealthy } so the timeout path can
  act on the same evidence it already collected; all user-facing strings
  are byte-identical to before.
- Rung 1: selfHealthy (health pid match on the preferred port) -> settle
  the zone sticky port, start the owner heartbeat, attach and return —
  no kill.
- Rung 2: the child may have laddered to another port after EADDRINUSE
  (src/server.ts ladder); find it via its own pid-tagged listen line in
  the per-port log (bytes written since spawn) and adopt it with the
  same strict proof (health ok + pid match).
- Unproven cases keep today's behavior exactly: burn the budget, taskkill
  /F /T, throw the same error text.
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-28_spawn-timeout-adopt-healthy-child (9180053)

Latest build — install this exact version

npm install -g billion-context@0.0.6807-pr.5010.812f1ac

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
9180053 billion-context@0.0.6807-pr.5010.812f1ac build
e331f44 billion-context@0.0.6800-pr.5010.1e80da2 build
52fbf56 billion-context@0.0.6799-pr.5010.e87dfe7 build
a5d0d81 billion-context@0.0.6753-pr.5010.5734693 build
2ec4a77 billion-context@0.0.6708-pr.5010.cfb2ba1 build
3a9bfe5 billion-context@0.0.6697-pr.5010.f8669a0 build
7c397e1 billion-context@0.0.6611-pr.5010.c910693 build
20e81b4 billion-context@0.0.6582-pr.5010.ba335d4 build
6710580 billion-context@0.0.6573-pr.5010.ed3fc83 build
082b5d4 billion-context@0.0.6570-pr.5010.9701428 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5010.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(launcher): adopt proven-healthy child at spawn timeout instead of killing it (#4994) [PR] fix(launcher): adopt proven-healthy child at spawn timeout instead of killing it (#4994) Oct 27, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 修的是 #4994 的 facet ④(60s 预算耗尽后超时路径无条件 taskkill 自己已证明健康的子进程);facet ①看门狗单次误判(#3890→#4572)、②give-up 后无重试(#4755/#4811→#4825)、③旧日志头采样(#4811→#4825)均有独立条目且已合 master,与本 PR 无机制重叠。检索 open+closed issue/PR(spawn timeout、killAbandonedChild、adopt healthy child 等签名)未见其他在途条目针对同一机制——这是 facet ④ 的唯一在途 PR。PR 头 5ae293d91 直接坐在当前 master head 9b6ec2345 上,无 stale base。

标签: bug / dsh / scope/wiring / p1 —— 压缩车道整体停摆到宿主重启 = 可用性类 p1;触发场景为 dsh 车道,修复点在 launcher bring-up 接线层。

本 PR 带 ework-agent-pr 标记(自有 agent 产出),不重做工作:正在拉分支核对代码(既有行为清单回归审查)+ 查 CI 状态,完成后按结构化八问给结论。

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 27, 2026
ci-check added 2 commits October 28, 2026 05:14
… timeout adoption rung 2 can find it (#4994)

Review finding on 5ae293d: rung 2 of the timeout adoption parses the child's pid-tagged listen line out of the per-port log, but post-#2864 a launcher-spawned child mutes its stderr leg into bili.log once the file leg is live - so the banner (the only producer of that line) never reaches the port log and rung 2 was unreachable for a healthy child in the primary starvation shape (clobbered records, healthy state dir). The listen-failed paths already carry this explicit write for exactly this reason; mirror it on the success path (one short line per spawn, best-effort append).

Behavior change (additive): launcher-spawned children now append ONE listen-success line to %TMPDIR%/bili-proxy-<port>.log per spawn (previous content unchanged; rotation at PORT_LOG_MAX_BYTES untouched). No config surface, wire, or persistence-format changes.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) CI 终局补记:最终 head 082b5d4a2 全套 30/30 全绿(此前因 LIVE_FILES tripwire 全红的 10 条 test 车道已随登记修复恢复)。分支就绪,可合并——merge 按钮留 owner。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Synced origin/master into this branch — clean merge, no conflicts. Head is now 6710580; mergeable_state went behind→blocked (required checks still running, none failed yet).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Push landed once connectivity recovered — head is now 7c397e1 (origin/master @ b08fbf3 merged in cleanly, #4994 changes intact). Post-push state: mergeable, no conflicts, behind→blocked (required checks running on the new head, none failed).

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

Synced this branch to current master. The only conflict was scripts/live-files.mjs — both sides appended a port-zone entry (#4982 draining-twin-ladder from master, #4994 spawn-timeout-adopt from here); resolved as the union keeping both entries in their original order, so no suite loses its zone band.

Local verdict on the merged head 3a9bfe52: typecheck clean; targeted suites green (issue4994-spawn-timeout-adopt 4/4, port-owner 24/24). Full-suite verdict = CI lanes on this push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Verified the sync above: scripts/live-files.mjs union confirmed — both zone entries present at head 3a9bfe5 (#4982 draining-twin-ladder and #4994 spawn-timeout-adopt), no suite lost its band. Master then picked up two trivial README-only commits (#5087/#5088 QQ group info) which I merged + pushed as well: head is now 2ec4a77, fully current with master — state behind→blocked (required checks running on the new head, none failed).

The previous sync commit resolved the live-files.hunk to the master side and dropped this PR's own registration row; CI's tripwire would have red every shard for it.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

同步一条:本分支已并入当前 master 推了 9180053(无冲突;scripts/live-files.mjs 五条 spawn 测试条目 #4982/#4994/#5012/preload-sandbox-dirs/event-loop-watchdog(如 master 侧已有则保留不重复) 校验无重复,共 54 条)。本地 tsc 干净、issue4994-spawn-timeout-adopt 4/4。顺带澄清此前一处读取坑:用 8 位短 SHA 查 actions/runs?head_sha= 会返回空,需完整 40 位——按完整 SHA 查,上一头 e331f44 的主 CI 其实是绿的(它 14:55:04Z 触发时 master 已含 cf99d93)。本轮 CI 结果以新 push 为准。

@ranxianglei
ranxianglei merged commit 3a5d532 into master Oct 29, 2026
55 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.916-master.3a5d532 (rolling master tag).

npm install -g billion-context@0.0.916-master.3a5d532  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dsh lane: parent watchdog false-positives kill healthy proxy; spawner then gives up permanently

1 participant