Skip to content

[PR] fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port (#5955) - #5972

Merged
ranxianglei merged 3 commits into
masterfrom
2026-11-08_wedged-same-lane-takeover
Nov 9, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-11-08_wedged-same-lane-takeover

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Closes #5955

Problem

The attach/rotation decision table had one uncovered cell: a same-lane resident on the canonical port running DIFFERENT bytes whose health probe never answers (BUSY — TCP connect succeeds while the event loop stays inside long sync slices, e.g. a multi-minute fold on an oversized session). Every existing surface skips such a holder:

No actor ever resolved the holder, so the port stayed wedged until the process died on its own — which the event-loop watchdog only guarantees for ≥5min continuous blocks; the saturation class never trips it. Reporter evidence (dsh 0.2.2-rc.1, Windows, v0.1.212): a zombie held 18787 for ~10+ minutes while every bootstrap round failed with 422 refusals, then the lane landed permanently on 18788.

Design

Bounded patience across bring-up cycles, then a supervised forced takeover:

  1. Cell check (isWedgedSameLaneCandidate, pure): same declared lane, our target port, fingerprint mismatch BEFORE any shape verdict, health silent + TCP alive. Anything that ANSWERS health is someone else's cell ([Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 adoption / 安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 rotation); matching bytes are attachable per dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811.
  2. Persisted observation ledger (wedged-takeover.json in the state dir, atomic write): during a wedge the client restarts every failed bootstrap round, so an in-process counter would reset each cycle and the window could never fill. Keyed origin#pid#birth-stamp (linux /proc/<pid>/stat starttime / win32 powershell StartTime / other POSIX ps lstart) so a REUSED pid starts its own window instead of inheriting the dead occupant's history (Windows recycles pids fast). Corrupt/missing file degrades loudly to a fresh window; write failure never blocks bring-up.
  3. Patience: WEDGED_TAKEOVER_WINDOW_MS = 600_000 (10 min = 2× the watchdog hard timeout) of continuous observations across ≥3 distinct cycles; an inter-observation gap >5min restarts the window. A user-visible line announces the window start (without it the user sees only 422s for 10 minutes).
  4. Takeover ladder (stage budgets field-tuned after the reporter's acceptance run — see below): graceful POST /__bili/shutdown with a 1s request budget (insurance for a loop that briefly unblocks) → wait 2s for the listener to close → SIGTERM (POSIX) + 1s → hard kill (POSIX SIGKILL / win32 taskkill /F /T, shared hardKillProxyTree) + 2s. Only after VERIFIED release does the normal spawn proceed; an abrupt kill gets a 2s settle for kernel socket state before the child binds (residual TIME_WAIT must not push the replacement to +1; measured release is ≤252ms, so this keeps an 8× margin on a rare path). Kill failures keep the ledger entry and fall through to the untouched spawn ladder — never worse than today.

Rationale for patience over immediate kill: BUSY may be slow-but-alive mid legitimate multi-minute fold; immediate kill loses in-flight work + cold start. The window bounds worst-case unavailability from ∞ to ~10min. Data-loss bound: the victim loses at most its in-flight turn (sessions persist per-turn).

Behavior changes (old → new)

Surface Old New Why
Wedged same-lane holder on canonical port Port stuck until the process dies on its own (∞ for the saturation class); sessions 422 forever; lane drifts to +1 sticky ports Forced takeover after 10 min of continuous observations (≥3 cycles, persisted across restarts); spawn proceeds on the freed port The deadlock cell had no resolver — availability decision (§7.3 availability-over-performance)
State dir no wedged-takeover.json New file, auto-unlinked when empty Ledger must survive client restarts
killAbandonedChild group kill + handle SIGKILL Same behavior, refactored onto the shared hard-kill helper; ESRCH (pid already gone) no longer warns on POSIX and win32 Dedup with the new kill path; master logged nothing for an already-gone pid either — the win32 false alarm fired on EVERY successful takeover in the field run
Wedged-takeover escalation ladder 3s shutdown + 5s wait + 2s term + 5s wait ≈ up to ~15s of pure spin against a structurally unresponsive holder 1s shutdown + 2s wait + 1s term + 2s wait ≈ up to ~6s to the release verdict (+2s settle unchanged) Field-measured: the port released in ~250ms after the hard kill — a holder silent on health for 10 minutes will not complete an HTTP round-trip in 3s; the graceful step is kept as bounded insurance (#3544 rotation keeps its own 3s default via the new optional timeout parameter)

No new config/env surface — all constants hardcoded (event-loop-watchdog precedent; Env Var Discipline). No wire/format changes.

Field-test feedback incorporated (commit 2826c51)

The issue reporter ran all three acceptance points on Windows 11 / node v24.15.0 against this branch: takeover PASS (port 18787 released, ledger entry deleted, replacement healthy), re-bind PASS (no more +1 drift), and the 422 trigger eliminated (oversize refusals only fire while the proxy is down). Two findings from that run are fixed here:

  1. ~12.4s of pure budget spin before the hard kill actually released the port (~250ms) → stage budgets tightened (table above); worst-case release verdict drops from ~15s to ~6s.
  2. False-alarm warn on every win32 takeover (direct SIGKILL … failed (kill ESRCH) — its listener may survive) → ESRCH suppressed in hardKillProxyTree's win32 fallback (taskkill already reaped the tree = goal met), mirroring the POSIX branch.

Out of scope (adjacent cells, deliberately left alone)

  • Same-bytes / shape-drifted BUSY holders (config edited mid-session): the child ladders away — degraded but working. Killing our own build is riskier and not needed for the reported defect.
  • CPU-time liveness probe (skip patience when CPU time is flat): real false-positive surface (an I/O-stalled legitimate fold looks identical to a hung one; a mis-kill loses the in-flight turn) — parked, not filed here.
  • Oversize-refusal as hard observation (compress the window to ≤60s once the gate refuses): needs a client→launcher cross-process channel design — separate issue.
  • Refusal-message operability / lane-stickiness observability / Fold-state reconciliation: classify resent-history divergence (edit/fork/new/system-only) and make compression blocks re-anchorable chain units #1921 fold-drift re-anchor: separate issues with 来源 markers.

Related work

Tests

Review tier

Human review required — a new kill path against a still-serving process is a product decision (auto-merge ineligible: new lifecycle behavior, new state file, kill semantics).

…canonical port (#5955)

The attach/rotation decision table had one uncovered cell: a same-lane
resident on the canonical port running DIFFERENT bytes whose health probe
never answers (BUSY: TCP connect succeeds while the event loop stays inside
long sync slices, e.g. a multi-minute fold on an oversized session). Attach
vetoes foreign bytes (#1225), #4125's upward-adoption veto-lift needs an
ANSWERED version a BUSY holder cannot give, #3544 rotation excludes
version-unverifiable residents by design (#4811), and the spawned child waits
5s behind a same-lane predecessor then ladders away (+1 sticky drift) or dies.
No actor ever resolved the holder, so the port stayed wedged until the process
died on its own — which the event-loop watchdog only guarantees for >=5min
CONTINUOUS blocks; the saturation class never trips it. Oversized sessions
then 422'd indefinitely on the stuck lane.

Resolution: bounded patience across bring-up cycles, then a supervised forced
takeover. A persisted observation ledger (state dir, atomic write, keyed
origin#pid#birth-stamp so a REUSED pid starts its own window instead of
inheriting the dead occupant's history) accumulates sightings across client
restarts. After WEDGED_TAKEOVER_WINDOW_MS (10min = 2x the watchdog hard
timeout) of continuous observations across >=3 distinct cycles, the launcher
requests a graceful shutdown, terminates if the listener survives, hard-kills
as last resort, verifies the listener closed, gives kernel socket state a
settle after an abrupt kill, and continues to the normal spawn. Any doubt
falls through to the untouched spawn ladder — never worse than today.

Adjacent surfaces deliberately unchanged: #3544 rotation (healthy older
holders), #4125 upward adoption (healthy newer holders), #4811 same-lane
attach (matching bytes), user-zone and foreign-lane holders all keep their
existing paths. killAbandonedChild keeps its exact behavior (group kill +
handle-level SIGKILL) via the new shared hard-kill helper.
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-08_wedged-same-lane-takeover (2826c51)

Latest build — install this exact version

npm install -g billion-context@0.0.8697-pr.5972.76ce338

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
2826c51 billion-context@0.0.8697-pr.5972.76ce338 build
e7d7179 billion-context@0.0.8689-pr.5972.20e0196 build
33bec44 billion-context@0.0.8682-pr.5972.13765a8 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5972.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Nov 8, 2026
@ranxianglei ranxianglei changed the title fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port (#5955) [PR] fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port (#5955) Nov 8, 2026
wedgedLedgerFilePath() is referenced only inside launcher.ts — the
deps-injection fallback at line 5691 — while tests supply their own path
via deps.wedgedLedgerPath. The leading `export` made it INTERNAL_ONLY under
scripts/check-unused-exports.mjs. Drop it to match the sibling module-local
ledger helpers (isWedgedLedgerEntry, readWedgedLedger). No behavior change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status (live-verified): PR #5972 is open, head branch 2026-11-08_wedged-same-lane-takeover sits directly on the current master tip (848f59b9) — no rebase needed. Labels already in place (bug / p1 / dsh / scope/wiring).

CI red — diagnosed, root-caused, fixed. The only failure on the previous head 33bec442d9 was the unused-exports gate. I did not assume it was inherited: ran node scripts/check-unused-exports.mjs on master and on the PR head in separate worktrees — master is clean (exit 0), the PR head fails with exactly one finding:

INTERNAL_ONLY src/launcher.ts: wedgedLedgerFilePath

So this one is OUR code, not master drift: the PR exports wedgedLedgerFilePath() (src/launcher.ts:5021), but every reference to it lives inside src/launcher.ts itself (tests inject their own path via deps.wedgedLedgerPath, nothing imports it). The fix is dropping the export — one line, zero behavior change.

That fix already landed on the branch: commit e7d717937c ("fix(launcher): stop exporting wedgedLedgerFilePath (unused-exports gate)", committed 2026-11-08T17:16Z by the ci-check worker). My own identical one-liner hit it as non-fast-forward when I pushed, so I verified the remote commit makes the same change (same line, same removal) and dropped my duplicate rather than stacking a redundant second commit. Re-ran the gate on the new head locally: clean (8 waived, exit 0).

Current CI on e7d717937c: in flight — 0 failures so far; build-artifact, both gate jobs, lane-guard, lifecycle-ledger-guard and mysql-contract have passed, the cross-platform test shards are still running. I'll report back when the full set settles.

Per the PR's stated review tier, human review is still required before merge (a kill path against a still-serving process + a new state file is a product decision). When CI is green, the merge call is yours: #5972

八问(中文)

  1. 是什么问题:CI 在本 PR 的头提交上红,唯一失败项是 unused-exports 门禁——本 PR 自己引入的 wedgedLedgerFilePath 导出无外部引用。
  2. 什么原因导致的:该函数只被 src/launcher.ts 内部调用(测试通过依赖注入自带路径,没有任何模块 import 它),导出属于多余表面;门禁按设计将此类判为 INTERNAL_ONLY。
  3. 影响范围与严重程度:仅 CI 门禁层面,无任何运行时行为影响;不修则本 PR 无法通过 CI(测试车道被 needs 连锁跳过)。
  4. 怎么修复的:去掉该函数的 export 关键字(一行小修,独立 commit),不改行为;修复已落在 PR 分支(e7d717937c),我在本地对新头复跑同一脚本确认干净。
  5. 是否有回归问题:无——纯去导出,全仓检索证实零外部引用;不碰线格式、配置、持久化;相邻面(安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 轮换 / [Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 采用 / dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 同指纹 attach)由原 PR 的 16 个测试钉住,本次未触碰。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:代码与分支状态就绪(基于最新 master、门禁本地干净、CI 重跑中),待 CI 全绿 + owner 人工评审后合并。

@TOSKA1226

Copy link
Copy Markdown
Contributor

两个优化点已作为 stacked PR 送上来:#5980(base = 本 PR 的分支,commit 055a51f,父 e7d7179)。

摘要:

  • BUSY 候选的接管预算:实测 13.8s 里有 12.4s 花在对一个由定义不会应答的 holder 上做优雅关闭(POST 3s + 观察 5s + SIGTERM 后 2s),然后照样得硬杀。现在只给 BUSY 路径换短预算(WEDGED_BUSY_REQUEST_TIMEOUT_MS = 750、两段观察各 500ms),一级都没删;硬杀后的 5s 验证和 2s settle 原样保留。同一 harness 复测:端到端 16727 → 7530 ms,force→释放 12547 → 4175 ms,force→health 13788 → 5545 ms。
  • win32 ESRCH 噪音:taskkill /F /T 收割后必刷的 direct SIGKILL of wedged holder … failed (kill ESRCH) — its listener may survive the cleanup (#4125) 现在静默,其它 errno 照旧上报。

测试 16 → 18 全绿(短预算断言 + ESRCH 静默断言),tsc --noEmit 干净,反向用例(优雅关闭仍走、SAME-build BUSY 仍 attach、user-zone/外车道/异端口不动)未受影响。

补充一句流程上的事实:本账号对 upstream 没有写权限(403 Permission to ranxianglei/billion-context.git denied to TOSKA1226,之前是经代理/维护者侧落地的),所以这次走 fork 分支 + stacked PR;你要直接合进本 PR 的话 cherry-pick 055a51f 即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Fixed the red unused-exports gate on this PR and pushed it green.

What was red: unused-exports failed at head 33bec442d9; every other early gate was already green/skipped.

Root cause: src/launcher.ts exported wedgedLedgerFilePath() (added for the #5955 wedged-takeover ledger), but nothing outside the file imports it — its only references are an internal fallback call at src/launcher.ts:5691 and a doc-comment mention at src/launcher.ts:366. Tests supply their own path via deps.wedgedLedgerPath, so the symbol has zero external consumers → the gate flags it INTERNAL_ONLY.

Fix: dropped the leading export (one line) so it's module-local, matching the sibling helpers isWedgedLedgerEntry / readWedgedLedger, which are already module-private. No WAIVER entry added — there's genuinely no external consumer to justify keeping it public.

Verification:

  • Local: ran the identical gate script (node scripts/check-unused-exports.mjs) on the pushed tree → clean, exit 0.
  • Live: after push (e7d717937c) the gate re-triggered and now reads success/completed; all other early gates (plan, version-guard, kernel-guard, pi-subagents-guard, lane-guard, lock-sync, test-import-graph, docs-sync, docs-coverage) are success too.
  • grep confirms the symbol has zero references outside src/launcher.ts (incl. dynamic property access), so removing the export cannot break any import or compile.
  • Note: the check-state mirror cache went stale/frozen while the run settled, so I confirmed the terminal gate state directly against the upstream check-run record for the exact head SHA rather than trusting the frozen snapshot. The remaining test matrix shards were still in-flight at last observation with no failures recorded; they're independent of this one-line change and out of scope for this gate fix.

Push discipline: fetched origin before pushing; remote head was unchanged (33bec442d9 == my base), so a clean fast-forward landed e7d717937c with no force-push and no conflicts.


结构化八问

  1. 是什么问题:PR [PR] fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port (#5955) #5972 的 unused-exports CI 门禁红——src/launcher.ts 里为 [Bug][dsh] 0.1.212: Zombie proxy holds port + fingerprint branch only skips without terminating → oversized sessions stu | [Bug][dsh] 0.1.212: 僵尸代理占端口 + fingerprint 分支只 skip 不 terminate -> 超大会话持续 422(BUSY 判定 × EADDRINUSE 死锁) #5955 楔死接管特性新增的 wedgedLedgerFilePath() 被 export 了,但全仓没有任何文件 import 它。
  2. 什么原因导致的:该函数所有引用(内部兜底调用 src/launcher.ts:5691 + 注释 src/launcher.ts:366)都在本文件内,测试又走 deps.wedgedLedgerPath 注入而非直接 import → 门禁判 INTERNAL_ONLY。根因是作者多写了一个 export,该符号本应模块私有。
  3. 影响范围与严重程度:仅 CI 门禁层,零运行时/用户面影响。纯导出卫生问题(一个不该公开的符号被公开)。严重度:仅内部实现,无外部影响。
  4. 怎么修复的:去掉 export 关键字,恢复为模块私有函数,与同文件既有的模块私有兄弟函数(isWedgedLedgerEntry、readWedgedLedger)一致;一行改动,不开 WAIVER(确无外部消费者)。
  5. 是否有回归问题:无。grep 全仓确认该符号在 src/launcher.ts 之外零引用(含动态属性访问),去 export 不可能破坏任何 import/编译;本地同脚本 EXIT=0。未触碰 wire/config/persistence/kernel。
  6. 是否破坏缓存:不涉及——纯源码导出可见性改动,不碰缓存/重试/游标/去重键。
  7. 是否改了设计方向:不涉及——无架构/边界/数据流转向,仅把误公开符号收回模块私有。
  8. 是否引入新的配置:不涉及——无新 env/config/默认值。

判定:unused-exports 已在 head e7d717937c 转绿(上游 check-run 记录 success/completed 确认),其余早期门禁全绿;CI 红已修复,可待剩余 test 分片收尾后进入合并评审。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI is now fully green on head e7d717937c (verified live, cache fetched 2026-11-08T17:51 UTC): every executed check passed — all 12 cross-platform test shards (ubuntu/windows × node 22/24), static-gates ×2, both gate jobs, mysql-contract, lifecycle-ledger-guard, build-artifact. The 10 skipped e2e lanes are path-gated (this PR only touches src/launcher.ts + its test file) and show the same skip pattern as the pre-fix run, so nothing was silently dropped by the one-line export removal.

Net state of PR #5972: open · head on current master tip (848f59b9) · gate red fixed (e7d717937c) · CI green · labels bug/p1/dsh/scope/wiring. Only remaining step is human review + merge, which per the PR's own tier note is yours (new kill path against a still-serving process = product decision): #5972

判定(中文):红因已除、CI 全绿、分支基于最新 master,代码侧无遗留;按 PR 自述评审档等 owner 人工评审后合并即可。

… alarm (#5955)

Acceptance run on Windows (issue reporter, branch @ 33bec44) passed all
three points but measured ~12.4s of pure budget spin in the escalation
ladder against a structurally unresponsive holder (the port released in
~250ms after the hard kill), plus a false-alarm warn on every successful
win32 takeover:

- Escalation stages tightened from 3s+5s+2s+5s to 1s+2s+1s+2s. The
  shutdown-request timeout is now parameterized (requestInstanceShutdown
  gains an optional budget; #3544 rotation keeps its 3s default).
  Worst-case release verdict drops from ~15s to ~6s (+2s post-kill settle
  unchanged). The graceful-first discipline is kept: after a 10-minute
  health silence it is insurance, bounded to at most ~3s of cost.
- hardKillProxyTree win32 fallback no longer warns when the direct SIGKILL
  throws ESRCH — taskkill already reaped the tree, which IS the goal;
  mirrors the POSIX branch suppression.
@TOSKA1226

Copy link
Copy Markdown
Contributor

Thanks — 2826c51 reads right and matches the field numbers (12.4s of a 13.8s takeover was pure budget on a holder that can never answer a graceful POST; ~6s worst-case staging is the correct shape, and silencing the win32 direct-SIGKILL ESRCH removes a false alarm on every takeover).

One gap worth closing before this merges: 2826c51 changes src/launcher.ts only, so the tuned ladder and the ESRCH silence have no regression guard. I put the coverage on a tests-only branch on top of 2826c51:

branch tests-wedged-budget @ 016d933 (TOSKA1226 fork, parent 2826c51, +61/−2)

  • tests/issue5955-wedged-same-lane-takeover.test.ts (18 cases) — candidate matrix, ledger window/stale-gap/persistence, pid-reuse birth stamps, escalation order, kill-failure fallback, port:0 lazy lane, and the negatives (healthy older/newer/same-build BUSY holders, user zone, foreign lane, other port, explicit attach are never touched)
  • src/launcher.ts: export function hardKillProxyTree (one token) so the win32 silence assertion can call it

The timing assertions are written against behaviour, not our old constants: shutdown POST budget must be > 0 && < 3000 (never the 3s graceful default) and total escalation staging must stay >= 2000 && <= 6000 — so re-tuning inside your envelope stays green, while a regression back to 3s+5s+2s+5s fails with the measured number in the message.

Verified locally against 2826c51: 18/18 pass (2.8s), tsc --noEmit exit 0, check-unused-exports clean, check-test-import-graph ok.

Two options, your call:

  1. Cherry-pick 016d933 into this branch now so the fix lands with its guard;
  2. I rebase onto master and open it as a separate PR once [PR] fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port (#5955) #5972 merges (base = master, so ci.yml's pull_request trigger runs the real core matrix instead of the ci:full-gated skip — I'd rather not repeat the "green gates" misread from [PR] fix(launcher): cap the BUSY takeover budget, stop reporting reaped pids (#5955) #5980).

@ranxianglei
ranxianglei merged commit 6d750ef into master Nov 9, 2026
48 checks passed
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1102-master.6d750ef (rolling master tag).

npm install -g billion-context@0.0.1102-master.6d750ef  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

2 participants