Repository navigation
[PR] fix(launcher): force-takeover the wedged same-lane proxy holding the canonical port (#5955) - #5972
Conversation
…canonical port (#5955) The attach/rotation decision table had one uncovered cell: a same-lane resident on the canonical port running DIFFERENT bytes whose health probe never answers (BUSY: TCP connect succeeds while the event loop stays inside long sync slices, e.g. a multi-minute fold on an oversized session). Attach vetoes foreign bytes (#1225), #4125's upward-adoption veto-lift needs an ANSWERED version a BUSY holder cannot give, #3544 rotation excludes version-unverifiable residents by design (#4811), and the spawned child waits 5s behind a same-lane predecessor then ladders away (+1 sticky drift) or dies. No actor ever resolved the holder, so the port stayed wedged until the process died on its own — which the event-loop watchdog only guarantees for >=5min CONTINUOUS blocks; the saturation class never trips it. Oversized sessions then 422'd indefinitely on the stuck lane. Resolution: bounded patience across bring-up cycles, then a supervised forced takeover. A persisted observation ledger (state dir, atomic write, keyed origin#pid#birth-stamp so a REUSED pid starts its own window instead of inheriting the dead occupant's history) accumulates sightings across client restarts. After WEDGED_TAKEOVER_WINDOW_MS (10min = 2x the watchdog hard timeout) of continuous observations across >=3 distinct cycles, the launcher requests a graceful shutdown, terminates if the listener survives, hard-kills as last resort, verifies the listener closed, gives kernel socket state a settle after an abrupt kill, and continues to the normal spawn. Any doubt falls through to the untouched spawn ladder — never worse than today. Adjacent surfaces deliberately unchanged: #3544 rotation (healthy older holders), #4125 upward adoption (healthy newer holders), #4811 same-lane attach (matching bytes), user-zone and foreign-lane holders all keep their existing paths. killAbandonedChild keeps its exact behavior (group kill + handle-level SIGKILL) via the new shared hard-kill helper.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8697-pr.5972.76ce338This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5972.tgz
npm install -g packageInstall instructions are refreshed on each push. |
wedgedLedgerFilePath() is referenced only inside launcher.ts — the deps-injection fallback at line 5691 — while tests supply their own path via deps.wedgedLedgerPath. The leading `export` made it INTERNAL_ONLY under scripts/check-unused-exports.mjs. Drop it to match the sibling module-local ledger helpers (isWedgedLedgerEntry, readWedgedLedger). No behavior change.
(ai: vllm-qwen/qwen3.8-27b) Status (live-verified): PR #5972 is open, head branch CI red — diagnosed, root-caused, fixed. The only failure on the previous head So this one is OUR code, not master drift: the PR exports That fix already landed on the branch: commit Current CI on Per the PR's stated review tier, human review is still required before merge (a kill path against a still-serving process + a new state file is a product decision). When CI is green, the merge call is yours: #5972 八问(中文)
判定:代码与分支状态就绪(基于最新 master、门禁本地干净、CI 重跑中),待 CI 全绿 + owner 人工评审后合并。 |
|
两个优化点已作为 stacked PR 送上来:#5980(base = 本 PR 的分支,commit 摘要:
测试 16 → 18 全绿(短预算断言 + ESRCH 静默断言), 补充一句流程上的事实:本账号对 upstream 没有写权限( |
(ai: vllm-qwen/qwen3.8-27b) Fixed the red What was red: Root cause: Fix: dropped the leading Verification:
Push discipline: fetched origin before pushing; remote head was unchanged ( 结构化八问
判定: |
(ai: vllm-qwen/qwen3.8-27b) CI is now fully green on head Net state of PR #5972: open · head on current master tip ( 判定(中文):红因已除、CI 全绿、分支基于最新 master,代码侧无遗留;按 PR 自述评审档等 owner 人工评审后合并即可。 |
… alarm (#5955) Acceptance run on Windows (issue reporter, branch @ 33bec44) passed all three points but measured ~12.4s of pure budget spin in the escalation ladder against a structurally unresponsive holder (the port released in ~250ms after the hard kill), plus a false-alarm warn on every successful win32 takeover: - Escalation stages tightened from 3s+5s+2s+5s to 1s+2s+1s+2s. The shutdown-request timeout is now parameterized (requestInstanceShutdown gains an optional budget; #3544 rotation keeps its 3s default). Worst-case release verdict drops from ~15s to ~6s (+2s post-kill settle unchanged). The graceful-first discipline is kept: after a 10-minute health silence it is insurance, bounded to at most ~3s of cost. - hardKillProxyTree win32 fallback no longer warns when the direct SIGKILL throws ESRCH — taskkill already reaped the tree, which IS the goal; mirrors the POSIX branch suppression.
|
Thanks — 2826c51 reads right and matches the field numbers (12.4s of a 13.8s takeover was pure budget on a holder that can never answer a graceful POST; ~6s worst-case staging is the correct shape, and silencing the win32 direct-SIGKILL ESRCH removes a false alarm on every takeover). One gap worth closing before this merges: 2826c51 changes branch
The timing assertions are written against behaviour, not our old constants: shutdown POST budget must be Verified locally against 2826c51: 18/18 pass (2.8s), Two options, your call:
|
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1102-master.6d750ef # this exact build
npm install -g billion-context@master # always the newest merged build
|
Closes #5955
Problem
The attach/rotation decision table had one uncovered cell: a same-lane resident on the canonical port running DIFFERENT bytes whose health probe never answers (BUSY — TCP connect succeeds while the event loop stays inside long sync slices, e.g. a multi-minute fold on an oversized session). Every existing surface skips such a holder:
code-fingerprint-mismatch, Launcher proxy reuse (ensureProxyRunning) breaks per-run isolation: shared-fate lifecycle + no code/client identity in compatibility check #1225 discipline);No actor ever resolved the holder, so the port stayed wedged until the process died on its own — which the event-loop watchdog only guarantees for ≥5min continuous blocks; the saturation class never trips it. Reporter evidence (dsh 0.2.2-rc.1, Windows, v0.1.212): a zombie held 18787 for ~10+ minutes while every bootstrap round failed with 422 refusals, then the lane landed permanently on 18788.
Design
Bounded patience across bring-up cycles, then a supervised forced takeover:
isWedgedSameLaneCandidate, pure): same declared lane, our target port, fingerprint mismatch BEFORE any shape verdict, health silent + TCP alive. Anything that ANSWERS health is someone else's cell ([Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 adoption / 安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 rotation); matching bytes are attachable per dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811.wedged-takeover.jsonin the state dir, atomic write): during a wedge the client restarts every failed bootstrap round, so an in-process counter would reset each cycle and the window could never fill. Keyedorigin#pid#birth-stamp(linux/proc/<pid>/statstarttime / win32 powershell StartTime / other POSIXps lstart) so a REUSED pid starts its own window instead of inheriting the dead occupant's history (Windows recycles pids fast). Corrupt/missing file degrades loudly to a fresh window; write failure never blocks bring-up.WEDGED_TAKEOVER_WINDOW_MS = 600_000(10 min = 2× the watchdog hard timeout) of continuous observations across ≥3 distinct cycles; an inter-observation gap >5min restarts the window. A user-visible line announces the window start (without it the user sees only 422s for 10 minutes).POST /__bili/shutdownwith a 1s request budget (insurance for a loop that briefly unblocks) → wait 2s for the listener to close → SIGTERM (POSIX) + 1s → hard kill (POSIX SIGKILL / win32taskkill /F /T, sharedhardKillProxyTree) + 2s. Only after VERIFIED release does the normal spawn proceed; an abrupt kill gets a 2s settle for kernel socket state before the child binds (residual TIME_WAIT must not push the replacement to +1; measured release is ≤252ms, so this keeps an 8× margin on a rare path). Kill failures keep the ledger entry and fall through to the untouched spawn ladder — never worse than today.Rationale for patience over immediate kill: BUSY may be slow-but-alive mid legitimate multi-minute fold; immediate kill loses in-flight work + cold start. The window bounds worst-case unavailability from ∞ to ~10min. Data-loss bound: the victim loses at most its in-flight turn (sessions persist per-turn).
Behavior changes (old → new)
wedged-takeover.jsonkillAbandonedChildNo new config/env surface — all constants hardcoded (event-loop-watchdog precedent; Env Var Discipline). No wire/format changes.
Field-test feedback incorporated (commit
2826c51)The issue reporter ran all three acceptance points on Windows 11 / node v24.15.0 against this branch: takeover PASS (port 18787 released, ledger entry deleted, replacement healthy), re-bind PASS (no more +1 drift), and the 422 trigger eliminated (oversize refusals only fire while the proxy is down). Two findings from that run are fixed here:
direct SIGKILL … failed (kill ESRCH) — its listener may survive) → ESRCH suppressed inhardKillProxyTree's win32 fallback (taskkill already reaped the tree = goal met), mirroring the POSIX branch.Out of scope (adjacent cells, deliberately left alone)
Related work
Tests
tests/issue5955-wedged-same-lane-takeover.test.ts(16 tests): pure decision-table rows (positive + every single-flip negative), ledger boundary arithmetic (window/observation/stale-gap edges), full-path escalation ladder ordering (graceful success / forced kill / kill-fail retry with ledger kept), cross-restart persistence, pid-reuse reset, and six negative rows pinning the ADJACENT surfaces unchanged (安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 rotation-only, [Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 adoption, dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 same-FP attach never killed, user-zone untouched, foreign-lane untouched, different-port untouched, attachUrl skips the step).Review tier
Human review required — a new kill path against a still-serving process is a product decision (auto-merge ineligible: new lifecycle behavior, new state file, kill semantics).