Repository navigation
[PR] fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) - #4825
Conversation
A runtime give-up lands the register base-less while toolsReady stays true (#3380). The recovery timer's tick cleared itself on toolsReady alone — exactly that shape — so an armed timer died on its first tick without calling maybeRetry, and fireGiveUp never armed one; degraded model traffic never reaches headersFor either. The lane stayed permanently direct/uncompressed until an /acp command or a host restart, so with a healthy replacement proxy already listening the client never re-attached, over-window bodies were refused locally (422), and the session became unrecoverable. - armRecoveryTimer: treat base-less as unhealthy even when tools are ready - fireGiveUp: arm the recovery driver at the moment of give-up
…hem (#4811) Discovery (probeLiveInstances) dropped a candidate whose health probe timed out identically to one whose process was dead — even when TCP connect proved the event loop merely saturated (#3654 three-state, discovery half deferred by #3673). Consequence chain: discovery reports empty → spawn a second proxy on the lane port → child opens the shared index.db before its listen fails → EADDRINUSE → TWIN_ADOPT_EXIT_CODE detour (7-27s measured) racing the client's hard ~13.5s give-up wall → lost ~half the time → session commits to direct (uncompressed) traffic permanently. The spawned child also created a second index.db writer during the window (#3923/#3959 family). - probeLiveInstances: keep candidates that answer TCP but not health probes (health === undefined && connectable); an explicit ok:false response still means "answered, draining" and stays dropped; diagnostics distinguish the three outcomes - pickAttachable: a BUSY candidate passes every health-free check verbatim (fingerprint, shape, strictPort, own-lane policy) and is attachable when its lifecycle is verifiable — own lane, attachExternal, or user-zone instance; a BUSY foreign declared lane is refused with a diagnostic instead of being silently dropped - attachGateAllows / rotateStalePredecessor widened to HealthInfo | undefined (a BUSY instance can never be a rotation target: version unverifiable) - isConnectable threaded through waitForStarterInstance / findLiveAttachable / rediscoverAfterChildExit call sites (deps override preserved)
logHeadSample read a fixed offset 0 of the append-only bili-proxy-<port>.log, which rotates only at PORT_LOG_MAX_BYTES (5 MB). Until rotation the "child output head" in every spawn-failure report showed the OLDEST line ever written for that port — in the #4811 field case a two-day-old EADDRINUSE from an older build, actively misdirecting triage of the current child. - capture the file size right after opening the log fd as this spawn's offset - sample [offset, offset+256); if the file shrank between open and stat (rotation under us) every byte present belongs to this spawn, so fall back to the last 256 bytes
bootstrap()'s catch logged "proxy bootstrap failed — model traffic goes direct (uncompressed)" and returned undefined without arming any retry driver. The plan-time maybeRetry call arms the timer only while the register still looks healthy-ish; once a one-shot bootstrap failure lands (port held, child dead), nothing re-drives ensureProxyRunning — the lane stayed direct until the host restarted (field case: 27 minutes of uncompressed traffic with a healthy replacement already available after restart). Arm the idempotent recovery timer at the failure site itself, mirroring the fireGiveUp arming from #4755: the driver now re-attempts bootstrap every recovery interval until a base binds and tools register.
…heal (#4811) - A rows: probeLiveInstances/pickAttachable keep a silent-but-TCP-alive candidate for own-lane / laneless-user-zone initiators (attached, no spawn); dead and answered-not-ok candidates still drop to spawn; BUSY foreign declared lane refuses with a diagnostic; BUSY user-zone under a laned initiator keeps #2877 own-lane policy parity - B row: spawn-failure reports sample THIS spawn's port-log bytes, not the oldest line ever written for the port - C row: two consecutive one-shot bootstrap failures are absorbed by the recovery driver and the lane self-heals to a healthy proxy (the issue-title contract: a one-shot failure must not become permanent)
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6337-pr.4825.63a476cThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4825.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) CI 自动修复完成(lane attempt 1/2,小修,无需决策请求)。本 PR 的 CI 红分两段,均已处理: 第一段 — lock-sync(继承红):master 的 kernel 版本 bump(package-lock 记录 acp-kernel 0.0.163 vs kernel/package.json 0.0.165)未重新生成 lockfile,PR 分支带上 master 后暴露。按 #4599 协调政策 merge 第二段 — 11 个 test shard 全红(本 PR 自伤):新增的 process-spawning 套件未登记 LIVE_FILES, 八问
当前状态(head 42533f9):lock-sync ✅ kernel-guard ✅ 全部 guard/gate ✅;11 个 test shard 排队中——tripwire 检查在每个 shard 测试前执行,本地已用同一检查逻辑定向复现通过(TRIPWIRE_CLEAN)。全量判定以远程 CI 为准;若排队跑完仍有红,按 #4599 政策继续跟进。已在 #4599 留一行状态。 |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。本 PR 是 #4811 的修复项(Closes #4811),并超集吸收 #4773(#4755,首 commit 为其 cherry-pick 标签: bug / dsh / scope/wiring / p1(可用性类:受影响会话不重启宿主无法恢复,与 #4811 裁定一致)。 当前状态: 正在做独立代码审核(逐路径读 diff + 触碰面既有行为清点)+ 核对 head |
(ai: vllm-qwen/qwen3.8-27b) CI 分账完成(head 42533f9,11 个 test shard 全红逐条核实):本 PR 自伤 = 0 条残留。 本 PR 唯一的自伤是上一轮报告的 LIVE_FILES 漏登记(tripwire),已在 42533f9 修掉——本轮 tripwire 不再触发,所有门禁(lock-sync/kernel-guard/guards/docs 等)全绿。剩余 11 个 test shard 的红全部来自 master 继承红与 flake,已逐条 A/B 验证(干净 master tip cdac1c0 定向复现):
结论:本 PR 侧 CI 问题已全部收敛(lock-sync 经 ci-fixes merge + kernel-change 声明转绿;自伤 tripwire 已修)。当前 11 个 test shard 的红均为 master 继承红(其中 3 条新签名已带 A/B 证据报到 #4599,请哨兵收敛进 ci-fixes/补录清单)或 flake,不属于本 PR 的修复范围。待 ci-fixes 收敛后 rebase/重触发即可全绿,PR 本体可进入审核合并流程。 |
(ai: vllm-qwen/qwen3.8-27b) 本条告警指向的是过期 head,无需新修复(attempt 2/2 无操作)。
当前 CI 全貌不变(详见上一条分账评论):门禁全绿;11 个 test shard 红 = master 继承红(其中 #1492 B / #4125 / #847 三条新签名已带干净 master A/B 证据报到 #4599)+ 2 条 flake,均非本 PR 自伤。待哨兵把继承红收敛进 ci-fixes 后 rebase/重触发即全绿。 |
…ous spawn's tail (#4811 review) logHeadSample's rotation fallback used 'start >= size', which also caught the zero-output subcase: a child that dies before writing a byte left size === spawnLogOffset, so the report quoted the PREVIOUS spawn's last 256 bytes as this child's head — the exact stale-head misdirection #4811 fixes (reachable when the child is SIGKILL'd/OOM'd before flushing). - fallback now triggers only on a shrunken file (size < start), where rotation under us really did happen and every byte left is ours - logTail says it explicitly for the zero-append case ('this child appended nothing of its own'), mirroring the existing EMPTY branch; guarded by .old absence so a real rotation never mislabels the claim - row B gains the zero-output edge assertion (fails on unmodified master)
…er (#4811 review) The header claimed row C pinned 'this issue's bootstrap-catch arming'. Verified on unmodified master: row C passes there — a FRESH register is base-less AND !toolsReady, which #2187's plan-time arming already drives (and even the old clear condition spared), so the catch arm is not what the row exercises. The field-case shape (given-up base-less + tools-ready plus further failures) is pinned by tests/issue4755-dsh-giveup-selfheal. Comment-only: no behavior change.
# Conflicts: # kernel/package-lock.json # package-lock.json
(ai: vllm-qwen/qwen3.8-27b) 独立审核完成,两处 review 小修已推上分支审核方式:拉分支逐路径通读 diff(probeLiveInstances / pickAttachable / attachGateAllows / rotateStalePredecessor / waitForStarterInstance / findLiveAttachableInstance / dsh-native 恢复驱动 / 两个测试文件 / LIVE_FILES 注册),独立判断,不复用 #4811 侧已有结论。 独立核过的关键点:
Review 中发现并已直接修掉的两处小问题(独立标注 commit,已推):
八问
CI 与门禁
提醒#4773 是本 PR 吸收的落选者(PR 描述:「merging either supersedes the other」)——合并本 PR 后请你自行关闭(agent 无关闭权)。 判定:代码本身可合——方向打根因、测试有牙(master 上 6/8 行红)、既有行为清点无回归(除已披露的诊断文本变更);合入时机等两个外部条件:#4814 合入(继承红收敛)+ #4828 拍板(kernel 门禁处置)。 |
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决并推送:merge origin/master 进分支,新 head ① 是什么问题:本 PR 分支与 master 产生合并冲突(dirty),GitHub 无法直接合并。 全量判定等新 head 上的 CI test lane(master 侧此前的继承红簇已在 e290b4f 修复,详见 #4599 floor 287)。 |
(ai: vllm-qwen/qwen3.8-27b) 状态更正(接我上一条八问回复):head 已变、CI 已全绿。 我推完两个 review 修复 commit( 由此两件事自动了结(逐文件 diff 核对过
八问增量:除「CI 状态」从"继承红待清"变为"全绿"外,各问结论不变;回归面未新增(合并进来的都是 master 既有已审内容);行为变更披露维持上一条所列两项(零输出子进程诊断文本、BUSY 候选保留语义)。 收工判定:代码可合、CI 全绿,本 PR 侧无遗留。 剩两件都在你手里:① 合入本 PR(我无法代合);② 关闭被吸收的落选者 #4773(agent 无关闭权)。 |
|
Field data from Windows + DSH desktop (bili 0.1.208) that may be useful for this PR — full writeup
I verified locally that 6000 ms + a bounded retry on the busy branch (250 ms backoff, 8 s cap, only Also for anyone triaging: the reason the loop saturates at all is the 512 MB Repro is reliable on this box within minutes under 2 concurrent large sessions — happy to test a |
# Conflicts: # kernel/package-lock.json # package-lock.json
(ai: vllm-qwen/qwen3.8-27b) 字段数据收到,三点全部在分支 head
层次判断:你的数据说明 1500ms 预算压在重负载冷加载的中位线上(median 1482ms)——不是长尾,是常态参数失配。但分层要说清:#4825 合入后,误判不再丢可用性(keep-BUSY → 排队等折叠),它退化为首请求排队延迟(最坏 ~11.5s)+ 门禁覆盖缺口。所以这是调优/门禁完整性问题,不是本 PR 的缺陷——keep-BUSY 保证不依赖探测预算取值。 处置:默认值变更 + 新配置面均属 owner 域(§7.4 human-only + Configuration Surface Discipline),且一个 issue 一个 scope,不塞进本 PR。已立决策 issue #4893(标签:p1 + scope/wiring —— 决策请求类,跨车道 launcher attach 路径),A/B/C/D 四选项 + 推荐 D(直接引用你本地验证的两个数字),并含配置面归属(raw env vs knobs/config 键)与附带小项(CONFIGURATION.md sessions 节补并发大会话调高 residentMaxBytes 的指引)。不阻塞 #4825 合并。 测试 offer 接受,有用。两种方式任选:① 从分支自构: 自查披露:处理过程中我探测镜像 API 的 issue 创建端点时,一次测试载荷意外落库为 #4892(标题 “x”,无内容)——已在该 issue 下评论说明是误建垃圾、待 owner 关闭,与任何工作项无关,抱歉造成噪音。 CI 状态:新 head |
(ai: vllm-qwen/qwen3.8-27b) 已同步 origin/master(23158db)进本分支并推送,新 head 8c500fb(原 5c7b8a3):lockfile 冲突按 master 侧解(kernel 0.0.166→0.0.167,本 PR 未动内核版本,纯 master 前进);本地 typecheck/build/定向 dsh 套件(issue4755/issue4811)全绿。新 head CI 全部完成:37 success + 14 skipped、0 失败,11 条 CORE test 腿 + 静态门禁全绿。当前 mergeable_state=blocked 为结构性状态而非本次同步引入——branch protection 要求的 3 条 windows node-24 test 腿只在 FULL scope(ci:full 标签 / release PR)产出,普通 PR 推送按设计不跑;behind 状态已清除(master 已是 head 祖先),分支侧无需再动。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.869-master.49a029b # this exact build
npm install -g billion-context@master # always the newest merged build
|
Closes #4811
Relation to #4773 (superset). This branch absorbs PR #4773 (#4755) as its first commit (cherry-pick of
497afe35d, author preserved). #4773 fixes the recovery-timer clear condition + give-up arming; this PR adds discovery-side BUSY retention, port-log sampling, and bootstrap-catch arming on top. Merging either supersedes the other — owner closes the loser.What was wrong (verified at tags v0.1.207/v0.1.208 AND master):
probeLiveInstancesdropped a candidate whose health probe timed out identically to a dead process — even when TCP connect proved the event loop merely saturated ([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 three-state, discovery half deferred by [PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) #3673). Discovery declared "empty" over a live same-lane holder → spawned a second proxy → the child opened the shared index.db before its listen failed (two-writer window, The same sessionId is served by two bili processes simultaneously: interleaved logs from global + pi-native dual instanc | 同一 sessionId 被两个 bili 进程同时服务:全局 + pi-native 双实例日志交织、重复 cold-load(split-brain) #3923/SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959 family) → EADDRINUSE → TWIN_ADOPT_EXIT_CODE detour (7–27 s measured in the field) racing the client's hard ~13.5 s give-up wall → lost ~half the time → session committed to direct (uncompressed) traffic permanently.bootstrap()'s catch returned undefined without arming any retry driver — a single one-shot bootstrap failure became permanent until host restart (field case: 27 minutes of direct lane traffic).logHeadSampleread a fixed offset 0 of the append-only port log (rotated only at 5 MB) — failure reports showed the OLDEST line ever written for that port (field case: a two-day-old EADDRINUSE from an older build), actively misdirecting triage.Fixes.
probeLiveInstances: keep candidates that answer TCP but not the health probe as{inst, health: undefined}; an explicitok:falseresponse still means "answered, draining" and stays dropped; diagnostics distinguish all three outcomes.pickAttachable: a BUSY candidate passes every health-free check verbatim (fingerprint / shape / strictPort / own-lane policy) and is attachable only when its lifecycle is verifiable — own lane, theattachExternalopt-in, or a user-zone instance; a BUSY foreign declared lane is refused with a diagnostic instead of silently dropped.attachGateAllows/rotateStalePredecessorwidened toHealthInfo | undefined; a BUSY instance can never be a rotation target (version unverifiable).isConnectablethreaded throughwaitForStarterInstance/findLiveAttachableInstance/rediscoverAfterChildExit(deps overrides preserved).[spawnOffset, spawnOffset+256)of the port log instead of byte 0; a shrunken file (rotation under us) falls back to the last 256 bytes — everything left belongs to this spawn.bootstrap()catch arms the idempotent recovery timer (mirrors the dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755 fireGiveUp arming), so a one-shot failure re-drivesensureProxyRunningevery recovery interval until a base binds.Behavior changes (old → new).
ensureProxyRunning(dsh / opencode / pi / codex / zcode exit-handoff / watchdog drift-repair), not just dsh.Residuals (documented, deliberately out of scope).
initSessions→store.bootinsidecreateEngineContext, beforeserver.listen). Reordering is a startup-order architecture change touching every lane (request semantics during the gap, watchdog arming timing, lineage reconcile) — owner-decision territory. BUSY retention removes the dominant trigger (single saturated same-lane holder); the residual window only exists for unverified contention ([Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 family), where the losing child serves no traffic at all (it exits on listen failure — no dual session commit possible; WAL serializes writers). Tracked under the [Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 failure domain — no new issue per granularity rules.Config surface: none — no new env var or config field; existing
native.attachExternal/BILI_NATIVE_ATTACH_EXTERNALsemantics unchanged.Kernel: untouched. Version: not bumped.
Tests. New
tests/issue4811-dsh-busy-drop-selfheal.test.ts(A rows: BUSY retention positive/negative matrix incl. #2877 own-lane-policy parity; B row: port-log sampling regression; C row: the one-shot-bootstrap self-heal contract from the issue title) plus the absorbedtests/issue4755-dsh-giveup-selfheal.test.ts. Targeted local runs green: issue4811 8/8, issue4755, issue3654, launcher 343 pass + 1 skip, dsh-native 68/68, dsh recovery/reactivation/spawn-retry/rereg 12/12. Full-suite verdict = remote CI.kernel-change: 无 kernel 源码/版本变更。PR diff 中的
kernel/package-lock.json(仅 2 行版本记录)来自按 CI 协调政策(#4599)mergeorigin/ci-fixes带入的 lockfile 同步(ci-fixes 哨兵修复 master lock-sync 继承红),非本 PR 的 kernel 改动;Rule 2 版本纪律不受影响(无 kernel/src、tsup、tsconfig 变更)。