Skip to content

[PR] fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) - #4825

Merged
ranxianglei merged 13 commits into
masterfrom
2026-10-26_dsh-busy-drop-recovery
Oct 27, 2026
Merged

ranxianglei merged 13 commits into
masterfrom
2026-10-26_dsh-busy-drop-recovery

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Closes #4811

Relation to #4773 (superset). This branch absorbs PR #4773 (#4755) as its first commit (cherry-pick of 497afe35d, author preserved). #4773 fixes the recovery-timer clear condition + give-up arming; this PR adds discovery-side BUSY retention, port-log sampling, and bootstrap-catch arming on top. Merging either supersedes the other — owner closes the loser.

What was wrong (verified at tags v0.1.207/v0.1.208 AND master):

  1. probeLiveInstances dropped a candidate whose health probe timed out identically to a dead process — even when TCP connect proved the event loop merely saturated ([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 three-state, discovery half deferred by [PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) #3673). Discovery declared "empty" over a live same-lane holder → spawned a second proxy → the child opened the shared index.db before its listen failed (two-writer window, The same sessionId is served by two bili processes simultaneously: interleaved logs from global + pi-native dual instanc | 同一 sessionId 被两个 bili 进程同时服务:全局 + pi-native 双实例日志交织、重复 cold-load(split-brain) #3923/SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959 family) → EADDRINUSE → TWIN_ADOPT_EXIT_CODE detour (7–27 s measured in the field) racing the client's hard ~13.5 s give-up wall → lost ~half the time → session committed to direct (uncompressed) traffic permanently.
  2. bootstrap()'s catch returned undefined without arming any retry driver — a single one-shot bootstrap failure became permanent until host restart (field case: 27 minutes of direct lane traffic).
  3. logHeadSample read a fixed offset 0 of the append-only port log (rotated only at 5 MB) — failure reports showed the OLDEST line ever written for that port (field case: a two-day-old EADDRINUSE from an older build), actively misdirecting triage.

Fixes.

  • probeLiveInstances: keep candidates that answer TCP but not the health probe as {inst, health: undefined}; an explicit ok:false response still means "answered, draining" and stays dropped; diagnostics distinguish all three outcomes.
  • pickAttachable: a BUSY candidate passes every health-free check verbatim (fingerprint / shape / strictPort / own-lane policy) and is attachable only when its lifecycle is verifiable — own lane, the attachExternal opt-in, or a user-zone instance; a BUSY foreign declared lane is refused with a diagnostic instead of silently dropped.
  • attachGateAllows / rotateStalePredecessor widened to HealthInfo | undefined; a BUSY instance can never be a rotation target (version unverifiable).
  • isConnectable threaded through waitForStarterInstance / findLiveAttachableInstance / rediscoverAfterChildExit (deps overrides preserved).
  • Spawn-failure reports sample [spawnOffset, spawnOffset+256) of the port log instead of byte 0; a shrunken file (rotation under us) falls back to the last 256 bytes — everything left belongs to this spawn.
  • dsh-native bootstrap() catch arms the idempotent recovery timer (mirrors the dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755 fireGiveUp arming), so a one-shot failure re-drives ensureProxyRunning every recovery interval until a base binds.

Behavior changes (old → new).

Residuals (documented, deliberately out of scope).

Config surface: none — no new env var or config field; existing native.attachExternal / BILI_NATIVE_ATTACH_EXTERNAL semantics unchanged.
Kernel: untouched. Version: not bumped.

Tests. New tests/issue4811-dsh-busy-drop-selfheal.test.ts (A rows: BUSY retention positive/negative matrix incl. #2877 own-lane-policy parity; B row: port-log sampling regression; C row: the one-shot-bootstrap self-heal contract from the issue title) plus the absorbed tests/issue4755-dsh-giveup-selfheal.test.ts. Targeted local runs green: issue4811 8/8, issue4755, issue3654, launcher 343 pass + 1 skip, dsh-native 68/68, dsh recovery/reactivation/spawn-retry/rereg 12/12. Full-suite verdict = remote CI.
kernel-change: 无 kernel 源码/版本变更。PR diff 中的 kernel/package-lock.json(仅 2 行版本记录)来自按 CI 协调政策(#4599)merge origin/ci-fixes 带入的 lockfile 同步(ci-fixes 哨兵修复 master lock-sync 继承红),非本 PR 的 kernel 改动;Rule 2 版本纪律不受影响(无 kernel/src、tsup、tsconfig 变更)。

ci-check added 5 commits October 26, 2026 16:03
A runtime give-up lands the register base-less while toolsReady stays true (#3380). The recovery timer's tick cleared itself on toolsReady alone — exactly that shape — so an armed timer died on its first tick without calling maybeRetry, and fireGiveUp never armed one; degraded model traffic never reaches headersFor either. The lane stayed permanently direct/uncompressed until an /acp command or a host restart, so with a healthy replacement proxy already listening the client never re-attached, over-window bodies were refused locally (422), and the session became unrecoverable.

- armRecoveryTimer: treat base-less as unhealthy even when tools are ready
- fireGiveUp: arm the recovery driver at the moment of give-up
…hem (#4811)

Discovery (probeLiveInstances) dropped a candidate whose health probe timed
out identically to one whose process was dead — even when TCP connect proved
the event loop merely saturated (#3654 three-state, discovery half deferred by
#3673). Consequence chain: discovery reports empty → spawn a second proxy on
the lane port → child opens the shared index.db before its listen fails →
EADDRINUSE → TWIN_ADOPT_EXIT_CODE detour (7-27s measured) racing the client's
hard ~13.5s give-up wall → lost ~half the time → session commits to direct
(uncompressed) traffic permanently. The spawned child also created a second
index.db writer during the window (#3923/#3959 family).

- probeLiveInstances: keep candidates that answer TCP but not health probes
  (health === undefined && connectable); an explicit ok:false response still
  means "answered, draining" and stays dropped; diagnostics distinguish the
  three outcomes
- pickAttachable: a BUSY candidate passes every health-free check verbatim
  (fingerprint, shape, strictPort, own-lane policy) and is attachable when its
  lifecycle is verifiable — own lane, attachExternal, or user-zone instance;
  a BUSY foreign declared lane is refused with a diagnostic instead of being
  silently dropped
- attachGateAllows / rotateStalePredecessor widened to HealthInfo | undefined
  (a BUSY instance can never be a rotation target: version unverifiable)
- isConnectable threaded through waitForStarterInstance / findLiveAttachable
  / rediscoverAfterChildExit call sites (deps override preserved)
logHeadSample read a fixed offset 0 of the append-only bili-proxy-<port>.log,
which rotates only at PORT_LOG_MAX_BYTES (5 MB). Until rotation the "child
output head" in every spawn-failure report showed the OLDEST line ever written
for that port — in the #4811 field case a two-day-old EADDRINUSE from an older
build, actively misdirecting triage of the current child.

- capture the file size right after opening the log fd as this spawn's offset
- sample [offset, offset+256); if the file shrank between open and stat
  (rotation under us) every byte present belongs to this spawn, so fall back
  to the last 256 bytes
bootstrap()'s catch logged "proxy bootstrap failed — model traffic goes
direct (uncompressed)" and returned undefined without arming any retry
driver. The plan-time maybeRetry call arms the timer only while the register
still looks healthy-ish; once a one-shot bootstrap failure lands (port held,
child dead), nothing re-drives ensureProxyRunning — the lane stayed direct
until the host restarted (field case: 27 minutes of uncompressed traffic
with a healthy replacement already available after restart).

Arm the idempotent recovery timer at the failure site itself, mirroring the
fireGiveUp arming from #4755: the driver now re-attempts bootstrap every
recovery interval until a base binds and tools register.
…heal (#4811)

- A rows: probeLiveInstances/pickAttachable keep a silent-but-TCP-alive
  candidate for own-lane / laneless-user-zone initiators (attached, no spawn);
  dead and answered-not-ok candidates still drop to spawn; BUSY foreign
  declared lane refuses with a diagnostic; BUSY user-zone under a laned
  initiator keeps #2877 own-lane policy parity
- B row: spawn-failure reports sample THIS spawn's port-log bytes, not the
  oldest line ever written for the port
- C row: two consecutive one-shot bootstrap failures are absorbed by the
  recovery driver and the lane self-heals to a healthy proxy (the issue-title
  contract: a one-shot failure must not become permanent)
@ranxianglei ranxianglei changed the title fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) [PR] fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-26_dsh-busy-drop-recovery (8c500fb)

Latest build — install this exact version

npm install -g billion-context@0.0.6337-pr.4825.63a476c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
8c500fb billion-context@0.0.6337-pr.4825.63a476c build
5c7b8a3 billion-context@0.0.6272-pr.4825.55be5f1 build
46eb24a billion-context@0.0.6241-pr.4825.d64a54a build
42533f9 billion-context@0.0.6225-pr.4825.3ff20fe build
909300a billion-context@0.0.6220-pr.4825.0533e14 build
254e678 billion-context@0.0.6218-pr.4825.213ad19 build
c32dc64 billion-context@0.0.6217-pr.4825.8c2c928 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4825.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 自动修复完成(lane attempt 1/2,小修,无需决策请求)。本 PR 的 CI 红分两段,均已处理:

第一段 — lock-sync(继承红):master 的 kernel 版本 bump(package-lock 记录 acp-kernel 0.0.163 vs kernel/package.json 0.0.165)未重新生成 lockfile,PR 分支带上 master 后暴露。按 #4599 协调政策 merge origin/ci-fixes(根因修复 dfadc21)进 PR 分支;PR body 已补 kernel-change 声明说明 kernel/package-lock.json 两行 diff 来自 ci-fixes 同步、非本 PR 的 kernel 改动。lock-sync、kernel-guard 现均绿。(此段由并行 worker 先行落地,head 909300a。)

第二段 — 11 个 test shard 全红(本 PR 自伤):新增的 process-spawning 套件未登记 LIVE_FILES,run-ci-tests.mjs 的 #1248 tripwire 判所有 shard 失败。已修:42533f9 在 scripts/live-files.mjs 末尾追加 tests/issue4811-dsh-busy-drop-selfheal.test.ts(append-only 惯例:slot=index 决定 zone port band,追加不重排既有槽位)。登记前已审并发假设:该套件 spawn 全部 mock(spawnImpl 计数)、真实 socket 全部 listen(0),并发安全;被吸收的姊妹套件 tests/issue4755-dsh-giveup-selfheal.test.ts 不匹配 SPAWN_RE,无需登记(全量扫描确认 0 个未登记 spawner)。

八问

  1. 是什么问题:PR CI 两段红——先是继承自 master 的 lock-sync lockfile 漂移,后是本 PR 自己的新测试套件漏登记导致 11 个 test shard 全灭。
  2. 什么原因导致的:① 版本 bump 提交未重新生成 package-lock.json(继承红,ci-fixes 已根治);② 新 process-spawning 测试文件未加入 LIVE_FILES 表,tripwire 门禁按设计 fail-fast。
  3. 影响范围与严重程度:纯 CI 基建问题,无产品行为/用户面影响;严重度 = 阻塞本 PR 合并,无运行时影响。
  4. 怎么修复的:merge ci-fixes 收敛 lockfile + PR body 声明;LIVE_FILES 追加登记(含并发假设审查)。
  5. 是否有回归问题:无。唯一代码改动是 live-files.mjs 加一条目+注释,不触碰任何运行时路径;既有条目顺序与端口 band 不变;lock-sync/kernel-guard 及全部门禁复查通过。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

当前状态(head 42533f9):lock-sync ✅ kernel-guard ✅ 全部 guard/gate ✅;11 个 test shard 排队中——tripwire 检查在每个 shard 测试前执行,本地已用同一检查逻辑定向复现通过(TRIPWIRE_CLEAN)。全量判定以远程 CI 为准;若排队跑完仍有红,按 #4599 政策继续跟进。已在 #4599 留一行状态。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。本 PR 是 #4811 的修复项(Closes #4811),并超集吸收 #4773(#4755,首 commit 为其 cherry-pick 497afe35d,原作者保留)——两个 PR 合任意一个即可,落选者 #4773 请 owner 关闭(agent 无关闭权)。相关但不同的条目:#4125(同症状家族,触发器是多版本 stale 实例而非单实例 BUSY drop,#4622 在途)、#3654/PR #3673(三态探测只落了 resolver 半边,discovery 半边当时明确 deferred——本 PR 补的正是那一半)。

标签: bug / dsh / scope/wiring / p1(可用性类:受影响会话不重启宿主无法恢复,与 #4811 裁定一致)。

当前状态: 正在做独立代码审核(逐路径读 diff + 触碰面既有行为清点)+ 核对 head 909300a0c 上 11 个红 test 分片与 master born-red 清单的关系(kernel-guard、lock-sync 已绿)。kernel-guard 此前自补声明的做法已被 owner 否定,按 2026-10-26 政策改走正规授权申请 #4828(锁文件漂移类,本 PR 属同类——其 kernel/package-lock.json diff 来自按 #4599 协议 merge ci-fixes 带入,零 kernel commit),会在 #4828 下补一条覆盖说明。结构化八问结论随后跟进。

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 26, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 分账完成(head 42533f9,11 个 test shard 全红逐条核实):本 PR 自伤 = 0 条残留。

本 PR 唯一的自伤是上一轮报告的 LIVE_FILES 漏登记(tripwire),已在 42533f9 修掉——本轮 tripwire 不再触发,所有门禁(lock-sync/kernel-guard/guards/docs 等)全绿。剩余 11 个 test shard 的红全部来自 master 继承红与 flake,已逐条 A/B 验证(干净 master tip cdac1c0 定向复现):

失败签名 shard 判定
storage-engine-proofs「full-mode restart is invisible (google)」 w22s4, u24s3, w22s3 继承——#4599 板上簇 1(首坏 380ef4d)
storage-render golden (google) u24s3, w22s5, w22s3 继承——#4599 板上簇 2(首坏 f590da9)
#1492 B preflight usage-sourced baseline u22s1, u24s1, w22s1 继承——干净 master 确定性红(新签名,已报 #4599)
#4125 interceptor give-up 'send-failed'(actual 'dead') w22s2, u24s3, w22s3 继承——干净 master 确定性红(新签名,已报 #4599)
#847 preflight honest exhaustion u22s2, u24s2, w22s3 继承——干净 master 确定性红(新签名,已报 #4599)
#1365 dsh apply() attach runtime death w22s1 flake——dsh-native 全套在 master 与 PR 分支本地均 68/68 绿;CI 呈 cancelledByParent 事件循环超时形态
issue4732 after-hook EBUSY unlink index.db w22s5 Windows 文件锁类 flake——子测试本身通过,仅 teardown rimraf 撞锁(cf #362)

结论:本 PR 侧 CI 问题已全部收敛(lock-sync 经 ci-fixes merge + kernel-change 声明转绿;自伤 tripwire 已修)。当前 11 个 test shard 的红均为 master 继承红(其中 3 条新签名已带 A/B 证据报到 #4599,请哨兵收敛进 ci-fixes/补录清单)或 flake,不属于本 PR 的修复范围。待 ci-fixes 收敛后 rebase/重触发即可全绿,PR 本体可进入审核合并流程。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

本条告警指向的是过期 head,无需新修复(attempt 2/2 无操作)。

  • 告警检查 = kernel-guard @ head 254e678(ci-fixes merge 后的中间提交),失败原文:This PR modifies kernel/ but does not declare 'kernel-change: <reason>' in its body. —— 触发原因:ci-fixes merge 把 kernel/package-lock.json 两行同步带进分支,而当时 PR body 还没补声明。
  • 该失败模式已在 909300a 解决(PR body 补上 kernel-change: 声明:仅 lockfile 元数据同步、无内核源码/版本变更)。
  • 当前分支最新 head 42533f9 上实测:kernel-guard ✅ / lock-sync ✅ / 全部 guard/gate ✅——已包含并超越告警所指 head。

当前 CI 全貌不变(详见上一条分账评论):门禁全绿;11 个 test shard 红 = master 继承红(其中 #1492 B / #4125 / #847 三条新签名已带干净 master A/B 证据报到 #4599)+ 2 条 flake,均非本 PR 自伤。待哨兵把继承红收敛进 ci-fixes 后 rebase/重触发即全绿。

ci-check added 3 commits October 26, 2026 18:25
…ous spawn's tail (#4811 review)

logHeadSample's rotation fallback used 'start >= size', which also caught
the zero-output subcase: a child that dies before writing a byte left
size === spawnLogOffset, so the report quoted the PREVIOUS spawn's last
256 bytes as this child's head — the exact stale-head misdirection #4811
fixes (reachable when the child is SIGKILL'd/OOM'd before flushing).

- fallback now triggers only on a shrunken file (size < start), where
  rotation under us really did happen and every byte left is ours
- logTail says it explicitly for the zero-append case ('this child
  appended nothing of its own'), mirroring the existing EMPTY branch;
  guarded by .old absence so a real rotation never mislabels the claim
- row B gains the zero-output edge assertion (fails on unmodified master)
…er (#4811 review)

The header claimed row C pinned 'this issue's bootstrap-catch arming'.
Verified on unmodified master: row C passes there — a FRESH register is
base-less AND !toolsReady, which #2187's plan-time arming already drives
(and even the old clear condition spared), so the catch arm is not what
the row exercises. The field-case shape (given-up base-less + tools-ready
plus further failures) is pinned by tests/issue4755-dsh-giveup-selfheal.
Comment-only: no behavior change.
# Conflicts:
#	kernel/package-lock.json
#	package-lock.json
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

独立审核完成,两处 review 小修已推上分支

审核方式:拉分支逐路径通读 diff(probeLiveInstances / pickAttachable / attachGateAllows / rotateStalePredecessor / waitForStarterInstance / findLiveAttachableInstance / dsh-native 恢复驱动 / 两个测试文件 / LIVE_FILES 注册),独立判断,不复用 #4811 侧已有结论。

独立核过的关键点:

  1. BUSY 保留与生命周期门控成立:same-lane BUSY 直接放行;foreign declared-lane BUSY 从「静默丢弃」变为带诊断拒绝;user-zone 豁免([Bug] attach gate: 手动 bili start 的 user-zone daemon 无视 BILI_NATIVE_ATTACH_EXTERNAL=0(文档称连 user-zone 一起关);并请求按 lane 控制 attach #2877 parity,A6 行钉住);answered-not-ok(draining)与真死实例的丢弃分支原样未动(A3/A4 钉住)。
  2. rotateStalePredecessor 用类型守卫排除 BUSY(版本不可验证,不能当轮换目标)——与旧行为等价(BUSY 原本就进不了该列表)。
  3. isConnectable 线程化覆盖全部调用点,deps override 语义保持。
  4. dsh-native:catch 里 armRecoveryTimer() 幂等(timer 存在即早退)、NODE_TEST_CONTEXT 守卫、unref、无后台 churn(C 行 churn-pin)。通读 apply() 后确认:生产路径上 base-less 状态由 Windows/dsh 宿主重启:代理就绪超出 SPAWN_WAIT_MS(20s) 导致 bootstrap 失败,本进程内无重试 → /bili/origin 恒为 null、设置面板永久"未绑定"、整场会话直连(压缩失效) #2187 plan-time arming + fireGiveUp arming 全覆盖,bootstrap-catch arming 是失败已知点的纵深保险(defense-in-depth),不是唯一驱动——这直接影响测试归属,见修复②。
  5. 测试是真测试:A1–A6 走真实 ensureProxyRunning 依赖缝的正负矩阵;B hermetic(listen(0) 取端口按 fix: emit /acp panel as persistent custom message and strip it from the proxy model context (#359) #360、env 覆盖+恢复);C 钉 churn-free 自愈。对未改动的 master 做了 A/B:6/8 行红(A1/A2/A4/A5/A6/B),A3 负门不变,C 在 master 上绿 → 见修复②。

Review 中发现并已直接修掉的两处小问题(独立标注 commit,已推):

  • ① 04984db — 零输出子进程被误报为上一 spawn 的输出:logHeadSample 的轮换回退条件写成 start >= size,多罩了「子进程一个字节都没写」的子情形(size === spawnLogOffset 时回退到文件尾部 256B = 引用上一个 spawn 的输出当本 child 的 head——正是本 PR 要消灭的陈旧头部误导类;子进程 flush 前被 SIGKILL/OOM 即可达)。改为仅文件收缩(size < start,真轮换)才回退;logTail 对零追加情形显式说「this child appended nothing of its own」(镜像既有 EMPTY 分支,.old 守卫保证真轮换时不误标);row B 补零输出边缘断言(master 上红)。行为变更披露(仅诊断文本):零输出子进程的失败报告 旧=引用上一 spawn 尾部 256B → 新=显式零输出注记;决策逻辑零变化。
  • ② 5e1a15b — row C 头注释机制归属失真:注释声称该行使「bootstrap-catch arming」生效,实测未改动 master 上该行即通过(fresh register 是 base-less+!toolsReady,Windows/dsh 宿主重启:代理就绪超出 SPAWN_WAIT_MS(20s) 导致 bootstrap 失败,本进程内无重试 → /bili/origin 恒为 null、设置面板永久"未绑定"、整场会话直连(压缩失效) #2187 plan-time arming 已驱动,连旧清除条件都不杀它);field-case 形态(give-up 后 base-less+tools-ready 再失败)由 tests/issue4755-dsh-giveup-selfheal.test.ts 钉住。纯注释修正,无行为变化。

八问

  1. 是什么问题:dsh 车道三个缺陷——①发现层把「健康探测超时但端口可连」(事件循环饱和,BUSY)等同死亡丢弃候选 → 重复 spawn → 子进程先开共享 index.db 再 listen 失败 EADDRINUSE → adopt 绕路 7–27s 对客户端放弃墙 ~13.5s,输一半 → 会话永久直连无压缩;②dsh bootstrap() 一次性失败不武装任何重试驱动,直到宿主重启(现场 27 分钟直连流量);③端口日志失败报告固定采 append-only 日志的 offset 0,报告两天前旧构建的 EADDRINUSE,误导 triage。
  2. 什么原因导致的:[Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/[PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) #3673 三态探测只落了 resolver 半边,discovery 半边当时明确延后——本 PR 补的就是那半边;bootstrap catch 只记日志不武装恢复驱动;offset-0 采样隐含「端口日志=本次子进程输出」的假设,与 append-only+延迟轮换的事实不符。
  3. 影响范围与严重程度:p1 可用性类——受影响会话不重启宿主无法恢复压缩;所有 native 车道共享 ensureProxyRunning,但 dsh 是唯一把这条链串成「永久直连」的车道(现场:4368 条消息/~2M token 的会话,provider 上限 1M,直连即不可用)。触发条件:宿主事件循环饱和(现场为 Windows DSH 桌面)+ 端口被活着但不响应的实例占用。
  4. 怎么修复的:BUSY 候选以 health-unknown 态保留,实例文件类兼容性检查照走,仅生命周期可验证时(own lane / attachExternal opt-in / user-zone)可附着,foreign declared-lane BUSY 带诊断拒绝而非静默丢;失败已知点武装幂等恢复驱动(+吸收 dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755 give-up arming,cherry-pick 保原作者);采样本 spawn 字节区间 [spawnOffset, +256),收缩回退全文件,零输出显式注记(review 补);rotateStalePredecessor 排除 BUSY。
  5. 是否有回归问题:触碰路径既有行为逐项清点——dead/draining 丢弃分支不变(A3/A4 钉);claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335 gate 对健康候选原样;[Bug] attach gate: 手动 bili start 的 user-zone daemon 无视 BILI_NATIVE_ATTACH_EXTERNAL=0(文档称连 user-zone 一起关);并请求按 lane 控制 attach #2877 own-lane parity 保持(A6 钉);watchdog 注册有界;attachGateAllows 单调用点(grep 核实);诊断文本变化已披露(仅零输出子情形,旧→新见修复①)。本人两个 review commit 不动决策逻辑,仅诊断措辞+注释。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无——补完 [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/[PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) #3673 既定方向的延后半边(discovery 三态),与 route-to-busy 语义一致;未采纳「开端口先于存储」的启动序变更(owner 决策区,[Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 家族另轨跟踪)。
  8. 是否引入了新的配置:无(无新字段/env/flag)。

CI 与门禁

提醒

#4773 是本 PR 吸收的落选者(PR 描述:「merging either supersedes the other」)——合并本 PR 后请你自行关闭(agent 无关闭权)。

判定:代码本身可合——方向打根因、测试有牙(master 上 6/8 行红)、既有行为清点无回归(除已披露的诊断文本变更);合入时机等两个外部条件:#4814 合入(继承红收敛)+ #4828 拍板(kernel 门禁处置)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决并推送:merge origin/master 进分支,新 head 46eb24a76,PR 状态 dirty → mergeable(现 behind,见 ⑤ 附注)。

① 是什么问题:本 PR 分支与 master 产生合并冲突(dirty),GitHub 无法直接合并。
② 什么原因导致的:双方同时改了 lockfile 里的 kernel 版本记录行——本分支早前 merge ci-fixes 时带入 acp-kernel 0.0.165 的记录,master 随后经 #4807 把 kernel 升到 0.0.166 并重生成 lockfile,两侧版本行不一致 → 文本冲突。代码文件零重叠(本 PR 只动 launcher/dsh-native/live-files/两个测试文件,master 只动 stream/preflight/persist/kernel 等)。
③ 影响范围与严重程度:纯分支状态问题,无用户面行为影响;仅 CI 级阻塞(冲突不解决就不能合并)。
④ 怎么修复的:把 master 合进分支;两个 lockfile 的冲突取 master 侧值(0.0.166)——已验证两份文件两侧版本只差版本串,且本 PR 自身没有任何主动的 kernel 版本变更(0.0.165 是此前 merge ci-fixes 被动带入的);其余全部 git 自动合并。合并后逐字节核对本 PR 自有的 5 个文件与合并前 tip 完全一致。
⑤ 是否有回归问题:无。本地验证:typecheck clean;定向套件 tests/issue4811-dsh-busy-drop-selfheal.test.ts 8/8、tests/issue4755-dsh-giveup-selfheal.test.ts 全绿。merge 带入的变更全部来自已评审合入 master 的提交(#4807 kernel 0.0.166、#4732 teardown 修复等),本次合并零语义引入。附注:推送后 master 又前进到 341e76c(#4802 429 governor 合入),PR 现显示 behind 而非冲突(GitHub 可自动合并);按「解完不重复 rebase」规则未追。
⑥ 是否破坏缓存:不涉及。
⑦ 是否改了设计方向:不涉及。
⑧ 是否引入了新的配置:不涉及。

全量判定等新 head 上的 CI test lane(master 侧此前的继承红簇已在 e290b4f 修复,详见 #4599 floor 287)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

状态更正(接我上一条八问回复):head 已变、CI 已全绿。

我推完两个 review 修复 commit(04984db5c / 5e1a15bdc)后,冲突解决车道把新 master(#4807 ref-chokepoint 等,kernel → 0.0.166、锁文件重新生成)合入了分支——当前 head 46eb24a76,我的两个 commit 完整在历史里(已用 merge-base --is-ancestor 核对),该合并与本 PR 自己的文件零重叠。

由此两件事自动了结(逐文件 diff 核对过 341e76c28..46eb24a76):

  1. PR 对 master 的 kernel/ 与根锁文件 diff 归零——之前的版本同步在分支追上 master 后变成空操作。kernel-guard 现在无需任何声明即绿,kernel authorization request: package-lock churn in PR branches tripping the kernel human-gate (8 PRs) #4828 的 A/B 决策不再覆盖本 PR(已在 kernel authorization request: package-lock churn in PR branches tripping the kernel human-gate (8 PRs) #4828 留更正评论,批准范围回到 7 个)。
  2. CI 全绿:51 项检查 = 37 success + 14 skipped,0 failure——包括之前 head 上红的全部测试分片。原因:master 侧随后修掉了 born-red 集群(issue847/issue1492/golden-storage-render 等测试文件更新),分支经那次合并拿到了修复。所有 e2e 真实客户端车道(e2e-native-dsh、e2e-dsh-plugin-lane 等 11 条)也全绿。

八问增量:除「CI 状态」从"继承红待清"变为"全绿"外,各问结论不变;回归面未新增(合并进来的都是 master 既有已审内容);行为变更披露维持上一条所列两项(零输出子进程诊断文本、BUSY 候选保留语义)。

收工判定:代码可合、CI 全绿,本 PR 侧无遗留。 剩两件都在你手里:① 合入本 PR(我无法代合);② 关闭被吸收的落选者 #4773(agent 无关闭权)。

@maozhuoshushu

Copy link
Copy Markdown

Field data from Windows + DSH desktop (bili 0.1.208) that may be useful for this PR — full writeup
including the measured tables is in #4811 (issuecomment-6431040047). Summary of the two points that
touch this branch:

  1. PROBE_TIMEOUT_MS is 1500 ms, hardcoded, and not env-overridable. It does not go through the
    envMillis() wrapper its neighbours use, so unlike BILI_ATTACH_HEALTH_DEADLINE_MS it can't be
    tuned. Measured synchronous cold-load blocking (n=439 pairs): median heavy load = 1482 ms,
    48.2% of heavy loads (refs>=5000) exceed 1500 ms, P90 = 4950 ms, max = 11563 ms. So the BUSY path
    you now retain is entered on ~half of heavy loads. Keeping it attachable fixes "declared dead",
    but the request still queues behind a saturated loop for up to ~11.5 s.

  2. Keeping health: undefined necessarily skips the checks that need health. In pickAttachable
    the instanceId verification and version-based gates (rotateStalePredecessor explicitly can't
    rotate a BUSY instance) become unrunnable for BUSY candidates. A longer budget keeps health
    populated and preserves those checks — the two changes look complementary rather than
    alternative.

I verified locally that 6000 ms + a bounded retry on the busy branch (250 ms backoff, 8 s cap, only
when TCP connect succeeded) takes heavy-load exceedance from 48.2% → 1.8%, and worst case
(2 instances × 8 s = 16 s) stays well inside SPAWN_BUDGET_MS = 60000, so it can't push startup
into timeout.

Also for anyone triaging: the reason the loop saturates at all is the 512 MB residentMaxBytes
default against ~136 MB for a single 14k-message session while inFlight reaches 7 → constant
evict/cold-load churn. Raising it to 2 GB removed the churn here; it's config, no code needed.

Repro is reliable on this box within minutes under 2 concurrent large sessions — happy to test a
build of this branch if that helps.

# Conflicts:
#	kernel/package-lock.json
#	package-lock.json
@ranxianglei ranxianglei mentioned this pull request Oct 27, 2026
Closed
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

字段数据收到,三点全部在分支 head 46eb24a76 上逐行核对过,均成立:

  1. PROBE_TIMEOUT_MS = 1500 硬编码、无 env 覆盖:src/launcher.ts:181 const PROBE_TIMEOUT_MS = 1500;,不走 envMillis()。你说的相邻旋钮 BILI_ATTACH_HEALTH_DEADLINE_MS 是客户端侧 attach 截止期(src/agent/shared.ts:25,473,默认 15000ms),确实与本项是两回事;TCP connect 探测又是另一个常量 CONNECT_PROBE_TIMEOUT_MS = 2000(src/liveness.ts:27)——「勿混」已记下。
  2. BUSY 候选结构性跳过健康依赖检查:instanceId 核对只在有健康应答时运行(src/launcher.ts:4109);watchdog/version 门禁对 BUSY 走独立分支(pickAttachable :4223-4239,own-lane/user-zone/attachExternal 放行、外来声明车道仍拒);rotateStalePredecessor 因 version 不可验证把无 health 候选显式排除出旋转目标(:4694-4702)。三处属实;own-lane 场景(dsh 主场景)生命周期另有同车道 spawn 纪律背书,缺口主要落在 user-zone/attachExternal 候选。
  3. residentMaxBytes 默认 512 MiB:src/knobs.ts:424-427(Bound startup session loading and preserve historical recovery #2803),纯配置旋钮,你提到 2 GB 消除 churn 的结论成立。这是饱和的上游诱因,也是最便宜的杠杆(降低饱和频率比抬高探测预算更有效),已写入决策 issue。

层次判断:你的数据说明 1500ms 预算压在重负载冷加载的中位线上(median 1482ms)——不是长尾,是常态参数失配。但分层要说清:#4825 合入后,误判不再丢可用性(keep-BUSY → 排队等折叠),它退化为首请求排队延迟(最坏 ~11.5s)+ 门禁覆盖缺口。所以这是调优/门禁完整性问题,不是本 PR 的缺陷——keep-BUSY 保证不依赖探测预算取值。

处置:默认值变更 + 新配置面均属 owner 域(§7.4 human-only + Configuration Surface Discipline),且一个 issue 一个 scope,不塞进本 PR。已立决策 issue #4893(标签:p1 + scope/wiring —— 决策请求类,跨车道 launcher attach 路径),A/B/C/D 四选项 + 推荐 D(直接引用你本地验证的两个数字),并含配置面归属(raw env vs knobs/config 键)与附带小项(CONFIGURATION.md sessions 节补并发大会话调高 residentMaxBytes 的指引)。不阻塞 #4825 合并。

测试 offer 接受,有用。两种方式任选:① 从分支自构:git clone -b 2026-10-26_dsh-busy-drop-recovery https://github.com/ranxianglei/billion-context && npm ci && npm run build,产物替换你 profile 里的 dist(你之前 instrument 过 dist,路径应熟);② 说一声我把分支 head 的 packed tarball 传本帖。预期观测(验收标准):你的复现场景(同车道单实例饱和)下日志出现 native-attach: keep ... BUSY ... not dead (#3654/#4811) + 直接 attach,全程无第二子进程、无 EADDRINUSE、无 adopting serving twin 迂回;首请求在折叠后完成属预期排队而非回归。若能顺手记录 attach 后的实际排队时长,就是 #4893 决策的直接输入。

自查披露:处理过程中我探测镜像 API 的 issue 创建端点时,一次测试载荷意外落库为 #4892(标题 “x”,无内容)——已在该 issue 下评论说明是误建垃圾、待 owner 关闭,与任何工作项无关,抱歉造成噪音。

CI 状态:新 head 46eb24a76 的全量判定仍在跑(冲突解决详情见上一条八问回复)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) 已同步 origin/master(23158db)进本分支并推送,新 head 8c500fb(原 5c7b8a3):lockfile 冲突按 master 侧解(kernel 0.0.166→0.0.167,本 PR 未动内核版本,纯 master 前进);本地 typecheck/build/定向 dsh 套件(issue4755/issue4811)全绿。新 head CI 全部完成:37 success + 14 skipped、0 失败,11 条 CORE test 腿 + 静态门禁全绿。当前 mergeable_state=blocked 为结构性状态而非本次同步引入——branch protection 要求的 3 条 windows node-24 test 腿只在 FULL scope(ci:full 标签 / release PR)产出,普通 PR 推送按设计不跑;behind 状态已清除(master 已是 head 祖先),分支侧无需再动。

@ranxianglei
ranxianglei merged commit 49a029b into master Oct 27, 2026
51 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.869-master.49a029b (rolling master tag).

npm install -g billion-context@0.0.869-master.49a029b  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

This was referenced Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent

2 participants