Repository navigation
Description
AI 声明:本 issue 由 AI(WorkBuddy)辅助报告人整理生成——日志引用均为
bili.log原文(仅掩码内网主机名、缩写会话 UUID),代码结论按【实测】/【推断】逐条标注证据等级;请以维护者的独立核实为准。
[Bug][dsh] 恢复竞态:serving twin 卡死后各客户端并发重拉,替补子进程在端口仍被僵尸(随后即将空闲)占据时按 twin-collision 自杀(code 3),全体客户端同时放弃 → 裸历史直连撞上游 1M token 上限;并发重拉还触发 index.db 双写者互斥(DIGEST DRIFT)
建议标签:bug / dsh / scope/wiring / p1
TL;DR (EN)
When a saturated serving twin wedges and its clients all respawn concurrently, each replacement child bind-fails with EADDRINUSE against the dying zombie (which still holds the port and the index.db write lock), and self-exits with code 3 (twin-collision signal) — by a protocol that assumes "port occupied = a healthy twin is serving". Seconds later the zombie fully dies and the port is empty ("nothing is listening on port 18788"), but every spawner has already concluded twin-collision and gave up simultaneously → all model traffic goes direct (uncompressed) and raw histories hit the upstream 1,000,000-token gate. The concurrent respawn window also trips the index.db two-writer guard and produces a unified shadow DIGEST DRIFT.
环境
- billion-context 0.1.208(npm,
node ...\node_modules\billion-context\dist\index.js start --host 127.0.0.1 --port 18788,lane dsh) - 宿主:DSH 桌面版(Windows 11 x64),同机多个并发 host 进程各带一个 in-process dsh-client,共享 18788
- 上游:OpenAI 兼容网关,输入硬上限 1,000,000 token(经本地 ccproxy 转发)
时间线(2026-10-27,bili.log 原文,UTC)
之前 serving twin 已历经多轮换代(pid 7388 → 22120 → 4012 → 25572 → 5148),每换代一次即一轮风暴;以下是最清晰的一轮:
| 时刻 | 事件 |
|---|---|
| 12:46:48 | native-attach: drop http://127.0.0.1:18788 (pid 4012): health probe failed but TCP connect succeeded — instance is BUSY (event loop saturated), not dead (#3654) ← 插件此时能正确区分饱和与死亡 |
| 13:20:43 | serving twin(pid 5148)僵死:20+ 条 [conn#NNNN] tcp closed reason=error(ECONNRESET);同会话 resident pool 达 417MB(evicted ... (weight 417301854 bytes)) |
| 13:20:49–51 | [storage] 49 tail pointer(s) ... reference missing rows + [storage] unified shadow DIGEST DRIFT ... unified replay != legacy load view |
| 13:20:53 | native-attach: drop http://127.0.0.1:18788 (pid 5148): health probe failed → 所有客户端同时 spawn 替补子进程 |
| 13:20:56 | 替补子进程撞上僵尸:[storage] another process (pid 5148) is already writing .../index.db — two processes on one index.db corrupt it (#3923/#3959)(端口同样仍被 5148 占据) |
| 13:20:57 | serving-twin adoption failed verification — continuing with the standard post-exit re-discovery |
| 13:21:00.977 | [dsh-client] proxy bootstrap failed — ...: bili: proxy child exited before becoming healthy (code 3 (twin-collision signal)) — nothing is listening on port 18788 + [dsh-client] proxy respawn gave up — model traffic goes direct (uncompressed) |
| 随后 | 5148 彻底退出、端口空闲,但所有客户端已放弃;裸流量直连上游。同日 12:47、13:17 各重演一轮(两条 gave up 均在 log 中) |
字节级对账(实测):直连期间 ccproxy 侧收到与 bili.log 会话 inbound=47.6MiB 完全一致的原始请求体(49,972,029B,≈5.2M tokens)→ 上游 1M token 校验拒绝;同一会话经 bili 折叠后仅 400–450KB(缓存命中 99%)。时拒时收(上游校验非确定)叠加直连间歇出现,用户侧表现即"多个会话同时报超出上下文、点继续又时好时坏"(与 #4381 症状一致)。
成因分析
- 【实测】code 3 自杀是协议内建行为(
EADDRINUSE → duplicate loses → exit code 3,spawner 读作 adopt 信号,Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341/autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044、[Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654)。 - 【实测】该协议隐含假设"端口被占 = 有健康孪生在服务"。恢复窗口内此假设必假:占端口的是僵死 serving twin(未释放 socket、未释放 index.db 写锁),几秒后它彻底退出、端口真空。
- 【推断】多 host 并发 respawn 使替补子进程之间也互为 duplicate:赢家的 spawner 收编成功,输家的 spawner 验证失败;一旦赢家也在初始化窗口内被 index.db 双写者守卫绊住,就出现"全员 code 3 + 端口空"的最坏分支。
- 【实测】12:46:48 表明三态健康探测([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 修复)能识别 "BUSY not dead",但该判定未接入 respawn 触发条件——13:20:53 仍按死亡处理并触发风暴。
与现有 issue / PR 的关系(查重)
- [bug] After event loop wedge, zombie still holds port 18789 — parent respawn loop spins on EADDRINUSE indefinitely; manu | [bug] 事件循环卡死后僵尸仍握 18789 端口,parent 重拉全部 EADDRINUSE 死循环,只能手工杀进程恢复(event loop wedge holds listen socket, respawn loop fails with EADDR… #4762(+ PR [PR] fix: wedged proxy self-exits instead of holding its port forever (#4762) #4784):卡死代理握端口不放、重拉 EADDRINUSE 循环;修复方向是"卡死代理自杀退出"。本 issue 报告的是该修复落地后的第二段竞态——卡死代理退出前后的并发重拉窗口里,替补子进程按 twin-collision 自杀、全体客户端同时放弃。两个 PR/issue 串起来才是完整故障域。
- [Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125:bootstrap 在旧实例占端口时新起实例 → goes direct → 裸历史撞 1M 上限。触发场景不同(陈旧常驻实例 vs 本 issue 的恢复竞态),后果链相同。
- dsh-native never reconnects after respawn gave up (healthy proxy listening) → unattributed over-window body refused locally (422) → session unrecoverable | [dsh-native] respawn 放弃后客户端永不重连(代理已健康监听)→ 未归属超窗 body 被本地 422 拒绝、会话不可恢复 #4755:give up 后客户端永不重连。本 issue 多个实例中客户端在代理恢复健康后仍有再次 adopt 成功的记录(12:47:04、13:18:14),不重连不是本案主因,但同一恢复协议域。
- [Bug][dsh] 0.1.208: attributed-but-not-ready direct send has no window guard, and takeover is declared before readiness — long sessions 400 with prompt_too_long | 归因正常但代理未就绪时直发无窗口护栏 + 就绪前就宣告接管 → 长会话必 400 #4894:direct send 无窗口守卫。本 issue 的 47.6MiB 直连泄漏正是该缺口的最大实锤样本。
- v0.1.203 (DSH edition) sporadically triggers "context window exceeded" errors; restarting DSH restores normal operation. | v0.1.203(DSH版)偶发出现“超出上下文窗口”的情况,重启DSH后恢复可用 #4381:同症状(偶发 context window exceeded、重启恢复)。
建议
- 替补 bind 失败先裁决再自杀:EADDRINUSE 时 HTTP 探测现有监听者(日志已有 pid);探测通过才 exit code 3,探测失败(僵尸)则退避重试等待端口释放,而非让位给空气。
- spawner 侧并发仲裁:code 3 的接收方先重试 adopt N 次(含对空口的探测),不要一次验证失败即永久放弃;或多个 host 间共享 spawn 意图(状态文件/锁)避免 N 路并发重拉。
- 把 [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 的 BUSY 判定接入 respawn 触发:TCP 可连 + probe 超时 = 饱和,不触发 respawn(至多背压)。
- give-up → direct 前加窗口守卫(呼应 [Bug][dsh] 0.1.208: attributed-but-not-ready direct send has no window guard, and takeover is declared before readiness — long sessions 400 with prompt_too_long | 归因正常但代理未就绪时直发无窗口护栏 + 就绪前就宣告接管 → 长会话必 400 #4894):未折叠体超过已知上游窗口时拒绝发送并等待重连,而不是把 5.2M tokens 的裸历史打出网。
- index.db 单写者仲裁:恢复窗口内多进程同时初始化持久层会触发 The same sessionId is served by two bili processes simultaneously: interleaved logs from global + pi-native dual instanc | 同一 sessionId 被两个 bili 进程同时服务:全局 + pi-native 双实例日志交织、重复 cold-load(split-brain) #3923/SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959 守卫并留下 DIGEST DRIFT(本轮已实测出现,对应会话 Fold-state reconciliation: classify resent-history divergence (edit/fork/new/system-only) and make compression blocks re-anchorable chain units #1921 fold silently lost)。
日志摘录
2026-10-27T12:46:48.570Z [info] native-attach: drop http://127.0.0.1:18788 (pid 4012): health probe failed but TCP connect succeeded — instance is BUSY (event loop saturated), not dead (#3654)
2026-10-27T13:20:53.882Z [info] native-attach: drop http://127.0.0.1:18788 (pid 5148): health probe failed
2026-10-27T13:20:56.934Z [error] [storage] another process (pid 5148) is already writing C:\...\index.db — two processes on one index.db corrupt it (#3923/#3959)
2026-10-27T13:21:00.977Z [warn] [dsh-client] proxy bootstrap failed — model traffic goes direct (uncompressed): bili: proxy child exited before becoming healthy (code 3 (twin-collision signal)) — nothing is listening on port 18788
2026-10-27T13:21:00.977Z [warn] [dsh-client] proxy respawn gave up — model traffic goes direct (uncompressed)
Activity
[-][Bug][dsh] 恢复竞态:卡死孪生死后并发重拉,替补子进程在端口仍被僵尸占据时按 twin-collision 自杀(code 3)→ 全体客户端同时放弃、裸历史直连撞上游 1M token 上限;并触发 index.db 双写者互斥(DIGEST DRIFT)[/-][+][Bug][dsh] Recovery race: after a stuck twin dies, concurrent re-pull causes the substitute child process to self-termin | [Bug][dsh] 恢复竞态:卡死孪生死后并发重拉,替补子进程在端口仍被僵尸占据时按 twin-collision 自杀(code 3)→ 全体客户端同时放弃、裸历史直连撞上游 1M token 上限;并触发 index.db 双写者互斥(DIGEST DRI…[/+]ranxianglei commentedon Oct 27, 2026
(ai: vllm-qwen/qwen3.8-27b)
查重:与 #4811 / #4894 同家族,但不判简单重复——本单报告的是那两个修复之后的「第二阶段竞态」,且 master 上仍有一个已复现的残留缺陷,保留本单跟踪并修复中。
一、你日志里的主链路,在你版本之后已经修掉了(版本滞后)
二、但有一处 master 至今仍在的残留缺陷,就是你日志里的 index.db 双写者 + DIGEST DRIFT 那半边
逐行核对 master 代码确认:替补子进程启动时只做了「健康探针答得上且身份匹配才算孪生」(#4341 verified-adopt)这一个前置判定。当端口被一个正在死掉/事件循环卡死的同构建孪生占据时——socket 还接受连接、但健康探针无响应(正是 #3654 三态探测定义过的"alive-but-unserviceable")——这个判定不命中,子进程就继续往下走:先完成持久化初始化(打开 index.db),然后自己的 listen 才报 EADDRINUSE,最后才按 twin-collision 退出 code 3。也就是说每一轮恢复风暴里,替补子进程都在僵尸底下开了一次数据库——两个进程共写一个 index.db 的窗口(#3923/#3959 守卫报的那条
another process is already writing)就是这么来的。最终退出码和修复前一样是 3,区别只是不再在僵尸底下碰库。补充两点边界说明:
[storage] another process … already writing守卫本身只是报告(advisory),不会阻止开库——所以它拦不住这个窗口,只能暴露它。三、修法(已在实现,PR 稍后给出)
把裁决前移到持久化之前:子进程启动时若发现 registry 里有同构建、同车道、同端口的孪生记录,且其 socket 接受连接但健康探针沉默(#3654 的 dying/wedged 签名),直接在持久化初始化前按 twin-collision 退出 code 3——结果与旧路径完全等价(旧路径迟早也会以同一个 code 3 退出),只是不再先开一次库。spawner 侧收到 code 3 走 #4811 已有的自愈重试,僵尸释放端口后下一轮绑定成功。
判定严格保持「正证据才裁决」:没有监听者 = 放行(stale marker 误杀正常启动是 #4341 的教训);探针有应答 = 交给既有 verified/mismatch 路径,本门不越权。
回归测试:单测真值表(socket 生死 × 探针应答/沉默 × 形状漂移/指纹/pid/port/lane 各维度)+ 真实 CLI 子进程的顺序验证(僵尸占端口 → code 3 且 index.db 不存在;ghost marker 占未监听端口 → 正常启动且 index.db 存在)。
标签:bug, dsh, scope/wiring, p1(报障时的用户体验属核心链路停摆级;残留的双写窗口归存储卫生类,随本单一并收敛)
建议:先升级下个 release 观察主症状是否消失;若升级后仍见 DIGEST DRIFT,请留 log 到这里,我们沿本单的残留修复继续追。
ranxianglei commentedon Oct 27, 2026
(ai: vllm-qwen/qwen3.8-27b)
收工:残留缺陷已修复,PR #5004 已提交(Closes #4977)。
按八问:
another process … is already writing与 DIGEST DRIFT 相关噪音。still held by a same-build bili instance … health probe is silent)。回归测试:单测真值表(socket 生死 × 探针应答/沉默 × 形状/指纹/pid/port/lane 各维度)+ 两行真实 CLI 子进程(僵尸占端口 → exit 3 且 BILI_SESSIONS_DIR 下 index.db 不存在,这是本缺陷的回归钉;ghost marker 占未监听端口 → 正常启动且 index.db 存在)。本地定向验证全过,全量回归交给 CI。
合并判定:等 CI 绿 + owner 合入。请你升级到包含本修复的 release 后观察主症状是否消失;若升级后仍见 DIGEST DRIFT,把 log 留到这里,我们沿 #4986 继续追。
ranxianglei commentedon Oct 27, 2026
(ai: vllm-qwen/qwen3.8-27b)
PR #5004 的 CI 已全绿(30/30:test ubuntu/windows × node 22/24 × 3 shards、static-gates、gate、mysql-contract 全部通过,0 失败)。可以合入了。
test(#4977): swallow zombie-socket errors in the issue4977 real-CLI rows
test(#5018): retarget the unverified-twin #4982 e2e row to the pre-pe…
Merge pull request #5004 from ranxianglei/2026-10-28_4977-present-twi…
fix(ranxianglei#4977): exit twin-collision before persistence init wh…