Skip to content

[Bug][dsh] Recovery race: after a stuck twin dies, concurrent re-pull causes the substitute child process to self-termin | [Bug][dsh] 恢复竞态:卡死孪生死后并发重拉,替补子进程在端口仍被僵尸占据时按 twin-collision 自杀(code 3)→ 全体客户端同时放弃、裸历史直连撞上游 1M token 上限;并触发 index.db 双写者互斥(DIGEST DRI… #4977

Closed

Description

@CelestNya

AI 声明:本 issue 由 AI(WorkBuddy)辅助报告人整理生成——日志引用均为 bili.log 原文(仅掩码内网主机名、缩写会话 UUID),代码结论按【实测】/【推断】逐条标注证据等级;请以维护者的独立核实为准。

[Bug][dsh] 恢复竞态:serving twin 卡死后各客户端并发重拉,替补子进程在端口仍被僵尸(随后即将空闲)占据时按 twin-collision 自杀(code 3),全体客户端同时放弃 → 裸历史直连撞上游 1M token 上限;并发重拉还触发 index.db 双写者互斥(DIGEST DRIFT)

建议标签:bug / dsh / scope/wiring / p1


TL;DR (EN)

When a saturated serving twin wedges and its clients all respawn concurrently, each replacement child bind-fails with EADDRINUSE against the dying zombie (which still holds the port and the index.db write lock), and self-exits with code 3 (twin-collision signal) — by a protocol that assumes "port occupied = a healthy twin is serving". Seconds later the zombie fully dies and the port is empty ("nothing is listening on port 18788"), but every spawner has already concluded twin-collision and gave up simultaneously → all model traffic goes direct (uncompressed) and raw histories hit the upstream 1,000,000-token gate. The concurrent respawn window also trips the index.db two-writer guard and produces a unified shadow DIGEST DRIFT.

环境

  • billion-context 0.1.208(npm,node ...\node_modules\billion-context\dist\index.js start --host 127.0.0.1 --port 18788,lane dsh)
  • 宿主:DSH 桌面版(Windows 11 x64),同机多个并发 host 进程各带一个 in-process dsh-client,共享 18788
  • 上游:OpenAI 兼容网关,输入硬上限 1,000,000 token(经本地 ccproxy 转发)

时间线(2026-10-27,bili.log 原文,UTC)

之前 serving twin 已历经多轮换代(pid 7388 → 22120 → 4012 → 25572 → 5148),每换代一次即一轮风暴;以下是最清晰的一轮:

时刻 事件
12:46:48 native-attach: drop http://127.0.0.1:18788 (pid 4012): health probe failed but TCP connect succeeded — instance is BUSY (event loop saturated), not dead (#3654) ← 插件此时能正确区分饱和与死亡
13:20:43 serving twin(pid 5148)僵死:20+ 条 [conn#NNNN] tcp closed reason=error(ECONNRESET);同会话 resident pool 达 417MB(evicted ... (weight 417301854 bytes))
13:20:49–51 [storage] 49 tail pointer(s) ... reference missing rows + [storage] unified shadow DIGEST DRIFT ... unified replay != legacy load view
13:20:53 native-attach: drop http://127.0.0.1:18788 (pid 5148): health probe failed → 所有客户端同时 spawn 替补子进程
13:20:56 替补子进程撞上僵尸:[storage] another process (pid 5148) is already writing .../index.db — two processes on one index.db corrupt it (#3923/#3959)(端口同样仍被 5148 占据)
13:20:57 serving-twin adoption failed verification — continuing with the standard post-exit re-discovery
13:21:00.977 [dsh-client] proxy bootstrap failed — ...: bili: proxy child exited before becoming healthy (code 3 (twin-collision signal)) — nothing is listening on port 18788 + [dsh-client] proxy respawn gave up — model traffic goes direct (uncompressed)
随后 5148 彻底退出、端口空闲,但所有客户端已放弃;裸流量直连上游。同日 12:47、13:17 各重演一轮(两条 gave up 均在 log 中)

字节级对账(实测):直连期间 ccproxy 侧收到与 bili.log 会话 inbound=47.6MiB 完全一致的原始请求体(49,972,029B,≈5.2M tokens)→ 上游 1M token 校验拒绝;同一会话经 bili 折叠后仅 400–450KB(缓存命中 99%)。时拒时收(上游校验非确定)叠加直连间歇出现,用户侧表现即"多个会话同时报超出上下文、点继续又时好时坏"(与 #4381 症状一致)。

成因分析

  1. 【实测】code 3 自杀是协议内建行为(EADDRINUSE → duplicate loses → exit code 3,spawner 读作 adopt 信号,Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341/autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044、[Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654)。
  2. 【实测】该协议隐含假设"端口被占 = 有健康孪生在服务"。恢复窗口内此假设必假:占端口的是僵死 serving twin(未释放 socket、未释放 index.db 写锁),几秒后它彻底退出、端口真空。
  3. 【推断】多 host 并发 respawn 使替补子进程之间也互为 duplicate:赢家的 spawner 收编成功,输家的 spawner 验证失败;一旦赢家也在初始化窗口内被 index.db 双写者守卫绊住,就出现"全员 code 3 + 端口空"的最坏分支。
  4. 【实测】12:46:48 表明三态健康探测([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 修复)能识别 "BUSY not dead",但该判定未接入 respawn 触发条件——13:20:53 仍按死亡处理并触发风暴。

与现有 issue / PR 的关系(查重)

建议

  1. 替补 bind 失败先裁决再自杀:EADDRINUSE 时 HTTP 探测现有监听者(日志已有 pid);探测通过才 exit code 3,探测失败(僵尸)则退避重试等待端口释放,而非让位给空气。
  2. spawner 侧并发仲裁:code 3 的接收方先重试 adopt N 次(含对空口的探测),不要一次验证失败即永久放弃;或多个 host 间共享 spawn 意图(状态文件/锁)避免 N 路并发重拉。
  3. 把 [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 的 BUSY 判定接入 respawn 触发:TCP 可连 + probe 超时 = 饱和,不触发 respawn(至多背压)。
  4. give-up → direct 前加窗口守卫(呼应 [Bug][dsh] 0.1.208: attributed-but-not-ready direct send has no window guard, and takeover is declared before readiness — long sessions 400 with prompt_too_long | 归因正常但代理未就绪时直发无窗口护栏 + 就绪前就宣告接管 → 长会话必 400 #4894):未折叠体超过已知上游窗口时拒绝发送并等待重连,而不是把 5.2M tokens 的裸历史打出网。
  5. index.db 单写者仲裁:恢复窗口内多进程同时初始化持久层会触发 The same sessionId is served by two bili processes simultaneously: interleaved logs from global + pi-native dual instanc | 同一 sessionId 被两个 bili 进程同时服务:全局 + pi-native 双实例日志交织、重复 cold-load(split-brain) #3923/SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959 守卫并留下 DIGEST DRIFT(本轮已实测出现,对应会话 Fold-state reconciliation: classify resent-history divergence (edit/fork/new/system-only) and make compression blocks re-anchorable chain units #1921 fold silently lost)。

日志摘录

2026-10-27T12:46:48.570Z [info] native-attach: drop http://127.0.0.1:18788 (pid 4012): health probe failed but TCP connect succeeded — instance is BUSY (event loop saturated), not dead (#3654)
2026-10-27T13:20:53.882Z [info] native-attach: drop http://127.0.0.1:18788 (pid 5148): health probe failed
2026-10-27T13:20:56.934Z [error] [storage] another process (pid 5148) is already writing C:\...\index.db — two processes on one index.db corrupt it (#3923/#3959)
2026-10-27T13:21:00.977Z [warn] [dsh-client] proxy bootstrap failed — model traffic goes direct (uncompressed): bili: proxy child exited before becoming healthy (code 3 (twin-collision signal)) — nothing is listening on port 18788
2026-10-27T13:21:00.977Z [warn] [dsh-client] proxy respawn gave up — model traffic goes direct (uncompressed)

Activity

  1. changed the title [-][Bug][dsh] 恢复竞态:卡死孪生死后并发重拉,替补子进程在端口仍被僵尸占据时按 twin-collision 自杀(code 3)→ 全体客户端同时放弃、裸历史直连撞上游 1M token 上限;并触发 index.db 双写者互斥(DIGEST DRIFT)[/-] [+][Bug][dsh] Recovery race: after a stuck twin dies, concurrent re-pull causes the substitute child process to self-termin | [Bug][dsh] 恢复竞态:卡死孪生死后并发重拉,替补子进程在端口仍被僵尸占据时按 twin-collision 自杀(code 3)→ 全体客户端同时放弃、裸历史直连撞上游 1M token 上限;并触发 index.db 双写者互斥(DIGEST DRI…[/+] on Oct 27, 2026
  2. ranxianglei commented on Oct 27, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重:与 #4811 / #4894 同家族,但不判简单重复——本单报告的是那两个修复之后的「第二阶段竞态」,且 master 上仍有一个已复现的残留缺陷,保留本单跟踪并修复中。

    一、你日志里的主链路,在你版本之后已经修掉了(版本滞后)

    二、但有一处 master 至今仍在的残留缺陷,就是你日志里的 index.db 双写者 + DIGEST DRIFT 那半边

    逐行核对 master 代码确认:替补子进程启动时只做了「健康探针答得上且身份匹配才算孪生」(#4341 verified-adopt)这一个前置判定。当端口被一个正在死掉/事件循环卡死的同构建孪生占据时——socket 还接受连接、但健康探针无响应(正是 #3654 三态探测定义过的"alive-but-unserviceable")——这个判定不命中,子进程就继续往下走:先完成持久化初始化(打开 index.db),然后自己的 listen 才报 EADDRINUSE,最后才按 twin-collision 退出 code 3。也就是说每一轮恢复风暴里,替补子进程都在僵尸底下开了一次数据库——两个进程共写一个 index.db 的窗口(#3923/#3959 守卫报的那条 another process is already writing)就是这么来的。最终退出码和修复前一样是 3,区别只是不再在僵尸底下碰库。

    补充两点边界说明:

    三、修法(已在实现,PR 稍后给出)

    把裁决前移到持久化之前:子进程启动时若发现 registry 里有同构建、同车道、同端口的孪生记录,且其 socket 接受连接但健康探针沉默(#3654 的 dying/wedged 签名),直接在持久化初始化前按 twin-collision 退出 code 3——结果与旧路径完全等价(旧路径迟早也会以同一个 code 3 退出),只是不再先开一次库。spawner 侧收到 code 3 走 #4811 已有的自愈重试,僵尸释放端口后下一轮绑定成功。
    判定严格保持「正证据才裁决」:没有监听者 = 放行(stale marker 误杀正常启动是 #4341 的教训);探针有应答 = 交给既有 verified/mismatch 路径,本门不越权。

    回归测试:单测真值表(socket 生死 × 探针应答/沉默 × 形状漂移/指纹/pid/port/lane 各维度)+ 真实 CLI 子进程的顺序验证(僵尸占端口 → code 3 且 index.db 不存在;ghost marker 占未监听端口 → 正常启动且 index.db 存在)。

    标签:bug, dsh, scope/wiring, p1(报障时的用户体验属核心链路停摆级;残留的双写窗口归存储卫生类,随本单一并收敛)

    建议:先升级下个 release 观察主症状是否消失;若升级后仍见 DIGEST DRIFT,请留 log 到这里,我们沿本单的残留修复继续追。

  3. ranxianglei commented on Oct 27, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    收工:残留缺陷已修复,PR #5004 已提交(Closes #4977)。

    按八问:

    1. 是什么问题:你日志里的链路一半是版本滞后(主链已在 v0.1.208 之后修掉,见上一条回复),另一半是 master 至今仍在的残留缺陷——恢复风暴里替补子进程在发现端口被僵尸占据之前先打开了 index.db,两个进程同写一个库,报出 another process … is already writing 与 DIGEST DRIFT 相关噪音。
    2. 什么原因导致的:子进程启动的前置裁决只认「健康探针有应答且身份匹配」的孪生(Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 verified-adopt)。占据者若是正在死掉的僵尸(socket 还接受连接、探针沉默),判定不命中,进程就一路走到持久化初始化、打开数据库,最后自己的 listen 才撞上 EADDRINUSE。裁决发生在开库之后,而不是之前。
    3. 影响范围与严重程度:触发条件 = 同构建同车道、占据者处于将死态的恢复轮次。后果是同库双写窗口 + 误导性存储日志 + 多余的启动/退出churn;你看到的实际数据损坏(fold 丢失、DIGEST DRIFT)主要来自僵尸自己写了一半——DRIFT 行出现在替补子进程出生之前——那部分的处置由 Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 单独跟踪。残留缺陷本身属存储卫生级;报障时的核心链路停摆(全体放弃 + 5MB 裸连撞 1M 上限)属版本滞后,已随 [PR] fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) #4825/[PR] fix(#4894): arm the direct-fallback oversize guard on dsh not-ready sends #4904 修复。
    4. 怎么修复的:把裁决前移——启动时发现「registry 里有同构建孪生记录、其 socket 接受连接但健康探针沉默」,直接在持久化初始化前以 code 3(twin-collision)退出。结果与旧路径完全等价(旧路径迟早也以同一 code 3 退出),只是更早决定且不再碰库;spawner 收到 code 3 走 dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 已有的自愈重试,僵尸释放端口后下一轮绑定成功。保持正证据裁决纪律:没有监听者 = 放行(防 stale marker 误杀正常启动),探针有应答 = 交还既有 verified/mismatch 路径。
    5. 是否有回归问题:逐项审过既有行为——verified-adopt 路径(Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 原逻辑未动,仅抽出公共尾部函数)、late EADDRINUSE 策略(未改,新门只在旧策略本就会以同 code 3 退出的情形触发)、stale marker 下的正常启动(负向测试行实证不受影响);兄弟套件 tests/issue4341-early-twin-adoption.test.ts 本地重跑全绿(7/7)。无阈值/时序/输出格式漂移。行为变更披露:原先「开库→锁文件覆盖→exit 3」的启动现在变成「直接 exit 3、不碰库」;该退出新增一条日志变体(still held by a same-build bili instance … health probe is silent)。
    6. 是否破坏缓存:不涉及——纯启动期路径,与前缀缓存/会话缓存/重试游标无关。
    7. 是否改了设计方向:没有。同一个协议(code 3 adopt 信号语义不变)内把一个已有决策点前移,未引入新机制。
    8. 是否引入新配置:没有。只复用既有 probeOwnerInfo / tcpConnectable 默认参数。

    回归测试:单测真值表(socket 生死 × 探针应答/沉默 × 形状/指纹/pid/port/lane 各维度)+ 两行真实 CLI 子进程(僵尸占端口 → exit 3 且 BILI_SESSIONS_DIR 下 index.db 不存在,这是本缺陷的回归钉;ghost marker 占未监听端口 → 正常启动且 index.db 存在)。本地定向验证全过,全量回归交给 CI。

    合并判定:等 CI 绿 + owner 合入。请你升级到包含本修复的 release 后观察主症状是否消失;若升级后仍见 DIGEST DRIFT,把 log 留到这里,我们沿 #4986 继续追。

  4. ranxianglei commented on Oct 27, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    PR #5004 的 CI 已全绿(30/30:test ubuntu/windows × node 22/24 × 3 shards、static-gates、gate、mysql-contract 全部通过,0 失败)。可以合入了。

  5. added 3 commits that reference this issue on Oct 27, 2026
  6. added a commit that references this issue on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions