Skip to content

Storage banner falsely warns «second process writing index.db» when same-lane bili instances share the machine-global sessions dir | 同车道多实例共享存储误报损坏告警 #5014

Open

Description

@ranxianglei

(ai: google/gemini-2.5-pro)

现象

同一台机器上同时运行多个同车道 bili 实例(日常形态:每个终端一个 bili pi -r,各自 spawn 代理),它们共享机器级 sessions 目录(~/.local/share/billion-context/sessions/,设计如此——AGENTS.md「machine-global facilities stay SHARED」)。任何一个实例在另一个存活实例之后打开统一存储时,Web 面板弹出红色告警:

检测到第二个进程在写同一个 index.db —— 另一个进程(pid N)正在写入本会话目录的数据库……请关闭多余实例:一个会话目录只能由一个实例服务。

日志同时打 [error] another process (pid N) is already writing <path> — two processes on one index.db corrupt it。

这是误报:每个实例只写自己会话的行,SQLite WAL + busy_timeout=5000 在多进程间串行化写入;真正危险的情形(同一个会话被两个实例提交)已有独立的 split-brain 检测与告警(#3952)。

触发条件

  1. 机器上 ≥2 个 bili 实例存活,共享同一个机器级 sessions 目录;
  2. 后开的实例打开 index.db 时,<db>.lock 里是另一个存活实例的 pid;
  3. 该 pid 能在实例注册表(<state>/instances/*.json)里找到,但 lane 相同(如都是 pi)。

根因

src/storage/store.ts(#4175 引入的 second-process 检测)把「良性」条件写成 不同车道:

const benign = peer !== undefined && ownLane !== undefined && peer.lane !== undefined && peer.lane !== ownLane;

同车道的已知实例全部落入「损坏告警」分支。跨车道共享是支持拓扑、同车道多实例同样是支持拓扑(多终端工作流的固有形态),两者在数据库层面的安全性完全相同。

修复方向

Activity

  1. ranxianglei commented on Oct 28, 2026

    @ranxianglei
    OwnerAuthor

    (ai: google/gemini-2.5-pro)

    修复已提:PR #5015(Closes #5014)。主题类别:E 存储层。

  2. ranxianglei commented on Oct 28, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    收工摘要(完整验证结论见上一条回复):

  3. aweds13 commented on Oct 31, 2026

    @aweds13

    Same-lane collision here is not "one terminal each" — one host process spawned both

    Confirming the benign condition described above bites even when there is exactly one user-visible host, and adding the consequence I could measure.

    Environment: bili 0.1.209 (39a0ada), Windows, single DSH (DeepSeek Harness) process.

    ParentProcessId  ProcessId  Endpoint
    29300            23420      127.0.0.1:18795   (this is the instance I query)
    29300            21708      127.0.0.1:18796   (working set 2.4 GB)
    

    Both children of the same parent pid 29300; no second terminal, no bili pi -r, no build change. On 18795, /__bili/status reports:

    "secondProcess": {"pid": 21708, "benign": false, "lane": "dsh"}

    Same lane ⇒ peer.lane !== ownLane is false ⇒ the "two processes on one index.db corrupt it" branch, exactly as the root cause in this issue predicts. So the trigger is not multiple terminals spawning proxies; a single host instance appears to be enough to produce the collision (possibly the +1 port ladder discussed in #4568).

    It is not only a banner

    While both were up, the store on 18795 reported a real degradation, not just noise:

    storage.requested: full
    storage.effective: legacy
    storage.reason: index database failed to open (Error: database is locked)
    

    It recovered to effective: full later, so this is transient lock contention rather than corruption — which supports the issue's premise that the same-lane case is benign at the database level. But in 0.1.209 the contention still lands the serving instance in legacy mode for a while, so the cost of misclassifying it is not cosmetic. (Related: #5324 covers the silent-degrade path; master reportedly refuses to start instead — this lane is on 0.1.209.)

    /__bili/status also flagged a split-brain claim on one session id from both instances, i.e. the second, independent channel this issue asks to keep unchanged.

    Question

    Is one host spawning two instances on consecutive ports (18795 + 18796) the intended topology, or a launch-ladder leak? If intended, the same-lane benign fix here matters more than the "multiple terminals" framing suggests, because every DSH user would hit it by default.

  4. ranxianglei commented on Oct 31, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    感谢这份证据——单宿主双实例的形态和实测降级都很关键。逐点回复:

    1. 你的 status JSON 正是本 issue 诊断的代码路径。同车道 peer(lane: "dsh")在旧代码里 peer.lane !== ownLane 为假 → benign: false → 红色损坏告警。PR #5015 修掉后,你这个场景(两个 pid 都在实例注册表里、都是 bili 子进程)会判为 benign → info 日志 + 信息样式横幅,不再报「corrupt it」。

    **2. 回答你的问题:不是预期拓扑,是已知缺陷,且已有修复在途。**分两层说:

    3. 你实测的降级是真实的,且已有记录:storage.effective: legacy + database is locked 这个签名在 #4566 的证据评论里一字不差地出现过(第二实例共目录的直接数据面后果)。两点澄清:

    4. split-brain 标记:独立通道按设计工作(#3952)——两个实例确实提交了同一个 session id,这正是本 issue 明确保留不动的那条告警。#5064 落地后双实例状态消失,这条也不会再触发。

    版本提示:你在 0.1.209 上;#5015 / #5064 / #4878 都会随后续 release 发布。在那之前,你这套 Windows 单宿主出现双实例属于上述已知缺陷,不是配置问题。

  5. ssamurias commented on Nov 1, 2026

    @ssamurias

    Data point from the dsh lane (Windows, 0.1.209, 2026-11-01 18:49 local UTC+8) — the banner may be benign, but same-lane sharing has a real consequence once the other instance is an unreachable orphan:

    18:49:48 [storage] unified sidecar failed to open …\sessions\index.db (Error: database is locked)
             — staying legacy-only; delete the file to reset the index
    18:49:51 port 18789: registry twin 0f60d531 (pid 20724) no longer answers as itself — the socket moved on
             while it drained; treating the port as foreign-held instead of dying twin-collision (#4982)
    18:49:52 [instances] another bili instance is running (pid 20724, http://127.0.0.1:18789) on lane "dsh" …
    18:52:10 [storage] second-process pid 20724 has exited — stale warning cleared   ← only after I killed it
    

    Two things this adds to the "false positive" framing:

    1. legacy-only is not just a banner — the replacement latched off the unified store for its entire life, so shadow verification and cross-process dedupe were gone for the whole session (which then also produced a unified shadow DIGEST DRIFT error once it re-opened).
    2. [instances] another bili instance is running (pid N) is reported for a peer that is provably unreachable and whose owner is dead. Knowing it is a known instance is not enough information here: the benign/malicious split needs a third state — known but wedged.

    Suggestion: when the peer pid maps to a known instance but its health has been unreachable for N seconds and its owner pid is not alive, say that explicitly (peer <pid> is wedged — takeover recommended) instead of folding it into either "benign" or "corruption", and don't fall back to legacy-only silently. Background and the reaping failure that produced the orphan: #5449.

  6. ranxianglei commented on Nov 1, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Both points verified against master — thanks for the clean log capture.

    1. legacy-only latch: confirmed permanent per process. The sidecar opens exactly once at store construction (src/persist.ts:498-500, one-shot UnifiedShadow.maybeOpen); the open-failure catch (src/storage/shadow.ts:237-254) logs the warn you saw, records the failure, and returns null — no retry, ever. The periodic recheck that printed your 18:52:10 line (src/storage/runtime-status.ts:272) only clears the warning when the peer pid exits; it never reopens the store. So yes: after you killed pid 20724, that instance stayed legacy-only until manual restart, even though the lock contention had already cleared. Your later DIGEST DRIFT on reopen is the known multi-instance consequence channel (shadow.ts:423 names a shared sessions dir as the usual cause).

    2. Third state "known but wedged": agreed in principle — with one important interaction. Today (your 0.1.209) that orphan reads as the red corruption alarm; once PR #5015 merges, the same scenario flips to benign info ("informational only") — i.e. the fix removes the over-alarm but would leave misleading "safe to ignore" advice for exactly this case, where the actionable fact is kill/takeover. Two feasibility notes from the code:

    • the instance registry does not record an owner/parent pid (RegistryEntry, src/instance.ts:473-499), so "owner pid is not alive" as stated is not evaluable today without a new field;
    • buildable signals already exist: health probing via the entry's origin (healthOf(origin), instance.ts:273 — already used by lane attach) plus the portReleasedAt marker ([PR] release v0.1.209 #4982), which is set exactly when a twin releases its socket while still draining — the machine-readable form of your "the socket moved on while it drained" line. A wedge verdict of "registry-known + health-unreachable-for-Ns (+ portReleasedAt)" needs no new data.

    Filed as issue #5452 (labels: bug / dsh / scope/storage / p2 — silent capability loss with a workaround, not data loss): checklist items D1 missing third state, D2 one-shot sidecar open with no recovery path, D3 dead-pid-only lock takeover (deliberately NOT "unreachable ⇒ takeover" — a slow-but-live writer is still the legitimate shared-dir topology; grabbing its lock is real corruption). Options A/B/C are listed there for owner decision.

    One correction: #5449 does not resolve to the reaping-failure background you cite — in this repo that number is an unrelated release-workflow PR (emergency publish channel restore). Orphan production (host dies mid-drain, child keeps running while holding locks) is a distinct upstream dependency of this whole scenario; if you have the right tracking number, drop it into #5452 so the chain closes. Until then the issue carries it as an open dependency.

    Workaround meanwhile: kill the orphan (as you did) and restart the replacement — the replacement will not pick up the unified store by itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions