Skip to content

[PR] fix: #7156 storage-corruption fallout — webSummaryRows json_valid guard, .chk close-time gate, first-write corruption classification - #7173

Merged
ranxianglei merged 6 commits into
masterfrom
2026-11-18_issue7156-corruption-fallback
Nov 20, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-11-18_issue7156-corruption-fallback

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Fixes #7156

Summary

#7156 reports two independent problems from a wedged-sidecar takeover incident (bili 0.1.216, old 0.1.215, single DSH host, storage:full, 573MiB index.db):

  1. Dual-writer corruption: the takeover spawned a new sidecar BEFORE the wedged predecessor was reclaimed, so two processes held index.db at once and corrupted it. The user's repro: a 3-row UPDATE from a second connection corrupts the live WAL + auto_vacuum=INCREMENTAL db; the same UPDATE on a snapshot copy is fine.
  2. Silent 0-historical: .recover left 3 blocks with payload_json='' (length 0), and webSummaryRows' json_extract threw malformed-JSON, which unifiedWebSummaries silently swallowed → the panel showed "5 live · 0 historical" (real: 222 sessions / 1700 blocks) with no log.

This PR fixes the non-decision fallout. The root cause of Problem 1 (the second-writer policy) is a resolved owner decision — this PR does NOT change the policy.

Fixes

  1. webSummaryRows (store.ts): wrap payload_json in a json_valid CTE so invalid/empty payloads degrade to NULL (skipped by filters, ignored by SUM) instead of throwing malformed-JSON and failing the whole .all().
  2. unifiedWebSummaries (persist.ts): log a warn when the catch swallows the error, so the "0 historical" fallback is visible in server logs.
  3. close() (store.ts): gate the close-time integrity-cache refresh on no live foreign writer (activeForeignWriter() === null). A process that booted healthy but whose file was corrupted by a concurrent second writer (the [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156 incident) would otherwise stamp a FRESH ok verdict onto the corrupted stats at close, so every later boot skips the rescan (cache hit) and dies on the first write.
  4. maybeOpen (shadow.ts): classify a first-post-open-write corruption (sqliteCorruptionDetail match) as db-corrupt → start WITHOUT the index (operator hand-fix) instead of REFUSING TO START (throw). The handle is abandoned (not close()'d) because close() would run the cache refresh and re-stamp the poisoned verdict.

Behavior changes (disclosed per owner policy 2026-09-22)

Root cause (resolved decision — new evidence warrants reconsideration)

The dual-writer window (Problem 1) is a designed consequence of #5955 (spawn-while-wedged-holder-alive) + the 2026-10-31 owner decision (#5068, closed; implemented as #5294, merged: warn-only, never block, "sharing a local-disk index.db is a supported topology"). The user's #7156 repro empirically REFUTES the "supported topology" premise (a 3-row UPDATE from a second connection corrupts the live WAL + auto_vacuum=INCREMENTAL db; the same UPDATE on a snapshot copy is fine). This is new evidence that may warrant reconsidering the decision. This PR fixes the fallout (symptoms) without changing the policy. The redundant option-B PR #5250 is still open.

Tests

  • tests/issue7156-storage-corruption.test.ts (4 tests: FIX 1 + control, FIX 2, FIX 4).
  • tests/storage-integrity-cache.test.ts (2 tests: FIX 3 + control).
  • All targeted suites green; typecheck clean.

…ty-payload failures

#7156 reports two independent problems from a wedged-sidecar takeover incident
(bili 0.1.216, old 0.1.215, single DSH host, storage:full, 573MiB index.db):
(1) the takeover spawned a new sidecar BEFORE the wedged predecessor was
reclaimed, so two processes held index.db at once and corrupted it (the user's
repro: a 3-row UPDATE from a second connection corrupts the live WAL +
auto_vacuum=INCREMENTAL db; the same UPDATE on a snapshot copy is fine);
(2) .recover left 3 blocks with payload_json='' (length 0), and webSummaryRows'
json_extract threw malformed-JSON, which unifiedWebSummaries silently swallowed
-> the panel showed "5 live / 0 historical" (real: 222 sessions / 1700 blocks)
with no log.

This PR fixes the non-decision fallout (the root cause — the second-writer
policy / takeover timing — is an open owner decision, #5281):

1. webSummaryRows: wrap payload_json in a json_valid CTE so invalid/empty
   payloads degrade to NULL (skipped by filters, ignored by SUM) instead of
   throwing malformed-JSON and failing the whole .all() (store.ts).
2. unifiedWebSummaries: log a warn when the catch swallows the error, so the
   "0 historical" fallback is visible in server logs (persist.ts).
3. close(): gate the close-time integrity-cache refresh on no live foreign
   writer (activeForeignWriter() === null). A process that booted healthy but
   whose file was corrupted by a concurrent second writer (the #7156 incident)
   would otherwise stamp a FRESH ok verdict onto the corrupted stats at close,
   so every later boot skips the rescan (cache hit) and dies on the first
   write. A live foreign writer means the file may have moved outside verified
   writes -> skip the refresh, next boot re-scans fresh (store.ts).
4. maybeOpen: classify a first-post-open-write corruption (sqliteCorruptionDetail
   match) as db-corrupt -> start WITHOUT the index (operator hand-fix) instead
   of REFUSING TO START (throw). The sync quick_check did not run (large store
   / cached verdict), so the corruption surfaced at the first write; the old
   comment "Corruption-shaped failures never reach here" was wrong. The handle
   is abandoned (not close()'d) because close() would run the cache refresh and
   re-stamp the poisoned verdict (shadow.ts).

Behavior changes (disclosed per owner policy 2026-09-22):
- FIX 3: a write-heavy store with a live foreign writer at close now re-scans
  on the next boot (cache miss) instead of a cache hit — aligned with the
  2026-10-30 owner policy "each next boot re-scans fresh"; cost is one extra
  rescan, boot is never blocked (#4609 preserved).
- FIX 4: a first-post-open-write corruption now starts WITHOUT the index
  (envelope-served) instead of REFUSING TO START (throw) — aligned with the
  2026-10-30 owner policy (proven-corrupt -> start without the index).

Tests: tests/issue7156-storage-corruption.test.ts (4 tests: FIX 1 + control,
FIX 2, FIX 4) + tests/storage-integrity-cache.test.ts (2 tests: FIX 3 + control).
All targeted suites green; typecheck clean.
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-18_issue7156-corruption-fallback (1b69c6e)

Latest build — install this exact version

npm install -g billion-context@0.0.11028-pr.7173.9c34b60

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
1b69c6e billion-context@0.0.11028-pr.7173.9c34b60 build
b4af60a billion-context@0.0.11027-pr.7173.9f26db5 build
3591d30 billion-context@0.0.11024-pr.7173.29230f1 build
ca4037e billion-context@0.0.10833-pr.7173.74d3fea build
7ec502f billion-context@0.0.10826-pr.7173.a4c7fa5 build
46c1521 billion-context@0.0.10819-pr.7173.1ab5125 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7173.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei added bug Something isn't working p1 scope/storage labels Nov 18, 2026
@ranxianglei ranxianglei changed the title fix: #7156 storage-corruption fallout — webSummaryRows json_valid guard, .chk close-time gate, first-write corruption classification [PR] fix: #7156 storage-corruption fallout — webSummaryRows json_valid guard, .chk close-time gate, first-write corruption classification Nov 18, 2026
…re, #1248)

The #7156 fix added a spawn to this suite (one bare `node -e` dummy
child as a live PID holder for the <db>.lock fixture in the close-time
cache-refresh gate test). scripts/run-ci-tests.mjs's SPAWN_RE tripwire
(#1248) requires every process-spawning test file to be registered in
scripts/live-files.mjs after reviewing its concurrency assumptions.

Concurrency review: single bare node child (setTimeout 30s backstop),
stdio ignore, no ports, no servers, mkdtemp-isolated per test,
SIGKILL'd in t.after — same pattern as the #5046/#4612 entries, so the
zone band is irrelevant.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Reviewed the PR and fixed its CI red. Summary:

Code review — all 4 fixes verified correct

  1. FIX 1 (webSummaryRows json_valid CTE) — correct. json_valid(payload_json) gates each extract: invalid/empty payloads degrade to NULL (skipped by the topic/summary filters, ignored by SUM), COUNT(*) is unchanged (counts all blocks), and behavior is identical for valid payloads. The CTE is referenced 4× so SQLite materializes it once — no worse than the previous 4 separate scans.
  2. FIX 2 (unifiedWebSummaries warn log) — correct; uses loggerLog per KDD feat: OpenAI Responses API support (/v1/responses) #4, so the "0 historical" fallback is now diagnosable in server logs.
  3. FIX 3 (close-time cache gate) — correct. activeForeignWriter() (src/storage/store.ts:804) excludes own pid (pid !== process.pid), so calling it while this process's own lockfile is still in place (close() releases the lock after this block) is safe. The disclosed behavior change (re-scan next boot when a live foreign writer is present at close) is bounded to one extra rescan and never blocks boot ([Bug] 0.1.205+: open-time quick_check is O(index.db) — 14 GB takes 260 s, so the 60 s spawn budget always kills the child and the host stays on direct | 0.1.205+ 开库 quick_check 使启动变成 O(库大小):14GB 库 260 秒 → 必超 60 秒 spawn 预算 → 整生命周期直连 #4609 preserved).
  4. FIX 4 (maybeOpen corruption classification) — correct. The outer catch classifies corruption first via sqliteCorruptionDetail (matches SQLITE_CORRUPT/SQLITE_NOTADB codes or message markers), then starts WITHOUT the index per the 2026-10-30 owner policy; the handle is abandoned (not close()'d) so the close-time cache refresh cannot re-stamp the poisoned verdict. Non-corruption errors keep the existing REFUSING-TO-START throw path.

CI red — root cause + fix

The red on head 46c1521 was PR-specific, not inherited: all 11 test shards died in ~17s at the pretest tripwire — process-spawning test files missing from LIVE_FILES in scripts/live-files.mjs: tests/storage-integrity-cache.test.ts (#1248). The PR added a spawn to that suite (one bare node -e child as a live PID holder for the <db>.lock fixture in the new close-time gate test) without registering it; the master version of the file has no spawn, so master is unaffected.

Fixed in-branch: appended tests/storage-integrity-cache.test.ts to LIVE_FILES (scripts/live-files.mjs) with the #1248 concurrency review — single bare node child (30s self-timeout backstop), stdio ignore, no ports/servers, mkdtemp-isolated per test, SIGKILL'd in t.after; same pattern as the #5046/#4612 entries, so the zone band is irrelevant. Own labeled commit 7ec502fba7, pushed to the PR branch. Targeted suites: 14/14 pass locally (both touched test files).

CI status (live, head 7ec502f): re-triggered by the push; static-gates (ubuntu/24) + mysql-contract already green, test shards in progress. Mergeable once the shards go green.

八问

  1. 是什么问题:PR 的 CI 全红——11 个 test 分片各 ~17s 即挂,不是测试失败,是 pretest tripwire(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248):PR 给 tests/storage-integrity-cache.test.ts 加了 spawn(close-time 缓存门控测试用的 PID-holder 子进程)却没登记进 scripts/live-files.mjs 的 LIVE_FILES。
  2. 什么原因导致的:PR 加了起子进程的测试但漏了 tripwire 要求的登记步骤。master 版该文件无 spawn,故为 PR 特有失败,非继承(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 当前已知红均为不同签名,已逐条核对)。
  3. 影响范围与严重程度:仅本 PR 的 CI 被阻塞;master 不受影响、无用户侧影响。严重程度:流程性(CI 红),但阻塞一个 p1 存储修复的合并。
  4. 怎么修复的:在 LIVE_FILES 追加登记该文件(append-only,不改 slot 编号),附 tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248 并发假设评审(单个裸 node 子进程、30s 自超时兜底、stdio ignore、无端口无服务、mkdtemp 隔离、t.after SIGKILL——与 SQLite cache has no graceful degradation when the driver is unavailable: llm/stream throws `Cannot read properties of un | sqlite 缓存在驱动不可用时没有降级:llm/stream 每回合抛 Cannot read properties of undefined (reading 'prepare')(DSH/Electron 打包环境) #5046/storage gc full VACUUM holds the write lock across the whole O(db) rewrite with no progress — online proxy shadow disabled mid-run | storage gc 全量 VACUUM 全程持写锁且无进度:在线代理影子层被降级 #4612 同模式,zone band 无关)。独立标注 commit 7ec502fba7,已推到 PR 分支;本地定向测试 14/14 绿。
  5. 是否有回归问题:无——改动只是 CI 脚本里的数组追加,不触碰任何运行时代码。PR 本身的 4 个修复也逐项审过(见上):对合法数据行为保持,两处行为变更均按 2026-10-30 owner policy 在 PR 正文披露,审过成立。
  6. 是否破坏缓存:不涉及。(PR 的 FIX 3 有意改变 integrity 判决缓存的写入时机,已披露,审过成立。)
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有。

判定:CI 修复已推送,等分片转绿即可合并;根因(双写者策略/takeover 时序)仍按 PR 正文所述在 #5281 待 owner 拍板,本 PR 不动策略。

…liberately abandoned handle)

CI (windows/22 shard) failed the FIX 4 test in its teardown:
  EBUSY: resource busy or locked, unlink '...\bili-issue7156-...\index.db'
  (tests/issue7156-storage-corruption.test.ts:209, the finally rmSync)

Root cause: the FIX 4 path under test abandons the store handle on purpose
(owner policy 2026-10-30 — a runtime corruption finding must not run
close()'s cache refresh, which would re-stamp the poisoned verdict).
Linux unlinks an open file fine; Windows cannot, so the teardown rmSync
throws EBUSY and the test fails although every assertion passed.

Fix: the repo's established pattern for exactly this (leaked ephemeral
handle on Windows) — rmBestEffort with bounded retries (maxRetries 10,
retryDelay 100ms), swallowed on persistent EBUSY/EPERM; same helper as
issue3397/issue4986/issue6780 suites. Only the FIX 4 test needed it: the
other three tests close their stores before the teardown rmSync.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI follow-up — both remaining reds accounted for; the PR-specific one is fixed.

After the LIVE_FILES fix (7ec502f), CI re-ran: 9 of 11 test shards green. Both reds are on windows/22:

  1. windows/22/shard5 (job 134884805861): the only real failure is #5955 processBirthStamp: own pid resolves to a stable non-empty stamp — the known [Bug][dsh] 0.1.212: Zombie proxy holds port + fingerprint branch only skips without terminating → oversized sessions stu | [Bug][dsh] 0.1.212: 僵尸代理占端口 + fingerprint 分支只 skip 不 terminate -> 超大会话持续 422(BUSY 判定 × EADDRINUSE 死锁) #5955 windows flake already recorded in CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 — plus 35 cancellations from a Responses WS suite hang cascade (event loop drained while the suite's promises were still pending; fail 1, cancelled 35). Neither PR test file appears in that shard; neither failure touches the PR diff.
  2. windows/22/shard3 (job 134884805624): 1 failure — the PR's own FIX 4 test, but failing in teardown, not in assertions:
    EBUSY: resource busy or locked, unlink '...\bili-issue7156-...\index.db' (tests/issue7156-storage-corruption.test.ts:209, the finally rmSync).
    Root cause: the FIX 4 path under test abandons the store handle on purpose (owner policy 2026-10-30 — a runtime corruption finding must not run close()'s cache refresh, which would re-stamp the poisoned verdict). Linux unlinks an open file fine; Windows keeps index.db locked → EBUSY → the test fails although every assertion passed.

Fix (commit ca4037e, pushed): the repo's established pattern for exactly this (leaked ephemeral handle on Windows) — rmBestEffort with bounded retries (maxRetries 10, retryDelay 100ms), swallowed on persistent EBUSY/EPERM; same helper as the issue3397/issue4986/issue6780 suites. Only the FIX 4 teardown needed it — the other three tests close their stores before the rmSync. Targeted suites 14/14 green locally; CI re-running on ca4037e (shards queued).

八问(更新):

  1. 是什么问题: PR 自带的 FIX 4 测试在 Windows CI 上 teardown 失败(EBUSY unlink index.db),测试断言本身全部通过。
  2. 什么原因导致的: FIX 4 被测路径按设计 abandon 而不 close store 句柄(owner policy 2026-10-30:运行时损坏发现不得跑 close() 的缓存刷新,否则重盖中毒的 .chk 判决);Linux 允许 unlink 打开中的文件,Windows 不允许 → 清理时 EBUSY。
  3. 影响范围与严重程度: 仅 Windows CI 上该测试的清理阶段;不影响产品行为,Linux 侧不受影响。
  4. 怎么修复的: 采用仓库既有的 rmBestEffort 模式(有界重试 + 吞掉持续 EBUSY/EPERM),与 Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397/Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986/0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 套件同构;只改 FIX 4 一处 teardown,不动其他测试。
  5. 是否有回归问题: 无——只改测试清理代码,不动产品代码;其余三个测试本就先 close 再删,未改动。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及——生产路径的 abandon 行为保持原样,那正是 owner policy 2026-10-30 要求的行为;改的只是测试侧清理。
  8. 是否引入了新的配置: 不涉及。

Verdict: both deterministic PR-specific reds (LIVE_FILES tripwire + Windows teardown EBUSY) are now fixed; the remaining win22-s5 red is the known inherited #5955 flake + a WS suite hang cascade, unrelated to this diff. Mergeable once this run's shards are green (modulo the known flake).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI is fully green on ca4037e — all 11 test shards passed (plus static-gates, all guards, mysql-contract; the 13 skipped lanes are the e2e suites that don't apply to this diff). The win22-s5 #5955 flake did not recur on this run.

Final state of the branch (3 commits on top of master):

Verdict: mergeable. The root-cause policy decision (second-writer policy / takeover timing) remains open in #5281, as the PR body states — this PR deliberately fixes only the fallout.

…e a poisoned integrity verdict

The FIX 3 gate only covered the close-time refresh. Two more cache-write
paths stamp an ok verdict while a live foreign writer is still active —
exactly the incident's shape (the wedged predecessor was still writing
when the new process's scan ran; the .chk ok at 19:19:23 came from that
window). If the foreign writer then dies without another write, the file
stats never change and every later boot hits the poisoned cache.

  * sync boot site (src/storage/store.ts): pre-read <db>.lock BEFORE the
    integrity block (the #4175 claim below overwrites the file with our
    own pid and would erase the evidence); an ok verdict is cached only
    when no live foreign pid was present
  * background completion (src/storage/integrity-check.ts): the ok
    verdict is cached only when no live foreign writer is active at
    completion — the incident's actual write path
  * sawForeignWriter lifetime flag: once a live foreign holder is
    observed (boot, background completion), no ok verdict is ever cached
    by this process, and the close-time refresh is skipped with a log
  * cache-hit boots no longer re-stamp at close: lastIntegrityOk is now
    armed only by a scan that ran in THIS lifetime; re-stamping a
    verdict we never re-verified would perpetuate a poisoned .chk
    forever (cache hit -> close re-stamp -> cache hit ...). Without it,
    the file set verified by a fresh scan stays trusted and any change
    outside verified writes invalidates the stats and forces a rescan

Tests: the #7156 close-time test now asserts the full suppression (no
verdict at boot, none at close, fresh rescan on the next boot once the
foreign writer is gone); new tests for the background-completion gate and
the no-re-stamp-on-cache-hit rule; the no-foreign-writer control is
unchanged (fast path preserved).

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结论:这个 PR 的四个修复本身都是对的,可以保留;但只修了"中毒"的其中一条路。我把剩下的路补齐了,commit 3591d300(已推到本分支),16/16 测试绿。等这个新 commit 的 CI 绿了再合并。

背景:#7156 里"缓存中毒"是怎么发生的

事故时间线里,新进程在 19:19:02 启动后台 quick_check,19:19:23 写出了 index.db.chk 的 ok 判定——而当时卡死的前一个 sidecar 还活着、还在写库。这次扫描和写入是"抢跑"的:它说 ok,但文件可能已经(或正在)被第二个写者弄坏。如果那个写者随后死掉、不再写,文件的 size/mtime 就再也不会变,于是之后每次开机都命中这个假 ok 缓存,跳过重新扫描,在第一条写语句上直接炸。

这个 PR 修了什么 / 漏了什么

FIX 3(close 时的门控)只挡住了"关闭时刷新缓存"这一条路。但写 ok 缓存一共有三条路,另外两条当时都开着:

  1. 后台 quick_check 完成时(就是事故里 19:19:23 那次写入的实际路径)——完成时只要扫出 ok 就无脑写缓存;
  2. 同步启动扫描通过后(小库走的这条路)——扫完 ok 就写缓存,而此刻第二个写者还可能在写;
  3. 缓存命中的启动在 close 时"再盖章"——启动时没做任何扫描(只是信任了旧缓存),close 时却把这个从未重新验证过的 ok 盖到新的统计信息上。这一条最隐蔽:中了毒的 .chk 会永远自我续命(命中缓存 → close 再盖章 → 命中缓存 → ……),下一轮扫描永远不会发生。

3591d300 的改动

  • 启动扫描前先读一次 .lock(#4175 的锁声明在完整性块之后、会用自己的 pid 覆盖文件,不先读就没证据了);有活的外部写者时,同步扫描的 ok 不写缓存,记一条 info 日志,下次启动重新扫;
  • 后台扫描完成时加同样的门控:有活的外部写者就不写缓存(补上事故的实际路径);
  • 新增生命周期标志:只要这个进程活着的任何时刻见过活的外部写者,它就永远不再写 ok 缓存(close 刷新也跳过,有日志);
  • 缓存命中的启动不再在 close 时再盖章:只有"本进程生命周期内真正扫过且 ok"才会武装 close 刷新。没中毒时没有任何行为变化(文件集被新鲜扫描验证过就继续信任;一旦有验证之外的改动,统计信息失配,自然触发重扫)。

误伤面:多进程共享 sessions 目录是受支持拓扑,外部只读 opener 也会让这次启动放弃缓存收益——代价只是下次启动多扫一次,永远不会导致错误判定,方向是安全的。

验证

  • 新增 3 个测试 + 重写 1 个:外部写者在场时三条路全部不写缓存、写者死后下次启动重扫并恢复快路径、缓存命中启动不再再盖章、无外部写者的对照组快路径不变;
  • storage-integrity-cache + issue7156-storage-corruption 合计 16/16 通过;storage-unified/storage-open-busy-lock/storage-ingest-watermark/extension-storage-contract/storage-backend-contract/storage-durability 全绿;typecheck 干净;
  • 本机有一个与 PR 无关的既有失败(fix-401-storage.test.ts,纯 JSON 旧 SessionStore,未改动的分支上同样失败,CI 上绿),不阻塞本 PR。

待办(根因,另议):双写者本身的修复方向(跨 lane 活锁拒绝 / sessionsDir 按 lane 隔离)还没定,见 issue 里的讨论。本 PR + 这个 commit 解决的是"坏了之后别被假缓存拖着走",不解决"别让它坏"。

…ne writer; never TRUNCATE the WAL under a live peer

Owner-approved 2026-11-20 ("1+2 改吧"), following the incident analysis in
issue #7156: healthy dual-writer topologies are fine (270/270 repro rounds
clean) — the corruption needs a predecessor holding a STALE view of the WAL
index (wedged or draining). These two fixes close both shapes at their
root, not their symptoms:

Fix 1 (launcher): a last gate before the spawn ladder.
  quiesceSameLanePredecessor() runs after attach / cross-lane fallback /
  #3544 rotation / #5955 wedge takeover / #707 marker-wait have all been
  exhausted, and refuses the spawn when the same-lane predecessor on the
  target port is still alive:
    - listener up + health answers  → healthy peer, allow (unchanged, #5082)
    - listener up + health silent   → wedged: actionable refusal (kill hint
      + the #5955 patience-window auto-takeover keeps owning the holder)
    - listener down + process alive → draining: bounded wait (10s,
      BILI_PREDECESSOR_DRAIN_GRACE_MS) for the PROCESS to exit — the final
      flush still writes index.db, so "port free" is not the safety
      condition (#5167); then spawn.
  Policy changes on purpose:
    * pre-escalation wedge cycles and failed kills now REFUSE the spawn
      instead of drifting a child to the next free port (that drift was the
      second writer); the takeover ladder retries on the next bring-up.
    * the attach-mode exemptions in rotateStalePredecessor and
      resolveWedgedSameLaneHolder are gone: an explicit attach pin (e.g. the
      machine-managed BILI_MCP_PROXY cc-lane pin) is a live predecessor, so
      pinned lanes get the same supervised takeover instead of spawning
      behind it — the exact path the incident took.

Fix 2 (storage): the open-time AND close-time
  PRAGMA wal_checkpoint(TRUNCATE) now skip while a live foreign writer is
  proven present (activeForeignWriter()). A TRUNCATE reclaims the WAL file
  and its -shm index out from under every peer; a peer holding a stale view
  then writes/checkpoints from the invalidated view and its frames land in
  reclaimed offsets — the incident damage shape. With no live foreign
  writer the behavior is byte-identical to before.
  NOTE: the close-time half goes one step beyond the literal ask ("skip on
  open") because the dying process's FINAL checkpoint is the proven
  incident vector; it is one block, trivially revertible if the owner
  disagrees.

Tests:
  - storage-integrity-cache: new case — a planted live foreign writer
    suppresses the TRUNCATE on open AND close (WAL stays non-empty, skip
    lines name the pid); the no-lock control still converges the WAL to 0.
  - issue5955 suite inverted to the new contract: pre-escalation cycles
    refuse (never spawn); the escalating cycle kills, waits for death,
    then spawns; kill-fail keeps the ledger and refuses every cycle.
  - issue3544 / issue4850 / launcher / issue5167 harnesses gain the
    isPidAlive (and isConnectable) seams: their in-process mock residents
    "die" with the listener, since the gate's safety condition is process
    death, which a mock can never really do.
  - full suite: 12486+ pass, 0 new failures vs the pristine 3591d30
    baseline (12 pre-existing env failures, individually verified on a
    stashed pristine tree; the advisory cluster that flaked once passes
    standalone 27/27 and on rerun).
@ranxianglei

Copy link
Copy Markdown
Owner Author

根因修复 1+2 已上(提交 b4af60ac)

owner 2026-11-20 批准的"同车道先杀后起 + 见到活的外部写者跳过 TRUNCATE",按事故分析(#7156)在根上堵了两种坏形状。健康的双写拓扑不受影响(270/270 复现轮零损坏已验证)。

修复 1(launcher):spawn 前最后一道闸
attach / 跨车道回退 / 旧版轮转 / 卡死接管 / 别家启动等待全部走完、即将 spawn 之前,检查目标端口上是否还活着同车道的前任:

  • 健康在服务的 → 放行(原有双写拓扑,不变);
  • 服务在但不应答健康检查(卡死)→ 拒绝 spawn,报错直接给 kill <pid> 和"耐心窗口到期自动接管"的说明;
  • 端口已释放但进程还在(排空中)→ 最多等 10s 让进程真正退出(排空阶段的最后 flush 还在写库,端口空了不等于安全),退出后放行。

两个有意的行为变化(都在 commit message 里写明):

  1. 卡死接管还没到耐心窗口的周期、以及杀不掉的场景,现在拒绝启动而不是把新进程漂到下一个空端口——那个漂移本身就是第二个写者;
  2. rotateStalePredecessor / 卡死接管里"显式 attach 就跳过"的豁免删掉了:显式 pin(BILI_MCP_PROXY 这类机器自管的车道 pin)就是一个活的前任,pin 住的车道从此也走受监督的接管,而不是背后直接开写——这正是事故走的那条路。

修复 2(storage):见到活的外部写者,跳过 wal_checkpoint(TRUNCATE)
TRUNCATE 不只是搬数据帧——它把 WAL 文件和 -shm 索引从所有同伴脚下收回。拿着过期索引视图的同伴(卡死中或排空中)再写/再 checkpoint,帧就落进已回收的偏移量上,即事故里的损坏形状。启动时的 checkpoint 现在跳过;另加了 close 时同样跳过——事故里旧 sidecar 的收尾 checkpoint就是实锤向量,这一小块单独成段,不同意可以只 revert 这处。无活外部写者时行为与以前逐字节一致。

测试

…config docs (#7156 gate knob)

The #7156 pre-spawn second-writer gate introduced BILI_PREDECESSOR_DRAIN_GRACE_MS
(src/launcher.ts) without registering it: lifecycle-ledger-guard and docs-coverage
went red. Added the ledger entry (docs/lifecycle.md §2.6), the env-only seed entry
(website/config-reference/env-only.yaml), regenerated the config-doc tables, and
added the long-form detail row in en/zh/ja alongside the sibling spawn-budget knobs.
@ranxianglei

Copy link
Copy Markdown
Owner Author

owner 2026-11-20 拍板:close 时跳过 TRUNCATE 保留(按建议)——它不是可选项,而是事故实锤向量(旧 sidecar 的收尾 checkpoint)。代码现状即拍板结果,无需改动。

@ranxianglei
ranxianglei merged commit 1e357bf into master Nov 20, 2026
87 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1239-master.1e357bf (rolling master tag).

npm install -g billion-context@0.0.1239-master.1e357bf  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant