Repository navigation
[PR] fix: #7156 storage-corruption fallout — webSummaryRows json_valid guard, .chk close-time gate, first-write corruption classification - #7173
Conversation
…ty-payload failures #7156 reports two independent problems from a wedged-sidecar takeover incident (bili 0.1.216, old 0.1.215, single DSH host, storage:full, 573MiB index.db): (1) the takeover spawned a new sidecar BEFORE the wedged predecessor was reclaimed, so two processes held index.db at once and corrupted it (the user's repro: a 3-row UPDATE from a second connection corrupts the live WAL + auto_vacuum=INCREMENTAL db; the same UPDATE on a snapshot copy is fine); (2) .recover left 3 blocks with payload_json='' (length 0), and webSummaryRows' json_extract threw malformed-JSON, which unifiedWebSummaries silently swallowed -> the panel showed "5 live / 0 historical" (real: 222 sessions / 1700 blocks) with no log. This PR fixes the non-decision fallout (the root cause — the second-writer policy / takeover timing — is an open owner decision, #5281): 1. webSummaryRows: wrap payload_json in a json_valid CTE so invalid/empty payloads degrade to NULL (skipped by filters, ignored by SUM) instead of throwing malformed-JSON and failing the whole .all() (store.ts). 2. unifiedWebSummaries: log a warn when the catch swallows the error, so the "0 historical" fallback is visible in server logs (persist.ts). 3. close(): gate the close-time integrity-cache refresh on no live foreign writer (activeForeignWriter() === null). A process that booted healthy but whose file was corrupted by a concurrent second writer (the #7156 incident) would otherwise stamp a FRESH ok verdict onto the corrupted stats at close, so every later boot skips the rescan (cache hit) and dies on the first write. A live foreign writer means the file may have moved outside verified writes -> skip the refresh, next boot re-scans fresh (store.ts). 4. maybeOpen: classify a first-post-open-write corruption (sqliteCorruptionDetail match) as db-corrupt -> start WITHOUT the index (operator hand-fix) instead of REFUSING TO START (throw). The sync quick_check did not run (large store / cached verdict), so the corruption surfaced at the first write; the old comment "Corruption-shaped failures never reach here" was wrong. The handle is abandoned (not close()'d) because close() would run the cache refresh and re-stamp the poisoned verdict (shadow.ts). Behavior changes (disclosed per owner policy 2026-09-22): - FIX 3: a write-heavy store with a live foreign writer at close now re-scans on the next boot (cache miss) instead of a cache hit — aligned with the 2026-10-30 owner policy "each next boot re-scans fresh"; cost is one extra rescan, boot is never blocked (#4609 preserved). - FIX 4: a first-post-open-write corruption now starts WITHOUT the index (envelope-served) instead of REFUSING TO START (throw) — aligned with the 2026-10-30 owner policy (proven-corrupt -> start without the index). Tests: tests/issue7156-storage-corruption.test.ts (4 tests: FIX 1 + control, FIX 2, FIX 4) + tests/storage-integrity-cache.test.ts (2 tests: FIX 3 + control). All targeted suites green; typecheck clean.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.11028-pr.7173.9c34b60This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr7173.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…re, #1248) The #7156 fix added a spawn to this suite (one bare `node -e` dummy child as a live PID holder for the <db>.lock fixture in the close-time cache-refresh gate test). scripts/run-ci-tests.mjs's SPAWN_RE tripwire (#1248) requires every process-spawning test file to be registered in scripts/live-files.mjs after reviewing its concurrency assumptions. Concurrency review: single bare node child (setTimeout 30s backstop), stdio ignore, no ports, no servers, mkdtemp-isolated per test, SIGKILL'd in t.after — same pattern as the #5046/#4612 entries, so the zone band is irrelevant.
(ai: vllm-qwen/qwen3.8-27b) Reviewed the PR and fixed its CI red. Summary: Code review — all 4 fixes verified correct
CI red — root cause + fix The red on head 46c1521 was PR-specific, not inherited: all 11 test shards died in ~17s at the pretest tripwire — Fixed in-branch: appended CI status (live, head 7ec502f): re-triggered by the push; static-gates (ubuntu/24) + mysql-contract already green, test shards in progress. Mergeable once the shards go green. 八问
判定:CI 修复已推送,等分片转绿即可合并;根因(双写者策略/takeover 时序)仍按 PR 正文所述在 #5281 待 owner 拍板,本 PR 不动策略。 |
…liberately abandoned handle) CI (windows/22 shard) failed the FIX 4 test in its teardown: EBUSY: resource busy or locked, unlink '...\bili-issue7156-...\index.db' (tests/issue7156-storage-corruption.test.ts:209, the finally rmSync) Root cause: the FIX 4 path under test abandons the store handle on purpose (owner policy 2026-10-30 — a runtime corruption finding must not run close()'s cache refresh, which would re-stamp the poisoned verdict). Linux unlinks an open file fine; Windows cannot, so the teardown rmSync throws EBUSY and the test fails although every assertion passed. Fix: the repo's established pattern for exactly this (leaked ephemeral handle on Windows) — rmBestEffort with bounded retries (maxRetries 10, retryDelay 100ms), swallowed on persistent EBUSY/EPERM; same helper as issue3397/issue4986/issue6780 suites. Only the FIX 4 test needed it: the other three tests close their stores before the teardown rmSync.
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI follow-up — both remaining reds accounted for; the PR-specific one is fixed. After the LIVE_FILES fix (7ec502f), CI re-ran: 9 of 11 test shards green. Both reds are on windows/22:
Fix (commit ca4037e, pushed): the repo's established pattern for exactly this (leaked ephemeral handle on Windows) — 八问(更新):
Verdict: both deterministic PR-specific reds (LIVE_FILES tripwire + Windows teardown EBUSY) are now fixed; the remaining win22-s5 red is the known inherited #5955 flake + a WS suite hang cascade, unrelated to this diff. Mergeable once this run's shards are green (modulo the known flake). |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI is fully green on ca4037e — all 11 test shards passed (plus static-gates, all guards, mysql-contract; the 13 skipped lanes are the e2e suites that don't apply to this diff). The win22-s5 #5955 flake did not recur on this run. Final state of the branch (3 commits on top of master):
Verdict: mergeable. The root-cause policy decision (second-writer policy / takeover timing) remains open in #5281, as the PR body states — this PR deliberately fixes only the fallout. |
…e a poisoned integrity verdict
The FIX 3 gate only covered the close-time refresh. Two more cache-write
paths stamp an ok verdict while a live foreign writer is still active —
exactly the incident's shape (the wedged predecessor was still writing
when the new process's scan ran; the .chk ok at 19:19:23 came from that
window). If the foreign writer then dies without another write, the file
stats never change and every later boot hits the poisoned cache.
* sync boot site (src/storage/store.ts): pre-read <db>.lock BEFORE the
integrity block (the #4175 claim below overwrites the file with our
own pid and would erase the evidence); an ok verdict is cached only
when no live foreign pid was present
* background completion (src/storage/integrity-check.ts): the ok
verdict is cached only when no live foreign writer is active at
completion — the incident's actual write path
* sawForeignWriter lifetime flag: once a live foreign holder is
observed (boot, background completion), no ok verdict is ever cached
by this process, and the close-time refresh is skipped with a log
* cache-hit boots no longer re-stamp at close: lastIntegrityOk is now
armed only by a scan that ran in THIS lifetime; re-stamping a
verdict we never re-verified would perpetuate a poisoned .chk
forever (cache hit -> close re-stamp -> cache hit ...). Without it,
the file set verified by a fresh scan stays trusted and any change
outside verified writes invalidates the stats and forces a rescan
Tests: the #7156 close-time test now asserts the full suppression (no
verdict at boot, none at close, fresh rescan on the next boot once the
foreign writer is gone); new tests for the background-completion gate and
the no-re-stamp-on-cache-hit rule; the no-foreign-writer control is
unchanged (fast path preserved).
ranxianglei
left a comment
There was a problem hiding this comment.
结论:这个 PR 的四个修复本身都是对的,可以保留;但只修了"中毒"的其中一条路。我把剩下的路补齐了,commit 3591d300(已推到本分支),16/16 测试绿。等这个新 commit 的 CI 绿了再合并。
背景:#7156 里"缓存中毒"是怎么发生的
事故时间线里,新进程在 19:19:02 启动后台 quick_check,19:19:23 写出了 index.db.chk 的 ok 判定——而当时卡死的前一个 sidecar 还活着、还在写库。这次扫描和写入是"抢跑"的:它说 ok,但文件可能已经(或正在)被第二个写者弄坏。如果那个写者随后死掉、不再写,文件的 size/mtime 就再也不会变,于是之后每次开机都命中这个假 ok 缓存,跳过重新扫描,在第一条写语句上直接炸。
这个 PR 修了什么 / 漏了什么
FIX 3(close 时的门控)只挡住了"关闭时刷新缓存"这一条路。但写 ok 缓存一共有三条路,另外两条当时都开着:
- 后台 quick_check 完成时(就是事故里 19:19:23 那次写入的实际路径)——完成时只要扫出 ok 就无脑写缓存;
- 同步启动扫描通过后(小库走的这条路)——扫完 ok 就写缓存,而此刻第二个写者还可能在写;
- 缓存命中的启动在 close 时"再盖章"——启动时没做任何扫描(只是信任了旧缓存),close 时却把这个从未重新验证过的 ok 盖到新的统计信息上。这一条最隐蔽:中了毒的
.chk会永远自我续命(命中缓存 → close 再盖章 → 命中缓存 → ……),下一轮扫描永远不会发生。
3591d300 的改动
- 启动扫描前先读一次
.lock(#4175 的锁声明在完整性块之后、会用自己的 pid 覆盖文件,不先读就没证据了);有活的外部写者时,同步扫描的 ok 不写缓存,记一条 info 日志,下次启动重新扫; - 后台扫描完成时加同样的门控:有活的外部写者就不写缓存(补上事故的实际路径);
- 新增生命周期标志:只要这个进程活着的任何时刻见过活的外部写者,它就永远不再写 ok 缓存(close 刷新也跳过,有日志);
- 缓存命中的启动不再在 close 时再盖章:只有"本进程生命周期内真正扫过且 ok"才会武装 close 刷新。没中毒时没有任何行为变化(文件集被新鲜扫描验证过就继续信任;一旦有验证之外的改动,统计信息失配,自然触发重扫)。
误伤面:多进程共享 sessions 目录是受支持拓扑,外部只读 opener 也会让这次启动放弃缓存收益——代价只是下次启动多扫一次,永远不会导致错误判定,方向是安全的。
验证
- 新增 3 个测试 + 重写 1 个:外部写者在场时三条路全部不写缓存、写者死后下次启动重扫并恢复快路径、缓存命中启动不再再盖章、无外部写者的对照组快路径不变;
storage-integrity-cache+issue7156-storage-corruption合计 16/16 通过;storage-unified/storage-open-busy-lock/storage-ingest-watermark/extension-storage-contract/storage-backend-contract/storage-durability全绿;typecheck 干净;- 本机有一个与 PR 无关的既有失败(
fix-401-storage.test.ts,纯 JSON 旧 SessionStore,未改动的分支上同样失败,CI 上绿),不阻塞本 PR。
待办(根因,另议):双写者本身的修复方向(跨 lane 活锁拒绝 / sessionsDir 按 lane 隔离)还没定,见 issue 里的讨论。本 PR + 这个 commit 解决的是"坏了之后别被假缓存拖着走",不解决"别让它坏"。
…ne writer; never TRUNCATE the WAL under a live peer
Owner-approved 2026-11-20 ("1+2 改吧"), following the incident analysis in
issue #7156: healthy dual-writer topologies are fine (270/270 repro rounds
clean) — the corruption needs a predecessor holding a STALE view of the WAL
index (wedged or draining). These two fixes close both shapes at their
root, not their symptoms:
Fix 1 (launcher): a last gate before the spawn ladder.
quiesceSameLanePredecessor() runs after attach / cross-lane fallback /
#3544 rotation / #5955 wedge takeover / #707 marker-wait have all been
exhausted, and refuses the spawn when the same-lane predecessor on the
target port is still alive:
- listener up + health answers → healthy peer, allow (unchanged, #5082)
- listener up + health silent → wedged: actionable refusal (kill hint
+ the #5955 patience-window auto-takeover keeps owning the holder)
- listener down + process alive → draining: bounded wait (10s,
BILI_PREDECESSOR_DRAIN_GRACE_MS) for the PROCESS to exit — the final
flush still writes index.db, so "port free" is not the safety
condition (#5167); then spawn.
Policy changes on purpose:
* pre-escalation wedge cycles and failed kills now REFUSE the spawn
instead of drifting a child to the next free port (that drift was the
second writer); the takeover ladder retries on the next bring-up.
* the attach-mode exemptions in rotateStalePredecessor and
resolveWedgedSameLaneHolder are gone: an explicit attach pin (e.g. the
machine-managed BILI_MCP_PROXY cc-lane pin) is a live predecessor, so
pinned lanes get the same supervised takeover instead of spawning
behind it — the exact path the incident took.
Fix 2 (storage): the open-time AND close-time
PRAGMA wal_checkpoint(TRUNCATE) now skip while a live foreign writer is
proven present (activeForeignWriter()). A TRUNCATE reclaims the WAL file
and its -shm index out from under every peer; a peer holding a stale view
then writes/checkpoints from the invalidated view and its frames land in
reclaimed offsets — the incident damage shape. With no live foreign
writer the behavior is byte-identical to before.
NOTE: the close-time half goes one step beyond the literal ask ("skip on
open") because the dying process's FINAL checkpoint is the proven
incident vector; it is one block, trivially revertible if the owner
disagrees.
Tests:
- storage-integrity-cache: new case — a planted live foreign writer
suppresses the TRUNCATE on open AND close (WAL stays non-empty, skip
lines name the pid); the no-lock control still converges the WAL to 0.
- issue5955 suite inverted to the new contract: pre-escalation cycles
refuse (never spawn); the escalating cycle kills, waits for death,
then spawns; kill-fail keeps the ledger and refuses every cycle.
- issue3544 / issue4850 / launcher / issue5167 harnesses gain the
isPidAlive (and isConnectable) seams: their in-process mock residents
"die" with the listener, since the gate's safety condition is process
death, which a mock can never really do.
- full suite: 12486+ pass, 0 new failures vs the pristine 3591d30
baseline (12 pre-existing env failures, individually verified on a
stashed pristine tree; the advisory cluster that flaked once passes
standalone 27/27 and on rerun).
根因修复 1+2 已上(提交
|
…config docs (#7156 gate knob) The #7156 pre-spawn second-writer gate introduced BILI_PREDECESSOR_DRAIN_GRACE_MS (src/launcher.ts) without registering it: lifecycle-ledger-guard and docs-coverage went red. Added the ledger entry (docs/lifecycle.md §2.6), the env-only seed entry (website/config-reference/env-only.yaml), regenerated the config-doc tables, and added the long-form detail row in en/zh/ja alongside the sibling spawn-budget knobs.
|
owner 2026-11-20 拍板:close 时跳过 TRUNCATE 保留(按建议)——它不是可选项,而是事故实锤向量(旧 sidecar 的收尾 checkpoint)。代码现状即拍板结果,无需改动。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1239-master.1e357bf # this exact build
npm install -g billion-context@master # always the newest merged build
|
Fixes #7156
Summary
#7156 reports two independent problems from a wedged-sidecar takeover incident (bili 0.1.216, old 0.1.215, single DSH host, storage:full, 573MiB index.db):
.recoverleft 3 blocks withpayload_json=''(length 0), andwebSummaryRows'json_extractthrew malformed-JSON, whichunifiedWebSummariessilently swallowed → the panel showed "5 live · 0 historical" (real: 222 sessions / 1700 blocks) with no log.This PR fixes the non-decision fallout. The root cause of Problem 1 (the second-writer policy) is a resolved owner decision — this PR does NOT change the policy.
Fixes
payload_jsonin ajson_validCTE so invalid/empty payloads degrade to NULL (skipped by filters, ignored by SUM) instead of throwing malformed-JSON and failing the whole.all().activeForeignWriter() === null). A process that booted healthy but whose file was corrupted by a concurrent second writer (the [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156 incident) would otherwise stamp a FRESH ok verdict onto the corrupted stats at close, so every later boot skips the rescan (cache hit) and dies on the first write.sqliteCorruptionDetailmatch) as db-corrupt → start WITHOUT the index (operator hand-fix) instead of REFUSING TO START (throw). The handle is abandoned (not close()'d) because close() would run the cache refresh and re-stamp the poisoned verdict.Behavior changes (disclosed per owner policy 2026-09-22)
Root cause (resolved decision — new evidence warrants reconsideration)
The dual-writer window (Problem 1) is a designed consequence of #5955 (spawn-while-wedged-holder-alive) + the 2026-10-31 owner decision (#5068, closed; implemented as #5294, merged: warn-only, never block, "sharing a local-disk index.db is a supported topology"). The user's #7156 repro empirically REFUTES the "supported topology" premise (a 3-row UPDATE from a second connection corrupts the live WAL + auto_vacuum=INCREMENTAL db; the same UPDATE on a snapshot copy is fine). This is new evidence that may warrant reconsidering the decision. This PR fixes the fallout (symptoms) without changing the policy. The redundant option-B PR #5250 is still open.
Tests
tests/issue7156-storage-corruption.test.ts(4 tests: FIX 1 + control, FIX 2, FIX 4).tests/storage-integrity-cache.test.ts(2 tests: FIX 3 + control).