Skip to content

[PR] fix: corrupted index.db now auto-quarantines and recovers instead of degrading forever (quick_check throw shapes) - #5123

Merged
ranxianglei merged 7 commits into
masterfrom
2026-10-29_integrity-throw-quarantine
Oct 31, 2026
Merged

ranxianglei merged 7 commits into
masterfrom
2026-10-29_integrity-throw-quarantine

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

A corrupted index.db whose corruption makes PRAGMA quick_check throw (truncated file, garbage header, page-content corruption) escaped the #4175 quarantine entirely: the shadow layer only caught StorageIntegrityError, but every throwing shape re-threw the raw driver error → permanent legacy-only degrade with no auto-heal, while the banner kept pointing users at "delete the file". The non-throwing path (quick_check returns rows) was quarantined correctly — this PR closes the gap so every corruption shape reaches the same quarantine decision.

Work consolidated under #4816 (same symptom + mechanism as #5108, which carries the duplicate label). 查重: 与 #4816 同症状同机制(后到并入先到);no in-flight fix PR references #4816 (searched open+closed issues and PRs, 2026-10-29).

Changes (one facet per commit)

  1. Corruption classification (src/storage/sqlite-errors.ts new, store.ts, integrity-check.ts): a small classifier matches exactly the two SQLite corruption shapes — SQLITE_CORRUPT / SQLITE_NOTADB error codes or the stable errmsg strings ("database disk image is malformed", "file is not a database") — across both engines (node:sqlite builtin, better-sqlite3, including [storage:<engine>]-wrapped messages). Transient shapes (SQLITE_BUSY/IOERR/CANTOPEN/SHORT_READ) deliberately do NOT match: quarantining a merely locked-but-healthy index over a transient AV lock (the Windows Defender class, [bug] 0.1.63 起 Windows persist EPERM 风暴:会话持久化静默丢数据(4.9k 次/117 会话)+ compress no-valid-ranges 拒绝 #362/SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, win32-x64) #3823) would destroy a working index.
    • sync quick_check catch: classified throws become StorageIntegrityError (quarantine-able) instead of escaping raw.
    • pragmas/migrate catch: same classification — stores above the 512 MiB async threshold skip the sync scan, so their corruption can surface here first.
    • background worker: corruption-shaped throws post a typed verdict; the parent caches a FAIL integrity verdict (was: warn-only "could not run", served unverified forever).
    • close-time symmetry: a FAIL verdict now rides a clean close onto the post-checkpoint stats (the ok-verdict refresh already existed), so large corrupt stores don't loop rescan→fail→rescan across clean restarts.
  2. Quarantine robustness (shadow.ts): rename failures are no longer swallowed silently. Missing sidecars are pre-checked (normal), each failure captures its errno, and a failed PRIMARY rename degrades loudly naming the errno and does NOT attempt a doomed fresh reopen; leftover -wal/-shm warn only (probe-verified: SQLite resets a stale WAL rather than replaying it onto a fresh file).
  3. Diagnostics (driver.ts getter, runtime-status.ts): /__bili/status storage.engine reports the resolved engine when the open failed AFTER resolution (was null → misdirected triage toward "install sqlite"); the diagnostics blob annotates the engine line accordingly. shortenError middle-truncates >160-char reasons so the errno at the tail survives long Windows paths.
  4. dsh child bootstrap sampling (launcher.ts, the observability half promised in the SQLite unified storage degraded to legacy-only (bili v0.1.208, node v24.21.0, win32-x64) #4816 thread): bootstrap-failure messages sample the child-log TAIL when output exceeds the 256-byte window (the "head" label is unchanged for ≤256B output).
  5. Tests: new tests/issue4816-integrity-throw-quarantine.test.ts (classifier truth table, both directions per Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617; store/shadow/worker/diagnostics facets incl. rename-failure via chmod and cross-restart FAIL-cache persistence); corrupt-index-child.ts scenarios repurposed — engproof now pins quarantine-and-recover to full mode with the leak guards re-targeted at the quarantined file, and the banner scenario switched to the deterministic no-engine failure (garbage-file corruption now heals itself, which is the point); parents updated accordingly.

Behavior changes (old → new)

# Old New Why
a quick_check/pragmas throw in a CORRUPT/NOTADB shape → raw escape → permanent legacy-only degrade; banner says "delete the file" classified → StorageIntegrityError → auto-quarantine (index.db.corrupt-<stamp> kept for forensics) + fresh full-mode index; old sessions import lazily from the legacy envelopes the #4175 design intent applied to the throwing shapes; the manual variant of exactly this operation was verified end-to-end by the reporter on win32 (their follow-up in #5108)
b quarantine rename failures silently caught; fresh reopen attempted regardless primary rename failure → loud degrade naming the errno, no doomed reopen; leftover sidecars warn only the silent catch hid the exact condition where auto-heal cannot proceed (reporter ask #2)
c background check throw → warn "could not run", store served unverified forever; only ok-verdicts refreshed the cache at close corruption throws cache a FAIL verdict; a clean close carries the FAIL verdict onto post-checkpoint stats (fail outranks ok) >512 MiB corrupt stores otherwise loop rescan→fail across restarts without ever quarantining
d storage.engine = null whenever the sidecar didn't open resolved engine reported when resolution succeeded but the open failed engine: none misdirected triage when the real problem was the file (reporter side note)
e diagnostics blob engine line bare gains (resolved, but index.db open failed — see degrade reason) annotation same as d
f degrade reasons >160 chars: head 157 + ... head 78 + … + tail 78 head-truncation cut off the actionable errno on long-path machines
g dsh bootstrap-failure message sampled the child-log HEAD (first 256B) samples the TAIL when output exceeds 256B (label switches to "tail") the failure reason accumulates at the end of the child log; head sampling showed pre-failure noise (#4816 face B)

No config/env surface added. No kernel/ changes.

Verification

  • Deterministic byte-corruption recipes probed locally (node v22 builtin sqlite): truncate-half / garbage-header / zeroed data page all make quick_check throw the CORRUPT/NOTADB shapes; zeroing page 2 keeps open() healthy so the async worker facet is exercised against a real throw.
  • Local targeted suites green: issue4816-integrity-throw-quarantine 7/7 · web-storage-banner 18/18 · storage-engine-proofs 12/12 · storage-integrity-cache + storage-diagnostics-lines 9/9 · issue4811-dsh-busy-drop-selfheal 1/1; typecheck + build clean. Full suite = remote CI.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-29_integrity-throw-quarantine (72d6b0a)

Latest build — install this exact version

npm install -g billion-context@0.0.6947-pr.5123.345ed03

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
72d6b0a billion-context@0.0.6947-pr.5123.345ed03 build
9dc0577 billion-context@0.0.6754-pr.5123.e28f37e build
6faac8f billion-context@0.0.6745-pr.5123.53f102f build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5123.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: corrupted index.db now auto-quarantines and recovers instead of degrading forever (quick_check throw shapes) [PR] fix: corrupted index.db now auto-quarantines and recovers instead of degrading forever (quick_check throw shapes) Oct 29, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。本 PR 是 #4816 的正规修复实现(后到并入先到,#5108 挂 duplicate)。我独立核对:#4816/#5108 描述的症状(throwing-shape corruption 绕过 #4175 隔离 → 永久 legacy-only degrade)与本 PR 修的路径一致;merge-tree 对当前 master 无文本冲突,CI 正跑在同一 sha 6faac8f。无其他在途 fix PR 引用 #4816。

方向关(①):正确,且是「疏」不是「堵」。根因定位准——shadow 隔离守卫只认 StorageIntegrityError,而 throwing 形态以 raw driver error 逃逸该守卫;PR 在错误产生源头(store.open 的 quick_check/pragmas catch + background worker)把 CORRUPT/NOTADB 分类转成 StorageIntegrityError,交给既有 #4175 决策接管,而非在 shadow 层再堵一层。transient(BUSY/IOERR/CANTOPEN)明确排除,与 #5082「锁住≠损坏」哲学一致。

回归审查(②)——逐项过既有行为:

  • non-throwing(返回坏行)损坏 → 原已隔离,不动,保持。
  • healthy 文件 open → 保持(test control 钉住)。
  • transient 失败(BUSY/IOERR/CANTOPEN)→ 仍 raw escape → legacy-only,不隔离(避免破坏健康库);classifier 负向行钉住。
  • 512MiB async 大库:原「worker throw → warn could not run,永不验证」→ 现 cache FAIL verdict + clean close 携带到 post-checkpoint stats。有意变更(c),已披露。

  • close-time 缓存刷新:ok 刷新已存在;新增 FAIL 刷新且 fail outranks ok。有意变更(c)。
  • /__bili/status storage.engine:null → resolution 成功后 open 失败报 resolved engine;仅展示面。有意变更(d/e)。
  • shortenError:head-truncate(157+…)→ middle-truncate(78…78),保留两端信号。有意变更(f),影响所有 >160 字符 degrade reason 文案。
  • dsh child bootstrap 采样 head→tail(>256B)。有意变更(g),观测面。
  • 高危模式排查:fail-fast 未变吞错(rename 失败反而从静默→loud);阈值未漂移(512MiB 不变);输出格式变化仅限 banner/diagnostics 文案(已披露)。未发现无意回归。

代码审查(③):sqlite-errors.ts code 优先(symbolic name)+ errmsg 子串兜底,双引擎覆盖含 [storage:<engine>] wrapper;worker 内联副本有 keep-in-sync 注释且被 worker-facet 测试端到端钉住。shadow.ts primary rename 失败 → 抛新 StorageIntegrityError → 落外层 catch → legacy-only loud degrade(命名 errno)、不做注定失败的 fresh reopen;sidecar(-wal/-shm)残留仅 warn(probe 验证 SQLite reset stale WAL 而非 replay);无死循环。runtime-status.ts 用 !isNoEngineError 守卫,只有「resolution 成功但 open 失败」才报 resolved engine,no-engine 仍 null(负向 test 钉住)。无 as any/@ts-ignore,旧的空 catch(正是被修缺陷)已消除。

测试有效性(④):新 test 7/7 真断言、双向(classifier 正+负;quarantine-recover + rename-fail-degrade);用真实字节损坏配方(truncate/garbage-header/zero-page-2),非 mock。engproof/banner 重定向自洽——garbage-file corruption 现在自愈,故 banner 场景改用确定性 no-engine 钉 wire 形状,engproof 改钉 quarantine-and-recover 且 leak guard 重定向到 quarantined 文件。本地定向 27/27 绿 + typecheck 绿;全量 = CI(在跑 sha 6faac8f,in_progress)。

diff 整洁度(⑤/⑤b):11 文件全部与目的相关,无无关 churn、无空白噪声、命名达意、注释讲为什么。launcher.ts 的 dsh 采样属 #4816 同家族可观测半边,边界上略扩但同一 issue 家族,可接受。


结构化八问

  1. 是什么问题:index.db 损坏若使 quick_check「抛异常」(截断/坏头/页内容损坏)而非返回坏行,会绕过 Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 自动隔离,永久降级 legacy-only 且不自动修复,横幅还误导用户「删文件」。
  2. 什么原因:shadow 隔离守卫只认 StorageIntegrityError,throwing 形态以 raw driver error 逃逸该守卫;Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 只在「返回坏行」那条路接住了。
  3. 影响范围与严重程度:所有走统一存储(full/shadow)且 index.db 发生 throwing 型损坏的环境;有绕行(手动 rename 可恢复,报告者已在 win32 验证),归 p1(存储可用性类)。
  4. 怎么修复:在错误源头把 CORRUPT/NOTADB 形状分类转成 StorageIntegrityError 交既有 Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 决策;同时加固隔离自身 rename 失败(loud + 不再注定失败 reopen)、补大库 FAIL 缓存跨重启、修正 status engine 误报与长文案截断。
  5. 是否回归:上述清单均「保持」或「有意变更且已披露」,fail-fast 未变吞错,未发现无意回归。
  6. 是否破坏缓存:完整性 verdict 缓存增强(FAIL 也缓存、clean close 携带到 post-checkpoint stats),方向是让下次启动正确隔离而非 rescan 死循环;prefix/WAL checkpoint 时序不变。
  7. 是否改设计方向:否,沿用 Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 既有隔离机制,只是把遗漏的 throwing 形态纳入同一决策。
  8. 是否引入新配置:无。

判定:质量达标,可合(待 CI 绿)。两点流程提醒,非阻塞:

… tests)

- runtime-status shortenError: raw .slice(-78) -> safeSuffix (and head via
  safePrefix) — the code-unit-safe clamp the #816/#1615 family gate requires
- issue4816 rename-fail test: pin BILI_STORAGE_UNIFIED=full around the wire
  assertion — the CI baseline preload pins legacy for the non-storage corpus
  (#2815), so requested resolved to legacy and nothing was reportable as
  degraded (passes locally without the preload, fails on every CI shard)
- storage-review-fixes review 3: re-pinned to the NEW intended behavior — the
  garbage-header corruption shape makes quick_check throw, so the store now
  quarantines the corrupt file set and recovers a fresh index instead of
  degrading legacy-only forever (PR behavior change (a)); asserts the
  forensic copy + fresh index.db exist
ranxianglei pushed a commit that referenced this pull request Oct 29, 2026
Phase 1 keyed the client's degrade decision off the additive parentBlocks
count plus a local branchPoint<core.length check. That re-derives server
state on the client and silently no-ops when a mixed-version peer lacks the
new field. Per review, move the judgment to the server: it computes
degraded = (inherited 0 blocks while the parent held some) from its own
ledger and sends an additive boolean; the client honors it directly and no
longer second-guesses with a truncation gate. Old clients ignore the flag.

Adds a test pinning that the client degrades on the flag even for a full
local match (no residual truncation gate), and pins degraded:false for a
healthy never-compressed parent clone.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 红已修,全绿。 修复 commit 9dc0577da 已推到 PR 分支,该 head 的完整 CI 矩阵:35 通过 / 12 跳过(按设计门控的 e2e lane)/ 0 失败。

6 条红的 test lane 归并为 3 个独立问题,全部在本 PR 自己的代码/测试里,master 是绿的(非继承性红):

  1. family gate 被新代码触发:shortenError 新增的裸 .slice(-78)(src/storage/runtime-status.ts:176)撞了 compactEntry summary clamp (slice(0,199)+'…') can strand a lone high surrogate → same deterministic UTF-8 gateway 500 family #816/[bug] summaryFingerprintLine 的 head/tail 裸码元切片产生孤立代理项 → 上游 JSON 400、会话永久卡死(#816/#828 同族第三站点;0.1.169 仍未修) #1615 字符串安全棘轮 → 改用 safePrefix/safeSuffix(src/text-safe.ts)。
  2. 新测试不 hermetic:issue4816 rename-fail 测试的 wire 断言依赖进程环境里的 BILI_STORAGE_UNIFIED,而 CI 基线 preload(Inbound markers: one declarative strip engine for all four wires (#2789, #2798, #2811 plan B) #2815)把非存储类测试文件钉在 legacy——legacy 下没有任何东西可报 degraded,reason 为空。本地(无 preload)过、CI 全红就是这个差异。修法:测试内钉 full、退出时还原。
  3. 旧测试钉着旧行为:storage-review-fixes「review 3」原本钉「损坏 index.db → 永久降级 legacy-only」——正是本 PR 有意改变的行为(行为变更 (a)):throw 型损坏现在隔离+恢复。PR 重钉了 e2e 场景但漏了这条单测。修法:重钉到新行为(隔离成功、取证副本 index.db.corrupt-* 存在、fresh index.db 重建、加载成功)。

结构化八问

  1. 是什么问题:PR head 6faac8f 上 6 条 test lane 红,PR 无法合入。
  2. 什么原因导致的:①新代码引入裸负切片,被仓库既有的字符串安全棘轮拦下;②新测试依赖的环境变量在 CI 基线里取值不同(local 无 preload = full,CI preload = legacy);③既有单测钉住的是本 PR 有意替换掉的旧行为,属于「修 A 必须同步改 B」里漏掉的 B。
  3. 影响范围与严重程度:仅 CI/测试面,零用户面影响;不修则本 PR 不可合并,生产行为完全不受影响。
  4. 怎么修复的:安全截断函数替换(机制对齐既有 compactEntry summary clamp (slice(0,199)+'…') can strand a lone high surrogate → same deterministic UTF-8 gateway 500 family #816/[bug] summaryFingerprintLine 的 head/tail 裸码元切片产生孤立代理项 → 上游 JSON 400、会话永久卡死(#816/#828 同族第三站点;0.1.169 仍未修) #1615 纪律)、测试环境变量钉死+还原(hermetic 化)、旧断言重钉到已披露的新行为并补强取证断言。一个 commit,3 文件,+21/−5,每行都服务于 CI 修复。
  5. 是否有回归问题:无。F1 对 ASCII 输出逐字节相同,仅当截断点落在 emoji/astral 字符上时比裸 slice 短至多 1 个 code unit(方向是「更短而非劈代理项」);F2/F3 纯测试改动。本地按 CI 精确条件(带 preload)先复现三处红、再验证修复后绿:text-safe 18/18、issue4816 7/7、storage-review-fixes 4/4,typecheck 干净;远端 CI 全矩阵绿为最终裁决。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。本修复不触碰 PR 已披露的行为变更 (a)–(g);第 3 项恰恰是把既有测试对齐到已披露的行为变更 (a)。
  8. 是否引入了新的配置:没有。

备注:分支仍落后 master 约 10 个提交(此前 review 确认 merge-tree 无文本冲突),CI 在当前 head 全绿。收工判定:可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Outcome: no action possible — PR #5123 upstream is merged & closed (merge commit 38ea945), so there's nothing to rebase/merge or push into an already-merged head branch. The repeated behind/conflict flags are a stale pre-merge snapshot; the merge-guard should stop firing on this closed PR.

ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
Resolve the fork-identity hash field-set conflict as the union of #5123 serialization-tolerance (drop contentType, JSON-canonicalize tool-call text) and #4266 signed-content carriers (drop thinkingTokens value; add thinkingSignature/googleThoughtSignature/redactedPayloadDigest). Pinned by tests/dsh-fork-adopt.test.ts parity + tests/issue4266-fork-thinking-gate.test.ts drift pins.
ranxianglei pushed a commit that referenced this pull request Nov 12, 2026
master moved past the previous sync (dc041dd) by the web handoff-render
(#6260) + anthropic trailing-system (#5432/#5441) + snapshot-write-lifecycle
(#6186) landings. None of those commits modify src/plugin.ts or
src/agent/fork-adopt.ts (master's version of both == merge-base), so this
merge is clean and preserves the #5123/#4266 union resolution untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant