Skip to content

[PR] feat(storage): index.db backup and corruption recovery commands bili storage backup / repair (#7156) | feat(storage): index.db 备份与损坏恢复命令 bili storage backup / repair(#7156) - #7431

Merged
ranxianglei merged 6 commits into
masterfrom
fix/storage-backup-repair-tools
Nov 21, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
fix/storage-backup-repair-tools

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

背景

2026-11-20 #7156 事故(3.2GB index.db 被多个长驻 bili 进程共写损坏)里,恢复手段是一段手写的 .recover 等价脚本 + 手工文件手术。这个 PR 把这两步做成自带验证的一等命令。

bili storage backup [--keep N]

bili storage repair

--from <backup.db>(有备份)

  1. 先验证备份(quick_check + sessions 表可读),坏备份/不存在 → 拒绝,什么都不动
  2. 坏库原地改名 index.db.corrupt-<ts> 隔离(-wal/-shm/.chk 一并移走,证据保留)
  3. 备份拷入(原件不动)
  4. 换装后再验一次 → 提示重启 bili

不带 --from(无备份 salvage)

安全护栏(全部有测试钉死)

  1. 另一活进程持有 index.db.lock(pid 存活判定,stale 自动放行)时拒绝换装(Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 pid 纪律)——隔离文件不能从活写者脚下抽走
  2. 当前 index.db 通过 quick_check 时 repair 拒绝执行——健康索引不能被一份可能更旧的备份静默顶掉;真要回滚是手工改名的事

回归

  • tests/storage-backup-repair.test.ts 7 用例:verdict 分类、活写者 pid 纪律、备份快照+retention、--from 换装(坏备份/不存在/健康库全拒绝)、无备份 salvage、活写者拒绝
  • 本地:tsc 干净 + cli/parseargs/storage 相邻套件 40/40 绿
  • 真实 CLI 冒烟:seed → backup → repair --from → 全绿

关联

…ed index.db snapshots and corruption recovery (#7156)

The 2026-11-20 production incident (#7156) left the operator with a hand-rolled
.recover-equivalent and manual file surgery as the only recovery path for a
proven-corrupt 3.2 GB index.db. Those moves are now first-class, self-verifying
commands:

  bili storage backup [--keep N]
    Online-safe consistent snapshot via VACUUM INTO (never rewrites the source,
    no exclusive lock — #4612 does not apply); the snapshot must pass its own
    quick_check before it counts; <sessions>/backups/ keeps the newest N (3).

  bili storage repair [--from <backup.db>]
    With --from: verify the backup (quick_check + readable sessions table),
    quarantine the corrupt index.db IN PLACE (index.db.corrupt-<ts>, -wal/-shm/
    .chk moved alongside — evidence kept), COPY the backup in (the backup stays
    intact), re-verify, tell the operator to restart.
    Without --from: salvage — read-only row-by-row extraction into a fresh
    schema (FK dependency order), bisecting broken page ranges to single rows
    and re-reading row columns individually (a corrupt overflow cell nulls its
    own column, the rest of the row survives — the #7156 field case lost only
    its big columns this way). Self-verified before the same swap.

Safety rails (all tested):
  * swap refuses while another live process holds index.db.lock (#4175 pid
    discipline — a quarantined file must not be stolen out from under a writer);
  * repair refuses to replace a HEALTHY index (passing quick_check) with a
    possibly-older backup — deliberate rollback stays a manual rename.

New module src/storage-backup-repair.ts + cli.ts wiring (bare 'bili storage'
stays a usage error — gc archives real sessions, so no action is the silent
default). README 'Storage maintenance CLI' section + CHANGELOG.
Regression: tests/storage-backup-repair.test.ts (7 cases: verdicts, live-writer
pid discipline, backup+retention, --from swap, healthy-index refusal,
no-backup salvage, --json).
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: fix/storage-backup-repair-tools (92eb029)

Latest build — install this exact version

npm install -g billion-context@0.0.11156-pr.7431.400d744

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
92eb029 billion-context@0.0.11156-pr.7431.400d744 build
447e8de billion-context@0.0.11154-pr.7431.1903697 build
3329c26 billion-context@0.0.11152-pr.7431.335182b build
842ddc2 billion-context@0.0.11139-pr.7431.2132e73 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7431.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title feat(storage): index.db 备份与损坏恢复命令 bili storage backup / repair(#7156) [PR] feat(storage): index.db backup and corruption recovery commands bili storage backup / repair (#7156) | feat(storage): index.db 备份与损坏恢复命令 bili storage backup / repair(#7156) Nov 21, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 是 #7156 事故家族的「运维防线」,与 #7424(运行时防线:拒启 + 信封导入保护)、#7425(大字段拆表,降低损坏代价)互补;无其他 open [PR] 针对本线程。

标签: scope/storage + p2(存储域运维工具,源自 p1 事故 #7156;命令本身 opt-in,不引入运行时行为)。

CI 红,两个根因(均为机械性登记遗漏;master 头 e691e6e 全绿 → PR 特有问题):

  1. 全部 12 个 test shard:tests/storage-backup-repair.test.ts 未登记进 scripts/live-files.mjs 的 LIVE_FILES——文件里有 spawnSync(stale-lock 用例的死 pid fixture),触发 tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248 process-spawning tripwire,套件在任何测试运行前就中止;
  2. architecture-docs:新增 src/storage-backup-repair.ts 未列入 reference/architecture.md 文件地图。

代码审核发现(独立于 CI):

  • --from 换装、活写者 pid 纪律、健康库拒绝执行等护栏合理,测试钉得实;
  • 一个实质缺陷:无备份 salvage 路径里,4 张 WITHOUT ROWID 表(conversations/session_ext/kv_store/message_media)上 SELECT max(rowid) 抛 no such column: rowid → 整表被静默跳过,且跳过不计入丢失 → exit 0 + ok:true,与 PR 正文「有丢失时 exit 1 并逐表报告」的承诺矛盾。即 schema v11 上每次无备份 salvage 都会丢掉这 4 张表。

计划(全部 additive commit 加在 PR 分支,不动你的提交):

  1. LIVE_FILES 登记测试文件(带 house-style 注释说明 spawnSync 来源);
  2. reference/architecture.md 补文件地图行;
  3. salvage 修复:WITHOUT ROWID 表走整表 INSERT...SELECT 拷贝(页完好即救回;页损坏则诚实报 skipped),任何被跳过的表计入丢失 → exit 1 + 显式报告;补测试用例(含确定性损坏 kv_store 根页的用例)。

本地定向验证后 push,CI 重绿后按八问体例回复审核结论。

ci-check added 4 commits November 21, 2026 12:32
The suite's freshDeadPid() fixture uses spawnSync("true") to mint a
guaranteed-dead pid, which trips the process-spawning tripwire in
run-ci-tests.mjs and fails every shard before any test runs. The only
child is that dead-pid holder: no ports, no servers, mkdtemp-isolated
per test — the entry is for the tripwire, like #5046/#4612.
…nt as loss (#7156)

No-backup salvage walked rowids (SELECT max(rowid) + per-row
re-read), but schema v11 has four WITHOUT ROWID tables
(conversations, session_ext, kv_store, message_media) where
max(rowid) throws 'no such column: rowid' — a structural property,
not corruption. The old code took that as 'unreadable' and skipped
the whole table silently, and 'lost' only counted per-row losses, so
every salvage exited 0 with ok:true while those tables came out
empty.

Now:
- isWithoutRowid() detects the table kind from the DDL in
  sqlite_master (not error-text matching);
- WITHOUT ROWID tables are copied whole: the read runs on the
  read-only src handle (a corrupt page throws mid-scan) and the
  write is one transaction on the fresh db — all rows land or none
  do. (A cross-connection INSERT...SELECT is not an option: without
  ATTACH the SELECT side resolves to the fresh db's own empty table
  and 'succeeds' with 0 rows.)
- any table the salvager cannot read (note set) now counts as loss:
  ok:false, exit 1, and the report names it in skippedTables (JSON)
  and a 'skipped tables:' line (text) — matching the stated contract
  'exit 1 and report per table when anything is lost'.

Behavior change: salvage that previously exited 0 with silently
emptied WITHOUT ROWID tables now exits 1 and names them; intact
WITHOUT ROWID tables are now preserved instead of dropped.
- buildSeedDb seeds real rows in conversations (1) and kv_store (2)
  so the salvage path has WITHOUT ROWID data to carry;
- the existing salvage test now asserts the kv_store report is
  self-consistent with the new db (salvaged count == actual count,
  note present when rows are lost) and that skippedTables exists;
- new deterministic test: corrupt EXACTLY the kv_store root page
  (rootpage from sqlite_master, whole page XOR 0xff — unlike the
  1/3 window, the target is exact) and assert exit 1, ok:false,
  kv_store salvaged 0 with a 'WITHOUT ROWID' note and named in
  skippedTables, while the intact conversations table is copied
  whole (1 row) and rowid tables are unaffected.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen/qwen3.8-27b)

CI fix run (attempt 1/6): architecture-docs fixed and pushed.

七问

  1. 是什么问题:PR 新增了 src/storage-backup-repair.ts,但 reference/architecture.md 的文件地图漏登记该文件,architecture-docs 守卫([PR] docs: live architecture hub — overall diagram + module pages with diagrams + file-map CI guard #5924 要求 src/ 全覆盖)判红。
  2. 什么原因导致的:功能提交只写了代码与测试,没有同步更新架构文件地图(AGENTS.md 规定「新增/删除源文件必须再生成地图」)。
  3. 影响范围与严重程度:仅 CI 门禁红,阻塞本 PR 合并;不影响任何运行时行为,无用户面影响。
  4. 怎么修复的:在 reference/architecture.md 按字母序补一行登记(含一句话职责描述与来源 issue 号),纯文档一行改动。
  5. 是否有回归问题:无。只动文件地图,不触碰任何代码/配置/协议;本地守卫全绿(326 个 src 文件全覆盖、无幽灵条目)。
  6. 是否破坏缓存:不涉及。
  7. 是否引入了新的配置:不涉及。

收工判定:architecture-docs 已修复并随新 head 447e8de3cd9 进入全量 CI,等待 CI 结果即可。

…7156)

The ③ salvage description predates the fix that makes WITHOUT ROWID
tables copy whole and counts skipped tables as loss (exit 1); the
entry now states both. The new tests live in the same file, under the
existing no-backup salvage item of the regression list.
@ranxianglei
ranxianglei force-pushed the fix/storage-backup-repair-tools branch from c85ddab to 92eb029 Compare November 21, 2026 04:40
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen/qwen3.8-27b)

Re-wake verification: the reported blocker is already fixed; full CI is running on the current head — no further work needed this run.

  • At wake, the mirror cache was stale (still showing the run on 447e8de3cd9); a one-shot GitHub API fetch confirmed the live state: the run on 447e8de3cd9 was cancelled because the lane agent pushed one more commit on top.
  • Current head: 92eb02923af (docs: CHANGELOG note for the WITHOUT ROWID salvage contract). It is unmarked → a full CI pass is in progress: 22 checks queued, 0 failed.
  • The originally reported failure (architecture-docs on 842ddc294d) was fixed in the previous run by commit 3329c2655ff (file-map registration of src/storage-backup-repair.ts), which is in the current head's history.
  • Re-ran node tools/check-architecture-docs.mjs on the current head 92eb02923af: all green (326 src files covered, no ghosts).
  • No other red checks; nothing to fix.

七问

  1. 是什么问题:本次唤醒报告的是旧 head 842ddc294d 上的 architecture-docs 红,上一轮已修复。
  2. 什么原因导致的:原因为 PR 新增 src/storage-backup-repair.ts 未登记进 reference/architecture.md 文件地图;已由 3329c2655ff 补上。
  3. 影响范围与严重程度:仅 CI 门禁,无运行时影响;当前已消除。
  4. 怎么修复的:本轮无需新修复——核实了修复已在当前 head 历史中,并在当前 head 本地复跑守卫确认全绿。
  5. 是否有回归问题:无;本轮零代码改动,仅验证。
  6. 是否破坏缓存:不涉及。
  7. 是否引入了新的配置:不涉及。

收工判定:当前 head 92eb02923af 的全量 CI 运行中(22 项 queued,0 红),architecture-docs 已确认修复,等待 CI 结果即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论:PR #7431(bili storage backup / repair)—— 发现 3 处问题,已全部修掉,CI 全绿

做了什么:在 PR 分支上追加 4 个 commit(未动原始 commit),推至 92eb02923af。

发现的 3 处问题:

  1. CI 登记遗漏 ×2(机械性):tests/storage-backup-repair.test.ts 未登记进 scripts/live-files.mjs → 测试里 freshDeadPid() 的 spawnSync("true") 死 pid 夹具触发 process-spawning tripwire,12 个 test shard 在跑任何测试之前就全红;src/storage-backup-repair.ts 未列入 reference/architecture.md 文件地图 → architecture-docs 车道红。
  2. 真实代码缺陷:无备份 salvage 会静默丢掉 schema v11 的全部 4 张 WITHOUT ROWID 表(conversations / session_ext / kv_store / message_media)且 exit 0。根因:salvage 只走 rowid(SELECT max(rowid) + 逐行重读),WITHOUT ROWID 表没有 rowid 列,结构性错误 no such column: rowid 被误诊为「不可读」整表跳过;且 lost 只统计行级丢失,表级跳过不计入 → exit 0 + ok:true。PR 自己声明的契约是「有丢失时 exit 1 并逐表报告」——被跳过的表就是丢失,却没被算。已实测复现(含一个关键发现:对坏表做跨连接 INSERT...SELECT 会静默拷 0 行而不抛错——SELECT 侧解析到新库自己的空表;count(*)/quick_check 才会抛)。

修复(4 个追加 commit):

  • 50060c9 ci: 测试登记进 LIVE_FILES(唯一子进程是死 pid 夹具,无端口,条目只为 tripwire)
  • 8f56752 fix(storage): isWithoutRowid() 从 sqlite_master 的 DDL 判定表类型(不做错误文本匹配);WITHOUT ROWID 表整表原子拷贝——读在只读 src 句柄上(坏页扫描时抛错),写是新库上单事务(全有或全无);任何读不了的表都算丢失 → ok:false + exit 1 + 报告逐表点名(JSON skippedTables / 文本 skipped tables:)
  • 447e8de test: seed 给两张 WITHOUT ROWID 表灌真实数据 + 新确定性测试(精确定位 kv_store root page 整页 XOR 损坏,断言 exit 1 / note / skippedTables,完好表 conversations 整表拷回,rowid 表不受影响)
  • 92eb029 docs: CHANGELOG 条目补上 salvage 契约的这两点
  • (architecture 地图行已被 ci-check 机器以 3329c26 先修,我的重复提交已弃)

本地验证:目标套件 8/8 绿、相邻 7 套件 66/67(1 skip = 可选 better-sqlite3 未装,既有)、tsc 干净、build 成功、architecture-docs guard 绿、LIVE_FILES tripwire 本地模拟通过。

八问:

  1. 是什么问题:本 PR 是功能(index.db 备份 + 损坏恢复的一等命令,[Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156 事故直接产物),不是缺陷。审核发现的真实缺陷:无备份 salvage 静默清空 4 张 WITHOUT ROWID 表并汇报成功。
  2. 什么原因导致的:salvage 路径只沿 rowid 走;WITHOUT ROWID 表无 rowid,结构性报错被误诊为损坏而跳表;丢失统计只算行级不算表级 → 退出码误报成功。
  3. 影响范围与严重程度:仅无备份 salvage 路径(没有备份时的最后防线)受影响;4 张表是次级状态(身份索引 / 宿主 ext / 车道 kv / 媒体映射),核心数据(sessions/content/messages/blocks)不受影响。严重度 p2——核心数据能救回,但 exit 0 会误导运维以为恢复完整,4 张表静默丢失。
  4. 怎么修复的:DDL 判定表类型 → WITHOUT ROWID 表整表拷贝(只读句柄读、坏页抛错、新库单事务写,全有或全无);读不了的表一律计为丢失(exit 1 + 点名)。
  5. 是否有回归问题:逐项审过既有行为——rowid 表 salvage 路径(二分 + 逐列重读)、--from 换装、活写者 pid 护栏、健康索引拒绝、backup 的 VACUUM INTO + retention,全部未动。行为变更(披露):无备份 salvage 的退出码语义从「只计行级丢失」变为「行级丢失 + 表级跳过」——旧行为 exit 0 + 静默清空表,新行为 exit 1 + 点名;完好的 WITHOUT ROWID 表现在被保留(旧行为丢弃)。这是向 PR 自声明契约的修复,不是语义漂移。
  6. 是否破坏缓存:不涉及——离线运维 CLI,不触碰请求管线 / 前缀缓存 / 重试策略。
  7. 是否改了设计方向:否。模块边界、quarantine+swap+verify 编排、只读纪律全部保持。
  8. 是否引入新配置:无。无新 config key / env;backup/repair 是既有 bili storage 的新子命令(--keep / --from 属 PR 自身功能面)。

CI(92eb029 实测):48 个 check run —— 36 success + 12 skipped(e2e 车道,本 diff 不触发),0 failed。前一轮红的两个根因(LIVE_FILES + architecture-docs)均已修复转绿。

判定:可合并——功能 + 护栏 + 回归测试闭环,2 处 CI 根因与 1 处代码缺陷全部修复;mergeable_state=behind 只是 master 前进(mergeable=true,无冲突)。合并留给 owner。

@ranxianglei
ranxianglei merged commit f46b8bd into master Nov 21, 2026
48 checks passed
ranxianglei pushed a commit that referenced this pull request Nov 21, 2026
…lict at 0.0.202

Master advanced 122 commits past the PR base (batch-seam double-claim fix
#7428 -> kernel 0.0.201, storage backup/repair #7156/#7431, ingest-band
default-off #7426, pf-tail-ids cap #7381, OpenAI reasoning alias #7432,
dsh-mneme scan #7430, and more). Content hunks auto-merged (kernel/src/
compress.ts, all four storage-render goldens); the only conflicts were the
three kernel version lines, resolved at 0.0.202 = master's 0.0.201 + this
PR's kernel/src change, per the CI-enforced kernel version discipline.

Targeted local verification: tests/golden-storage-render.test.ts +
tests/storage-state-fields.test.ts 7/7, kernel issue6116/issue3769/
hide-consumed suites 27/27.
ranxianglei pushed a commit that referenced this pull request Nov 21, 2026
…te refusal test

tests/issue7156-pure-sqlite-refusal.test.ts (added to master via #7431) carries the same hardcoded zeroPage(p, 2) recipe that 484a3d3 already fixed in tests/issue7156-storage-corruption.test.ts: page 3 of the file only lands on the sessions data page under the 12-table schema; this PR's schema (15 tables after the three-table retirement) shifts it onto an empty page, so the first post-open write no longer throws CORRUPT and the test fails on every CI shard.

Look the sessions root page up from sqlite_master and zero THAT page — stable across schema sizes (same pattern as 484a3d3).

Verified locally (node 22, linux): without the fix the test fails with 'Missing expected exception: a first-write corruption in the pure-sqlite stance refuses to start' (the exact ubuntu CI failure); with the fix 5/5 pass.

PR #6785 CI fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant