Skip to content

Storage review fixes: authoritative refs/blocks, corrupt-index guard, BEGIN IMMEDIATE, restore 409 + CCR sidecar (#2690 review) - #2768

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-11_storage-review-fixes
Oct 11, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-11_storage-review-fixes

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Addresses every blocking finding from the owner review of #2690 (issuecomment-6108173553). Scope is strictly the new storage surface; the knob-off default path is untouched (cleared twice by independent review).

Blocking findings → fixes

# Finding Fix
1 refs (session_id, ord) PK conflict when the byRaw map renumbers (native compaction drops a front key) → 3-strike self-disable refs rows authoritatively replaced per ingest (DELETE-then-INSERT); ghost rows cleared, ords compact 0..n-1
2 same-family silent shapes: non-collision prune leaves ghost refs rows; #2695 zombie blocks resurrect (upsert-only) refs + blocks both authoritative; destroyed-substrate block rows deleted with CAS refcounts released
3 corrupt index.db bricks the proxy (unguarded UnifiedStore.open) guarded open → one warn [storage] unified sidecar FAILED to open (…) — staying legacy-only; delete or repair index.db to re-enable, shadow null, legacy keeps working; 3-strike backstop unchanged
4 listSessions leaks one SQLite connection per call (re-breaks the Windows EPERM fix) try/finally store.closeQuietly() in src/export.ts
5 deferred read-then-write transactions → SQLITE_BUSY_SNAPSHOT under the supported multi-process topology (#394) BEGIN IMMEDIATE at top level (better-sqlite3 .transaction(fn).immediate(), node:sqlite BEGIN IMMEDIATE); savepoints unchanged for nesting
6 restoreReplica with a caller-chosen id silently overwrites a live session; restore loses the content-store envelope → CCR can't fetch originals after rollback exists-check → throw already exists → HTTP 409; createReplica freezes the disk content-store envelope into the checkpoint blob; restoreReplica materializes the sidecar for the restored id (survives the next live save, which drops forkContentStore)

Also from the review notes (non-blocking but cheap):

  • conversations table dropped before schema v1 ships (zero writers).
  • --jsonl degrades to the legacy-tail renderer on an unreadable index (guarded open) and is gated on BILI_ENCRYPTION_KEY (index.db holds cleartext).
  • exportSession closes its one-shot handle.
  • PLUGIN.md registration example updated to fork protocol v2.
  • CONFIGURATION.md BILI_STORAGE_UNIFIED row documents the shadow/full disk expectation (≈2× unique-content bytes, no auto-VACUUM).

Tests

New tests/storage-review-fixes.test.ts (hermetic pins, same harness as storage-replicas):

  1. renumbered refs re-ingest without PK conflicts, ghost rows cleared, ords compact
  2. destroyed-substrate blocks deleted and stay dead across a second save
  3. corrupt index.db degrades to legacy-only in both first and later processes (no throw, save+load still work)
  4. restore refuses live-id overwrite (/already exists/) and the restored id round-trips CCR originals through the real load path (loadSync + loadContentStore on a fresh handle)

Full storage family green locally: unified 13, replicas 4, corpus 9, export-jsonl 4, cli-export 2; tsc clean.

prepared by an agent under the local git identity (ranxianglei), human merge required

…nt, corrupt-index guard, BEGIN IMMEDIATE, restore 409 + CCR sidecar

Owner review (blocking findings, issuecomment-6108173553):

1. refs rows: DELETE-then-INSERT per ingest — a renumbered byRaw map
   (native compaction drops a front key) no longer trips the
   (session_id, ord) PK; ghost rows cleared, ords compact.
2. blocks rows: authoritative replacement — destroyed-substrate rows
   are deleted (CAS refcounts released); #2695 zombies cannot
   resurrect on later saves.
3. conversations table dropped before schema v1 ships (zero writers).
4. corrupt index.db: guarded open — warn + legacy-only, never bricks
   the proxy (3-strike self-disable stays as backstop).
5. BEGIN IMMEDIATE at top-level transactions (multi-process
   SQLITE_BUSY_SNAPSHOT, #394 topology).
6. restoreReplica: refuses live-id overwrite (409 via admin mapping);
   freezes the disk content-store envelope into the checkpoint and
   materializes the sidecar for the restored id (CCR originals survive
   the next live save — buildRecord drops forkContentStore).
7. export: listSessions/exportSession close one-shot handles (Windows
   EPERM regression); --jsonl degrades to legacy renderer on
   unreadable/encrypted index.
8. PLUGIN.md registration example updated to fork protocol v2.

Tests: tests/storage-review-fixes.test.ts (4) + full storage family
green locally (unified 13, replicas 4, corpus 9, export-jsonl 4,
cli-export 2).
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-11_storage-review-fixes (069f13c)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3049-pr.2768.e0dd559

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2768.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 ## PR #2768 review — 可合(待新 head CI 全绿)

审原 issue(方向关):owner 对 #2690 的三遍 review 给出 7 项 blocking + 若干 pre-merge 纪律项。本 PR 逐条对应、scope 严格锁在新存储面(11 文件全部在 storage surface + docs + tests,无蔓延),且没有一项是治标:fix 1 比建议方案更强(整表替换而非 ord >= len 尾删)、fix 3 按建议原文实现、fix 5 选对了隔离级别(WAL 下顶层 BEGIN IMMEDIATE 正是正确配对)。方向正确,修的是机制不是症状。master == PR base(eacbb58,无漂移,merge-tree 干净)。

逐项核实(代码级):

  • B1 refs PK 冲突:src/storage/store.ts ingestSession 事务内先 DELETE FROM refs WHERE session_id=? 再插入——幽灵行清除、ords 紧凑 0..n-1。✓
  • B2 僵尸块:同事务 SELECT stale blocks → releaseContent(summary_hash/one_hash) → DELETE。引用计数对称性我单独核过:checkpoint 存的是 whole-state blob(state_hash),不引 per-block hash,释放块级 ref 不会误删 checkpoint 内容;同文本被 CCR 与 block 共享时 refcount=2,只减一。✓
  • B3 corrupt index.db:maybeOpen try/catch → 一条可操作 warn(delete or repair index.db to re-enable)+ shadow=null + legacy 继续工作;每 SessionStore 构造一次(persist.ts:353);3-strike backstop 未动;replicasAvailable()→false。✓
  • B4 连接泄漏:export.ts listSessions try/finally closeQuietly()。✓
  • B5 BUSY_SNAPSHOT:better-sqlite3 .transaction(fn).immediate()、node:sqlite BEGIN IMMEDIATE,嵌套 savepoint 不变;WAL 已启用(schema.ts:40)。✓
  • B6 restore:exists-check → throw already exists → admin 映射 409(与既有 checkpoint-not-found→404 同风格);createReplica 冻结磁盘 CCR sidecar 进 record.forkContentStore,restoreReplica 为恢复 id 物化 sidecar(tmp+rename+codec)——buildRecord 在无 fork receipt 时会丢弃该字段,所以 sidecar 文件才是持久层,下次 live save 不丢。✓
  • B7(storage-mode-label hermetic pin):base 上已修(Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 分支在 review 快照后进化过),无需再动。✓
  • 非阻塞项逐条属实:conversations 表已删(schema + counts() 两处)、--jsonl 加密门槛 + 不可读时降级 legacy-tail renderer、exportSession 关 handle、PLUGIN.md 注册示例 v2、CONFIGURATION.md 磁盘期望(≈2× unique-content bytes、无 auto-VACUUM)。✓

回归审查(既有行为清单):knob-off 默认路径零触碰(maybeOpen 早退 null、不开句柄)✓;两种压缩模式的 wire/rebuild/preflight/nudge 完全未涉及(纯存储面)✓;unified lane 既有行为(digest reconcile、organic import、GC parity、加密 opt-out、3-strike、replica 往返)由 17 个既有存储测试覆盖,本地全绿 ✓。高危模式逐项过:无 fail-fast 变吞错(corrupt-guard 是显式 warn 降级而非静默吞)、无阈值/时序漂移、无输出格式变化。行为变更披露(旧→新+原因):① restore 到已存在 id:静默覆盖 → 409 拒绝(默认 fresh-id 路径不变);② corrupt index.db:进程启动即挂 → warn + legacy-only 降级(可用性修复);③ --jsonl(index 不可读):报错 → 降级 legacy-tail renderer。残留缺口(不拦):restore 的 exists-check 走 StateStore.loadSync 候选(discovered map + _unknown hint + flat name,kernel/src/persist/store.ts:340)— 另一进程在 boot 后持久化的同 id 会话(#394 多进程拓扑)不在本进程 discovered 中,409 不会触发;writeNow 按 record.meta 派生路径,同 meta 会覆盖对方文件、异 meta 产生兄弟文件由下次 loadAll 按 savedAt 调和。触发需 loopback admin + 操作者指定 id + 双进程撞同一 id,极窄,建议作 follow-up 跟踪(与 #394 多进程告警同机制族)。

测试有效性:4 个新测试是真测试——我做了负向对照:把 tests/storage-review-fixes.test.ts 拷到 base eacbb58(含 #2690)跑,4/4 全挂且死因正确(destroyed block row must be deleted、Missing expected rejection 等);head 上全过。断言是行为级(PK 冲突、幽灵行集合、二次 save 复活、corrupt db 双进程 boot、409 正则、CCR 原文经真 loadSync+loadContentStore fresh handle 往返),非常识断言;hermetic(pin BILI_CONFIG_FILE、删 BILI_STORAGE_UNIFIED、mkdtemp、golden corpus 复用)。缺口:跨进程 BUSY_SNAPSHOT 无测试(需真并发,成本高)、better-sqlite3 驱动 lane 靠 CI(本机无 native 模块,skip 正常)。可接受。

小问题直接修(已在 PR 分支补两个独立标注 commit,push 至新 head 8d7bb6631):

  1. docs: CONFIGURATION.md:1740 本 PR 引入的意外双空格("drops it.␣␣Pinned")
  2. docs(PLUGIN): PLUGIN.md:3 头部 "Status: protocol v1" 过期(manifest 已 advertise protocolVersion 2,owner review pre-merge 项的漏网一处;其余 :51/:169/:184/§8 在 base 上已是 v2)

遗留待 owner 拍板(源自 #2690 review pre-merge 清单,本 PR 未触及):resolveStorageMode 直读 env(shadow.ts:37-38、persist.ts:1280)而未经 src/knobs.ts(reference/env-vars.md 硬规则 2)。属 #2690 原有范围,建议小 follow-up PR 或并入后续 knobs 整理。


结构化八问

  1. 是什么问题:Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 统一存储 sidecar 在新存储面上的五类缺陷——非权威 ingest(refs PK 冲突→3-strike 自废 / 幽灵行 / 僵尸块复活)、open 无守卫(corrupt index.db 砖掉整个代理)、连接泄漏(re-break Windows EPERM 修复)、deferred 事务(多进程下 SQLITE_BUSY_SNAPSHOT)、restore 语义(静默覆盖活会话 + 回滚后丢 CCR 原文)。
  2. 什么原因导致的:ingest 被建模成 append-only 而非权威快照替换;sidecar open 缺少 by-design 的降级守卫;事务隔离级别与支持的多进程拓扑(无单实例守卫:多个 bili 代理进程并存共写 sessions 目录,伴随孤儿 ref、有注册表无 transcript、计数自相矛盾的污染簇 #394)不匹配;restore 路径既无存在检查也不冻结磁盘内容包络。
  3. 影响范围与严重程度:仅 storage.unified=shadow/full lane;默认 knob-off 路径零触碰。unified lane 内:压缩重编号可静默杀死 sidecar(3-strike,功能降级)、corrupt index.db = 代理整体不可用(严重)、restore 到活 id = 静默数据覆盖(数据丢失级,但触发面窄:loopback admin + 操作者指定 id)。
  4. 怎么修复的:ingest 改同事务内整表权威替换(DELETE-then-INSERT + CAS 引用计数释放);guarded open 降级 legacy-only 并留一条可操作 warn;顶层 BEGIN IMMEDIATE;restore 加 409 存在检查 + CCR 包络冻结/物化;listSessions 关句柄。
  5. 是否有回归问题:无。knob-off 路径零触碰;wire/压缩(两种模式)不涉及;unified lane 既有行为 17 个测试全绿;无 fail-fast 变吞错、无阈值/时序/格式漂移。行为变更已按第 4 条披露(restore 409、corrupt 降级、--jsonl 降级三项,均为可用性/安全性改善)。
  6. 是否破坏缓存:否。CAS 引用计数释放严格对称(只释放实际删除行所引用的 hash;checkpoint 持有 whole-state blob 不受影响);wire 字节稳定与前缀缓存不受影响(无 wire 变更)。
  7. 是否改了设计方向:否。仍是 Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 的 sidecar 架构 + knob 门控迁移路线,本次是加固 pass。唯一语义澄清:refs/blocks 现在被明确定义为 live state 的权威投影(本来就是这个意图,代码之前没做到)。
  8. 是否引入了新的配置:无新配置键/env/默认值。BILI_STORAGE_UNIFIED 语义不变(legacy|shadow|full,默认关);CONFIGURATION.md 仅新增磁盘期望说明句。

判定:可合 —— 待新 head 8d7bb6631(在原 head 24/28 绿的基础上仅加两个 doc nit)CI 全绿即可;合并仍归 human。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🔗 Heads-up from the #2771 thread: sibling PR #2772 (fix: survive a corrupt unified index by degrading to legacy-only (#2771) + storage-engine proof battery) makes an equivalent change to the same src/storage/shadow.ts lines as this PR's item #3 (the guarded UnifiedStore.open). Both branches sit directly on master eacbb584 and neither is an ancestor of the other, so merging both will conflict in src/storage/shadow.ts. Suggest reconciling to a single carrier before either merges — detail in #2771.

@ranxianglei
ranxianglei merged commit da6315e into master Oct 11, 2026
26 of 28 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.279-master.da6315e (rolling master tag).

npm install -g billion-context@0.0.279-master.da6315e  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant