Skip to content

[PR] refactor(storage): extract StorageBackend contract behind UnifiedStore (#3388 Phase A) - #3432

Closed
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-16_storage-backend-interface
Closed

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-16_storage-backend-interface

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Phase A of #3388 — storage backends beyond SQLite (MySQL / Greenplum / extensible), aimed at separating the proxy layer from the DB service so bili can run distributed.

This PR extracts UnifiedStore's synchronous instance-method surface into a new StorageBackend interface (src/storage/backend.ts), makes UnifiedStore implement it, and widens export.ts renderSessionJsonlFromTables(store: ...) from UnifiedStore to StorageBackend so the contract has a genuine second consumer across a module boundary (not a decorative type alias).

Design decisions

  • Operation-level seam, not a SQL-dialect shim. Operations such as ingestSession commit CAS refcount + authoritative-set replace + watermark atomically in one transaction. That atomicity cannot survive dialect translation, so each backend implements the same operations natively — SQLite keeps its proven SQL, MySQL/Greenplum get fresh implementations against the same contract.
  • Synchronous interface now, async later. The interface mirrors today's store exactly (synchronous). It widens to async in Phase C alongside the first networked backend (pg / mysql2 are Promise-based). Keeping Phase A purely synchronous means zero signature/caller changes — a true pure refactor.
  • Engine details excluded. Static UnifiedStore.open(path) and the SQLite-specific engine / engineSource getters are deliberately left out of the interface — they are implementation details; a networked backend selects its target via the Phase-B config knob, not a file path.

Behavior change

None. Pure refactor. No runtime path changed, no config added, no wire / persistence-format change. UnifiedStore is the same class now carrying an implements clause; renderSessionJsonlFromTables accepts a strictly wider type while every existing call still passes a UnifiedStore.

Verification

  • Rebased onto current master (384bdf6f4, merge of 定时全量失败自动开 issue,转绿自动关 #3428). The three touched files were byte-identical across the 34-commit gap, so no conflicts.
  • tsc --noEmit -p tsconfig.json clean. The implements clause forces every method signature to match the interface.
  • Targeted suites green: full storage + export family (tests/*storage*.test.ts, tests/export-*.test.ts, tests/unified-acp-invariants.test.ts, tests/cli-export.test.ts) → 148 tests, 147 pass, 0 fail, 1 skip. Full-suite verdict deferred to CI on the pushed head.

Scope / what's next (all tracked in #3388)

  • A (this PR): StorageBackend contract + UnifiedStore implements it.
  • B: add storage.backend config knob (default sqlite) — owner-gated config surface, reported in-thread.
  • C: first networked backend (MySQL vs Greenplum — pending owner pick + a target instance) + async widening + factory wiring.
  • D: distributed hardening (connection pool, cross-machine writer trust, MVCC refcount/watermark, crash recovery).

Refs #3388

Phase A of #3388 (storage backends beyond SQLite: MySQL/Greenplum/extensible). Pulls UnifiedStore's synchronous instance-method surface into a StorageBackend interface (src/storage/backend.ts), makes UnifiedStore implement it, and widens export.ts renderSessionJsonlFromTables to accept the interface so the contract has a genuine second consumer across a module boundary.

Pure refactor: zero behavior change, no signature or caller changes. The interface is intentionally synchronous (mirrors today's store) and widens to async in Phase C alongside the first networked backend. This is the seam that lets MySQL/Greenplum backends be added without touching consumers (persist/SessionStore/export/web-list/GC/runtime-status).

Refs #3388
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-16_storage-backend-interface (729ebfa)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4049-pr.3432.15f019d

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3432.tgz
npm install -g package

Install instructions are refreshed on each push.

reference/architecture.md had no entry for the unified storage layer (#2671): the whole src/storage/ directory was unlisted and the Persistence orientation bullet described only persist.ts + session.ts. Add the src/storage/ section (all 9 files incl. the StorageBackend contract from #3388 Phase A) and correct the Persistence bullet to point at the unified store + contract. Doc-only; no code or behavior change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到,开始 review PR #3432(#3388 Phase A)。

查重: 不重复 —— 开着的 issue/PR 里没有其它条目做「StorageBackend 契约提取」;同区域的在途工作(#3419 并发覆盖、#3423 shadow digest 漂移、#3348 sqlite bootstrap 超时)机制与症状都不同,不构成重复。父 issue 只有 #3388。

接下来按规范走:方向审(对照 #3388)、回归清点、代码/测试/diff 审查,结论用八问体例回帖。

…not-throw contract

Review follow-up on #3432 Phase A: the new contract documents close() as
best-effort ("implementations must not throw into the caller") and every
call site already defends (UnifiedShadow.close swallows; persist flushAll/
closeQuietly treat it as never-throwing; export degrades to the legacy
renderer), but the SQLite impl only guarded the WAL checkpoint — the final
db handle release could still throw. Same isolation pattern as the existing
checkpoint guard: warn-log instead of propagate.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 结论(#3388 Phase A)

独立审完(方向、回归、代码、测试、diff 五关),并已在 PR 分支补了一个小修 commit 729ebface。总判定:可合(等 CI 全量绿)。

① 方向审(对照 #3388):#3388 要的是 sqlite 之外的存储后端 + 扩展性。Phase A 选「操作级接缝而非 SQL 方言 shim」是对的——ingestSession 的 CAS refcount + 权威集替换 + watermark 单事务原子性跨方言翻译必然失真,每个后端原生实现同一组操作是唯一稳的路。「同步先行、Phase C 随第一个网络后端一起加 async」也合理:Phase A 零调用方改动是纯重构,async 扩散不会留下死中间态。engine 细节(open(path) / engine / engineSource)留在契约外正确——网络后端走配置不走路径。方向无大问题。

② 回归清点(逐项既有行为):

  • src/export.ts renderSessionJsonlFromTables 参数放宽 UnifiedStore→StorageBackend:纯类型变更,运行期字节不变;外部调用方仅 tests/export-jsonl.test.ts:79(传 UnifiedStore),保持;
  • src/storage/store.ts 只加了 implements 子句 + type-only import:编译期擦除,运行时零变化,保持;
  • 新增 src/storage/backend.ts:纯类型模块,bundle 里擦除,无运行时影响;
  • reference/architecture.md:文档。文件清单与 src/storage/ 实际 9 个文件逐一核对一致,描述与各模块头部注释一致。
    高危模式逐项过:无语义漂移、无 fail-fast 变吞错(见下披露)、无阈值/时序/输出格式变化。回归:无。

③ 代码审查发现 + 已修:契约文档说 close() "implementations must not throw into the caller",但 UnifiedStore.close() 只 guard 了 WAL checkpoint,末尾的 this.db.close() 裸奔可抛。所有现有调用方本已按 never-throw 防御(UnifiedShadow.close swallows、persist closeQuietly 注释 "Never throws"、export finally 外层 catch 降级 legacy renderer),无测试钉住抛错行为——已按同一隔离模式补 try/catch(warn 日志),独立 commit 729ebface。
行为变更披露(Deliverables #4):UnifiedStore.close() 旧行为 = 底层句柄释放失败时异常可传播给调用方;新行为 = 记一行 warn 不再抛出。原因 = 对齐本 PR 新声明的契约,且全部现有调用路径本来就已吞掉或降级,实际可观察行为不变。

④ 测试有效性:PR 未新增测试——对纯类型提取可接受:implements 子句由 tsc 强制签名逐一对齐,行为由既有 storage/export 语料钉住。我本地复跑定向家族(storage + export + unified-acp-invariants + cli-export):148 tests, 147 pass, 0 fail, 1 skip,与 PR 声明完全一致;tsc --noEmit clean;对当前 master(ce9bc90a3)merge-tree 干净无冲突。契约完备性我也核过:接口 37 个方法与 UnifiedStore 公有实例方法一一对应,排除项均有文档说明;export.ts 里 UnifiedStore import 仍被 L358 使用,非死代码。全量判定以推送 head 的 CI test lanes 为准。

⑤/⑤b diff 与代码质量:4 文件 115 行,每行都服务 PR 目的,无空行/重排/lockfile/version 污染;命名达意、无死代码、注释讲为什么、风格与仓库一致、零新依赖。通过。

结构化八问

  1. 是什么问题:不是缺陷修复,是 Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储,比如mysql,greenplum等,并保持扩展性 #3388(存储层支持 MySQL/Greenplum 等)的第一步——把 UnifiedStore 的同步实例方法面抽成 StorageBackend 契约,并让 export.ts 成为跨模块真实消费方。
  2. 什么原因导致的:架构缺口——proxy 层直接耦合 SQLite 具体类,没有操作级接缝,后续加后端只能重写 load-bearing store。
  3. 影响范围与严重程度:仅内部实现;纯编译期重构,零运行时路径改动,用户不可见。
  4. 怎么修复的:抽操作级契约(原子性论证支撑「不做方言 shim」)、同步先行 async 后补、engine 细节留契约外、放宽一个导出函数签名;外加我补的 close() 契约一致性小修。
  5. 是否有回归问题:无。清单见上②;唯一行为微变(close() 由可抛变为 warn 日志)已披露且与既有调用方防御一致。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/重试/游标/去重键。
  7. 是否改了设计方向:有,且是本 PR 的目的——新增 StorageBackend 抽象层作为多后端扩展点;数据流不变,SQLite 仍是唯一实现。两个非阻塞观察留给 Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储,比如mysql,greenplum等,并保持扩展性 #3388 后续阶段:(a) 行类型目前从 ./store.js type-import,Phase C 落地第二个后端时可考虑挪到独立 types 模块;(b) 第二个实现落地时应配一套共享 conformance 测试。
  8. 是否引入了新的配置:无。(storage.backend 旋钮属 Phase B,owner-gated,另行报告。)

判定:可合 —— 方向成立、回归为零、验证齐备(tsc + 定向 148 项 + 干净 merge);CI 全量绿后即可由你合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Superseded by the re-scoped two-step plan the owner set on #3388: (1) redo the StorageBackend seam against current master — this PR's interface list predates schema v6/v7 (reapBlocks / archive / restore / gc / media hydration / incremental pointers are missing from the contract), (2) pluggable backends with sqlite itself as the default registered backend, MySQL as the first real one. Fresh PR will carry the full operation surface; closing this one rather than force-rebasing a contract that is incomplete by construction.

@ranxianglei ranxianglei changed the title refactor(storage): extract StorageBackend contract behind UnifiedStore (#3388 Phase A) [PR] refactor(storage): extract StorageBackend contract behind UnifiedStore (#3388 Phase A) Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant