Skip to content

[PR] storage: StorageBackend contract (step 1) — sqlite through the registry, mysql a registered stub (#3931) - #3932

Merged
ranxianglei merged 4 commits into
masterfrom
2026-10-20_storage-backend-seam
Oct 20, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-10-20_storage-backend-seam

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Implements #3931 — step 1 of the #3388 pluggable-storage plan, scoped by the owner: land the abstraction seam, keep every test green, re-wire sqlite through it, leave MySQL for step 2.

What

  • StorageBackend contract (src/storage/backend.ts) — operation-level and async: ~40 operations (ingest, message/block/ref reads, session_ext/kv, ingest watermark + incremental pointers, the Reaped blocks resurrect after restart in full mode: stale index row outlives the #2695 reap and poisons the JSON envelope #3841 reap journal, archive/restore/gc, checkpoints/replicas, summaries, refcount diagnostics, close). The seam is operations, not SQL dialects — a dialect shim cannot preserve ingestSession's single-transaction CAS refcount discipline, so each backend implements the same operations natively. The interface is async because step-2 network backends are; the sqlite implementation stays synchronous under the hood with a Promise.resolve surface (hot-path cost: one microtask).
  • sqlite re-wired through the registry — SqliteBackend wraps the existing UnifiedStore; openBackend("sqlite", …) resolves via the registry. Zero schema change (stays v7), zero migration, zero behavior change.
  • mysql is a registered stub — the name validates (configs won't bounce between "unknown" and "unimplemented" when step 2 lands), but opening it throws one documented line; a non-sqlite selection at lane boot logs a single notice and stays on sqlite (boots never crash); mysql2 is never loaded.
  • Seam proof — the --jsonl export renderer now consumes the contract end-to-end (its behavior is pinned by the existing export tests, unmodified); tests/storage-backend-contract.test.ts runs the same assertions against ANY backend instance via the registry — step 2 runs it unchanged against mysql.
  • Config — storage.backend file key + BILI_STORAGE_BACKEND env (default sqlite), documented via the yaml seeds → regenerated CONFIGURATION en/zh.

Explicitly not here (step 2, #3388)

  • No real mysql implementation, no mysql2 dependency (optional-dependency + self-download discipline already established for better-sqlite3 will be reused).
  • No async conversion of the persist/shadow hot path (interface being async is enough; callers convert per-surface starting step 2).
  • No dynamic third-party backend loading (built-in registry only; the shape does not preclude opening registration later).
  • No multi-writer/concurrency rework (CAS refcount + watermark under network MVCC ships WITH the mysql backend).

Verification

Stakes first (storage-backend-contract + storage-backend-knob, red on the missing module before implementation). Green locally: contract 6/6, knob 3/3, export-jsonl 4/4 (one mechanical adaptation: the renderer is async now), storage-unified 14/14, archive-gc 4/4, reap-journal 2/2, golden-storage-render 5/5, storage-ext 5/5 — plus the full suite run attached to CI. tsc clean (the renderNudgeText 4-arg errors on current master are stale-kernel-dist artifacts, not from this branch).

Fixes #3931.

@ranxianglei ranxianglei changed the title storage: StorageBackend contract (step 1) — sqlite through the registry, mysql a registered stub (#3931) [PR] storage: StorageBackend contract (step 1) — sqlite through the registry, mysql a registered stub (#3931) Oct 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 这是 #3931 的唯一实现 PR;open+closed 里检索 StorageBackend / storage seam / #3388 step-1,无其他在途实现。其余 open 的 storage 相关 PR(#3144 尾部缺陷、#3930 引擎 bootstrap 回归测试、#2681 重构前置等)都是不同的缺陷/功能域,与本 seam 无重叠。

标签: pr(既有) + scope/storage + p3 — 存储层抽象缝(step 1),零行为变更的架构铺路,非缺陷、不涉具体客户端,严重度取最低档。

状态: CI gate/version-guard 已绿,test 各 lane 还在排队;我已完成分支拉取,正在按规范做独立 review(原 issue 方向 → 回归清点 → 代码/测试/diff)。结论稍后按八问体例给出。

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_storage-backend-seam (5d665fa)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4722-pr.3932.b3cbc12

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3932.tgz
npm install -g package

Install instructions are refreshed on each push.

ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…#3932 review)

resolveBackendName collapses unregistered names to sqlite before the boot
notice consults it, so a typo'd selection (env or file) fell back silently
while CONFIGURATION.md promised a notice; the unknown-name branch of
backendSelectionNotice was dead production code. The notice now judges the
raw winning tier via selectedBackendRaw (resolveBackendName builds on it,
behavior-identical), and treats "" as unset so the mapper is total over its
string domain. Behavior change: unknown names log ONE warn line at boot
(previously silent); effective backend unchanged (sqlite).
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…3932 review)

The --jsonl path hardcoded new SqliteBackend(UnifiedStore.open(index)), so a
deliberate mysql selection silently exported from sqlite instead of failing
with the documented stub error that CONFIGURATION.md and #3931 promise. The
selection is now enforced before the guarded open: registered-but-unshipped
names go through openBackend and throw their documented line before any file
access; typo'd names warn once and read sqlite. The guarded try/catch is
untouched, so the corrupt-index legacy-tail degrade contract (#2690) on the
sqlite path is byte-for-byte preserved.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review complete — found 2 real defects in the selection surface, fixed directly on the PR branch and pushed (two labeled commits, additive, no rebase-replace).

Independent review (re-derived from the diff, not read off the issue)

  1. Direction (①): Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388) #3931 acceptance items all met — knob matrix, contract suite run through the registry, export consuming the interface, docs via yaml seeds + CHANGELOG entry, zero schema change (v7). The "operations, not SQL dialects" seam decision is sound: a dialect shim cannot preserve ingestSession's single-transaction CAS refcount discipline, so native-per-backend is the right call. Step-1 scoping matches Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储,比如mysql,greenplum等,并保持扩展性 #3388 exactly.
  2. Regression audit (②), existing behaviors on touched paths, item by item:
    • persist/shadow hot path: maybeOpen's new param is optional; sole caller src/persist.ts:466 passes fileStorageBackend() through; unset-selection path is byte-identical to old behavior ✓
    • schema v7 / migrations: untouched ✓
    • export --jsonl: sqlite default path unchanged, incl. the Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 corrupt-index → legacy-tail degrade and the encrypted-lane skip-index gate ✓
    • golden / wire-contract / kernel: untouched (golden-storage-render 5/5 green) ✓
    • nudge cadence, both compression modes: untouched ✓
  3. Code (③): SqliteBackend wrapper, registry, and both new test files are clean; the contract suite imports interface-only (nothing from store internals), so step 2 can run it against mysql unchanged.
  4. Tests (④): assertions have teeth (refcount before/after −1, archive→restore→gc state machine, watermark clear→null, stub error text pinned); the four pre-existing export-jsonl tests are unmodified and keep pinning renderer behavior.
  5. Diff cleanliness (⑤/⑤b): 14 files +632/-12, every line serves the purpose; style/naming match repo conventions; no dead code (author's follow-up f7ce1e910 already dropped the unused RegisteredBackendName export for the unused-exports gate).

Defects found & fixed (pushed as labeled commits)

  • D1 — unknown backend names fell back silently (39b8beb20). resolveBackendName collapses unregistered names to sqlite before the boot notice consults it, so a typo'd storage.backend (env or file) produced zero log output, while CONFIGURATION.md written by this same PR promises "Unknown names also fall back to sqlite with a notice" — and backendSelectionNotice's unknown-name branch was dead production code (only reachable via direct test calls). Fix: new exported selectedBackendRaw(env, file) returns the winning tier's raw value; the boot notice judges the raw selection; resolveBackendName now builds on it (behavior-identical); the notice mapper treats "" as unset so it's total over its string domain. New knob-test rows pin the composition (positive: typo'd env name surfaces the unknown-name notice; negative: unset and normalized-sqlite stay silent).
    Behavior change (old→new): unknown name at lane boot: silent sqlite fallback → one warn line naming the bad value + valid list. Effective backend unchanged (sqlite either way).
  • D2 — export ignored the selection (5e3a52ca3). The --jsonl path hardcoded new SqliteBackend(UnifiedStore.open(index)), so a deliberate mysql selection silently exported from sqlite instead of failing with the documented stub error that this PR's own CONFIGURATION.md text and Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388) #3931 promise. A naive fix inside the existing guarded try would have been swallowed by its catch and misreported as "unified index unreadable". Fix: the selection is enforced before the guarded open — a registered-but-unshipped name goes through openBackend and throws its documented line before any file access; a typo'd name warns once to stderr and reads sqlite. The guarded try/catch is byte-for-byte untouched, so the Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 corrupt-sqlite-index degrade contract survives exactly. New export-jsonl row pins the mysql-failure path (negative direction already covered by the unmodified test 1: default selection renders from unified tables).
    Behavior change (old→new): --jsonl under a mysql selection: silently exports from sqlite → fails at open with the documented stub error. Matches the documented promise; no user has a working mysql yet, so no live workflow changes.

Verification

Local targeted runs (full-suite verdict belongs to CI, per local-testing discipline): 15/15 green — storage-backend-knob 4/4 (incl. new row), storage-backend-contract 6/6, export-jsonl 5/5 (incl. new row); tsc --noEmit clean with freshly built kernel/pi-subagents dists (no stale-dist artifacts). Remote CI is running on the new head 5e3a52ca3: gates 8/8 success, guards (kernel/lane/pi-subagents/config-surface/docs-sync/build-artifact/version-guard/lock-sync) in flight, e2e lanes skipped by design (file-trigger mismatch). Note: the branch is currently BEHIND master (master moved after the PR was cut) — a rebase before merge is advisable; BEHIND alone doesn't block.

结构化八问

  1. 是什么问题:可插拔存储 step-1 接缝 PR 的审查——方向与实现整体正确,但选择面有两个小缺陷:拼错的 backend 名静默回退、export 路径绕过选择直接读 sqlite。
  2. 什么原因导致的:通知逻辑引用的是「解析后」的名字(未知名已被折叠成 sqlite,对通知不可见);export 路径按具体实现类直构,没走 registry。
  3. 影响范围与严重程度:体验瑕疵级,不影响数据与可用性——仅在用户主动误配或提前配置 mysql 时触发;默认用户零感知。
  4. 怎么修复的:以「胜出层的原始值」为选择判据,boot 通知与 export 开库门禁共用同一 helper;注册未落地(stub)的名字在开库前 fail-fast 抛文档化错误,拼错的名字告警一次后照常走 sqlite。
  5. 是否有回归问题:无。既有行为逐项审过(persist/shadow 热路径、schema v7、export 默认路径含 Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 降级、加密车道、golden、nudge/双压缩模式)均保持;两处行为变更是有意且已披露(旧→新见上),sqlite 默认路径逐字节不变。
  6. 是否破坏缓存:不涉及(纯选择面修正,不触碰缓存/重试/游标/去重键)。
  7. 是否改了设计方向:不涉及——仍严格在 Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储,比如mysql,greenplum等,并保持扩展性 #3388 step-1 范围内,接缝形状未变。
  8. 是否引入了新的配置:无新增(PR 自带的 storage.backend + BILI_STORAGE_BACKEND 已在 PR 描述与 CONFIGURATION en/zh 中记录,默认 sqlite,无需迁移)。

判定:两个缺陷已就地修复并推送(39b8beb20、5e3a52ca3),建议 rebase 当前 master 后等 CI 全绿即可合并;合并动作本身留给人工。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 终态跟进(head 5e3a52ca3,接上条 review 回复)

CI 主工作流:全绿。 19/20 通过 + red-signal 按设计 skipped(仅在测试失败时才跑)。首轮唯一红的一个 job —— test (windows-latest, 22, 1) —— 重跑后通过(同 commit,新 job)。全部 e2e 车道(real codex / codex-persona / native opencode·dsh·pi / fake-codex / pi-codex-ws / local-registry / Windows codex overlay)、PR Build Artifact、各静态门禁(plan/kernel-guard/lane-guard/config-surface-guard/docs-sync/lock-sync/version-guard/unused-exports/…)均绿。

那个红是什么、为什么判为 flake:

另一处红(不属本 PR): .github/workflows/ci-gate-plugins.yml(手动真模型车道)在本分支三个 head(bdd97ef 原始 head / f7ce1e9 / 5e3a52c)上以完全相同的方式失败——零 job 创建,在步骤执行前就挂掉。发生在任何 review-fix 之前、且与 PR 内容无交集,属触发/环境侧问题,不计入本 PR 门禁;如需排查请另行安排。

八问增量(相对上条回复):代码零新增变更;回归结论不变(Windows 分片 flake 经重跑证伪,不构成回归;既有行为审计清单见上条);缓存不涉及;设计方向不变;配置面不变(storage.backend / BILI_STORAGE_BACKEND 默认 sqlite,无迁移)。

判定更新:CI 门禁已全部转绿,本 PR 具备合并条件;合并动作本身留给人工。

ranxianglei and others added 4 commits October 20, 2026 19:45
…he registry, mysql a registered stub (#3931)

- src/storage/backend.ts: operation-level async contract (~40 ops: ingest,
  reads, session_ext/kv, watermark/pointers, reap journal, archive/gc,
  checkpoints, summaries, diagnostics) — the seam is operations, not SQL
  dialects, so per-backend transaction semantics survive. SqliteBackend
  wraps the existing UnifiedStore (sync under the hood, Promise surface).
- registry: openBackend()/listBackendNames(); 'mysql' validates but throws
  the documented step-2 error at open (mysql2 never loaded in step 1);
  resolveBackendName (env BILI_STORAGE_BACKEND > storage.backend > sqlite)
  + backendSelectionNotice keep non-sqlite boots on sqlite with ONE warn.
- shadow.maybeOpen takes the file backend tier and logs the notice.
- export.ts jsonl renderer consumes the contract (seam proof); guarded
  open wraps UnifiedStore in the adapter — behavior pinned by existing
  export tests unchanged.
- tests: storage-backend-contract (runs against ANY backend via the
  registry — reused verbatim for mysql in step 2) + knob matrix.
- docs: storage.backend seeds (process.yaml + env-only.yaml), regenerated
  CONFIGURATION en/zh; CHANGELOG.

Zero schema change, zero behavior change for sqlite users (#3931).
…#3932 review)

resolveBackendName collapses unregistered names to sqlite before the boot
notice consults it, so a typo'd selection (env or file) fell back silently
while CONFIGURATION.md promised a notice; the unknown-name branch of
backendSelectionNotice was dead production code. The notice now judges the
raw winning tier via selectedBackendRaw (resolveBackendName builds on it,
behavior-identical), and treats "" as unset so the mapper is total over its
string domain. Behavior change: unknown names log ONE warn line at boot
(previously silent); effective backend unchanged (sqlite).
…3932 review)

The --jsonl path hardcoded new SqliteBackend(UnifiedStore.open(index)), so a
deliberate mysql selection silently exported from sqlite instead of failing
with the documented stub error that CONFIGURATION.md and #3931 promise. The
selection is now enforced before the guarded open: registered-but-unshipped
names go through openBackend and throw their documented line before any file
access; typo'd names warn once and read sqlite. The guarded try/catch is
untouched, so the corrupt-index legacy-tail degrade contract (#2690) on the
sqlite path is byte-for-byte preserved.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_storage-backend-seam branch from 5e3a52c to 5d665fa Compare October 20, 2026 11:47
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

分支已更新到最新 master(接上条):上一条回复后 master 前进了 56 个 commit(包含触碰本 PR 重叠文件的变更:src/config.ts、CONFIGURATION en/zh)。已把分支 rebase 到 77e2eb4b1——重叠文件自动合并、无冲突;新 head 5d665fabe(原 4 个 commit 原样重放,rebase 本身零代码/行为变化)。rebase 后本地验证:typecheck 干净,定向测试 15/15(knob 4 + contract 6 + export-jsonl 5)。CI 正在新 head 上跑,全绿后我再补一行终态。

@ranxianglei
ranxianglei merged commit 45ca4b8 into master Oct 20, 2026
22 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.667-master.45ca4b8 (rolling master tag).

npm install -g billion-context@0.0.667-master.45ca4b8  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388)

1 participant