Skip to content

[PR] feat(storage): run the store on bun's builtin bun:sqlite - #5435

Merged
ranxianglei merged 5 commits into
ranxianglei:masterfrom
chidea:master
Nov 4, 2026
Merged

ranxianglei merged 5 commits into
ranxianglei:masterfrom
chidea:master

Conversation

@chidea

@chidea chidea commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

What this does

bili running on bun can now actually use SQLite: the engine ladder gains a step 0 — bun:sqlite (bun's builtin) — instead of degrading to legacy-only storage. Byte-for-byte unchanged on Node: step 0 is required only when process.versions.bun exists.

No new dependency: bun:sqlite ships inside bun, better-sqlite3 stays an optionalDependency, and the self-bootstrap download stays a Node-only affair.

Why the ladder needs the extra step

Measured on bun 1.3.14:

  • require("node:sqlite") → module does not exist (bun has no node:sqlite) → ladder step 1 empty;
  • require("better-sqlite3") → Error: Could not load the default exports from better-sqlite3 (an N-API prebuild cannot be loaded by bun) → steps 2 and 3 empty as well;
  • so probeBuiltinSqlite()/probeHostBetterSqlite3() both return false → bootstrap downloads a prebuild for nothing, the load then fails anyway → the whole proxy falls back to legacy-only (fix(storage): drop better-sqlite3 optional dep (#2823) #2825 behaves exactly as contracted, but for bun users it is a dead end).

This is not hypothetical: omp (#2690 multi-process writer topology) starts bili with bun by default, and on that machine index.db stayed empty.

Key changes

  • src/storage/driver.ts: ladder step 0 = probeBunSqlite(). bun's builtin is synchronous and already shaped like better-sqlite3 (Database, sync run/get/all, .transaction()), so the adapter only covers the three measured divergences:
    • Named parameters: bun binds by the sigil written in the SQL (:a/@b/$c), while all SQL in this codebase is better-sqlite3 style and passes bare bags ({ a: 1 }). Bare keys are duplicated per statement into exactly the sigil forms that statement really uses (string literals are blanked first, so a :name inside a value is never mistaken for a parameter); all three sigils are written (bun honours only the first match) instead of reaching for allowBareNamedParameters.
    • BLOBs: bun hands back a bare Uint8Array, while the media/payload readers are written against the Buffer contract (buffer.equals, subarray slicing depend on Buffer) → normalized to Buffer on read.
    • run() return: changes/lastInsertRowid are BigInt → narrowed to number; bun has no .open flag → synthesized (two handle.open checks depend on it).
    • Transactions: writes take .immediate(), read-only takes .deferred() (Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 discipline: a deferred transaction that upgrades to the write lock mid-flight hits SQLITE_BUSY_SNAPSHOT, which busy_timeout cannot resolve); close() is idempotent (bun throws on an already-closed database, and callers may close twice).
    • Probe seam: a bag that pins only {builtin, host} is now treated as a Node host (bun only when a test opts in with bun: true), otherwise every existing ladder suite drifts on a machine whose runtime happens to be bun.
  • src/storage/engine-bootstrap.ts: on bun the bootstrap returns not-needed — downloading an N-API prebuild for bun is wasted bandwidth that ends in a load failure; bun never needs a bootstrap.
  • src/storage/runtime-status.ts: nodeSqliteUpgradeHint and the degrade banner name bun:sqlite under bun instead of telling operators to upgrade Node (a lever that does not exist there).
  • src/storage/integrity-check.ts: the whole-history background worker is a self-contained source string that does not reuse the ladder, so it selects its engine by process.versions.bun; bun's read-only option is spelled readonly (Node spells it readOnly).
  • Tests: tests/storage-bun-sqlite.test.ts (new, 9 tests). The fake-Database group runs through the new test seam setBunDatabaseFactoryForTests and therefore runs on Node CI as well (sigil duplication, keys outside literals rewritten exactly once, BigInt/Uint8Array/null normalization, transactions must take .immediate(), step 0 wins the ladder, a pinned {builtin, host} bag means a Node host). The real-builtin group runs only under bun (real BEGIN/rollback, nested transactions must not commit early, the write lock is released immediately after the write, and a real UnifiedStore round-trip: open with the PRAGMA ladder + quick_check + migrations, named-parameter insert, read back, close triggering the WAL checkpoint, reopen and the data is still there). Additionally, 7 raw test handles stop hard-coding openSqliteDatabase(path, "node:sqlite") and follow the automatic probe; the three node:sqlite-only legs in storage-unified now skip by probe instead of blowing up at load.
  • CHANGELOG: new ### Storage — bun 运行时原生支持(bun:sqlite) section under ## [Unreleased].

Measured (same machine, bun 1.3.14, against the pre-change baseline)

suite before after
storage-bun-sqlite (new) — 9 pass / 0 fail
storage-unified 4 pass / 13 fail 14 pass / 4 skip / 0 fail
storage-ingest-watermark 1 / 15 15 / 1
storage-bounded-load 1 / 4 4 / 1
storage-engine-proofs 1 / 11 7 / 5
plugin-fork 35 / 21 51 / 5
block-bounds / fork-clone-media / issue4982 / unified-remap fail to load 1/0, 3/0, 18/0, 3/0
storage-engine-bootstrap 43 / 1 43 / 1 (untouched)

24 more tests pass, zero regressions. tsc --noEmit clean; tsup build plus both dist gates (check-dist-import-annotations, check-dist-external) pass.

Real-store end-to-end: a copy of the production sessions/index.db (13 sessions / 2430 messages), served by this repo's bun-built dist:

[storage] unified FULL mode ENABLED (…) [engine: bun:sqlite/builtin] [wal-cap: 8388608]

PRAGMA quick_check = ok, the ladder ran, schema user_version 8 → 10, all data intact (blocks 41, content 2469, refs 2433, conversations 13), and the original store untouched.

Test verdict

  • tests/storage-bun-sqlite.test.ts 9/9 (including the real-store UnifiedStore round-trip).
  • Every storage-related suite re-measured per file matches the "after" column above.
  • tests/golden/storage-render/*.json, freshly regenerated upstream, under bun: zero byte mismatches across the four wire cases (chat/anthropic/google/responses). The only failure is closeServer throwing ERR_SERVER_NOT_RUNNING in tests/golden-storage-drive.ts:420-422 during teardown (bun's node:http shim) — i.e. outbound bodies and persisted state are byte-identical on bun:sqlite, so the BLOB→Buffer and BigInt→number normalizations change no observable byte.
  • After merging upstream v0.1.211: storage-bun-sqlite + storage-unified + storage-engine-bootstrap + issue5408 + issue5412 + issue5415 = 80 pass / 4 skip / 1 fail (that 1 is the pre-existing engine-bootstrap known failure).
  • Left to CI: the full Node matrix — this machine has no real Node (~/.local/bin/node is a symlink to bun).

Known residue

  • bun's node:test shim has no t.mock (Support the node:test built-in API oven-sh/bun#5090): suites that rely on it still fail under bun, unrelated to this PR.
  • Some Node-only suites statically import("node:sqlite") at module scope (e.g. issue4949) and report Could not resolve: "node:sqlite" under bun. They should either skip by driver probe or move onto driver.ts; this PR only fixes the 7 hard-coded engine names and does not spread further.
  • tests/golden-storage-drive.ts's closeServer throws ERR_SERVER_NOT_RUNNING under bun (teardown only; assertions unaffected).
  • The Node path is guarded locally only by tsc plus the fake-Database suite (which does not need bun and will run on Node CI): step 0 is strictly gated on process.versions.bun, so Node never reaches the new code.

bun has no node:sqlite, and a better-sqlite3 N-API prebuild cannot be
loaded there at all ("Could not load the default exports from
better-sqlite3", measured on bun 1.3.14), so the ranxianglei#2833 ladder resolved
to no engine and every bun-run proxy degraded to legacy-only storage.

- driver.ts: ladder step 0 = bun:sqlite, required only when
  process.versions.bun exists. bun's builtin is synchronous and already
  shaped like better-sqlite3, so the adapter covers the three measured
  divergences: named params bind by the sigil written in the SQL (the
  store's SQL is better-sqlite3 style and passes bare { a: 1 } bags, so
  bare keys are duplicated into the sigil forms each statement really
  uses, string literals blanked first); BLOB columns come back as a bare
  Uint8Array and are normalized to Buffer; run() reports BigInt
  changes/lastInsertRowid (now Number) and there is no .open flag.
  Writes take .immediate(), reads .deferred() (ranxianglei#2690: a deferred
  transaction that upgrades mid-flight hits SQLITE_BUSY_SNAPSHOT, which
  busy_timeout cannot resolve); close() is idempotent.
- probe seam: a bag pinning only { builtin, host } now simulates a NODE
  host (bun off unless the test opts in with bun: true), so the ladder
  suites keep passing on a machine whose runtime happens to be bun.
- engine-bootstrap.ts: bun never downloads a prebuild (not-needed);
  nodeSqliteUpgradeHint and the degrade banner name bun:sqlite instead
  of the Node lever; the self-contained integrity worker picks its
  engine by the runtime it runs on ({ readonly: true } on bun).
- tests: new tests/storage-bun-sqlite.test.ts (9 tests: fake-Database
  contract pins that also run on Node CI, plus real-bun lock/rollback
  and a real UnifiedStore round-trip through open, ingest, reads, close
  and reopen). Seven raw test handles stop pinning node:sqlite so those
  suites run on any runtime; the three node:sqlite-only legs in
  storage-unified skip by probe instead of failing to load.

Measured under bun against the pre-change baseline on this machine: 24
more storage tests pass, zero regressions (storage-unified 4/13 -> 14
pass + 4 skip + 0 fail; ingest-watermark 1/15 -> 15/1; bounded-load
1/4 -> 4/1; engine-proofs 1/11 -> 7/5; plugin-fork 35/21 -> 51/5;
engine-bootstrap unchanged at 43/1). Remaining bun failures are the
node:test shim's missing t.mock (oven-sh/bun#5090), Node-only suites
that statically import node:sqlite, and ERR_SERVER_NOT_RUNNING in the
child-process harness.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

收到,开始独立审阅 PR #5435。

查重:不重复。 检索 open+closed(关键词 bun、node:sqlite、storage engine ladder):无其他 issue/PR 在做 bun:sqlite 支持。PR 正文引用的 #2825(legacy-only 降级契约)与 #2690(omp 多进程 writer 拓扑)是背景证据,不是平行修复——本 PR 是该机制的唯一在途工作项。

正在做:rebase 新鲜度核对 → 逐文件通读 diff(driver.ts 为核心)→ 既有行为回归清点 → CI 检查读取。结论按结构化八问回复。

ci-check added 2 commits November 1, 2026 18:27
Review finding on ranxianglei#5435 (ai-aligned fix, no production code touched):
the 'bun probe is false on real Node' test called
openSqliteDatabase("/unused/index.db") to assert the picked engine is
not bun:sqlite. On every Node runtime the ladder resolves a real engine
(node:sqlite builtin or better-sqlite3) and then genuinely tries to
open the nonexistent path, throwing 'unable to open database file'
before it can report anything — the assertion could never pass under
Node. Pin probeBunSqlite() === false instead: that IS step 0 of the
ladder, and it needs no database at all.

Verified locally: Node 22 (6 pass / 3 bun-only skips) and bun 1.4.0
(9/9 pass).
The check-unused-exports CI gate fails on this head: BunSqliteCtor is
exported but only used inside driver.ts itself (the require() cast at
openBunSqlite). Drop the leading export per the gate's INTERNAL_ONLY
suggestion — no importers exist.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR Review — feat(storage): run the store on bun's builtin bun:sqlite

独立审完 diff(12 文件 +586/-30)+ 本地双运行时实测(Node v22.23.2 / bun 1.4.0)。方向正确、适配层干净,但发现 2 处会让 CI 红的缺陷,已按外部 PR 规范直接在 PR 分支修掉并推送(两个标注 commit,见下)。

结构化八问

  1. 是什么问题:bun 运行时下(omp 默认用 bun 拉起 bili)引擎阶梯三步全部落空——bun 没有 node:sqlite,better-sqlite3 N-API 预编译在 bun 下必加载失败——自举白白下载后仍只能 legacy-only,index.db 永远空着。本 PR 在阶梯加第 0 步 bun:sqlite(仅 process.versions.bun 存在时探测),让统一存储在 bun 上真正跑起来。
  2. 什么原因导致的:根因是引擎解析只认 Node 世界的两种形态,bun 的内置引擎(better-sqlite3 同形 API,但命名参数按 SQL 书写记号绑定、BLOB 回裸 Uint8Array、run() 回 BigInt、无 .open)无任何适配。修在源头(detectEngine + 一层薄适配器),不是症状层打补丁。
  3. 影响范围与严重程度:仅 bun 运行时受益(从「存储完全不可用」到全功能);Node 路径零变化(第 0 步被运行时门控死,本地实测 probeBunSqlite() 在 Node 恒 false,阶梯次序不变)。
  4. 怎么修复的:三处实测差异各归其位——命名参数按语句把裸键复制成该语句真正用到的记号形态(单引号字面量先抹平;失败模式只有「多给一个 bun 会忽略的键」,单向安全)、BLOB/BigInt/null 归一到 better-sqlite3 契约、写事务 .immediate()/只读 .deferred() 保住 Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 写锁拓扑;自举在 bun 直接 not-needed;状态文案与整史完整性 worker 按运行时选引擎(bun 选项名是 readonly)。
  5. 是否有回归问题:无。逐项过:① Node 路径字节级不变(门控 + 既有 storage 套件本地复跑,storage-engine-proofs 里 4 条 goldens 失败在 master 上同样失败=本沙箱环境因素,非本 PR);② 引擎名消费面全仓 grep——store/backend/shadow/runtime-status 全是透明字符串透传(日志 tag、状态面、重开路径),没有任何假设「恰好两个值」的 switch,持久化格式不存引擎名;③ 探针缝语义变更(钉死的 {builtin,host} 袋视为模拟 Node 主机)对既有测试袋向后兼容;④ wire 零触碰,双压缩模式不受影响;⑤ dist 两道门本地构建实测通过,bun:sqlite 字面量在产物里只以不透明常量/数据串出现,import 位置零出现;⑥ kernel/ 与 pi-subagents/ 未触碰;⑦ 配置面零新增。
  6. 是否破坏缓存:不涉及(纯存储层,无缓存/重试/游标/去重逻辑)。
  7. 是否改了设计方向:不转向——彻底解决 dsh/pnpm 构建脚本门导致 build-blocked:SQLite 引擎自举(内置优先 → 运行时自下载 → 复用宿主) #2833 阶梯原样保留,只在「旧三步结构性不可能工作」的运行时上前置一步;降级链语义不变。
  8. 是否引入新配置:无(无新 env/config/CLI flag)。

行为变更披露(Deliverables 第 4 条)

  • bun 上(本特性目的):统一存储 legacy-only → 全功能;状态/日志文案从指 Node 改为讲 bun:sqlite(不再让运维去升 Node);完整性 worker 改用 bun 只读打开。
  • Node 上:无变化(含日志与 /__bili/status 输出)。

审核发现(均已修复,push 到 PR 分支)

commit 缺陷 处置
0a46006d3 fix(test) tests/storage-bun-sqlite.test.ts:178 调 openSqliteDatabase("/unused/index.db") 断言引擎不是 bun:sqlite——但任何 Node 上阶梯都会解析出真引擎并真的去开这个不存在的路径,抛 unable to open database file,该断言在任何 Node 上都无法通过,Node CI 车道必红 改钉 probeBunSqlite() === false(那就是阶梯第 0 步本身,无需开库)
1d11fd4e5 refactor(storage) src/storage/driver.ts 导出 BunSqliteCtor 但仅文件内部使用,check-unused-exports 门禁红(CI 在作者 head c012442 上确实失败于该 job) 去掉 export(门禁自身的 INTERNAL_ONLY 建议)

验证证据(本地定向,双运行时)

  • 新增套件:Node v22.23.2 → 6 pass / 0 fail / 3 skip(bun-only 组正确跳过);bun 1.4.0 → 9/9 pass(含真 BEGIN/回滚/嵌套事务、两连接写锁立刻交还、真 UnifiedStore 端到端 close→reopen)。
  • 回归钉:PR 触碰的全部测试文件本地复跑绿(除上述 master 同样红的环境性 goldens)。
  • tsc --noEmit 干净;npm run build + check-dist-import-annotations + check-dist-external + check-unused-exports 全过。
  • 作者 head 的 CI:除 unused-exports(已修)外全绿/skip。

判定

两处修复已推送(chidea:master @ 1d11fd4e5),等 CI 在新 head 上转绿即可合并(合并权在人)。测试质量有牙(fake 组钉适配契约、真内置组钉运行时行为),适配层无坏味道,纪律项(dist 门/kernel 边界/配置面)全部守住了。可合,待 CI。

@ranxianglei
ranxianglei merged commit 1e20c69 into ranxianglei:master Nov 4, 2026
86 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants