Repository navigation
[PR] feat(storage): run the store on bun's builtin bun:sqlite - #5435
Merged
Merged
Conversation
bun has no node:sqlite, and a better-sqlite3 N-API prebuild cannot be
loaded there at all ("Could not load the default exports from
better-sqlite3", measured on bun 1.3.14), so the ranxianglei#2833 ladder resolved
to no engine and every bun-run proxy degraded to legacy-only storage.
- driver.ts: ladder step 0 = bun:sqlite, required only when
process.versions.bun exists. bun's builtin is synchronous and already
shaped like better-sqlite3, so the adapter covers the three measured
divergences: named params bind by the sigil written in the SQL (the
store's SQL is better-sqlite3 style and passes bare { a: 1 } bags, so
bare keys are duplicated into the sigil forms each statement really
uses, string literals blanked first); BLOB columns come back as a bare
Uint8Array and are normalized to Buffer; run() reports BigInt
changes/lastInsertRowid (now Number) and there is no .open flag.
Writes take .immediate(), reads .deferred() (ranxianglei#2690: a deferred
transaction that upgrades mid-flight hits SQLITE_BUSY_SNAPSHOT, which
busy_timeout cannot resolve); close() is idempotent.
- probe seam: a bag pinning only { builtin, host } now simulates a NODE
host (bun off unless the test opts in with bun: true), so the ladder
suites keep passing on a machine whose runtime happens to be bun.
- engine-bootstrap.ts: bun never downloads a prebuild (not-needed);
nodeSqliteUpgradeHint and the degrade banner name bun:sqlite instead
of the Node lever; the self-contained integrity worker picks its
engine by the runtime it runs on ({ readonly: true } on bun).
- tests: new tests/storage-bun-sqlite.test.ts (9 tests: fake-Database
contract pins that also run on Node CI, plus real-bun lock/rollback
and a real UnifiedStore round-trip through open, ingest, reads, close
and reopen). Seven raw test handles stop pinning node:sqlite so those
suites run on any runtime; the three node:sqlite-only legs in
storage-unified skip by probe instead of failing to load.
Measured under bun against the pre-change baseline on this machine: 24
more storage tests pass, zero regressions (storage-unified 4/13 -> 14
pass + 4 skip + 0 fail; ingest-watermark 1/15 -> 15/1; bounded-load
1/4 -> 4/1; engine-proofs 1/11 -> 7/5; plugin-fork 35/21 -> 51/5;
engine-bootstrap unchanged at 43/1). Remaining bun failures are the
node:test shim's missing t.mock (oven-sh/bun#5090), Node-only suites
that statically import node:sqlite, and ERR_SERVER_NOT_RUNNING in the
child-process harness.
Owner
[bot] 🏷 收到,开始独立审阅 PR #5435。 查重:不重复。 检索 open+closed(关键词 正在做:rebase 新鲜度核对 → 逐文件通读 diff(driver.ts 为核心)→ 既有行为回归清点 → CI 检查读取。结论按结构化八问回复。 |
added 2 commits
November 1, 2026 18:27
Review finding on ranxianglei#5435 (ai-aligned fix, no production code touched): the 'bun probe is false on real Node' test called openSqliteDatabase("/unused/index.db") to assert the picked engine is not bun:sqlite. On every Node runtime the ladder resolves a real engine (node:sqlite builtin or better-sqlite3) and then genuinely tries to open the nonexistent path, throwing 'unable to open database file' before it can report anything — the assertion could never pass under Node. Pin probeBunSqlite() === false instead: that IS step 0 of the ladder, and it needs no database at all. Verified locally: Node 22 (6 pass / 3 bun-only skips) and bun 1.4.0 (9/9 pass).
The check-unused-exports CI gate fails on this head: BunSqliteCtor is exported but only used inside driver.ts itself (the require() cast at openBunSqlite). Drop the leading export per the gate's INTERNAL_ONLY suggestion — no importers exist.
Owner
(ai: vllm-qwen/qwen3.8-27b) PR Review — feat(storage): run the store on bun's builtin bun:sqlite独立审完 diff(12 文件 +586/-30)+ 本地双运行时实测(Node v22.23.2 / bun 1.4.0)。方向正确、适配层干净,但发现 2 处会让 CI 红的缺陷,已按外部 PR 规范直接在 PR 分支修掉并推送(两个标注 commit,见下)。 结构化八问
行为变更披露(Deliverables 第 4 条)
审核发现(均已修复,push 到 PR 分支)
验证证据(本地定向,双运行时)
判定两处修复已推送( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
bili running on bun can now actually use SQLite: the engine ladder gains a step 0 —
bun:sqlite(bun's builtin) — instead of degrading to legacy-only storage. Byte-for-byte unchanged on Node: step 0 isrequired only whenprocess.versions.bunexists.No new dependency:
bun:sqliteships inside bun,better-sqlite3stays an optionalDependency, and the self-bootstrap download stays a Node-only affair.Why the ladder needs the extra step
Measured on bun 1.3.14:
require("node:sqlite")→ module does not exist (bun has nonode:sqlite) → ladder step 1 empty;require("better-sqlite3")→Error: Could not load the default exports from better-sqlite3(an N-API prebuild cannot be loaded by bun) → steps 2 and 3 empty as well;probeBuiltinSqlite()/probeHostBetterSqlite3()both return false →bootstrapdownloads a prebuild for nothing, the load then fails anyway → the whole proxy falls back to legacy-only (fix(storage): drop better-sqlite3 optional dep (#2823) #2825 behaves exactly as contracted, but for bun users it is a dead end).This is not hypothetical: omp (#2690 multi-process writer topology) starts bili with bun by default, and on that machine
index.dbstayed empty.Key changes
src/storage/driver.ts: ladder step 0 =probeBunSqlite(). bun's builtin is synchronous and already shaped like better-sqlite3 (Database, syncrun/get/all,.transaction()), so the adapter only covers the three measured divergences::a/@b/$c), while all SQL in this codebase is better-sqlite3 style and passes bare bags ({ a: 1 }). Bare keys are duplicated per statement into exactly the sigil forms that statement really uses (string literals are blanked first, so a:nameinside a value is never mistaken for a parameter); all three sigils are written (bun honours only the first match) instead of reaching forallowBareNamedParameters.Uint8Array, while the media/payload readers are written against theBuffercontract (buffer.equals,subarrayslicing depend on Buffer) → normalized toBufferon read.run()return:changes/lastInsertRowidareBigInt→ narrowed tonumber; bun has no.openflag → synthesized (twohandle.openchecks depend on it)..immediate(), read-only takes.deferred()(Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 discipline: a deferred transaction that upgrades to the write lock mid-flight hitsSQLITE_BUSY_SNAPSHOT, whichbusy_timeoutcannot resolve);close()is idempotent (bun throws on an already-closed database, and callers may close twice).{builtin, host}is now treated as a Node host (bun only when a test opts in withbun: true), otherwise every existing ladder suite drifts on a machine whose runtime happens to be bun.src/storage/engine-bootstrap.ts: on bun the bootstrap returnsnot-needed— downloading an N-API prebuild for bun is wasted bandwidth that ends in a load failure; bun never needs a bootstrap.src/storage/runtime-status.ts:nodeSqliteUpgradeHintand the degrade banner namebun:sqliteunder bun instead of telling operators to upgrade Node (a lever that does not exist there).src/storage/integrity-check.ts: the whole-history background worker is a self-contained source string that does not reuse the ladder, so it selects its engine byprocess.versions.bun; bun's read-only option is spelledreadonly(Node spells itreadOnly).tests/storage-bun-sqlite.test.ts(new, 9 tests). The fake-Databasegroup runs through the new test seamsetBunDatabaseFactoryForTestsand therefore runs on Node CI as well (sigil duplication, keys outside literals rewritten exactly once, BigInt/Uint8Array/null normalization, transactions must take.immediate(), step 0 wins the ladder, a pinned{builtin, host}bag means a Node host). The real-builtin group runs only under bun (realBEGIN/rollback, nested transactions must not commit early, the write lock is released immediately after the write, and a realUnifiedStoreround-trip: open with the PRAGMA ladder +quick_check+ migrations, named-parameter insert, read back, close triggering the WAL checkpoint, reopen and the data is still there). Additionally, 7 raw test handles stop hard-codingopenSqliteDatabase(path, "node:sqlite")and follow the automatic probe; the threenode:sqlite-only legs instorage-unifiednow skip by probe instead of blowing up at load.### Storage — bun 运行时原生支持(bun:sqlite)section under## [Unreleased].Measured (same machine, bun 1.3.14, against the pre-change baseline)
24 more tests pass, zero regressions.
tsc --noEmitclean;tsupbuild plus both dist gates (check-dist-import-annotations,check-dist-external) pass.Real-store end-to-end: a copy of the production
sessions/index.db(13 sessions / 2430 messages), served by this repo's bun-built dist:PRAGMA quick_check= ok, the ladder ran, schemauser_version8 → 10, all data intact (blocks 41, content 2469, refs 2433, conversations 13), and the original store untouched.Test verdict
tests/storage-bun-sqlite.test.ts9/9 (including the real-storeUnifiedStoreround-trip).tests/golden/storage-render/*.json, freshly regenerated upstream, under bun: zero byte mismatches across the four wire cases (chat/anthropic/google/responses). The only failure iscloseServerthrowingERR_SERVER_NOT_RUNNINGintests/golden-storage-drive.ts:420-422during teardown (bun'snode:httpshim) — i.e. outbound bodies and persisted state are byte-identical on bun:sqlite, so the BLOB→Buffer and BigInt→number normalizations change no observable byte.storage-bun-sqlite + storage-unified + storage-engine-bootstrap + issue5408 + issue5412 + issue5415= 80 pass / 4 skip / 1 fail (that 1 is the pre-existing engine-bootstrap known failure).~/.local/bin/nodeis a symlink to bun).Known residue
node:testshim has not.mock(Support thenode:testbuilt-in API oven-sh/bun#5090): suites that rely on it still fail under bun, unrelated to this PR.import("node:sqlite")at module scope (e.g.issue4949) and reportCould not resolve: "node:sqlite"under bun. They should either skip by driver probe or move ontodriver.ts; this PR only fixes the 7 hard-coded engine names and does not spread further.tests/golden-storage-drive.ts'scloseServerthrowsERR_SERVER_NOT_RUNNINGunder bun (teardown only; assertions unaffected).tscplus the fake-Databasesuite (which does not need bun and will run on Node CI): step 0 is strictly gated onprocess.versions.bun, so Node never reaches the new code.