From 5b9212965dda2af1bf116d25021c01dfaa103c36 Mon Sep 17 00:00:00 2001 From: zypNB123 Date: Tue, 20 Oct 2026 10:20:22 +0800 Subject: [PATCH 01/10] fix: gate the self-downloaded SQLite engine on the artifact's own Node floor better-sqlite3 v13 declares engines: node >= 22, but the bootstrap pinned only version + sha512 and registered any copy that require() accepted. On Node 20 the win32-x64 prebuild faults inside process.dlopen (access violation) and the linux-x64 one faults on the first new Database() - uncatchable either way, so a bootstrap probe cannot defend against it. The crash loops the host process instead of degrading to legacy storage. The pin now carries the artifact's own Node major floor and runBootstrap() refuses BOTH the download and a cached-copy registration below it, returning unsupported-node with an explicit upgrade reason. The gate sits after the builtin/host probes (a host-installed better-sqlite3@12 on Node 20 is fine) and before the cache check (the cached 13.0.3 copy is the crash source). A cached copy is left on disk so it becomes usable again after a Node upgrade. No new config surface, no cooldown (a too-old Node is not a transient download failure). Fixes #3894. Refs #3341. --- CONFIGURATION.md | 3 +- CONFIGURATION.zh-CN.md | 3 +- src/storage/engine-bootstrap.ts | 35 ++++++- src/storage/runtime-status.ts | 2 + tests/storage-engine-bootstrap.test.ts | 137 ++++++++++++++++++++++--- website/config-reference/process.yaml | 4 +- 6 files changed, 158 insertions(+), 26 deletions(-) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 56a191b3d3..afcb6c1236 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,8 +175,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Below the pinned artifact's own Node floor (22 for better-sqlite3 v13) neither the download nor a cached copy is registered - loading that binding there crashes the process instead of failing gracefully (#3341). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 727a85219c..6379bf8051 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,8 +175,7 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择,可插拔存储计划第一步(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约,sqlite 为默认且是当前唯一真实现(现有 UnifiedStore 套一层 Promise 包装适配器 —— 零行为变化、零 schema 变化)。mysql 是已注册占位:该名字能通过校验(真实后端随第二步落地 —— mysql2 客户端,走可选依赖+自下载纪律 —— 期间配置不会在 unknown/unimplemented 之间来回弹),但选中它只会在 lane 启动时打一行提示并继续用 sqlite;直接开后端的路径(export)按文档化的错误失败。未知名字同样回退 sqlite 并提示。每个 lane 启动时解析一次。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。在钉死产物自身的 Node 下限(better-sqlite3 v13 为 22)以下,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | diff --git a/src/storage/engine-bootstrap.ts b/src/storage/engine-bootstrap.ts index dccb15c25a..2f6c8a3052 100644 --- a/src/storage/engine-bootstrap.ts +++ b/src/storage/engine-bootstrap.ts @@ -14,8 +14,16 @@ * * Artifact: better-sqlite3 v13 ships N-API prebuilds for all eight * platform/arch combinations INSIDE its npm tarball (no install script, no - * per-ABI split — N-API is ABI-stable across Node versions). We pin version + - * sha512 in-repo; bumping either is a human-visible PR. Download egress honors + * per-ABI split). N-API is ABI-stable across Node versions, but ABI stability + * is NOT a support guarantee: v13 declares `engines: node >= 22`, and loading + * its binding outside that range is undefined behavior rather than a clean + * failure — on Node 20 the win32-x64 prebuild faults inside `process.dlopen` + * (access violation, uncatchable) and the linux-x64 one faults on the first + * `new Database()` (#3341). So the pin carries the artifact's OWN Node floor + * next to version + sha512, and the floor gates BOTH the download and the + * cached-copy registration: a copy that cannot legally load must never be + * registered. We pin version + sha512 + floor in-repo; bumping any of the + * three is a human-visible PR. Download egress honors * the same proxy decision chain as model traffic (#609) and the configured * registry mirror (BILI_UPDATE_REGISTRY / update.registry). * @@ -43,12 +51,15 @@ interface EngineBootstrapSpec { version: string; /** sha512 base64 digest of the registry tarball — pinned in-repo, verified on every download. */ sha512Base64: string; + /** Node major floor copied from the artifact's own `engines.node`; below it the binding is undefined behavior (process crash), not a graceful failure. */ + minNodeMajor: number; } -/** Pinned engine artifact (#2833). Bump version AND digest together in one PR. */ +/** Pinned engine artifact (#2833). Bump version, digest AND the Node floor together in one PR. */ const DEFAULT_ENGINE_SPEC: Readonly = Object.freeze({ version: "13.0.3", sha512Base64: "RbOBxmLBG8uvFUc15X9+9SFemKcQ0WBuISBVkpuiaUB2qblC8UWlHEjdWVoZ8AdhSwmoEgsiXKfopX0CQxaACQ==", + minNodeMajor: 22, }); const SUPPORTED_TARGETS = [ @@ -71,6 +82,7 @@ type BootstrapStatus = | "downloaded" | "disabled" | "unsupported-platform" + | "unsupported-node" | "cooldown" | "failed"; @@ -109,6 +121,18 @@ function activeSpec(): EngineBootstrapSpec { return specOverride ?? DEFAULT_ENGINE_SPEC; } +/** + * Pure floor predicate (version in -> verdict out) so tests pin the threshold + * boundary without depending on the Node running them. An unparseable major + * counts as satisfied: `process.versions.node` is semver by contract, and + * bricking unified storage on an exotic runtime would be the worse failure. + */ +export function nodeMajorSatisfiesFloor(minNodeMajor: number, nodeVersion: string = process.versions.node): boolean { + const major = Number.parseInt(nodeVersion.split(".")[0] ?? "", 10); + if (!Number.isFinite(major)) return true; + return major >= minNodeMajor; +} + /** Platform key matching better-sqlite3's own binding.js musl detection. */ export function platformTargetKey(overrides?: { platform?: NodeJS.Platform; arch?: string }): string { const platform = overrides?.platform ?? process.platform; @@ -223,6 +247,11 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise= ${spec.minNodeMajor}, this is ${process.versions.node}`; + loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${detail}; a self-downloaded engine would crash this process on load, staying on legacy storage (upgrade Node.js to ${spec.minNodeMajor}+ for unified storage)`); + return { status: "unsupported-node", detail }; + } const target = platformOverride ?? platformTargetKey(); if (cacheUsable(finalDir(spec.version), target)) { if (!tryRegisterFromCache(finalDir(spec.version))) { diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index 32967d0774..0205ce170c 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -99,6 +99,8 @@ function degradeReason(): string { return `SQLite self-download blocked (${b.detail ?? "unknown"}), retrying after cooldown`; case "unsupported-platform": return `unsupported platform for SQLite self-download (${b.detail ?? "?"})`; + case "unsupported-node": + return `self-downloaded SQLite engine requires a newer Node.js (${b.detail ?? "unknown"})`; case "disabled": return "SQLite self-download disabled by configuration"; case "failed": diff --git a/tests/storage-engine-bootstrap.test.ts b/tests/storage-engine-bootstrap.test.ts index f398ac9cfa..7e9cb627ce 100644 --- a/tests/storage-engine-bootstrap.test.ts +++ b/tests/storage-engine-bootstrap.test.ts @@ -44,6 +44,7 @@ import { BOOTSTRAP_TIMEOUT_MS, ensureSqliteEngine, extractTgzToDir, + nodeMajorSatisfiesFloor, nodeSqliteUpgradeHint, platformTargetKey, resetEngineBootstrapForTests, @@ -331,7 +332,7 @@ test("bootstrap: happy path downloads, verifies sha, extracts, registers loader; try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -377,7 +378,7 @@ test("#3818 unloadable cached copy is removed and a pinned download is retried ( try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -416,7 +417,7 @@ test("bootstrap: sha512 mismatch quarantines the download and arms the cooldown" const good = fakePackageTgz(); const tampered = Buffer.from(good); tampered[tampered.length - 1] ^= 0xff; - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(good) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(good), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tampered); await withRegistry(reg.base, async () => { @@ -448,7 +449,7 @@ test("bootstrap: expired cooldown re-attempts the download", async () => { try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); const sqliteDir = path.join(cachePrev, "billion-context", "sqlite"); fs.mkdirSync(sqliteDir, { recursive: true }); @@ -473,7 +474,7 @@ test("bootstrap: stale unusable cache dir is replaced instead of looping the coo try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); // Partial/corrupt publish from a killed run: dir exists but lacks the // target prebuilt, so cacheUsable() is false and rename-over would ENOTEMPTY. @@ -506,7 +507,7 @@ test("bootstrap: a download that outlasts its timeout budget aborts fail-safe an try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); const delayMs = 800; setBootstrapTimeoutMsForTests(Math.floor(delayMs / 5)); @@ -532,7 +533,7 @@ test("bootstrap: a slow-but-completing download succeeds once the budget covers try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); const delayMs = 400; setBootstrapTimeoutMsForTests(delayMs * 10); @@ -554,7 +555,7 @@ test("bootstrap: offline (connection refused) degrades fail-safe with cooldown, const cachePrev = isolateCache(); try { resetAllSeams(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: "irrelevant" }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: "irrelevant", minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); await withRegistry("http://127.0.0.1:1", async () => { const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); @@ -576,7 +577,7 @@ test("bootstrap: disabled knob gates the fetch (cached copies would still be use try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -598,7 +599,7 @@ test("bootstrap: legacy mode short-circuits before any probe or fetch", async () try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -623,7 +624,7 @@ test("bootstrap: builtin present -> not-needed without touching the network", as try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: true, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -646,7 +647,7 @@ test("bootstrap: host-provided engine -> not-needed without touching the network try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: true }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -669,7 +670,7 @@ test("bootstrap: unsupported platform key -> no download attempt", async () => { try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); setPlatformKeyForTests("freebsd-arm64"); reg = await startMockRegistry(tgz); @@ -693,7 +694,7 @@ test("bootstrap: concurrent callers share one attempt (process-dedup)", async () try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -821,7 +822,7 @@ test("mirror fallback: a failing first mirror falls to the next and the bootstra try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); bad = await startMockRegistry(tgz, 500); // registry up, tarball broken at the HTTP layer good = await startMockRegistry(tgz); @@ -847,7 +848,7 @@ test("mirror exhaustion: the cooldown reason lists every mirror attempted (#3338 try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); bad1 = await startMockRegistry(tgz, 500); bad2 = await startMockRegistry(tgz, 503); @@ -875,7 +876,7 @@ test("mirror list: a configured registry is attempted before the builtins (#3338 try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); // Deliberately NO mirrors override: the production list must put the @@ -913,3 +914,105 @@ test("nodeSqliteUpgradeHint: pins the Node >= 22.5 threshold and the levers (#33 assert.match(nodeSqliteUpgradeHint("19.9.9"), /Node\.js >= 22\.5/, "19.x predates node:sqlite"); assert.equal(nodeSqliteUpgradeHint("22.5.1"), "", "22.5.1 ships node:sqlite — no hint"); }); + +// --------------------------------------------------------------------------- +// Artifact Node floor (#3341): better-sqlite3 v13 declares `engines: node >= 22` +// and faults the process on Node 20 instead of failing cleanly, so the pinned +// spec carries that floor and the bootstrap refuses BOTH the download and the +// cached-copy registration below it. +// --------------------------------------------------------------------------- + +test("nodeMajorSatisfiesFloor: both directions of the artifact floor (#3341)", () => { + assert.equal(nodeMajorSatisfiesFloor(22, "20.19.5"), false, "Node 20 is below the v13 floor"); + assert.equal(nodeMajorSatisfiesFloor(22, "18.20.8"), false, "Node 18 is below the v13 floor"); + assert.equal(nodeMajorSatisfiesFloor(22, "22.0.0"), true, "Node 22 satisfies the floor"); + assert.equal(nodeMajorSatisfiesFloor(22, "22.4.9"), true, "the floor is a MAJOR floor - 22.x minors all pass"); + assert.equal(nodeMajorSatisfiesFloor(22, "24.11.0"), true, "Node 24 satisfies the floor"); + assert.equal(nodeMajorSatisfiesFloor(20, "20.19.5"), true, "the predicate tracks the floor it is given, not a hardcoded 22"); + assert.equal(nodeMajorSatisfiesFloor(22, "garbage"), true, "an unparseable runtime must not brick unified storage"); +}); + +test("bootstrap: below the artifact Node floor -> unsupported-node, no download, no cache registration (#3341)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + // minNodeMajor above ANY Node that could run this suite: the gate must + // fire regardless of which runtime CI uses (ubuntu/windows x 22/24). + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 99 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + reg = await startMockRegistry(tgz); + await withRegistry(reg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `expected the floor gate, got ${JSON.stringify(out)}`); + assert.match(out.detail ?? "", /better-sqlite3@13\.0\.3 needs Node\.js >= 99/, "the detail names the artifact and its floor"); + assert.equal(reg!.requests(), 0, "below the floor the ~11 MB download must never start"); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3")), "nothing materialized into the cache"); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", ".bootstrap-failed.json")), "a too-old Node is not a download failure - no cooldown armed"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); + }); + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: a cache that already exists is left in place (and unregistered) below the floor (#3341)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 99 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + const target = platformTargetKey(); + const cached = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3"); + fs.mkdirSync(path.join(cached, "lib"), { recursive: true }); + fs.mkdirSync(path.join(cached, "prebuilds"), { recursive: true }); + fs.writeFileSync(path.join(cached, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "13.0.3" })); + fs.writeFileSync(path.join(cached, "lib", "index.js"), FAKE_LIB_INDEX); + fs.writeFileSync(path.join(cached, "prebuilds", `${target}.node`), Buffer.from("FAKE-NODE-BINARY")); + reg = await startMockRegistry(tgz); + await withRegistry(reg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `expected the floor gate, got ${JSON.stringify(out)}`); + assert.equal(reg!.requests(), 0, "a cache hit below the floor must not trigger a re-download either"); + assert.ok(fs.existsSync(path.join(cached, "prebuilds", `${target}.node`)), "the cached copy must survive: a Node upgrade makes it valid again"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "a cache below the floor must never be registered"); + }); + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: at/above the artifact Node floor the gate does NOT fire (#3341 negative case)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + reg = await startMockRegistry(tgz); + await withRegistry(reg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `a satisfied floor must not block the bootstrap, got ${JSON.stringify(out)}`); + assert.equal(reg!.requests(), 1, "the download proceeded"); + const o = openSqliteDatabase(":memory:"); + assert.equal(o.engine, "better-sqlite3"); + assert.equal(o.source, "self-downloaded"); + o.db.close(); + }); + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 8c004ae2ef..da8927b52d 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -264,8 +264,8 @@ entries: default: "true" env: BILI_SQLITE_AUTO_DOWNLOAD group: process - desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." - desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" + desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Below the pinned artifact's own Node floor (22 for better-sqlite3 v13) neither the download nor a cached copy is registered - loading that binding there crashes the process instead of failing gracefully (#3341). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." + desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。在钉死产物自身的 Node 下限(better-sqlite3 v13 为 22)以下,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" - key: storage.jsonWriter type: "on|off" default: "on" From a6f7f5c1f5ee6535b674ab9f8d5ea945d85fdc55 Mon Sep 17 00:00:00 2001 From: awork Date: Tue, 20 Oct 2026 11:16:21 +0800 Subject: [PATCH 02/10] fix(runtime-status): surface the unsupported-node floor reason before the sidecar-open-error branch (#3894) Review fix for #3896. The new case "unsupported-node" in degradeReason() was unreachable: when bootstrap returns unsupported-node no loader is registered, so the sidecar open ALSO records a "no SQLite engine available" open-error, and the existing sidecarOpenError branch returned first - shadowing the specific reason. Users saw "better-sqlite3 not installed" instead of the real cause (a too-old Node). Promote the floor reason ahead of that branch and drop the now-dead switch case. Regression test pins the acp_status reason for the unsupported-node scenario. --- src/storage/runtime-status.ts | 8 ++++-- tests/acp-status-storage-degraded.test.ts | 31 +++++++++++++++++++++++ 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index 0205ce170c..6303da848a 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -83,6 +83,12 @@ function degradeReason(): string { const last = sidecar.stats.lastError ? shortenError(sidecar.stats.lastError) : "unknown"; return `unified store auto-disabled after ${MAX_CONSECUTIVE_FAILURES} consecutive failures (last: ${last})`; } + // A too-old Node is refused BEFORE any engine load, so the sidecar open ALSO + // fails with the generic no-engine error — surface the real floor reason first, + // or the sidecar-open-error branch below would shadow it (#3894). + if (state.bootstrap?.status === "unsupported-node") { + return `self-downloaded SQLite engine requires a newer Node.js (${state.bootstrap.detail ?? "unknown"})`; + } if (state.sidecarOpenError) { const e = shortenError(state.sidecarOpenError); // The driver's no-engine error surfaces through the same open path — @@ -99,8 +105,6 @@ function degradeReason(): string { return `SQLite self-download blocked (${b.detail ?? "unknown"}), retrying after cooldown`; case "unsupported-platform": return `unsupported platform for SQLite self-download (${b.detail ?? "?"})`; - case "unsupported-node": - return `self-downloaded SQLite engine requires a newer Node.js (${b.detail ?? "unknown"})`; case "disabled": return "SQLite self-download disabled by configuration"; case "failed": diff --git a/tests/acp-status-storage-degraded.test.ts b/tests/acp-status-storage-degraded.test.ts index eda94fd3b5..ba6e107b4c 100644 --- a/tests/acp-status-storage-degraded.test.ts +++ b/tests/acp-status-storage-degraded.test.ts @@ -111,3 +111,34 @@ test("#3385: healthy storage stays quiet — no degrade line", () => { rmrf(base); } }); + +test("#3894: below the artifact Node floor the degrade reason names the Node upgrade, not generic no-engine", () => { + // Unsupported-node refuses the engine BEFORE any load, so the sidecar open + // ALSO records a "no SQLite engine available" open-error. The floor reason + // must outrank that generic one — otherwise the user sees "better-sqlite3 + // not installed" when the real cause is a too-old Node (#3894 dead-code fix). + const base = mkdtempSync(path.join(tmpdir(), "acp-status-3894-")); + const snap = isolate(base); + let store: SessionStore | null = null; + try { + setSqliteEnginesUnavailableForTests(true); + _resetStorageRuntimeStatusForTest(); + recordEngineBootstrap({ status: "unsupported-node", detail: "better-sqlite3@13.0.3 needs Node.js >= 22, this is 20.19.5" }); + store = new SessionStore({ dir: process.env.BILI_SESSIONS_DIR, enabled: true }); + _setStoreForTest(store); + + const out = handleAcpStatus({}, makeCtx()); + const text = out.text ?? ""; + assert.ok(text.includes("STORAGE —"), "the storage line is present"); + assert.ok(text.includes("⚠ degraded —"), `the degrade warning line is present (got: ${text.slice(-400)})`); + assert.ok(text.includes("requires a newer Node.js"), `the specific floor reason surfaces (got: ${text.slice(-400)})`); + assert.ok(/Node\.js >= 22/.test(text), "the reason carries the artifact's own Node floor"); + } finally { + store?.closeQuietly(); + _setStoreForTest(null as unknown as SessionStore); + setSqliteEnginesUnavailableForTests(false); + _resetStorageRuntimeStatusForTest(); + restore(snap); + rmrf(base); + } +}); From 044e5246d76154589f45300d6edb63b510f4aaf3 Mon Sep 17 00:00:00 2001 From: awork Date: Tue, 20 Oct 2026 11:33:19 +0800 Subject: [PATCH 03/10] docs(config): BILI_STORAGE_UNIFIED row keeps both the Node-floor clause and the #3818 default wording Rebase follow-up for #3896: the hand-maintained env-var row must carry both the PR's Node-floor sentence and master's post-#3818 'Defaults to full when unset' tail - the two independent conflict resolutions had split them. Same fix in CONFIGURATION.zh-CN.md. --- CONFIGURATION.md | 2 +- CONFIGURATION.zh-CN.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index afcb6c1236..5dfc1c3ff6 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -1888,7 +1888,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_PERSIST_DEBOUNCE_MS` | Debounce window for persistence writes to disk, in ms (default `500`). | | `BILI_PERSIST_TAIL_TOKENS` | Token budget for the persisted conversation snapshot (#401). The disk record stores the **folded view** (block summaries in place of compressed ranges) truncated to the newest messages within this budget — never the raw full history. Default `16384`; `0` disables message persistence entirely (block summaries and compressed originals still persist; `bili export` falls back to block-only rendering). Live in-memory sessions are unaffected — `bili export` of a live session is always complete. | | `BILI_PERSIST_ZSTD` | Set `1`/`true` to enable zstd compression of session files (#1080, owner decision: **default off** — plain JSON maximizes recoverability: jq/grep-debuggable, and immune to the downgrade tail risk). When enabled, every session file that actually shrinks is written as `BILIZSTD1` — a small header (magic + format version + mode byte) over the JSON body, zstd-compressed on Node ≥ 22.15 and stored raw on older runtimes; readers accept both bodies (native zstd, else the bundled WASM fallback), so files stay readable across runtimes and versions. Payloads the framing cannot shrink (tiny sessions) and key-less raw bodies stay bare JSON on disk. Independent of `BILI_ENCRYPTION_KEY` — with both active, the compression choice is recorded in the mode byte inside `BILIENC1`. Existing plain-JSON files are NEVER rewritten at boot (downgrade safety — a mass re-encode would make rolling back to an older bili read its own writes as "corrupt"); they convert organically on their next save. Note: once a session has been saved as `BILIZSTD1`, an OLDER bili (< 0.1.135) cannot read it — this caveat applies only to deployments that opted in; unset or any other value keeps plain JSON. | -| `BILI_STORAGE_UNIFIED` | Tiered storage-migration switch (#2671). Config-file tier: `{"storage":{"unified":"legacy|shadow|full"}}` selects the same modes (strict spellings) and applies whenever the env is unset; `BILI_STORAGE_UNIFIED` (which also accepts `1`/`true` = shadow and `unified` = full) always wins over the file. The mode is resolved once per lane boot — hot edits to the file take effect on the next restart. Set `1`/`true` to enable the **unified SQLite storage dry-run** (Phase 1). Every session write is double-ingested into `/index.db` (engine resolution order since #2833: built-in `node:sqlite` when the runtime provides it (Node >= 22.5) -> an already-installed host `better-sqlite3` native module -> a previously self-downloaded cached copy -> a one-time self-download of the pinned `better-sqlite3` prebuilt tarball from the npm registry (~11 MB, sha512-verified, gated by `storage.sqliteAutoDownload`, unsupported platforms never download) -> none, in which case the sidecar stays legacy-only per the #2825 degrade contract; engine acquisition never depends on the host's install-time build-script approval) and the unified replay is digest-reconciled against the legacy load view — a mismatch logs `[storage] unified shadow DIGEST DRIFT …` and never blocks or alters the write path (legacy files stay the single source of truth). Set `full`/`unified` to enable the **drop-in read path** (Phase 1.5): `loadSync`/`boot`/`loadAll`/`loadStateForSearch` go unified-first — a legacy-file session is organically imported into the index on first open (`[storage] unified import on first open: …`) and later loads are served from `index.db`, with automatic fallback to the legacy file on any unified failure. Legacy files keep being double-written, so **rollback = unset the knob** (env and/or file value). Hard encryption opt-out: while `BILI_ENCRYPTION_KEY` is active the whole sidecar refuses to open (index.db would hold cleartext ingest payloads; a `unified sidecar DISABLED` warning is logged and the store stays legacy-only). Enumeration/GC and the web session list remain file-backed; `removeLegacyFile` now also deletes the index rows and releases CAS refcounts. Digests treat the last-writer `biliVersion` stamp and wall-clock timestamps as environment (`__VER__`/`__TS__`), everything else must match byte-for-byte. The shadow self-disables after 3 consecutive internal failures (in full mode that also flips reads back to legacy-only). **Replica/checkpoint REST (Phase 2)**: on any unified lane (shadow or full), `POST /__bili/replicas {sessionId, label?, pinned?}` freezes the session's persisted state as one content-addressed checkpoint blob + row (the resident session is flushed first), `GET /__bili/replicas[?sessionId=]` lists them, `GET /__bili/replicas/` returns detail + byte length, `POST /__bili/replicas//restore {sessionId?}` materializes the checkpoint as a NEW first-class session through the normal dual write path (JSON + unified ingest; `meta.restoredFrom`/`restoredCheckpoint` record provenance), and `DELETE /__bili/replicas/` drops it. Pinned checkpoints hold a CAS refcount and survive `deleteSession` GC (the whole point: a copy taken before a destructive experiment outlives it); explicit DELETE still removes even pinned ones. Disk expectation: shadow/full roughly doubles unique-content bytes (every session has a JSON file AND index.db rows sharing a content CAS with no dedup across the two stores) and there is no auto-VACUUM — size the sessions dir accordingly while the migration runs. Legacy-only lanes answer 501 with the knob hint. Routes are loopback-only like the rest of `__bili`. Defaults to `full` when unset (#3818). | +| `BILI_STORAGE_UNIFIED` | Tiered storage-migration switch (#2671). Config-file tier: `{"storage":{"unified":"legacy|shadow|full"}}` selects the same modes (strict spellings) and applies whenever the env is unset; `BILI_STORAGE_UNIFIED` (which also accepts `1`/`true` = shadow and `unified` = full) always wins over the file. The mode is resolved once per lane boot — hot edits to the file take effect on the next restart. Set `1`/`true` to enable the **unified SQLite storage dry-run** (Phase 1). Every session write is double-ingested into `/index.db` (engine resolution order since #2833: built-in `node:sqlite` when the runtime provides it (Node >= 22.5) -> an already-installed host `better-sqlite3` native module -> a previously self-downloaded cached copy -> a one-time self-download of the pinned `better-sqlite3` prebuilt tarball from the npm registry (~11 MB, sha512-verified, gated by `storage.sqliteAutoDownload`, unsupported platforms never download; below the pinned artifact's own Node floor (22 for better-sqlite3 v13) neither the download nor a cached copy is registered, since loading it there crashes the process (#3341)) -> none, in which case the sidecar stays legacy-only per the #2825 degrade contract; engine acquisition never depends on the host's install-time build-script approval) and the unified replay is digest-reconciled against the legacy load view — a mismatch logs `[storage] unified shadow DIGEST DRIFT …` and never blocks or alters the write path (legacy files stay the single source of truth). Set `full`/`unified` to enable the **drop-in read path** (Phase 1.5): `loadSync`/`boot`/`loadAll`/`loadStateForSearch` go unified-first — a legacy-file session is organically imported into the index on first open (`[storage] unified import on first open: …`) and later loads are served from `index.db`, with automatic fallback to the legacy file on any unified failure. Legacy files keep being double-written, so **rollback = unset the knob** (env and/or file value). Hard encryption opt-out: while `BILI_ENCRYPTION_KEY` is active the whole sidecar refuses to open (index.db would hold cleartext ingest payloads; a `unified sidecar DISABLED` warning is logged and the store stays legacy-only). Enumeration/GC and the web session list remain file-backed; `removeLegacyFile` now also deletes the index rows and releases CAS refcounts. Digests treat the last-writer `biliVersion` stamp and wall-clock timestamps as environment (`__VER__`/`__TS__`), everything else must match byte-for-byte. The shadow self-disables after 3 consecutive internal failures (in full mode that also flips reads back to legacy-only). **Replica/checkpoint REST (Phase 2)**: on any unified lane (shadow or full), `POST /__bili/replicas {sessionId, label?, pinned?}` freezes the session's persisted state as one content-addressed checkpoint blob + row (the resident session is flushed first), `GET /__bili/replicas[?sessionId=]` lists them, `GET /__bili/replicas/` returns detail + byte length, `POST /__bili/replicas//restore {sessionId?}` materializes the checkpoint as a NEW first-class session through the normal dual write path (JSON + unified ingest; `meta.restoredFrom`/`restoredCheckpoint` record provenance), and `DELETE /__bili/replicas/` drops it. Pinned checkpoints hold a CAS refcount and survive `deleteSession` GC (the whole point: a copy taken before a destructive experiment outlives it); explicit DELETE still removes even pinned ones. Disk expectation: shadow/full roughly doubles unique-content bytes (every session has a JSON file AND index.db rows sharing a content CAS with no dedup across the two stores) and there is no auto-VACUUM — size the sessions dir accordingly while the migration runs. Legacy-only lanes answer 501 with the knob hint. Routes are loopback-only like the rest of `__bili`. Defaults to `full` when unset (#3818). | | `BILI_PERSIST_EPERM_ALERT_THRESHOLD` | N consecutive persist write failures (`EPERM`/`EBUSY`/`EACCES`) on one session before the one-time "add this dir to antivirus exclusions" alert fires (default `5`). Windows only. See [Windows: exclude the sessions dir](#windows-exclude-the-sessions-dir-from-antivirus-362). | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | Re-alert window for the persist EPERM alert, in ms. `0` (default) = alert once then stay silent; `>0` = re-alert at most every that many ms while the failures continue. | | `BILI_TUNNEL_ALLOWED_HOSTS` | `/bili/` tunnel admission for **remote clients** (#409): comma-separated `host` or `host:port` entries that unlock loopback/private destinations (e.g. a LAN relay or the machine's own sglang) for non-loopback clients. The proxy itself and link-local/metadata addresses are always denied; local (loopback) clients always pass. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 6379bf8051..446c02359a 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -1885,7 +1885,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_PERSIST_DEBOUNCE_MS` | 持久化写盘的防抖窗口(毫秒,默认 `500`)。 | | `BILI_PERSIST_TAIL_TOKENS` | 持久化会话快照的 token 预算(#401)。盘上记录的是**折叠视图**(压缩范围以块摘要替代)并截断到该预算内的最新消息 —— 不再存全量原始历史。默认 `16384`;设 `0` 彻底不持久化消息(块摘要与压缩原件仍会持久化,`bili export` 退回块级渲染)。活会话内存不受影响 —— 活会话的 `bili export` 始终完整。 | | `BILI_PERSIST_ZSTD` | 设为 `1`/`true` 启用会话文件的 zstd 压缩(#1080,owner 决定:**默认关闭**——纯 JSON 可恢复性最强:可用 jq/grep 调试,且无降级尾部风险)。启用后,每个确实能压缩变小的会话文件均以 `BILIZSTD1` 格式写入——即在 JSON 主体之上附加一个小头部(魔数 + 格式版本 + 模式字节),在 Node ≥ 22.15 上以 zstd 压缩存储,在旧版运行时上则原样存储;读取端兼容两种主体格式(优先使用原生 zstd,否则回退至内置的 WASM 实现),因此文件在不同运行时和版本间均可正常读取。压不小的小会话与无密钥的原始主体以裸 JSON 落盘。与 `BILI_ENCRYPTION_KEY` 相互独立——两者同时生效时,压缩方式会记录在 `BILIENC1` 内部的模式字节中。已有的纯 JSON 文件**永不在启动时改写**(降级安全——批量重编码会让回退到旧版 bili 时把自己的写入当成“损坏文件”);它们在其下一次保存时自然转换。注意:一旦会话已保存为 `BILIZSTD1`,旧版 bili(< 0.1.135)无法读取——该限制仅对显式启用的部署生效;未设置或其他值均保持纯 JSON。 | -| `BILI_STORAGE_UNIFIED` | 存储迁移分层开关(#2671)。配置文件层:`{"storage":{"unified":"legacy|shadow|full"}}` 选用相同模式(严格拼写),在环境变量未设时生效;`BILI_STORAGE_UNIFIED`(额外接受 `1`/`true`=shadow、`unified`=full)始终优先于文件。模式在每个 lane 启动时解析一次——文件热改动在下次重启生效。设 `1`/`true` 启用**统一 SQLite 存储干跑**(Phase 1):每次会话写入都会双写到 `/index.db`(引擎解析顺序自 #2833 起:运行时提供内置 `node:sqlite` 时优先(Node >= 22.5)-> 宿主已装的 `better-sqlite3` 原生模块 -> 先前自下载的缓存副本 -> 一次性自下载钉死的 `better-sqlite3` 预编译 tarball(约 11 MB,sha512 校验,受 `storage.sqliteAutoDownload` 控制,不支持的平台永不下载)-> 无引擎则按 #2825 降级契约保持仅 legacy;引擎获取永不依赖宿主安装期的构建脚本审批),并将统一层回放与旧格式加载视图做 digest 对账——不匹配时记录 `[storage] unified shadow DIGEST DRIFT …`,绝不阻塞或改写主写路径(旧格式文件仍是唯一事实源)。设 `full`/`unified` 启用**平替读路径**(Phase 1.5):`loadSync`/`boot`/`loadAll`/`loadStateForSearch` 改为 unified-first——首次打开旧文件会话时有机导入索引(`[storage] unified import on first open: …`),后续加载直接由 `index.db` 提供,任意统一层失败自动回退旧文件;旧格式文件照常双写,**回滚 = 去掉开关(环境变量与/或文件值)**。加密硬互斥:`BILI_ENCRYPTION_KEY` 激活时整个 sidecar 拒绝开启(index.db 明文存 ingest 载荷,记录 `unified sidecar DISABLED` 警告后保持 legacy-only)。枚举/GC 与 web 列表仍以文件为基底;`removeLegacyFile` 同步删除索引行并释放 CAS 引用计数。digest 把末次写入者版本戳 `biliVersion` 与墙钟时间戳视为环境(`__VER__`/`__TS__`),其余字节必须逐一致。连续 3 次内部失败后自动停用(full 模式下停用即回退 legacy 读)。**副本/检查点 REST(Phase 2)**:任意统一层 lane(shadow 或 full)上,`POST /__bili/replicas {sessionId, label?, pinned?}` 把会话持久化状态冻结为一条内容寻址的 checkpoint blob+行(先强制刷新驻留会话),`GET /__bili/replicas[?sessionId=]` 列出副本,`GET /__bili/replicas/` 返回详情+字节数,`POST /__bili/replicas//restore {sessionId?}` 把检查点物化为全新的一等会话(走正常双写路径:JSON+统一层 ingest;`meta.restoredFrom`/`restoredCheckpoint` 记录来源),`DELETE /__bili/replicas/` 删除。pinned 副本持有 CAS 引用计数,可在 `deleteSession` GC 中存活(本功能的意义:破坏性实验前抓的副本活得过实验本身);显式 DELETE 仍可删 pinned。legacy-only lane 返回 501 并附开关提示。路由与 `__bili` 其余部分一样仅限 loopback。未设置时默认 `full`(#3818)。 | +| `BILI_STORAGE_UNIFIED` | 存储迁移分层开关(#2671)。配置文件层:`{"storage":{"unified":"legacy|shadow|full"}}` 选用相同模式(严格拼写),在环境变量未设时生效;`BILI_STORAGE_UNIFIED`(额外接受 `1`/`true`=shadow、`unified`=full)始终优先于文件。模式在每个 lane 启动时解析一次——文件热改动在下次重启生效。设 `1`/`true` 启用**统一 SQLite 存储干跑**(Phase 1):每次会话写入都会双写到 `/index.db`(引擎解析顺序自 #2833 起:运行时提供内置 `node:sqlite` 时优先(Node >= 22.5)-> 宿主已装的 `better-sqlite3` 原生模块 -> 先前自下载的缓存副本 -> 一次性自下载钉死的 `better-sqlite3` 预编译 tarball(约 11 MB,sha512 校验,受 `storage.sqliteAutoDownload` 控制,不支持的平台永不下载;在钉死产物自身的 Node 下限(better-sqlite3 v13 为 22)以下,下载与缓存副本注册都不进行,因为在那里加载会让进程崩溃(#3341))-> 无引擎则按 #2825 降级契约保持仅 legacy;引擎获取永不依赖宿主安装期的构建脚本审批),并将统一层回放与旧格式加载视图做 digest 对账——不匹配时记录 `[storage] unified shadow DIGEST DRIFT …`,绝不阻塞或改写主写路径(旧格式文件仍是唯一事实源)。设 `full`/`unified` 启用**平替读路径**(Phase 1.5):`loadSync`/`boot`/`loadAll`/`loadStateForSearch` 改为 unified-first——首次打开旧文件会话时有机导入索引(`[storage] unified import on first open: …`),后续加载直接由 `index.db` 提供,任意统一层失败自动回退旧文件;旧格式文件照常双写,**回滚 = 去掉开关(环境变量与/或文件值)**。加密硬互斥:`BILI_ENCRYPTION_KEY` 激活时整个 sidecar 拒绝开启(index.db 明文存 ingest 载荷,记录 `unified sidecar DISABLED` 警告后保持 legacy-only)。枚举/GC 与 web 列表仍以文件为基底;`removeLegacyFile` 同步删除索引行并释放 CAS 引用计数。digest 把末次写入者版本戳 `biliVersion` 与墙钟时间戳视为环境(`__VER__`/`__TS__`),其余字节必须逐一致。连续 3 次内部失败后自动停用(full 模式下停用即回退 legacy 读)。**副本/检查点 REST(Phase 2)**:任意统一层 lane(shadow 或 full)上,`POST /__bili/replicas {sessionId, label?, pinned?}` 把会话持久化状态冻结为一条内容寻址的 checkpoint blob+行(先强制刷新驻留会话),`GET /__bili/replicas[?sessionId=]` 列出副本,`GET /__bili/replicas/` 返回详情+字节数,`POST /__bili/replicas//restore {sessionId?}` 把检查点物化为全新的一等会话(走正常双写路径:JSON+统一层 ingest;`meta.restoredFrom`/`restoredCheckpoint` 记录来源),`DELETE /__bili/replicas/` 删除。pinned 副本持有 CAS 引用计数,可在 `deleteSession` GC 中存活(本功能的意义:破坏性实验前抓的副本活得过实验本身);显式 DELETE 仍可删 pinned。legacy-only lane 返回 501 并附开关提示。路由与 `__bili` 其余部分一样仅限 loopback。未设置时默认 `full`(#3818)。 | | `BILI_PERSIST_EPERM_ALERT_THRESHOLD` | 同一会话连续 N 次持久化写失败(`EPERM`/`EBUSY`/`EACCES`)后触发一次性「把该目录加入杀软排除项」告警的阈值(默认 `5`)。仅 Windows。见下文「Windows:把会话目录加入杀软排除项」章节。 | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | persist EPERM 告警的重复窗口(毫秒)。`0`(默认)= 只告警一次后静默;`>0` = 失败持续期间最多每这么久重复告警一次。 | | `BILI_MAX_SESSIONS` | 内存中最多保留的会话数(默认 `256`;LRU 淘汰 —— 磁盘是事实源)。 | From 111b8ea2ebda8ed6766bf63b74777491d97c11a3 Mon Sep 17 00:00:00 2001 From: zypNB123 Date: Tue, 20 Oct 2026 13:53:11 +0800 Subject: [PATCH 04/10] fix: keep low Node versions on the unified lane with a per-major engine artifact MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit better-sqlite3 v13 declares `engines: node >= 22` and compiles its prebuilds for N-API 10, while Node 20 only provides N-API 9. Loading that binding below its floor is undefined behavior, not a clean failure: the win32-x64 prebuild faults inside `process.dlopen` (access violation, uncatchable) and the linux-x64 one faults on the first `new Database()` (#3341, #3894). The self-bootstrap registered such a copy whenever it happened to be cached, which crash-looped the proxy child (1485 spawns, exit 0xC0000005) on Node 20. The previous revision of this branch refused the artifact below its floor and degraded to legacy JSON storage. That is the wrong answer: legacy storage is scheduled for removal, so "low Node" must stay on the unified lane with a lower artifact rather than fall off it. Pin one artifact per Node major band instead: * Node >= 22 -> better-sqlite3 13.0.3, unchanged: one tarball that carries its own eight prebuilds. * Node 20 -> better-sqlite3 12.9.0, the last 12.x line that publishes an ABI 115 binding. Its npm tarball is pure JS (no binding, no prebuilds); the native half is a separate ABI-keyed asset (`better-sqlite3-v12.9.0-node-v115-.tar.gz`) that the bootstrap now fetches from the binary mirror and materializes as `prebuilds/.node`. v12's `lib/database.js` accepts an injected `nativeBinding`, so the two halves compose without `bindings`, `prebuild-install` or any install script — the #2833 contract holds: pinned artifacts, sha512 verified on every download, no host build approval. A spec is eligible only when its Node floor AND (when it has one) its prebuild ABI match the running Node. A runtime no pinned artifact serves reports `unsupported-node` with the full table in the detail instead of registering a binding that would crash the process. Pinned sibling cache dirs are no longer pruned, so switching Node majors does not re-download an artifact this build already knows how to load. Verified on a real Node 20.19.5 (ABI 115): downloads 12.9.0 + its ABI 115 prebuild, binds through nativeBinding, runs DDL/DML/transactions, process survives. On Node 22.22.3 the same cache resolves to builtin node:sqlite and the v12 copy is left untouched. Tests: 5 new cases (JS-only artifact binds its ABI prebuild; a mismatched ABI is refused before any download; a self-contained artifact never touches the binary mirror; the prebuild extractor lands and fails loud). Both directions of the gate are asserted per #3617. Fixes #3894. Refs #3341. --- CONFIGURATION.md | 4 +- CONFIGURATION.zh-CN.md | 4 +- src/storage/engine-bootstrap.ts | 337 +++++++++++++++++----- src/storage/runtime-status.ts | 9 +- tests/acp-status-storage-degraded.test.ts | 14 +- tests/storage-engine-bootstrap.test.ts | 206 +++++++++++++ website/config-reference/process.yaml | 4 +- 7 files changed, 481 insertions(+), 97 deletions(-) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 5dfc1c3ff6..8c8744e6c2 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,7 +175,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Below the pinned artifact's own Node floor (22 for better-sqlite3 v13) neither the download nor a cached copy is registered - loading that binding there crashes the process instead of failing gracefully (#3341). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER NODE MAJOR, because a binding cannot be loaded outside the range its package declares: Node >= 22 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), Node 20 uses v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding) - so a low Node stays on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | @@ -1888,7 +1888,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_PERSIST_DEBOUNCE_MS` | Debounce window for persistence writes to disk, in ms (default `500`). | | `BILI_PERSIST_TAIL_TOKENS` | Token budget for the persisted conversation snapshot (#401). The disk record stores the **folded view** (block summaries in place of compressed ranges) truncated to the newest messages within this budget — never the raw full history. Default `16384`; `0` disables message persistence entirely (block summaries and compressed originals still persist; `bili export` falls back to block-only rendering). Live in-memory sessions are unaffected — `bili export` of a live session is always complete. | | `BILI_PERSIST_ZSTD` | Set `1`/`true` to enable zstd compression of session files (#1080, owner decision: **default off** — plain JSON maximizes recoverability: jq/grep-debuggable, and immune to the downgrade tail risk). When enabled, every session file that actually shrinks is written as `BILIZSTD1` — a small header (magic + format version + mode byte) over the JSON body, zstd-compressed on Node ≥ 22.15 and stored raw on older runtimes; readers accept both bodies (native zstd, else the bundled WASM fallback), so files stay readable across runtimes and versions. Payloads the framing cannot shrink (tiny sessions) and key-less raw bodies stay bare JSON on disk. Independent of `BILI_ENCRYPTION_KEY` — with both active, the compression choice is recorded in the mode byte inside `BILIENC1`. Existing plain-JSON files are NEVER rewritten at boot (downgrade safety — a mass re-encode would make rolling back to an older bili read its own writes as "corrupt"); they convert organically on their next save. Note: once a session has been saved as `BILIZSTD1`, an OLDER bili (< 0.1.135) cannot read it — this caveat applies only to deployments that opted in; unset or any other value keeps plain JSON. | -| `BILI_STORAGE_UNIFIED` | Tiered storage-migration switch (#2671). Config-file tier: `{"storage":{"unified":"legacy|shadow|full"}}` selects the same modes (strict spellings) and applies whenever the env is unset; `BILI_STORAGE_UNIFIED` (which also accepts `1`/`true` = shadow and `unified` = full) always wins over the file. The mode is resolved once per lane boot — hot edits to the file take effect on the next restart. Set `1`/`true` to enable the **unified SQLite storage dry-run** (Phase 1). Every session write is double-ingested into `/index.db` (engine resolution order since #2833: built-in `node:sqlite` when the runtime provides it (Node >= 22.5) -> an already-installed host `better-sqlite3` native module -> a previously self-downloaded cached copy -> a one-time self-download of the pinned `better-sqlite3` prebuilt tarball from the npm registry (~11 MB, sha512-verified, gated by `storage.sqliteAutoDownload`, unsupported platforms never download; below the pinned artifact's own Node floor (22 for better-sqlite3 v13) neither the download nor a cached copy is registered, since loading it there crashes the process (#3341)) -> none, in which case the sidecar stays legacy-only per the #2825 degrade contract; engine acquisition never depends on the host's install-time build-script approval) and the unified replay is digest-reconciled against the legacy load view — a mismatch logs `[storage] unified shadow DIGEST DRIFT …` and never blocks or alters the write path (legacy files stay the single source of truth). Set `full`/`unified` to enable the **drop-in read path** (Phase 1.5): `loadSync`/`boot`/`loadAll`/`loadStateForSearch` go unified-first — a legacy-file session is organically imported into the index on first open (`[storage] unified import on first open: …`) and later loads are served from `index.db`, with automatic fallback to the legacy file on any unified failure. Legacy files keep being double-written, so **rollback = unset the knob** (env and/or file value). Hard encryption opt-out: while `BILI_ENCRYPTION_KEY` is active the whole sidecar refuses to open (index.db would hold cleartext ingest payloads; a `unified sidecar DISABLED` warning is logged and the store stays legacy-only). Enumeration/GC and the web session list remain file-backed; `removeLegacyFile` now also deletes the index rows and releases CAS refcounts. Digests treat the last-writer `biliVersion` stamp and wall-clock timestamps as environment (`__VER__`/`__TS__`), everything else must match byte-for-byte. The shadow self-disables after 3 consecutive internal failures (in full mode that also flips reads back to legacy-only). **Replica/checkpoint REST (Phase 2)**: on any unified lane (shadow or full), `POST /__bili/replicas {sessionId, label?, pinned?}` freezes the session's persisted state as one content-addressed checkpoint blob + row (the resident session is flushed first), `GET /__bili/replicas[?sessionId=]` lists them, `GET /__bili/replicas/` returns detail + byte length, `POST /__bili/replicas//restore {sessionId?}` materializes the checkpoint as a NEW first-class session through the normal dual write path (JSON + unified ingest; `meta.restoredFrom`/`restoredCheckpoint` record provenance), and `DELETE /__bili/replicas/` drops it. Pinned checkpoints hold a CAS refcount and survive `deleteSession` GC (the whole point: a copy taken before a destructive experiment outlives it); explicit DELETE still removes even pinned ones. Disk expectation: shadow/full roughly doubles unique-content bytes (every session has a JSON file AND index.db rows sharing a content CAS with no dedup across the two stores) and there is no auto-VACUUM — size the sessions dir accordingly while the migration runs. Legacy-only lanes answer 501 with the knob hint. Routes are loopback-only like the rest of `__bili`. Defaults to `full` when unset (#3818). | +| `BILI_STORAGE_UNIFIED` | Tiered storage-migration switch (#2671). Config-file tier: `{"storage":{"unified":"legacy|shadow|full"}}` selects the same modes (strict spellings) and applies whenever the env is unset; `BILI_STORAGE_UNIFIED` (which also accepts `1`/`true` = shadow and `unified` = full) always wins over the file. The mode is resolved once per lane boot — hot edits to the file take effect on the next restart. Set `1`/`true` to enable the **unified SQLite storage dry-run** (Phase 1). Every session write is double-ingested into `/index.db` (engine resolution order since #2833: built-in `node:sqlite` when the runtime provides it (Node >= 22.5) -> an already-installed host `better-sqlite3` native module -> a previously self-downloaded cached copy -> a one-time self-download of the pinned `better-sqlite3` prebuilt tarball from the npm registry (~11 MB, sha512-verified, gated by `storage.sqliteAutoDownload`, unsupported platforms never download; engine artifacts are pinned per Node major (Node >= 22: better-sqlite3 v13; Node 20: v12.9.0 plus its ABI-keyed prebuild), and a runtime no pinned artifact serves is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process (#3341, #3894)) -> none, in which case the sidecar stays legacy-only per the #2825 degrade contract; engine acquisition never depends on the host's install-time build-script approval) and the unified replay is digest-reconciled against the legacy load view — a mismatch logs `[storage] unified shadow DIGEST DRIFT …` and never blocks or alters the write path (legacy files stay the single source of truth). Set `full`/`unified` to enable the **drop-in read path** (Phase 1.5): `loadSync`/`boot`/`loadAll`/`loadStateForSearch` go unified-first — a legacy-file session is organically imported into the index on first open (`[storage] unified import on first open: …`) and later loads are served from `index.db`, with automatic fallback to the legacy file on any unified failure. Legacy files keep being double-written, so **rollback = unset the knob** (env and/or file value). Hard encryption opt-out: while `BILI_ENCRYPTION_KEY` is active the whole sidecar refuses to open (index.db would hold cleartext ingest payloads; a `unified sidecar DISABLED` warning is logged and the store stays legacy-only). Enumeration/GC and the web session list remain file-backed; `removeLegacyFile` now also deletes the index rows and releases CAS refcounts. Digests treat the last-writer `biliVersion` stamp and wall-clock timestamps as environment (`__VER__`/`__TS__`), everything else must match byte-for-byte. The shadow self-disables after 3 consecutive internal failures (in full mode that also flips reads back to legacy-only). **Replica/checkpoint REST (Phase 2)**: on any unified lane (shadow or full), `POST /__bili/replicas {sessionId, label?, pinned?}` freezes the session's persisted state as one content-addressed checkpoint blob + row (the resident session is flushed first), `GET /__bili/replicas[?sessionId=]` lists them, `GET /__bili/replicas/` returns detail + byte length, `POST /__bili/replicas//restore {sessionId?}` materializes the checkpoint as a NEW first-class session through the normal dual write path (JSON + unified ingest; `meta.restoredFrom`/`restoredCheckpoint` record provenance), and `DELETE /__bili/replicas/` drops it. Pinned checkpoints hold a CAS refcount and survive `deleteSession` GC (the whole point: a copy taken before a destructive experiment outlives it); explicit DELETE still removes even pinned ones. Disk expectation: shadow/full roughly doubles unique-content bytes (every session has a JSON file AND index.db rows sharing a content CAS with no dedup across the two stores) and there is no auto-VACUUM — size the sessions dir accordingly while the migration runs. Legacy-only lanes answer 501 with the knob hint. Routes are loopback-only like the rest of `__bili`. Defaults to `full` when unset (#3818). | | `BILI_PERSIST_EPERM_ALERT_THRESHOLD` | N consecutive persist write failures (`EPERM`/`EBUSY`/`EACCES`) on one session before the one-time "add this dir to antivirus exclusions" alert fires (default `5`). Windows only. See [Windows: exclude the sessions dir](#windows-exclude-the-sessions-dir-from-antivirus-362). | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | Re-alert window for the persist EPERM alert, in ms. `0` (default) = alert once then stay silent; `>0` = re-alert at most every that many ms while the failures continue. | | `BILI_TUNNEL_ALLOWED_HOSTS` | `/bili/` tunnel admission for **remote clients** (#409): comma-separated `host` or `host:port` entries that unlock loopback/private destinations (e.g. a LAN relay or the machine's own sglang) for non-loopback clients. The proxy itself and link-local/metadata addresses are always denied; local (loopback) clients always pass. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 446c02359a..4cd7236616 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,7 +175,7 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。在钉死产物自身的 Node 下限(better-sqlite3 v13 为 22)以下,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按 Node 大版本分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),Node 20 用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | @@ -1885,7 +1885,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_PERSIST_DEBOUNCE_MS` | 持久化写盘的防抖窗口(毫秒,默认 `500`)。 | | `BILI_PERSIST_TAIL_TOKENS` | 持久化会话快照的 token 预算(#401)。盘上记录的是**折叠视图**(压缩范围以块摘要替代)并截断到该预算内的最新消息 —— 不再存全量原始历史。默认 `16384`;设 `0` 彻底不持久化消息(块摘要与压缩原件仍会持久化,`bili export` 退回块级渲染)。活会话内存不受影响 —— 活会话的 `bili export` 始终完整。 | | `BILI_PERSIST_ZSTD` | 设为 `1`/`true` 启用会话文件的 zstd 压缩(#1080,owner 决定:**默认关闭**——纯 JSON 可恢复性最强:可用 jq/grep 调试,且无降级尾部风险)。启用后,每个确实能压缩变小的会话文件均以 `BILIZSTD1` 格式写入——即在 JSON 主体之上附加一个小头部(魔数 + 格式版本 + 模式字节),在 Node ≥ 22.15 上以 zstd 压缩存储,在旧版运行时上则原样存储;读取端兼容两种主体格式(优先使用原生 zstd,否则回退至内置的 WASM 实现),因此文件在不同运行时和版本间均可正常读取。压不小的小会话与无密钥的原始主体以裸 JSON 落盘。与 `BILI_ENCRYPTION_KEY` 相互独立——两者同时生效时,压缩方式会记录在 `BILIENC1` 内部的模式字节中。已有的纯 JSON 文件**永不在启动时改写**(降级安全——批量重编码会让回退到旧版 bili 时把自己的写入当成“损坏文件”);它们在其下一次保存时自然转换。注意:一旦会话已保存为 `BILIZSTD1`,旧版 bili(< 0.1.135)无法读取——该限制仅对显式启用的部署生效;未设置或其他值均保持纯 JSON。 | -| `BILI_STORAGE_UNIFIED` | 存储迁移分层开关(#2671)。配置文件层:`{"storage":{"unified":"legacy|shadow|full"}}` 选用相同模式(严格拼写),在环境变量未设时生效;`BILI_STORAGE_UNIFIED`(额外接受 `1`/`true`=shadow、`unified`=full)始终优先于文件。模式在每个 lane 启动时解析一次——文件热改动在下次重启生效。设 `1`/`true` 启用**统一 SQLite 存储干跑**(Phase 1):每次会话写入都会双写到 `/index.db`(引擎解析顺序自 #2833 起:运行时提供内置 `node:sqlite` 时优先(Node >= 22.5)-> 宿主已装的 `better-sqlite3` 原生模块 -> 先前自下载的缓存副本 -> 一次性自下载钉死的 `better-sqlite3` 预编译 tarball(约 11 MB,sha512 校验,受 `storage.sqliteAutoDownload` 控制,不支持的平台永不下载;在钉死产物自身的 Node 下限(better-sqlite3 v13 为 22)以下,下载与缓存副本注册都不进行,因为在那里加载会让进程崩溃(#3341))-> 无引擎则按 #2825 降级契约保持仅 legacy;引擎获取永不依赖宿主安装期的构建脚本审批),并将统一层回放与旧格式加载视图做 digest 对账——不匹配时记录 `[storage] unified shadow DIGEST DRIFT …`,绝不阻塞或改写主写路径(旧格式文件仍是唯一事实源)。设 `full`/`unified` 启用**平替读路径**(Phase 1.5):`loadSync`/`boot`/`loadAll`/`loadStateForSearch` 改为 unified-first——首次打开旧文件会话时有机导入索引(`[storage] unified import on first open: …`),后续加载直接由 `index.db` 提供,任意统一层失败自动回退旧文件;旧格式文件照常双写,**回滚 = 去掉开关(环境变量与/或文件值)**。加密硬互斥:`BILI_ENCRYPTION_KEY` 激活时整个 sidecar 拒绝开启(index.db 明文存 ingest 载荷,记录 `unified sidecar DISABLED` 警告后保持 legacy-only)。枚举/GC 与 web 列表仍以文件为基底;`removeLegacyFile` 同步删除索引行并释放 CAS 引用计数。digest 把末次写入者版本戳 `biliVersion` 与墙钟时间戳视为环境(`__VER__`/`__TS__`),其余字节必须逐一致。连续 3 次内部失败后自动停用(full 模式下停用即回退 legacy 读)。**副本/检查点 REST(Phase 2)**:任意统一层 lane(shadow 或 full)上,`POST /__bili/replicas {sessionId, label?, pinned?}` 把会话持久化状态冻结为一条内容寻址的 checkpoint blob+行(先强制刷新驻留会话),`GET /__bili/replicas[?sessionId=]` 列出副本,`GET /__bili/replicas/` 返回详情+字节数,`POST /__bili/replicas//restore {sessionId?}` 把检查点物化为全新的一等会话(走正常双写路径:JSON+统一层 ingest;`meta.restoredFrom`/`restoredCheckpoint` 记录来源),`DELETE /__bili/replicas/` 删除。pinned 副本持有 CAS 引用计数,可在 `deleteSession` GC 中存活(本功能的意义:破坏性实验前抓的副本活得过实验本身);显式 DELETE 仍可删 pinned。legacy-only lane 返回 501 并附开关提示。路由与 `__bili` 其余部分一样仅限 loopback。未设置时默认 `full`(#3818)。 | +| `BILI_STORAGE_UNIFIED` | 存储迁移分层开关(#2671)。配置文件层:`{"storage":{"unified":"legacy|shadow|full"}}` 选用相同模式(严格拼写),在环境变量未设时生效;`BILI_STORAGE_UNIFIED`(额外接受 `1`/`true`=shadow、`unified`=full)始终优先于文件。模式在每个 lane 启动时解析一次——文件热改动在下次重启生效。设 `1`/`true` 启用**统一 SQLite 存储干跑**(Phase 1):每次会话写入都会双写到 `/index.db`(引擎解析顺序自 #2833 起:运行时提供内置 `node:sqlite` 时优先(Node >= 22.5)-> 宿主已装的 `better-sqlite3` 原生模块 -> 先前自下载的缓存副本 -> 一次性自下载钉死的 `better-sqlite3` 预编译 tarball(约 11 MB,sha512 校验,受 `storage.sqliteAutoDownload` 控制,不支持的平台永不下载;引擎产物按 Node 大版本分别钉死(Node >= 22 用 better-sqlite3 v13;Node 20 用 v12.9.0 及其 ABI 匹配的 prebuild),任何钉死产物都不支持的运行时会在加载之前被拒绝,下载与缓存副本注册都不进行,因为在那里加载该 binding 会让进程崩溃(#3341、#3894))-> 无引擎则按 #2825 降级契约保持仅 legacy;引擎获取永不依赖宿主安装期的构建脚本审批),并将统一层回放与旧格式加载视图做 digest 对账——不匹配时记录 `[storage] unified shadow DIGEST DRIFT …`,绝不阻塞或改写主写路径(旧格式文件仍是唯一事实源)。设 `full`/`unified` 启用**平替读路径**(Phase 1.5):`loadSync`/`boot`/`loadAll`/`loadStateForSearch` 改为 unified-first——首次打开旧文件会话时有机导入索引(`[storage] unified import on first open: …`),后续加载直接由 `index.db` 提供,任意统一层失败自动回退旧文件;旧格式文件照常双写,**回滚 = 去掉开关(环境变量与/或文件值)**。加密硬互斥:`BILI_ENCRYPTION_KEY` 激活时整个 sidecar 拒绝开启(index.db 明文存 ingest 载荷,记录 `unified sidecar DISABLED` 警告后保持 legacy-only)。枚举/GC 与 web 列表仍以文件为基底;`removeLegacyFile` 同步删除索引行并释放 CAS 引用计数。digest 把末次写入者版本戳 `biliVersion` 与墙钟时间戳视为环境(`__VER__`/`__TS__`),其余字节必须逐一致。连续 3 次内部失败后自动停用(full 模式下停用即回退 legacy 读)。**副本/检查点 REST(Phase 2)**:任意统一层 lane(shadow 或 full)上,`POST /__bili/replicas {sessionId, label?, pinned?}` 把会话持久化状态冻结为一条内容寻址的 checkpoint blob+行(先强制刷新驻留会话),`GET /__bili/replicas[?sessionId=]` 列出副本,`GET /__bili/replicas/` 返回详情+字节数,`POST /__bili/replicas//restore {sessionId?}` 把检查点物化为全新的一等会话(走正常双写路径:JSON+统一层 ingest;`meta.restoredFrom`/`restoredCheckpoint` 记录来源),`DELETE /__bili/replicas/` 删除。pinned 副本持有 CAS 引用计数,可在 `deleteSession` GC 中存活(本功能的意义:破坏性实验前抓的副本活得过实验本身);显式 DELETE 仍可删 pinned。legacy-only lane 返回 501 并附开关提示。路由与 `__bili` 其余部分一样仅限 loopback。未设置时默认 `full`(#3818)。 | | `BILI_PERSIST_EPERM_ALERT_THRESHOLD` | 同一会话连续 N 次持久化写失败(`EPERM`/`EBUSY`/`EACCES`)后触发一次性「把该目录加入杀软排除项」告警的阈值(默认 `5`)。仅 Windows。见下文「Windows:把会话目录加入杀软排除项」章节。 | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | persist EPERM 告警的重复窗口(毫秒)。`0`(默认)= 只告警一次后静默;`>0` = 失败持续期间最多每这么久重复告警一次。 | | `BILI_MAX_SESSIONS` | 内存中最多保留的会话数(默认 `256`;LRU 淘汰 —— 磁盘是事实源)。 | diff --git a/src/storage/engine-bootstrap.ts b/src/storage/engine-bootstrap.ts index 2f6c8a3052..cf3f6fa8fd 100644 --- a/src/storage/engine-bootstrap.ts +++ b/src/storage/engine-bootstrap.ts @@ -12,18 +12,30 @@ * contract). This module owns ONLY the download/cache step; driver.ts owns * resolution order and opening. * - * Artifact: better-sqlite3 v13 ships N-API prebuilds for all eight - * platform/arch combinations INSIDE its npm tarball (no install script, no - * per-ABI split). N-API is ABI-stable across Node versions, but ABI stability - * is NOT a support guarantee: v13 declares `engines: node >= 22`, and loading - * its binding outside that range is undefined behavior rather than a clean - * failure — on Node 20 the win32-x64 prebuild faults inside `process.dlopen` - * (access violation, uncatchable) and the linux-x64 one faults on the first - * `new Database()` (#3341). So the pin carries the artifact's OWN Node floor - * next to version + sha512, and the floor gates BOTH the download and the - * cached-copy registration: a copy that cannot legally load must never be - * registered. We pin version + sha512 + floor in-repo; bumping any of the - * three is a human-visible PR. Download egress honors + * Artifacts are pinned PER NODE MAJOR, because the binding is not portable + * below the floor the package declares (#3341, #3894). better-sqlite3 v13 + * ships N-API prebuilds for all eight platform/arch combinations INSIDE its + * npm tarball, but it declares `engines: node >= 22` and compiles them for + * N-API 10, while Node 20 only provides N-API 9 — loading that binding on + * Node 20 is undefined behavior rather than a clean failure: the win32-x64 + * prebuild faults inside `process.dlopen` (access violation, uncatchable) and + * the linux-x64 one faults on the first `new Database()`. Node 20 therefore + * gets the last 12.x line, which declares `engines: 20.x || 22.x || ...` and + * ships no binding of its own: its npm tarball is pure JS and the native half + * is an ABI-keyed prebuild (`better-sqlite3-v-node-v115-.tar.gz`) + * published on the GitHub release / npmmirror binary mirrors. v12's + * `lib/database.js` accepts an injected `nativeBinding`, so the two halves + * compose without `bindings`, `prebuild-install` or any install script — the + * #2833 contract holds: pinned artifacts, sha512 verified on every download, + * no host build approval. Downgrading such users to legacy JSON storage would + * be the wrong answer: legacy is scheduled for removal, so "low Node" must + * stay on the unified lane with a lower artifact, not fall off it. + * + * A spec is eligible only when its Node floor AND (when it has one) its + * prebuild ABI match the running Node. When nothing is eligible the bootstrap + * reports `unsupported-node` instead of registering a binding that would crash + * the process. We pin version + sha512 + floor + per-target prebuild digests + * in-repo; bumping any of them is a human-visible PR. Download egress honors * the same proxy decision chain as model traffic (#609) and the configured * registry mirror (BILI_UPDATE_REGISTRY / update.registry). * @@ -47,20 +59,57 @@ import { fileStorageAutoDownload, fileStorageUnified, resolveStorageMode, resolv import { normalizeRegistryBase } from "../update.js"; import { proxyDispatcher } from "../upstream-proxy.js"; +/** The separate ABI-keyed binding artifact a JS-only npm tarball needs. */ +interface EnginePrebuild { + /** Node `process.versions.modules` ABI the binaries were built for (115 = Node 20). */ + nodeAbi: string; + /** sha512 base64 per platform target — pinned in-repo, verified on every download. */ + sha512Base64ByTarget: Readonly>; +} + interface EngineBootstrapSpec { version: string; /** sha512 base64 digest of the registry tarball — pinned in-repo, verified on every download. */ sha512Base64: string; /** Node major floor copied from the artifact's own `engines.node`; below it the binding is undefined behavior (process crash), not a graceful failure. */ minNodeMajor: number; + /** Present when the npm tarball carries no binding of its own (v12 and earlier). */ + prebuild?: EnginePrebuild; } -/** Pinned engine artifact (#2833). Bump version, digest AND the Node floor together in one PR. */ -const DEFAULT_ENGINE_SPEC: Readonly = Object.freeze({ - version: "13.0.3", - sha512Base64: "RbOBxmLBG8uvFUc15X9+9SFemKcQ0WBuISBVkpuiaUB2qblC8UWlHEjdWVoZ8AdhSwmoEgsiXKfopX0CQxaACQ==", - minNodeMajor: 22, -}); +/** + * Pinned engine artifacts (#2833), one per Node major band. Bump version, + * digests, floor and ABI together in one PR. + * + * Order is preference order: the newest artifact that serves the running Node + * wins, so a Node 22+ machine keeps the single-tarball v13 lane even though + * v12 could also serve it. + */ +const ENGINE_SPECS: readonly EngineBootstrapSpec[] = Object.freeze([ + { + version: "13.0.3", + sha512Base64: "RbOBxmLBG8uvFUc15X9+9SFemKcQ0WBuISBVkpuiaUB2qblC8UWlHEjdVVoZ8AdhSwmoEgsiXKfopX0CQxaACQ==", + minNodeMajor: 22, + }, + { + version: "12.9.0", + sha512Base64: "wqUv4Gm3toFpHDQmaKD4QhZm3g1DjUBI0yzS4UBl6lElUmXFYdTQmmEDpAFa5o8FiFiymURypEnfVHzILKaxqQ==", + minNodeMajor: 20, + prebuild: { + nodeAbi: "115", + sha512Base64ByTarget: Object.freeze({ + "linux-x64": "fNMKurhPKb0ki9mnZJK9PVazCiRugcdmauo5qGjqQjUGIrdaVWrb4nlLwU7IiyppcfLu8n/qUsQ5CEZ36uoGaw==", + "linux-arm64": "7q7wZe0+5LLQt2YUkocGb41B9TvEMLKIaNI+CiQUQQ2DFBO1RPr3ixj/Gtb+0noMYHjJrqtqKUMO1xMQXnE0iw==", + "linuxmusl-x64": "GCriICDKw7K5dNjAVo8xFpSJXcYXXBAK9RDoeuSUv6rB5IVRP96IwxeynTy4Xc21l1507Efz+I2yy6FW5gKQ/w==", + "linuxmusl-arm64": "2OWfVP1nklnuR8ZfCwozJcLB0ZpOTRHqAkDpsLonEF5uHZMBnOAzGtd5d+nuPU+KaS5czciU0tXZFhsCUw7dbg==", + "darwin-x64": "tW2Vb19+Bh+PLfcGVnrERhc9Mk50Vi8HciIc0nwIJXlPm21xzKL6uEQHMoyKIAxF+vNXxDsUoJyNRg/TgCXjkA==", + "darwin-arm64": "H6jehr7s1D/RSTRplCL++hneLTgpqVzgzjrX/NJCsf0OMBSKNGCd9kMYAZJjMlp0IetS39sG5PTG7AcLCTQKeg==", + "win32-x64": "r+BAeN980mQEdIwDcrNU3iJqzAryuOyfeICFfYHuHVclonTr2L0IlzsaGp2D50KNWUeLQrIhBCFuNoT2+iBDTw==", + "win32-arm64": "jMSa36VrAiLsWFtnwWX+FEs4JWIjHeduVITY44ZcHsqn7BzPcMljL8LdJ0tVVGssV8x9PuqgIqMD5gFmzcYh9w==", + }), + }, + }, +]); const SUPPORTED_TARGETS = [ "linux-x64", "linux-arm64", @@ -69,10 +118,11 @@ const SUPPORTED_TARGETS = [ "win32-x64", "win32-arm64", ]; -// Whole-download budget for the ~11 MB engine tarball. Must cover a multi-MB -// registry fetch over a non-trivial network (corporate proxy / slow link); -// 10 s locked Node<=21 machines into a 6 h legacy-only cooldown after one slow -// boot (#3338). Aligned with the updater's own tarball budget (src/update.ts). +// Whole-download budget for the engine artifacts (v13: one ~11 MB tarball; +// v12: ~2.6 MB of JS plus a ~1 MB prebuild). Must cover a multi-MB registry +// fetch over a non-trivial network (corporate proxy / slow link); 10 s locked +// Node<=21 machines into a 6 h legacy-only cooldown after one slow boot +// (#3338). Aligned with the updater's own tarball budget (src/update.ts). export const BOOTSTRAP_TIMEOUT_MS = 60_000; export const COOLDOWN_MS = 6 * 60 * 60 * 1000; @@ -95,7 +145,7 @@ let specOverride: EngineBootstrapSpec | null = null; let platformOverride: string | null = null; let inFlight: Promise | null = null; -/** Test seam: swap the pinned artifact (version/sha) for a hermetic fixture. */ +/** Test seam: swap the pinned artifact (version/digests/floor) for a hermetic fixture. */ export function setEngineBootstrapSpecForTests(spec: EngineBootstrapSpec | null): void { specOverride = spec; } @@ -117,10 +167,6 @@ export function setBootstrapTimeoutMsForTests(ms: number | null): void { bootstrapTimeoutOverrideMs = ms; } -function activeSpec(): EngineBootstrapSpec { - return specOverride ?? DEFAULT_ENGINE_SPEC; -} - /** * Pure floor predicate (version in -> verdict out) so tests pin the threshold * boundary without depending on the Node running them. An unparseable major @@ -133,6 +179,33 @@ export function nodeMajorSatisfiesFloor(minNodeMajor: number, nodeVersion: strin return major >= minNodeMajor; } +/** Why this spec cannot serve the running Node, or null when it can. Both + * halves are load-bearing: the floor is what the package declares, the ABI is + * what the binary was actually built against — a JS-only tarball whose + * prebuild targets a different ABI loads just as badly as one below its floor. */ +function specUnusableReason(spec: EngineBootstrapSpec, nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules): string | null { + if (!nodeMajorSatisfiesFloor(spec.minNodeMajor, nodeVersion)) { + return `better-sqlite3@${spec.version} needs Node.js >= ${spec.minNodeMajor}, this is ${nodeVersion}`; + } + if (spec.prebuild && spec.prebuild.nodeAbi !== nodeAbi) { + return `better-sqlite3@${spec.version} ships no binding for Node ABI ${nodeAbi} (its prebuild is pinned to ABI ${spec.prebuild.nodeAbi}), this is Node.js ${nodeVersion}`; + } + return null; +} + +function pinnedSpecForThisNode(): EngineBootstrapSpec | null { + return ENGINE_SPECS.find((s) => specUnusableReason(s) === null) ?? null; +} + +/** Honest "nothing fits" text: names the running Node/ABI and every pinned + * band, so the report explains itself instead of blaming the network. */ +function unsupportedNodeDetail(nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules): string { + const covered = ENGINE_SPECS.map((s) => (s.prebuild + ? `better-sqlite3@${s.version} on Node.js ${s.minNodeMajor}.x (ABI ${s.prebuild.nodeAbi})` + : `better-sqlite3@${s.version} on Node.js >= ${s.minNodeMajor}`)).join(" or "); + return `no pinned SQLite engine runs on Node.js ${nodeVersion} (ABI ${nodeAbi}) — the pinned artifacts cover ${covered}`; +} + /** Platform key matching better-sqlite3's own binding.js musl detection. */ export function platformTargetKey(overrides?: { platform?: NodeJS.Platform; arch?: string }): string { const platform = overrides?.platform ?? process.platform; @@ -246,15 +319,22 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise= ${spec.minNodeMajor}, this is ${process.versions.node}`; - loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${detail}; a self-downloaded engine would crash this process on load, staying on legacy storage (upgrade Node.js to ${spec.minNodeMajor}+ for unified storage)`); + const spec = specOverride ?? pinnedSpecForThisNode(); + if (!spec) { + const detail = unsupportedNodeDetail(); + loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${detail}; staying on legacy storage`); return { status: "unsupported-node", detail }; } + const unusable = specUnusableReason(spec); + if (unusable) { + // Never register a binding outside the range it declares: the load + // faults uncatchably, so a probe cannot save us — only refusing can. + loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${unusable}; a self-downloaded engine would crash this process on load, staying on legacy storage`); + return { status: "unsupported-node", detail: unusable }; + } const target = platformOverride ?? platformTargetKey(); if (cacheUsable(finalDir(spec.version), target)) { - if (!tryRegisterFromCache(finalDir(spec.version))) { + if (!tryRegisterFromCache(finalDir(spec.version), spec, target)) { // #3135 ⑤/#3818: the copy's bytes were sha-verified when cached, // so "unloadable" means the environment changed under it (major // Node upgrade, truncated cache write, platform drift). A cache @@ -273,6 +353,9 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise= 22.5; upgrade Node.js for the builtin engine (engine tarball mirrors tried in order: npm + npmmirror; set BILI_UPDATE_REGISTRY or update.registry to pin one)"; } -function tryRegisterFromCache(dir: string): boolean { +/** v12 and earlier construct with `(file, { nativeBinding })`; the extra option + * is what lets a JS-only tarball bind the pinned ABI prebuild without the + * `bindings` resolver or an install script. A path (not the addon object) is + * passed so the binding is loaded by v12 at construction time, keeping + * `dlopen` inside the one call the ABI gate has already cleared. */ +type BetterSqlite3CtorWithBinding = new (file: string, options: { nativeBinding: string }) => InstanceType; + +function withNativeBinding(Database: BetterSqlite3CtorWithBinding, addonPath: string): BetterSqlite3Ctor { + const Bound = function BoundDatabase(this: unknown, file: string): InstanceType { + return new Database(file, { nativeBinding: addonPath }); + }; + return Bound as unknown as BetterSqlite3Ctor; +} + +function tryRegisterFromCache(dir: string, spec: EngineBootstrapSpec, target: string): boolean { try { const r = createRequire(path.join(dir, "package.json")); const mod = r("./lib/index.js") as unknown as BetterSqlite3Ctor; - registerSelfDownloadedBetterSqlite3(() => mod); + if (!spec.prebuild) { + registerSelfDownloadedBetterSqlite3(() => mod); + return true; + } + const addonPath = path.join(dir, "prebuilds", `${target}.node`); + if (!existsSync(addonPath)) return false; + registerSelfDownloadedBetterSqlite3(() => withNativeBinding(mod as unknown as BetterSqlite3CtorWithBinding, addonPath)); return true; } catch (err) { loggerLog("debug", `[storage] sqlite engine self-bootstrap: cache registration failed (${String(err)})`); @@ -363,12 +487,18 @@ function tryRegisterFromCache(dir: string): boolean { } } +const PINNED_VERSIONS = new Set(ENGINE_SPECS.map((s) => s.version)); + async function pruneSiblingVersions(keepDir: string): Promise { try { const entries = await fsp.readdir(cacheRoot()); await Promise.all(entries.flatMap((e) => { - if (!e.startsWith("better-sqlite3-") || path.join(cacheRoot(), e) === keepDir) return []; - return [fsp.rm(path.join(cacheRoot(), e), { recursive: true, force: true }).catch(() => {})]; + const dir = path.join(cacheRoot(), e); + if (!e.startsWith("better-sqlite3-") || dir === keepDir) return []; + // Pinned siblings stay: switching Node majors (20 <-> 22) must not + // re-download an artifact this build already knows how to load. + if (PINNED_VERSIONS.has(e.slice("better-sqlite3-".length))) return []; + return [fsp.rm(dir, { recursive: true, force: true }).catch(() => {})]; })); } catch { // pruning is housekeeping only — never fail the bootstrap over it @@ -382,31 +512,30 @@ async function pruneSiblingVersions(keepDir: string): Promise { async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, resolveProxy?: (url: string) => string | undefined): Promise { const root = cacheRoot(); await fsp.mkdir(root, { recursive: true }); - const tmpTgz = path.join(root, `.dl-${randomBytes(6).toString("hex")}.tgz`); const tmpDir = path.join(root, `.wip-${randomBytes(6).toString("hex")}`); try { // #3338 directive 2: walk the mirror list; the FIRST mirror that // yields a sha512-verified tarball wins. Fetch + integrity are // per-mirror; extraction/publishing happens once, mirror-independent. - let tgz: Buffer | null = null; - const attempted: string[] = []; - let lastErr: unknown = null; - for (const base of engineMirrors()) { - const url = `${base}/better-sqlite3/-/better-sqlite3-${spec.version}.tgz`; - attempted.push(base); - try { - tgz = await fetchVerifiedTarball(url, spec, tmpTgz, resolveProxy); - break; - } catch (err) { - lastErr = err; - loggerLog("warn", `[storage] sqlite engine self-bootstrap: mirror ${base} failed (${err instanceof Error ? err.message : String(err)}) — trying next mirror`); - } - } - if (!tgz) { - const reason = lastErr instanceof Error ? lastErr.message : String(lastErr); - throw new Error(`${reason} (tried mirrors: ${attempted.join(", ")})`); + const npmTgz = await fetchFirstMirror( + engineMirrors().map((base) => ({ url: `${base}/better-sqlite3/-/better-sqlite3-${spec.version}.tgz`, label: base })), + spec.sha512Base64, + root, + resolveProxy, + ); + await extractTgzToDir(npmTgz, tmpDir, target, { withPrebuilds: !spec.prebuild }); + if (spec.prebuild) { + const pinned = spec.prebuild.sha512Base64ByTarget[target]; + if (!pinned) throw new Error(`no pinned prebuild digest for ${target}`); + const fileName = `better-sqlite3-v${spec.version}-node-v${spec.prebuild.nodeAbi}-${target}.tar.gz`; + const prebuildTgz = await fetchFirstMirror( + prebuildMirrors().map((base) => ({ url: `${base}/v${spec.version}/${fileName}`, label: base })), + pinned, + root, + resolveProxy, + ); + await extractPrebuildTgz(prebuildTgz, tmpDir, target); } - await extractTgzToDir(tgz, tmpDir, target); const dest = finalDir(spec.version); if (existsSync(dest)) { // rename(2) over a non-empty dir is ENOTEMPTY: a stale final dir would @@ -418,15 +547,41 @@ async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, res } await fsp.rename(tmpDir, dest); } finally { - await fsp.rm(tmpTgz, { force: true }).catch(() => {}); await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => {}); } } +interface MirrorAttempt { + url: string; + /** Shown in the retry log — the mirror root, not the full asset URL. */ + label: string; +} + +async function fetchFirstMirror(attempts: readonly MirrorAttempt[], expectedSha512Base64: string, scratchDir: string, resolveProxy?: (url: string) => string | undefined): Promise { + const tmpTgz = path.join(scratchDir, `.dl-${randomBytes(6).toString("hex")}.tgz`); + const attempted: string[] = []; + let lastErr: unknown = null; + try { + for (const attempt of attempts) { + attempted.push(attempt.label); + try { + return await fetchVerifiedTarball(attempt.url, expectedSha512Base64, tmpTgz, resolveProxy); + } catch (err) { + lastErr = err; + loggerLog("warn", `[storage] sqlite engine self-bootstrap: mirror ${attempt.label} failed (${err instanceof Error ? err.message : String(err)}) — trying next mirror`); + } + } + const reason = lastErr instanceof Error ? lastErr.message : String(lastErr); + throw new Error(`${reason} (tried mirrors: ${attempted.join(", ")})`); + } finally { + await fsp.rm(tmpTgz, { force: true }).catch(() => {}); + } +} + /** One mirror attempt: stream the tarball to `tmpTgz` while hashing; return * the bytes only when the pinned sha512 matches. Throws on timeout / HTTP * error / integrity mismatch so the caller can advance to the next mirror. */ -async function fetchVerifiedTarball(url: string, spec: EngineBootstrapSpec, tmpTgz: string, resolveProxy?: (url: string) => string | undefined): Promise { +async function fetchVerifiedTarball(url: string, expectedSha512Base64: string, tmpTgz: string, resolveProxy?: (url: string) => string | undefined): Promise { const dispatcher = proxyDispatcher(resolveProxy?.(url)); // @types/node types RequestInit.dispatcher as its internal Dispatcher; // cast once here (no `as any`), same pattern as src/update.ts fetchWithEgress. @@ -451,36 +606,61 @@ async function fetchVerifiedTarball(url: string, spec: EngineBootstrapSpec, tmpT await out.close(); } const digest = hash.digest("base64"); - if (digest !== spec.sha512Base64) { - throw new Error(`sha512 mismatch (got ${digest.slice(0, 16)}..., pinned ${spec.sha512Base64.slice(0, 16)}...) — quarantining download`); + if (digest !== expectedSha512Base64) { + throw new Error(`sha512 mismatch (got ${digest.slice(0, 16)}..., pinned ${expectedSha512Base64.slice(0, 16)}...) — quarantining download`); } return await fsp.readFile(tmpTgz); } // --------------------------------------------------------------------------- // Minimal tgz extractor (devDeps are unavailable at runtime — bili ships a -// self-contained bundle). Supports exactly what npm tarballs of the pinned -// package contain: regular files, directories, GNU long names ('L') and PAX -// path records ('x'). Anything else fails loud rather than half-extracting. -// Only lib/, prebuilds/ and package.json are materialized (~1.5 MB of the -// 27 MB unpacked tarball — the C++ sources are build-time only). +// self-contained bundle). Supports exactly what the pinned artifacts contain: +// regular files, directories, GNU long names ('L') and PAX path records ('x'). +// Anything else fails loud rather than half-extracting. // --------------------------------------------------------------------------- const INCLUDE_EXACT = new Set(["package/package.json"]); -const INCLUDE_PREFIXES = ["package/lib/", "package/prebuilds/"]; -function included(name: string): boolean { - return INCLUDE_EXACT.has(name) || INCLUDE_PREFIXES.some((p) => name.startsWith(p)); +function included(name: string, withPrebuilds: boolean): boolean { + if (INCLUDE_EXACT.has(name) || name.startsWith("package/lib/")) return true; + return withPrebuilds && name.startsWith("package/prebuilds/"); +} + +interface ExtractTgzOptions { + /** The npm tarball carries `prebuilds/.node` itself (v13). Off for a + * JS-only tarball whose binding arrives as a separate ABI prebuild (v12). */ + withPrebuilds?: boolean; } -export async function extractTgzToDir(tgz: Buffer, destDir: string, target: string): Promise { +export async function extractTgzToDir(tgz: Buffer, destDir: string, target: string, opts: ExtractTgzOptions = {}): Promise { + const withPrebuilds = opts.withPrebuilds ?? true; + await fsp.mkdir(destDir, { recursive: true }); + await extractTgzEntries(tgz, (name) => (included(name, withPrebuilds) ? destFor(name, destDir) : null)); + const required = withPrebuilds + ? ["package.json", path.join("lib", "index.js"), path.join("prebuilds", `${target}.node`)] + : ["package.json", path.join("lib", "index.js")]; + for (const rel of required) { + if (!existsSync(path.join(destDir, rel))) throw new Error(`tarball missing required file: ${rel}`); + } +} + +/** Extract the ABI-keyed prebuild tarball (a single `build/Release/*.node`) to + * `prebuilds/.node` — the location cacheUsable() and the v12 loader + * both expect, regardless of the source layout. */ +export async function extractPrebuildTgz(tgz: Buffer, destDir: string, target: string): Promise { + const dest = path.join(destDir, "prebuilds", `${target}.node`); + await fsp.mkdir(path.dirname(dest), { recursive: true }); + await extractTgzEntries(tgz, (name) => (name.endsWith(".node") ? dest : null)); + if (!existsSync(dest)) throw new Error(`prebuild tarball missing a .node binding for ${target}`); +} + +async function extractTgzEntries(tgz: Buffer, resolveDest: (name: string) => string | null): Promise { let data: Buffer; try { data = gunzipSync(tgz); } catch (err) { throw new Error(`gunzip failed: ${String(err)}`); } - await fsp.mkdir(destDir, { recursive: true }); let off = 0; let pendingName: string | null = null; while (off + 512 <= data.length) { @@ -504,20 +684,17 @@ export async function extractTgzToDir(tgz: Buffer, destDir: string, target: stri if (paxPath) pendingName = paxPath; continue; } - if (!included(name)) continue; + const dest = resolveDest(name); + if (dest === null) continue; if (typeflag === 0x30 || typeflag === 0x00) { - const dest = destFor(name, destDir); await fsp.mkdir(path.dirname(dest), { recursive: true }); await fsp.writeFile(dest, payload); } else if (typeflag === 0x35) { - await fsp.mkdir(destFor(name, destDir), { recursive: true }); + await fsp.mkdir(dest, { recursive: true }); } else { throw new Error(`unsupported tar entry type 0x${typeflag.toString(16)} for ${name}`); } } - for (const required of ["package.json", path.join("lib", "index.js"), path.join("prebuilds", `${target}.node`)]) { - if (!existsSync(path.join(destDir, required))) throw new Error(`tarball missing required file: ${required}`); - } } function destFor(name: string, destDir: string): string { diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index 6303da848a..ff0947cf62 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -83,11 +83,12 @@ function degradeReason(): string { const last = sidecar.stats.lastError ? shortenError(sidecar.stats.lastError) : "unknown"; return `unified store auto-disabled after ${MAX_CONSECUTIVE_FAILURES} consecutive failures (last: ${last})`; } - // A too-old Node is refused BEFORE any engine load, so the sidecar open ALSO - // fails with the generic no-engine error — surface the real floor reason first, - // or the sidecar-open-error branch below would shadow it (#3894). + // No pinned artifact serves this runtime (below every Node floor, or a Node + // ABI no prebuild targets), so the engine is refused BEFORE any load and the + // sidecar open ALSO fails with the generic no-engine error — surface the real + // reason first, or the sidecar-open-error branch below would shadow it (#3894). if (state.bootstrap?.status === "unsupported-node") { - return `self-downloaded SQLite engine requires a newer Node.js (${state.bootstrap.detail ?? "unknown"})`; + return `no self-downloaded SQLite engine supports this Node.js (${state.bootstrap.detail ?? "unknown"})`; } if (state.sidecarOpenError) { const e = shortenError(state.sidecarOpenError); diff --git a/tests/acp-status-storage-degraded.test.ts b/tests/acp-status-storage-degraded.test.ts index ba6e107b4c..27bcb1a41a 100644 --- a/tests/acp-status-storage-degraded.test.ts +++ b/tests/acp-status-storage-degraded.test.ts @@ -112,18 +112,18 @@ test("#3385: healthy storage stays quiet — no degrade line", () => { } }); -test("#3894: below the artifact Node floor the degrade reason names the Node upgrade, not generic no-engine", () => { +test("#3894: an unsupported Node surfaces the artifact-table reason, not generic no-engine", () => { // Unsupported-node refuses the engine BEFORE any load, so the sidecar open - // ALSO records a "no SQLite engine available" open-error. The floor reason - // must outrank that generic one — otherwise the user sees "better-sqlite3 - // not installed" when the real cause is a too-old Node (#3894 dead-code fix). + // ALSO records a "no SQLite engine available" open-error. That reason must + // outrank the generic one — otherwise the user sees "better-sqlite3 not + // installed" when the real cause is the running Node (#3894 dead-code fix). const base = mkdtempSync(path.join(tmpdir(), "acp-status-3894-")); const snap = isolate(base); let store: SessionStore | null = null; try { setSqliteEnginesUnavailableForTests(true); _resetStorageRuntimeStatusForTest(); - recordEngineBootstrap({ status: "unsupported-node", detail: "better-sqlite3@13.0.3 needs Node.js >= 22, this is 20.19.5" }); + recordEngineBootstrap({ status: "unsupported-node", detail: "no pinned SQLite engine runs on Node.js 21.7.3 (ABI 120)" }); store = new SessionStore({ dir: process.env.BILI_SESSIONS_DIR, enabled: true }); _setStoreForTest(store); @@ -131,8 +131,8 @@ test("#3894: below the artifact Node floor the degrade reason names the Node upg const text = out.text ?? ""; assert.ok(text.includes("STORAGE —"), "the storage line is present"); assert.ok(text.includes("⚠ degraded —"), `the degrade warning line is present (got: ${text.slice(-400)})`); - assert.ok(text.includes("requires a newer Node.js"), `the specific floor reason surfaces (got: ${text.slice(-400)})`); - assert.ok(/Node\.js >= 22/.test(text), "the reason carries the artifact's own Node floor"); + assert.ok(text.includes("no self-downloaded SQLite engine supports this Node.js"), `the specific reason surfaces (got: ${text.slice(-400)})`); + assert.ok(/Node\.js 21\.7\.3/.test(text), "the reason carries the runtime the table could not serve"); } finally { store?.closeQuietly(); _setStoreForTest(null as unknown as SessionStore); diff --git a/tests/storage-engine-bootstrap.test.ts b/tests/storage-engine-bootstrap.test.ts index 7e9cb627ce..67d5c8153c 100644 --- a/tests/storage-engine-bootstrap.test.ts +++ b/tests/storage-engine-bootstrap.test.ts @@ -11,6 +11,12 @@ * re-fetch), integrity mismatch (quarantine + cooldown), * cooldown skip + expiry, offline fail-safe, disabled * knob, legacy-mode short-circuit, unsupported platform. + * NODE BANDS — the #3894 per-major artifact table: Node 20 downloads + * the JS-only 12.9.0 tarball PLUS its ABI-keyed prebuild + * and binds it through `nativeBinding`; a prebuild whose + * ABI does not match the running Node is refused before + * any download; a pinned prebuild-less artifact keeps the + * single-tarball lane and never touches the binary mirror. * EXTRACTOR — include filtering, required-file validation, path * traversal rejection, symlink rejection, non-gzip input. * KNOB — BILI_SQLITE_AUTO_DOWNLOAD > storage.sqliteAutoDownload @@ -43,6 +49,7 @@ import { import { BOOTSTRAP_TIMEOUT_MS, ensureSqliteEngine, + extractPrebuildTgz, extractTgzToDir, nodeMajorSatisfiesFloor, nodeSqliteUpgradeHint, @@ -52,6 +59,7 @@ import { setEngineBootstrapSpecForTests, setEngineMirrorsForTests, setPlatformKeyForTests, + setPrebuildMirrorsForTests, type BootstrapOutcome, } from "../src/storage/engine-bootstrap.ts"; import { resolveSqliteAutoDownload } from "../src/knobs.ts"; @@ -206,6 +214,7 @@ function resetAllSeams(): void { clearSelfDownloadedEngineForTests(); setEngineBootstrapSpecForTests(null); setEngineMirrorsForTests(null); + setPrebuildMirrorsForTests(null); setPlatformKeyForTests(null); setBootstrapTimeoutMsForTests(null); resetEngineBootstrapForTests(); @@ -1016,3 +1025,200 @@ test("bootstrap: at/above the artifact Node floor the gate does NOT fire (#3341 rmrf(cachePrev); } }); + +// --------------------------------------------------------------------------- +// Per-Node-major artifact table (#3894): a JS-only npm tarball (v12 and +// earlier) ships no binding at all, so the bootstrap must ALSO fetch the +// ABI-keyed prebuild and bind it through v12's `nativeBinding` option. Getting +// this wrong is what the whole issue is about: registering a binding whose +// floor/ABI does not match the running Node kills the process uncatchably. +// --------------------------------------------------------------------------- + +/** Stub v12 lib: throws unless the loader injected a `nativeBinding` PATH — + * the injection is the contract under test, and a path (not an addon object) + * keeps `dlopen` inside the constructor v12 itself owns. */ +const FAKE_V12_LIB_INDEX = `const fs = require('node:fs'); +module.exports = class FakeBetterSqlite3V12 { + constructor(p, options) { + if (!options || typeof options.nativeBinding !== 'string') { + throw new Error('FAKE v12: nativeBinding path was not injected'); + } + if (!fs.existsSync(options.nativeBinding)) { + throw new Error('FAKE v12: nativeBinding path does not exist: ' + options.nativeBinding); + } + this.bindingPath = options.nativeBinding; + this.p = p; + this.open = true; + } + exec() {} + prepare() { return { run: () => ({ changes: 1, lastInsertRowid: 1 }), get: () => ({ ok: true }), all: () => [] }; } + transaction(fn) { const w = () => fn(); w.immediate = w; w.deferred = w; w.exclusive = w; return w; } + pragma() {} + close() { this.open = false; } +};`; + +/** JS-only npm tarball shape (v12): package.json + lib/, NO prebuilds/. */ +function fakeJsOnlyPackageTgz(): Buffer { + return buildTgz([ + { name: "package/package.json", content: JSON.stringify({ name: "better-sqlite3", version: "12.9.0" }) }, + { name: "package/lib/index.js", content: FAKE_V12_LIB_INDEX }, + { name: "package/lib/database.js", content: "module.exports = require('./index.js');" }, + { name: "package/deps/sqlite3.c", content: "int unused;" }, + ]); +} + +/** ABI-keyed prebuild shape: one `build/Release/better_sqlite3.node`. */ +function fakePrebuildTgz(): Buffer { + return buildTgz([{ name: "build/Release/better_sqlite3.node", content: Buffer.from("FAKE-ABI-NODE-BINARY") }]); +} + +async function startMockBinaryMirror(body: Buffer, version: string, abi: string, target: string): Promise { + const expected = `/v${version}/better-sqlite3-v${version}-node-v${abi}-${target}.tar.gz`; + let count = 0; + const server = http.createServer((req, res) => { + count++; + if (req.url !== expected) { + res.writeHead(404); + res.end(); + return; + } + res.writeHead(200, { "content-type": "application/octet-stream" }); + res.end(body); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const addr = server.address() as AddressInfo; + return { + base: `http://127.0.0.1:${addr.port}`, + requests: () => count, + close: () => new Promise((r) => { + (server as unknown as { closeAllConnections?: () => void }).closeAllConnections?.(); + server.close(() => r()); + }), + }; +} + +test("extractor: ABI prebuild tarball lands at prebuilds/.node", async () => { + const dest = fs.mkdtempSync(path.join(os.tmpdir(), "bili-boot-extract-")); + try { + const target = platformTargetKey(); + await extractPrebuildTgz(fakePrebuildTgz(), dest, target); + assert.ok(fs.existsSync(path.join(dest, "prebuilds", `${target}.node`)), "the binding is published where cacheUsable() and the v12 loader look"); + } finally { + rmrf(dest); + } +}); + +test("extractor: an ABI prebuild tarball with no .node binding fails loud", async () => { + const dest = fs.mkdtempSync(path.join(os.tmpdir(), "bili-boot-extract-")); + try { + const tgz = buildTgz([{ name: "build/Release/README.md", content: "no binding here" }]); + await assert.rejects(() => extractPrebuildTgz(tgz, dest, platformTargetKey()), /missing a \.node binding/); + } finally { + rmrf(dest); + } +}); + +test("bootstrap: a JS-only artifact downloads its ABI prebuild and binds it via nativeBinding (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const abi = process.versions.modules; + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { nodeAbi: abi, sha512Base64ByTarget: { [target]: sha512B64(preTgz) } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", abi, target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `the JS-only lane must install, got ${JSON.stringify(out)}`); + assert.equal(npmReg!.requests(), 1, "the npm tarball was fetched"); + assert.equal(binReg!.requests(), 1, "the ABI prebuild was fetched from the binary mirror"); + const cached = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-12.9.0"); + assert.ok(fs.existsSync(path.join(cached, "prebuilds", `${target}.node`)), "the binding is cached next to the JS package"); + const o = openSqliteDatabase(":memory:"); + assert.equal(o.engine, "better-sqlite3"); + assert.equal(o.source, "self-downloaded"); + o.db.close(); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: a prebuild whose ABI is not this Node is refused before any download (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { nodeAbi: "999", sha512Base64ByTarget: { [target]: sha512B64(preTgz) } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", "999", target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `an ABI mismatch must be refused, got ${JSON.stringify(out)}`); + assert.match(out.detail ?? "", /no binding for Node ABI \d+/, "the detail names the ABI gap"); + assert.equal(npmReg!.requests(), 0, "neither artifact may be downloaded"); + assert.equal(binReg!.requests(), 0); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-12.9.0")), "nothing materialized into the cache"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: a self-contained artifact never touches the binary mirror (#3894 negative case)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(tgz); + binReg = await startMockBinaryMirror(fakePrebuildTgz(), "13.0.3", process.versions.modules, platformTargetKey()); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `the single-tarball lane must keep working, got ${JSON.stringify(out)}`); + assert.equal(npmReg!.requests(), 1); + assert.equal(binReg!.requests(), 0, "a tarball that carries its own prebuilds must not hit the binary mirror"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index da8927b52d..8a7a7a79dd 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -264,8 +264,8 @@ entries: default: "true" env: BILI_SQLITE_AUTO_DOWNLOAD group: process - desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Below the pinned artifact's own Node floor (22 for better-sqlite3 v13) neither the download nor a cached copy is registered - loading that binding there crashes the process instead of failing gracefully (#3341). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." - desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。在钉死产物自身的 Node 下限(better-sqlite3 v13 为 22)以下,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" + desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER NODE MAJOR, because a binding cannot be loaded outside the range its package declares: Node >= 22 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), Node 20 uses v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding) - so a low Node stays on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." + desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按 Node 大版本分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),Node 20 用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" - key: storage.jsonWriter type: "on|off" default: "on" From 6c185bc603e8050b4c64884d633b97069e9b6a75 Mon Sep 17 00:00:00 2001 From: zypNB123 Date: Tue, 20 Oct 2026 15:01:52 +0800 Subject: [PATCH 05/10] test: stop a stray Windows handle from failing the full-flip suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `test (windows-latest, 24, 3)` leg went red on `#3818 review: reap-shape records ... stay envelope-served across restart` with `EPERM, Permission denied` from `rmrf()` — not an assertion failure. Windows can still hold an ephemeral handle on `index.db`/`-wal` right after a store closes (Defender real-time scan, delayed close), and a bare `rmSync({recursive:true, force:true})` throws instead of waiting. Retry with the same `maxRetries`/`retryDelay` shape the other Windows-touching suites already use (`tests/export-jsonl.test.ts`, `tests/identity-registry.test.ts`), then swallow the residual error: the dir lives under `HERMETIC`, which the `after` hook clears anyway, so a leaked handle can only leak a temp dir — never fail an otherwise green run. Refs #3894. --- tests/storage-full-flip.test.ts | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/storage-full-flip.test.ts b/tests/storage-full-flip.test.ts index ef15357b5b..61b80044b2 100644 --- a/tests/storage-full-flip.test.ts +++ b/tests/storage-full-flip.test.ts @@ -44,8 +44,16 @@ delete process.env.BILI_STORAGE_JSON_WRITER; const PREV_UNIFIED = process.env.BILI_STORAGE_UNIFIED; const PREV_JSON_WRITER = process.env.BILI_STORAGE_JSON_WRITER; +// Windows can still hold an ephemeral handle on index.db/-wal right after a +// store closes (Defender scan, delayed close), so a bare rmSync throws EPERM +// and fails an otherwise green run. Retry, then give up quietly: the dir lives +// under HERMETIC, which the after hook clears anyway. function rmrf(p: string): void { - fs.rmSync(p, { recursive: true, force: true }); + try { + fs.rmSync(p, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + /* Windows stray handle */ + } } function tmpDir(tag: string): string { From 80f52f71a18103924fb4804ae5a27ad0fedc6ceb Mon Sep 17 00:00:00 2001 From: awork Date: Tue, 20 Oct 2026 15:06:53 +0800 Subject: [PATCH 06/10] test: cover pinned-sibling cache retention in the post-publish prune (#3894) Additive review commit (ai: vllm-qwen/qwen3.8-27b): the PINNED_VERSIONS prune-retention behavior introduced by the per-major artifact table had no test row - assert the other pinned version' dir survives a successful download while an unpinned legacy version is still removed. --- tests/storage-engine-bootstrap.test.ts | 46 ++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/tests/storage-engine-bootstrap.test.ts b/tests/storage-engine-bootstrap.test.ts index 67d5c8153c..8cad547e28 100644 --- a/tests/storage-engine-bootstrap.test.ts +++ b/tests/storage-engine-bootstrap.test.ts @@ -1222,3 +1222,49 @@ test("bootstrap: a self-contained artifact never touches the binary mirror (#389 rmrf(cachePrev); } }); + +test("bootstrap: pinned sibling cache dirs survive the prune; unpinned ones are still removed (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const abi = process.versions.modules; + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { nodeAbi: abi, sha512Base64ByTarget: { [target]: sha512B64(preTgz) } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + // Pre-existing siblings before this download publishes: the OTHER pinned + // version's dir must survive the post-publish prune (switching Node + // majors must not force a re-download), an unpinned legacy version must + // still be cleaned up. + const pinnedSibling = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3"); + const staleSibling = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-9.9.9"); + fs.mkdirSync(path.join(pinnedSibling, "lib"), { recursive: true }); + fs.writeFileSync(path.join(pinnedSibling, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "13.0.3" })); + fs.writeFileSync(path.join(pinnedSibling, "lib", "index.js"), FAKE_LIB_INDEX); + fs.mkdirSync(staleSibling, { recursive: true }); + fs.writeFileSync(path.join(staleSibling, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "9.9.9" })); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", abi, target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `the JS-only lane must install, got ${JSON.stringify(out)}`); + assert.ok(fs.existsSync(path.join(pinnedSibling, "package.json")), "the other pinned version's cache dir survives the prune"); + assert.ok(!fs.existsSync(staleSibling), "an unpinned legacy version is still pruned"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); From 78f25119e2d4949430b814b00179ac35c0d6a984 Mon Sep 17 00:00:00 2001 From: zypNB123 Date: Tue, 20 Oct 2026 18:22:56 +0800 Subject: [PATCH 07/10] fix: pick the SQLite artifact by N-API version and cache it per ABI The per-major band table from the previous commit only asked two questions - the package's own engines floor and the ABI of the binary - and that leaves a hole big enough to crash the process: better-sqlite3 v13 declares `engines.node >= 22`, but its binding.gyp compiles with NAPI_VERSION=10, and Node 22.0 through 22.12 ship N-API 9. Those releases satisfy the major floor, pass every existing check, and then fault uncatchably inside process.dlopen when the v13 prebuild is loaded. `minNodeMajor: 22` cannot express this: the gap is in the N-API axis, not the major axis. Serving is therefore decided on three axes now - engines floor, N-API version the binding was compiled against, and the ABI of the binary we would register: * `EngineBootstrapSpec.minNapiVersion` carries the artifact's NAPI_VERSION. v13.0.3 declares 10. A runtime below it is refused with a reason naming the gap instead of being handed an addon it cannot load. * `EnginePrebuild.sha512Base64ByAbi` replaces the single-ABI `nodeAbi`/`sha512Base64ByTarget` pair with an ABI-keyed digest table, so v12.9.0 now serves both Node 20 (ABI 115) and the Node 22.0-22.12 band (ABI 127). Eight new pins, taken from the npmmirror binary mirror. * The cache directory is ABI-keyed (`better-sqlite3-12.9.0-abi115`): a runtime that switches Node minors must fetch the binding that matches it rather than inherit the previous one. The prune keeps every pinned ABI variant, so switching back does not re-download. `pinnedEngineVersionFor()` is exported as a pure selector so the whole band matrix is testable without a download. Verified end to end on real runtimes against the real registry: Node 22.0.0 (ABI 127, N-API 9) now downloads v12.9.0 plus its ABI 127 prebuild, opens a database, runs DDL/DML and survives - the same binary that segfaults when the v13 prebuild is loaded. Node 20.19.5 stays on the ABI 115 lane; Node 22.13+ and 22.14+ take the builtin or the v13 single-tarball lane as before. Fixes #3894. Refs #3341. --- CONFIGURATION.md | 2 +- CONFIGURATION.zh-CN.md | 2 +- src/storage/engine-bootstrap.ts | 112 ++++++++++++++++--------- tests/storage-engine-bootstrap.test.ts | 107 +++++++++++++++++++++-- website/config-reference/process.yaml | 4 +- 5 files changed, 179 insertions(+), 48 deletions(-) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 8c8744e6c2..88a9a5a12d 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,7 +175,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER NODE MAJOR, because a binding cannot be loaded outside the range its package declares: Node >= 22 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), Node 20 uses v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding) - so a low Node stays on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding); a runtime no pinned artifact serves is refused before any load. That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 4cd7236616..4ee7b74a07 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,7 +175,7 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按 Node 大版本分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),Node 20 用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入);任何钉死产物都不服务的运行时会在加载前被拒绝 —— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | diff --git a/src/storage/engine-bootstrap.ts b/src/storage/engine-bootstrap.ts index cf3f6fa8fd..e056b7a0d3 100644 --- a/src/storage/engine-bootstrap.ts +++ b/src/storage/engine-bootstrap.ts @@ -61,10 +61,8 @@ import { proxyDispatcher } from "../upstream-proxy.js"; /** The separate ABI-keyed binding artifact a JS-only npm tarball needs. */ interface EnginePrebuild { - /** Node `process.versions.modules` ABI the binaries were built for (115 = Node 20). */ - nodeAbi: string; - /** sha512 base64 per platform target — pinned in-repo, verified on every download. */ - sha512Base64ByTarget: Readonly>; + /** sha512 base64 per Node ABI (`process.versions.modules`) and platform target - pinned in-repo, verified on every download. */ + sha512Base64ByAbi: Readonly>>>; } interface EngineBootstrapSpec { @@ -73,6 +71,8 @@ interface EngineBootstrapSpec { sha512Base64: string; /** Node major floor copied from the artifact's own `engines.node`; below it the binding is undefined behavior (process crash), not a graceful failure. */ minNodeMajor: number; + /** Node N-API floor the binding was compiled against (its binding.gyp NAPI_VERSION). A runtime below it cannot load the addon: `process.dlopen` faults uncatchably instead of reporting a missing symbol. Absent = no N-API floor. */ + minNapiVersion?: number; /** Present when the npm tarball carries no binding of its own (v12 and earlier). */ prebuild?: EnginePrebuild; } @@ -83,29 +83,45 @@ interface EngineBootstrapSpec { * * Order is preference order: the newest artifact that serves the running Node * wins, so a Node 22+ machine keeps the single-tarball v13 lane even though - * v12 could also serve it. + * v12 could also serve it. Serving is decided on THREE axes, not just the + * major: the package's own engines floor, the N-API version the binding was + * compiled against, and the ABI of the binary we would register. Node + * 22.0-22.12 passes the v13 major floor but only ships N-API 9, so it falls + * through to the v12 ABI 127 band instead of dlopen'ing a N-API 10 addon. */ const ENGINE_SPECS: readonly EngineBootstrapSpec[] = Object.freeze([ { version: "13.0.3", sha512Base64: "RbOBxmLBG8uvFUc15X9+9SFemKcQ0WBuISBVkpuiaUB2qblC8UWlHEjdVVoZ8AdhSwmoEgsiXKfopX0CQxaACQ==", minNodeMajor: 22, + minNapiVersion: 10, }, { version: "12.9.0", sha512Base64: "wqUv4Gm3toFpHDQmaKD4QhZm3g1DjUBI0yzS4UBl6lElUmXFYdTQmmEDpAFa5o8FiFiymURypEnfVHzILKaxqQ==", minNodeMajor: 20, prebuild: { - nodeAbi: "115", - sha512Base64ByTarget: Object.freeze({ - "linux-x64": "fNMKurhPKb0ki9mnZJK9PVazCiRugcdmauo5qGjqQjUGIrdaVWrb4nlLwU7IiyppcfLu8n/qUsQ5CEZ36uoGaw==", - "linux-arm64": "7q7wZe0+5LLQt2YUkocGb41B9TvEMLKIaNI+CiQUQQ2DFBO1RPr3ixj/Gtb+0noMYHjJrqtqKUMO1xMQXnE0iw==", - "linuxmusl-x64": "GCriICDKw7K5dNjAVo8xFpSJXcYXXBAK9RDoeuSUv6rB5IVRP96IwxeynTy4Xc21l1507Efz+I2yy6FW5gKQ/w==", - "linuxmusl-arm64": "2OWfVP1nklnuR8ZfCwozJcLB0ZpOTRHqAkDpsLonEF5uHZMBnOAzGtd5d+nuPU+KaS5czciU0tXZFhsCUw7dbg==", - "darwin-x64": "tW2Vb19+Bh+PLfcGVnrERhc9Mk50Vi8HciIc0nwIJXlPm21xzKL6uEQHMoyKIAxF+vNXxDsUoJyNRg/TgCXjkA==", - "darwin-arm64": "H6jehr7s1D/RSTRplCL++hneLTgpqVzgzjrX/NJCsf0OMBSKNGCd9kMYAZJjMlp0IetS39sG5PTG7AcLCTQKeg==", - "win32-x64": "r+BAeN980mQEdIwDcrNU3iJqzAryuOyfeICFfYHuHVclonTr2L0IlzsaGp2D50KNWUeLQrIhBCFuNoT2+iBDTw==", - "win32-arm64": "jMSa36VrAiLsWFtnwWX+FEs4JWIjHeduVITY44ZcHsqn7BzPcMljL8LdJ0tVVGssV8x9PuqgIqMD5gFmzcYh9w==", + sha512Base64ByAbi: Object.freeze({ + "115": Object.freeze({ + "linux-x64": "fNMKurhPKb0ki9mnZJK9PVazCiRugcdmauo5qGjqQjUGIrdaVWrb4nlLwU7IiyppcfLu8n/qUsQ5CEZ36uoGaw==", + "linux-arm64": "7q7wZe0+5LLQt2YUkocGb41B9TvEMLKIaNI+CiQUQQ2DFBO1RPr3ixj/Gtb+0noMYHjJrqtqKUMO1xMQXnE0iw==", + "linuxmusl-x64": "GCriICDKw7K5dNjAVo8xFpSJXcYXXBAK9RDoeuSUv6rB5IVRP96IwxeynTy4Xc21l1507Efz+I2yy6FW5gKQ/w==", + "linuxmusl-arm64": "2OWfVP1nklnuR8ZfCwozJcLB0ZpOTRHqAkDpsLonEF5uHZMBnOAzGtd5d+nuPU+KaS5czciU0tXZFhsCUw7dbg==", + "darwin-x64": "tW2Vb19+Bh+PLfcGVnrERhc9Mk50Vi8HciIc0nwIJXlPm21xzKL6uEQHMoyKIAxF+vNXxDsUoJyNRg/TgCXjkA==", + "darwin-arm64": "H6jehr7s1D/RSTRplCL++hneLTgpqVzgzjrX/NJCsf0OMBSKNGCd9kMYAZJjMlp0IetS39sG5PTG7AcLCTQKeg==", + "win32-x64": "r+BAeN980mQEdIwDcrNU3iJqzAryuOyfeICFfYHuHVclonTr2L0IlzsaGp2D50KNWUeLQrIhBCFuNoT2+iBDTw==", + "win32-arm64": "jMSa36VrAiLsWFtnwWX+FEs4JWIjHeduVITY44ZcHsqn7BzPcMljL8LdJ0tVVGssV8x9PuqgIqMD5gFmzcYh9w==", + }), + "127": Object.freeze({ + "linux-x64": "jpVKlRCjwmKDIfohRK4Zf/potrKjl2CuvKiWm2SOL1eY7pTRQrGBc1DSEr4OBulYHE6PTl59w7hoAlzNOefG3g==", + "linux-arm64": "ei++r3DQbhwXsP97PHYENHnNd3jrai5dy11SCDZqzmqEm6mjloT1qwrpglVJmW4TLliYBJ8U9bMG/6Y8nbDBCQ==", + "linuxmusl-x64": "eFglLYVSmdQo7ehLh30dOF4H9csX92XIn1oKB3TdtTZ+7TTh7cfViO7TQIC4NPt6Ufbl3cos9y6/pU8FNL1dQg==", + "linuxmusl-arm64": "e72bURjJ4SYseIMY7uW1IkubTqzuKzD6NIVP0InsD8+DJBnRwi8+yIEEcODIK4xsFKSO6tE8TLuvPmetXEk8aA==", + "darwin-x64": "GxXg+WENfyP9DGnvOSMnPge+kDkEZmU8cF7EFOoFSzAFhEwYOBvFB4VTTqhZ6G4y54I7ufkjgkOxZmafdSOXOw==", + "darwin-arm64": "V1mPpKKnv3AvfZb/++CSOx49tmUi4fba4vORsD8GTaJVwvKmveJ/uPqXaEVhYQgDkao5VLiWL6gN2wk9rQXfPw==", + "win32-x64": "7chlb4LP9ETESK6djkGxDq4aDTTByiG1VoUwadJk7JZFhfn2u7vblPCulZaM2C3Ju3jHowc8LBKKQZiH52Uc5g==", + "win32-arm64": "0mD1nLGHG066cUFdIgMTEJfhbIeOgD52+dE0r8LAALarhU5JD0uyKPvpQH/V3rBjvwTOtaQtYqeEFgRj5NUL/g==", + }), }), }, }, @@ -183,12 +199,19 @@ export function nodeMajorSatisfiesFloor(minNodeMajor: number, nodeVersion: strin * halves are load-bearing: the floor is what the package declares, the ABI is * what the binary was actually built against — a JS-only tarball whose * prebuild targets a different ABI loads just as badly as one below its floor. */ -function specUnusableReason(spec: EngineBootstrapSpec, nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules): string | null { +function specUnusableReason(spec: EngineBootstrapSpec, nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules, napiVersion: string = process.versions.napi ?? ""): string | null { if (!nodeMajorSatisfiesFloor(spec.minNodeMajor, nodeVersion)) { return `better-sqlite3@${spec.version} needs Node.js >= ${spec.minNodeMajor}, this is ${nodeVersion}`; } - if (spec.prebuild && spec.prebuild.nodeAbi !== nodeAbi) { - return `better-sqlite3@${spec.version} ships no binding for Node ABI ${nodeAbi} (its prebuild is pinned to ABI ${spec.prebuild.nodeAbi}), this is Node.js ${nodeVersion}`; + if (spec.minNapiVersion !== undefined) { + const napi = Number.parseInt(napiVersion, 10); + if (!Number.isFinite(napi)) return null; + if (napi < spec.minNapiVersion) { + return `better-sqlite3@${spec.version} is built against N-API ${spec.minNapiVersion}, this Node.js ${nodeVersion} ships N-API ${napiVersion}`; + } + } + if (spec.prebuild && !spec.prebuild.sha512Base64ByAbi[nodeAbi]) { + return `better-sqlite3@${spec.version} ships no binding for Node ABI ${nodeAbi} (pinned ABIs: ${Object.keys(spec.prebuild.sha512Base64ByAbi).join(", ")}), this is Node.js ${nodeVersion}`; } return null; } @@ -197,12 +220,19 @@ function pinnedSpecForThisNode(): EngineBootstrapSpec | null { return ENGINE_SPECS.find((s) => specUnusableReason(s) === null) ?? null; } +/** Pure band selector (runtime triple in -> pinned version out, null = nothing + * fits) so tests pin the whole matrix — including the Node 22.0-22.12 band + * that passes v13's major floor but predates N-API 10 — without downloads. */ +export function pinnedEngineVersionFor(nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules, napiVersion: string = process.versions.napi ?? ""): string | null { + return ENGINE_SPECS.find((s) => specUnusableReason(s, nodeVersion, nodeAbi, napiVersion) === null)?.version ?? null; +} + /** Honest "nothing fits" text: names the running Node/ABI and every pinned * band, so the report explains itself instead of blaming the network. */ function unsupportedNodeDetail(nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules): string { const covered = ENGINE_SPECS.map((s) => (s.prebuild - ? `better-sqlite3@${s.version} on Node.js ${s.minNodeMajor}.x (ABI ${s.prebuild.nodeAbi})` - : `better-sqlite3@${s.version} on Node.js >= ${s.minNodeMajor}`)).join(" or "); + ? `better-sqlite3@${s.version} on Node.js ${s.minNodeMajor}.x (ABI ${Object.keys(s.prebuild.sha512Base64ByAbi).join("/")})` + : `better-sqlite3@${s.version} on Node.js >= ${s.minNodeMajor} (N-API >= ${s.minNapiVersion ?? 0})`)).join(" or "); return `no pinned SQLite engine runs on Node.js ${nodeVersion} (ABI ${nodeAbi}) — the pinned artifacts cover ${covered}`; } @@ -228,8 +258,9 @@ function cacheRoot(): string { return path.join(cacheDir(), "sqlite"); } -function finalDir(version: string): string { - return path.join(cacheRoot(), `better-sqlite3-${version}`); +function finalDir(spec: EngineBootstrapSpec, nodeAbi: string = process.versions.modules): string { + const suffix = spec.prebuild ? `-abi${nodeAbi}` : ""; + return path.join(cacheRoot(), `better-sqlite3-${spec.version}${suffix}`); } function cooldownFile(): string { @@ -333,8 +364,9 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise { + await fsp.rm(dest, { recursive: true, force: true }).catch(() => { // best-effort cleanup; the pinned download re-verifies anyway }); } else { @@ -353,7 +385,7 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise s.version)); +const PINNED_DIR_PREFIXES = ENGINE_SPECS.map((s) => `better-sqlite3-${s.version}`); async function pruneSiblingVersions(keepDir: string): Promise { try { @@ -496,8 +529,9 @@ async function pruneSiblingVersions(keepDir: string): Promise { const dir = path.join(cacheRoot(), e); if (!e.startsWith("better-sqlite3-") || dir === keepDir) return []; // Pinned siblings stay: switching Node majors (20 <-> 22) must not - // re-download an artifact this build already knows how to load. - if (PINNED_VERSIONS.has(e.slice("better-sqlite3-".length))) return []; + // re-download an artifact this build already knows how to load. The + // prefix match also keeps every pinned ABI variant of one version. + if (PINNED_DIR_PREFIXES.some((p) => e === p || e.startsWith(`${p}-abi`))) return []; return [fsp.rm(dir, { recursive: true, force: true }).catch(() => {})]; })); } catch { @@ -509,7 +543,7 @@ async function pruneSiblingVersions(keepDir: string): Promise { // Download + verify + extract // --------------------------------------------------------------------------- -async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, resolveProxy?: (url: string) => string | undefined): Promise { +async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, nodeAbi: string, resolveProxy?: (url: string) => string | undefined): Promise { const root = cacheRoot(); await fsp.mkdir(root, { recursive: true }); const tmpDir = path.join(root, `.wip-${randomBytes(6).toString("hex")}`); @@ -525,9 +559,9 @@ async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, res ); await extractTgzToDir(npmTgz, tmpDir, target, { withPrebuilds: !spec.prebuild }); if (spec.prebuild) { - const pinned = spec.prebuild.sha512Base64ByTarget[target]; - if (!pinned) throw new Error(`no pinned prebuild digest for ${target}`); - const fileName = `better-sqlite3-v${spec.version}-node-v${spec.prebuild.nodeAbi}-${target}.tar.gz`; + const pinned = spec.prebuild.sha512Base64ByAbi[nodeAbi]?.[target]; + if (!pinned) throw new Error(`no pinned prebuild digest for ${target} on ABI ${nodeAbi}`); + const fileName = `better-sqlite3-v${spec.version}-node-v${nodeAbi}-${target}.tar.gz`; const prebuildTgz = await fetchFirstMirror( prebuildMirrors().map((base) => ({ url: `${base}/v${spec.version}/${fileName}`, label: base })), pinned, @@ -536,7 +570,7 @@ async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, res ); await extractPrebuildTgz(prebuildTgz, tmpDir, target); } - const dest = finalDir(spec.version); + const dest = finalDir(spec, nodeAbi); if (existsSync(dest)) { // rename(2) over a non-empty dir is ENOTEMPTY: a stale final dir would // fail every attempt and re-arm the 6h cooldown indefinitely. A usable diff --git a/tests/storage-engine-bootstrap.test.ts b/tests/storage-engine-bootstrap.test.ts index 8cad547e28..8fc9c89cfc 100644 --- a/tests/storage-engine-bootstrap.test.ts +++ b/tests/storage-engine-bootstrap.test.ts @@ -17,6 +17,10 @@ * ABI does not match the running Node is refused before * any download; a pinned prebuild-less artifact keeps the * single-tarball lane and never touches the binary mirror. + * A Node that passes the major floor but predates the + * artifact's N-API version falls through to the next + * band; ABI-keyed artifacts cache per ABI so a second + * runtime never inherits the wrong binding. * EXTRACTOR — include filtering, required-file validation, path * traversal rejection, symlink rejection, non-gzip input. * KNOB — BILI_SQLITE_AUTO_DOWNLOAD > storage.sqliteAutoDownload @@ -52,6 +56,7 @@ import { extractPrebuildTgz, extractTgzToDir, nodeMajorSatisfiesFloor, + pinnedEngineVersionFor, nodeSqliteUpgradeHint, platformTargetKey, resetEngineBootstrapForTests, @@ -1132,7 +1137,7 @@ test("bootstrap: a JS-only artifact downloads its ABI prebuild and binds it via version: "12.9.0", sha512Base64: sha512B64(npmTgz), minNodeMajor: 20, - prebuild: { nodeAbi: abi, sha512Base64ByTarget: { [target]: sha512B64(preTgz) } }, + prebuild: { sha512Base64ByAbi: { [abi]: { [target]: sha512B64(preTgz) } } }, }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); @@ -1143,7 +1148,7 @@ test("bootstrap: a JS-only artifact downloads its ABI prebuild and binds it via assert.equal(out.status, "downloaded", `the JS-only lane must install, got ${JSON.stringify(out)}`); assert.equal(npmReg!.requests(), 1, "the npm tarball was fetched"); assert.equal(binReg!.requests(), 1, "the ABI prebuild was fetched from the binary mirror"); - const cached = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-12.9.0"); + const cached = path.join(cachePrev, "billion-context", "sqlite", `better-sqlite3-12.9.0-abi${abi}`); assert.ok(fs.existsSync(path.join(cached, "prebuilds", `${target}.node`)), "the binding is cached next to the JS package"); const o = openSqliteDatabase(":memory:"); assert.equal(o.engine, "better-sqlite3"); @@ -1172,7 +1177,7 @@ test("bootstrap: a prebuild whose ABI is not this Node is refused before any dow version: "12.9.0", sha512Base64: sha512B64(npmTgz), minNodeMajor: 20, - prebuild: { nodeAbi: "999", sha512Base64ByTarget: { [target]: sha512B64(preTgz) } }, + prebuild: { sha512Base64ByAbi: { "999": { [target]: sha512B64(preTgz) } } }, }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); @@ -1184,7 +1189,7 @@ test("bootstrap: a prebuild whose ABI is not this Node is refused before any dow assert.match(out.detail ?? "", /no binding for Node ABI \d+/, "the detail names the ABI gap"); assert.equal(npmReg!.requests(), 0, "neither artifact may be downloaded"); assert.equal(binReg!.requests(), 0); - assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-12.9.0")), "nothing materialized into the cache"); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", `better-sqlite3-12.9.0-abi${process.versions.modules}`)), "nothing materialized into the cache"); assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); }); } finally { @@ -1223,6 +1228,98 @@ test("bootstrap: a self-contained artifact never touches the binary mirror (#389 } }); + +// --------------------------------------------------------------------------- +// N-API floor (#3894 follow-up): a binding compiled against N-API 10 cannot be +// dlopen'd by a runtime that only ships N-API 9 - the load faults uncatchably, +// so the band table must fall through to an artifact that DOES match instead +// of trusting the package's `engines.node` major alone. Node 22.0-22.12 is +// exactly that trap: `>= 22` is satisfied, N-API is still 9. +// --------------------------------------------------------------------------- + +test("pinnedEngineVersionFor: every Node band picks a loadable artifact (#3894)", () => { + assert.equal(pinnedEngineVersionFor("20.19.5", "115", "9"), "12.9.0", "Node 20 -> the ABI 115 JS-only lane"); + assert.equal(pinnedEngineVersionFor("22.0.0", "127", "9"), "12.9.0", "Node 22.0-22.12 passes v13's major floor but predates N-API 10 -> must fall back, not dlopen"); + assert.equal(pinnedEngineVersionFor("22.13.0", "127", "9"), "12.9.0", "the whole pre-N-API-10 22.x band falls back"); + assert.equal(pinnedEngineVersionFor("22.14.0", "127", "10"), "13.0.3", "N-API 10 arrives -> the single-tarball v13 lane"); + assert.equal(pinnedEngineVersionFor("23.11.1", "131", "10"), "13.0.3", "Node 23 is above the floor with N-API 10"); + assert.equal(pinnedEngineVersionFor("24.11.0", "137", "10"), "13.0.3", "Node 24 is above the floor with N-API 10"); + assert.equal(pinnedEngineVersionFor("25.2.1", "141", "10"), "13.0.3", "Node 25 is above the floor with N-API 10"); + assert.equal(pinnedEngineVersionFor("21.7.3", "120", "9"), null, "Node 21 has no pinned band -> honest unsupported-node"); + assert.equal(pinnedEngineVersionFor("19.9.0", "111", "8"), null, "Node 19 is below the engines floor"); + assert.equal(pinnedEngineVersionFor("18.20.8", "108", "8"), null, "Node 18 is below the engines floor"); +}); + +test("bootstrap: an artifact whose N-API floor is above this runtime is refused before any download (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20, minNapiVersion: 99 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(tgz); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `an N-API gap must be refused, got ${JSON.stringify(out)}`); + assert.match(out.detail ?? "", /built against N-API 99/, "the detail names the N-API gap"); + assert.equal(npmReg!.requests(), 0, "the tarball must not be downloaded"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); + }); + } finally { + await npmReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: an ABI-keyed artifact keeps one cache dir per ABI so a second runtime fetches its own binding (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const abi = process.versions.modules; + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + // A cache dir from the SAME version but a DIFFERENT ABI: reusing it would + // hand this runtime the wrong binding, so the bootstrap must not treat it + // as a cache hit. + const wrongAbiDir = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-12.9.0-abi999"); + fs.mkdirSync(path.join(wrongAbiDir, "lib"), { recursive: true }); + fs.mkdirSync(path.join(wrongAbiDir, "prebuilds"), { recursive: true }); + fs.writeFileSync(path.join(wrongAbiDir, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "12.9.0" })); + fs.writeFileSync(path.join(wrongAbiDir, "lib", "index.js"), FAKE_V12_LIB_INDEX); + fs.writeFileSync(path.join(wrongAbiDir, "prebuilds", `${target}.node`), Buffer.from("WRONG-ABI")); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { sha512Base64ByAbi: { [abi]: { [target]: sha512B64(preTgz) } } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", abi, target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `a foreign-ABI cache dir must not be reused, got ${JSON.stringify(out)}`); + assert.equal(npmReg!.requests(), 1, "the JS tarball was fetched into this ABI's own dir"); + const ownDir = path.join(cachePrev, "billion-context", "sqlite", `better-sqlite3-12.9.0-abi${abi}`); + assert.ok(fs.existsSync(path.join(ownDir, "prebuilds", `${target}.node`)), "this runtime's own binding was published"); + assert.ok(fs.existsSync(wrongAbiDir), "the other ABI's cache dir survives the prune"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + test("bootstrap: pinned sibling cache dirs survive the prune; unpinned ones are still removed (#3894)", async () => { const cachePrev = isolateCache(); let npmReg: MockRegistry | null = null; @@ -1237,7 +1334,7 @@ test("bootstrap: pinned sibling cache dirs survive the prune; unpinned ones are version: "12.9.0", sha512Base64: sha512B64(npmTgz), minNodeMajor: 20, - prebuild: { nodeAbi: abi, sha512Base64ByTarget: { [target]: sha512B64(preTgz) } }, + prebuild: { sha512Base64ByAbi: { [abi]: { [target]: sha512B64(preTgz) } } }, }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); // Pre-existing siblings before this download publishes: the OTHER pinned diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 8a7a7a79dd..6ed1e68ed5 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -264,8 +264,8 @@ entries: default: "true" env: BILI_SQLITE_AUTO_DOWNLOAD group: process - desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER NODE MAJOR, because a binding cannot be loaded outside the range its package declares: Node >= 22 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), Node 20 uses v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding) - so a low Node stays on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." - desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按 Node 大版本分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),Node 20 用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" + desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding); a runtime no pinned artifact serves is refused before any load. That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." + desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入);任何钉死产物都不服务的运行时会在加载前被拒绝 —— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" - key: storage.jsonWriter type: "on|off" default: "on" From 3312e1be5c783e43e70725505b4d3c4562446d25 Mon Sep 17 00:00:00 2001 From: awork Date: Tue, 20 Oct 2026 18:50:26 +0800 Subject: [PATCH 08/10] docs(config): drop the duplicated 'refused before any load' clause from the sqliteAutoDownload description (#3894) In-passing nit from review of bb534002b: the rev3 prose stated the refusal clause twice in a row (short form + detailed form) in both languages. Keep the detailed sentence only; regenerate CONFIGURATION.md en/zh from the seed. --- CONFIGURATION.md | 2 +- CONFIGURATION.zh-CN.md | 2 +- website/config-reference/process.yaml | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 88a9a5a12d..e8fa75e23f 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,7 +175,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding); a runtime no pinned artifact serves is refused before any load. That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). Env wins over the file tier. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 4ee7b74a07..dac0ff06d9 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,7 +175,7 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入);任何钉死产物都不服务的运行时会在加载前被拒绝 —— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。环境变量优先于配置文件层。 | diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 6ed1e68ed5..58f74498f1 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -264,8 +264,8 @@ entries: default: "true" env: BILI_SQLITE_AUTO_DOWNLOAD group: process - desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding); a runtime no pinned artifact serves is refused before any load. That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." - desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入);任何钉死产物都不服务的运行时会在加载前被拒绝 —— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" + desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." + desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" - key: storage.jsonWriter type: "on|off" default: "on" From 41c10e068c571f3e93c12338c674c573e43d5f67 Mon Sep 17 00:00:00 2001 From: zypNB123 Date: Tue, 20 Oct 2026 19:47:43 +0800 Subject: [PATCH 09/10] test: skip the solver-backed verif cases when no SMT solver is installed #3853 landed three solver-backed verif tests without the availability gate the other verif suites carry, so on a runner with neither cvc5 nor z3 they fail instead of skipping - `no solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)`. That contradicts the module's own contract (AGENTS.md rule 14: an environment without a solver must skip, never go red) and it is what turned every `test` leg red on this head, on ubuntu and windows alike. Adds the same probe-then-skip the sibling suites use (verif-bug3769, verif-generic-append-stability, verif-symbolic-smoke) to bug3676, bug3691 and the induction step. With a solver present nothing changes; without one the three cases report skipped and the rest of the suite still runs. Verified locally: the 8 verif files report 5 pass / 10 skipped / 0 fail with no solver on PATH, and the probe returns null as expected. --- tests/verif-bug3676-wire-reorder.test.ts | 4 +++- tests/verif-bug3691-oscillation.test.ts | 2 ++ tests/verif-induction-append-stability.test.ts | 5 ++++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/tests/verif-bug3676-wire-reorder.test.ts b/tests/verif-bug3676-wire-reorder.test.ts index de3992289b..e86f5d00b5 100644 --- a/tests/verif-bug3676-wire-reorder.test.ts +++ b/tests/verif-bug3676-wire-reorder.test.ts @@ -101,7 +101,9 @@ function partType(content: SymArr, i: number): Seq<"main"> { return t.t; } -test("bug3676: solver finds the [file,text]→[text,file] part-order reversal on the real wire codec", { timeout: 240_000 }, async () => { +test("bug3676: solver finds the [file,text]→[text,file] part-order reversal on the real wire codec", { timeout: 240_000 }, async (t) => { + const solver = await (async () => { try { return await solveSmt2("(set-logic ALL)(check-sat)", []); } catch { return null; } })(); + if (!solver) return t.skip("no SMT solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)"); const ctx = newCtx(); const ports = { ...makeCounterPorts(ctx), deriveCanonicalMessageId: makeIdPort(ctx) }; diff --git a/tests/verif-bug3691-oscillation.test.ts b/tests/verif-bug3691-oscillation.test.ts index 7a17c3fa9f..c0ae1d6484 100644 --- a/tests/verif-bug3691-oscillation.test.ts +++ b/tests/verif-bug3691-oscillation.test.ts @@ -32,6 +32,8 @@ const mkMsgs = (T: string): { id: string; role: "tool" | "user"; contentType: "t ]; test("bug3691: solver finds a clip-triggering witness on the real truncation code", { timeout: 240_000 }, async (t) => { + const solver = await (async () => { try { return await solveSmt2("(set-logic ALL)(check-sat)", []); } catch { return null; } })(); + if (!solver) return t.skip("no SMT solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)"); const ctx = newCtx(); const T = ctx.String.const("T"); const tok1 = ctx.Real.const("tok1"); diff --git a/tests/verif-induction-append-stability.test.ts b/tests/verif-induction-append-stability.test.ts index 8ffebecc42..8aa2224d3a 100644 --- a/tests/verif-induction-append-stability.test.ts +++ b/tests/verif-induction-append-stability.test.ts @@ -42,7 +42,10 @@ import type { SymVal } from "../verif/symbolic.ts"; const REFS_FILE = "kernel/src/refs.ts"; -test("induction step: appending a distinct-id message cannot touch any observed binding (arbitrary n)", { timeout: 240_000 }, () => { +test("induction step: appending a distinct-id message cannot touch any observed binding (arbitrary n)", { timeout: 240_000 }, async (t) => { + const solver = await (async () => { try { return await solveSmt2("(set-logic ALL)(check-sat)", []); } catch { return null; } })(); + if (!solver) return t.skip("no SMT solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)"); + const ctx = newCtx(); // Abstract prefix state: one observed binding under a FREE key/value. const idP = ctx.String.const("id_p"); From 1ca3bfe0bf4fc00237681613c6af1be7cacc2033 Mon Sep 17 00:00:00 2001 From: zypNB123 Date: Tue, 20 Oct 2026 19:57:47 +0800 Subject: [PATCH 10/10] docs(config): regenerate the index after the seed edits (#3894) --- CHANGELOG.md | 4 ++++ CONFIGURATION.md | 1 + CONFIGURATION.zh-CN.md | 1 + 3 files changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index eca1a2bf80..b06eb1c364 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ All notable changes to **billion-context** are documented here. Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag. ## [Unreleased] + +### 存储 + +- **SQLite 引擎自举按运行时分级钉死产物(#3894,作者 @zypNB123)**:Node ≥ 22 且 N-API ≥ 10 继续用单个 v13 tarball;Node 20 与 22.0–22.12(只有 N-API 9)改用 better-sqlite3 v12.9.0(纯 JS tarball + ABI 匹配的 prebuild,经 npmmirror 二进制镜像获取、sha512 校验、nativeBinding 注入)——低版本 Node 留在统一存储车道而不是降级 legacy。任何钉死产物都不支持的运行时在**加载之前**拒绝(`unsupported-node`,不下载、不注册缓存副本),因为在那里 dlopen 该 binding 会直接崩溃进程(win32 上 1485 次连续 0xC0000005 崩溃环的根因);同版本多 ABI 缓存共存,切换 Node 大版本不重下。 - 存储后端契约第一步(#3931, #3388):统一存储抽出操作级异步 `StorageBackend` 契约(`src/storage/backend.ts`),sqlite 经注册表以默认后端身份重新接入(Promise 包装适配器,零行为/零 schema 变化);`mysql` 为已注册占位 —— 选中时启动提示一行并继续用 sqlite,直开后端的路径(export)按文档化错误失败。新配置 `storage.backend`(env `BILI_STORAGE_BACKEND`,缺省 sqlite)。`--jsonl` 导出渲染器改吃契约(接缝证真);契约测试套件对任意后端实例跑同一断言,第二步 mysql 落地时原样复用。 ### 存储收尾九项一车(#3901,修复 #3463/#3762/#3841/#3798/#3633;#3723 落地) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index e8fa75e23f..86ca56af85 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,6 +175,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index dac0ff06d9..f933061232 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,6 +175,7 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择,可插拔存储计划第一步(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约,sqlite 为默认且是当前唯一真实现(现有 UnifiedStore 套一层 Promise 包装适配器 —— 零行为变化、零 schema 变化)。mysql 是已注册占位:该名字能通过校验(真实后端随第二步落地 —— mysql2 客户端,走可选依赖+自下载纪律 —— 期间配置不会在 unknown/unimplemented 之间来回弹),但选中它只会在 lane 启动时打一行提示并继续用 sqlite;直接开后端的路径(export)按文档化的错误失败。未知名字同样回退 sqlite 并提示。每个 lane 启动时解析一次。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 |