diff --git a/CHANGELOG.md b/CHANGELOG.md index 610a941968..499c901fe3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,9 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ## [Unreleased] +### 存储 + +- **SQLite 引擎自举按运行时分级钉死产物(#3894,作者 @zypNB123)**:Node ≥ 22 且 N-API ≥ 10 继续用单个 v13 tarball;Node 20 与 22.0–22.12(只有 N-API 9)改用 better-sqlite3 v12.9.0(纯 JS tarball + ABI 匹配的 prebuild,经 npmmirror 二进制镜像获取、sha512 校验、nativeBinding 注入)——低版本 Node 留在统一存储车道而不是降级 legacy。任何钉死产物都不支持的运行时在**加载之前**拒绝(`unsupported-node`,不下载、不注册缓存副本),因为在那里 dlopen 该 binding 会直接崩溃进程(win32 上 1485 次连续 0xC0000005 崩溃环的根因);同版本多 ABI 缓存共存,切换 Node 大版本不重下。 ### 存储后端热路径接线(#3388 第三步 / #3931) - `storage.backend="mysql"` 接进代理热路径:保存经异步影子 facade 进串行后端队列(水位 delta、#3841 收割台账、digest 采样、三振自禁用,与 sqlite 路径逐条 lockstep);sqlite 保持零 await 同步快车道,行为逐字节不变。 diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 4d6da96800..be710a94c8 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,6 +175,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot. | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is FULLY WIRED (steps 2+3): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path is wired through the async shadow facade: saves enqueue onto a serial backend queue (watermark deltas, orphan/reap journal, digest sampling — sqlite keeps its zero-await sync fast path unchanged), SYNC reads ride the JSON envelope (the sidecar attaches asynchronously via ready(); viewFor/importRecord sync facades miss by design), replicas/ext/GC/export speak the async twins. writerOff stays sqlite-only (the envelope is the mysql sync read path). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. | | `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (hot path included — the async shadow queue). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | @@ -1895,7 +1897,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_PERSIST_DEBOUNCE_MS` | Debounce window for persistence writes to disk, in ms (default `500`). | | `BILI_PERSIST_TAIL_TOKENS` | Token budget for the persisted conversation snapshot (#401). The disk record stores the **folded view** (block summaries in place of compressed ranges) truncated to the newest messages within this budget — never the raw full history. Default `16384`; `0` disables message persistence entirely (block summaries and compressed originals still persist; `bili export` falls back to block-only rendering). Live in-memory sessions are unaffected — `bili export` of a live session is always complete. | | `BILI_PERSIST_ZSTD` | Set `1`/`true` to enable zstd compression of session files (#1080, owner decision: **default off** — plain JSON maximizes recoverability: jq/grep-debuggable, and immune to the downgrade tail risk). When enabled, every session file that actually shrinks is written as `BILIZSTD1` — a small header (magic + format version + mode byte) over the JSON body, zstd-compressed on Node ≥ 22.15 and stored raw on older runtimes; readers accept both bodies (native zstd, else the bundled WASM fallback), so files stay readable across runtimes and versions. Payloads the framing cannot shrink (tiny sessions) and key-less raw bodies stay bare JSON on disk. Independent of `BILI_ENCRYPTION_KEY` — with both active, the compression choice is recorded in the mode byte inside `BILIENC1`. Existing plain-JSON files are NEVER rewritten at boot (downgrade safety — a mass re-encode would make rolling back to an older bili read its own writes as "corrupt"); they convert organically on their next save. Note: once a session has been saved as `BILIZSTD1`, an OLDER bili (< 0.1.135) cannot read it — this caveat applies only to deployments that opted in; unset or any other value keeps plain JSON. | -| `BILI_STORAGE_UNIFIED` | Tiered storage-migration switch (#2671). Config-file tier: `{"storage":{"unified":"legacy|shadow|full"}}` selects the same modes (strict spellings) and applies whenever the env is unset; `BILI_STORAGE_UNIFIED` (which also accepts `1`/`true` = shadow and `unified` = full) always wins over the file. The mode is resolved once per lane boot — hot edits to the file take effect on the next restart. Set `1`/`true` to enable the **unified SQLite storage dry-run** (Phase 1). Every session write is double-ingested into `/index.db` (engine resolution order since #2833: built-in `node:sqlite` when the runtime provides it (Node >= 22.5) -> an already-installed host `better-sqlite3` native module -> a previously self-downloaded cached copy -> a one-time self-download of the pinned `better-sqlite3` prebuilt tarball from the npm registry (~11 MB, sha512-verified, gated by `storage.sqliteAutoDownload`, unsupported platforms never download) -> none, in which case the sidecar stays legacy-only per the #2825 degrade contract; engine acquisition never depends on the host's install-time build-script approval) and the unified replay is digest-reconciled against the legacy load view — a mismatch logs `[storage] unified shadow DIGEST DRIFT …` and never blocks or alters the write path (legacy files stay the single source of truth). Set `full`/`unified` to enable the **drop-in read path** (Phase 1.5): `loadSync`/`boot`/`loadAll`/`loadStateForSearch` go unified-first — a legacy-file session is organically imported into the index on first open (`[storage] unified import on first open: …`) and later loads are served from `index.db`, with automatic fallback to the legacy file on any unified failure. Legacy files keep being double-written, so **rollback = unset the knob** (env and/or file value). Hard encryption opt-out: while `BILI_ENCRYPTION_KEY` is active the whole sidecar refuses to open (index.db would hold cleartext ingest payloads; a `unified sidecar DISABLED` warning is logged and the store stays legacy-only). Enumeration/GC and the web session list remain file-backed; `removeLegacyFile` now also deletes the index rows and releases CAS refcounts. Digests treat the last-writer `biliVersion` stamp and wall-clock timestamps as environment (`__VER__`/`__TS__`), everything else must match byte-for-byte. The shadow self-disables after 3 consecutive internal failures (in full mode that also flips reads back to legacy-only). **Replica/checkpoint REST (Phase 2)**: on any unified lane (shadow or full), `POST /__bili/replicas {sessionId, label?, pinned?}` freezes the session's persisted state as one content-addressed checkpoint blob + row (the resident session is flushed first), `GET /__bili/replicas[?sessionId=]` lists them, `GET /__bili/replicas/` returns detail + byte length, `POST /__bili/replicas//restore {sessionId?}` materializes the checkpoint as a NEW first-class session through the normal dual write path (JSON + unified ingest; `meta.restoredFrom`/`restoredCheckpoint` record provenance), and `DELETE /__bili/replicas/` drops it. Pinned checkpoints hold a CAS refcount and survive `deleteSession` GC (the whole point: a copy taken before a destructive experiment outlives it); explicit DELETE still removes even pinned ones. Disk expectation: shadow/full roughly doubles unique-content bytes (every session has a JSON file AND index.db rows sharing a content CAS with no dedup across the two stores) and there is no auto-VACUUM — size the sessions dir accordingly while the migration runs. Legacy-only lanes answer 501 with the knob hint. Routes are loopback-only like the rest of `__bili`. Defaults to `full` when unset (#3818). | +| `BILI_STORAGE_UNIFIED` | Tiered storage-migration switch (#2671). Config-file tier: `{"storage":{"unified":"legacy|shadow|full"}}` selects the same modes (strict spellings) and applies whenever the env is unset; `BILI_STORAGE_UNIFIED` (which also accepts `1`/`true` = shadow and `unified` = full) always wins over the file. The mode is resolved once per lane boot — hot edits to the file take effect on the next restart. Set `1`/`true` to enable the **unified SQLite storage dry-run** (Phase 1). Every session write is double-ingested into `/index.db` (engine resolution order since #2833: built-in `node:sqlite` when the runtime provides it (Node >= 22.5) -> an already-installed host `better-sqlite3` native module -> a previously self-downloaded cached copy -> a one-time self-download of the pinned `better-sqlite3` prebuilt tarball from the npm registry (~11 MB, sha512-verified, gated by `storage.sqliteAutoDownload`, unsupported platforms never download; engine artifacts are pinned per Node major (Node >= 22: better-sqlite3 v13; Node 20: v12.9.0 plus its ABI-keyed prebuild), and a runtime no pinned artifact serves is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process (#3341, #3894)) -> none, in which case the sidecar stays legacy-only per the #2825 degrade contract; engine acquisition never depends on the host's install-time build-script approval) and the unified replay is digest-reconciled against the legacy load view — a mismatch logs `[storage] unified shadow DIGEST DRIFT …` and never blocks or alters the write path (legacy files stay the single source of truth). Set `full`/`unified` to enable the **drop-in read path** (Phase 1.5): `loadSync`/`boot`/`loadAll`/`loadStateForSearch` go unified-first — a legacy-file session is organically imported into the index on first open (`[storage] unified import on first open: …`) and later loads are served from `index.db`, with automatic fallback to the legacy file on any unified failure. Legacy files keep being double-written, so **rollback = unset the knob** (env and/or file value). Hard encryption opt-out: while `BILI_ENCRYPTION_KEY` is active the whole sidecar refuses to open (index.db would hold cleartext ingest payloads; a `unified sidecar DISABLED` warning is logged and the store stays legacy-only). Enumeration/GC and the web session list remain file-backed; `removeLegacyFile` now also deletes the index rows and releases CAS refcounts. Digests treat the last-writer `biliVersion` stamp and wall-clock timestamps as environment (`__VER__`/`__TS__`), everything else must match byte-for-byte. The shadow self-disables after 3 consecutive internal failures (in full mode that also flips reads back to legacy-only). **Replica/checkpoint REST (Phase 2)**: on any unified lane (shadow or full), `POST /__bili/replicas {sessionId, label?, pinned?}` freezes the session's persisted state as one content-addressed checkpoint blob + row (the resident session is flushed first), `GET /__bili/replicas[?sessionId=]` lists them, `GET /__bili/replicas/` returns detail + byte length, `POST /__bili/replicas//restore {sessionId?}` materializes the checkpoint as a NEW first-class session through the normal dual write path (JSON + unified ingest; `meta.restoredFrom`/`restoredCheckpoint` record provenance), and `DELETE /__bili/replicas/` drops it. Pinned checkpoints hold a CAS refcount and survive `deleteSession` GC (the whole point: a copy taken before a destructive experiment outlives it); explicit DELETE still removes even pinned ones. Disk expectation: shadow/full roughly doubles unique-content bytes (every session has a JSON file AND index.db rows sharing a content CAS with no dedup across the two stores) and there is no auto-VACUUM — size the sessions dir accordingly while the migration runs. Legacy-only lanes answer 501 with the knob hint. Routes are loopback-only like the rest of `__bili`. Defaults to `full` when unset (#3818). | | `BILI_PERSIST_EPERM_ALERT_THRESHOLD` | N consecutive persist write failures (`EPERM`/`EBUSY`/`EACCES`) on one session before the one-time "add this dir to antivirus exclusions" alert fires (default `5`). Windows only. See [Windows: exclude the sessions dir](#windows-exclude-the-sessions-dir-from-antivirus-362). | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | Re-alert window for the persist EPERM alert, in ms. `0` (default) = alert once then stay silent; `>0` = re-alert at most every that many ms while the failures continue. | | `BILI_TUNNEL_ALLOWED_HOSTS` | `/bili/` tunnel admission for **remote clients** (#409): comma-separated `host` or `host:port` entries that unlock loopback/private destinations (e.g. a LAN relay or the machine's own sglang) for non-loopback clients. The proxy itself and link-local/metadata addresses are always denied; local (loopback) clients always pass. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 0d26a0a739..5ae78cbb7f 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,6 +175,8 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择,可插拔存储计划第一步(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约,sqlite 为默认且是当前唯一真实现(现有 UnifiedStore 套一层 Promise 包装适配器 —— 零行为变化、零 schema 变化)。mysql 是已注册占位:该名字能通过校验(真实后端随第二步落地 —— mysql2 客户端,走可选依赖+自下载纪律 —— 期间配置不会在 unknown/unimplemented 之间来回弹),但选中它只会在 lane 启动时打一行提示并继续用 sqlite;直接开后端的路径(export)按文档化的错误失败。未知名字同样回退 sqlite 并提示。每个 lane 启动时解析一次。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已完整接线(第二+三步):openBackend("mysql", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径经异步影子 facade 接线:保存进串行后端队列(水位 delta、孤儿/收割台账、digest 采样 —— sqlite 保持零 await 同步快车道不变),同步读走 JSON 信封(sidecar 经 ready() 异步挂载;viewFor/importRecord 同步面按设计 miss),replicas/ext/GC/export 走异步孪生。writerOff 仅 sqlite(信封就是 mysql 的同步读路径)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。 | | `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | storage.backend="mysql" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(含热路径 —— 异步影子队列)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | @@ -1892,7 +1894,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_PERSIST_DEBOUNCE_MS` | 持久化写盘的防抖窗口(毫秒,默认 `500`)。 | | `BILI_PERSIST_TAIL_TOKENS` | 持久化会话快照的 token 预算(#401)。盘上记录的是**折叠视图**(压缩范围以块摘要替代)并截断到该预算内的最新消息 —— 不再存全量原始历史。默认 `16384`;设 `0` 彻底不持久化消息(块摘要与压缩原件仍会持久化,`bili export` 退回块级渲染)。活会话内存不受影响 —— 活会话的 `bili export` 始终完整。 | | `BILI_PERSIST_ZSTD` | 设为 `1`/`true` 启用会话文件的 zstd 压缩(#1080,owner 决定:**默认关闭**——纯 JSON 可恢复性最强:可用 jq/grep 调试,且无降级尾部风险)。启用后,每个确实能压缩变小的会话文件均以 `BILIZSTD1` 格式写入——即在 JSON 主体之上附加一个小头部(魔数 + 格式版本 + 模式字节),在 Node ≥ 22.15 上以 zstd 压缩存储,在旧版运行时上则原样存储;读取端兼容两种主体格式(优先使用原生 zstd,否则回退至内置的 WASM 实现),因此文件在不同运行时和版本间均可正常读取。压不小的小会话与无密钥的原始主体以裸 JSON 落盘。与 `BILI_ENCRYPTION_KEY` 相互独立——两者同时生效时,压缩方式会记录在 `BILIENC1` 内部的模式字节中。已有的纯 JSON 文件**永不在启动时改写**(降级安全——批量重编码会让回退到旧版 bili 时把自己的写入当成“损坏文件”);它们在其下一次保存时自然转换。注意:一旦会话已保存为 `BILIZSTD1`,旧版 bili(< 0.1.135)无法读取——该限制仅对显式启用的部署生效;未设置或其他值均保持纯 JSON。 | -| `BILI_STORAGE_UNIFIED` | 存储迁移分层开关(#2671)。配置文件层:`{"storage":{"unified":"legacy|shadow|full"}}` 选用相同模式(严格拼写),在环境变量未设时生效;`BILI_STORAGE_UNIFIED`(额外接受 `1`/`true`=shadow、`unified`=full)始终优先于文件。模式在每个 lane 启动时解析一次——文件热改动在下次重启生效。设 `1`/`true` 启用**统一 SQLite 存储干跑**(Phase 1):每次会话写入都会双写到 `/index.db`(引擎解析顺序自 #2833 起:运行时提供内置 `node:sqlite` 时优先(Node >= 22.5)-> 宿主已装的 `better-sqlite3` 原生模块 -> 先前自下载的缓存副本 -> 一次性自下载钉死的 `better-sqlite3` 预编译 tarball(约 11 MB,sha512 校验,受 `storage.sqliteAutoDownload` 控制,不支持的平台永不下载)-> 无引擎则按 #2825 降级契约保持仅 legacy;引擎获取永不依赖宿主安装期的构建脚本审批),并将统一层回放与旧格式加载视图做 digest 对账——不匹配时记录 `[storage] unified shadow DIGEST DRIFT …`,绝不阻塞或改写主写路径(旧格式文件仍是唯一事实源)。设 `full`/`unified` 启用**平替读路径**(Phase 1.5):`loadSync`/`boot`/`loadAll`/`loadStateForSearch` 改为 unified-first——首次打开旧文件会话时有机导入索引(`[storage] unified import on first open: …`),后续加载直接由 `index.db` 提供,任意统一层失败自动回退旧文件;旧格式文件照常双写,**回滚 = 去掉开关(环境变量与/或文件值)**。加密硬互斥:`BILI_ENCRYPTION_KEY` 激活时整个 sidecar 拒绝开启(index.db 明文存 ingest 载荷,记录 `unified sidecar DISABLED` 警告后保持 legacy-only)。枚举/GC 与 web 列表仍以文件为基底;`removeLegacyFile` 同步删除索引行并释放 CAS 引用计数。digest 把末次写入者版本戳 `biliVersion` 与墙钟时间戳视为环境(`__VER__`/`__TS__`),其余字节必须逐一致。连续 3 次内部失败后自动停用(full 模式下停用即回退 legacy 读)。**副本/检查点 REST(Phase 2)**:任意统一层 lane(shadow 或 full)上,`POST /__bili/replicas {sessionId, label?, pinned?}` 把会话持久化状态冻结为一条内容寻址的 checkpoint blob+行(先强制刷新驻留会话),`GET /__bili/replicas[?sessionId=]` 列出副本,`GET /__bili/replicas/` 返回详情+字节数,`POST /__bili/replicas//restore {sessionId?}` 把检查点物化为全新的一等会话(走正常双写路径:JSON+统一层 ingest;`meta.restoredFrom`/`restoredCheckpoint` 记录来源),`DELETE /__bili/replicas/` 删除。pinned 副本持有 CAS 引用计数,可在 `deleteSession` GC 中存活(本功能的意义:破坏性实验前抓的副本活得过实验本身);显式 DELETE 仍可删 pinned。legacy-only lane 返回 501 并附开关提示。路由与 `__bili` 其余部分一样仅限 loopback。未设置时默认 `full`(#3818)。 | +| `BILI_STORAGE_UNIFIED` | 存储迁移分层开关(#2671)。配置文件层:`{"storage":{"unified":"legacy|shadow|full"}}` 选用相同模式(严格拼写),在环境变量未设时生效;`BILI_STORAGE_UNIFIED`(额外接受 `1`/`true`=shadow、`unified`=full)始终优先于文件。模式在每个 lane 启动时解析一次——文件热改动在下次重启生效。设 `1`/`true` 启用**统一 SQLite 存储干跑**(Phase 1):每次会话写入都会双写到 `/index.db`(引擎解析顺序自 #2833 起:运行时提供内置 `node:sqlite` 时优先(Node >= 22.5)-> 宿主已装的 `better-sqlite3` 原生模块 -> 先前自下载的缓存副本 -> 一次性自下载钉死的 `better-sqlite3` 预编译 tarball(约 11 MB,sha512 校验,受 `storage.sqliteAutoDownload` 控制,不支持的平台永不下载;引擎产物按 Node 大版本分别钉死(Node >= 22 用 better-sqlite3 v13;Node 20 用 v12.9.0 及其 ABI 匹配的 prebuild),任何钉死产物都不支持的运行时会在加载之前被拒绝,下载与缓存副本注册都不进行,因为在那里加载该 binding 会让进程崩溃(#3341、#3894))-> 无引擎则按 #2825 降级契约保持仅 legacy;引擎获取永不依赖宿主安装期的构建脚本审批),并将统一层回放与旧格式加载视图做 digest 对账——不匹配时记录 `[storage] unified shadow DIGEST DRIFT …`,绝不阻塞或改写主写路径(旧格式文件仍是唯一事实源)。设 `full`/`unified` 启用**平替读路径**(Phase 1.5):`loadSync`/`boot`/`loadAll`/`loadStateForSearch` 改为 unified-first——首次打开旧文件会话时有机导入索引(`[storage] unified import on first open: …`),后续加载直接由 `index.db` 提供,任意统一层失败自动回退旧文件;旧格式文件照常双写,**回滚 = 去掉开关(环境变量与/或文件值)**。加密硬互斥:`BILI_ENCRYPTION_KEY` 激活时整个 sidecar 拒绝开启(index.db 明文存 ingest 载荷,记录 `unified sidecar DISABLED` 警告后保持 legacy-only)。枚举/GC 与 web 列表仍以文件为基底;`removeLegacyFile` 同步删除索引行并释放 CAS 引用计数。digest 把末次写入者版本戳 `biliVersion` 与墙钟时间戳视为环境(`__VER__`/`__TS__`),其余字节必须逐一致。连续 3 次内部失败后自动停用(full 模式下停用即回退 legacy 读)。**副本/检查点 REST(Phase 2)**:任意统一层 lane(shadow 或 full)上,`POST /__bili/replicas {sessionId, label?, pinned?}` 把会话持久化状态冻结为一条内容寻址的 checkpoint blob+行(先强制刷新驻留会话),`GET /__bili/replicas[?sessionId=]` 列出副本,`GET /__bili/replicas/` 返回详情+字节数,`POST /__bili/replicas//restore {sessionId?}` 把检查点物化为全新的一等会话(走正常双写路径:JSON+统一层 ingest;`meta.restoredFrom`/`restoredCheckpoint` 记录来源),`DELETE /__bili/replicas/` 删除。pinned 副本持有 CAS 引用计数,可在 `deleteSession` GC 中存活(本功能的意义:破坏性实验前抓的副本活得过实验本身);显式 DELETE 仍可删 pinned。legacy-only lane 返回 501 并附开关提示。路由与 `__bili` 其余部分一样仅限 loopback。未设置时默认 `full`(#3818)。 | | `BILI_PERSIST_EPERM_ALERT_THRESHOLD` | 同一会话连续 N 次持久化写失败(`EPERM`/`EBUSY`/`EACCES`)后触发一次性「把该目录加入杀软排除项」告警的阈值(默认 `5`)。仅 Windows。见下文「Windows:把会话目录加入杀软排除项」章节。 | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | persist EPERM 告警的重复窗口(毫秒)。`0`(默认)= 只告警一次后静默;`>0` = 失败持续期间最多每这么久重复告警一次。 | | `BILI_MAX_SESSIONS` | 内存中最多保留的会话数(默认 `256`;LRU 淘汰 —— 磁盘是事实源)。 | diff --git a/src/storage/engine-bootstrap.ts b/src/storage/engine-bootstrap.ts index dccb15c25a..e056b7a0d3 100644 --- a/src/storage/engine-bootstrap.ts +++ b/src/storage/engine-bootstrap.ts @@ -12,10 +12,30 @@ * contract). This module owns ONLY the download/cache step; driver.ts owns * resolution order and opening. * - * Artifact: better-sqlite3 v13 ships N-API prebuilds for all eight - * platform/arch combinations INSIDE its npm tarball (no install script, no - * per-ABI split — N-API is ABI-stable across Node versions). We pin version + - * sha512 in-repo; bumping either is a human-visible PR. Download egress honors + * Artifacts are pinned PER NODE MAJOR, because the binding is not portable + * below the floor the package declares (#3341, #3894). better-sqlite3 v13 + * ships N-API prebuilds for all eight platform/arch combinations INSIDE its + * npm tarball, but it declares `engines: node >= 22` and compiles them for + * N-API 10, while Node 20 only provides N-API 9 — loading that binding on + * Node 20 is undefined behavior rather than a clean failure: the win32-x64 + * prebuild faults inside `process.dlopen` (access violation, uncatchable) and + * the linux-x64 one faults on the first `new Database()`. Node 20 therefore + * gets the last 12.x line, which declares `engines: 20.x || 22.x || ...` and + * ships no binding of its own: its npm tarball is pure JS and the native half + * is an ABI-keyed prebuild (`better-sqlite3-v-node-v115-.tar.gz`) + * published on the GitHub release / npmmirror binary mirrors. v12's + * `lib/database.js` accepts an injected `nativeBinding`, so the two halves + * compose without `bindings`, `prebuild-install` or any install script — the + * #2833 contract holds: pinned artifacts, sha512 verified on every download, + * no host build approval. Downgrading such users to legacy JSON storage would + * be the wrong answer: legacy is scheduled for removal, so "low Node" must + * stay on the unified lane with a lower artifact, not fall off it. + * + * A spec is eligible only when its Node floor AND (when it has one) its + * prebuild ABI match the running Node. When nothing is eligible the bootstrap + * reports `unsupported-node` instead of registering a binding that would crash + * the process. We pin version + sha512 + floor + per-target prebuild digests + * in-repo; bumping any of them is a human-visible PR. Download egress honors * the same proxy decision chain as model traffic (#609) and the configured * registry mirror (BILI_UPDATE_REGISTRY / update.registry). * @@ -39,17 +59,73 @@ import { fileStorageAutoDownload, fileStorageUnified, resolveStorageMode, resolv import { normalizeRegistryBase } from "../update.js"; import { proxyDispatcher } from "../upstream-proxy.js"; +/** The separate ABI-keyed binding artifact a JS-only npm tarball needs. */ +interface EnginePrebuild { + /** sha512 base64 per Node ABI (`process.versions.modules`) and platform target - pinned in-repo, verified on every download. */ + sha512Base64ByAbi: Readonly>>>; +} + interface EngineBootstrapSpec { version: string; /** sha512 base64 digest of the registry tarball — pinned in-repo, verified on every download. */ sha512Base64: string; + /** Node major floor copied from the artifact's own `engines.node`; below it the binding is undefined behavior (process crash), not a graceful failure. */ + minNodeMajor: number; + /** Node N-API floor the binding was compiled against (its binding.gyp NAPI_VERSION). A runtime below it cannot load the addon: `process.dlopen` faults uncatchably instead of reporting a missing symbol. Absent = no N-API floor. */ + minNapiVersion?: number; + /** Present when the npm tarball carries no binding of its own (v12 and earlier). */ + prebuild?: EnginePrebuild; } -/** Pinned engine artifact (#2833). Bump version AND digest together in one PR. */ -const DEFAULT_ENGINE_SPEC: Readonly = Object.freeze({ - version: "13.0.3", - sha512Base64: "RbOBxmLBG8uvFUc15X9+9SFemKcQ0WBuISBVkpuiaUB2qblC8UWlHEjdWVoZ8AdhSwmoEgsiXKfopX0CQxaACQ==", -}); +/** + * Pinned engine artifacts (#2833), one per Node major band. Bump version, + * digests, floor and ABI together in one PR. + * + * Order is preference order: the newest artifact that serves the running Node + * wins, so a Node 22+ machine keeps the single-tarball v13 lane even though + * v12 could also serve it. Serving is decided on THREE axes, not just the + * major: the package's own engines floor, the N-API version the binding was + * compiled against, and the ABI of the binary we would register. Node + * 22.0-22.12 passes the v13 major floor but only ships N-API 9, so it falls + * through to the v12 ABI 127 band instead of dlopen'ing a N-API 10 addon. + */ +const ENGINE_SPECS: readonly EngineBootstrapSpec[] = Object.freeze([ + { + version: "13.0.3", + sha512Base64: "RbOBxmLBG8uvFUc15X9+9SFemKcQ0WBuISBVkpuiaUB2qblC8UWlHEjdVVoZ8AdhSwmoEgsiXKfopX0CQxaACQ==", + minNodeMajor: 22, + minNapiVersion: 10, + }, + { + version: "12.9.0", + sha512Base64: "wqUv4Gm3toFpHDQmaKD4QhZm3g1DjUBI0yzS4UBl6lElUmXFYdTQmmEDpAFa5o8FiFiymURypEnfVHzILKaxqQ==", + minNodeMajor: 20, + prebuild: { + sha512Base64ByAbi: Object.freeze({ + "115": Object.freeze({ + "linux-x64": "fNMKurhPKb0ki9mnZJK9PVazCiRugcdmauo5qGjqQjUGIrdaVWrb4nlLwU7IiyppcfLu8n/qUsQ5CEZ36uoGaw==", + "linux-arm64": "7q7wZe0+5LLQt2YUkocGb41B9TvEMLKIaNI+CiQUQQ2DFBO1RPr3ixj/Gtb+0noMYHjJrqtqKUMO1xMQXnE0iw==", + "linuxmusl-x64": "GCriICDKw7K5dNjAVo8xFpSJXcYXXBAK9RDoeuSUv6rB5IVRP96IwxeynTy4Xc21l1507Efz+I2yy6FW5gKQ/w==", + "linuxmusl-arm64": "2OWfVP1nklnuR8ZfCwozJcLB0ZpOTRHqAkDpsLonEF5uHZMBnOAzGtd5d+nuPU+KaS5czciU0tXZFhsCUw7dbg==", + "darwin-x64": "tW2Vb19+Bh+PLfcGVnrERhc9Mk50Vi8HciIc0nwIJXlPm21xzKL6uEQHMoyKIAxF+vNXxDsUoJyNRg/TgCXjkA==", + "darwin-arm64": "H6jehr7s1D/RSTRplCL++hneLTgpqVzgzjrX/NJCsf0OMBSKNGCd9kMYAZJjMlp0IetS39sG5PTG7AcLCTQKeg==", + "win32-x64": "r+BAeN980mQEdIwDcrNU3iJqzAryuOyfeICFfYHuHVclonTr2L0IlzsaGp2D50KNWUeLQrIhBCFuNoT2+iBDTw==", + "win32-arm64": "jMSa36VrAiLsWFtnwWX+FEs4JWIjHeduVITY44ZcHsqn7BzPcMljL8LdJ0tVVGssV8x9PuqgIqMD5gFmzcYh9w==", + }), + "127": Object.freeze({ + "linux-x64": "jpVKlRCjwmKDIfohRK4Zf/potrKjl2CuvKiWm2SOL1eY7pTRQrGBc1DSEr4OBulYHE6PTl59w7hoAlzNOefG3g==", + "linux-arm64": "ei++r3DQbhwXsP97PHYENHnNd3jrai5dy11SCDZqzmqEm6mjloT1qwrpglVJmW4TLliYBJ8U9bMG/6Y8nbDBCQ==", + "linuxmusl-x64": "eFglLYVSmdQo7ehLh30dOF4H9csX92XIn1oKB3TdtTZ+7TTh7cfViO7TQIC4NPt6Ufbl3cos9y6/pU8FNL1dQg==", + "linuxmusl-arm64": "e72bURjJ4SYseIMY7uW1IkubTqzuKzD6NIVP0InsD8+DJBnRwi8+yIEEcODIK4xsFKSO6tE8TLuvPmetXEk8aA==", + "darwin-x64": "GxXg+WENfyP9DGnvOSMnPge+kDkEZmU8cF7EFOoFSzAFhEwYOBvFB4VTTqhZ6G4y54I7ufkjgkOxZmafdSOXOw==", + "darwin-arm64": "V1mPpKKnv3AvfZb/++CSOx49tmUi4fba4vORsD8GTaJVwvKmveJ/uPqXaEVhYQgDkao5VLiWL6gN2wk9rQXfPw==", + "win32-x64": "7chlb4LP9ETESK6djkGxDq4aDTTByiG1VoUwadJk7JZFhfn2u7vblPCulZaM2C3Ju3jHowc8LBKKQZiH52Uc5g==", + "win32-arm64": "0mD1nLGHG066cUFdIgMTEJfhbIeOgD52+dE0r8LAALarhU5JD0uyKPvpQH/V3rBjvwTOtaQtYqeEFgRj5NUL/g==", + }), + }), + }, + }, +]); const SUPPORTED_TARGETS = [ "linux-x64", "linux-arm64", @@ -58,10 +134,11 @@ const SUPPORTED_TARGETS = [ "win32-x64", "win32-arm64", ]; -// Whole-download budget for the ~11 MB engine tarball. Must cover a multi-MB -// registry fetch over a non-trivial network (corporate proxy / slow link); -// 10 s locked Node<=21 machines into a 6 h legacy-only cooldown after one slow -// boot (#3338). Aligned with the updater's own tarball budget (src/update.ts). +// Whole-download budget for the engine artifacts (v13: one ~11 MB tarball; +// v12: ~2.6 MB of JS plus a ~1 MB prebuild). Must cover a multi-MB registry +// fetch over a non-trivial network (corporate proxy / slow link); 10 s locked +// Node<=21 machines into a 6 h legacy-only cooldown after one slow boot +// (#3338). Aligned with the updater's own tarball budget (src/update.ts). export const BOOTSTRAP_TIMEOUT_MS = 60_000; export const COOLDOWN_MS = 6 * 60 * 60 * 1000; @@ -71,6 +148,7 @@ type BootstrapStatus = | "downloaded" | "disabled" | "unsupported-platform" + | "unsupported-node" | "cooldown" | "failed"; @@ -83,7 +161,7 @@ let specOverride: EngineBootstrapSpec | null = null; let platformOverride: string | null = null; let inFlight: Promise | null = null; -/** Test seam: swap the pinned artifact (version/sha) for a hermetic fixture. */ +/** Test seam: swap the pinned artifact (version/digests/floor) for a hermetic fixture. */ export function setEngineBootstrapSpecForTests(spec: EngineBootstrapSpec | null): void { specOverride = spec; } @@ -105,8 +183,57 @@ export function setBootstrapTimeoutMsForTests(ms: number | null): void { bootstrapTimeoutOverrideMs = ms; } -function activeSpec(): EngineBootstrapSpec { - return specOverride ?? DEFAULT_ENGINE_SPEC; +/** + * Pure floor predicate (version in -> verdict out) so tests pin the threshold + * boundary without depending on the Node running them. An unparseable major + * counts as satisfied: `process.versions.node` is semver by contract, and + * bricking unified storage on an exotic runtime would be the worse failure. + */ +export function nodeMajorSatisfiesFloor(minNodeMajor: number, nodeVersion: string = process.versions.node): boolean { + const major = Number.parseInt(nodeVersion.split(".")[0] ?? "", 10); + if (!Number.isFinite(major)) return true; + return major >= minNodeMajor; +} + +/** Why this spec cannot serve the running Node, or null when it can. Both + * halves are load-bearing: the floor is what the package declares, the ABI is + * what the binary was actually built against — a JS-only tarball whose + * prebuild targets a different ABI loads just as badly as one below its floor. */ +function specUnusableReason(spec: EngineBootstrapSpec, nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules, napiVersion: string = process.versions.napi ?? ""): string | null { + if (!nodeMajorSatisfiesFloor(spec.minNodeMajor, nodeVersion)) { + return `better-sqlite3@${spec.version} needs Node.js >= ${spec.minNodeMajor}, this is ${nodeVersion}`; + } + if (spec.minNapiVersion !== undefined) { + const napi = Number.parseInt(napiVersion, 10); + if (!Number.isFinite(napi)) return null; + if (napi < spec.minNapiVersion) { + return `better-sqlite3@${spec.version} is built against N-API ${spec.minNapiVersion}, this Node.js ${nodeVersion} ships N-API ${napiVersion}`; + } + } + if (spec.prebuild && !spec.prebuild.sha512Base64ByAbi[nodeAbi]) { + return `better-sqlite3@${spec.version} ships no binding for Node ABI ${nodeAbi} (pinned ABIs: ${Object.keys(spec.prebuild.sha512Base64ByAbi).join(", ")}), this is Node.js ${nodeVersion}`; + } + return null; +} + +function pinnedSpecForThisNode(): EngineBootstrapSpec | null { + return ENGINE_SPECS.find((s) => specUnusableReason(s) === null) ?? null; +} + +/** Pure band selector (runtime triple in -> pinned version out, null = nothing + * fits) so tests pin the whole matrix — including the Node 22.0-22.12 band + * that passes v13's major floor but predates N-API 10 — without downloads. */ +export function pinnedEngineVersionFor(nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules, napiVersion: string = process.versions.napi ?? ""): string | null { + return ENGINE_SPECS.find((s) => specUnusableReason(s, nodeVersion, nodeAbi, napiVersion) === null)?.version ?? null; +} + +/** Honest "nothing fits" text: names the running Node/ABI and every pinned + * band, so the report explains itself instead of blaming the network. */ +function unsupportedNodeDetail(nodeVersion: string = process.versions.node, nodeAbi: string = process.versions.modules): string { + const covered = ENGINE_SPECS.map((s) => (s.prebuild + ? `better-sqlite3@${s.version} on Node.js ${s.minNodeMajor}.x (ABI ${Object.keys(s.prebuild.sha512Base64ByAbi).join("/")})` + : `better-sqlite3@${s.version} on Node.js >= ${s.minNodeMajor} (N-API >= ${s.minNapiVersion ?? 0})`)).join(" or "); + return `no pinned SQLite engine runs on Node.js ${nodeVersion} (ABI ${nodeAbi}) — the pinned artifacts cover ${covered}`; } /** Platform key matching better-sqlite3's own binding.js musl detection. */ @@ -131,8 +258,9 @@ function cacheRoot(): string { return path.join(cacheDir(), "sqlite"); } -function finalDir(version: string): string { - return path.join(cacheRoot(), `better-sqlite3-${version}`); +function finalDir(spec: EngineBootstrapSpec, nodeAbi: string = process.versions.modules): string { + const suffix = spec.prebuild ? `-abi${nodeAbi}` : ""; + return path.join(cacheRoot(), `better-sqlite3-${spec.version}${suffix}`); } function cooldownFile(): string { @@ -222,10 +350,23 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise { + await fsp.rm(dest, { recursive: true, force: true }).catch(() => { // best-effort cleanup; the pinned download re-verifies anyway }); } else { @@ -244,6 +385,9 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise= 22.5; upgrade Node.js for the builtin engine (engine tarball mirrors tried in order: npm + npmmirror; set BILI_UPDATE_REGISTRY or update.registry to pin one)"; } -function tryRegisterFromCache(dir: string): boolean { +/** v12 and earlier construct with `(file, { nativeBinding })`; the extra option + * is what lets a JS-only tarball bind the pinned ABI prebuild without the + * `bindings` resolver or an install script. A path (not the addon object) is + * passed so the binding is loaded by v12 at construction time, keeping + * `dlopen` inside the one call the ABI gate has already cleared. */ +type BetterSqlite3CtorWithBinding = new (file: string, options: { nativeBinding: string }) => InstanceType; + +function withNativeBinding(Database: BetterSqlite3CtorWithBinding, addonPath: string): BetterSqlite3Ctor { + const Bound = function BoundDatabase(this: unknown, file: string): InstanceType { + return new Database(file, { nativeBinding: addonPath }); + }; + return Bound as unknown as BetterSqlite3Ctor; +} + +function tryRegisterFromCache(dir: string, spec: EngineBootstrapSpec, target: string): boolean { try { const r = createRequire(path.join(dir, "package.json")); const mod = r("./lib/index.js") as unknown as BetterSqlite3Ctor; - registerSelfDownloadedBetterSqlite3(() => mod); + if (!spec.prebuild) { + registerSelfDownloadedBetterSqlite3(() => mod); + return true; + } + const addonPath = path.join(dir, "prebuilds", `${target}.node`); + if (!existsSync(addonPath)) return false; + registerSelfDownloadedBetterSqlite3(() => withNativeBinding(mod as unknown as BetterSqlite3CtorWithBinding, addonPath)); return true; } catch (err) { loggerLog("debug", `[storage] sqlite engine self-bootstrap: cache registration failed (${String(err)})`); @@ -334,12 +520,19 @@ function tryRegisterFromCache(dir: string): boolean { } } +const PINNED_DIR_PREFIXES = ENGINE_SPECS.map((s) => `better-sqlite3-${s.version}`); + async function pruneSiblingVersions(keepDir: string): Promise { try { const entries = await fsp.readdir(cacheRoot()); await Promise.all(entries.flatMap((e) => { - if (!e.startsWith("better-sqlite3-") || path.join(cacheRoot(), e) === keepDir) return []; - return [fsp.rm(path.join(cacheRoot(), e), { recursive: true, force: true }).catch(() => {})]; + const dir = path.join(cacheRoot(), e); + if (!e.startsWith("better-sqlite3-") || dir === keepDir) return []; + // Pinned siblings stay: switching Node majors (20 <-> 22) must not + // re-download an artifact this build already knows how to load. The + // prefix match also keeps every pinned ABI variant of one version. + if (PINNED_DIR_PREFIXES.some((p) => e === p || e.startsWith(`${p}-abi`))) return []; + return [fsp.rm(dir, { recursive: true, force: true }).catch(() => {})]; })); } catch { // pruning is housekeeping only — never fail the bootstrap over it @@ -350,35 +543,34 @@ async function pruneSiblingVersions(keepDir: string): Promise { // Download + verify + extract // --------------------------------------------------------------------------- -async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, resolveProxy?: (url: string) => string | undefined): Promise { +async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, nodeAbi: string, resolveProxy?: (url: string) => string | undefined): Promise { const root = cacheRoot(); await fsp.mkdir(root, { recursive: true }); - const tmpTgz = path.join(root, `.dl-${randomBytes(6).toString("hex")}.tgz`); const tmpDir = path.join(root, `.wip-${randomBytes(6).toString("hex")}`); try { // #3338 directive 2: walk the mirror list; the FIRST mirror that // yields a sha512-verified tarball wins. Fetch + integrity are // per-mirror; extraction/publishing happens once, mirror-independent. - let tgz: Buffer | null = null; - const attempted: string[] = []; - let lastErr: unknown = null; - for (const base of engineMirrors()) { - const url = `${base}/better-sqlite3/-/better-sqlite3-${spec.version}.tgz`; - attempted.push(base); - try { - tgz = await fetchVerifiedTarball(url, spec, tmpTgz, resolveProxy); - break; - } catch (err) { - lastErr = err; - loggerLog("warn", `[storage] sqlite engine self-bootstrap: mirror ${base} failed (${err instanceof Error ? err.message : String(err)}) — trying next mirror`); - } - } - if (!tgz) { - const reason = lastErr instanceof Error ? lastErr.message : String(lastErr); - throw new Error(`${reason} (tried mirrors: ${attempted.join(", ")})`); + const npmTgz = await fetchFirstMirror( + engineMirrors().map((base) => ({ url: `${base}/better-sqlite3/-/better-sqlite3-${spec.version}.tgz`, label: base })), + spec.sha512Base64, + root, + resolveProxy, + ); + await extractTgzToDir(npmTgz, tmpDir, target, { withPrebuilds: !spec.prebuild }); + if (spec.prebuild) { + const pinned = spec.prebuild.sha512Base64ByAbi[nodeAbi]?.[target]; + if (!pinned) throw new Error(`no pinned prebuild digest for ${target} on ABI ${nodeAbi}`); + const fileName = `better-sqlite3-v${spec.version}-node-v${nodeAbi}-${target}.tar.gz`; + const prebuildTgz = await fetchFirstMirror( + prebuildMirrors().map((base) => ({ url: `${base}/v${spec.version}/${fileName}`, label: base })), + pinned, + root, + resolveProxy, + ); + await extractPrebuildTgz(prebuildTgz, tmpDir, target); } - await extractTgzToDir(tgz, tmpDir, target); - const dest = finalDir(spec.version); + const dest = finalDir(spec, nodeAbi); if (existsSync(dest)) { // rename(2) over a non-empty dir is ENOTEMPTY: a stale final dir would // fail every attempt and re-arm the 6h cooldown indefinitely. A usable @@ -389,15 +581,41 @@ async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, res } await fsp.rename(tmpDir, dest); } finally { - await fsp.rm(tmpTgz, { force: true }).catch(() => {}); await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => {}); } } +interface MirrorAttempt { + url: string; + /** Shown in the retry log — the mirror root, not the full asset URL. */ + label: string; +} + +async function fetchFirstMirror(attempts: readonly MirrorAttempt[], expectedSha512Base64: string, scratchDir: string, resolveProxy?: (url: string) => string | undefined): Promise { + const tmpTgz = path.join(scratchDir, `.dl-${randomBytes(6).toString("hex")}.tgz`); + const attempted: string[] = []; + let lastErr: unknown = null; + try { + for (const attempt of attempts) { + attempted.push(attempt.label); + try { + return await fetchVerifiedTarball(attempt.url, expectedSha512Base64, tmpTgz, resolveProxy); + } catch (err) { + lastErr = err; + loggerLog("warn", `[storage] sqlite engine self-bootstrap: mirror ${attempt.label} failed (${err instanceof Error ? err.message : String(err)}) — trying next mirror`); + } + } + const reason = lastErr instanceof Error ? lastErr.message : String(lastErr); + throw new Error(`${reason} (tried mirrors: ${attempted.join(", ")})`); + } finally { + await fsp.rm(tmpTgz, { force: true }).catch(() => {}); + } +} + /** One mirror attempt: stream the tarball to `tmpTgz` while hashing; return * the bytes only when the pinned sha512 matches. Throws on timeout / HTTP * error / integrity mismatch so the caller can advance to the next mirror. */ -async function fetchVerifiedTarball(url: string, spec: EngineBootstrapSpec, tmpTgz: string, resolveProxy?: (url: string) => string | undefined): Promise { +async function fetchVerifiedTarball(url: string, expectedSha512Base64: string, tmpTgz: string, resolveProxy?: (url: string) => string | undefined): Promise { const dispatcher = proxyDispatcher(resolveProxy?.(url)); // @types/node types RequestInit.dispatcher as its internal Dispatcher; // cast once here (no `as any`), same pattern as src/update.ts fetchWithEgress. @@ -422,36 +640,61 @@ async function fetchVerifiedTarball(url: string, spec: EngineBootstrapSpec, tmpT await out.close(); } const digest = hash.digest("base64"); - if (digest !== spec.sha512Base64) { - throw new Error(`sha512 mismatch (got ${digest.slice(0, 16)}..., pinned ${spec.sha512Base64.slice(0, 16)}...) — quarantining download`); + if (digest !== expectedSha512Base64) { + throw new Error(`sha512 mismatch (got ${digest.slice(0, 16)}..., pinned ${expectedSha512Base64.slice(0, 16)}...) — quarantining download`); } return await fsp.readFile(tmpTgz); } // --------------------------------------------------------------------------- // Minimal tgz extractor (devDeps are unavailable at runtime — bili ships a -// self-contained bundle). Supports exactly what npm tarballs of the pinned -// package contain: regular files, directories, GNU long names ('L') and PAX -// path records ('x'). Anything else fails loud rather than half-extracting. -// Only lib/, prebuilds/ and package.json are materialized (~1.5 MB of the -// 27 MB unpacked tarball — the C++ sources are build-time only). +// self-contained bundle). Supports exactly what the pinned artifacts contain: +// regular files, directories, GNU long names ('L') and PAX path records ('x'). +// Anything else fails loud rather than half-extracting. // --------------------------------------------------------------------------- const INCLUDE_EXACT = new Set(["package/package.json"]); -const INCLUDE_PREFIXES = ["package/lib/", "package/prebuilds/"]; -function included(name: string): boolean { - return INCLUDE_EXACT.has(name) || INCLUDE_PREFIXES.some((p) => name.startsWith(p)); +function included(name: string, withPrebuilds: boolean): boolean { + if (INCLUDE_EXACT.has(name) || name.startsWith("package/lib/")) return true; + return withPrebuilds && name.startsWith("package/prebuilds/"); +} + +interface ExtractTgzOptions { + /** The npm tarball carries `prebuilds/.node` itself (v13). Off for a + * JS-only tarball whose binding arrives as a separate ABI prebuild (v12). */ + withPrebuilds?: boolean; } -export async function extractTgzToDir(tgz: Buffer, destDir: string, target: string): Promise { +export async function extractTgzToDir(tgz: Buffer, destDir: string, target: string, opts: ExtractTgzOptions = {}): Promise { + const withPrebuilds = opts.withPrebuilds ?? true; + await fsp.mkdir(destDir, { recursive: true }); + await extractTgzEntries(tgz, (name) => (included(name, withPrebuilds) ? destFor(name, destDir) : null)); + const required = withPrebuilds + ? ["package.json", path.join("lib", "index.js"), path.join("prebuilds", `${target}.node`)] + : ["package.json", path.join("lib", "index.js")]; + for (const rel of required) { + if (!existsSync(path.join(destDir, rel))) throw new Error(`tarball missing required file: ${rel}`); + } +} + +/** Extract the ABI-keyed prebuild tarball (a single `build/Release/*.node`) to + * `prebuilds/.node` — the location cacheUsable() and the v12 loader + * both expect, regardless of the source layout. */ +export async function extractPrebuildTgz(tgz: Buffer, destDir: string, target: string): Promise { + const dest = path.join(destDir, "prebuilds", `${target}.node`); + await fsp.mkdir(path.dirname(dest), { recursive: true }); + await extractTgzEntries(tgz, (name) => (name.endsWith(".node") ? dest : null)); + if (!existsSync(dest)) throw new Error(`prebuild tarball missing a .node binding for ${target}`); +} + +async function extractTgzEntries(tgz: Buffer, resolveDest: (name: string) => string | null): Promise { let data: Buffer; try { data = gunzipSync(tgz); } catch (err) { throw new Error(`gunzip failed: ${String(err)}`); } - await fsp.mkdir(destDir, { recursive: true }); let off = 0; let pendingName: string | null = null; while (off + 512 <= data.length) { @@ -475,20 +718,17 @@ export async function extractTgzToDir(tgz: Buffer, destDir: string, target: stri if (paxPath) pendingName = paxPath; continue; } - if (!included(name)) continue; + const dest = resolveDest(name); + if (dest === null) continue; if (typeflag === 0x30 || typeflag === 0x00) { - const dest = destFor(name, destDir); await fsp.mkdir(path.dirname(dest), { recursive: true }); await fsp.writeFile(dest, payload); } else if (typeflag === 0x35) { - await fsp.mkdir(destFor(name, destDir), { recursive: true }); + await fsp.mkdir(dest, { recursive: true }); } else { throw new Error(`unsupported tar entry type 0x${typeflag.toString(16)} for ${name}`); } } - for (const required of ["package.json", path.join("lib", "index.js"), path.join("prebuilds", `${target}.node`)]) { - if (!existsSync(path.join(destDir, required))) throw new Error(`tarball missing required file: ${required}`); - } } function destFor(name: string, destDir: string): string { diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index bc4ec3300f..d47a0421a8 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -83,6 +83,13 @@ function degradeReason(): string { const last = sidecar.stats.lastError ? shortenError(sidecar.stats.lastError) : "unknown"; return `unified store auto-disabled after ${MAX_CONSECUTIVE_FAILURES} consecutive failures (last: ${last})`; } + // No pinned artifact serves this runtime (below every Node floor, or a Node + // ABI no prebuild targets), so the engine is refused BEFORE any load and the + // sidecar open ALSO fails with the generic no-engine error — surface the real + // reason first, or the sidecar-open-error branch below would shadow it (#3894). + if (state.bootstrap?.status === "unsupported-node") { + return `no self-downloaded SQLite engine supports this Node.js (${state.bootstrap.detail ?? "unknown"})`; + } if (state.sidecarOpenError) { const e = shortenError(state.sidecarOpenError); // The driver's no-engine error surfaces through the same open path — diff --git a/tests/acp-status-storage-degraded.test.ts b/tests/acp-status-storage-degraded.test.ts index eda94fd3b5..27bcb1a41a 100644 --- a/tests/acp-status-storage-degraded.test.ts +++ b/tests/acp-status-storage-degraded.test.ts @@ -111,3 +111,34 @@ test("#3385: healthy storage stays quiet — no degrade line", () => { rmrf(base); } }); + +test("#3894: an unsupported Node surfaces the artifact-table reason, not generic no-engine", () => { + // Unsupported-node refuses the engine BEFORE any load, so the sidecar open + // ALSO records a "no SQLite engine available" open-error. That reason must + // outrank the generic one — otherwise the user sees "better-sqlite3 not + // installed" when the real cause is the running Node (#3894 dead-code fix). + const base = mkdtempSync(path.join(tmpdir(), "acp-status-3894-")); + const snap = isolate(base); + let store: SessionStore | null = null; + try { + setSqliteEnginesUnavailableForTests(true); + _resetStorageRuntimeStatusForTest(); + recordEngineBootstrap({ status: "unsupported-node", detail: "no pinned SQLite engine runs on Node.js 21.7.3 (ABI 120)" }); + store = new SessionStore({ dir: process.env.BILI_SESSIONS_DIR, enabled: true }); + _setStoreForTest(store); + + const out = handleAcpStatus({}, makeCtx()); + const text = out.text ?? ""; + assert.ok(text.includes("STORAGE —"), "the storage line is present"); + assert.ok(text.includes("⚠ degraded —"), `the degrade warning line is present (got: ${text.slice(-400)})`); + assert.ok(text.includes("no self-downloaded SQLite engine supports this Node.js"), `the specific reason surfaces (got: ${text.slice(-400)})`); + assert.ok(/Node\.js 21\.7\.3/.test(text), "the reason carries the runtime the table could not serve"); + } finally { + store?.closeQuietly(); + _setStoreForTest(null as unknown as SessionStore); + setSqliteEnginesUnavailableForTests(false); + _resetStorageRuntimeStatusForTest(); + restore(snap); + rmrf(base); + } +}); diff --git a/tests/storage-engine-bootstrap.test.ts b/tests/storage-engine-bootstrap.test.ts index f398ac9cfa..8fc9c89cfc 100644 --- a/tests/storage-engine-bootstrap.test.ts +++ b/tests/storage-engine-bootstrap.test.ts @@ -11,6 +11,16 @@ * re-fetch), integrity mismatch (quarantine + cooldown), * cooldown skip + expiry, offline fail-safe, disabled * knob, legacy-mode short-circuit, unsupported platform. + * NODE BANDS — the #3894 per-major artifact table: Node 20 downloads + * the JS-only 12.9.0 tarball PLUS its ABI-keyed prebuild + * and binds it through `nativeBinding`; a prebuild whose + * ABI does not match the running Node is refused before + * any download; a pinned prebuild-less artifact keeps the + * single-tarball lane and never touches the binary mirror. + * A Node that passes the major floor but predates the + * artifact's N-API version falls through to the next + * band; ABI-keyed artifacts cache per ABI so a second + * runtime never inherits the wrong binding. * EXTRACTOR — include filtering, required-file validation, path * traversal rejection, symlink rejection, non-gzip input. * KNOB — BILI_SQLITE_AUTO_DOWNLOAD > storage.sqliteAutoDownload @@ -43,7 +53,10 @@ import { import { BOOTSTRAP_TIMEOUT_MS, ensureSqliteEngine, + extractPrebuildTgz, extractTgzToDir, + nodeMajorSatisfiesFloor, + pinnedEngineVersionFor, nodeSqliteUpgradeHint, platformTargetKey, resetEngineBootstrapForTests, @@ -51,6 +64,7 @@ import { setEngineBootstrapSpecForTests, setEngineMirrorsForTests, setPlatformKeyForTests, + setPrebuildMirrorsForTests, type BootstrapOutcome, } from "../src/storage/engine-bootstrap.ts"; import { resolveSqliteAutoDownload } from "../src/knobs.ts"; @@ -205,6 +219,7 @@ function resetAllSeams(): void { clearSelfDownloadedEngineForTests(); setEngineBootstrapSpecForTests(null); setEngineMirrorsForTests(null); + setPrebuildMirrorsForTests(null); setPlatformKeyForTests(null); setBootstrapTimeoutMsForTests(null); resetEngineBootstrapForTests(); @@ -331,7 +346,7 @@ test("bootstrap: happy path downloads, verifies sha, extracts, registers loader; try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -377,7 +392,7 @@ test("#3818 unloadable cached copy is removed and a pinned download is retried ( try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -416,7 +431,7 @@ test("bootstrap: sha512 mismatch quarantines the download and arms the cooldown" const good = fakePackageTgz(); const tampered = Buffer.from(good); tampered[tampered.length - 1] ^= 0xff; - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(good) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(good), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tampered); await withRegistry(reg.base, async () => { @@ -448,7 +463,7 @@ test("bootstrap: expired cooldown re-attempts the download", async () => { try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); const sqliteDir = path.join(cachePrev, "billion-context", "sqlite"); fs.mkdirSync(sqliteDir, { recursive: true }); @@ -473,7 +488,7 @@ test("bootstrap: stale unusable cache dir is replaced instead of looping the coo try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); // Partial/corrupt publish from a killed run: dir exists but lacks the // target prebuilt, so cacheUsable() is false and rename-over would ENOTEMPTY. @@ -506,7 +521,7 @@ test("bootstrap: a download that outlasts its timeout budget aborts fail-safe an try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); const delayMs = 800; setBootstrapTimeoutMsForTests(Math.floor(delayMs / 5)); @@ -532,7 +547,7 @@ test("bootstrap: a slow-but-completing download succeeds once the budget covers try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); const delayMs = 400; setBootstrapTimeoutMsForTests(delayMs * 10); @@ -554,7 +569,7 @@ test("bootstrap: offline (connection refused) degrades fail-safe with cooldown, const cachePrev = isolateCache(); try { resetAllSeams(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: "irrelevant" }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: "irrelevant", minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); await withRegistry("http://127.0.0.1:1", async () => { const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); @@ -576,7 +591,7 @@ test("bootstrap: disabled knob gates the fetch (cached copies would still be use try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -598,7 +613,7 @@ test("bootstrap: legacy mode short-circuits before any probe or fetch", async () try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -623,7 +638,7 @@ test("bootstrap: builtin present -> not-needed without touching the network", as try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: true, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -646,7 +661,7 @@ test("bootstrap: host-provided engine -> not-needed without touching the network try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: true }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -669,7 +684,7 @@ test("bootstrap: unsupported platform key -> no download attempt", async () => { try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); setPlatformKeyForTests("freebsd-arm64"); reg = await startMockRegistry(tgz); @@ -693,7 +708,7 @@ test("bootstrap: concurrent callers share one attempt (process-dedup)", async () try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); await withRegistry(reg.base, async () => { @@ -821,7 +836,7 @@ test("mirror fallback: a failing first mirror falls to the next and the bootstra try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); bad = await startMockRegistry(tgz, 500); // registry up, tarball broken at the HTTP layer good = await startMockRegistry(tgz); @@ -847,7 +862,7 @@ test("mirror exhaustion: the cooldown reason lists every mirror attempted (#3338 try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); bad1 = await startMockRegistry(tgz, 500); bad2 = await startMockRegistry(tgz, 503); @@ -875,7 +890,7 @@ test("mirror list: a configured registry is attempted before the builtins (#3338 try { resetAllSeams(); const tgz = fakePackageTgz(); - setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); setSqliteProbeOverridesForTests({ builtin: false, host: false }); reg = await startMockRegistry(tgz); // Deliberately NO mirrors override: the production list must put the @@ -913,3 +928,440 @@ test("nodeSqliteUpgradeHint: pins the Node >= 22.5 threshold and the levers (#33 assert.match(nodeSqliteUpgradeHint("19.9.9"), /Node\.js >= 22\.5/, "19.x predates node:sqlite"); assert.equal(nodeSqliteUpgradeHint("22.5.1"), "", "22.5.1 ships node:sqlite — no hint"); }); + +// --------------------------------------------------------------------------- +// Artifact Node floor (#3341): better-sqlite3 v13 declares `engines: node >= 22` +// and faults the process on Node 20 instead of failing cleanly, so the pinned +// spec carries that floor and the bootstrap refuses BOTH the download and the +// cached-copy registration below it. +// --------------------------------------------------------------------------- + +test("nodeMajorSatisfiesFloor: both directions of the artifact floor (#3341)", () => { + assert.equal(nodeMajorSatisfiesFloor(22, "20.19.5"), false, "Node 20 is below the v13 floor"); + assert.equal(nodeMajorSatisfiesFloor(22, "18.20.8"), false, "Node 18 is below the v13 floor"); + assert.equal(nodeMajorSatisfiesFloor(22, "22.0.0"), true, "Node 22 satisfies the floor"); + assert.equal(nodeMajorSatisfiesFloor(22, "22.4.9"), true, "the floor is a MAJOR floor - 22.x minors all pass"); + assert.equal(nodeMajorSatisfiesFloor(22, "24.11.0"), true, "Node 24 satisfies the floor"); + assert.equal(nodeMajorSatisfiesFloor(20, "20.19.5"), true, "the predicate tracks the floor it is given, not a hardcoded 22"); + assert.equal(nodeMajorSatisfiesFloor(22, "garbage"), true, "an unparseable runtime must not brick unified storage"); +}); + +test("bootstrap: below the artifact Node floor -> unsupported-node, no download, no cache registration (#3341)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + // minNodeMajor above ANY Node that could run this suite: the gate must + // fire regardless of which runtime CI uses (ubuntu/windows x 22/24). + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 99 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + reg = await startMockRegistry(tgz); + await withRegistry(reg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `expected the floor gate, got ${JSON.stringify(out)}`); + assert.match(out.detail ?? "", /better-sqlite3@13\.0\.3 needs Node\.js >= 99/, "the detail names the artifact and its floor"); + assert.equal(reg!.requests(), 0, "below the floor the ~11 MB download must never start"); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3")), "nothing materialized into the cache"); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", ".bootstrap-failed.json")), "a too-old Node is not a download failure - no cooldown armed"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); + }); + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: a cache that already exists is left in place (and unregistered) below the floor (#3341)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 99 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + const target = platformTargetKey(); + const cached = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3"); + fs.mkdirSync(path.join(cached, "lib"), { recursive: true }); + fs.mkdirSync(path.join(cached, "prebuilds"), { recursive: true }); + fs.writeFileSync(path.join(cached, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "13.0.3" })); + fs.writeFileSync(path.join(cached, "lib", "index.js"), FAKE_LIB_INDEX); + fs.writeFileSync(path.join(cached, "prebuilds", `${target}.node`), Buffer.from("FAKE-NODE-BINARY")); + reg = await startMockRegistry(tgz); + await withRegistry(reg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `expected the floor gate, got ${JSON.stringify(out)}`); + assert.equal(reg!.requests(), 0, "a cache hit below the floor must not trigger a re-download either"); + assert.ok(fs.existsSync(path.join(cached, "prebuilds", `${target}.node`)), "the cached copy must survive: a Node upgrade makes it valid again"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "a cache below the floor must never be registered"); + }); + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: at/above the artifact Node floor the gate does NOT fire (#3341 negative case)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + reg = await startMockRegistry(tgz); + await withRegistry(reg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `a satisfied floor must not block the bootstrap, got ${JSON.stringify(out)}`); + assert.equal(reg!.requests(), 1, "the download proceeded"); + const o = openSqliteDatabase(":memory:"); + assert.equal(o.engine, "better-sqlite3"); + assert.equal(o.source, "self-downloaded"); + o.db.close(); + }); + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +// --------------------------------------------------------------------------- +// Per-Node-major artifact table (#3894): a JS-only npm tarball (v12 and +// earlier) ships no binding at all, so the bootstrap must ALSO fetch the +// ABI-keyed prebuild and bind it through v12's `nativeBinding` option. Getting +// this wrong is what the whole issue is about: registering a binding whose +// floor/ABI does not match the running Node kills the process uncatchably. +// --------------------------------------------------------------------------- + +/** Stub v12 lib: throws unless the loader injected a `nativeBinding` PATH — + * the injection is the contract under test, and a path (not an addon object) + * keeps `dlopen` inside the constructor v12 itself owns. */ +const FAKE_V12_LIB_INDEX = `const fs = require('node:fs'); +module.exports = class FakeBetterSqlite3V12 { + constructor(p, options) { + if (!options || typeof options.nativeBinding !== 'string') { + throw new Error('FAKE v12: nativeBinding path was not injected'); + } + if (!fs.existsSync(options.nativeBinding)) { + throw new Error('FAKE v12: nativeBinding path does not exist: ' + options.nativeBinding); + } + this.bindingPath = options.nativeBinding; + this.p = p; + this.open = true; + } + exec() {} + prepare() { return { run: () => ({ changes: 1, lastInsertRowid: 1 }), get: () => ({ ok: true }), all: () => [] }; } + transaction(fn) { const w = () => fn(); w.immediate = w; w.deferred = w; w.exclusive = w; return w; } + pragma() {} + close() { this.open = false; } +};`; + +/** JS-only npm tarball shape (v12): package.json + lib/, NO prebuilds/. */ +function fakeJsOnlyPackageTgz(): Buffer { + return buildTgz([ + { name: "package/package.json", content: JSON.stringify({ name: "better-sqlite3", version: "12.9.0" }) }, + { name: "package/lib/index.js", content: FAKE_V12_LIB_INDEX }, + { name: "package/lib/database.js", content: "module.exports = require('./index.js');" }, + { name: "package/deps/sqlite3.c", content: "int unused;" }, + ]); +} + +/** ABI-keyed prebuild shape: one `build/Release/better_sqlite3.node`. */ +function fakePrebuildTgz(): Buffer { + return buildTgz([{ name: "build/Release/better_sqlite3.node", content: Buffer.from("FAKE-ABI-NODE-BINARY") }]); +} + +async function startMockBinaryMirror(body: Buffer, version: string, abi: string, target: string): Promise { + const expected = `/v${version}/better-sqlite3-v${version}-node-v${abi}-${target}.tar.gz`; + let count = 0; + const server = http.createServer((req, res) => { + count++; + if (req.url !== expected) { + res.writeHead(404); + res.end(); + return; + } + res.writeHead(200, { "content-type": "application/octet-stream" }); + res.end(body); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const addr = server.address() as AddressInfo; + return { + base: `http://127.0.0.1:${addr.port}`, + requests: () => count, + close: () => new Promise((r) => { + (server as unknown as { closeAllConnections?: () => void }).closeAllConnections?.(); + server.close(() => r()); + }), + }; +} + +test("extractor: ABI prebuild tarball lands at prebuilds/.node", async () => { + const dest = fs.mkdtempSync(path.join(os.tmpdir(), "bili-boot-extract-")); + try { + const target = platformTargetKey(); + await extractPrebuildTgz(fakePrebuildTgz(), dest, target); + assert.ok(fs.existsSync(path.join(dest, "prebuilds", `${target}.node`)), "the binding is published where cacheUsable() and the v12 loader look"); + } finally { + rmrf(dest); + } +}); + +test("extractor: an ABI prebuild tarball with no .node binding fails loud", async () => { + const dest = fs.mkdtempSync(path.join(os.tmpdir(), "bili-boot-extract-")); + try { + const tgz = buildTgz([{ name: "build/Release/README.md", content: "no binding here" }]); + await assert.rejects(() => extractPrebuildTgz(tgz, dest, platformTargetKey()), /missing a \.node binding/); + } finally { + rmrf(dest); + } +}); + +test("bootstrap: a JS-only artifact downloads its ABI prebuild and binds it via nativeBinding (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const abi = process.versions.modules; + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { sha512Base64ByAbi: { [abi]: { [target]: sha512B64(preTgz) } } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", abi, target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `the JS-only lane must install, got ${JSON.stringify(out)}`); + assert.equal(npmReg!.requests(), 1, "the npm tarball was fetched"); + assert.equal(binReg!.requests(), 1, "the ABI prebuild was fetched from the binary mirror"); + const cached = path.join(cachePrev, "billion-context", "sqlite", `better-sqlite3-12.9.0-abi${abi}`); + assert.ok(fs.existsSync(path.join(cached, "prebuilds", `${target}.node`)), "the binding is cached next to the JS package"); + const o = openSqliteDatabase(":memory:"); + assert.equal(o.engine, "better-sqlite3"); + assert.equal(o.source, "self-downloaded"); + o.db.close(); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: a prebuild whose ABI is not this Node is refused before any download (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { sha512Base64ByAbi: { "999": { [target]: sha512B64(preTgz) } } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", "999", target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `an ABI mismatch must be refused, got ${JSON.stringify(out)}`); + assert.match(out.detail ?? "", /no binding for Node ABI \d+/, "the detail names the ABI gap"); + assert.equal(npmReg!.requests(), 0, "neither artifact may be downloaded"); + assert.equal(binReg!.requests(), 0); + assert.ok(!fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", `better-sqlite3-12.9.0-abi${process.versions.modules}`)), "nothing materialized into the cache"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: a self-contained artifact never touches the binary mirror (#3894 negative case)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(tgz); + binReg = await startMockBinaryMirror(fakePrebuildTgz(), "13.0.3", process.versions.modules, platformTargetKey()); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `the single-tarball lane must keep working, got ${JSON.stringify(out)}`); + assert.equal(npmReg!.requests(), 1); + assert.equal(binReg!.requests(), 0, "a tarball that carries its own prebuilds must not hit the binary mirror"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + + +// --------------------------------------------------------------------------- +// N-API floor (#3894 follow-up): a binding compiled against N-API 10 cannot be +// dlopen'd by a runtime that only ships N-API 9 - the load faults uncatchably, +// so the band table must fall through to an artifact that DOES match instead +// of trusting the package's `engines.node` major alone. Node 22.0-22.12 is +// exactly that trap: `>= 22` is satisfied, N-API is still 9. +// --------------------------------------------------------------------------- + +test("pinnedEngineVersionFor: every Node band picks a loadable artifact (#3894)", () => { + assert.equal(pinnedEngineVersionFor("20.19.5", "115", "9"), "12.9.0", "Node 20 -> the ABI 115 JS-only lane"); + assert.equal(pinnedEngineVersionFor("22.0.0", "127", "9"), "12.9.0", "Node 22.0-22.12 passes v13's major floor but predates N-API 10 -> must fall back, not dlopen"); + assert.equal(pinnedEngineVersionFor("22.13.0", "127", "9"), "12.9.0", "the whole pre-N-API-10 22.x band falls back"); + assert.equal(pinnedEngineVersionFor("22.14.0", "127", "10"), "13.0.3", "N-API 10 arrives -> the single-tarball v13 lane"); + assert.equal(pinnedEngineVersionFor("23.11.1", "131", "10"), "13.0.3", "Node 23 is above the floor with N-API 10"); + assert.equal(pinnedEngineVersionFor("24.11.0", "137", "10"), "13.0.3", "Node 24 is above the floor with N-API 10"); + assert.equal(pinnedEngineVersionFor("25.2.1", "141", "10"), "13.0.3", "Node 25 is above the floor with N-API 10"); + assert.equal(pinnedEngineVersionFor("21.7.3", "120", "9"), null, "Node 21 has no pinned band -> honest unsupported-node"); + assert.equal(pinnedEngineVersionFor("19.9.0", "111", "8"), null, "Node 19 is below the engines floor"); + assert.equal(pinnedEngineVersionFor("18.20.8", "108", "8"), null, "Node 18 is below the engines floor"); +}); + +test("bootstrap: an artifact whose N-API floor is above this runtime is refused before any download (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz), minNodeMajor: 20, minNapiVersion: 99 }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(tgz); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "unsupported-node", `an N-API gap must be refused, got ${JSON.stringify(out)}`); + assert.match(out.detail ?? "", /built against N-API 99/, "the detail names the N-API gap"); + assert.equal(npmReg!.requests(), 0, "the tarball must not be downloaded"); + assert.throws(() => openSqliteDatabase(":memory:"), /no SQLite engine available/i, "no loader was registered"); + }); + } finally { + await npmReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: an ABI-keyed artifact keeps one cache dir per ABI so a second runtime fetches its own binding (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const abi = process.versions.modules; + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + // A cache dir from the SAME version but a DIFFERENT ABI: reusing it would + // hand this runtime the wrong binding, so the bootstrap must not treat it + // as a cache hit. + const wrongAbiDir = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-12.9.0-abi999"); + fs.mkdirSync(path.join(wrongAbiDir, "lib"), { recursive: true }); + fs.mkdirSync(path.join(wrongAbiDir, "prebuilds"), { recursive: true }); + fs.writeFileSync(path.join(wrongAbiDir, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "12.9.0" })); + fs.writeFileSync(path.join(wrongAbiDir, "lib", "index.js"), FAKE_V12_LIB_INDEX); + fs.writeFileSync(path.join(wrongAbiDir, "prebuilds", `${target}.node`), Buffer.from("WRONG-ABI")); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { sha512Base64ByAbi: { [abi]: { [target]: sha512B64(preTgz) } } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", abi, target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `a foreign-ABI cache dir must not be reused, got ${JSON.stringify(out)}`); + assert.equal(npmReg!.requests(), 1, "the JS tarball was fetched into this ABI's own dir"); + const ownDir = path.join(cachePrev, "billion-context", "sqlite", `better-sqlite3-12.9.0-abi${abi}`); + assert.ok(fs.existsSync(path.join(ownDir, "prebuilds", `${target}.node`)), "this runtime's own binding was published"); + assert.ok(fs.existsSync(wrongAbiDir), "the other ABI's cache dir survives the prune"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("bootstrap: pinned sibling cache dirs survive the prune; unpinned ones are still removed (#3894)", async () => { + const cachePrev = isolateCache(); + let npmReg: MockRegistry | null = null; + let binReg: MockRegistry | null = null; + try { + resetAllSeams(); + const target = platformTargetKey(); + const abi = process.versions.modules; + const npmTgz = fakeJsOnlyPackageTgz(); + const preTgz = fakePrebuildTgz(); + setEngineBootstrapSpecForTests({ + version: "12.9.0", + sha512Base64: sha512B64(npmTgz), + minNodeMajor: 20, + prebuild: { sha512Base64ByAbi: { [abi]: { [target]: sha512B64(preTgz) } } }, + }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + // Pre-existing siblings before this download publishes: the OTHER pinned + // version's dir must survive the post-publish prune (switching Node + // majors must not force a re-download), an unpinned legacy version must + // still be cleaned up. + const pinnedSibling = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3"); + const staleSibling = path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-9.9.9"); + fs.mkdirSync(path.join(pinnedSibling, "lib"), { recursive: true }); + fs.writeFileSync(path.join(pinnedSibling, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "13.0.3" })); + fs.writeFileSync(path.join(pinnedSibling, "lib", "index.js"), FAKE_LIB_INDEX); + fs.mkdirSync(staleSibling, { recursive: true }); + fs.writeFileSync(path.join(staleSibling, "package.json"), JSON.stringify({ name: "better-sqlite3", version: "9.9.9" })); + npmReg = await startMockRegistry(npmTgz, 200, "12.9.0"); + binReg = await startMockBinaryMirror(preTgz, "12.9.0", abi, target); + setPrebuildMirrorsForTests([binReg.base]); + await withRegistry(npmReg.base, async () => { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `the JS-only lane must install, got ${JSON.stringify(out)}`); + assert.ok(fs.existsSync(path.join(pinnedSibling, "package.json")), "the other pinned version's cache dir survives the prune"); + assert.ok(!fs.existsSync(staleSibling), "an unpinned legacy version is still pruned"); + }); + } finally { + await npmReg?.close(); + await binReg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); diff --git a/tests/storage-full-flip.test.ts b/tests/storage-full-flip.test.ts index ef15357b5b..61b80044b2 100644 --- a/tests/storage-full-flip.test.ts +++ b/tests/storage-full-flip.test.ts @@ -44,8 +44,16 @@ delete process.env.BILI_STORAGE_JSON_WRITER; const PREV_UNIFIED = process.env.BILI_STORAGE_UNIFIED; const PREV_JSON_WRITER = process.env.BILI_STORAGE_JSON_WRITER; +// Windows can still hold an ephemeral handle on index.db/-wal right after a +// store closes (Defender scan, delayed close), so a bare rmSync throws EPERM +// and fails an otherwise green run. Retry, then give up quietly: the dir lives +// under HERMETIC, which the after hook clears anyway. function rmrf(p: string): void { - fs.rmSync(p, { recursive: true, force: true }); + try { + fs.rmSync(p, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + /* Windows stray handle */ + } } function tmpDir(tag: string): string { diff --git a/tests/verif-bug3676-wire-reorder.test.ts b/tests/verif-bug3676-wire-reorder.test.ts index de3992289b..e86f5d00b5 100644 --- a/tests/verif-bug3676-wire-reorder.test.ts +++ b/tests/verif-bug3676-wire-reorder.test.ts @@ -101,7 +101,9 @@ function partType(content: SymArr, i: number): Seq<"main"> { return t.t; } -test("bug3676: solver finds the [file,text]→[text,file] part-order reversal on the real wire codec", { timeout: 240_000 }, async () => { +test("bug3676: solver finds the [file,text]→[text,file] part-order reversal on the real wire codec", { timeout: 240_000 }, async (t) => { + const solver = await (async () => { try { return await solveSmt2("(set-logic ALL)(check-sat)", []); } catch { return null; } })(); + if (!solver) return t.skip("no SMT solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)"); const ctx = newCtx(); const ports = { ...makeCounterPorts(ctx), deriveCanonicalMessageId: makeIdPort(ctx) }; diff --git a/tests/verif-bug3691-oscillation.test.ts b/tests/verif-bug3691-oscillation.test.ts index 7a17c3fa9f..c0ae1d6484 100644 --- a/tests/verif-bug3691-oscillation.test.ts +++ b/tests/verif-bug3691-oscillation.test.ts @@ -32,6 +32,8 @@ const mkMsgs = (T: string): { id: string; role: "tool" | "user"; contentType: "t ]; test("bug3691: solver finds a clip-triggering witness on the real truncation code", { timeout: 240_000 }, async (t) => { + const solver = await (async () => { try { return await solveSmt2("(set-logic ALL)(check-sat)", []); } catch { return null; } })(); + if (!solver) return t.skip("no SMT solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)"); const ctx = newCtx(); const T = ctx.String.const("T"); const tok1 = ctx.Real.const("tok1"); diff --git a/tests/verif-induction-append-stability.test.ts b/tests/verif-induction-append-stability.test.ts index 8ffebecc42..8aa2224d3a 100644 --- a/tests/verif-induction-append-stability.test.ts +++ b/tests/verif-induction-append-stability.test.ts @@ -42,7 +42,10 @@ import type { SymVal } from "../verif/symbolic.ts"; const REFS_FILE = "kernel/src/refs.ts"; -test("induction step: appending a distinct-id message cannot touch any observed binding (arbitrary n)", { timeout: 240_000 }, () => { +test("induction step: appending a distinct-id message cannot touch any observed binding (arbitrary n)", { timeout: 240_000 }, async (t) => { + const solver = await (async () => { try { return await solveSmt2("(set-logic ALL)(check-sat)", []); } catch { return null; } })(); + if (!solver) return t.skip("no SMT solver available (set VERIF_SOLVER_CMD / VERIF_PYTHON)"); + const ctx = newCtx(); // Abstract prefix state: one observed binding under a FREE key/value. const idP = ctx.String.const("id_p"); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index ab3b6fd819..e5c59f1bb9 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -271,8 +271,8 @@ entries: default: "true" env: BILI_SQLITE_AUTO_DOWNLOAD group: process - desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." - desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" + desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788)." + desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。" - key: storage.jsonWriter type: "on|off" default: "on"