diff --git a/CONFIGURATION.md b/CONFIGURATION.md index ab12a3029b..25f336b9a4 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -170,7 +170,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to shadow (JSON stays the sole source of truth; the index is digest-reconciled telemetry - set legacy to opt out). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index b7d26def9f..43d366b1e1 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -170,7 +170,7 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 shadow(JSON 仍是唯一事实源,索引只做对账遥测 —— 设 legacy 可退出)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 | +| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | diff --git a/src/acp-status.ts b/src/acp-status.ts index d8da280a54..17ed8d0455 100644 --- a/src/acp-status.ts +++ b/src/acp-status.ts @@ -24,6 +24,7 @@ import { getUpdateVisibility } from "./update-notes.js"; import { VERSION, BUILD_COMMIT } from "./version.js"; import { isExternalSummaryBlock } from "./external-summary-marker.js"; import { storageStatusLabel } from "./persist.js"; +import { storageStatusWire } from "./storage/runtime-status.js"; import { toolOk, type ProxyToolResult } from "./proxy-tool-result.js"; interface AcpStatusCtx { @@ -362,6 +363,15 @@ export function handleAcpStatus(args: Record, ctx: AcpStatusCtx // am I on" never requires shell access to the lane log. extra.push(""); extra.push(`STORAGE — ${storageStatusLabel()}`); + // #3385 (owner directive 3, acp_status surface): when the unified store is + // degraded (engine missing, self-download cooldown, ...), surface the SAME + // reason string the web banner and diagnostics carry — on Node < 22.5 it + // contains the explicit Node upgrade lever, so the MODEL can tell the + // user instead of the fact living only in lane logs the user never opens. + const storageWire = storageStatusWire(); + if (storageWire.degraded && storageWire.reason) { + extra.push(` ⚠ degraded — ${storageWire.reason} (relay this to the user)`); + } const archive = preCompactionArchiveOf(ctx.session); if (Object.keys(archive).length > 0) { extra.push(""); diff --git a/src/storage/engine-bootstrap.ts b/src/storage/engine-bootstrap.ts index 28e9d15c66..b95795545a 100644 --- a/src/storage/engine-bootstrap.ts +++ b/src/storage/engine-bootstrap.ts @@ -237,7 +237,7 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise= 20 is in-contract, but node:sqlite ships at >= 22.5, so + * Node 20 users see "no SQLite engine" unless the tarball download succeeds. + * Pure (version in → text out) so tests pin the threshold logic; empty string + * on Nodes that already have the builtin. + */ +export function nodeSqliteUpgradeHint(nodeVersion: string = process.versions.node): string { + const parts = nodeVersion.split(".").map((p) => Number.parseInt(p, 10)); + const major = parts[0]; + const minor = parts[1]; + const hasBuiltin = typeof major === "number" && typeof minor === "number" && !Number.isNaN(major) && !Number.isNaN(minor) && (major > 22 || (major === 22 && minor >= 5)); + if (hasBuiltin) return ""; + return " — node:sqlite needs Node.js >= 22.5; upgrade Node.js for the builtin engine (engine tarball mirrors tried in order: npm + npmmirror; set BILI_UPDATE_REGISTRY or update.registry to pin one)"; } function tryRegisterFromCache(dir: string): boolean { @@ -288,41 +327,32 @@ async function pruneSiblingVersions(keepDir: string): Promise { // --------------------------------------------------------------------------- async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, resolveProxy?: (url: string) => string | undefined): Promise { - const base = registryBaseFor(spec); - const url = `${base}/better-sqlite3/-/better-sqlite3-${spec.version}.tgz`; const root = cacheRoot(); await fsp.mkdir(root, { recursive: true }); const tmpTgz = path.join(root, `.dl-${randomBytes(6).toString("hex")}.tgz`); const tmpDir = path.join(root, `.wip-${randomBytes(6).toString("hex")}`); try { - const dispatcher = proxyDispatcher(resolveProxy?.(url)); - // @types/node types RequestInit.dispatcher as its internal Dispatcher; - // cast once here (no `as any`), same pattern as src/update.ts fetchWithEgress. - const res = await fetch(url, { - signal: AbortSignal.timeout(bootstrapTimeoutOverrideMs ?? BOOTSTRAP_TIMEOUT_MS), - ...(dispatcher ? { dispatcher } : {}), - } as RequestInit); - if (!res.ok || !res.body) throw new Error(`registry returned HTTP ${res.status} for ${url}`); - const hash = createHash("sha512"); - const out = await fsp.open(tmpTgz, "w"); - try { - const reader = res.body.getReader(); - for (;;) { - const { done, value } = await reader.read(); - if (done) break; - if (value) { - hash.update(value); - await out.write(value); - } + // #3338 directive 2: walk the mirror list; the FIRST mirror that + // yields a sha512-verified tarball wins. Fetch + integrity are + // per-mirror; extraction/publishing happens once, mirror-independent. + let tgz: Buffer | null = null; + const attempted: string[] = []; + let lastErr: unknown = null; + for (const base of engineMirrors()) { + const url = `${base}/better-sqlite3/-/better-sqlite3-${spec.version}.tgz`; + attempted.push(base); + try { + tgz = await fetchVerifiedTarball(url, spec, tmpTgz, resolveProxy); + break; + } catch (err) { + lastErr = err; + loggerLog("warn", `[storage] sqlite engine self-bootstrap: mirror ${base} failed (${err instanceof Error ? err.message : String(err)}) — trying next mirror`); } - } finally { - await out.close(); } - const digest = hash.digest("base64"); - if (digest !== spec.sha512Base64) { - throw new Error(`sha512 mismatch (got ${digest.slice(0, 16)}..., pinned ${spec.sha512Base64.slice(0, 16)}...) — quarantining download`); + if (!tgz) { + const reason = lastErr instanceof Error ? lastErr.message : String(lastErr); + throw new Error(`${reason} (tried mirrors: ${attempted.join(", ")})`); } - const tgz = await fsp.readFile(tmpTgz); await extractTgzToDir(tgz, tmpDir, target); const dest = finalDir(spec.version); if (existsSync(dest)) { @@ -340,6 +370,40 @@ async function downloadAndInstall(spec: EngineBootstrapSpec, target: string, res } } +/** One mirror attempt: stream the tarball to `tmpTgz` while hashing; return + * the bytes only when the pinned sha512 matches. Throws on timeout / HTTP + * error / integrity mismatch so the caller can advance to the next mirror. */ +async function fetchVerifiedTarball(url: string, spec: EngineBootstrapSpec, tmpTgz: string, resolveProxy?: (url: string) => string | undefined): Promise { + const dispatcher = proxyDispatcher(resolveProxy?.(url)); + // @types/node types RequestInit.dispatcher as its internal Dispatcher; + // cast once here (no `as any`), same pattern as src/update.ts fetchWithEgress. + const res = await fetch(url, { + signal: AbortSignal.timeout(bootstrapTimeoutOverrideMs ?? BOOTSTRAP_TIMEOUT_MS), + ...(dispatcher ? { dispatcher } : {}), + } as RequestInit); + if (!res.ok || !res.body) throw new Error(`registry returned HTTP ${res.status} for ${url}`); + const hash = createHash("sha512"); + const out = await fsp.open(tmpTgz, "w"); + try { + const reader = res.body.getReader(); + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + if (value) { + hash.update(value); + await out.write(value); + } + } + } finally { + await out.close(); + } + const digest = hash.digest("base64"); + if (digest !== spec.sha512Base64) { + throw new Error(`sha512 mismatch (got ${digest.slice(0, 16)}..., pinned ${spec.sha512Base64.slice(0, 16)}...) — quarantining download`); + } + return await fsp.readFile(tmpTgz); +} + // --------------------------------------------------------------------------- // Minimal tgz extractor (devDeps are unavailable at runtime — bili ships a // self-contained bundle). Supports exactly what npm tarballs of the pinned diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index e33846db11..9f71260242 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -24,7 +24,7 @@ */ import type { BootstrapOutcome, CooldownMarker } from "./engine-bootstrap.js"; -import { COOLDOWN_MS, readBootstrapFailureMarker } from "./engine-bootstrap.js"; +import { COOLDOWN_MS, nodeSqliteUpgradeHint, readBootstrapFailureMarker } from "./engine-bootstrap.js"; import { fileStorageAutoDownload, fileStorageJsonWriter, fileStorageUnified, fileStorageWalJournalSizeLimit, resolveStorageMode } from "../knobs.js"; import { MAX_CONSECUTIVE_FAILURES, type UnifiedShadow } from "./shadow.js"; import type { StorageMode } from "./shadow.js"; @@ -85,7 +85,7 @@ function degradeReason(): string { // The driver's no-engine error surfaces through the same open path — // say what it actually is instead of "index database failed to open". if (/no SQLite engine available/i.test(e)) { - return "no SQLite engine available (node:sqlite missing, better-sqlite3 not installed)"; + return `no SQLite engine available (node:sqlite missing, better-sqlite3 not installed)${nodeSqliteUpgradeHint()}`; } return `index database failed to open (${e})`; } @@ -104,7 +104,7 @@ function degradeReason(): string { break; } } - return "no SQLite engine available (node:sqlite missing, better-sqlite3 not installed)"; + return `no SQLite engine available (node:sqlite missing, better-sqlite3 not installed)${nodeSqliteUpgradeHint()}`; } const COOLDOWN_CACHE_MS = 30_000; diff --git a/tests/acp-status-storage-degraded.test.ts b/tests/acp-status-storage-degraded.test.ts new file mode 100644 index 0000000000..eda94fd3b5 --- /dev/null +++ b/tests/acp-status-storage-degraded.test.ts @@ -0,0 +1,113 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import * as path from "node:path"; +import { handleAcpStatus } from "../src/acp-status.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { setSqliteEnginesUnavailableForTests } from "../src/storage/driver.ts"; +import { _resetStorageRuntimeStatusForTest, recordEngineBootstrap } from "../src/storage/runtime-status.ts"; +import { createCore, createInitialState, defaultConfig, type CompressionCore, type Config, type CoreMessage } from "acp-kernel"; +import type { Session } from "../src/session.ts"; +import { rmrf } from "./tmp-rm.ts"; + +/** + * #3385 (owner directive 3, acp_status surface): the STORAGE section of the + * acp_status tool must surface the runtime degrade reason verbatim — on + * Node < 22.5 with no engine it carries the explicit "upgrade Node.js" + * lever (see nodeSqliteUpgradeHint in src/storage/engine-bootstrap.ts), so + * the MODEL can relay the fix to the user instead of the fact living only + * in lane logs the user never opens. Healthy storage stays quiet and an + * intentional legacy opt-out is never nagged. + * + * The store is built for real (engines forced unavailable) so the recording + * seams fire exactly as they do in production; _setStoreForTest then keeps + * handleAcpStatus's ccrLoopConfig from rebuilding a healthy store mid-call + * and clobbering the degraded state (that clobber is also why the store is + * constructed BEFORE handleAcpStatus here). + */ + +function makeCtx(): { core: CompressionCore; config: Config; messages: CoreMessage[]; session: Session } { + const session = { id: "acp-status-3385", meta: {}, metadata: {}, stats: { requests: 0 }, state: createInitialState() } as unknown as Session; + return { core: createCore(), config: defaultConfig(200000), messages: [], session }; +} + +const ENV_KEYS = ["BILI_SESSIONS_DIR", "BILI_CONFIG_FILE", "XDG_CACHE_HOME", "XDG_STATE_HOME", "BILI_STORAGE_UNIFIED", "BILI_ENCRYPTION_KEY"] as const; + +function isolate(base: string): Record { + const snap: Record = {}; + for (const k of ENV_KEYS) snap[k] = process.env[k]; + const sessions = path.join(base, "sessions"); + mkdtempSync(sessions); // replace mkdirSync recursive — sessions parent exists + process.env.BILI_SESSIONS_DIR = sessions; + process.env.BILI_CONFIG_FILE = path.join(base, "absent-config.json"); + process.env.XDG_CACHE_HOME = path.join(base, "cache"); + process.env.XDG_STATE_HOME = path.join(base, "state"); + delete process.env.BILI_STORAGE_UNIFIED; + delete process.env.BILI_ENCRYPTION_KEY; + return snap; +} + +function restore(snap: Record): void { + for (const k of ENV_KEYS) { + if (snap[k] === undefined) delete process.env[k]; + else process.env[k] = snap[k]; + } +} + +test("#3385: acp_status STORAGE section surfaces the degrade reason so the model can relay the lever", () => { + const base = mkdtempSync(path.join(tmpdir(), "acp-status-3385-")); + const snap = isolate(base); + let store: SessionStore | null = null; + try { + setSqliteEnginesUnavailableForTests(true); + _resetStorageRuntimeStatusForTest(); + recordEngineBootstrap({ status: "failed", detail: "The operation was aborted due to timeout" }); + store = new SessionStore({ dir: process.env.BILI_SESSIONS_DIR, enabled: true }); + _setStoreForTest(store); + + const out = handleAcpStatus({}, makeCtx()); + const text = out.text ?? ""; + assert.ok(text.includes("STORAGE —"), "the storage line is present"); + assert.ok(text.includes("⚠ degraded —"), `the degrade warning line is present (got: ${text.slice(-400)})`); + assert.ok(text.includes("no SQLite engine available"), "the reason names the missing engine"); + assert.ok(text.includes("(relay this to the user)"), "the model is told to relay it"); + // The Node >= 22.5 lever is part of the reason string ONLY on Nodes + // that lack the builtin (nodeSqliteUpgradeHint is empty on >= 22.5 — + // pinned separately in tests/storage-engine-bootstrap.test.ts). + if (Number(process.versions.node.split(".")[0]) < 22 || (Number(process.versions.node.split(".")[0]) === 22 && Number(process.versions.node.split(".")[1]) < 5)) { + assert.ok(/Node\.js >= 22\.5/.test(text), "this Node lacks the builtin — the reason must carry the upgrade lever"); + } + } finally { + store?.closeQuietly(); + _setStoreForTest(null as unknown as SessionStore); + setSqliteEnginesUnavailableForTests(false); + _resetStorageRuntimeStatusForTest(); + restore(snap); + rmrf(base); + } +}); + +test("#3385: healthy storage stays quiet — no degrade line", () => { + // No engine forcing: on any runner the builtin/host/cached ladder serves + // a real engine, the sidecar opens, and the STORAGE line must NOT be + // followed by a degrade warning. + const base = mkdtempSync(path.join(tmpdir(), "acp-status-3385-ok-")); + const snap = isolate(base); + let store: SessionStore | null = null; + try { + _resetStorageRuntimeStatusForTest(); + store = new SessionStore({ dir: process.env.BILI_SESSIONS_DIR, enabled: true }); + _setStoreForTest(store); + const out = handleAcpStatus({}, makeCtx()); + const text = out.text ?? ""; + assert.ok(text.includes("STORAGE —"), "the storage line is present"); + assert.ok(!text.includes("⚠ degraded"), `healthy storage shows no degrade warning (got: ${text.slice(-400)})`); + } finally { + store?.closeQuietly(); + _setStoreForTest(null as unknown as SessionStore); + _resetStorageRuntimeStatusForTest(); + restore(snap); + rmrf(base); + } +}); diff --git a/tests/storage-engine-bootstrap.test.ts b/tests/storage-engine-bootstrap.test.ts index 0af9555c9d..d0f5a3c6e9 100644 --- a/tests/storage-engine-bootstrap.test.ts +++ b/tests/storage-engine-bootstrap.test.ts @@ -44,10 +44,12 @@ import { BOOTSTRAP_TIMEOUT_MS, ensureSqliteEngine, extractTgzToDir, + nodeSqliteUpgradeHint, platformTargetKey, resetEngineBootstrapForTests, setBootstrapTimeoutMsForTests, setEngineBootstrapSpecForTests, + setEngineMirrorsForTests, setPlatformKeyForTests, type BootstrapOutcome, } from "../src/storage/engine-bootstrap.ts"; @@ -201,6 +203,7 @@ function resetAllSeams(): void { setSqliteProbeOverridesForTests(null); clearSelfDownloadedEngineForTests(); setEngineBootstrapSpecForTests(null); + setEngineMirrorsForTests(null); setPlatformKeyForTests(null); setBootstrapTimeoutMsForTests(null); resetEngineBootstrapForTests(); @@ -209,13 +212,19 @@ function resetAllSeams(): void { const UNIFIED_ENV = { BILI_STORAGE_UNIFIED: "full" }; async function withRegistry(base: string, fn: () => Promise): Promise { + // #3338: the engine download now walks a mirror list (configured first, + // then builtin npm + npmmirror). Pin the list to this one mock so the + // pre-mirror-era tests stay hermetic — a failing configured mock must + // fail the bootstrap instead of falling through to the real registries. const prev = process.env.BILI_UPDATE_REGISTRY; process.env.BILI_UPDATE_REGISTRY = base; + setEngineMirrorsForTests([base]); try { await fn(); } finally { if (prev === undefined) delete process.env.BILI_UPDATE_REGISTRY; else process.env.BILI_UPDATE_REGISTRY = prev; + setEngineMirrorsForTests(null); } } @@ -760,3 +769,107 @@ test("platformTargetKey: musl detection mirrors binding.js (glibc presence decid assert.equal(platformTargetKey({ platform: "darwin", arch: "arm64" }), "darwin-arm64"); assert.equal(platformTargetKey({ platform: "win32", arch: "x64" }), "win32-x64"); }); + +// --------------------------------------------------------------------------- +// Multi-mirror engine download (#3338 directive 2) + Node upgrade hint (directive 3) +// --------------------------------------------------------------------------- + +test("mirror fallback: a failing first mirror falls to the next and the bootstrap succeeds (#3338)", async () => { + const cachePrev = isolateCache(); + let bad: MockRegistry | null = null; + let good: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + bad = await startMockRegistry(tgz, 500); // registry up, tarball broken at the HTTP layer + good = await startMockRegistry(tgz); + setEngineMirrorsForTests([bad.base, good.base]); + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `expected fallback success, got ${JSON.stringify(out)}`); + assert.ok(bad.requests() >= 1, "the failing mirror was attempted first"); + assert.ok(good.requests() >= 1, "the fallback mirror served the tarball"); + assert.ok(fs.existsSync(path.join(cachePrev, "billion-context", "sqlite", "better-sqlite3-13.0.3", "package.json")), "fallback download materialized"); + } finally { + await bad?.close(); + await good?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("mirror exhaustion: the cooldown reason lists every mirror attempted (#3338)", async () => { + const cachePrev = isolateCache(); + let bad1: MockRegistry | null = null; + let bad2: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + bad1 = await startMockRegistry(tgz, 500); + bad2 = await startMockRegistry(tgz, 503); + setEngineMirrorsForTests([bad1.base, bad2.base]); + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "failed", `expected exhaustion failure, got ${JSON.stringify(out)}`); + const detail = out.detail ?? ""; + assert.ok(detail.includes(bad1.base), `reason must name the first mirror: ${detail}`); + assert.ok(detail.includes(bad2.base), `reason must name the second mirror: ${detail}`); + assert.match(detail, /tried mirrors: /, "reason enumerates the mirror list"); + const marker = path.join(cachePrev, "billion-context", "sqlite", ".bootstrap-failed.json"); + assert.ok(fs.existsSync(marker), "exhaustion arms the cooldown"); + } finally { + await bad1?.close(); + await bad2?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("mirror list: a configured registry is attempted before the builtins (#3338)", async () => { + const cachePrev = isolateCache(); + let reg: MockRegistry | null = null; + try { + resetAllSeams(); + const tgz = fakePackageTgz(); + setEngineBootstrapSpecForTests({ version: "13.0.3", sha512Base64: sha512B64(tgz) }); + setSqliteProbeOverridesForTests({ builtin: false, host: false }); + reg = await startMockRegistry(tgz); + // Deliberately NO mirrors override: the production list must put the + // configured registry first. The mock SUCCEEDS, so the loop breaks + // before ever reaching the builtin production registries (no network). + // If the ordering regressed (builtins first), CI's real egress would + // serve the tarball and this mock would record zero requests. + const prev = process.env.BILI_UPDATE_REGISTRY; + process.env.BILI_UPDATE_REGISTRY = reg.base; + try { + const out = await ensureSqliteEngine({ env: { ...UNIFIED_ENV } }); + assert.equal(out.status, "downloaded", `expected configured-first success, got ${JSON.stringify(out)}`); + assert.equal(reg!.requests(), 1, "the configured registry was the one (and only) mirror hit"); + } finally { + if (prev === undefined) delete process.env.BILI_UPDATE_REGISTRY; + else process.env.BILI_UPDATE_REGISTRY = prev; + } + } finally { + await reg?.close(); + resetAllSeams(); + restoreCache(cachePrev); + rmrf(cachePrev); + } +}); + +test("nodeSqliteUpgradeHint: pins the Node >= 22.5 threshold and the levers (#3338 directive 3)", () => { + assert.equal(nodeSqliteUpgradeHint("22.5.0"), "", "22.5.0 ships node:sqlite — no hint"); + assert.equal(nodeSqliteUpgradeHint("23.0.0"), "", "23.x ships node:sqlite — no hint"); + assert.equal(nodeSqliteUpgradeHint("24.11.0"), "", "24.x ships node:sqlite — no hint"); + const old = nodeSqliteUpgradeHint("20.20.2"); + assert.match(old, /Node\.js >= 22\.5/, "the #3385 node version gets the upgrade lever"); + assert.match(old, /npmmirror/, "the hint names the builtin mirrors"); + assert.match(old, /BILI_UPDATE_REGISTRY/, "the hint names the registry override"); + assert.match(nodeSqliteUpgradeHint("22.4.9"), /Node\.js >= 22\.5/, "22.4.x predates node:sqlite"); + assert.match(nodeSqliteUpgradeHint("19.9.9"), /Node\.js >= 22\.5/, "19.x predates node:sqlite"); + assert.equal(nodeSqliteUpgradeHint("22.5.1"), "", "22.5.1 ships node:sqlite — no hint"); +}); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 4df4cb4870..f12948a09c 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -243,8 +243,8 @@ entries: default: "true" env: BILI_SQLITE_AUTO_DOWNLOAD group: process - desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download." - desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。" + desc_en: "Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385)." + desc_zh: "SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。" - key: storage.jsonWriter type: "on|off" default: "on"