diff --git a/docs/configuration.md b/docs/configuration.md index 45bea523e4..f7a2439279 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -207,7 +207,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is FULLY WIRED (steps 2+3): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path is wired through the async shadow facade: saves enqueue onto a serial backend queue (watermark deltas, orphan/reap journal, digest sampling — sqlite keeps its zero-await sync fast path unchanged), SYNC reads ride the JSON envelope (the sidecar attaches asynchronously via ready(); viewFor/importRecord sync facades miss by design), replicas/ext/GC/export/fork speak the async twins (#4911: the fork lane awaits lastLiveSeq + the row clone through them; the sync window reads for fold-anchor backfill / snapshot render stay envelope-served by design). writerOff stays sqlite-only (the envelope is the mysql sync read path). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. | | `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (hot path included — the async shadow queue). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | -| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | +| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session CCR content-store sidecar stops being written as well - in full mode the index's ccr mirror is consulted first on every read, so the mirror is the durable copy. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | diff --git a/docs/configuration.zh-CN.md b/docs/configuration.zh-CN.md index c6d3c537c9..01a4d3a61c 100644 --- a/docs/configuration.zh-CN.md +++ b/docs/configuration.zh-CN.md @@ -207,7 +207,7 @@ | `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已完整接线(第二+三步):openBackend("mysql", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径经异步影子 facade 接线:保存进串行后端队列(水位 delta、孤儿/收割台账、digest 采样 —— sqlite 保持零 await 同步快车道不变),同步读走 JSON 信封(sidecar 经 ready() 异步挂载;viewFor/importRecord 同步面按设计 miss),replicas/ext/GC/export/fork 走异步孪生(#4911:fork 车道经它们 await lastLiveSeq + 行克隆;fold-anchor 回填/快照渲染用的同步窗口读按设计仍走信封)。writerOff 仅 sqlite(信封就是 mysql 的同步读路径)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。 | | `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | storage.backend="mysql" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(含热路径 —— 异步影子队列)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | -| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | +| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 CCR content-store 边车也不再写 —— full 模式下读取时先查索引里的 ccr 镜像,镜像才是持久副本。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | diff --git a/src/config.ts b/src/config.ts index b65c3d544f..f25212e7da 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2226,10 +2226,11 @@ type FileConfig = { * wins. The proxy hot path stays on sqlite until the step-3 async * swap; the boot notice names that boundary. */ mysql?: { dsn?: string }; - /** Phase 3 (#2960): "off" stops writing the legacy JSON envelope while - * storage.unified is "full" — the SQLite index becomes the only - * persistent copy (reads were already unified-first; boot enumerates - * the index union). Ignored with a one-time warning under + /** Phase 3 (#2960): "off" stops writing the legacy JSON envelope AND + * the per-session CCR content-store sidecar while storage.unified is + * "full" — the SQLite index (session row + ccr mirror) becomes the + * only persistent copy (reads were already unified-first; boot + * enumerates the index union). Ignored with a one-time warning under * shadow/legacy, where the JSON file remains the sole source of * truth. Env BILI_STORAGE_JSON_WRITER wins; default "on". Rollback = * unset (JSON writing resumes; index-only sessions keep loading via diff --git a/src/persist.ts b/src/persist.ts index fe03806106..24fb99f43b 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -844,6 +844,23 @@ export class SessionStore { session.contentStoreDirty = false; return true; } + if (this.writerOffActive()) { + // #2960 Phase 3 extension: the off switch covers the CCR + // content-store sidecar as well — in effective-full mode + // the unified ccr mirror (ingested by this same save's + // builder, which runs right after this gate returns) is the + // durable copy, and contentStoreOf() reads it first, so the + // file is never consulted. contentStoreDirty is cleared for + // the same reason: this save drops the resident store right + // after the ingest (dropResidentContentStore), and a stale + // dirty=true with contentStore=null would make a later + // writer-resume save hit the emptied-store branch and + // DELETE the sidecar file while the table may be dead. + // Convergence after a writer-resume comes from the kernel + // re-adopting its in-memory store on the next turn. + session.contentStoreDirty = false; + return true; + } mkdirSync(path.dirname(abs), { recursive: true }); const text = JSON.stringify(session.contentStore); durableWriteFile(tmp, this.codec ? this.codec.encode(text) : text, 0o600); diff --git a/tests/storage-json-writer-off-json-writes.test.ts b/tests/storage-json-writer-off-json-writes.test.ts new file mode 100644 index 0000000000..0c54252cf3 --- /dev/null +++ b/tests/storage-json-writer-off-json-writes.test.ts @@ -0,0 +1,254 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { SessionStore, contentStoreRelPathFor, relPathFor, storageStatusLabel } from "../src/persist.ts"; + + +/** + * #2960 Phase 3 / #5988 contract gate: with `storage.jsonWriter = "off"` + * (unified full), the sessions directory may gain a .json file ONLY through + * the whitelisted materialization paths below. The off switch promises "the + * SQLite index is the ONLY persistent copy" — any code that starts writing + * JSON there (the CCR content-store sidecar used to, #5988) is a regression + * of that contract and must fail CI, not wait for a human audit. + * + * Mechanism: snapshot diff, not fs interception. Before and after each phase + * every *.json file in the store dir is hashed; added/modified entries are + * checked against the whitelist. Whatever API the write path uses + * (writeFileSync, rename, fd writes) shows up in the diff, so a new writer + * cannot hide behind an un-intercepted syscall. + * + * Phases: + * A first save of a fresh store → ZERO .json writes + * B restart, unified-first adopt + save → ZERO .json writes + * C replica + restore → the restored id's own + * materialized files ONLY + * D control: switch ON, same first save → the snapshot MUST see writes + * (guards against a vacuous gate) + * + * The whitelist: the ONLY .json writes off mode may make are the + * materialization files of a restored id (its portable envelope + its CCR + * sidecar), named by relPathFor / contentStoreRelPathFor — the index is not + * a restore source, and it does not carry forked/restored CCR stores. Any + * .json write outside that set (including one for a NON-restored id) is a + * contract regression and fails here. Adding a new legitimate writer: name + * the code path (file:function) and why the index cannot carry the bytes in + * JSON_WRITE_WHITELIST and in the PR description. + * + * Scope: the sessions directory (this store). CLI-level state files under + * stateDir() are a separate concern (#5757 migrates them into kv_store). + */ + +// Same hermeticity pin as the sibling storage suites: pin BOTH tiers so each +// leg controls its own mode explicitly (absent now defaults to full, #3818). +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jwgate-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +/** The categories of .json writes the off mode is allowed to make in the + * sessions dir. Each entry names the code path and why the index cannot + * carry the bytes. The concrete per-run file set is derived from the + * restored id via materializedFilesFor() — session FILES are named by a + * hash of the id (relPathFor), never by the id itself, so a regex over file + * names cannot anchor a restore. */ +const JSON_WRITE_WHITELIST: ReadonlyArray<{ re: RegExp; reason: string }> = [ + // restoreReplica materialization (persist.ts restoreReplica → kernel + // store.writeNow + writeContentStoreSidecar): the index is not a + // restore source, and it does not carry forked/restored CCR stores — + // buildRecord drops forkContentStore on the next live save, so the + // sidecar is the durable copy for the restored id. + { re: /^restore-.*\.json$/, reason: "restore materialization (portable envelope + CCR sidecar)" }, +]; + +/** The concrete whitelist set for one restore: the restored id's envelope + * and CCR sidecar, named exactly as the writers name them. */ +function materializedFilesFor(id: string, protocol: string, origin: string): Set { + return new Set([ + relPathFor(id, protocol, origin).split(path.sep).join("/"), + contentStoreRelPathFor(id, protocol, origin).split(path.sep).join("/"), + ]); +} + +const CORPUS_ROOT = path.resolve(import.meta.dirname, "golden/storage-corpus"); + +function loadCorpusCase(dir: string, name: string, id: string, protocol: string) { + fs.cpSync(path.join(CORPUS_ROOT, name), path.join(dir, name), { recursive: true }); + const loader = new SessionStore({ dir: path.join(dir, name), debounceMs: 1, enabled: true }); + const loaded = loader.loadSync(id, { protocol, upstreamOrigin: "http://127.0.0.1:8199" }); + loader.cancelAll(); + loader.closeQuietly(); + assert.ok(loaded, `${name}: corpus loadSync returned null`); + return loaded; +} + +/** relpath → sha256 of every *.json file under dir (sidecars included). */ +function jsonSnapshot(dir: string): Map { + const out = new Map(); + const walk = (p: string) => { + for (const e of fs.readdirSync(p, { withFileTypes: true })) { + const full = path.join(p, e.name); + if (e.isDirectory()) walk(full); + else if (e.name.endsWith(".json")) + out.set(path.relative(dir, full).split(path.sep).join("/"), + createHash("sha256").update(fs.readFileSync(full)).digest("hex")); + } + }; + walk(dir); + return out; +} + +function jsonDiff(before: Map, after: Map) { + const added: string[] = []; + const modified: string[] = []; + const removed: string[] = []; + for (const [f, h] of after) { + if (!before.has(f)) added.push(f); + else if (before.get(f) !== h) modified.push(f); + } + for (const f of before.keys()) if (!after.has(f)) removed.push(f); + return { added, modified, removed }; +} + +function whitelistText(): string { + return JSON_WRITE_WHITELIST.map((w) => `${w.re} — ${w.reason}`).join("\n"); +} + +/** Strict leg: a phase that may not touch ANY .json file. */ +function assertNoJsonWrites(phase: string, diff: ReturnType): void { + const offenders = [...diff.added, ...diff.modified]; + assert.deepEqual(offenders.sort(), [], + `${phase}: jsonWriter=off must write NO .json files into the sessions dir — offenders:\n ${offenders.join("\n ")}\nWhitelist (allowed only in the restore phase):\n${whitelistText()}`); +} + +/** Restore leg: every write must be one of the restored id's own + * materialized files. */ +function assertWhitelistedJsonWrites(phase: string, diff: ReturnType, allowed: Set): void { + const offenders = [...diff.added, ...diff.modified].filter((f) => !allowed.has(f)).sort(); + assert.deepEqual(offenders, [], + `${phase}: .json write outside the whitelist:\n ${offenders.join("\n ")}\nWhitelist categories:\n${whitelistText()}\nAllowed files for this restore:\n ${[...allowed].join("\n ")}\nIf the write is legitimate, extend JSON_WRITE_WHITELIST with the code path and the reason the index cannot carry the bytes.`); +} + +/** Windows: an ephemeral stray handle must not fail CI — best-effort delete + * with bounded retries, swallowed on persistent EBUSY/EPERM. */ +function rmBestEffort(p: string): void { + try { + fs.rmSync(p, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + /* leaked temp dir on a Windows runner is harmless */ + } +} + +test("jsonWriter=off: only whitelisted .json files may be written in the sessions dir", async () => { + const prevMode = process.env.BILI_STORAGE_UNIFIED; + process.env.BILI_STORAGE_UNIFIED = "full"; + process.env.BILI_STORAGE_JSON_WRITER = "off"; + try { + assert.equal(storageStatusLabel(), "storage: full (unified-only, json envelope off)"); + const seed = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jwgate-seed-")); + // The ccr corpus case: a non-trivial content store, so any CCR write + // regression shows up as bytes in the diff. + const session = loadCorpusCase(seed, "ccr", "corpus-ccr", "responses"); + session.contentStore = { + version: 1, + byHash: { h1: "THE ORIGINAL LONG TEXT" }, + byRef: { m00001: { hash: "h1", rawId: "raw-1", kind: "file read", tokens: 5, chars: 22, head: "ORIGINAL HEAD" } }, + } as typeof session.contentStore; + session.contentStoreDirty = true; + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jwgate-full-")); + try { + // Phase A: first save of a fresh store — the index-only landing. + { + const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); + const before = jsonSnapshot(dir); + await store.writeNow(session); + store.cancelAll(); + store.closeQuietly(); + assertNoJsonWrites("phase A (first save)", jsonDiff(before, jsonSnapshot(dir))); + assert.ok(fs.existsSync(path.join(dir, "index.db")), "the durable copy must be the index"); + } + + // Phase B: restart — unified-first adopt of the index-only + // session, then a save that does real work. + { + const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); + const before = jsonSnapshot(dir); + const adopted = store.loadSync(session.id, { protocol: "responses", upstreamOrigin: "http://127.0.0.1:8199" }); + assert.ok(adopted, "index-only session must adopt after restart"); + await store.writeNow(adopted); + store.cancelAll(); + store.closeQuietly(); + assertNoJsonWrites("phase B (restart adopt + save)", jsonDiff(before, jsonSnapshot(dir))); + } + + // Phase C: replica + restore — the ONLY phase allowed .json + // writes, and only the whitelisted materialization. + { + const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); + const before = jsonSnapshot(dir); + await store.loadSync(session.id, { protocol: "responses", upstreamOrigin: "http://127.0.0.1:8199" }); + const cp = await store.createReplica(session.id, { label: "json-writer-off-gate" }); + // NO sessionId — the default `restore-` id. + const restored = await store.restoreReplica(cp.cp_id); + assert.ok(restored.sessionId.startsWith("restore-"), "default restore id must start with restore-"); + assert.equal(restored.restoredFrom, session.id); + store.cancelAll(); + store.closeQuietly(); + const diff = jsonDiff(before, jsonSnapshot(dir)); + const allowed = materializedFilesFor(restored.sessionId, "responses", "http://127.0.0.1:8199"); + assertWhitelistedJsonWrites("phase C (replica + restore)", diff, allowed); + // Portability is the reason the materialization is + // whitelisted — it must actually happen, not just be allowed. + assert.ok(diff.added.includes(relPathFor(restored.sessionId, "responses", "http://127.0.0.1:8199").split(path.sep).join("/")), + "restore must materialize a portable JSON envelope"); + } + + // No transient files may be left behind by any phase. + const leftover: string[] = []; + const walk = (p: string) => { + for (const e of fs.readdirSync(p, { withFileTypes: true })) { + const full = path.join(p, e.name); + if (e.isDirectory()) walk(full); + else if (e.name.startsWith(".tmp-")) leftover.push(full); + } + }; + walk(dir); + assert.deepEqual(leftover, [], "no .tmp-* transients may remain"); + } finally { + rmBestEffort(dir); + rmBestEffort(seed); + } + } finally { + delete process.env.BILI_STORAGE_JSON_WRITER; + process.env.BILI_STORAGE_UNIFIED = prevMode; + } +}); + +test("control: with jsonWriter=on the same first save DOES write .json (gate is not vacuous)", async () => { + const prevMode = process.env.BILI_STORAGE_UNIFIED; + process.env.BILI_STORAGE_UNIFIED = "full"; + delete process.env.BILI_STORAGE_JSON_WRITER; // switch on + try { + const seed = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jwgate-seed-")); + const session = loadCorpusCase(seed, "folded", "corpus-folded", "anthropic"); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jwgate-on-")); + try { + const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); + const before = jsonSnapshot(dir); + await store.writeNow(session); + store.cancelAll(); + store.closeQuietly(); + const diff = jsonDiff(before, jsonSnapshot(dir)); + assert.ok([...diff.added, ...diff.modified].length > 0, + "the on-control must produce .json writes — an empty diff means the snapshot mechanism is broken"); + } finally { + rmBestEffort(dir); + rmBestEffort(seed); + } + } finally { + process.env.BILI_STORAGE_UNIFIED = prevMode; + } +}); diff --git a/tests/storage-json-writer-off.test.ts b/tests/storage-json-writer-off.test.ts index eca1f87781..9d7f8a860c 100644 --- a/tests/storage-json-writer-off.test.ts +++ b/tests/storage-json-writer-off.test.ts @@ -9,8 +9,10 @@ import { UnifiedStore } from "../src/storage/store.ts"; /** * Phase 3 (#2960): `storage.jsonWriter = "off"` — while the unified store - * runs in full mode, the legacy JSON envelope is no longer written; the - * SQLite index is the only persistent copy. Reads were already + * runs in full mode, the legacy JSON envelope AND the per-session CCR + * content-store sidecar are no longer written; the SQLite index (session + * row + ccr mirror, ingested from the same record) is the only persistent + * copy. Reads were already * unified-first (#2671 Phase 1.5) and boot enumerates the index union * (#2992), so an index-only session must survive a store restart with its * folds intact. Safety legs: the switch is ignored (warned once) under @@ -37,10 +39,11 @@ function loadCorpusCase(dir: string, name: string, id: string, protocol: string) return loaded; } -/** Session envelopes live at /_.json; the per-session - * content-store sidecar (<...>.content-store.json) is intentionally still - * written under jsonWriter=off (cheap, keeps CCR portable), so the probe - * counts only real session records. */ +/** Session envelopes live at /_.json; the probe counts + * only real session records. (The CCR content-store sidecar is not + * written under jsonWriter=off either — asserted separately — and a + * replica/export re-materialization may legitimately add a NEW envelope + * name that an original-absence assertion must not see.) */ function envelopeFiles(dir: string): string[] { const out: string[] = []; const walk = (p: string) => { @@ -134,6 +137,90 @@ test("jsonWriter=off (full): no envelope written, index-only session survives re } }); +test("jsonWriter=off (full): CCR content-store sidecar is not written; the index mirror is the durable copy", async () => { + const prevMode = process.env.BILI_STORAGE_UNIFIED; + process.env.BILI_STORAGE_UNIFIED = "full"; + process.env.BILI_STORAGE_JSON_WRITER = "off"; + try { + const seed = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jw-ccr-seed-")); + const session = loadCorpusCase(seed, "ccr", "corpus-ccr", "responses"); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jw-ccr-full-")); + try { + // Lay the corpus envelope + sidecar out as THIS store's sessions + // dir (loadCorpusCase nests them under /ccr/). + fs.cpSync(path.join(seed, "ccr"), dir, { recursive: true }); + const sidecars: string[] = []; + const walk = (p: string) => { + for (const e of fs.readdirSync(p, { withFileTypes: true })) { + const full = path.join(p, e.name); + if (e.isDirectory()) walk(full); + else if (e.name.endsWith(".content-store.json")) sidecars.push(full); + } + }; + walk(dir); + assert.equal(sidecars.length, 1, "the ccr corpus case ships exactly one sidecar"); + const before = fs.readFileSync(sidecars[0]!, "utf8"); + // Graft a DIFFERENT content store: if the off leg ever wrote the + // sidecar, its bytes would change (the corpus store holds two + // original entries, the graft holds only h1). + session.contentStore = { + version: 1, + byHash: { h1: "THE ORIGINAL LONG TEXT" }, + byRef: { m00001: { hash: "h1", rawId: "raw-1", kind: "file read", tokens: 5, chars: 22, head: "ORIGINAL HEAD" } }, + } as typeof session.contentStore; + session.contentStoreDirty = true; + const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); + await store.writeNow(session); + const storeRec = store as unknown as { shadow?: unknown }; + assert.ok(storeRec.shadow, "the unified sidecar must be alive for the off switch to engage"); + store.cancelAll(); + store.closeQuietly(); + + // 1) The sidecar is byte-identical — the off gate skipped the write. + assert.equal(fs.readFileSync(sidecars[0]!, "utf8"), before, "jsonWriter=off must not write the CCR content-store sidecar"); + + // 2) The index is the durable copy: a fresh handle resolves the + // grafted originals from the unified ccr mirror (sessions.ccr_json). + const fresh = new SessionStore({ dir, debounceMs: 1, enabled: true }); + try { + const mirror = fresh.unifiedCcrFor(session.id); + assert.ok(mirror, "the unified ccr mirror must carry the store"); + assert.equal(mirror!.byHash["h1"], "THE ORIGINAL LONG TEXT", "CCR originals must survive via the index mirror"); + fresh.cancelAll(); + } finally { + fresh.closeQuietly(); + } + + // 3) Control: with the switch back on, the same kind of save + // DOES write the sidecar — proving the skip is exactly the + // off gate, not an incidental no-op. + delete process.env.BILI_STORAGE_JSON_WRITER; + const store2 = new SessionStore({ dir, debounceMs: 1, enabled: true }); + try { + const again = store2.loadSync(session.id, { protocol: "responses", upstreamOrigin: "http://127.0.0.1:8199" }); + assert.ok(again, "session must still load for the control leg"); + again.contentStore = { + version: 1, + byHash: { h2: "SHADOW WRITE" }, + byRef: { m00002: { hash: "h2", rawId: "raw-2", kind: "tool result", tokens: 3, chars: 12, head: "SHADOW HEAD" } }, + } as typeof again.contentStore; + again.contentStoreDirty = true; + await store2.writeNow(again); + assert.notEqual(fs.readFileSync(sidecars[0]!, "utf8"), before, "with jsonWriter=on the sidecar must be written"); + store2.cancelAll(); + } finally { + store2.closeQuietly(); + } + } finally { + rmBestEffort(dir); + rmBestEffort(seed); + } + } finally { + delete process.env.BILI_STORAGE_JSON_WRITER; + process.env.BILI_STORAGE_UNIFIED = prevMode; + } +}); + test("jsonWriter=off is ignored under shadow (JSON stays the sole source of truth)", async () => { const prevMode = process.env.BILI_STORAGE_UNIFIED; process.env.BILI_STORAGE_UNIFIED = "1"; // shadow diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 29e8811f1e..7ba754c171 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -370,8 +370,8 @@ entries: default: "on" env: BILI_STORAGE_JSON_WRITER group: process - desc_en: "Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes)." - desc_zh: "Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。" + desc_en: "Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session CCR content-store sidecar stops being written as well - in full mode the index's ccr mirror is consulted first on every read, so the mirror is the durable copy. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes)." + desc_zh: "Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 CCR content-store 边车也不再写 —— full 模式下读取时先查索引里的 ccr 镜像,镜像才是持久副本。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。" - key: storage.walJournalSizeLimitBytes type: number default: "8388608"