diff --git a/README.md b/README.md index 92526d0ebe..82180a62c6 100644 --- a/README.md +++ b/README.md @@ -729,6 +729,7 @@ bili sessions restore # bring an archived session back from the log l bili sessions foreign-blocks # (#5312) list fold blocks whose covered messages never belonged to this transcript — dry-run; --apply reversibly archives them, --restore : brings one back bili storage gc [--days N] [--vacuum full|incremental] [--dry-run] # print the sweep plan (sessions past retention, zero-ref content rows, ~pages reclaimable), then run the same gc() the HTTP endpoint calls; --dry-run stops after the plan +bili storage migrate [--dry-run] # (#5599) one-shot bulk import of every legacy JSON session into the unified SQLite index with per-session digest reconciliation (imported / indexed / drifted / failed); read-only over the JSON files — archive/delete them yourself once the report is clean; --dry-run plans without writing bili storage backup [--keep N] # online-consistent index.db snapshot into /backups/ (VACUUM INTO — # never rewrites the source, no exclusive lock, safe while a proxy runs); the # snapshot is quick_check-verified before it counts; keeps the newest N (default 3) @@ -740,7 +741,7 @@ bili storage repair [--from f.db] # with --from: verify the backup (quick_check # survives). Both swap paths refuse to run while another live process holds index.db.lock. ``` -`archive`/`restore`/`gc` require the unified store (`BILI_STORAGE_UNIFIED=sqlite|shadow|full` or `storage.unified` in the config file — the default is now pure sqlite) — legacy-only installs refuse with a hint. `--days` defaults to `storage.archiveAfterDays` (365). No scheduler: these run on demand. All commands accept `--json`. +`archive`/`restore`/`gc`/`migrate` require the unified store (`BILI_STORAGE_UNIFIED=sqlite|shadow|full` or `storage.unified` in the config file — the default is now pure sqlite) — legacy-only installs refuse with a hint. `--days` defaults to `storage.archiveAfterDays` (365). No scheduler: these run on demand. All commands accept `--json`. **Backup & repair (#7156, 2026-11-20 production incident).** A 3.2 GB index.db proved corrupt (foreign second writer, #7156) and the only recovery path was a hand-rolled `.recover`-equivalent plus manual file surgery — those two manual moves are now the `backup`/`repair` commands above. Operational discipline that keeps the incident from recurring: run `bili storage backup` on a schedule (the retention flag is the only state it keeps), and keep **exactly one** long-lived bili process per sessions dir — a second live writer is the corruption mode, and every storage command now detects a live foreign writer and refuses rather than contending. diff --git a/README.zh-CN.md b/README.zh-CN.md index ee02e832f4..2586bc5cee 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -503,11 +503,13 @@ bili sessions ls # id、label、解码后大小、最后更新 bili sessions rm # 完整删除(释放内容引用计数,含媒体);随后收缩 index.db(--no-vacuum 跳过) bili sessions archive # 导出到 archive/archive.jsonl + 清行;tombstone 保留 label 与身份 bili sessions restore # 从日志行恢复已归档的会话 +bili sessions foreign-blocks # (#5312) 列出折叠块覆盖的消息从未属于本会话的记录 —— dry-run;--apply 可逆地归档它们,--restore : 找回其中一条 bili storage gc [--days N] [--vacuum full|incremental] [--dry-run] # 先打印清扫计划(超期会话、零引用内容行、可回收页数估算),再执行与 HTTP 端点相同的 gc();--dry-run 只打印计划 +bili storage migrate [--dry-run] # (#5599) 一次性把全部 legacy JSON 会话批量导入统一 SQLite 索引,逐会话做摘要对账(imported / indexed / drifted / failed);对 JSON 文件只读 —— 报告干净后归档/删除由你自己决定;--dry-run 只打印计划不写入 ``` -`archive`/`restore`/`gc` 需要统一存储(`BILI_STORAGE_UNIFIED=sqlite|shadow|full` 或配置文件里的 `storage.unified` —— 缺省现在就是纯 sqlite)—— legacy-only 安装会拒绝并给出提示。`--days` 默认取 `storage.archiveAfterDays`(365)。没有调度器:全部按需运行。所有命令都支持 `--json`。 +`archive`/`restore`/`gc`/`migrate` 需要统一存储(`BILI_STORAGE_UNIFIED=sqlite|shadow|full` 或配置文件里的 `storage.unified` —— 缺省现在就是纯 sqlite)—— legacy-only 安装会拒绝并给出提示。`--days` 默认取 `storage.archiveAfterDays`(365)。没有调度器:全部按需运行。所有命令都支持 `--json`。 **大库上 full vacuum 是离线操作(#4612)。** `--vacuum full`(默认)会在独占写锁下重写整个 index.db —— O(库大小),多 GB 的库要几分钟到几十分钟,且没有中间进度(SQLite 的 VACUUM 是原子操作,拆不开)。在有代理正从同一会话目录服务时跑它,会把那个代理的写入饿死到其存储层降级为止,所以 `storage gc --vacuum full` **在另一个存活进程持有该数据库时会拒绝启动**(`index.db.lock` pid 纪律,#4175):CLI 以退出码 1 给出可操作提示,面板端点返回 HTTP 409。先停掉另一个实例(或等它退出)再重试;要在线安全地缩库用 `--vacuum incremental`,它以短事务回收空闲页、不拿独占锁。零引用删除清扫无论哪种模式都分批执行并逐批打印进度。 diff --git a/docs/configuration.md b/docs/configuration.md index a2ba64df70..46f818c9bc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -214,7 +214,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960) — default flipped this release: UNSET now defaults to OFF for the pure-sqlite selection (storage.unified absent or "sqlite") — the SQLite index is the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Explicit full keeps the historical #3818 dual-write default (envelopes stay written unless this key says otherwise). An explicit on/off at either tier always wins in effective-full mode: under shadow/legacy (or with encryption active) a requested off is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings ("off"/"on"). Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. To restore dual-write: set storage.jsonWriter=on (or BILI_STORAGE_JSON_WRITER=1) — index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session CCR content-store sidecar stops being written as well - in full mode the index's ccr mirror is consulted first on every read, so the mirror is the durable copy. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | -| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier. | +| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore , and `bili storage migrate [--dry-run]` (#5599: one-shot bulk import of every legacy JSON session into the unified index with per-session digest reconciliation — read-only over the JSON files; README 'Storage maintenance CLI' section). Env wins over the file tier. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. | diff --git a/docs/configuration.zh-CN.md b/docs/configuration.zh-CN.md index 94d38a80f2..90341c4536 100644 --- a/docs/configuration.zh-CN.md +++ b/docs/configuration.zh-CN.md @@ -214,7 +214,7 @@ | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960)—— 本次发布翻转缺省:UNSET(未设置)现在对纯 SQLite 选择(storage.unified 未设置或 "sqlite")缺省为 OFF —— SQLite 索引成为唯一持久副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。显式 full 保留 #3818 的历史双写缺省(除非本键另说,信封照写)。有效 full 模式下任一层的显式 on/off 总是生效:shadow/legacy(或加密激活)下请求的 off 会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层("off"/"on")。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。恢复双写:设 storage.jsonWriter=on(或 BILI_STORAGE_JSON_WRITER=1)—— 纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 CCR content-store 边车也不再写 —— full 模式下读取时先查索引里的 ccr 镜像,镜像才是持久副本。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | -| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。 | +| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore ,还有 `bili storage migrate [--dry-run]`(#5599:一次性把全部 legacy JSON 会话批量导入统一索引并逐会话摘要对账——对 JSON 文件只读;README「存储维护 CLI」一节)。环境变量优先于配置文件层。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 | diff --git a/scripts/check-dist-external.mjs b/scripts/check-dist-external.mjs index fd71a56127..7abe2d6e19 100644 --- a/scripts/check-dist-external.mjs +++ b/scripts/check-dist-external.mjs @@ -85,6 +85,14 @@ const STATIC_REF_PATTERNS = [ ]; const DYNAMIC_REF_PATTERN = /\bimport\s*\(\s*["']([^"'\n]+)["']\s*\)/g; +// A capture outside the module-specifier character set is a regex artifact, +// not an import: #5636 — an error message ending in the word "import" fused +// its closing quote with the next string literal on the same MINIFIED line +// into a bogus side-effect ref (`static );decodedBytes=... @ line 935`). +// Every char below cannot occur in a resolvable bare specifier, so dropping +// such captures cannot hide a real external import. +const NOT_A_SPECIFIER = /[\s"'`(){}[\];=<>$\\|&!,#`]/; + function lineOf(text, index) { return text.slice(0, index).split("\n").length; } @@ -96,6 +104,7 @@ function collectRefs(text, re, kind) { while ((m = re.exec(text)) !== null) { const spec = m[1]; if (!spec || spec.startsWith(".") || spec.startsWith("/")) continue; + if (NOT_A_SPECIFIER.test(spec)) continue; const base = spec.replace(/^node:/, "").split("/")[0]; if (NODE_BUILTINS.has(base)) continue; refs.push({ spec, kind, line: lineOf(text, m.index) }); diff --git a/src/cli.ts b/src/cli.ts index 475738511c..0951e6a1be 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -21,6 +21,7 @@ * bili acp-cache diff attribute cache breaks from ACP_DUMP_BODY dumps * bili sessions ls|rm|archive|restore [id] manage persisted sessions * bili storage gc [--days N] run the storage sweep now (--dry-run plans only) + * bili storage migrate bulk-import legacy JSON sessions into the unified index (#5599) * bili storage backup [--keep N] consistent index.db snapshot (+retention) * bili storage repair [--from ] restore from backup / salvage no-backup * bili pack install install a custom compress prompt pack from npm (#5319) @@ -56,7 +57,7 @@ import { runMcpStdio } from "./mcp.js"; import { PLUGIN_AGENTS, isPluginAgent, pluginInstall, pluginRemove, pluginStatusAll, pluginUpdate, renderPluginList, warnActivePluginSessions, type PluginAgent } from "./plugin-install.js"; import { runLaunch, runTestPi, isLaunchClient, type ClientName } from "./launcher.js"; import { exportSession } from "./export.js"; -import { runSessionsCommand, runStorageGcCommand } from "./storage-maintenance.js"; +import { runSessionsCommand, runStorageGcCommand, runStorageMigrateCommand } from "./storage-maintenance.js"; import { runStorageBackupCommand, runStorageRepairCommand } from "./storage-backup-repair.js"; import { installPackFromNpm, removePackFile } from "./pack-install.js"; import { listResolvedPacks } from "./compress-settings.js"; @@ -118,6 +119,13 @@ Usage: release unreferenced content, shrink index.db (--days N overrides the age; --vacuum full|incremental; prints the sweep plan first, --dry-run stops there) + bili storage migrate bulk-import every legacy JSON session into the unified + SQLite index with per-session digest reconciliation (#5599): + imports what is missing or newer than the index, then + verifies every session against its source file and lists + drifted/failed ones. Read-only over the JSON files + (--dry-run plans without writing; --json machine-readable). + Safest while no host process is running. bili storage backup online-consistent index.db snapshot into /backups/ (VACUUM INTO; verified with quick_check; --keep N retention, default 3) — safe while the proxy runs @@ -255,7 +263,7 @@ type Parsed = { storageDays?: number; storageVacuum?: "full" | "incremental"; storageDryRun?: boolean; - storageSub?: "gc" | "backup" | "repair"; + storageSub?: "gc" | "migrate" | "backup" | "repair"; storageKeep?: number; storageFrom?: string; jsonOutput?: boolean; @@ -619,8 +627,8 @@ export function parseArgs(argv: string[]): Parsed { // A bare `bili storage` stays an error (the existing contract, // pinned by cli-parseargs): gc archives real sessions, so no // action may be the silent default. - if (sub !== "gc" && sub !== "backup" && sub !== "repair") { - console.error(`bili storage: unknown action "${sub ?? ""}" (try "bili storage gc|backup|repair")`); + if (sub !== "gc" && sub !== "migrate" && sub !== "backup" && sub !== "repair") { + console.error(`bili storage: unknown action "${sub ?? ""}" (try "bili storage gc|migrate|backup|repair")`); process.exit(2); } storageSub = sub as Parsed["storageSub"]; @@ -824,7 +832,9 @@ export async function main(): Promise { try { const r = command === "sessions" ? await runSessionsCommand({ sub: sessionsSub!, target: sessionsTarget, json: jsonOutput, noVacuum: sessionsNoVacuum, apply: sessionsApply, restoreTarget: sessionsRestoreTarget }) - : storageSub === "backup" + : storageSub === "migrate" + ? await runStorageMigrateCommand({ dryRun: storageDryRun, json: jsonOutput }) + : storageSub === "backup" ? await runStorageBackupCommand({ keep: storageKeep, json: jsonOutput }) : storageSub === "repair" ? await runStorageRepairCommand({ from: storageFrom, json: jsonOutput }) diff --git a/src/persist.ts b/src/persist.ts index e31208952c..373a3c04ef 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -426,6 +426,24 @@ const STALE_COLD_LOAD_WARN_MIN = 30; // #5408: how often the frozen-skip ERROR may repeat per session. const FROZEN_SKIP_NOTICE_COOLDOWN_MS = 10 * 60_000; +/** #5599 `bili storage migrate`: per-session outcome of the bulk legacy→unified import. */ +interface LegacyMigrateSessionResult { + id: string; + /** Representative state file (relative to the sessions dir) for this id. */ + file: string; + status: "imported" | "indexed" | "drifted" | "failed" | "planned"; + reason?: string; + /** Same-id route files that were NOT used as representative (left untouched). */ + alternates?: number; + decodedBytes?: number; +} + +export interface LegacyMigrateReport { + scannedFiles: number; + results: LegacyMigrateSessionResult[]; + counts: { imported: number; indexed: number; drifted: number; failed: number; planned: number }; +} + export class SessionStore { /** #5188: args of the SYNCHRONOUS first-open path only. When that path ran * and failed, the save path retries maybeOpen on a cooldown; intentional @@ -520,6 +538,7 @@ export class SessionStore { // #3931 step 3: assigned asynchronously for network backends (mysql) — // see the constructor and ready(). private shadow?: UnifiedShadow; + private readonly originalsBasis = new WeakMap; hashes: Set }>(); /** Only alive while one synchronous record build hydrates both lazy fields. */ private residentFieldsForBuild?: { id: string; @@ -767,6 +786,10 @@ export class SessionStore { * loadEnvelope). Bounded server startup refuses corrupt originals; * legacy direct callers retain their warned fresh-store fallback. */ loadContentStore(session: Session): MessageContentStore | null { + if (this.writerOffActive() || this.envelopeFrozen(session.id)) { + const originals = this.shadow?.getCcr(session.id, true); + if (originals != null) return originals; + } return this.readContentStoreRels(session.id, [ contentStoreRelPathFor(session.id, session.meta.protocol, session.meta.upstreamOrigin), contentStoreRelPathFor(session.id), @@ -777,6 +800,10 @@ export class SessionStore { * checkpoint path has no live Session object, only the frozen record's * meta — same rel-path candidates, same parse-and-degrade semantics. */ private contentStoreById(id: string, meta?: { protocol?: string; upstreamOrigin?: string } | null): MessageContentStore | null { + if (this.writerOffActive() || this.envelopeFrozen(id)) { + const originals = this.shadow?.getCcr(id, true); + if (originals != null) return originals; + } return this.readContentStoreRels(id, [ contentStoreRelPathFor(id, meta?.protocol, meta?.upstreamOrigin), contentStoreRelPathFor(id), @@ -876,6 +903,10 @@ export class SessionStore { * save retries; the previous envelope stays byte-identical. */ private saveContentStore(session: Session): boolean { if (!this.enabled || !session.contentStoreDirty) return true; + if (this.writerOffActive() || this.envelopeFrozen(session.id)) { + session.contentStore ??= { version: 1, byHash: {}, byRef: {} }; + return true; + } const rel = contentStoreRelPathFor(session.id, session.meta.protocol, session.meta.upstreamOrigin); const abs = path.join(this.sessionsDir, rel); const tmp = path.join(path.dirname(abs), `.tmp-${path.basename(abs, ".json")}-${process.pid}-${this.tmpSeq++}`); @@ -1997,6 +2028,129 @@ export class SessionStore { return this.afterLoad(id, view); } + /** #5599 `bili storage migrate`: bulk-import every legacy JSON session into + * the unified index, then digest-reconcile each against its source file — + * the same verifier the steady-state sample uses, run once per session. + * Read-only over the JSON files (never deletes/renames/quarantines); the + * re-import decision mirrors loadSync's ahead-probe, so the outcome equals + * what a live open would converge to. Re-runs are safe: consistent + * sessions report "indexed" without re-writing. */ + async migrateLegacyToUnified(opts: { dryRun?: boolean } = {}): Promise { + const dryRun = opts.dryRun === true; + const counts = { imported: 0, indexed: 0, drifted: 0, failed: 0, planned: 0 }; + const results: LegacyMigrateSessionResult[] = []; + const byId = new Map>(); + const badFiles: Array<{ file: string; reason: string }> = []; + let scannedFiles = 0; + for (const root of [this.sessionsDir, `${this.sessionsDir}-cold`] as string[]) { + if (!existsSync(root)) continue; + for (const file of walkJsonFilesSync(root)) { + if (path.basename(file).endsWith(".content-store.json")) continue; + scannedFiles++; + const parsed = await this.readRawFileWithBytes(file); + if (!parsed) { + badFiles.push({ file, reason: "unreadable or undecodable" }); + continue; + } + const rec = this.parseBootPayload(parsed.value); + if (!rec) { + badFiles.push({ file, reason: "invalid session record" }); + continue; + } + if (!fileNameMatchesId(path.basename(file), rec.payload.id, rec.meta.protocol, rec.meta.upstreamOrigin)) { + badFiles.push({ file, reason: `filename does not match the record id (${rec.payload.id})` }); + continue; + } + const list = byId.get(rec.payload.id) ?? []; + list.push({ file, savedAt: rec.payload.savedAt ?? 0 }); + byId.set(rec.payload.id, list); + } + } + const relOf = (file: string): string => + file.startsWith(`${this.sessionsDir}-cold${path.sep}`) + ? `cold:${path.relative(`${this.sessionsDir}-cold`, file)}` + : path.relative(this.sessionsDir, file); + const indexedBefore = new Set(this.unifiedListSessionIds() ?? []); + for (const id of [...byId.keys()].sort()) { + const files = byId.get(id)!; + // Boot's survivor rule: newest savedAt wins, lexicographic path breaks ties. + const rep = [...files].sort((a, b) => b.savedAt - a.savedAt || a.file.localeCompare(b.file))[0]; + let status: LegacyMigrateSessionResult["status"] = "failed"; + let reason: string | undefined; + let decodedBytes: number | undefined; + if (dryRun) { + try { + const parsed = await this.readRawFileWithBytes(rep.file); + const rec = parsed ? this.parseBootPayload(parsed.value) : null; + if (!parsed || !rec || rec.payload.id !== id) throw new Error("representative file changed between scan and import attempt"); + decodedBytes = parsed.bytes; + this.migrationOriginals(rep.file, rec.payload.forkContentStore, rec.payload.state.messageRefs); + status = indexedBefore.has(id) ? "indexed" : "planned"; + } catch (err) { + status = "failed"; + reason = String(err instanceof Error ? err.message : err); + } + } else { + try { + const parsed = await this.readRawFileWithBytes(rep.file); + const rec = parsed ? this.parseBootPayload(parsed.value) : null; + if (!parsed || !rec || rec.payload.id !== id) throw new Error("representative file changed between scan and import attempt"); + decodedBytes = parsed.bytes; + const shadow = this.shadow; + if (!shadow) throw new Error("unified sidecar unavailable"); + const hadRows = indexedBefore.has(id); + const indexed = hadRows ? this.loadUnified(id) : null; + const envReq = Number(rec.payload.stats?.requests ?? 0); + const idxReq = indexed ? Number(indexed.stats?.requests ?? 0) : 0; + const ahead = indexed !== null && + (envReq > idxReq || (envReq === idxReq && Number(rec.payload.state?.nextBlockId ?? 0) > Number(indexed.state?.nextBlockId ?? 0))); + const originals = this.migrationOriginals(rep.file, rec.payload.forkContentStore, rec.payload.state.messageRefs); + const completeOriginals = this.mergeOriginalStores([hadRows ? shadow.getCcr(id, true) : null, originals]); + const complete = completeOriginals ? { ...rec.payload, forkContentStore: completeOriginals } : rec.payload; + let wrote = false; + if (!hadRows || indexed === null || ahead) { + if (!shadow.importRecord(complete)) throw new Error("import failed (see log)"); + wrote = true; + } + const legacy = recordToLegacyView(complete); + if (!legacy) throw new Error("legacy view build failed"); + const verdict = shadow.verifyImported(legacy); + if (verdict === null) { + status = wrote ? "imported" : "indexed"; + reason = "verification unavailable (network backend)"; + } else if (verdict.matches) { + status = wrote ? "imported" : "indexed"; + } else { + status = "drifted"; + const parts = [`diverged: ${verdict.diagnostics.keys.join(", ")}`]; + if (verdict.diagnostics.stateKeys) parts.push(`state: ${verdict.diagnostics.stateKeys.join(", ")}`); + reason = `digest mismatch (${parts.join("; ")})`; + } + if (canonicalJson(shadow.getCcr(id, true)) !== canonicalJson(completeOriginals ?? null)) { + status = "drifted"; + reason = "CCR originals differ from the migration source"; + } + } catch (err) { + status = "failed"; + reason = String(err instanceof Error ? err.message : err); + } + } + const entry: LegacyMigrateSessionResult = { id, file: relOf(rep.file), status }; + if (reason) entry.reason = reason; + if (decodedBytes !== undefined) entry.decodedBytes = decodedBytes; + if (files.length > 1) entry.alternates = files.length - 1; + results.push(entry); + counts[status]++; + } + for (const bad of badFiles) { + const entry: LegacyMigrateSessionResult = { id: path.basename(bad.file), file: relOf(bad.file), status: "failed", reason: bad.reason }; + results.push(entry); + counts.failed++; + } + this.log("info", `[persist] storage migrate${dryRun ? " (dry run)" : ""}: ${scannedFiles} files scanned, ${counts.imported} imported, ${counts.indexed} indexed, ${counts.drifted} drifted, ${counts.failed} failed`); + return { scannedFiles, results, counts }; + } + /** Shared post-load fixups for the unified read path: rebuilds the * Session from a store view (viewToRecord → buildSession, the same * forward-compat pass as a legacy load) and applies the #408 clamp @@ -2046,6 +2200,19 @@ export class SessionStore { try { const seqByRawId = this.unifiedSeqByRaw(session.id); if (seqByRawId === null || seqByRawId.size === 0) return; + // #5599: stamp only a COMPLETE ledger — every mapped id must own a + // row (live or retired). An imported collapsed ledger (folded + // ghosts have no rows; the tail re-based at seq 0) would let the + // stamp evict the ghost bindings onto fresh numbers and re-target + // saved citations at newer bodies. Incomplete → keep legacy refs + // (JSON-load parity); the next fold retries once rows exist. + // Query failure fails safe to skip: never restamp on uncertain input. + const mapped = Object.keys(state.messageRefs.byRaw).filter((id) => state.messageRefs.byRaw[id] !== "BLOCKED"); + const backed = this.unifiedBackedMsgIds(session.id, mapped); + if (backed === null || mapped.some((id) => !backed.has(id))) { + this.log("info", `[persist] ${session.id}: unified ledger incomplete (${mapped.length} mapped ids checked) — keeping legacy refs (#5599)`); + return; + } const outcome = restampRefsToLedger(state, seqByRawId); // #4733: the ref restamp alone leaves the CCR content store keyed by // store-time refs — a stored original would sit under a number the @@ -2296,6 +2463,18 @@ export class SessionStore { } } + /** #5599: msg ids owning ≥1 unified row (live or retired) — the re-stamp + * completeness check's data source. Null when the unified store cannot + * answer; the caller then keeps legacy refs (fail-safe direction). */ + private unifiedBackedMsgIds(id: string, msgIds: string[]): Set | null { + if (!this.shadow || this.shadow.stats.disabled) return null; + try { + return this.shadow.backedMsgIds(id, msgIds); + } catch { + return null; + } + } + /** #4730/#4911: highest live (non-retired) seq — the fork endpoint's * inclusive throughSeq, awaited from the (async) fork lane. NOT count - 1 * from unifiedLiveRowCount: tombstones keep their seq, so a retired @@ -3038,15 +3217,121 @@ export class SessionStore { /** #3818: decoded-size proxy for budget arbitration of index-only * sessions — the serialized record the JSON writer would emit, which * is what readRawFileWithBytes measures for file-backed candidates - * (#2803: the budget is a RAM proxy, not a disk one). */ + * (#2803: the budget is a RAM proxy, not a disk one). + * #6773 review: measure from the pure record projection, not the + * committing guardedBuild — the CCR generation guard throws for a + * cold session whose contentStore is not yet materialized (the + * request path materializes it via contentStoreOf, but the warm loop + * invokes the builder directly), so the catch returned + * MAX_SAFE_INTEGER and silently skipped every CCR-bearing session + * from pre-warming. buildRecordForSave is the same record guardedBuild + * returns on the normal path, without the commit side effect. */ private estimateDecodedBytes(session: Session): number { try { - return JSON.stringify(this.guardedBuild(session)()).length; + return JSON.stringify(this.buildRecordForSave(session, this.liveStorageStamp)).length; } catch { return Number.MAX_SAFE_INTEGER; } } + private mergeOriginalStores(stores: Array): MessageContentStore | undefined { + let merged: MessageContentStore | undefined; + for (const store of stores) { + if (store == null) continue; + if (typeof store !== "object" || store.version !== 1 || !store.byHash || !store.byRef + || typeof store.byHash !== "object" || typeof store.byRef !== "object" + || Array.isArray(store.byHash) || Array.isArray(store.byRef)) throw new Error("malformed CCR originals store"); + merged ??= { version: 1, byHash: {}, byRef: {} }; + for (const [hash, text] of Object.entries(store.byHash)) { + if (typeof text !== "string" || createHash("sha256").update(text, "utf8").digest("hex") !== hash) { + throw new Error("CCR original body has an invalid content hash"); + } + if (Object.hasOwn(merged.byHash, hash) && merged.byHash[hash] !== text) throw new Error("conflicting CCR original body"); + merged.byHash[hash] = text; + } + for (const [ref, entry] of Object.entries(store.byRef)) { + if (!entry || typeof entry !== "object" || typeof entry.hash !== "string" + || typeof entry.rawId !== "string" || !Object.hasOwn(store.byHash, entry.hash)) throw new Error("CCR reference has no valid original body"); + if (Object.hasOwn(merged.byRef, ref) && canonicalJson(merged.byRef[ref]) !== canonicalJson(entry)) throw new Error(`conflicting CCR original reference: ${ref}`); + Object.defineProperty(merged.byRef, ref, { value: entry, enumerable: true, writable: true, configurable: true }); + } + } + return merged; + } + + private migrationOriginals(file: string, embedded?: MessageContentStore, refs?: Session["state"]["messageRefs"]): MessageContentStore | undefined { + const validateOwners = (originals: MessageContentStore | undefined): MessageContentStore | undefined => { + for (const [ref, entry] of Object.entries(originals?.byRef ?? {})) { + // Match the existing fork alias contract: pruned IDs retain + // frozen original keys; still-mapped IDs must agree both ways. + if (refs?.byRaw && Object.hasOwn(refs.byRaw, entry.rawId) + && (refs.byRaw[entry.rawId] !== ref || refs.byRef?.[ref] !== entry.rawId)) { + throw new Error(`CCR original reference owner conflicts with session: ${ref}`); + } + } + return originals; + }; + // #5485: legacy-mode sessions never reach the fold-time re-key, so their + // originals can still sit under refs from before a renumbering. Apply the + // same re-key here; the source files stay untouched. + const rekeyed = (originals: MessageContentStore | undefined): MessageContentStore | undefined => + originals && refs?.byRaw ? restampContentStore(originals, refs.byRaw) : originals; + const sidecar = file.endsWith(".fb.json") + ? `${file.slice(0, -".fb.json".length)}.content-store.json` + : file.replace(/\.json$/, ".content-store.json"); + let bytes: Buffer; + try { bytes = readFileSync(sidecar); } + catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") return validateOwners(rekeyed(this.mergeOriginalStores([embedded]))); + throw new Error("CCR originals sidecar cannot be read", { cause: err }); + } + let parsed: MessageContentStore; + try { parsed = JSON.parse(this.readCodec.decode(bytes)) as MessageContentStore; } + catch (err) { throw new Error("CCR originals sidecar cannot be decoded", { cause: err }); } + if (parsed == null) throw new Error("malformed CCR originals sidecar"); + return validateOwners(rekeyed(this.mergeOriginalStores([embedded, parsed]))); + } + + private mergeDurableOriginals(session: Session, disk: MessageContentStore | null): MessageContentStore | undefined { + const resident = session.contentStore; + if (session.contentStoreDirty === true && resident + && Object.keys(resident.byRef).length === 0 && Object.keys(resident.byHash).length === 0) return resident; + const basis = this.originalsBasis.get(session); + if (basis) { + for (const [ref, entry] of basis.refs) { + if (!disk?.byRef[ref] || canonicalJson(disk.byRef[ref]) !== entry) { + throw new Error(`Index commit refused: CCR generation conflict at ${ref}`); + } + } + for (const hash of basis.hashes) { + if (!disk || !Object.hasOwn(disk.byHash, hash)) throw new Error("Index commit refused: CCR generation lost an original body"); + } + } else if (disk) { + for (const [ref, entry] of Object.entries(disk.byRef)) { + if (!resident?.byRef[ref] || canonicalJson(resident.byRef[ref]) !== canonicalJson(entry)) { + throw new Error(`Index commit refused: unproven CCR generation at ${ref}`); + } + } + for (const hash of Object.keys(disk.byHash)) { + if (!resident || !Object.hasOwn(resident.byHash, hash)) throw new Error("Index commit refused: unproven CCR original body generation"); + } + } + const merged = this.mergeOriginalStores([disk, resident]); + if (merged) { + session.contentStore = merged; + session.contentStoreDirty = true; + } + return merged; + } + + private rememberOriginalsBasis(session: Session): void { + const store = session.contentStore; + this.originalsBasis.set(session, { + refs: new Map(Object.entries(store?.byRef ?? {}).map(([ref, entry]) => [ref, canonicalJson(entry)])), + hashes: new Set(Object.keys(store?.byHash ?? {})), + }); + } + private guardedBuild(session: Session): () => PersistedSession { // #2671: capture the storage-mode stamp EAGERLY, at enqueue time — // the kernel write chain may invoke the builder on a later tick than @@ -3060,6 +3345,16 @@ export class SessionStore { // the JSON envelope, and the stamp must say legacy for that copy. const stamp = this.liveStorageStamp; return () => { + const requireCommit = this.writerOffActive() || this.envelopeFrozen(session.id); + if (requireCommit && !this.shadow) throw new Error("Index commit refused: unified storage unavailable"); + const settleCcr: Parameters[4] = requireCommit && this.shadow && !this.shadow.isAsyncBackend + ? (current, counters) => { + if (counters && (session.stats.requests < counters.requests + || (session.stats.requests === counters.requests && session.state.nextBlockId < counters.nextBlockId))) { + throw new Error("Index commit refused: session generation advanced while preparing originals"); + } + return this.mergeDurableOriginals(session, current); + } : undefined; const record = this.buildRecordForSave(session, stamp); const disk = this.staleDiskPayload(record); if (disk === null) { @@ -3070,7 +3365,6 @@ export class SessionStore { // re-ingests forever. buildRecord omits it, so the disk file // never carries it. const payload = session.decompositionLog; - if (payload !== undefined) session.decompositionLog = undefined; // #2960 ⑤: the table holds the RAW tail (full fidelity); the // envelope keeps its bounded folded view for legacy readers. // ②a caveat: once foldResidentTail has folded the resident @@ -3087,7 +3381,12 @@ export class SessionStore { this.shadow?.record(payload !== undefined ? { ...record, decompositionLog: payload } : record, tail && tail.length > 0 ? { lastMessages: tail, lastMessagesFolded: session.lastMessagesFolded } - : undefined, session.contentStore ?? undefined); + : undefined, session.contentStore ?? undefined, requireCommit, settleCcr); + if (payload !== undefined) session.decompositionLog = undefined; + if (settleCcr) { + this.rememberOriginalsBasis(session); + session.contentStoreDirty = false; + } this.noteOwnCommit(session.id, record); return record; } @@ -3107,7 +3406,23 @@ export class SessionStore { // to the live session so serving converges on the same newest-wins // arbitration the write side just used. this.convergeToDisk(session, disk); - this.shadow?.record(disk); + if (settleCcr && this.shadow) { + // #6773 review: convergence discards the in-memory state, so + // the stale in-memory originals are discarded with it — rebind + // from the index mirror (the newer generation we just converged + // to) and seed the basis so the guarded commit merges against + // it instead of refusing the unproven generation (a stale + // resident with no basis wedges every subsequent save until + // restart). + session.contentStore = this.shadow.getCcr(session.id, true) ?? undefined; + session.contentStoreDirty = false; + this.rememberOriginalsBasis(session); + } + this.shadow?.record(disk, undefined, undefined, requireCommit, settleCcr); + if (settleCcr) { + this.rememberOriginalsBasis(session); + session.contentStoreDirty = false; + } // Rewrite the disk's own payload: content-identical no-op that // preserves the newer state while satisfying the write chain. this.noteOwnCommit(session.id, disk); diff --git a/src/storage-maintenance.ts b/src/storage-maintenance.ts index ebfd2348f8..dca41530aa 100644 --- a/src/storage-maintenance.ts +++ b/src/storage-maintenance.ts @@ -1,12 +1,12 @@ /** - * `bili sessions …` / `bili storage gc` — CLI plumbing over the existing + * `bili sessions …` / `bili storage gc|migrate` — CLI plumbing over the existing * persistence + #3911 store maintenance APIs. No new store semantics: every * mutation here is the same call the HTTP surface makes (admin.ts), so the * two lanes stay structurally identical. */ import { existsSync, statSync } from "node:fs"; import { join } from "node:path"; -import { SessionStore } from "./persist.js"; +import { SessionStore, type LegacyMigrateReport } from "./persist.js"; import type { GcPlan } from "./storage/store.js"; import { fileStorageArchiveAfterDays, resolveArchiveAfterDays } from "./knobs.js"; import { scanForeignBlocks, applyForeignBlockSweep, type ForeignBlockPlan } from "./foreign-block-sweep.js"; @@ -356,3 +356,58 @@ export async function runStorageGcCommand(opts: { store.closeQuietly(); } } + +function renderMigrate(report: LegacyMigrateReport, dryRun: boolean): string { + const lines = [ + `storage migrate${dryRun ? " (dry run)" : ""}: ${report.scannedFiles} files scanned`, + ` imported: ${report.counts.imported} indexed: ${report.counts.indexed}${dryRun ? ` planned: ${report.counts.planned}` : ""} drifted: ${report.counts.drifted} failed: ${report.counts.failed}`, + ]; + const problems = report.results.filter((r) => r.status === "drifted" || r.status === "failed"); + if (problems.length > 0) { + lines.push(""); + lines.push("problems:"); + for (const p of problems) { + lines.push(` ${p.status.toUpperCase()} ${p.id}`); + lines.push(` ${p.file}`); + if (p.reason) lines.push(` ${p.reason}`); + if (p.alternates) lines.push(` (${p.alternates} same-id route file(s) left untouched)`); + } + } else if (report.results.length > 0) { + lines.push(` all ${report.results.length} session(s) reconciled`); + } else { + lines.push(" no legacy JSON sessions found"); + } + if (dryRun) lines.push("dry-run: nothing written"); + else lines.push("note: JSON files were not modified — archive or delete them yourself once this report is clean."); + return lines.join("\n"); +} + +async function runMigrate(store: SessionStore, dryRun: boolean, json: boolean): Promise { + if (!store.replicasAvailable()) return fail(`storage migrate ${UNIFIED_REQUIRED}`, json); + const dbBefore = indexDbSize(store.dir); + const report = await store.migrateLegacyToUnified({ dryRun }); + const dbAfter = indexDbSize(store.dir); + const code = dryRun || (report.counts.drifted + report.counts.failed) === 0 ? 0 : 1; + if (json) { + return { code, text: `${JSON.stringify({ dryRun, ...report, indexDbBefore: dbBefore, indexDbAfter: dbAfter }, null, 2)}\n` }; + } + let text = renderMigrate(report, dryRun); + if (!dryRun && dbBefore !== undefined && dbAfter !== undefined) text += `\n index.db size: ${formatBytes(dbBefore)} -> ${formatBytes(dbAfter)}`; + return { code, text: `${text}\n` }; +} + +/** #5599 one-shot bulk import of every legacy JSON session into the unified + * SQLite index with per-session digest reconciliation. Read-only over the + * JSON files; exit 1 when any session is drifted or failed. */ +export async function runStorageMigrateCommand(opts: { + dryRun?: boolean; + json?: boolean; + dir?: string; +}): Promise { + const store = new SessionStore({ dir: opts.dir, enabled: true }); + try { + return await runMigrate(store, !!opts.dryRun, !!opts.json); + } finally { + store.closeQuietly(); + } +} diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 0e306de770..915d173901 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -568,7 +568,7 @@ export class UnifiedShadow { // engine/engineSource introspection lives in engineInfo (sqlite-only // diagnostics widened for network backends); runtime-status reads that. - /** Ingest + verify one committed record. Never throws. + /** Ingest + verify one committed record. Required commits propagate failures. * * `rawTail` (#2960 ⑤): the live in-memory conversation tail when the * caller has one. The envelope persists only a pruned+truncated view of @@ -576,20 +576,37 @@ export class UnifiedShadow { * message log authoritatively, so the ingest uses the raw tail whenever * available; both digest legs go through the same override, so parity * accounting stays honest. */ - record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore): void { + record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore, + requireCommit = false, + settleCcr?: (current: MessageContentStore | null, counters: { requests: number; nextBlockId: number } | null) => MessageContentStore | undefined): void { // #3931 step 3: network backend — the ingest runs on the serial // queue (fire-and-forget; failures are logged once per distinct message — never disabled). if (this.sync === null) { + if (requireCommit) throw new Error("Originals guard requires synchronous SQLite storage"); this.enqueueAsyncRecord(record, rawTail, liveCcr); return; } + if (settleCcr) { + if (!requireCommit) throw new Error("Originals guard requires a durable commit"); + const loaded = this.legacyLoad(record); + if (!loaded) throw new Error("Index commit refused: session record could not be loaded"); + const sync = this.sync; + sync.withOriginalsTransaction(() => { + const originals = settleCcr(sync.getCcr(loaded.id, true), sync.sessionCounters(loaded.id)); + this.record(record, rawTail, originals, true); + }); + return; + } // #5627: which stage is running — the failure forensics must name // the operation that died, not just the engine's one-line verdict. let phase = "legacy-load"; let sessionId: string | undefined; try { const loaded = this.legacyLoad(record); - if (!loaded) return; + if (!loaded) { + if (requireCommit) throw new Error("Index commit refused: session record could not be loaded"); + return; + } sessionId = loaded.id; // #4216/#4242: the reap shape (blockContents without their // state.blocks entries) is now REPRESENTABLE — planIngest @@ -772,6 +789,7 @@ export class UnifiedShadow { this.consecutiveFailures = 0; } catch (err) { this.noteIngestFailure(err, phase, sessionId); + if (requireCommit) throw err; } } @@ -942,6 +960,27 @@ export class UnifiedShadow { } } + /** #5599 bulk migration: one-shot digest verification of a freshly imported + * session against its legacy source view — the same verifier as the + * steady-state sample in record(), without the latch/freeze side effects + * (the migrate tool reports, it does not heal). Null on network backends + * (no sync read path); throws only when the ingest vanished its own row. */ + verifyImported(legacy: LegacySessionLike): ReturnType | null { + if (this.sync === null) return null; + const sessionRow = this.sync.getSessionRow(legacy.id); + if (!sessionRow) throw new Error(`session row missing after import: ${legacy.id}`); + const verifier = new BatchedViewVerification(legacy, sessionRow, this.sync.getRefs(legacy.id), this.sync.getDeadRefs(legacy.id)); + for (let offset = 0; ; offset += VERIFY_BATCH_SIZE) { + const rows = this.sync.getBlocks(legacy.id, { limit: VERIFY_BATCH_SIZE, offset }); + verifier.addBlocks(rows); + if (rows.length < VERIFY_BATCH_SIZE) break; + } + for (let i = 0; i < verifier.messageSeqs.length; i += VERIFY_BATCH_SIZE) { + verifier.addMessages(this.sync.getMessages(legacy.id, { seqs: verifier.messageSeqs.slice(i, i + VERIFY_BATCH_SIZE) })); + } + return verifier.finish(); + } + /** GC parity: remove the session's rows when the legacy files are deleted. */ remove(id: string): void { if (this.sync === null) return; @@ -1174,9 +1213,9 @@ export class UnifiedShadow { } /** #2960 ③: whole-store CCR read from the sessions row's ccr mirror. */ - getCcr(sessionId: string): MessageContentStore | null { + getCcr(sessionId: string, requireValid = false): MessageContentStore | null { if (this.sync === null) return null; - return this.sync.getCcr(sessionId); + return this.sync.getCcr(sessionId, requireValid); } /** #4739: narrow single-column metadata read for lazy-field restore. @@ -1213,6 +1252,18 @@ export class UnifiedShadow { return this.sync.seqByRaw(sessionId); } + /** #5599: row-backed msg ids (live OR retired) for the re-stamp + * completeness check — same full-trust gate as seqByRaw. Null when the + * store cannot answer; the caller fails safe to keeping legacy refs. */ + backedMsgIds(sessionId: string, msgIds: string[]): Set | null { + if (this.sync === null) return null; + try { + return this.sync.backedMsgIds(sessionId, msgIds); + } catch { + return null; + } + } + /** #4316 PR3/3: live (non-retired) message row count. */ liveRowCount(sessionId: string): number { if (this.sync !== null) return this.sync.liveRowCount(sessionId); diff --git a/src/storage/store.ts b/src/storage/store.ts index 0dcf1f4835..cadcfe00ac 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -3375,6 +3375,29 @@ export class UnifiedStore { return out; } + /** #5599: which of these msg_ids own at least ONE row (live OR retired). + * Completeness check for the one-time ledger re-stamp: an imported + * collapsed ledger (folded ghosts never got rows; the tail re-based at + * seq 0) must not be stamped, because the stamp evicts the ghost + * bindings onto fresh numbers and re-targets saved citations at newer + * bodies. Native ledgers always qualify — the historian sweep retires + * covered rows instead of deleting them (#4390), so every mapped id + * keeps its row. */ + backedMsgIds(sessionId: string, msgIds: string[]): Set { + const out = new Set(); + if (msgIds.length === 0) return out; + const CHUNK = 500; + for (let i = 0; i < msgIds.length; i += CHUNK) { + const chunk = msgIds.slice(i, i + CHUNK); + const ph = chunk.map(() => "?").join(", "); + const rows = this.db + .prepare(`SELECT DISTINCT msg_id FROM messages WHERE session_id = ? AND msg_id IN (${ph})`) + .all(sessionId, ...chunk) as Record[]; + for (const r of rows) out.add(r.msg_id as string); + } + return out; + } + /** #4316 PR3/3: fork adoption clones the parent's rows wholesale so the * child's cloned kernel refs keep pointing at the same ledger positions * (owner decision 2026-10-22: fork copies every number). The child @@ -3769,21 +3792,57 @@ export class UnifiedStore { })); } - getCcr(sessionId: string): MessageContentStore | null { + withOriginalsTransaction(action: () => T): T { + return this.db.transaction(action); + } + + getCcr(sessionId: string, requireValid = false): MessageContentStore | null { const row = this.db .prepare("SELECT ccr_json FROM sessions WHERE session_id = ?") .get(sessionId) as { ccr_json: string | null } | undefined; - if (!row || row.ccr_json === null) return null; + if (!row || row.ccr_json === null) { + if (requireValid) { + const indexed = this.db.prepare("SELECT count(*) AS n FROM ccr_entries WHERE session_id = ?").get(sessionId) as { n: number }; + if (Number(indexed.n) !== 0) throw new Error("Index commit refused: missing CCR mirror with durable bindings"); + } + return null; + } try { const parsed = JSON.parse(row.ccr_json) as MessageContentStore; if (parsed && typeof parsed === "object" && parsed.version === 1 && parsed.byHash && typeof parsed.byHash === "object" && parsed.byRef && typeof parsed.byRef === "object") { + if (requireValid) { + if (Array.isArray(parsed.byHash) || Array.isArray(parsed.byRef)) throw new Error("unreadable CCR shape"); + for (const [hash, text] of Object.entries(parsed.byHash)) { + if (typeof text !== "string" || createHash("sha256").update(text).digest("hex") !== hash) { + throw new Error("unreadable CCR original body checksum"); + } + } + for (const entry of Object.values(parsed.byRef)) { + if (!entry || typeof entry.hash !== "string" || typeof entry.rawId !== "string" + || !Object.hasOwn(parsed.byHash, entry.hash)) throw new Error("unreadable CCR original body binding"); + } + const entries = this.db.prepare(`SELECT e.ccr_key, e.content_hash, c.bytes + FROM ccr_entries e LEFT JOIN content c ON c.hash = e.content_hash + WHERE e.session_id = ?`).all(sessionId) as { ccr_key: string; content_hash: string; bytes: Uint8Array | null }[]; + if (entries.length !== Object.keys(parsed.byRef).length) throw new Error("unreadable CCR binding count"); + for (const entry of entries) { + if (entry.bytes === null) throw new Error("unreadable CCR binding content"); + const bytes = Buffer.from(entry.bytes); + if (createHash("sha256").update(bytes).digest("hex") !== entry.content_hash + || JSON.stringify(parsed.byRef[entry.ccr_key]) !== bytes.toString("utf8")) { + throw new Error("unreadable CCR binding content"); + } + } + } return parsed; } - } catch { + } catch (err) { + if (requireValid) throw new Error("Index commit refused: unreadable CCR mirror", { cause: err }); // fall through: a corrupt mirror degrades to the file path } + if (requireValid) throw new Error("Index commit refused: unreadable CCR mirror"); return null; } diff --git a/tests/check-dist-external.test.ts b/tests/check-dist-external.test.ts index 60a06a6285..ec6c7f5552 100644 --- a/tests/check-dist-external.test.ts +++ b/tests/check-dist-external.test.ts @@ -53,6 +53,20 @@ test("the sanctioned opaque path and incidental strings are invisible", () => { assert.equal(clean.length, 0); }); +test("the word 'import' inside a string literal is not an import (the #5636 false positive)", () => { + // The minified shape that tripped the build: an error message ending in + // the word import, then more code and another string literal on the SAME + // line. The side-effect pattern used to fuse the two quotes into a bogus + // ref — pinning both directions (#3617): no ref here, and a real + // side-effect import on the next line is still caught. + const text = + 'if(!rec)throw new Error("representative file changed between scan and import");decodedBytes=parsed.bytes;\n' + + 'import"left-pad";'; + const refs = findExternalRefs(text); + assert.equal(refs.length, 1); + assert.deepEqual({ spec: refs[0].spec, kind: refs[0].kind }, { spec: "left-pad", kind: "static" }); +}); + test("node builtins (with and without node: prefix) are clean", () => { const clean = findExternalRefs( 'import{readFile}from"node:fs";import crypto from"crypto";const u=require("util");', diff --git a/tests/issue6780-pre-forward-dedup.test.ts b/tests/issue6780-pre-forward-dedup.test.ts index 606ffdeb98..59c0661b04 100644 --- a/tests/issue6780-pre-forward-dedup.test.ts +++ b/tests/issue6780-pre-forward-dedup.test.ts @@ -106,7 +106,16 @@ async function withHarness(run: (h: Harness) => Promise): Promise { ingestRowsNow: () => anyStore.ingestRowsNow(session), spy: () => { const real = anyStore.shadow.record.bind(anyStore.shadow); - anyStore.shadow.record = (...args: unknown[]) => { ingests++; return real(...args); }; + anyStore.shadow.record = (...args: unknown[]) => { + // #6773: the writer-off originals commit (requireCommit + + // settleCcr) wraps the actual ingest in a transaction and + // re-enters record() for the inner write — count only the + // inner (settleCcr-less) ingest so one full build is one + // count, preserving this suite's pre-#6773 counting shape. + if (args[4] !== undefined) return real(...args); + ingests++; + return real(...args); + }; return () => ingests; }, markDirty: () => { diff --git a/tests/persist-foreign-originals-control.test.ts b/tests/persist-foreign-originals-control.test.ts new file mode 100644 index 0000000000..72ab6d8f41 --- /dev/null +++ b/tests/persist-foreign-originals-control.test.ts @@ -0,0 +1,365 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createHash } from 'node:crypto'; +import { DatabaseSync } from 'node:sqlite'; +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { createInitialState, storeOriginal } from 'acp-kernel'; +import { SessionStore } from '../src/persist.ts'; +import { UnifiedShadow } from '../src/storage/shadow.ts'; +import { UnifiedStore } from '../src/storage/store.ts'; +import { type Session } from '../src/session.ts'; +import { rmrf } from './tmp-rm.ts'; + +const root = mkdtempSync(join(tmpdir(), 'bili-index-resident-')); +function loadWithOriginals(store: SessionStore, original: Session): Session | null { + const loaded = store.loadSync(original.id, original.meta); + if (loaded) loaded.contentStore = store.loadContentStore(loaded) ?? undefined; + return loaded; +} +process.env.BILI_CONFIG_FILE = join(root, 'no-config.json'); +process.env.BILI_PERSIST_ZSTD = '0'; +test.after(() => rmrf(root)); + +function fixture(id: string): Session { + const original = 'opaque signed original λ\u0000'; + const hash = createHash('sha256').update(original).digest('hex'); + return { id, meta: { protocol: 'openai', upstreamOrigin: 'http://upstream' }, + stats: { requests: 0, tokensSaved: 0, inputTokens: 0, cachedTokens: 0, outputTokens: 0, + cacheSamples: 0, lastInputTokens: 0, contextTokens: 0, compressCreditTokens: 0, + retrieveCalls: 0, retrieveHits: 0, retrieveMisses: 0, storedBytes: 0, + storeBytesSaved: 0, rangeRestores: 0 }, + state: createInitialState(), metadata: { exact: 'metadata' }, createdAt: 1, lastSeen: 2, + inFlight: 0, persisted: false, pendingRetrievals: [], + blockContents: new Map([['orphan', { one: null, full: { text: original, count: 3 } }]]), + contentStore: { version: 1, byHash: { [hash]: original }, byRef: { + m00001: { rawId: 'raw', hash, head: 'opaque', kind: 'tool', tokens: 5, chars: original.length }, + } }, contentStoreDirty: true }; +} + +for (const inFlight of [0, 1]) { +test(`a stale resident save preserves another SQL writer same-counter originals (inFlight=${inFlight})`, async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'foreign-originals-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('shared'); + a.inFlight = inFlight; + let second: SessionStore | undefined; + try { + await first.writeNow(a); + second = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const b = loadWithOriginals(second, a); + assert.ok(b?.contentStore); + b.contentStore = storeOriginal(b.contentStore, { ref: 'm00002', rawId: 'foreign', + text: 'Exact foreign original λ\u0000', kind: 'tool', tokens: 8, head: 'foreign' }); + b.contentStoreDirty = true; + assert.equal(b.stats.requests, a.stats.requests); + assert.equal(b.state.nextBlockId, a.state.nextBlockId); + await second.writeNow(b); + const expected = structuredClone(second.unifiedCcrFor(a.id)); + await first.writeNow(a); + assert.deepEqual(first.unifiedCcrFor(a.id), expected, + 'same-counter external originals must survive a stale resident save'); + } finally { second?.closeQuietly(); first.closeQuietly(); } +}); +} + +for (const replacement of [false, true]) { +test(`a stale in-flight writer cannot resurrect originals after an external reset (replacement=${replacement})`, async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'foreign-reset-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('shared-reset'); + a.inFlight = 1; + let second: SessionStore | undefined; + try { + await first.writeNow(a); + second = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const b = loadWithOriginals(second, a); + assert.ok(b); + b.contentStore = undefined; + b.contentStoreDirty = true; + b.revisionEpoch = (b.revisionEpoch ?? 0) + 1; + await second.writeNow(b); + const cleared = structuredClone(second.unifiedCcrFor(a.id)); + assert.deepEqual(cleared, { version: 1, byHash: {}, byRef: {} }); + if (replacement) { + b.contentStore = storeOriginal({ version: 1, byHash: {}, byRef: {} }, { + ref: 'm00001', rawId: 'replacement-generation', text: 'Replacement opaque original Ω\u0000', + kind: 'tool', tokens: 9, head: 'replacement', + }); + b.contentStoreDirty = true; + b.revisionEpoch = (b.revisionEpoch ?? 0) + 1; + await second.writeNow(b); + } + const winning = structuredClone(second.unifiedCcrFor(a.id)); + await assert.rejects(first.writeNow(a), /generation|conflict/i); + assert.deepEqual(first.unifiedCcrFor(a.id), winning, 'stale originals must not resurrect after reset'); + assert.ok(a.contentStore?.byRef.m00001, 'refused writer must retain its unsaved resident evidence'); + } finally { second?.closeQuietly(); first.closeQuietly(); } +}); +} + +test('an outer commit failure preserves pending originals and publishes no ownership until retry', async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'foreign-commit-fault-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('shared-commit-fault'); + a.inFlight = 1; + let second: SessionStore | undefined; + const exec = DatabaseSync.prototype.exec; + try { + await first.writeNow(a); + second = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const b = loadWithOriginals(second, a); + assert.ok(b?.contentStore); + b.contentStore = storeOriginal(b.contentStore, { ref: 'm00002', rawId: 'foreign', + text: 'External original λ\u0000', kind: 'tool', tokens: 8, head: 'external' }); + b.contentStoreDirty = true; + await second.writeNow(b); + const before = structuredClone(second.unifiedCcrFor(a.id)); + assert.ok(before); + const added = { ref: 'm00003', rawId: 'pending-local', text: 'Pending local original Ω\u0000', kind: 'tool', tokens: 9, head: 'pending' }; + assert.ok(a.contentStore); + a.contentStore = storeOriginal(a.contentStore, added); + a.contentStoreDirty = true; + let published = 0; + first.setSessionCommitListener(() => { published++; }); + let faults = 0; + DatabaseSync.prototype.exec = function (sql: string) { + if (sql === 'COMMIT' && faults++ === 0) throw new Error('injected outer commit failure'); + return exec.call(this, sql); + }; + await assert.rejects(first.writeNow(a), /injected outer commit failure/); + DatabaseSync.prototype.exec = exec; + assert.equal(faults, 1, 'the fault must reach the actual outer COMMIT'); + assert.equal(published, 0, 'a rolled-back transaction must not publish ownership'); + assert.deepEqual(first.unifiedCcrFor(a.id), before, 'failed commit must leave the external durable store exact'); + assert.ok(a.contentStoreDirty, 'pending originals must remain dirty after failure'); + assert.ok(a.contentStore?.byRef.m00003, 'pending local original must remain resident'); + await first.writeNow(a); + assert.deepEqual(first.unifiedCcrFor(a.id), storeOriginal(before, added)); + assert.equal(published, 1); + assert.equal(a.contentStoreDirty, false); + } finally { DatabaseSync.prototype.exec = exec; second?.closeQuietly(); first.closeQuietly(); } +}); + +for (const kind of ['malformed', 'null', 'empty', 'binding', 'CAS', 'body']) { +test(`a fresh writer refuses an unreadable originals mirror with durable bindings (${kind})`, async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'unreadable-originals-')); + const seed = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const pending = fixture('unreadable-originals'); + await seed.writeNow(pending); + seed.closeQuietly(); + const db = new DatabaseSync(join(dir, 'index.db')); + let writer: SessionStore | undefined; + try { + assert.equal(Number(db.prepare('SELECT count(*) AS n FROM ccr_entries WHERE session_id=?').get(pending.id)?.n), 1); + let corrupt = db.prepare('SELECT ccr_json FROM sessions WHERE session_id=?').get(pending.id)?.ccr_json as string | null; + if (kind === 'malformed') corrupt = 'not valid JSON'; + if (kind === 'null') corrupt = null; + if (kind === 'empty') corrupt = JSON.stringify({ version: 1, byHash: {}, byRef: {} }); + if (kind === 'binding') { + assert.ok(corrupt); + const changed = JSON.parse(corrupt); + changed.byRef.m00001.head = 'different durable entry metadata'; + corrupt = JSON.stringify(changed); + } + if (kind === 'body') { + assert.ok(corrupt); + const changed = JSON.parse(corrupt); + changed.byHash[Object.keys(changed.byHash)[0]!] = 'Different original body under its unchanged hash'; + corrupt = JSON.stringify(changed); + } + db.prepare('UPDATE sessions SET ccr_json=? WHERE session_id=?').run(corrupt, pending.id); + if (kind === 'CAS') db.prepare(`UPDATE content SET bytes=? WHERE hash IN + (SELECT content_hash FROM ccr_entries WHERE session_id=?)`).run(Buffer.from('changed binding bytes'), pending.id); + const bindings = db.prepare('SELECT * FROM ccr_entries WHERE session_id=?').all(pending.id); + const bodies = db.prepare(`SELECT c.hash, c.bytes FROM content c JOIN ccr_entries e + ON c.hash=e.content_hash WHERE e.session_id=?`).all(pending.id); + writer = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + assert.throws(() => writer!.loadContentStore(pending), /unreadable|missing CCR/, + 'unreadable authoritative SQL originals must not degrade to an empty or legacy read'); + assert.ok(pending.contentStore); + pending.contentStore = storeOriginal(pending.contentStore, { ref: 'm00002', rawId: 'new-after-corruption', + text: 'New exact pending original Ω\u0000', kind: 'tool', tokens: 7, head: 'pending' }); + pending.contentStoreDirty = true; + await assert.rejects(writer.writeNow(pending), /unreadable|missing CCR/, + 'unreadable authoritative originals must not be replaced by a fresh writer'); + assert.equal(db.prepare('SELECT ccr_json FROM sessions WHERE session_id=?').get(pending.id)?.ccr_json, corrupt); + assert.deepEqual(db.prepare('SELECT * FROM ccr_entries WHERE session_id=?').all(pending.id), bindings); + assert.deepEqual(db.prepare(`SELECT c.hash, c.bytes FROM content c JOIN ccr_entries e + ON c.hash=e.content_hash WHERE e.session_id=?`).all(pending.id), bodies); + assert.equal(pending.contentStoreDirty, true); + } finally { writer?.closeQuietly(); db.close(); } +}); +} + +test('a writer advancing after the stale check is refused at the atomic commit boundary', async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'foreign-counter-race-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('shared-counter-race'); + a.inFlight = 1; + let second: SessionStore | undefined; + const record = UnifiedShadow.prototype.record; + let competing: Promise | undefined; + try { + await first.writeNow(a); + second = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const b = loadWithOriginals(second, a); + assert.ok(b?.contentStore); + b.contentStore = storeOriginal(b.contentStore, { ref: 'm00002', rawId: 'advanced', + text: 'Advanced original Ω\u0000', kind: 'tool', tokens: 9, head: 'advanced' }); + b.contentStoreDirty = true; + b.stats.requests++; + let injected = false; + let published = 0; + first.setSessionCommitListener(() => { published++; }); + const other = second; + UnifiedShadow.prototype.record = function (...args) { + if (!injected && args[4]) { + injected = true; + competing = other.writeNow(b); // writer-off executes synchronously + } + return record.apply(this, args); + }; + await assert.rejects(first.writeNow(a), /session generation advanced|SQLite ownership changed/); + UnifiedShadow.prototype.record = record; + await competing; + assert.ok(injected, 'competing commit must land after the host stale check'); + assert.equal(published, 0); + assert.deepEqual(first.unifiedCcrFor(a.id), second.unifiedCcrFor(a.id)); + assert.ok(first.unifiedCcrFor(a.id)?.byRef.m00002); + } finally { UnifiedShadow.prototype.record = record; second?.closeQuietly(); first.closeQuietly(); } +}); + +test('a newer request may intentionally reset block numbering and originals', async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'newer-reset-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('newer-reset'); + a.inFlight = 1; + a.state.nextBlockId = 2; + try { + await first.writeNow(a); + a.state = createInitialState(); + a.contentStore = undefined; + a.contentStoreDirty = true; + a.stats.requests++; + a.revisionEpoch = (a.revisionEpoch ?? 0) + 1; + await first.writeNow(a); + assert.deepEqual(first.unifiedCcrFor(a.id), { version: 1, byHash: {}, byRef: {} }); + } finally { first.closeQuietly(); } +}); + +test('the originals read and ingest exclude a competing SQLite writer in one transaction', async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'foreign-atomic-lock-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('shared-atomic-lock'); + a.inFlight = 1; + let outsider: DatabaseSync | undefined; + const record = UnifiedShadow.prototype.record; + const getCcr = UnifiedStore.prototype.getCcr; + try { + await first.writeNow(a); + outsider = new DatabaseSync(join(dir, 'index.db')); + outsider.exec('PRAGMA busy_timeout = 0'); + outsider.exec('UPDATE sessions SET updated_at = updated_at + 1'); + let armed = false; + let attempted = false; + UnifiedShadow.prototype.record = function (...args) { + if (args[4]) armed = true; + return record.apply(this, args); + }; + const connection = outsider; + UnifiedStore.prototype.getCcr = function (...args) { + const current = getCcr.apply(this, args); + if (armed) { + armed = false; + attempted = true; + try { + assert.throws(() => connection.exec('BEGIN IMMEDIATE'), /locked|busy/i, + 'a second writer must be excluded after the originals read and before ingest'); + } finally { + try { connection.exec('ROLLBACK'); } catch { /* no transaction when correctly locked */ } + } + } + return current; + }; + await first.writeNow(a); + assert.ok(attempted, 'the lock control must reach the actual durable-originals read'); + } finally { + UnifiedShadow.prototype.record = record; + UnifiedStore.prototype.getCcr = getCcr; + outsider?.close(); + first.closeQuietly(); + } +}); + +test('changed original bytes under an unchanged hash are refused', async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'invalid-original-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('invalid-original'); + a.inFlight = 1; + try { + await first.writeNow(a); + const durable = structuredClone(first.unifiedCcrFor(a.id)); + assert.ok(a.contentStore); + const hash = a.contentStore.byRef.m00001.hash; + a.contentStore.byHash[hash] = 'Changed bytes under an unchanged hash'; + a.contentStoreDirty = true; + await assert.rejects(first.writeNow(a), /invalid content hash/); + assert.deepEqual(first.unifiedCcrFor(a.id), durable); + assert.equal(a.contentStoreDirty, true); + } finally { first.closeQuietly(); } +}); + +test('ownership cannot adopt a foreign commit landing immediately after its COMMIT', async () => { + process.env.BILI_STORAGE_UNIFIED = 'full'; + process.env.BILI_STORAGE_JSON_WRITER = 'off'; + const dir = mkdtempSync(join(root, 'owned-version-stamp-race-')); + const first = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const a = fixture('owned-version-stamp-race'); + a.inFlight = 1; + const exec = DatabaseSync.prototype.exec; + let second: SessionStore | undefined; + let competing: Promise | undefined; + try { + await first.writeNow(a); + let armed = true; + DatabaseSync.prototype.exec = function (sql: string) { + const result = exec.call(this, sql); + if (sql === 'COMMIT' && armed) { + armed = false; + second = new SessionStore({ dir, enabled: true, debounceMs: 1, log: () => {} }); + const b = loadWithOriginals(second, a); + assert.ok(b?.contentStore); + b.contentStore = storeOriginal(b.contentStore, { ref: 'm00002', rawId: 'after-commit', + text: 'Foreign commit after lock release Ω\u0000', kind: 'tool', tokens: 9, head: 'foreign' }); + b.contentStoreDirty = true; + competing = second.writeNow(b); + } + return result; + }; + await first.writeNow(a); + DatabaseSync.prototype.exec = exec; + assert.ok(competing, 'the foreign commit must land inside the post-COMMIT stamping window'); + await competing; + assert.ok(first.unifiedCcrFor(a.id)?.byRef.m00002); + await first.writeNow(a); + assert.ok(first.unifiedCcrFor(a.id)?.byRef.m00002, 'the next save must not erase the foreign original'); + } finally { DatabaseSync.prototype.exec = exec; second?.closeQuietly(); first.closeQuietly(); } +}); diff --git a/tests/seq-ledger-pr3.test.ts b/tests/seq-ledger-pr3.test.ts index 0bd9b6b27c..2275ba9f1c 100644 --- a/tests/seq-ledger-pr3.test.ts +++ b/tests/seq-ledger-pr3.test.ts @@ -15,6 +15,10 @@ import { join } from "node:path"; * is refused with renumber guidance instead of folding the wrong range. * S15: cold load re-stamps legacy counter refs onto seq+1 exactly once and * records the superseded map on the session. + * S16: cold load KEEPS legacy refs when mapped ids own no rows (#5599: an + * imported collapsed ledger must not be stamped — the stamp would + * evict ghost bindings and re-target saved citations at newer bodies); + * a complete ledger still aligns (control half). */ process.env.NODE_ENV = "test"; process.env.BILI_STORAGE_UNIFIED = "full"; @@ -245,3 +249,101 @@ test("S15: citation-space alignment defers from cold load to the next fold save } rmSync(dir, { recursive: true, force: true }); }); + +test("S16: deferred alignment keeps legacy refs when mapped ids own no rows (#5599 folded-tail import)", async () => { + const refOf = (i: number): string => `m${String(i).padStart(5, "0")}`; + const idOf = (i: number): string => (i <= 6 ? `o${i}` : `t${i - 6}`); + + const dir = join(testRoot, `s16-${Math.random().toString(36).slice(2)}`); + const { SessionStore, _setStoreForTest } = await import("../src/persist.ts"); + try { + // Imported collapsed ledger: o1..o6 are folded (bindings live in the + // ref ledger, NO message rows), t1..t3 are the resident tail re-based + // at seq 0..2, and the saved summary cites m00001. Seeded exactly like + // an import does — one ingest of the record as-is. + const legacy = legacyLike("pr3-folded-tail", [msg("t1", "assistant"), msg("t2"), msg("t3")]); + const byRaw: Record = {}; + const byRef: Record = {}; + for (let i = 1; i <= 9; i++) { + byRaw[idOf(i)] = refOf(i); + byRef[refOf(i)] = idOf(i); + } + legacy.state.messageRefs = { byRaw, byRef }; + legacy.state.blocks = [{ + blockId: "b1", + runId: "r1", + tier: 1, + summary: "Earlier evidence: m00001.", + directMessageIds: ["o1", "o2", "o3", "o4", "o5", "o6"], + effectiveMessageIds: ["o1", "o2", "o3", "o4", "o5", "o6"], + directBlockIds: [], + compressedTokens: 60, + createdAt: Date.now(), + survivedCount: 0, + generation: "young", + active: true, + startRef: "m00001", + endRef: "m00006", + }]; + legacy.state.deadRefs = ["m00001", "m00002", "m00003", "m00004", "m00005", "m00006"]; + legacy.contentStore = { + version: 1, + byHash: {}, + byRef: { + m00001: msg("o1", "user", "original body one"), + m00007: msg("t1", "assistant", "tail body one"), + }, + }; + const st = UnifiedStore.open(join(dir, "index.db"), undefined as never) as unknown as UnifiedStore; + st.ingestSession(planIngest(legacy)); + st.close(); + + // Cold load from the index alone (fresh store over the same dir). + const store = new SessionStore({ enabled: true, dir, debounceMs: 0 }); + _setStoreForTest(store); + const reloaded = store.loadSync("pr3-folded-tail"); + assert.ok(reloaded, "session loads from the index"); + // #5608-D: alignment runs from the post-fold save, not at cold load. + (reloaded as { pendingLedgerAlign?: boolean }).pendingLedgerAlign = true; + assert.ok(store.flushSync(reloaded!), "post-fold save commits"); + for (let i = 1; i <= 9; i++) { + assert.equal(reloaded!.state.messageRefs.byRaw[idOf(i)], refOf(i), `${idOf(i)} keeps its legacy ref`); + } + assert.notEqual((reloaded!.state as { refLedgerAligned?: boolean }).refLedgerAligned, true, "incomplete ledger: alignment skipped"); + assert.equal(reloaded!.restampSuperseded, undefined); + // The saved citation still resolves to the original body, not the tail. + const ccr = reloaded!.contentStore as { byRef?: Record } | undefined; + assert.equal(ccr?.byRef?.["m00001"]?.text, "original body one", "m00001 retrieves o1's original"); + assert.equal(ccr?.byRef?.["m00007"]?.text, "tail body one", "tail binding untouched"); + store.closeQuietly(); + + // Control: the SAME ref map over a COMPLETE ledger (every mapped id + // owns a live row) must still align — here it converges to a no-op, + // proving the guard discriminates on row coverage, not on folds. + const dir2 = join(testRoot, `s16c-${Math.random().toString(36).slice(2)}`); + try { + const complete = legacyLike("pr3-complete", Array.from({ length: 9 }, (_, k) => msg(idOf(k + 1)))); + complete.state.messageRefs = { byRaw, byRef }; + const st2 = UnifiedStore.open(join(dir2, "index.db"), undefined as never) as unknown as UnifiedStore; + st2.ingestSession(planIngest(complete)); + st2.close(); + const store2 = new SessionStore({ enabled: true, dir: dir2, debounceMs: 0 }); + _setStoreForTest(store2); + const aligned = store2.loadSync("pr3-complete"); + assert.ok(aligned, "complete-ledger session loads"); + (aligned as { pendingLedgerAlign?: boolean }).pendingLedgerAlign = true; + assert.ok(store2.flushSync(aligned!), "post-fold save commits"); + assert.equal((aligned!.state as { refLedgerAligned?: boolean }).refLedgerAligned, true, "complete ledger still aligns"); + for (let i = 1; i <= 9; i++) { + assert.equal(aligned!.state.messageRefs.byRaw[idOf(i)], refOf(i), `${idOf(i)} already at seq+1 — no-op alignment`); + } + store2.closeQuietly(); + } finally { + _setStoreForTest({} as never); + rmSync(dir2, { recursive: true, force: true }); + } + } finally { + _setStoreForTest({} as never); + } + rmSync(dir, { recursive: true, force: true }); +}); diff --git a/tests/storage-durability.test.ts b/tests/storage-durability.test.ts index cb672b6328..b952c1bd5c 100644 --- a/tests/storage-durability.test.ts +++ b/tests/storage-durability.test.ts @@ -18,6 +18,33 @@ delete process.env.BILI_STORAGE_UNIFIED; import { UnifiedStore } from "../src/storage/store.ts"; import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { statSync, utimesSync } from "node:fs"; + +/** Damage a referenced page, not potentially unused bytes in an empty DB. */ +function corruptSessionsPage(path: string): Buffer { + const { db } = openSqliteDatabase(path); + let rootpage: number; + try { + const row = db.prepare("SELECT rootpage FROM sqlite_master WHERE type = 'table' AND name = 'sessions'").get() as { rootpage: number }; + assert.ok(row?.rootpage > 0, "sessions must have a real b-tree root"); + rootpage = row.rootpage; + } finally { + db.close(); + } + const raw = readFileSync(path); + const storedPageSize = raw.readUInt16BE(16); + const pageSize = storedPageSize === 1 ? 65536 : storedPageSize; + const header = (rootpage - 1) * pageSize + (rootpage === 1 ? 100 : 0); + assert.ok([2, 5, 10, 13].includes(raw[header]), "fixture starts with a valid b-tree page type"); + raw[header] = 0xff; + const before = statSync(path); + writeFileSync(path, raw); + // Force a new integrity scan after this same-size write. Deliberately + // preserving metadata is a separate integrity-cache contract. + utimesSync(path, before.atime, new Date(before.mtimeMs + 2000)); + return raw; +} function freshDir(): string { return mkdtempSync(join(tmpdir(), "storage-durability-")); @@ -54,16 +81,8 @@ test("quick_check: a corrupt db fails open with a typed error, files left UNTOUC const store = UnifiedStore.open(path); store.close(); - // 2. Corrupt the main db file (flip bytes in the middle of the page area). - const raw = readFileSync(path); - for (let i = Math.floor(raw.length / 3); i < Math.floor(raw.length / 3) + 4096; i++) { - raw[i] = raw[i] ^ 0xff; - } - writeFileSync(path, raw); - // The flip keeps size + mtime (4 ms granularity), so the #4609 verdict - // cache (index.db.chk) would still match and skip quick_check — drop it - // so the reopen re-scans from fresh evidence (#4599). - rmSync(`${path}.chk`, { force: true }); + // 2. Damage a live page and invalidate the previous clean verdict. + corruptSessionsPage(path); // 3. Reopen — must fail with a typed, greppable integrity error // (migrations must never run against a corrupt file). @@ -144,16 +163,8 @@ test("shadow open ALARMS on a corrupt index.db set — nothing renamed, boot pro const store = UnifiedStore.open(path); store.close(); - // Corrupt it (page area flip) and remember its exact bytes. - const raw = readFileSync(path); - for (let i = Math.floor(raw.length / 3); i < Math.floor(raw.length / 3) + 4096; i++) { - raw[i] = raw[i] ^ 0xff; - } - writeFileSync(path, raw); - // Drop the #4609 verdict cache: the flip keeps size + mtime, so a - // cached "ok" would skip quick_check and the shadow would open the - // corrupt set instead of alarming (#4599). - rmSync(`${path}.chk`, { force: true }); + // Damage a live page and remember its exact bytes. + const raw = corruptSessionsPage(path); const beforeHash = createHash("sha256").update(raw).digest("hex"); // The shadow's maybeOpen (full mode) must ALARM, not refuse — come diff --git a/tests/storage-json-writer-off-json-writes.test.ts b/tests/storage-json-writer-off-json-writes.test.ts index da8fea4e6e..8047accfd5 100644 --- a/tests/storage-json-writer-off-json-writes.test.ts +++ b/tests/storage-json-writer-off-json-writes.test.ts @@ -5,6 +5,10 @@ import * as fs from "node:fs"; import * as os from "node:os"; import * as path from "node:path"; import { SessionStore, contentStoreRelPathFor, relPathFor, storageStatusLabel } from "../src/persist.ts"; +// Strict SQL reads verify every original against its sha256 key, so the +// grafted stores use real content hashes. +const sha256 = (text: string): string => createHash("sha256").update(text, "utf8").digest("hex"); +const H1 = sha256("THE ORIGINAL LONG TEXT"); /** @@ -154,8 +158,8 @@ test("jsonWriter=off: only whitelisted .json files may be written in the session const session = loadCorpusCase(seed, "ccr", "corpus-ccr", "responses"); session.contentStore = { version: 1, - byHash: { h1: "THE ORIGINAL LONG TEXT" }, - byRef: { m00001: { hash: "h1", rawId: "raw-1", kind: "file read", tokens: 5, chars: 22, head: "ORIGINAL HEAD" } }, + byHash: { [H1]: "THE ORIGINAL LONG TEXT" }, + byRef: { m00001: { hash: H1, rawId: "raw-1", kind: "file read", tokens: 5, chars: 22, head: "ORIGINAL HEAD" } }, } as typeof session.contentStore; session.contentStoreDirty = true; const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jwgate-full-")); diff --git a/tests/storage-json-writer-off.test.ts b/tests/storage-json-writer-off.test.ts index d5fd32d0cc..a983ae8370 100644 --- a/tests/storage-json-writer-off.test.ts +++ b/tests/storage-json-writer-off.test.ts @@ -1,10 +1,16 @@ import { test } from "node:test"; import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; import * as fs from "node:fs"; import * as os from "node:os"; import * as path from "node:path"; import { SessionStore, storageStatusLabel } from "../src/persist.ts"; import { UnifiedStore } from "../src/storage/store.ts"; +// Strict SQL reads verify every original against its sha256 key, so the +// grafted stores use real content hashes. +const sha256 = (text: string): string => createHash("sha256").update(text, "utf8").digest("hex"); +const H1 = sha256("THE ORIGINAL LONG TEXT"); +const H2 = sha256("SHADOW WRITE"); /** @@ -165,8 +171,8 @@ test("jsonWriter=off (full): CCR content-store sidecar is not written; the index // original entries, the graft holds only h1). session.contentStore = { version: 1, - byHash: { h1: "THE ORIGINAL LONG TEXT" }, - byRef: { m00001: { hash: "h1", rawId: "raw-1", kind: "file read", tokens: 5, chars: 22, head: "ORIGINAL HEAD" } }, + byHash: { [H1]: "THE ORIGINAL LONG TEXT" }, + byRef: { m00001: { hash: H1, rawId: "raw-1", kind: "file read", tokens: 5, chars: 22, head: "ORIGINAL HEAD" } }, } as typeof session.contentStore; session.contentStoreDirty = true; const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); @@ -185,7 +191,7 @@ test("jsonWriter=off (full): CCR content-store sidecar is not written; the index try { const mirror = fresh.unifiedCcrFor(session.id); assert.ok(mirror, "the unified ccr mirror must carry the store"); - assert.equal(mirror!.byHash["h1"], "THE ORIGINAL LONG TEXT", "CCR originals must survive via the index mirror"); + assert.equal(mirror!.byHash[H1], "THE ORIGINAL LONG TEXT", "CCR originals must survive via the index mirror"); fresh.cancelAll(); } finally { fresh.closeQuietly(); @@ -201,8 +207,8 @@ test("jsonWriter=off (full): CCR content-store sidecar is not written; the index assert.ok(again, "session must still load for the control leg"); again.contentStore = { version: 1, - byHash: { h2: "SHADOW WRITE" }, - byRef: { m00002: { hash: "h2", rawId: "raw-2", kind: "tool result", tokens: 3, chars: 12, head: "SHADOW HEAD" } }, + byHash: { [H2]: "SHADOW WRITE" }, + byRef: { m00002: { hash: H2, rawId: "raw-2", kind: "tool result", tokens: 3, chars: 12, head: "SHADOW HEAD" } }, } as typeof again.contentStore; again.contentStoreDirty = true; await store2.writeNow(again); diff --git a/tests/storage-migrate-originals.test.ts b/tests/storage-migrate-originals.test.ts new file mode 100644 index 0000000000..d9e08d7008 --- /dev/null +++ b/tests/storage-migrate-originals.test.ts @@ -0,0 +1,293 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createContentStore, createInitialState, storeOriginal, type MessageContentStore } from "acp-kernel"; +import { SessionStore, contentStoreRelPathFor, relPathFor } from "../src/persist.ts"; +import { createStorageCodec } from "../src/encrypt.ts"; +import { runStorageMigrateCommand } from "../src/storage-maintenance.ts"; +import type { Session } from "../src/session.ts"; +import { rmrf } from "./tmp-rm.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const id = "migration-originals"; +const meta: Session["meta"] = { protocol: "anthropic", upstreamOrigin: "https://originals.example" }; +const originals = storeOriginal(createContentStore(), { + ref: "m90001", rawId: "original-message", text: "café\nexact second line\r\n🧪", + kind: "shell output", tokens: 12, head: "exact original", command: "printf original", +}); + +async function seed(dir: string, content = originals, requests = 1): Promise { + process.env.BILI_STORAGE_UNIFIED = "legacy"; + const store = new SessionStore({ dir, enabled: true }); + const session: Session = { + id, meta, stats: { requests, tokensSaved: 0, inputTokens: 0, cachedTokens: 0, outputTokens: 0, cacheSamples: 0, lastInputTokens: 0, contextTokens: 0, compressCreditTokens: 0, retrieveCalls: 0, retrieveHits: 0, retrieveMisses: 0, storedBytes: 0, storeBytesSaved: 0, rangeRestores: 0 }, + metadata: {}, state: createInitialState(), createdAt: Date.now(), lastSeen: Date.now(), + blockContents: new Map(), inFlight: 0, persisted: false, pendingRetrievals: [], + contentStore: content, contentStoreDirty: true, + }; + try { await store.writeNow(session); } + finally { await store.closeQuietlyAsync(); process.env.BILI_STORAGE_UNIFIED = "full"; } +} + +async function indexedOriginals(dir: string): Promise { + const store = new SessionStore({ dir, enabled: true }); + try { return structuredClone(store.unifiedCcrFor(id)); } + finally { await store.closeQuietlyAsync(); } +} + +test("migration preserves paired originals and rerun verifies without rewriting", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-")); + try { + await seed(dir); + const source = join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)); + const before = readFileSync(source); + const first = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(first.code, 0, first.text); + assert.deepEqual(await indexedOriginals(dir), originals, "SQLite must retain exact paired originals"); + const second = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(second.code, 0, second.text); + assert.equal(JSON.parse(second.text).counts.indexed, 1); + assert.equal(JSON.parse(second.text).counts.imported, 0); + assert.deepEqual(readFileSync(source), before, "source originals remain untouched"); + } finally { rmrf(dir); } +}); + +test("corrupt paired originals fail migration before a session is acknowledged", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-corrupt-")); + try { + await seed(dir); + const source = join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)); + writeFileSync(source, "not valid JSON"); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 1, "corrupt originals cannot report successful migration"); + const store = new SessionStore({ dir, enabled: true }); + try { assert.deepEqual(store.unifiedListSessionIds(), [], "failed source must not be partially imported"); } + finally { await store.closeQuietlyAsync(); } + assert.equal(readFileSync(source, "utf8"), "not valid JSON"); + } finally { rmrf(dir); } +}); + +test("paired originals retain an independently matching session reference owner", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-owner-match-")); + try { + await seed(dir); + const file = join(dir, relPathFor(id, meta.protocol, meta.upstreamOrigin)); + const record = JSON.parse(readFileSync(file, "utf8")); + record.payload.state.messageRefs.byRef.m90001 = originals.byRef.m90001.rawId; + record.payload.state.messageRefs.byRaw[originals.byRef.m90001.rawId] = "m90001"; + writeFileSync(file, JSON.stringify(record)); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 0, result.text); + assert.deepEqual(await indexedOriginals(dir), originals); + } finally { rmrf(dir); } +}); + +test("paired originals retain frozen aliases for pruned message IDs without rekeying", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-frozen-alias-")); + try { + await seed(dir); + const file = join(dir, relPathFor(id, meta.protocol, meta.upstreamOrigin)); + const record = JSON.parse(readFileSync(file, "utf8")); + delete record.payload.state.messageRefs.byRaw[originals.byRef.m90001.rawId]; + record.payload.state.messageRefs.byRef.m90001 = "a-different-known-message"; + record.payload.state.messageRefs.byRaw["a-different-known-message"] = "m90001"; + writeFileSync(file, JSON.stringify(record)); + const before = readFileSync(file); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 0, "an unmapped frozen original is allowed by the existing fork alias contract"); + assert.deepEqual(await indexedOriginals(dir), originals, "migration must preserve the original keys and bodies exactly"); + assert.deepEqual(readFileSync(file), before, "source refs must not be repaired or reassigned"); + } finally { rmrf(dir); } +}); + +test("paired originals under a pre-renumbering ref are re-keyed to the message's current ref", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-rekey-")); + try { + await seed(dir); + const file = join(dir, relPathFor(id, meta.protocol, meta.upstreamOrigin)); + const source = join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)); + const record = JSON.parse(readFileSync(file, "utf8")); + record.payload.state.messageRefs.byRef.m90001 = "a-different-known-message"; + record.payload.state.messageRefs.byRaw["a-different-known-message"] = "m90001"; + record.payload.state.messageRefs.byRaw[originals.byRef.m90001.rawId] = "m90002"; + record.payload.state.messageRefs.byRef.m90002 = originals.byRef.m90001.rawId; + writeFileSync(file, JSON.stringify(record)); + const before = readFileSync(file); + const sourceBefore = readFileSync(source); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 0, result.text); + const indexed = await indexedOriginals(dir); + assert.deepEqual(indexed?.byRef, { m90002: originals.byRef.m90001 }, "the original follows its message, never the other owner of m90001"); + assert.deepEqual(indexed?.byHash, originals.byHash); + assert.deepEqual(readFileSync(file), before, "source refs stay untouched"); + assert.deepEqual(readFileSync(source), sourceBefore, "source originals stay untouched"); + } finally { rmrf(dir); } +}); + +test("paired originals cannot attach to an inconsistent session ref map", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-owner-")); + try { + await seed(dir); + const file = join(dir, relPathFor(id, meta.protocol, meta.upstreamOrigin)); + const record = JSON.parse(readFileSync(file, "utf8")); + record.payload.state.messageRefs.byRaw[originals.byRef.m90001.rawId] = "m90001"; + record.payload.state.messageRefs.byRef.m90001 = "a-different-known-message"; + record.payload.state.messageRefs.byRaw["a-different-known-message"] = "m90001"; + writeFileSync(file, JSON.stringify(record)); + const before = readFileSync(file); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 1, "a ref the map assigns both ways to different messages must not be imported"); + assert.equal(JSON.parse(result.text).counts.failed, 1); + const store = new SessionStore({ dir, enabled: true }); + try { assert.deepEqual(store.unifiedListSessionIds(), [], "wrong-owner source must not be partially imported"); } + finally { await store.closeQuietlyAsync(); } + assert.deepEqual(readFileSync(file), before); + } finally { rmrf(dir); } +}); + +test("already-indexed sessions cannot hide changed originals on rerun", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-drift-")); + try { + await seed(dir); + assert.equal((await runStorageMigrateCommand({ dir, json: true })).code, 0); + const changed = storeOriginal(createContentStore(), { + ref: "m90001", rawId: "different-message", text: "different bytes", + kind: "shell output", tokens: 4, head: "different", + }); + writeFileSync(join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)), JSON.stringify(changed)); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 1, "source/index originals divergence must be reported"); + assert.deepEqual(await indexedOriginals(dir), originals, "rerun must not replace indexed originals"); + } finally { rmrf(dir); } +}); + +test("missing body in a paired originals store is a failed migration", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-missing-")); + try { + await seed(dir); + const broken = { ...originals, byHash: {} }; + writeFileSync(join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)), JSON.stringify(broken)); + assert.equal((await runStorageMigrateCommand({ dir, json: true })).code, 1); + } finally { rmrf(dir); } +}); + +test("dry-run reports corrupt originals and does not import them", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-dry-")); + try { + await seed(dir); + writeFileSync(join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)), "broken"); + const result = await runStorageMigrateCommand({ dir, json: true, dryRun: true }); + assert.equal(result.code, 0, "dry-run retains the existing CLI exit-code contract"); + assert.equal(JSON.parse(result.text).counts.failed, 1); + } finally { rmrf(dir); } +}); + +test("cold compressed paired originals migrate exactly", async () => { + const root = mkdtempSync(join(tmpdir(), "bili-migrate-originals-cold-")); + const dir = join(root, "sessions"); + try { + await seed(dir); + const sidecar = join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)); + const codec = createStorageCodec({ compress: true }); + assert.ok(codec); + writeFileSync(sidecar, codec.encode(JSON.stringify(originals))); + renameSync(dir, `${dir}-cold`); + mkdirSync(dir); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 0, result.text); + assert.deepEqual(await indexedOriginals(dir), originals); + assert.ok(JSON.parse(result.text).results[0].file.startsWith("cold:")); + } finally { rmrf(root); } +}); + +test("ahead import retains prior indexed originals alongside new source originals", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-ahead-")); + try { + await seed(dir); + assert.equal((await runStorageMigrateCommand({ dir, json: true })).code, 0); + const next = storeOriginal(createContentStore(), { + ref: "m90002", rawId: "next-message", text: "new original\n", + kind: "file read", tokens: 4, head: "next", + }); + await seed(dir, next, 2); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 0, result.text); + assert.deepEqual(await indexedOriginals(dir), { + version: 1, byHash: { ...originals.byHash, ...next.byHash }, byRef: { ...originals.byRef, ...next.byRef }, + }); + const rerun = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(rerun.code, 0, rerun.text); + assert.equal(JSON.parse(rerun.text).counts.indexed, 1); + assert.equal(JSON.parse(rerun.text).counts.imported, 0); + assert.deepEqual(await indexedOriginals(dir), { + version: 1, byHash: { ...originals.byHash, ...next.byHash }, byRef: { ...originals.byRef, ...next.byRef }, + }); + } finally { rmrf(dir); } +}); + +test("ahead import refuses conflicting indexed references without replacing them", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-conflict-")); + try { + await seed(dir); + assert.equal((await runStorageMigrateCommand({ dir, json: true })).code, 0); + const conflict = storeOriginal(createContentStore(), { + ref: "m90001", rawId: "conflicting-message", text: "conflicting bytes", + kind: "file read", tokens: 4, head: "conflict", + }); + await seed(dir, conflict, 2); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 1); + assert.match(result.text, /conflicting CCR original reference/); + assert.deepEqual(await indexedOriginals(dir), originals); + } finally { rmrf(dir); } +}); + +test("embedded fork originals cannot silently conflict with the paired sidecar", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-fork-")); + try { + await seed(dir); + const file = join(dir, relPathFor(id, meta.protocol, meta.upstreamOrigin)); + const record = JSON.parse(readFileSync(file, "utf8")); + record.payload.forkContentStore = storeOriginal(createContentStore(), { + ref: "m90001", rawId: "fork-message", text: "fork bytes", + kind: "file read", tokens: 4, head: "fork", + }); + writeFileSync(file, JSON.stringify(record)); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 1); + assert.match(result.text, /conflicting CCR original reference/); + } finally { rmrf(dir); } +}); + +test("an old partial index is reported drifted when its originals are absent", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-partial-")); + try { + await seed(dir, createContentStore()); + assert.equal((await runStorageMigrateCommand({ dir, json: true })).code, 0); + await seed(dir); + const result = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(result.code, 1); + assert.equal(JSON.parse(result.text).counts.drifted, 1); + assert.match(result.text, /CCR originals/); + } finally { rmrf(dir); } +}); + +test("UUID-only original reference drift is rejected without replacing indexed originals", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-originals-uuid-")); + try { + const initial = storeOriginal(createContentStore(), { + ref: "m90001", rawId: "00000000-0000-4000-8000-000000000001", text: "stable original bytes", + kind: "file read", tokens: 4, head: "stable", + }); + await seed(dir, initial); + assert.equal((await runStorageMigrateCommand({ dir, json: true })).code, 0); + const changed = structuredClone(initial); + changed.byRef.m90001.rawId = "00000000-0000-4000-8000-000000000002"; + writeFileSync(join(dir, contentStoreRelPathFor(id, meta.protocol, meta.upstreamOrigin)), JSON.stringify(changed)); + const rerun = await runStorageMigrateCommand({ dir, json: true }); + assert.equal(rerun.code, 1, "UUID normalization must not hide exact original identity drift"); + assert.deepEqual(await indexedOriginals(dir), initial); + } finally { rmrf(dir); } +}); diff --git a/tests/storage-migrate.test.ts b/tests/storage-migrate.test.ts new file mode 100644 index 0000000000..bac5ff0636 --- /dev/null +++ b/tests/storage-migrate.test.ts @@ -0,0 +1,259 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { tmpdir } from "node:os"; +import { appendFileSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { join, relative, sep } from "node:path"; +import { createInitialState } from "acp-kernel"; +import { SessionStore } from "../src/persist.ts"; +import type { Session } from "../src/session.ts"; +import { runStorageMigrateCommand } from "../src/storage-maintenance.ts"; +import { rmrf } from "./tmp-rm.ts"; + +// Plain-JSON envelopes (bypass the codec) and the v0.1.203+ default mode: +// migrate is tested against a store configured exactly like a current install. +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_STORAGE_UNIFIED = "full"; + +function makeSession(id: string, meta: Session["meta"] = {}, requests = 0): Session { + return { + id, + meta, + stats: { requests, tokensSaved: 0, inputTokens: 0, cachedTokens: 0, outputTokens: 0, cacheSamples: 0, lastInputTokens: 0, contextTokens: 0, compressCreditTokens: 0, retrieveCalls: 0, retrieveHits: 0, retrieveMisses: 0, storedBytes: 0, storeBytesSaved: 0, rangeRestores: 0 }, + metadata: {}, + state: createInitialState(), + createdAt: Date.now(), + lastSeen: Date.now(), + blockContents: new Map(), + inFlight: 0, + persisted: false, + pendingRetrievals: [], + }; +} + +async function writeSessions(dir: string, sessions: Array<{ id: string; meta?: Session["meta"]; requests?: number }>): Promise { + const store = new SessionStore({ dir, debounceMs: 5, enabled: true }); + try { + for (const s of sessions.map((x) => makeSession(x.id, x.meta, x.requests))) await store.writeNow(s); + } finally { + await store.closeQuietlyAsync(); + } +} + +/** Every file under dir (any name) keyed by relative path, minus the index.db + * family (the migration target, expected to appear). */ +function snapshotFiles(dir: string): Map { + const out = new Map(); + const walk = (cur: string): void => { + for (const name of readdirSync(cur)) { + if (name.startsWith("index.db")) continue; + const full = join(cur, name); + const st = statSync(full); + if (st.isDirectory()) walk(full); + else out.set(relative(dir, full), { size: st.size, mtimeMs: st.mtimeMs }); + } + }; + walk(dir); + return out; +} + +/** Fresh-install simulation: the writing instance's index.db is gone, only the + * legacy JSON files remain. */ +function dropIndex(dir: string): void { + for (const suffix of ["", "-wal", "-shm"]) rmSync(join(dir, `index.db${suffix}`), { force: true }); +} + +interface MigrateResultEntry { + id: string; + file: string; + status: "imported" | "indexed" | "drifted" | "failed" | "planned"; + reason?: string; + alternates?: number; + decodedBytes?: number; +} + +interface MigrateJson { + dryRun: boolean; + scannedFiles: number; + counts: { imported: number; indexed: number; drifted: number; failed: number; planned: number }; + results: MigrateResultEntry[]; + indexDbBefore?: number; + indexDbAfter?: number; +} + +async function runMigrate(dir: string, dryRun = false): Promise<{ code: number; out: MigrateJson }> { + const r = await runStorageMigrateCommand({ dir, dryRun, json: true }); + return { code: r.code, out: JSON.parse(r.text) as MigrateJson }; +} + +const byId = (out: MigrateJson, id: string): MigrateResultEntry => { + const hit = out.results.find((r) => r.id === id); + assert.ok(hit, `result entry for ${id}`); + return hit!; +}; + +test("bulk-imports legacy JSON sessions into a fresh index and reconciles clean", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [ + { id: "mig-a" }, + { id: "mig-b" }, + { id: "mig-c", meta: { protocol: "anthropic", upstreamOrigin: "https://a.example" } }, + ]); + dropIndex(dir); + const first = await runMigrate(dir); + assert.equal(first.code, 0); + assert.equal(first.out.counts.imported, 3); + assert.equal(first.out.counts.failed + first.out.counts.drifted, 0); + assert.equal(first.out.scannedFiles, 3); + for (const id of ["mig-a", "mig-b", "mig-c"]) assert.equal(byId(first.out, id).status, "imported"); + + // Delete the JSON corpus entirely: the data must be servable from the + // index alone (the jsonWriter-off end-state). + for (const name of readdirSync(dir)) { + const full = join(dir, name); + if (statSync(full).isDirectory()) rmrf(full); + } + const reader = new SessionStore({ dir, enabled: true }); + try { + for (const id of ["mig-a", "mig-b", "mig-c"]) assert.ok(reader.loadSync(id), `${id} served from the index alone`); + } finally { + await reader.closeQuietlyAsync(); + } + + // Re-run on the now file-less dir: nothing left to scan, clean exit. + const second = await runMigrate(dir); + assert.equal(second.code, 0); + assert.equal(second.out.scannedFiles, 0); + } finally { + rmrf(dir); + } +}); + +test("already-indexed sessions report indexed without re-writing", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [{ id: "keep-1" }, { id: "keep-2" }]); + // Without this the writing instance's own first-open auto-import has + // already filled the index, and the first migrate run only verifies. + dropIndex(dir); + const first = await runMigrate(dir); + assert.equal(first.out.counts.imported, 2); + const second = await runMigrate(dir); + assert.equal(second.code, 0); + assert.equal(second.out.counts.indexed, 2, "consistent sessions verify clean on re-run"); + assert.equal(second.out.counts.imported, 0, "no re-write of consistent sessions"); + } finally { + rmrf(dir); + } +}); + +test("dry-run plans without writing; the real run then imports", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [{ id: "dry-1" }, { id: "dry-2" }]); + dropIndex(dir); + const plan = await runMigrate(dir, true); + assert.equal(plan.code, 0); + assert.equal(plan.out.dryRun, true); + assert.equal(plan.out.counts.planned, 2); + assert.equal(plan.out.counts.imported, 0); + const real = await runMigrate(dir); + assert.equal(byId(real.out, "dry-1").status, "imported", "dry-run wrote nothing — the real run still has to import"); + assert.equal(byId(real.out, "dry-2").status, "imported"); + } finally { + rmrf(dir); + } +}); + +test("corrupt file is reported failed without stopping the rest", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [{ id: "ok-1" }, { id: "bad-1" }]); + const protoDir = join(dir, "_unknown"); + const badFile = join(protoDir, readdirSync(protoDir).find((n) => { + try { + return JSON.parse(readFileSync(join(protoDir, n), "utf8")).payload.id === "bad-1"; + } catch { + return false; + } + })!); + appendFileSync(badFile, "\nGARBAGE{{not-json"); + dropIndex(dir); + const r = await runMigrate(dir); + assert.equal(r.code, 1, "failed results exit non-zero"); + assert.equal(r.out.counts.failed, 1); + assert.equal(r.out.counts.imported, 1, "healthy sessions still import"); + const failed = r.out.results.find((x) => x.status === "failed")!; + assert.ok(failed.reason?.includes("unreadable"), `reason names the failure: ${failed.reason}`); + assert.equal(byId(r.out, "ok-1").status, "imported"); + } finally { + rmrf(dir); + } +}); + +test("same-id route files arbitrate to newest savedAt; alternates left untouched", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [{ id: "dup-1", meta: { protocol: "anthropic", upstreamOrigin: "https://a.example" } }]); + await new Promise((res) => setTimeout(res, 25)); + await writeSessions(dir, [{ id: "dup-1", meta: { protocol: "openai", upstreamOrigin: "https://b.example" }, requests: 1 }]); + dropIndex(dir); + const r = await runMigrate(dir); + assert.equal(r.code, 0); + const dup = byId(r.out, "dup-1"); + assert.equal(dup.alternates, 1, "the losing route file is reported, not merged"); + assert.equal(dup.status, "imported"); + assert.ok(dup.file.startsWith(`openai${sep}`), `newer route wins: ${dup.file}`); + } finally { + rmrf(dir); + } +}); + +test("migrate never modifies any file under the sessions dir", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [{ id: "ro-1" }, { id: "ro-2" }]); + dropIndex(dir); + const before = snapshotFiles(dir); + const r = await runMigrate(dir); + assert.equal(r.code, 0); + const after = snapshotFiles(dir); + assert.equal(after.size, before.size, "no files added or removed (quarantine renames included)"); + for (const [rel, meta] of before) assert.deepEqual(after.get(rel), meta, `${rel} untouched`); + } finally { + rmrf(dir); + } +}); + +test("already-indexed session whose file diverged reports drifted without re-writing", async () => { + const dir = mkdtempSync(join(tmpdir(), "bili-migrate-")); + try { + await writeSessions(dir, [{ id: "drift-1", requests: 1 }]); + dropIndex(dir); + const first = await runMigrate(dir); + assert.equal(byId(first.out, "drift-1").status, "imported"); + + // Diverge the source file WITHOUT making it ahead: bump a non-request + // stat so its digest no longer matches the index, while requests and + // nextBlockId stay put so the ahead-probe does not re-import it. + const protoDir = join(dir, "_unknown"); + const file = join(protoDir, readdirSync(protoDir).find((n) => { + try { + return JSON.parse(readFileSync(join(protoDir, n), "utf8")).payload.id === "drift-1"; + } catch { + return false; + } + })!); + const obj = JSON.parse(readFileSync(file, "utf8")); + obj.payload.stats.tokensSaved = 999999; + writeFileSync(file, JSON.stringify(obj)); + + const second = await runMigrate(dir); + assert.equal(second.code, 1, "drifted results exit non-zero"); + const d = byId(second.out, "drift-1"); + assert.equal(d.status, "drifted", `expected drifted, got ${d.status}: ${d.reason}`); + assert.equal(second.out.counts.imported, 0, "diverged-but-not-ahead session is not re-imported"); + } finally { + rmrf(dir); + } +}); diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index 8e4908ba87..a2f9f89c22 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -417,8 +417,11 @@ test("resolveStorageWalJournalSizeLimit: env wins, garbage falls through, file t * the LEGACY-mode view of the same files plus the deterministic one-time * ledger alignment (same seqByRaw the full-mode load aligned against). The * corpus manifest keeps pinning the PURE view (storage-corpus suite); this - * oracle pins the equivalence "full serve = legacy serve + alignment". */ -function alignedLegacyDigest(caseDir: string, meta: { protocol: string; id: string }, seqByRawId: Map): string { + * oracle pins the equivalence "full serve = legacy serve + alignment". + * #5599: the alignment is applied only when the one-time re-stamp's + * completeness guard would pass (every mapped id owns a row) — an imported + * collapsed ledger keeps its legacy refs, mirroring the cold-load path. */ +function alignedLegacyDigest(caseDir: string, meta: { protocol: string; id: string }, probe: UnifiedStore): string { const legacyDir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-legacy-oracle-")); try { fs.cpSync(caseDir, path.join(legacyDir, "case"), { recursive: true }); @@ -435,7 +438,16 @@ function alignedLegacyDigest(caseDir: string, meta: { protocol: string; id: stri assert.ok(pure, "legacy oracle load returned null"); const view = toLegacyLike(pure); const state = structuredClone(view.state as Record); - restampRefsToLedger(state as never, seqByRawId); + // #5599 completeness mirror of the cold-load re-stamp gate: an + // imported collapsed ledger (folded ghosts own no rows) keeps its + // legacy refs instead of being stamped. + const byRaw = (state.messageRefs as { byRaw: Record }).byRaw; + const mapped = Object.keys(byRaw).filter((rawId) => byRaw[rawId] !== "BLOCKED"); + const seqByRawId = probe.seqByRaw(meta.id); + const backed = probe.backedMsgIds(meta.id, mapped); + if (seqByRawId.size > 0 && backed !== null && !mapped.some((rawId) => !backed.has(rawId))) { + restampRefsToLedger(state as never, seqByRawId); + } delete state.refLedgerAligned; return sessionDigest({ ...view, state: state as never }); } finally { @@ -475,11 +487,19 @@ test("storage unified FULL mode: read path serves from the index with organic im // #5609 CI red: the deferred-alignment high-water mark must ride // kstate_json — dropping it diverged the second full-mode read // from the legacy view on exactly this key. + // #5599: the folded corpus imports a collapsed ledger (folded ghosts + // own no rows), so the completeness guard keeps its legacy refs and + // no high-water mark is minted; storage-state-fields pins the + // kstate_json round trip itself. const rowKstate = JSON.parse(probe.getSessionRow(meta.id)!.kstateJson) as Record; - assert.equal(typeof rowKstate.timestampsHighWaterRefSeq, "number", "timestampsHighWaterRefSeq must persist through kstate_json (#5608-C)"); + if ((session.state as { refLedgerAligned?: boolean }).refLedgerAligned === true) { + assert.equal(typeof rowKstate.timestampsHighWaterRefSeq, "number", "timestampsHighWaterRefSeq must persist through kstate_json (#5608-C)"); + } else { + assert.equal(rowKstate.timestampsHighWaterRefSeq, undefined, "unaligned ledger mints no high-water mark (#5599)"); + } // #4316 PR3/3: expected = legacy view of the same files + the // deterministic alignment against these very rows. - expectedDigest = alignedLegacyDigest(caseDir, meta, probe.seqByRaw(meta.id)); + expectedDigest = alignedLegacyDigest(caseDir, meta, probe); } finally { probe.close(); } @@ -581,7 +601,7 @@ test("storage unified: file-config storage.unified selects the mode; env tier st const probeStore = UnifiedStore.open(path.join(caseDir, "index.db")); let fileTierDigest = ""; try { - fileTierDigest = alignedLegacyDigest(caseDir, meta, probeStore.seqByRaw(meta.id)); + fileTierDigest = alignedLegacyDigest(caseDir, meta, probeStore); } finally { probeStore.close(); } diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 6d152d5023..4eca2815f6 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -444,8 +444,8 @@ entries: default: "365" env: BILI_STORAGE_ARCHIVE_AFTER_DAYS group: process - desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier." - desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。" + desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore , and `bili storage migrate [--dry-run]` (#5599: one-shot bulk import of every legacy JSON session into the unified index with per-session digest reconciliation — read-only over the JSON files; README 'Storage maintenance CLI' section). Env wins over the file tier." + desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore ,还有 `bili storage migrate [--dry-run]`(#5599:一次性把全部 legacy JSON 会话批量导入统一索引并逐会话摘要对账——对 JSON 文件只读;README「存储维护 CLI」一节)。环境变量优先于配置文件层。" desc_ja: "日単位の GC 年齢スイープウィンドウ (#3463, #3901): これより長くアイドル状態のセッションはアーカイブされる - 単一のアーカイブログ (セッションディレクトリ配下の archive/archive.jsonl — セッションごとのファイルスプレーなし) に 1 行の JSON が追記され、トombストーン行がラベルとアイデンティティを保持し、コンテンツ/メディア参照が解放され、リストアは 1 つの POST のみで可能。0 は年齢スイープを無効にする; POST /__bili/storage/gc は常に実行され、既存の index.db を縮小するには ?vacuum=full を取る (新規書き込みには常に auto_vacuum=INCREMENTAL が有効)。同じ操作に CLI レイヤーがある (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]、および bili sessions ls/rm/archive/restore (README の 'Storage maintenance CLI' セクション)。環境変数がファイル階層より優先される。" - key: sessions type: object