diff --git a/CHANGELOG.md b/CHANGELOG.md index d282ef99a7..610a941968 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,16 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ## [Unreleased] +### 存储后端热路径接线(#3388 第三步 / #3931) + +- `storage.backend="mysql"` 接进代理热路径:保存经异步影子 facade 进串行后端队列(水位 delta、#3841 收割台账、digest 采样、三振自禁用,与 sqlite 路径逐条 lockstep);sqlite 保持零 await 同步快车道,行为逐字节不变。 +- 同步读在 mysql 上走 JSON 信封(sidecar 经 `ready()` 异步挂载;`viewFor`/`importRecord` 同步面按设计 miss);`loadEnvelope` 在异步后端下惰性播种磁盘索引,裸 id 冷加载能找到命名空间文件(仅 mysql,sqlite 保持 #1937 姿态)。 +- replicas/ext/GC/export 走异步孪生(`createReplica`/`listReplicas`/`deleteReplica`/`replicaByteLen`/`restoreReplica`/`unifiedGc`/`unifiedRestoreArchived` 全 async;admin 面 await 接线);`maybeCutForkBase` 异步体。 +- `writerOff` 仅 sqlite:mysql 的同步读路径就是信封,信封必须存在(启动提示写明)。 +- v1 降级面:incremental lane 与快照渲染在 mysql 上同步读 miss → 诚实降级(409 resync / storage-off);web 会话列表走磁盘 walk。 +- 测试:mysql e2e(热路径保存/信封纪律/replicas/ext/重启回温)+ contract 7/7 对真 MySQL 8.0;sqlite 家族全绿(replicas/unified/writer-off/full-flip/archive/reap/bounded/web-sessions 24/golden/export)。 + + ### 存储:MySQL 后端落地(#3388 第二步,#3931) - `src/storage/mysql.ts`:MysqlBackend 完整实现 StorageBackend 操作契约(全 ~40 方法:带 CAS 引用计数的 ingest(delta/全量/completeLog)、读取、session_ext/kv、收割台账、checkpoint、会话绑定、归档/恢复/GC、诊断)—— MySQL 8.0 方言,mysql2 连接池(打进 dependencies,#3931 拍板:纯 JS ~500KB,锁文件钉版本),DDL 为 v7 形状直接建表(bili_meta 表存 schema 版本替 PRAGMA user_version;JSON 列一律 LONGTEXT 保字节序,MySQL JSON 类型会重排键序破坏 digest)。 diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 8513b95715..635455d04c 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,8 +175,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is IMPLEMENTED (step 2): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path (shadow ingest) stays on sqlite until the step-3 async swap — the boot notice names that boundary; export speaks mysql today (storage.backend="mysql" + a DSN exports straight from MySQL). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. | -| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (export today; the proxy hot path swaps in step 3). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is FULLY WIRED (steps 2+3): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path is wired through the async shadow facade: saves enqueue onto a serial backend queue (watermark deltas, orphan/reap journal, digest sampling — sqlite keeps its zero-await sync fast path unchanged), SYNC reads ride the JSON envelope (the sidecar attaches asynchronously via ready(); viewFor/importRecord sync facades miss by design), replicas/ext/GC/export speak the async twins. writerOff stays sqlite-only (the envelope is the mysql sync read path). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. | +| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (hot path included — the async shadow queue). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | @@ -335,8 +335,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| | `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase. | -| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931): sqlite (default) or mysql (implemented, #3388 step 2 - openBackend speaks the full contract against MySQL 8.0; export works today, the proxy hot path swaps in step 3). Wins over the file tier. | -| `BILI_MYSQL_DSN` | string | unset | — | MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend="mysql" (#3388 step 2, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs. | +| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931): sqlite (default) or mysql (fully wired, #3388 steps 2+3 - saves ingest through the async shadow queue, sync reads ride the JSON envelope, replicas/ext/GC via the async twins; export too). Wins over the file tier. | +| `BILI_MYSQL_DSN` | string | unset | — | MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend="mysql" (#3388, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs. The pool opens lazily; closeQuietlyAsync drains the ingest queue and closes it. | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (one JSON line appended to the single archive log (archive/archive.jsonl) + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): compute the outbound snapshot view on every armed turn WITHOUT using it, classify the diff against the legacy outbound, and surface per-session counters in the web panel + the snapshot-shadow diagnostics topic. 0/false/off disables; 1/true/on enables; otherwise the config-file snapshot.shadow boolean applies (default on). The render lane itself stays rollout-gated (BILI_ROLLOUT_SNAPSHOT_RENDER), so the shadow lane never changes outbound bytes - it is the dry-run evidence that flips the rollout later. | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | Override the config file path (path relocation). | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index a90a146f75..6af262aaf7 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,8 +175,8 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已实现(第二步):openBackend("mysql", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径(shadow ingest)在第三步异步化改造前仍走 sqlite —— 启动提示会写明这一边界;export 今天就能说 mysql(storage.backend="mysql" + DSN 直接从 MySQL 导出)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。 | -| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | storage.backend="mysql" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(今天为 export;代理热路径第三步接入)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。 | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已完整接线(第二+三步):openBackend("mysql", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径经异步影子 facade 接线:保存进串行后端队列(水位 delta、孤儿/收割台账、digest 采样 —— sqlite 保持零 await 同步快车道不变),同步读走 JSON 信封(sidecar 经 ready() 异步挂载;viewFor/importRecord 同步面按设计 miss),replicas/ext/GC/export 走异步孪生。writerOff 仅 sqlite(信封就是 mysql 的同步读路径)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。 | +| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | storage.backend="mysql" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(含热路径 —— 异步影子队列)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | @@ -335,8 +335,8 @@ | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| | `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | 让 bili 本地应答 dsh 压缩摘要请求(返回折叠快照而非拒绝,#3733):身份头 + 内核渲染格式的块摘要,零出口流量;重放经 #2432 再基线识别。 | -| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已实现,#3388 第二步 —— openBackend 对 MySQL 8.0 说完整契约;export 今天可用,代理热路径第三步接入)。优先于文件层。 | -| `BILI_MYSQL_DSN` | string | unset | — | storage.backend="mysql" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388 第二步,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。 | +| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已接线,#3388 第二+三步 —— 保存经异步影子队列入 mysql,同步读走 JSON 信封,replicas/ext/GC 走异步孪生;export 同样支持)。优先于文件层。 | +| `BILI_MYSQL_DSN` | string | unset | — | storage.backend="mysql" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。连接池惰性打开;closeQuietlyAsync 排空入队事务并关闭。 | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(单文件归档日志 archive/archive.jsonl 追加一行 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。 | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | 快照渲染影子对账(#3756 后续):每轮已折叠会话都用引擎现算一次出站快照视图但不使用,与 legacy 实际出站比对分类,逐会话计数进 web 面板与 snapshot-shadow 诊断主题。0/false/off 关闭;1/true/on 开启;否则读配置文件 snapshot.shadow 布尔(默认开)。渲染车道本身仍受灰度开关控制(BILI_ROLLOUT_SNAPSHOT_RENDER),影子车道绝不改变出站字节 —— 它是后续翻转灰度的干跑证据。 | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | 覆盖配置文件路径(路径重定位)。 | diff --git a/package-lock.json b/package-lock.json index a7c080ece5..e8860f80fc 100644 --- a/package-lock.json +++ b/package-lock.json @@ -4248,7 +4248,7 @@ }, "node_modules/aws-ssl-profiles": { "version": "1.1.2", - "resolved": "http://127.0.0.1:4873/aws-ssl-profiles/-/aws-ssl-profiles-1.1.2.tgz", + "resolved": "https://registry.npmjs.org/aws-ssl-profiles/-/aws-ssl-profiles-1.1.2.tgz", "integrity": "sha512-NZKeq9AfyQvEeNlN0zSYAaWrmBffJh3IELMZfRpJVWgrpEbtEpnjvzqBPf+mxoI287JohRDoa+/nsfqqiZmF6g==", "license": "MIT", "engines": { @@ -4946,7 +4946,7 @@ }, "node_modules/denque": { "version": "2.1.0", - "resolved": "http://127.0.0.1:4873/denque/-/denque-2.1.0.tgz", + "resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz", "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==", "license": "Apache-2.0", "engines": { @@ -5596,7 +5596,7 @@ }, "node_modules/generate-function": { "version": "2.3.1", - "resolved": "http://127.0.0.1:4873/generate-function/-/generate-function-2.3.1.tgz", + "resolved": "https://registry.npmjs.org/generate-function/-/generate-function-2.3.1.tgz", "integrity": "sha512-eeB5GfMNeevm/GRYq20ShmsaGcmI81kIX2K9XQx5miC8KdHaC6Jm0qQ8ZNeGOi7wYB8OsdxKs+Y2oVuTFuVwKQ==", "license": "MIT", "dependencies": { @@ -6081,7 +6081,7 @@ }, "node_modules/is-property": { "version": "1.0.2", - "resolved": "http://127.0.0.1:4873/is-property/-/is-property-1.0.2.tgz", + "resolved": "https://registry.npmjs.org/is-property/-/is-property-1.0.2.tgz", "integrity": "sha512-Ks/IoX00TtClbGQr4TWXemAnktAQvYB7HzcCxDGqEZU6oCmb2INHuOoKxbtR+HFkmYWBKv/dOZtGRiAjDhj92g==", "license": "MIT" }, @@ -6430,7 +6430,7 @@ }, "node_modules/long": { "version": "5.3.2", - "resolved": "http://127.0.0.1:4873/long/-/long-5.3.2.tgz", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", "license": "Apache-2.0" }, @@ -6476,7 +6476,7 @@ }, "node_modules/lru.min": { "version": "1.1.5", - "resolved": "http://127.0.0.1:4873/lru.min/-/lru.min-1.1.5.tgz", + "resolved": "https://registry.npmjs.org/lru.min/-/lru.min-1.1.5.tgz", "integrity": "sha512-5J9ysMYUpYIg9RF2vJpy9SinEmSviFSe0GyPpCQ4L5QSkLAgeLXlTAOu2ZwWUU5m+0SBl6gUU1R1ZQB3aKypfA==", "license": "MIT", "engines": { @@ -6732,7 +6732,7 @@ }, "node_modules/mysql2": { "version": "3.15.3", - "resolved": "http://127.0.0.1:4873/mysql2/-/mysql2-3.15.3.tgz", + "resolved": "https://registry.npmjs.org/mysql2/-/mysql2-3.15.3.tgz", "integrity": "sha512-FBrGau0IXmuqg4haEZRBfHNWB5mUARw6hNwPDXXGg0XzVJ50mr/9hb267lvpVMnhZ1FON3qNd4Xfcez1rbFwSg==", "license": "MIT", "dependencies": { @@ -6752,7 +6752,7 @@ }, "node_modules/mysql2/node_modules/iconv-lite": { "version": "0.7.3", - "resolved": "http://127.0.0.1:4873/iconv-lite/-/iconv-lite-0.7.3.tgz", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz", "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==", "license": "MIT", "dependencies": { @@ -6780,7 +6780,7 @@ }, "node_modules/named-placeholders": { "version": "1.1.6", - "resolved": "http://127.0.0.1:4873/named-placeholders/-/named-placeholders-1.1.6.tgz", + "resolved": "https://registry.npmjs.org/named-placeholders/-/named-placeholders-1.1.6.tgz", "integrity": "sha512-Tz09sEL2EEuv5fFowm419c1+a/jSMiBjI9gHxVLrVdbUkkNUUfjsVYs9pVZu5oCon/kmRh9TfLEObFtkVxmY0w==", "license": "MIT", "dependencies": { @@ -7737,7 +7737,7 @@ }, "node_modules/seq-queue": { "version": "0.0.5", - "resolved": "http://127.0.0.1:4873/seq-queue/-/seq-queue-0.0.5.tgz", + "resolved": "https://registry.npmjs.org/seq-queue/-/seq-queue-0.0.5.tgz", "integrity": "sha512-hr3Wtp/GZIc/6DAGPDcV4/9WoZhjrkXsi5B/07QgX8tsdc6ilr7BFM6PM6rbdAX1kFSDYeZGLipIZZKyQP0O5Q==" }, "node_modules/serve-static": { @@ -7898,7 +7898,7 @@ }, "node_modules/sqlstring": { "version": "2.3.3", - "resolved": "http://127.0.0.1:4873/sqlstring/-/sqlstring-2.3.3.tgz", + "resolved": "https://registry.npmjs.org/sqlstring/-/sqlstring-2.3.3.tgz", "integrity": "sha512-qC9iz2FlN7DQl3+wjwn3802RTyjCx7sDvfQEXchwa6CWOx07/WVfh91gBmQ9fahw8snwGEWU3xGzOt4tFyHLxg==", "license": "MIT", "engines": { diff --git a/package.json b/package.json index 62f5d07c41..1e172e5d32 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "billion-context", "version": "0.1.202", - "description": "A context-compression plugin — small context windows (a 100K context is enough), 5x fewer tokens, month-long single sessions (billions of tokens), and compression quality — for all agents: pi, OpenCode, Codex, Claude Code, and more. billion-context is all you need.", + "description": "A context-compression plugin \u2014 small context windows (a 100K context is enough), 5x fewer tokens, month-long single sessions (billions of tokens), and compression quality \u2014 for all agents: pi, OpenCode, Codex, Claude Code, and more. billion-context is all you need.", "type": "module", "main": "dist/index.js", "exports": { @@ -60,6 +60,7 @@ "url": "https://github.com/ranxianglei/billion-context/issues" }, "dependencies": { + "mysql2": "^3.15.3", "zod": "4.1.8" }, "optionalDependencies": { diff --git a/src/persist.ts b/src/persist.ts index 6759f0101a..608e5a8650 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -1,5 +1,5 @@ import { createHash, randomUUID } from "node:crypto"; -import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; import { readdir, readFile, rm } from "node:fs/promises"; import * as path from "node:path"; const { join } = path; @@ -19,6 +19,7 @@ import { fileStorageArchiveAfterDays, fileStorageWalJournalSizeLimit, fileStorageBackend, + fileStorageMysqlDsn, } from "./knobs.js"; import { log as loggerLog } from "./logger.js"; import { recordRequestedMode } from "./storage/runtime-status.js"; @@ -30,6 +31,7 @@ import { markDirty, scrubTurnSeparatorIds } from "./session.js"; import { LazyBlockContents, isDroppedMetadata, lazyMetadata } from "./session.js"; import type { Session, BlockContent, BlockView } from "./session.js"; import { orphanContents, resolveStorageMode, UnifiedShadow, type StorageMode } from "./storage/shadow.js"; +import { resolveBackendName } from "./storage/backend.js"; import { toLegacyLike, type LegacySessionLike } from "./storage/ingest.js"; import { sha256Hex, type CheckpointRow } from "./storage/store.js"; import type { WireProtocol } from "./util.js"; @@ -370,6 +372,10 @@ export class SessionStore { private readonly readCodec: StateStoreCodec; private boundedReads = false; private diskFiles: string[] = []; + /** #3931 step 3: loadEnvelope lazily seeds the disk index on first use + * (network backends make the envelope the sync read path — bare-id + * loads must find namespaced files without a bootBounded walk). */ + private diskFilesSeeded = false; private readonly legacyAliases = new Map(); private readonly sourceFiles = new Map(); /** #3199: post-boot background alias scan (unified modes only) — null in @@ -379,7 +385,9 @@ export class SessionStore { * names (pid + seq makes collisions across processes/rapid saves * unreachable). */ private tmpSeq = 0; - private readonly shadow?: UnifiedShadow; + // #3931 step 3: assigned asynchronously for network backends (mysql) — + // see the constructor and ready(). + private shadow?: UnifiedShadow; /** #2960 ②: sessions whose fork_base checkpoint has been cut (or was * found already present) — memoized so steady-state writes of receipt * sessions do not re-query the checkpoints table every flush. */ @@ -391,6 +399,14 @@ export class SessionStore { * and file writes resume on the very next save (fail-safe: never a * window with no persistent copy at all). */ private readonly jsonWriterOff: boolean; + /** #3931 step 3: resolves when the async backend (mysql) opened (or + * failed to — the sidecar just stays undefined). Resolved immediately + * for sqlite/legacy boots. */ + private readonly mysqlReady: Promise = Promise.resolve(); + /** Await the storage backend before first use (mysql pool + DDL). */ + ready(): Promise { + return this.mysqlReady; + } /** #3133: writerOff pending builds, keyed by session id. markDirty fires * several times per request and the writerOff branch used to build * synchronously on EVERY call; a microtask-batched run per turn keeps @@ -458,16 +474,38 @@ export class SessionStore { // under shadow/legacy/encryption — there the JSON file is the sole // source of truth. const wantJsonWriterOff = knobResolveJsonWriterOff(process.env, fileStorageJsonWriter()); - this.jsonWriterOff = wantJsonWriterOff && storageMode === "full" && !unifiedBlockedByEncryption; + this.jsonWriterOff = wantJsonWriterOff && storageMode === "full" && !unifiedBlockedByEncryption && resolveBackendName(process.env, fileStorageBackend()) === "sqlite"; if (wantJsonWriterOff && !this.jsonWriterOff) { - baseLog("warn", "[storage] storage.jsonWriter=off ignored — requires storage.unified=full (under shadow/legacy the JSON envelope stays the sole source of truth)"); - } - this.shadow = this.enabled && !unifiedBlockedByEncryption - ? (UnifiedShadow.maybeOpen(this.sessionsDir, process.env, (rec) => { - const s = buildSession(rec as PersistedSession); - return toLegacyLike(s); - }, fileStorageUnified(), fileStorageWalJournalSizeLimit(), fileStorageBackend()) ?? undefined) - : undefined; + baseLog("warn", "[storage] storage.jsonWriter=off ignored — requires storage.unified=full on the sqlite backend (network backends keep the JSON envelope as the sync read path)"); + } + const legacyLoad = (rec: unknown): LegacySessionLike | null => { + const s = buildSession(rec as PersistedSession); + return toLegacyLike(s); + }; + // #3931 step 3: network backends (mysql) can only open asynchronously + // (pool + DDL), so the sidecar attaches when it resolves — until then + // this.shadow stays undefined and every surface keeps its documented + // no-sidecar posture (envelope reads/writes, banner, replicas off). + // `ready()` gates the async consumers (saves, boot, admin). + const backendName = resolveBackendName(process.env, fileStorageBackend()); + if (this.enabled && !unifiedBlockedByEncryption && backendName === "mysql") { + this.mysqlReady = UnifiedShadow.openAsync( + this.sessionsDir, + process.env, + legacyLoad, + fileStorageUnified(), + fileStorageWalJournalSizeLimit(), + fileStorageBackend(), + fileStorageMysqlDsn(), + ).then((sh) => { + this.shadow = sh ?? undefined; + }); + this.shadow = undefined; + } else { + this.shadow = this.enabled && !unifiedBlockedByEncryption + ? (UnifiedShadow.maybeOpen(this.sessionsDir, process.env, legacyLoad, fileStorageUnified(), fileStorageWalJournalSizeLimit(), fileStorageBackend()) ?? undefined) + : undefined; + } this.store = new StateStore({ dir: this.sessionsDir, version: PERSIST_VERSION, @@ -714,6 +752,7 @@ export class SessionStore { this.boundedReads = true; mkdirSync(this.sessionsDir, { recursive: true }); this.diskFiles = []; + this.diskFilesSeeded = true; this.legacyAliases.clear(); const warm: Array<{ file: string; id: string; meta: PersistedSession["meta"]; savedAt: number; bytes: number }> = []; const canonical = new Set(); @@ -1190,6 +1229,19 @@ export class SessionStore { return envelope; } } + // #3931 step 3: network backends make the envelope the sync read + // path — their viewFor/importRecord sync facades always miss, so + // bare-id loads must find namespaced files without a bootBounded + // walk. Seeded once, sync, ASYNC BACKENDS ONLY: the sqlite path + // keeps its boot-walk posture (#1937 detail cost accounting rides + // the web disk index, not this read). + if (!this.diskFilesSeeded && this.shadow?.isAsyncBackend) { + this.diskFiles = [ + ...walkJsonFilesSync(this.sessionsDir), + ...(existsSync(`${this.sessionsDir}-cold`) ? walkJsonFilesSync(`${this.sessionsDir}-cold`) : []), + ].filter((f) => !path.basename(f).endsWith(".content-store.json")); + this.diskFilesSeeded = true; + } // A bounded boot does not seed the kernel's discovered-path map. // Probe indexed paths lazily, live before cold, and reject corrupt // matching files instead of silently resuming with empty fold state. @@ -1543,6 +1595,12 @@ export class SessionStore { * rows whose file vanished). null = legacy/degraded — plain file walk. */ webSummaryBaseline(): "authoritative" | "reconcile" | null { if (!this.shadow || this.shadow.stats.disabled) return null; + // #3931 step 3 (v1): the web summary surface reads the sync facade — + // on a network backend those rows are unavailable, and an EMPTY + // authoritative row set would blank the session list. Ride the disk + // walk (envelopes are always written there) until the async twin is + // wired into this surface. + if (this.shadow.isAsyncBackend) return null; return this.shadow.mode === "full" ? "authoritative" : "reconcile"; } @@ -1630,13 +1688,20 @@ export class SessionStore { * restoreReplica even after the session row and envelope are gone — * the substrate Phase 3 needs to stop writing the envelope blobs. */ private maybeCutForkBase(session: Session): void { + void this.maybeCutForkBaseAsync(session); + } + + /** #3931 step 3: the async twin body — sqlite resolves through the same + * sync calls (identical behavior, one promise hop); mysql rides the + * backend queue. Fire-and-forget from the save paths. */ + private async maybeCutForkBaseAsync(session: Session): Promise { if (!this.shadow || this.shadow.stats.disabled) return; const retained = session.metadata.publicForkReceipt !== undefined || session.metadata.publicSnapshotRetained === true; if (!retained || !Array.isArray(session.pluginSnapshot) || session.pluginSnapshot.length === 0) return; if (this.forkBaseCut.has(session.id)) return; try { - if (this.shadow.listCheckpoints(session.id).some((cp) => cp.kind === "fork_base")) { + if ((await this.shadow.listCheckpointsAsync(session.id)).some((cp) => cp.kind === "fork_base")) { this.forkBaseCut.add(session.id); return; } @@ -1645,7 +1710,7 @@ export class SessionStore { .filter((b) => (b as { active?: boolean }).active !== false) .map((b) => b.blockId) .sort(); - this.shadow.createCheckpoint({ + await this.shadow.insertCheckpointAsync({ cpId: `cp-${randomUUID()}`, sessionId: session.id, seqAt: session.pluginSnapshot!.length, @@ -1662,9 +1727,9 @@ export class SessionStore { } } - createReplica(sessionId: string, opts: { label?: string; pinned?: boolean; kind?: "auto" | "manual" | "replica" | "fork_base" } = {}): CheckpointRow { + async createReplica(sessionId: string, opts: { label?: string; pinned?: boolean; kind?: "auto" | "manual" | "replica" | "fork_base" } = {}): Promise { const shadow = this.liveShadow(); - const view = shadow.viewFor(sessionId); + const view = await shadow.viewForAsync(sessionId); if (!view) throw new Error(`[storage] session ${sessionId} is not in the unified index (unknown, or never persisted under this mode)`); const record = viewToRecord(view); // CCR originals must survive the rollback the replica exists for: @@ -1679,10 +1744,10 @@ export class SessionStore { .filter((b) => (b as { active?: boolean }).active !== false) .map((b) => b.blockId) .sort(); - shadow.createCheckpoint({ + await shadow.insertCheckpointAsync({ cpId, sessionId, - seqAt: this.shadow!.sessionTipSeq(sessionId), + seqAt: await this.shadow!.sessionTipSeqAsync(sessionId), blockSetHash: sha256Hex(JSON.stringify(activeBlockIds)), kind: opts.kind ?? "replica", label: opts.label ?? null, @@ -1690,18 +1755,19 @@ export class SessionStore { createdAt: Date.now(), stateJson: JSON.stringify(record), }); - const row = shadow.getCheckpoint(cpId); + const row = await shadow.getCheckpointAsync(cpId); if (!row) throw new Error(`[storage] checkpoint ${cpId} missing right after insert (index inconsistency)`); return row; } - listReplicas(sessionId?: string): CheckpointRow[] { - return this.liveShadow().listCheckpoints(sessionId); + async listReplicas(sessionId?: string): Promise { + await this.ready(); + return this.liveShadow().listCheckpointsAsync(sessionId); } /** Blob size of one checkpoint's frozen record (for detail views). */ - replicaByteLen(cpId: string): number { - return this.liveShadow().checkpointState(cpId).byteLen; + async replicaByteLen(cpId: string): Promise { + return (await this.liveShadow().checkpointStateAsync(cpId)).byteLen; } /** Materializes a checkpoint as a NEW session (the original is never @@ -1710,9 +1776,9 @@ export class SessionStore { * first-class citizen from the first read. */ async restoreReplica(cpId: string, opts: { sessionId?: string } = {}): Promise<{ sessionId: string; restoredFrom: string; seqAt: number }> { const shadow = this.liveShadow(); - const cp = shadow.getCheckpoint(cpId); + const cp = await shadow.getCheckpointAsync(cpId); if (!cp) throw new Error(`[storage] checkpoint not found: ${cpId}`); - const { stateJson } = shadow.checkpointState(cpId); + const { stateJson } = await shadow.checkpointStateAsync(cpId); const record = JSON.parse(stateJson) as PersistedSession; const newId = opts.sessionId ?? `restore-${randomUUID().slice(0, 8)}`; // A caller-chosen id must never silently overwrite a live session — @@ -1744,8 +1810,8 @@ export class SessionStore { /** Drops a checkpoint. Explicit operator action overrides `pinned` — * pinning protects against GC, not against deletion. */ - deleteReplica(cpId: string): boolean { - return this.liveShadow().deleteCheckpoint(cpId); + async deleteReplica(cpId: string): Promise { + return this.liveShadow().deleteCheckpointAsync(cpId); } /** #1937: sessions directory this store resolves against (the web summary @@ -2347,6 +2413,17 @@ export class SessionStore { this.shadow?.close(); } + /** #3931 step 3: async twin of closeQuietly for network backends — + * awaits the mysql sidecar open (no-op for sqlite/legacy), cancels + * pending saves, then drains the ingest queue and closes the pool. + * Tests and short-lived processes on mysql MUST use this or the pool + * keeps the event loop alive forever. */ + async closeQuietlyAsync(): Promise { + await this.ready(); + this.cancelAll(); + await this.shadow?.closeAsync(); + } + /** Whether a write is currently pending (debounce timer armed) for a id. */ hasPending(id: string): boolean { return this.saveTimers.has(id) || this.writerOffPending.has(id); @@ -2692,6 +2769,22 @@ function isStaleTempName(name: string): boolean { return name.startsWith(".tmp-") || /\.tmp-enc-\d+-\d+$/.test(name); } +/** #3931 step 3 sync twin — the lazy disk index for loadEnvelope. */ +function walkJsonFilesSync(dir: string, out: string[] = []): string[] { + let entries; + try { + entries = readdirSync(dir, { withFileTypes: true }); + } catch { + return out; + } + for (const e of entries) { + const full = path.join(dir, e.name); + if (e.isDirectory()) walkJsonFilesSync(full, out); + else if (e.isFile() && (isStaleTempName(e.name) || e.name.endsWith(".json"))) out.push(full); + } + return out; +} + async function walkJsonFiles(dir: string): Promise { const entries = await readdir(dir, { withFileTypes: true }); const out: string[] = []; diff --git a/src/server/admin.ts b/src/server/admin.ts index b95a71984b..9da0777bc5 100644 --- a/src/server/admin.ts +++ b/src/server/admin.ts @@ -467,7 +467,7 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv } const qVacuum = (() => { try { return new URL(req.url!, "http://localhost").searchParams.get("vacuum"); } catch { return null; } })(); const vacuum = qVacuum === "full" || qVacuum === "incremental" ? qVacuum : parsed.vacuum === "full" ? "full" : parsed.vacuum === "incremental" ? "incremental" : undefined; - const result = store.unifiedGc({ + const result = await store.unifiedGc({ archiveAfterDays: typeof parsed.archiveAfterDays === "number" && Number.isFinite(parsed.archiveAfterDays) && parsed.archiveAfterDays >= 0 ? parsed.archiveAfterDays : undefined, vacuum, }); @@ -500,7 +500,7 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv res.end(JSON.stringify({ ok: false, error: "restore requires the unified store" })); return; } - const result = store.unifiedRestoreArchived(parsed.sessionId, typeof parsed.archiveDir === "string" && parsed.archiveDir ? parsed.archiveDir : undefined); + const result = await store.unifiedRestoreArchived(parsed.sessionId, typeof parsed.archiveDir === "string" && parsed.archiveDir ? parsed.archiveDir : undefined); log("info", `[admin] POST /__bili/storage/restore ${parsed.sessionId} → ${JSON.stringify(result)}`); res.writeHead(result?.ok ? 200 : 404, { "content-type": "application/json" }); res.end(JSON.stringify({ ok: result?.ok === true, error: result?.reason ?? (result ? undefined : "unified store unavailable") })); @@ -536,7 +536,7 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv const resident = peekSession(parsed.sessionId); if (resident) await store.writeNow(resident); try { - const row = store.createReplica(parsed.sessionId, { + const row = await store.createReplica(parsed.sessionId, { label: typeof parsed.label === "string" ? parsed.label : undefined, pinned: parsed.pinned === true, }); @@ -610,7 +610,7 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv res.end(JSON.stringify({ ok: false, error: "replicas require the unified store — start this instance with BILI_STORAGE_UNIFIED=shadow|full" })); return; } - const deleted = store.deleteReplica(cpId); + const deleted = await store.deleteReplica(cpId); res.writeHead(deleted ? 200 : 404, { "content-type": "application/json" }); res.end(JSON.stringify({ ok: deleted, deleted })); return; @@ -870,7 +870,7 @@ async function sendReplicasList(res: http.ServerResponse, url: string): Promise< return; } const q = new URL(url, "http://localhost").searchParams.get("sessionId"); - const rows = store.listReplicas(q ?? undefined); + const rows = await store.listReplicas(q ?? undefined); res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ ok: true, replicas: rows.map(serializeReplica) }, null, 2)); } @@ -889,13 +889,13 @@ async function sendReplicaDetail(res: http.ServerResponse, url: string): Promise return; } try { - const row = store.listReplicas().find((r) => r.cp_id === cpId); + const row = (await store.listReplicas()).find((r) => r.cp_id === cpId); if (!row) { res.writeHead(404, { "content-type": "application/json" }); res.end(JSON.stringify({ ok: false, error: `checkpoint not found: ${cpId}` })); return; } - const byteLen = store.replicaByteLen(cpId); + const byteLen = await store.replicaByteLen(cpId); res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ ok: true, replica: { ...serializeReplica(row), byteLen } }, null, 2)); } catch (err) { diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index 32967d0774..bc4ec3300f 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -167,7 +167,7 @@ function getStorageRuntimeStatus(): StorageRuntimeStatus { degraded, intentionalOptOut, reason: degraded ? degradeReason() : null, - engine: sidecar ? { name: sidecar.engine, source: sidecar.engineSource } : null, + engine: sidecar ? sidecar.engineInfo : null, bootstrap: state.bootstrap?.status ?? null, bootstrapFailure: cd.marker, cooldownRemainingMs: cd.remainingMs, diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index b3f50449f2..d4315d9af9 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -26,7 +26,7 @@ import { planIngest, planIngestDelta, loadLegacyView, completeBlockContents, typ import type { BlockView } from "../session.js"; import { resolveStorageMode, resolveStorageWalJournalSizeLimit, type StorageMode } from "../knobs.ts"; import { UnifiedStore, type CheckpointRow, type SessionSummaryRow, type WebSummaryRow } from "./store.ts"; -import { backendSelectionNotice, selectedBackendRaw } from "./backend.ts"; +import { resolveBackendName, backendSelectionNotice, selectedBackendRaw, openBackend, type StorageBackend } from "./backend.ts"; import { canonicalize, sha256 } from "./canonical.ts"; import { recordSidecarInstance, recordSidecarOpenFailure } from "./runtime-status.ts"; import type { SqliteEngineName, SqliteEngineSource } from "./driver.ts"; @@ -104,19 +104,47 @@ export class UnifiedShadow { * until a save verifies clean again; surfaced on /__bili/status wire as * driftedSessions so drift is observable, not just logged. */ readonly driftedSessions = new Set(); - private readonly store: UnifiedStore; + /** #3931 step 3: the sqlite fast lane. Null when the selected backend + * is a network backend (mysql) — then every sync facade method below + * returns its miss value (null / [] / no-op) and callers fall back to + * the JSON envelope exactly as they do when no sidecar opened, while + * the async twins route through `backend`. The sqlite path stays + * byte-identical to the pre-step-3 behavior. */ + private readonly sync: UnifiedStore | null; + /** Async contract backend (mysql when sync === null; SqliteBackend + * wrapper is NOT constructed on the sqlite path — record()/viewFor() + * keep calling the raw store synchronously). */ + private readonly backend: StorageBackend | null = null; + /** Serializes async-backend work (record queue) the way the sync store + * implicitly serialized inside one event-loop turn. */ + private queueTail: Promise = Promise.resolve(); private readonly legacyLoad: (record: unknown) => LegacySessionLike | null; private consecutiveFailures = 0; /** #3133: per-session ingest counters (verify sampling). Per-process * only — nothing persisted, no schema change. */ private readonly ingestCounts = new Map(); - private constructor(mode: StorageMode, store: UnifiedStore, legacyLoad: (record: unknown) => LegacySessionLike | null) { + private constructor(mode: StorageMode, store: UnifiedStore | null, legacyLoad: (record: unknown) => LegacySessionLike | null, backend?: StorageBackend) { this.mode = mode; - this.store = store; + this.sync = store; + if (store === null) this.backend = backend ?? null; this.legacyLoad = legacyLoad; } + /** #3931 step 3: true when this shadow rides a network backend (mysql). + * Sync facade methods miss; async twins carry the traffic. */ + get isAsyncBackend(): boolean { + return this.sync === null; + } + + /** Honest engine label for status surfaces (sqlite keeps the raw store + * introspection; mysql reports itself). */ + get engineInfo(): { name: string; source: string } { + return this.sync + ? { name: this.sync.engine, source: this.sync.engineSource } + : { name: "mysql", source: "network" }; + } + /** Opens the sidecar when a knob asks for one; null (no cost) otherwise. */ static maybeOpen( dir: string, @@ -175,18 +203,52 @@ export class UnifiedShadow { return shadow; } + /** #3931 step 3: async open for network backends. sqlite resolves + * synchronously through maybeOpen (same instance, zero behavior + * change); mysql opens the pool + runs DDL via backend.init() and can + * only be awaited — SessionStore assigns this.shadow when it resolves + * and `ready()` gates the async consumers (saves/boot/admin). */ + static async openAsync( + dir: string, + env: NodeJS.ProcessEnv = process.env, + legacyLoad?: (record: unknown) => LegacySessionLike | null, + fileUnified?: string, + fileWalLimit?: number, + fileBackend?: string, + fileMysqlDsn?: string, + ): Promise { + const backendName = resolveBackendName(env, fileBackend); + if (backendName !== "mysql") return UnifiedShadow.maybeOpen(dir, env, legacyLoad, fileUnified, fileWalLimit, fileBackend); + const mode = resolveStorageMode(env, fileUnified); + if (mode === "legacy") return null; + const dsn = (env.BILI_MYSQL_DSN ?? fileMysqlDsn ?? "").trim(); + if (!dsn) { + loggerLog("warn", "[storage] storage.backend=mysql selected but no DSN is configured (BILI_MYSQL_DSN or storage.mysql.dsn) — staying legacy-only; see the CONFIGURATION storage section"); + recordSidecarOpenFailure("mysql backend selected without a DSN"); + return null; + } + try { + const backend = await openBackend("mysql", dsn); + await backend.init?.(); + const shadow = new UnifiedShadow(mode, null, legacyLoad ?? (() => null), backend); + recordSidecarInstance(shadow); + loggerLog("info", `[storage] unified ${mode === "full" ? "FULL" : "shadow"} mode on the MYSQL backend (dsn host ${dsn.replace(/:[^:@/]*@/, ":***@")}) — async shadow facade; JSON envelopes keep being written (writerOff is sqlite-only)`); + return shadow; + } catch (err) { + const errStr = String(err); + loggerLog("warn", `[storage] mysql backend failed to open (${errStr}) — staying legacy-only`); + recordSidecarOpenFailure(errStr); + return null; + } + } + /** True when reads should go unified-first (full mode, sidecar healthy). */ get readsUnified(): boolean { return this.mode === "full" && !this.stats.disabled; } - get engine(): SqliteEngineName { - return this.store.engine; - } - - get engineSource(): SqliteEngineSource { - return this.store.engineSource; - } + // engine/engineSource introspection lives in engineInfo (sqlite-only + // diagnostics widened for network backends); runtime-status reads that. /** Ingest + verify one committed record. Never throws. * @@ -197,6 +259,12 @@ export class UnifiedShadow { * available; both digest legs go through the same override, so parity * accounting stays honest. */ record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore): void { + // #3931 step 3: network backend — the ingest runs on the serial + // queue (fire-and-forget; failures feed the same 3-strike disable). + if (this.sync === null) { + this.enqueueAsyncRecord(record, rawTail, liveCcr); + return; + } if (this.stats.disabled) return; try { const loaded = this.legacyLoad(record); @@ -222,13 +290,13 @@ export class UnifiedShadow { const reapedIds = new Set(); const orphans = new Map(); try { - for (const row of this.store.getBlocks(loaded.id)) { + for (const row of this.sync.getBlocks(loaded.id)) { if (!claimed.has(row.blockId)) reapedIds.add(row.blockId); } for (const [blockId, view] of loaded.blockContents) { if (!claimed.has(blockId)) orphans.set(blockId, view); } - if (reapedIds.size > 0 || orphans.size > 0) this.store.reapBlocks(loaded.id, reapedIds, orphans); + if (reapedIds.size > 0 || orphans.size > 0) this.sync.reapBlocks(loaded.id, reapedIds, orphans); } catch { // journal failure must not break the save path } @@ -259,7 +327,7 @@ export class UnifiedShadow { const plan = planIngest(legacy); // #3285: only a raw-tail plan is the complete log — that is // what lets the full path sweep foreign rows as stale. - this.store.ingestSession(plan, undefined, { completeLog: hasRawTail }); + this.sync.ingestSession(plan, undefined, { completeLog: hasRawTail }); } this.stats.ingests++; // #3133 sampled verification: ingest integrity is already guarded @@ -271,14 +339,14 @@ export class UnifiedShadow { this.ingestCounts.set(legacy.id, count); if ((count - 1) % VERIFY_EVERY_INGESTS === 0) { this.stats.verifySamples++; - const sessionRow = this.store.getSessionRow(legacy.id); + const sessionRow = this.sync.getSessionRow(legacy.id); if (!sessionRow) throw new Error(`session row missing after ingest: ${legacy.id}`); const rows = { session: sessionRow, - messages: this.store.getMessages(legacy.id), - blocks: this.store.getBlocks(legacy.id), - deadRefs: this.store.getDeadRefs(legacy.id), - refs: this.store.getRefs(legacy.id), + messages: this.sync.getMessages(legacy.id), + blocks: this.sync.getBlocks(legacy.id), + deadRefs: this.sync.getDeadRefs(legacy.id), + refs: this.sync.getRefs(legacy.id), }; const loaded = loadLegacyView(rows); const expect = shadowDigest({ ...legacy, blockContents: orderedBlockContents(completeBlockContents(legacy.blockContents, legacy.state.blocks)) }); @@ -297,7 +365,7 @@ export class UnifiedShadow { // Drop the watermark so the next save re-points every // tail slot through the exact full path (divergence // window bounded to ≤ VERIFY_EVERY_INGESTS saves). - this.store.clearIngestWatermark(legacy.id); + this.sync.clearIngestWatermark(legacy.id); loggerLog("error", `[storage] unified shadow DIGEST DRIFT for ${legacy.id}: unified replay != legacy load view — dropped the ingest watermark, the next save re-ingests the full tail`); } else { this.stats.verifyOk++; @@ -324,17 +392,18 @@ export class UnifiedShadow { * never throw into record(), whose catch counts toward disabling the * shadow — a watermark hiccup is not a storage health problem. */ private ingestDelta(legacy: LegacySessionLike, hasRawTail: boolean): boolean { + if (this.sync === null) return false; try { - const mark = this.store.ingestWatermark(legacy.id); + const mark = this.sync.ingestWatermark(legacy.id); if (!mark) return false; // The pre-transaction probe is an optimization (skip building a // doomed plan); ingestSession re-verifies the boundary inside // BEGIN IMMEDIATE, which is the authoritative check. - const boundary = this.store.probeIngestBoundary(legacy.id, mark.seq); + const boundary = this.sync.probeIngestBoundary(legacy.id, mark.seq); if (!boundary) return false; const plan = planIngestDelta(legacy, mark.seq, mark.tailSeqs, { msgId: boundary.msgId, contentHash: boundary.contentHash }, hasRawTail); if (!plan) return false; - const applied = this.store.ingestSession(plan, { + const applied = this.sync.ingestSession(plan, { trustedSeq: mark.seq, boundaryMsgId: boundary.msgId, boundaryHash: boundary.contentHash, @@ -350,16 +419,17 @@ export class UnifiedShadow { /** Loads the unified view of one session (full mode read path). * Returns null on miss — the caller falls back to the legacy file. */ viewFor(id: string): LegacySessionLike | null { + if (this.sync === null) return null; if (this.stats.disabled) return null; try { - const sessionRow = this.store.getSessionRow(id); + const sessionRow = this.sync.getSessionRow(id); if (!sessionRow) return null; const rows = { session: sessionRow, - messages: this.store.getMessages(id), - blocks: this.store.getBlocks(id), - deadRefs: this.store.getDeadRefs(id), - refs: this.store.getRefs(id), + messages: this.sync.getMessages(id), + blocks: this.sync.getBlocks(id), + deadRefs: this.sync.getDeadRefs(id), + refs: this.sync.getRefs(id), }; // The view's `state` object carries every MERGE_STATE_FIELDS key by // construction; the declared Record just needs a @@ -374,6 +444,7 @@ export class UnifiedShadow { /** Organic import (full mode): ingest a legacy record the index has not * seen so future loads serve from the unified store. Never throws. */ importRecord(record: unknown): LegacySessionLike | null { + if (this.sync === null) return null; if (this.stats.disabled) return null; try { const legacy = this.legacyLoad(record); @@ -385,7 +456,7 @@ export class UnifiedShadow { this.stats.orphanSkips++; return null; } - this.store.ingestSession(planIngest(legacy)); + this.sync.ingestSession(planIngest(legacy)); this.stats.imports++; loggerLog("info", `[storage] unified import on first open: ${legacy.id}`); return legacy; @@ -397,10 +468,11 @@ export class UnifiedShadow { /** GC parity: remove the session's rows when the legacy files are deleted. */ remove(id: string): void { + if (this.sync === null) return; this.ingestCounts.delete(id); if (this.stats.disabled) return; try { - this.store.deleteSession(id); + this.sync.deleteSession(id); } catch (err) { loggerLog("warn", `[storage] unified remove failed for ${id} (${String(err)})`); } @@ -412,43 +484,50 @@ export class UnifiedShadow { // operator action, unlike the never-throw shadow mirrors above. createCheckpoint(cp: Parameters[0]): void { + if (this.sync === null) throw new Error('[storage] sync checkpoint API unavailable on a network backend — use the async twin'); this.assertLive(); - this.store.insertCheckpoint(cp); + this.sync.insertCheckpoint(cp); } listCheckpoints(sessionId?: string): CheckpointRow[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.listCheckpoints(sessionId); + return this.sync.listCheckpoints(sessionId); } getCheckpoint(cpId: string): CheckpointRow | undefined { + if (this.sync === null) return undefined; this.assertLive(); - return this.store.getCheckpoint(cpId); + return this.sync.getCheckpoint(cpId); } checkpointState(cpId: string): { stateJson: string; byteLen: number } { + if (this.sync === null) throw new Error('[storage] sync checkpoint API unavailable on a network backend — use the async twin'); this.assertLive(); - return this.store.checkpointState(cpId); + return this.sync.checkpointState(cpId); } deleteCheckpoint(cpId: string): boolean { + if (this.sync === null) return false; this.assertLive(); - return this.store.deleteCheckpoint(cpId); + return this.sync.deleteCheckpoint(cpId); } /** Ingest freshness watermark for load-side import gating (null = never * ingested). See SessionStore.buildForLoadAll for the race this guards. */ sessionUpdatedAt(sessionId: string): number | null { + if (this.sync === null) return null; this.assertLive(); - return this.store.sessionUpdatedAt(sessionId); + return this.sync.sessionUpdatedAt(sessionId); } /** Absolute-seq tip of the session's message log (+1 = next free seq). * Used as a checkpoint's seq_at. */ sessionTipSeq(sessionId: string): number { + if (this.sync === null) return 0; this.assertLive(); let tip = 0; - for (const m of this.store.getMessages(sessionId)) if (m.seq + 1 > tip) tip = m.seq + 1; + for (const m of this.sync.getMessages(sessionId)) if (m.seq + 1 > tip) tip = m.seq + 1; return tip; } @@ -456,14 +535,16 @@ export class UnifiedShadow { * full-mode boot union so rows whose legacy file disappeared remain * enumerable. */ listSessionIds(): string[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.listSessionIds(); + return this.sync.listSessionIds(); } /** Identity lookup passthrough (#2960 R2.1). */ lookupConversation(namespace: string, conversationId: string): string | null { + if (this.sync === null) return null; this.assertLive(); - return this.store.lookupConversation(namespace, conversationId); + return this.sync.lookupConversation(namespace, conversationId); } /** #3049: O(1) counter probe backing the persist stale-guard fast path. @@ -471,13 +552,15 @@ export class UnifiedShadow { * (this throws like every other pass-through; the guard treats any * failure as "no fast path" and falls back to the full envelope read). */ sessionCounters(sessionId: string): { requests: number; nextBlockId: number } | null { + if (this.sync === null) return null; this.assertLive(); - return this.store.sessionCounters(sessionId); + return this.sync.sessionCounters(sessionId); } listConversationBindings(namespace?: string): { namespace: string; conversationId: string; sessionId: string; claimedAt: number }[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.listConversationBindings(namespace); + return this.sync.listConversationBindings(namespace); } /** Summary fast-path rows (#2960 R2.1): export/web lists serve from one @@ -486,23 +569,27 @@ export class UnifiedShadow { * message JSON in seq order — exactly what planIngest interned). */ /** #3901/#3463: pass-through GC + restore (see UnifiedStore.gc). */ gc(opts: { archiveAfterDays: number; archiveDir: string; vacuum?: "full" | "incremental" }): { archived: number; contentRowsDeleted: number } { - return this.store.gc(opts); + if (this.sync === null) return { archived: 0, contentRowsDeleted: 0 }; + return this.sync.gc(opts); } restoreArchived(sessionId: string, opts: { archiveDir: string }): { ok: boolean; reason?: string } { - return this.store.restoreArchived(sessionId, opts); + if (this.sync === null) return { ok: false, reason: 'async backend — use restoreArchivedAsync' }; + return this.sync.restoreArchived(sessionId, opts); } rawMessages(sessionId: string): { seq: number; content: string }[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.getMessages(sessionId).map((m) => ({ seq: m.seq, content: m.content })); + return this.sync.getMessages(sessionId).map((m) => ({ seq: m.seq, content: m.content })); } /** #3756 snapshot render: the logical view's seq pointer list * (tail_json.tailSeqs). Null when absent/malformed. */ tailSeqs(sessionId: string): number[] | null { + if (this.sync === null) return null; this.assertLive(); - return this.store.tailSeqs(sessionId); + return this.sync.tailSeqs(sessionId); } /** #3756 snapshot render: full row view (id/ref/hash included) of the @@ -512,25 +599,29 @@ export class UnifiedShadow { * upgrade-tolerant interval path, contentHash for the exact-content * dedup path). */ tailViewRows(sessionId: string): MessageRow[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.getMessages(sessionId); + return this.sync.getMessages(sessionId); } sessionSummaries(): SessionSummaryRow[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.sessionSummaries(); + return this.sync.sessionSummaries(); } /** #2960 ③: whole-store CCR read from the sessions row's ccr mirror. */ getCcr(sessionId: string): MessageContentStore | null { + if (this.sync === null) return null; this.assertLive(); - return this.store.getCcr(sessionId); + return this.sync.getCcr(sessionId); } /** #2960 R2.2: web list rows (stats/metadata JSON + block aggregates). */ webSummaryRows(): WebSummaryRow[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.webSummaryRows(); + return this.sync.webSummaryRows(); } // ------------------------------------------------------------------ @@ -540,43 +631,332 @@ export class UnifiedShadow { // ------------------------------------------------------------------ setSessionExt(sessionId: string, key: string, value: unknown): void { + if (this.sync === null) return; this.assertLive(); - this.store.setSessionExt(sessionId, key, value); + this.sync.setSessionExt(sessionId, key, value); } getSessionExt(sessionId: string, key: string): unknown { + if (this.sync === null) return null; this.assertLive(); - return this.store.getSessionExt(sessionId, key); + return this.sync.getSessionExt(sessionId, key); } listSessionExt(sessionId: string): { key: string; value: unknown; updatedAt: number }[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.listSessionExt(sessionId); + return this.sync.listSessionExt(sessionId); } deleteSessionExt(sessionId: string, key?: string): number { + if (this.sync === null) return 0; this.assertLive(); - return this.store.deleteSessionExt(sessionId, key); + return this.sync.deleteSessionExt(sessionId, key); } setKv(namespace: string, key: string, value: unknown): void { + if (this.sync === null) return; this.assertLive(); - this.store.setKv(namespace, key, value); + this.sync.setKv(namespace, key, value); } getKv(namespace: string, key: string): unknown { + if (this.sync === null) return null; this.assertLive(); - return this.store.getKv(namespace, key); + return this.sync.getKv(namespace, key); } listKv(namespace?: string, keyPrefix?: string): { namespace: string; key: string; value: unknown; updatedAt: number }[] { + if (this.sync === null) return []; this.assertLive(); - return this.store.listKv(namespace, keyPrefix); + return this.sync.listKv(namespace, keyPrefix); } deleteKv(namespace: string, key?: string): number { + if (this.sync === null) return 0; this.assertLive(); - return this.store.deleteKv(namespace, key); + return this.sync.deleteKv(namespace, key); + } + + // ---- #3931 step 3: async facade (network backends) ---------------------- + // On sqlite every twin delegates to the sync method above (zero behavior + // change). On mysql the sync facades MISS by design (callers fall back to + // JSON envelopes exactly as they do when no sidecar opened) and these + // twins carry the traffic through the async StorageBackend contract. + + private enqueue(work: () => Promise): Promise { + const backend = this.backend; + if (!backend) return Promise.reject(new Error("[storage] async shadow work without a backend")); + const next = this.queueTail.then(work, work); + this.queueTail = next.then(() => undefined, () => undefined); + return next; + } + + /** record()'s async port — MUST stay in lockstep with the sync body + * above (orphan guard + #3841 reap journal, #3133 watermark delta, + * full ingest, #3133 sampled verify + drift latch). Serialized by the + * shadow queue; failures feed the same 3-strike disable. */ + private enqueueAsyncRecord(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore): void { + void this.enqueue(async () => { + if (this.stats.disabled) return; + try { + const backend = this.backend!; + const loaded0 = this.legacyLoad(record); + if (!loaded0) return; + let legacy = loaded0; + let hasRawTail = false; + if (rawTail && rawTail.lastMessages.length > 0) { + hasRawTail = true; + legacy = { ...legacy, lastMessages: rawTail.lastMessages, lastMessagesFolded: rawTail.lastMessagesFolded === true }; + } + if (liveCcr) legacy = { ...legacy, contentStore: liveCcr }; + // #3818/#3841: same orphan-guard + reap-journal semantics. + if (orphanContents(legacy)) { + const claimed = new Set(legacy.state.blocks.map((b) => b.blockId)); + const reapedIds = new Set(); + const orphans = new Map(); + for (const row of await backend.getBlocks(legacy.id)) { + if (!claimed.has(row.blockId)) reapedIds.add(row.blockId); + } + for (const [blockId, view] of legacy.blockContents) { + if (!claimed.has(blockId)) orphans.set(blockId, view); + } + if (reapedIds.size > 0 || orphans.size > 0) await backend.reapBlocks(legacy.id, reapedIds, orphans); + this.stats.orphanSkips++; + return; + } + // #3133 watermark delta. + let applied = false; + try { + const mark = await backend.ingestWatermark(legacy.id); + if (mark) { + const boundary = await backend.probeIngestBoundary(legacy.id, mark.seq); + if (boundary) { + const plan = planIngestDelta(legacy, mark.seq, mark.tailSeqs, { msgId: boundary.msgId, contentHash: boundary.contentHash }, hasRawTail); + if (plan) { + applied = await backend.ingestSession(plan, { trustedSeq: mark.seq, boundaryMsgId: boundary.msgId, boundaryHash: boundary.contentHash, trustedTailSeqs: mark.tailSeqs }); + if (applied) this.stats.deltaIngests++; + } + } + } + } catch { + applied = false; + } + if (!applied) await backend.ingestSession(planIngest(legacy), undefined, { completeLog: hasRawTail }); + this.stats.ingests++; + const count = (this.ingestCounts.get(legacy.id) ?? 0) + 1; + this.ingestCounts.set(legacy.id, count); + if ((count - 1) % VERIFY_EVERY_INGESTS === 0) { + this.stats.verifySamples++; + const sessionRow = await backend.getSessionRow(legacy.id); + if (!sessionRow) throw new Error(`session row missing after ingest: ${legacy.id}`); + const rows = { + session: sessionRow, + messages: await backend.getMessages(legacy.id), + blocks: await backend.getBlocks(legacy.id), + deadRefs: await backend.getDeadRefs(legacy.id), + refs: await backend.getRefs(legacy.id), + }; + const loaded = loadLegacyView(rows); + const expect = shadowDigest({ ...legacy, blockContents: orderedBlockContents(completeBlockContents(legacy.blockContents, legacy.state.blocks)) }); + const got = shadowDigest({ ...loaded, blockContents: orderedBlockContents(loaded.blockContents) }); + if (got !== expect) { + this.stats.verifyDrift++; + this.driftedSessions.add(legacy.id); + this.stats.lastError = `digest drift for ${legacy.id}: unified replay != legacy load view`; + await backend.clearIngestWatermark(legacy.id); + loggerLog("error", `[storage] unified shadow DIGEST DRIFT for ${legacy.id}: unified replay != legacy load view — dropped the ingest watermark, the next save re-ingests the full tail`); + } else { + this.stats.verifyOk++; + this.driftedSessions.delete(legacy.id); + } + } + this.consecutiveFailures = 0; + } catch (err) { + this.consecutiveFailures++; + this.stats.lastError = String(err); + if (this.consecutiveFailures >= MAX_CONSECUTIVE_FAILURES) { + this.stats.disabled = true; + loggerLog("warn", `[storage] unified shadow disabled after ${this.consecutiveFailures} consecutive failures (last: ${String(err)})`); + } + } + }); + } + + private viaAsync(miss: T, syncValue: () => T, via: () => Promise): Promise { + return this.sync === null ? via() : Promise.resolve(syncValue()); + } + + viewForAsync(id: string): Promise { + return this.viaAsync(null, () => this.viewFor(id), async () => { + if (this.stats.disabled) return null; + try { + const backend = this.backend!; + const sessionRow = await backend.getSessionRow(id); + if (!sessionRow) return null; + return loadLegacyView({ + session: sessionRow, + messages: await backend.getMessages(id), + blocks: await backend.getBlocks(id), + deadRefs: await backend.getDeadRefs(id), + refs: await backend.getRefs(id), + }) as LegacySessionLike; + } catch (err) { + loggerLog("warn", `[storage] unified load failed for ${id} (${String(err)}) — falling back to the legacy file`); + return null; + } + }); + } + + importRecordAsync(record: unknown): Promise { + return this.viaAsync(null, () => this.importRecord(record), async () => { + if (this.stats.disabled) return null; + try { + const legacy = this.legacyLoad(record); + if (!legacy) return null; + if (orphanContents(legacy)) { + this.stats.orphanSkips++; + return null; + } + await this.enqueue(() => this.backend!.ingestSession(planIngest(legacy))); + this.stats.imports++; + loggerLog("info", `[storage] unified import on first open: ${legacy.id}`); + return legacy; + } catch (err) { + loggerLog("warn", `[storage] unified import failed for a legacy record (${String(err)}) — serving the legacy view`); + return null; + } + }); + } + + removeAsync(id: string): Promise { + return this.viaAsync(undefined, () => this.remove(id), () => this.enqueue(() => this.backend!.deleteSession(id)).then(() => undefined)); + } + + listSessionIdsAsync(): Promise { + return this.viaAsync([], () => this.listSessionIds(), () => this.backend!.listSessionIds()); + } + + sessionSummariesAsync(): Promise { + return this.viaAsync([], () => this.sessionSummaries(), () => this.backend!.sessionSummaries()); + } + + webSummaryRowsAsync(): Promise { + return this.viaAsync([], () => this.webSummaryRows(), () => this.backend!.webSummaryRows()); + } + + lookupConversationAsync(namespace: string, conversationId: string): Promise { + return this.viaAsync(null, () => this.lookupConversation(namespace, conversationId), () => this.backend!.lookupConversation(namespace, conversationId)); + } + + listConversationBindingsAsync(namespace?: string): Promise<{ namespace: string; conversationId: string; sessionId: string; claimedAt: number }[]> { + return this.viaAsync([], () => this.listConversationBindings(namespace), () => this.backend!.listConversationBindings(namespace)); + } + + sessionCountersAsync(sessionId: string): Promise<{ requests: number; nextBlockId: number } | null> { + return this.viaAsync(null, () => this.sessionCounters(sessionId), () => this.backend!.sessionCounters(sessionId)); + } + + sessionUpdatedAtAsync(sessionId: string): Promise { + return this.viaAsync(null, () => this.sessionUpdatedAt(sessionId), () => this.backend!.sessionUpdatedAt(sessionId)); + } + + sessionTipSeqAsync(sessionId: string): Promise { + return this.viaAsync(0, () => this.sessionTipSeq(sessionId), async () => { + let tip = 0; + for (const m of await this.backend!.getMessages(sessionId)) if (m.seq + 1 > tip) tip = m.seq + 1; + return tip; + }); + } + + insertCheckpointAsync(cp: Parameters[0]): Promise { + return this.viaAsync(undefined, () => this.createCheckpoint(cp), () => this.enqueue(() => this.backend!.insertCheckpoint(cp)).then(() => undefined)); + } + + listCheckpointsAsync(sessionId?: string): Promise { + return this.viaAsync([], () => this.listCheckpoints(sessionId), () => this.backend!.listCheckpoints(sessionId)); + } + + getCheckpointAsync(cpId: string): Promise { + return this.viaAsync(undefined, () => this.getCheckpoint(cpId), () => this.backend!.getCheckpoint(cpId)); + } + + checkpointStateAsync(cpId: string): Promise<{ stateJson: string; byteLen: number }> { + return this.viaAsync({ stateJson: "", byteLen: 0 }, () => this.checkpointState(cpId), () => this.backend!.checkpointState(cpId)); + } + + deleteCheckpointAsync(cpId: string): Promise { + return this.viaAsync(false, () => this.deleteCheckpoint(cpId), () => this.backend!.deleteCheckpoint(cpId)); + } + + gcAsync(opts: { archiveAfterDays: number; archiveDir: string; vacuum?: "full" | "incremental" }): Promise<{ archived: number; contentRowsDeleted: number }> { + return this.viaAsync({ archived: 0, contentRowsDeleted: 0 }, () => this.gc(opts), () => this.enqueue(() => this.backend!.gc(opts))); + } + + restoreArchivedAsync(sessionId: string, opts: { archiveDir: string }): Promise<{ ok: boolean; reason?: string }> { + return this.viaAsync({ ok: false, reason: "async backend" }, () => this.restoreArchived(sessionId, opts), () => this.enqueue(() => this.backend!.restoreArchived(sessionId, opts))); + } + + rawMessagesAsync(sessionId: string): Promise<{ seq: number; content: string }[]> { + return this.viaAsync([], () => this.rawMessages(sessionId), async () => (await this.backend!.getMessages(sessionId)).map((m) => ({ seq: m.seq, content: m.content }))); + } + + tailSeqsAsync(sessionId: string): Promise { + return this.viaAsync(null, () => this.tailSeqs(sessionId), () => this.backend!.tailSeqs(sessionId)); + } + + tailViewRowsAsync(sessionId: string): Promise { + return this.viaAsync([], () => this.tailViewRows(sessionId), () => this.backend!.getMessages(sessionId)); + } + + getCcrAsync(sessionId: string): Promise { + return this.viaAsync(null, () => this.getCcr(sessionId), () => this.backend!.getCcr(sessionId)); + } + + setSessionExtAsync(sessionId: string, key: string, value: unknown): Promise { + return this.viaAsync(undefined, () => this.setSessionExt(sessionId, key, value), () => this.enqueue(() => this.backend!.setSessionExt(sessionId, key, value)).then(() => undefined)); + } + + getSessionExtAsync(sessionId: string, key: string): Promise { + return this.viaAsync(null, () => this.getSessionExt(sessionId, key), () => this.backend!.getSessionExt(sessionId, key)); + } + + listSessionExtAsync(sessionId: string): Promise<{ key: string; value: unknown; updatedAt: number }[]> { + return this.viaAsync([], () => this.listSessionExt(sessionId), () => this.backend!.listSessionExt(sessionId)); + } + + deleteSessionExtAsync(sessionId: string, key?: string): Promise { + return this.viaAsync(0, () => this.deleteSessionExt(sessionId, key), () => this.backend!.deleteSessionExt(sessionId, key)); + } + + setKvAsync(namespace: string, key: string, value: unknown): Promise { + return this.viaAsync(undefined, () => this.setKv(namespace, key, value), () => this.enqueue(() => this.backend!.setKv(namespace, key, value)).then(() => undefined)); + } + + getKvAsync(namespace: string, key: string): Promise { + return this.viaAsync(null, () => this.getKv(namespace, key), () => this.backend!.getKv(namespace, key)); + } + + listKvAsync(namespace?: string, keyPrefix?: string): Promise<{ namespace: string; key: string; value: unknown; updatedAt: number }[]> { + return this.viaAsync([], () => this.listKv(namespace, keyPrefix), () => this.backend!.listKv(namespace, keyPrefix)); + } + + deleteKvAsync(namespace: string, key?: string): Promise { + return this.viaAsync(0, () => this.deleteKv(namespace, key), () => this.backend!.deleteKv(namespace, key)); + } + + /** Drains the async record queue, then closes the backend pool. */ + async closeAsync(): Promise { + if (this.sync !== null) { + this.close(); + return; + } + try { + await this.queueTail; + } catch { + // queue work reports its own failures + } + await this.backend?.close(); } private assertLive(): void { @@ -584,8 +964,9 @@ export class UnifiedShadow { } close(): void { + if (this.sync === null) return; try { - this.store.close(); + this.sync.close(); } catch (err) { // shutdown path — never throw, but a swallowed close failure leaves // the sqlite handle locked (windows EBUSY on temp-dir teardown); log diff --git a/tests/export-jsonl.test.ts b/tests/export-jsonl.test.ts index 2a124cf3b9..ef01ca5312 100644 --- a/tests/export-jsonl.test.ts +++ b/tests/export-jsonl.test.ts @@ -173,7 +173,6 @@ test("renderSessionJsonlFromSession dereferences via an injected content store", test("--jsonl enforces the backend selection at open time: the mysql stub fails with its documented error (#3932 review)", async () => { await withMode("full", async () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-jsonl-backend-")); - process.env.BILI_STORAGE_BACKEND = "mysql"; try { const session = await loadCorpusCase(dir, "folded", "corpus-folded", "anthropic"); const store = new SessionStore({ dir: path.join(dir, "folded"), debounceMs: 1, enabled: true }); @@ -182,6 +181,13 @@ test("--jsonl enforces the backend selection at open time: the mysql stub fails store.cancelAll(); assert.ok(fs.existsSync(path.join(dir, "folded", "index.db")), "full-mode corpus ingest must leave index.db behind"); + // #3931 step 3: the backend selection binds at STORE construction — + // a store built under "mysql" without a DSN has no sqlite shadow at + // all (boot warns; the envelope is the only sink), so this leg + // ingests under the sqlite default and selects mysql only for the + // export call, which is the surface under test here. + process.env.BILI_STORAGE_BACKEND = "mysql"; + await assert.rejects( () => exportSession(session.id, { dir: path.join(dir, "folded"), jsonl: true }), (err: unknown) => { diff --git a/tests/storage-backend-mysql-e2e.test.ts b/tests/storage-backend-mysql-e2e.test.ts new file mode 100644 index 0000000000..e44dc0e0bd --- /dev/null +++ b/tests/storage-backend-mysql-e2e.test.ts @@ -0,0 +1,137 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { SessionStore } from "../src/persist.ts"; +import type { LegacySessionLike } from "../src/storage/ingest.ts"; + +/** + * #3931 step 3: the mysql backend on the REAL SessionStore hot path. + * + * The contract suite (storage-backend-contract.test.ts) proves the backend + * satisfies the operation contract against a real MySQL. This suite proves + * the WIRING: a SessionStore booted with storage.backend=mysql + full mode + * - attaches its sidecar asynchronously (ready()) and keeps writing JSON + * envelopes (writerOff is sqlite-only), + * - ingests saves onto the serial backend queue (rows appear), + * - serves cold loads from the envelope (the sync read path), + * - exposes replicas/ext through the async twins, + * - restarts warm on the same DSN. + * + * Skipped unless BILI_MYSQL_TEST_DSN is set (same discipline as the + * contract suite; the CI mysql-contract job runs both). + */ + +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_CONFIG_FILE = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), "bili-mysql-e2e-hermetic-")), + "no-config.json", +); + +const DSN = process.env.BILI_MYSQL_TEST_DSN ?? ""; +const CORPUS_ROOT = path.resolve(import.meta.dirname, "golden/storage-corpus"); + +async function loadCorpusCase(dir: string, name: string, id: string, protocol: string) { + fs.cpSync(path.join(CORPUS_ROOT, name), path.join(dir, name), { recursive: true }); + const loader = new SessionStore({ dir: path.join(dir, name), debounceMs: 1, enabled: true }); + const loaded = loader.loadSync(id, { protocol, upstreamOrigin: "http://127.0.0.1:8199" }); + loader.cancelAll(); + // With the mysql env pinned this loader also opens a pool — close it or + // the process hangs on the open connection. + await loader.closeQuietlyAsync(); + assert.ok(loaded, `${name}: corpus loadSync returned null`); + return loaded; +} + +function mysqlEnv(dsn: string): NodeJS.ProcessEnv { + return { + ...process.env, + BILI_STORAGE_UNIFIED: "full", + BILI_STORAGE_BACKEND: "mysql", + BILI_MYSQL_DSN: dsn, + }; +} + +/** Poll the async twin until the session row lands (mysql network + queue + * latency). tail_json may legitimately carry null tailSeqs (a corpus + * session with empty lastMessages), so the session ROW is the wait signal. */ +async function untilRows(store: SessionStore, id: string, ms = 5000): Promise { + const deadline = Date.now() + ms; + while (Date.now() < deadline) { + const shadow = (store as unknown as { shadow?: { viewForAsync(sid: string): Promise<{ id?: string } | null> } }).shadow; + const view = shadow ? await shadow.viewForAsync(id) : null; + if (view) return; + await new Promise((r) => setTimeout(r, 50)); + } + assert.fail(`mysql rows for ${id} did not land within ${ms}ms`); +} + +test("mysql hot path: saves land on the backend, envelopes keep being written, restart stays warm", { skip: DSN ? false : "BILI_MYSQL_TEST_DSN not set" }, async () => { + const prevEnv = { ...process.env }; + Object.assign(process.env, mysqlEnv(DSN)); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-mysql-e2e-")); + const sid = `mysql-e2e-${Date.now().toString(36)}`; + try { + // A corpus session folded once (deadrefs case has blocks + contents). + const corpus = await loadCorpusCase(dir, "deadrefs", "corpus-deadrefs", "anthropic"); + corpus.id = sid; + corpus.metadata.label = sid; + + const store = new SessionStore({ dir, debounceMs: 0, enabled: true }); + await store.ready(); + const shadow = (store as unknown as { shadow?: { isAsyncBackend: boolean; viewForAsync(id: string): Promise } }).shadow; + assert.ok(shadow, "mysql sidecar must attach after ready()"); + assert.equal(shadow.isAsyncBackend, true, "the shadow must ride the async backend"); + + // Hot-path write: the save runs the async ingest queue. + await store.writeNow(corpus); + await untilRows(store, sid); + + // Envelope discipline: writerOff is sqlite-only — the JSON file is + // the sync read path on mysql, so it MUST exist. + const all: string[] = []; + const walk = (d: string) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const full = path.join(d, e.name); if (e.isDirectory()) walk(full); else all.push(e.name); } }; + walk(dir); + const envelope = all.find((f) => f.endsWith(".json") && !f.endsWith(".content-store.json")); + assert.ok(envelope, `a JSON envelope must be written under mysql (${all.join(",")})`); + + // The backend holds the session view (async twin): same id, fold + // blocks intact (the corpus case carries blocks + contents). + const view = await shadow.viewForAsync(sid); + assert.ok(view, "viewForAsync must serve the ingested session"); + const viewAny = view as unknown as { id?: string; state?: { blocks?: unknown[] } }; + assert.equal(viewAny.id, sid, "backend view id"); + assert.ok(Array.isArray(viewAny.state?.blocks) && (viewAny.state?.blocks?.length ?? 0) > 0, "backend view keeps the fold blocks"); + + // Replicas through the async twins. + const cp = await store.createReplica(sid, { label: "mysql-e2e" }); + assert.match(cp.cp_id, /^cp-/); + const listed = await store.listReplicas(sid); + assert.ok(listed.some((r) => r.cp_id === cp.cp_id)); + const restored = await store.restoreReplica(cp.cp_id); + assert.equal(restored.restoredFrom, sid); + assert.equal(await store.deleteReplica(cp.cp_id), true); + + // ext substrate (incremental pointers) round-trips async. + const shExt = (store as unknown as { shadow: { setSessionExtAsync(id: string, k: string, v: unknown): Promise; getSessionExtAsync(id: string, k: string): Promise } }).shadow; + await shExt.setSessionExtAsync(sid, "inc:test-key", [1, 2, 3]); + assert.deepEqual(await shExt.getSessionExtAsync(sid, "inc:test-key"), [1, 2, 3]); + + await store.closeQuietlyAsync(); + + // Restart on the same DSN: the sidecar reattaches, the session is + // still fully served (envelope + backend rows), and a second save + // lands again (watermark path on the async backend). + const reopen = new SessionStore({ dir, debounceMs: 0, enabled: true }); + await reopen.ready(); + const warm = reopen.loadSync(sid); + assert.ok(warm, "cold load after restart must serve (envelope path)"); + await reopen.writeNow(warm); + await untilRows(reopen, sid); + await reopen.closeQuietlyAsync(); + } finally { + Object.assign(process.env, prevEnv); + fs.rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/tests/storage-json-writer-off.test.ts b/tests/storage-json-writer-off.test.ts index 44c5b3ff75..eca1f87781 100644 --- a/tests/storage-json-writer-off.test.ts +++ b/tests/storage-json-writer-off.test.ts @@ -115,7 +115,7 @@ test("jsonWriter=off (full): no envelope written, index-only session survives re // Rollback story: replicas keep materializing JSON envelopes // (restore writes through the kernel store directly) — a // portable export even while the writer is off. - const cp = reopen.createReplica(session.id, { label: "json-writer-off-rollback" }); + const cp = await reopen.createReplica(session.id, { label: "json-writer-off-rollback" }); const restored = await reopen.restoreReplica(cp.cp_id, { sessionId: session.id + "-rb" }); assert.equal(restored.restoredFrom, session.id); const rb = envelopeFiles(dir).map((f) => path.basename(f)); diff --git a/tests/storage-replicas.test.ts b/tests/storage-replicas.test.ts index bcb56d37d8..409fcf2568 100644 --- a/tests/storage-replicas.test.ts +++ b/tests/storage-replicas.test.ts @@ -85,7 +85,7 @@ test("replicas: create → list → detail round-trip (shadow mode, real persist store.cancelAll(); assert.ok(store.replicasAvailable(), "shadow mode must expose replicas"); - const row = store.createReplica(session.id, { label: "before-experiment" }); + const row = await store.createReplica(session.id, { label: "before-experiment" }); assert.match(row.cp_id, /^cp-/); assert.equal(row.session_id, session.id); assert.ok(row.seq_at > 0, `seq_at must be the absolute tip, got ${row.seq_at}`); @@ -94,10 +94,10 @@ test("replicas: create → list → detail round-trip (shadow mode, real persist assert.equal(row.label, "before-experiment"); assert.ok(row.block_set_hash.length >= 16, "block set hash is a sha256 prefix at least"); - const listed = store.listReplicas(session.id); + const listed = await store.listReplicas(session.id); assert.equal(listed.length, 1); assert.equal(listed[0]!.cp_id, row.cp_id); - assert.ok(store.replicaByteLen(row.cp_id) > 0, "frozen record blob must have size"); + assert.ok(await store.replicaByteLen(row.cp_id) > 0, "frozen record blob must have size"); store.closeQuietly(); } finally { try { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } catch { /* Windows: an ephemeral stray handle must not fail CI */ } @@ -114,7 +114,7 @@ test("replicas: restore materializes a NEW session with identical content, even await store.writeNow(session); store.cancelAll(); - const pinned = store.createReplica(session.id, { pinned: true, label: "safe copy" }); + const pinned = await store.createReplica(session.id, { pinned: true, label: "safe copy" }); const before = (() => { const unified = UnifiedStore.open(path.join(dir, "index.db")); try { @@ -200,8 +200,8 @@ test("replicas: pinned survives deleteSession GC, unpinned is swept with it", as const store = new SessionStore({ dir, debounceMs: 1, enabled: true }); await store.writeNow(session); store.cancelAll(); - const pinned = store.createReplica(session.id, { pinned: true, label: "keep" }); - const loose = store.createReplica(session.id, {}); + const pinned = await store.createReplica(session.id, { pinned: true, label: "keep" }); + const loose = await store.createReplica(session.id, {}); const unified = UnifiedStore.open(path.join(dir, "index.db")); try { @@ -232,8 +232,10 @@ test("replicas: legacy mode fails fast (501-class), explicit delete overrides pi await legacyStore.writeNow(session); legacyStore.cancelAll(); assert.equal(legacyStore.replicasAvailable(), false); - assert.throws(() => legacyStore.createReplica(session.id), /legacy-only/); - assert.throws(() => legacyStore.listReplicas(), /legacy-only/); + // #3931 step 3: the replica surface is async (network backends) — + // refusals surface as rejections, not sync throws. + await assert.rejects(legacyStore.createReplica(session.id), /legacy-only/); + await assert.rejects(legacyStore.listReplicas(), /legacy-only/); // Now flip the same directory into shadow mode: organic import on // first write brings the session into the index, and the full API @@ -245,10 +247,10 @@ test("replicas: legacy mode fails fast (501-class), explicit delete overrides pi assert.ok(s2); await store.writeNow(s2); store.cancelAll(); - const pinned = store.createReplica(session.id, { pinned: true }); - assert.equal(store.deleteReplica(pinned.cp_id), true); - assert.equal(store.listReplicas().length, 0); - assert.equal(store.deleteReplica(pinned.cp_id), false, "second delete: gone is gone"); + const pinned = await store.createReplica(session.id, { pinned: true }); + assert.equal(await store.deleteReplica(pinned.cp_id), true); + assert.equal((await store.listReplicas()).length, 0); + assert.equal(await store.deleteReplica(pinned.cp_id), false, "second delete: gone is gone"); }); legacyStore.closeQuietly(); } finally { diff --git a/tests/storage-review-fixes.test.ts b/tests/storage-review-fixes.test.ts index 30df6de902..ea5feb6d54 100644 --- a/tests/storage-review-fixes.test.ts +++ b/tests/storage-review-fixes.test.ts @@ -190,7 +190,7 @@ test("reviews 4+5: restore refuses live-id overwrite and materializes the CCR si await store.writeNow(session); store.cancelAll(); - const replica = store.createReplica(session.id, { pinned: true, label: "safe copy" }); + const replica = await store.createReplica(session.id, { pinned: true, label: "safe copy" }); // Review 6 (HTTP half): caller-chosen id that already exists must // throw (admin maps to 409), never silently clobber a live session. diff --git a/website/config-reference/env-only.yaml b/website/config-reference/env-only.yaml index ebbe693037..4356fcdb87 100644 --- a/website/config-reference/env-only.yaml +++ b/website/config-reference/env-only.yaml @@ -16,16 +16,16 @@ entries: kind: env-only audience: user group: envonly - desc_en: "Env tier for storage.backend (#3931): sqlite (default) or mysql (implemented, #3388 step 2 - openBackend speaks the full contract against MySQL 8.0; export works today, the proxy hot path swaps in step 3). Wins over the file tier." - desc_zh: "storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已实现,#3388 第二步 —— openBackend 对 MySQL 8.0 说完整契约;export 今天可用,代理热路径第三步接入)。优先于文件层。" + desc_en: "Env tier for storage.backend (#3931): sqlite (default) or mysql (fully wired, #3388 steps 2+3 - saves ingest through the async shadow queue, sync reads ride the JSON envelope, replicas/ext/GC via the async twins; export too). Wins over the file tier." + desc_zh: "storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已接线,#3388 第二+三步 —— 保存经异步影子队列入 mysql,同步读走 JSON 信封,replicas/ext/GC 走异步孪生;export 同样支持)。优先于文件层。" - key: BILI_MYSQL_DSN type: string default: "unset" kind: env-only audience: user group: envonly - desc_en: "MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend=\"mysql\" (#3388 step 2, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs." - desc_zh: "storage.backend=\"mysql\" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388 第二步,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。" + desc_en: "MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend=\"mysql\" (#3388, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs. The pool opens lazily; closeQuietlyAsync drains the ingest queue and closes it." + desc_zh: "storage.backend=\"mysql\" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。连接池惰性打开;closeQuietlyAsync 排空入队事务并关闭。" - key: BILI_STORAGE_ARCHIVE_AFTER_DAYS type: integer default: "365" diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index e21b7f75ed..ab3b6fd819 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -257,15 +257,15 @@ entries: default: "unset (sqlite)" env: BILI_STORAGE_BACKEND group: process - desc_en: "Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is IMPLEMENTED (step 2): openBackend(\"mysql\", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path (shadow ingest) stays on sqlite until the step-3 async swap — the boot notice names that boundary; export speaks mysql today (storage.backend=\"mysql\" + a DSN exports straight from MySQL). Unknown names fall back to sqlite with a notice. Resolved once per lane boot." - desc_zh: "存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已实现(第二步):openBackend(\"mysql\", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径(shadow ingest)在第三步异步化改造前仍走 sqlite —— 启动提示会写明这一边界;export 今天就能说 mysql(storage.backend=\"mysql\" + DSN 直接从 MySQL 导出)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。" + desc_en: "Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is FULLY WIRED (steps 2+3): openBackend(\"mysql\", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path is wired through the async shadow facade: saves enqueue onto a serial backend queue (watermark deltas, orphan/reap journal, digest sampling — sqlite keeps its zero-await sync fast path unchanged), SYNC reads ride the JSON envelope (the sidecar attaches asynchronously via ready(); viewFor/importRecord sync facades miss by design), replicas/ext/GC/export speak the async twins. writerOff stays sqlite-only (the envelope is the mysql sync read path). Unknown names fall back to sqlite with a notice. Resolved once per lane boot." + desc_zh: "存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已完整接线(第二+三步):openBackend(\"mysql\", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径经异步影子 facade 接线:保存进串行后端队列(水位 delta、孤儿/收割台账、digest 采样 —— sqlite 保持零 await 同步快车道不变),同步读走 JSON 信封(sidecar 经 ready() 异步挂载;viewFor/importRecord 同步面按设计 miss),replicas/ext/GC/export 走异步孪生。writerOff 仅 sqlite(信封就是 mysql 的同步读路径)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。" - key: storage.mysql.dsn type: string default: "unset" env: BILI_MYSQL_DSN group: process - desc_en: "MySQL DSN for storage.backend=\"mysql\" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (export today; the proxy hot path swaps in step 3). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default." - desc_zh: "storage.backend=\"mysql\" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(今天为 export;代理热路径第三步接入)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。" + desc_en: "MySQL DSN for storage.backend=\"mysql\" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (hot path included — the async shadow queue). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default." + desc_zh: "storage.backend=\"mysql\" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(含热路径 —— 异步影子队列)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。" - key: storage.sqliteAutoDownload type: boolean default: "true"