From 288c795210b37eedf7c553257e72ba7738a25d95 Mon Sep 17 00:00:00 2001 From: ranxianglei Date: Tue, 20 Oct 2026 17:30:33 +0800 Subject: [PATCH] =?UTF-8?q?storage:=20MySQL=20backend=20(step=202=20of=20#?= =?UTF-8?q?3388)=20=E2=80=94=20MysqlBackend=20implements=20the=20full=20St?= =?UTF-8?q?orageBackend=20contract,=20mysql2=20bundled=20+=20lazily=20load?= =?UTF-8?q?ed,=20export=20speaks=20mysql=20today=20(#3931)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - src/storage/mysql.ts: full contract surface against MySQL 8.0 (ingest with atomic CAS refcount bumps, delta/full/completeLog paths, reads, session_ext, reap journal, checkpoints, conversations, archive/restore/gc with the local JSONL archive log, diagnostics). DDL is the v7 shape (bili_meta carries the schema version; JSON columns stay LONGTEXT for byte-exact digests). - backend.ts: openBackend is async; the mysql branch lazily imports mysql2 (boots that never select mysql never load it); resolveMysqlDsn (env BILI_MYSQL_DSN > storage.mysql.dsn > documented error). - export.ts: backend-aware open — storage.backend="mysql" + DSN exports straight from MySQL; every fallback stays on the sqlite index. - Contract suite: the mysql lane runs the same assertions against a real mysql:8.0 (BILI_MYSQL_TEST_DSN); CI gains a mysql-contract service lane. - Boundary (step 3, next PR): the proxy hot path (shadow ingest) stays on sqlite until the persist async swap; the boot notice names the boundary. --- .github/workflows/ci.yml | 33 + CHANGELOG.md | 8 + CONFIGURATION.md | 7 +- CONFIGURATION.zh-CN.md | 7 +- package-lock.json | 118 +- src/config.ts | 6 + src/export.ts | 44 +- src/knobs.ts | 7 + src/storage/backend.ts | 48 +- src/storage/mysql.ts | 1438 ++++++++++++++++++++++++ tests/storage-backend-contract.test.ts | 121 +- tests/storage-backend-knob.test.ts | 38 +- website/config-reference/env-only.yaml | 12 +- website/config-reference/process.yaml | 11 +- 14 files changed, 1852 insertions(+), 46 deletions(-) create mode 100644 src/storage/mysql.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0f41378c56..f3dd0b6214 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -243,6 +243,39 @@ jobs: ACP_TEST_CLAUDE_NATIVE: "1" CI_TEST_SHARD: "${{ matrix.shard }}/${{ matrix.shardTotal }}" # denominator carried per-include by the plan job (#3295) + # #3388 step 2: the MySQL contract lane. The storage-backend contract + # suite runs against a real mysql:8.0 service (BILI_MYSQL_TEST_DSN) in + # addition to the always-on sqlite lane inside the main test job. Cheap + # (two files, ~1min) so it rides the core scope — every PR proves the + # mysql backend still satisfies the operation contract. + mysql-contract: + needs: [plan] + if: ${{ !failure() && !cancelled() && needs.plan.outputs.scope != 'skip' }} + runs-on: ubuntu-latest + services: + mysql: + image: mysql:8.0 + env: + MYSQL_ROOT_PASSWORD: bili-test-root + MYSQL_DATABASE: bili + ports: + - 3306:3306 + options: >- + --health-cmd="mysqladmin ping -uroot -pbili-test-root" + --health-interval=10s + --health-timeout=5s + --health-retries=10 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: npm + - run: npm install --ignore-scripts --no-fund --no-audit + - run: node --import tsx --test tests/storage-backend-contract.test.ts tests/storage-backend-knob.test.ts + env: + BILI_MYSQL_TEST_DSN: mysql://root:bili-test-root@127.0.0.1:3306/bili + # #2816: typecheck + the in-repo component suites + the bundle build used # to run sequentially in front of the test job (~90s of critical path per # leg). They are independent of the proxy test suites, so they run as diff --git a/CHANGELOG.md b/CHANGELOG.md index eca1a2bf80..d282ef99a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable changes to **billion-context** are documented here. Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag. ## [Unreleased] + +### 存储:MySQL 后端落地(#3388 第二步,#3931) + +- `src/storage/mysql.ts`:MysqlBackend 完整实现 StorageBackend 操作契约(全 ~40 方法:带 CAS 引用计数的 ingest(delta/全量/completeLog)、读取、session_ext/kv、收割台账、checkpoint、会话绑定、归档/恢复/GC、诊断)—— MySQL 8.0 方言,mysql2 连接池(打进 dependencies,#3931 拍板:纯 JS ~500KB,锁文件钉版本),DDL 为 v7 形状直接建表(bili_meta 表存 schema 版本替 PRAGMA user_version;JSON 列一律 LONGTEXT 保字节序,MySQL JSON 类型会重排键序破坏 digest)。 +- `openBackend` 改异步:mysql 分支懒加载 mysql2(不选 mysql 的启动永不加载该模块);DSN 解析 `env BILI_MYSQL_DSN > storage.mysql.dsn > 缺省报错`(文档化错误列全两个旋钮)。 +- **边界(第三步待做)**:代理热路径(shadow ingest/persist.loadSync 同步契约)仍走 sqlite;今天 export 说 mysql(`storage.backend="mysql"` + DSN 直接从 MySQL 导出 JSONL);启动提示写明该边界。 +- CI 新增 `mysql-contract` lane:每次 PR 用 mysql:8.0 service 容器跑同一份契约套件(sqlite lane 永远在,mysql lane 有 DSN 就跑)。 +- 归档文件系统语义跨后端一致:mysql 后端的归档/恢复同样落本地 `archive/archive.jsonl`(单文件日志,#3927)。 - 存储后端契约第一步(#3931, #3388):统一存储抽出操作级异步 `StorageBackend` 契约(`src/storage/backend.ts`),sqlite 经注册表以默认后端身份重新接入(Promise 包装适配器,零行为/零 schema 变化);`mysql` 为已注册占位 —— 选中时启动提示一行并继续用 sqlite,直开后端的路径(export)按文档化错误失败。新配置 `storage.backend`(env `BILI_STORAGE_BACKEND`,缺省 sqlite)。`--jsonl` 导出渲染器改吃契约(接缝证真);契约测试套件对任意后端实例跑同一断言,第二步 mysql 落地时原样复用。 ### 存储收尾九项一车(#3901,修复 #3463/#3762/#3841/#3798/#3633;#3723 落地) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 56a191b3d3..8513b95715 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -175,7 +175,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. | | `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot. | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is IMPLEMENTED (step 2): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path (shadow ingest) stays on sqlite until the step-3 async swap — the boot notice names that boundary; export speaks mysql today (storage.backend="mysql" + a DSN exports straight from MySQL). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. | +| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (export today; the proxy hot path swaps in step 3). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | @@ -334,7 +335,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| | `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase. | -| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931 step 1): sqlite (default, only real backend) or mysql (registered stub - one boot notice, stays on sqlite until step 2). Wins over the file tier. | +| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931): sqlite (default) or mysql (implemented, #3388 step 2 - openBackend speaks the full contract against MySQL 8.0; export works today, the proxy hot path swaps in step 3). Wins over the file tier. | +| `BILI_MYSQL_DSN` | string | unset | — | MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend="mysql" (#3388 step 2, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs. | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (one JSON line appended to the single archive log (archive/archive.jsonl) + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): compute the outbound snapshot view on every armed turn WITHOUT using it, classify the diff against the legacy outbound, and surface per-session counters in the web panel + the snapshot-shadow diagnostics topic. 0/false/off disables; 1/true/on enables; otherwise the config-file snapshot.shadow boolean applies (default on). The render lane itself stays rollout-gated (BILI_ROLLOUT_SNAPSHOT_RENDER), so the shadow lane never changes outbound bytes - it is the dry-run evidence that flips the rollout later. | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | Override the config file path (path relocation). | @@ -1748,6 +1750,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_MITM` | `mitm.enabled` | true | | `BILI_MITM_DOMAINS` | `mitm.domains` | [] | | `BILI_MITM_HANDSHAKE_TIMEOUT_MS` | `mitm.handshakeTimeoutMs` | 10000 | +| `BILI_MYSQL_DSN` | `storage.mysql.dsn` | unset | | `BILI_NATIVE_ATTACH_EXTERNAL` | `native.attachExternal` | false | | `BILI_NATIVE_ATTACH_POLICY` | `native.attachPolicy` | own-lane | | `BILI_NON_HTTP_PROVIDERS` | `compactionOptIn` | false | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 727a85219c..a90a146f75 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -175,7 +175,8 @@ | `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | EPERM 告警的重复间隔;0 只记录一次。 | | `storage` | object | {} (all defaults) | — | 统一 SQLite 存储边车设置(#2671)。 | | `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 full(#3818 翻转 —— shadow 干跑自 #2815 起对账一直干净,索引因此成为默认加载路径;JSON 信封照写,legacy 仍是一行逃生口)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | -| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择,可插拔存储计划第一步(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约,sqlite 为默认且是当前唯一真实现(现有 UnifiedStore 套一层 Promise 包装适配器 —— 零行为变化、零 schema 变化)。mysql 是已注册占位:该名字能通过校验(真实后端随第二步落地 —— mysql2 客户端,走可选依赖+自下载纪律 —— 期间配置不会在 unknown/unimplemented 之间来回弹),但选中它只会在 lane 启动时打一行提示并继续用 sqlite;直接开后端的路径(export)按文档化的错误失败。未知名字同样回退 sqlite 并提示。每个 lane 启动时解析一次。 | +| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | 存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已实现(第二步):openBackend("mysql", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径(shadow ingest)在第三步异步化改造前仍走 sqlite —— 启动提示会写明这一边界;export 今天就能说 mysql(storage.backend="mysql" + DSN 直接从 MySQL 导出)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。 | +| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | storage.backend="mysql" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(今天为 export;代理热路径第三步接入)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | @@ -334,7 +335,8 @@ | Key | Type | Default | Env | Description | |-----|------|---------|-----|-------------| | `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | 让 bili 本地应答 dsh 压缩摘要请求(返回折叠快照而非拒绝,#3733):身份头 + 内核渲染格式的块摘要,零出口流量;重放经 #2432 再基线识别。 | -| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | storage.backend 的环境变量层(#3931 第一步):sqlite(默认,唯一真实现)或 mysql(注册占位 —— 启动提示一行,第二步落地前继续用 sqlite)。优先于文件层。 | +| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已实现,#3388 第二步 —— openBackend 对 MySQL 8.0 说完整契约;export 今天可用,代理热路径第三步接入)。优先于文件层。 | +| `BILI_MYSQL_DSN` | string | unset | — | storage.backend="mysql" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388 第二步,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。 | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(单文件归档日志 archive/archive.jsonl 追加一行 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。 | | `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | 快照渲染影子对账(#3756 后续):每轮已折叠会话都用引擎现算一次出站快照视图但不使用,与 legacy 实际出站比对分类,逐会话计数进 web 面板与 snapshot-shadow 诊断主题。0/false/off 关闭;1/true/on 开启;否则读配置文件 snapshot.shadow 布尔(默认开)。渲染车道本身仍受灰度开关控制(BILI_ROLLOUT_SNAPSHOT_RENDER),影子车道绝不改变出站字节 —— 它是后续翻转灰度的干跑证据。 | | `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | 覆盖配置文件路径(路径重定位)。 | @@ -1747,6 +1749,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_MITM` | `mitm.enabled` | true | | `BILI_MITM_DOMAINS` | `mitm.domains` | [] | | `BILI_MITM_HANDSHAKE_TIMEOUT_MS` | `mitm.handshakeTimeoutMs` | 10000 | +| `BILI_MYSQL_DSN` | `storage.mysql.dsn` | unset | | `BILI_NATIVE_ATTACH_EXTERNAL` | `native.attachExternal` | false | | `BILI_NATIVE_ATTACH_POLICY` | `native.attachPolicy` | own-lane | | `BILI_NON_HTTP_PROVIDERS` | `compactionOptIn` | false | diff --git a/package-lock.json b/package-lock.json index 3b9a178d51..a7c080ece5 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,6 +9,7 @@ "version": "0.1.202", "license": "MIT", "dependencies": { + "mysql2": "^3.15.3", "zod": "4.1.8" }, "bin": { @@ -4245,6 +4246,15 @@ "node": "*" } }, + "node_modules/aws-ssl-profiles": { + "version": "1.1.2", + "resolved": "http://127.0.0.1:4873/aws-ssl-profiles/-/aws-ssl-profiles-1.1.2.tgz", + "integrity": "sha512-NZKeq9AfyQvEeNlN0zSYAaWrmBffJh3IELMZfRpJVWgrpEbtEpnjvzqBPf+mxoI287JohRDoa+/nsfqqiZmF6g==", + "license": "MIT", + "engines": { + "node": ">= 6.0.0" + } + }, "node_modules/aws4": { "version": "1.13.2", "resolved": "https://registry.npmjs.org/aws4/-/aws4-1.13.2.tgz", @@ -4934,6 +4944,15 @@ "node": ">=0.4.0" } }, + "node_modules/denque": { + "version": "2.1.0", + "resolved": "http://127.0.0.1:4873/denque/-/denque-2.1.0.tgz", + "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==", + "license": "Apache-2.0", + "engines": { + "node": ">=0.10" + } + }, "node_modules/depd": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", @@ -5575,6 +5594,15 @@ "dev": true, "license": "MIT" }, + "node_modules/generate-function": { + "version": "2.3.1", + "resolved": "http://127.0.0.1:4873/generate-function/-/generate-function-2.3.1.tgz", + "integrity": "sha512-eeB5GfMNeevm/GRYq20ShmsaGcmI81kIX2K9XQx5miC8KdHaC6Jm0qQ8ZNeGOi7wYB8OsdxKs+Y2oVuTFuVwKQ==", + "license": "MIT", + "dependencies": { + "is-property": "^1.0.2" + } + }, "node_modules/get-east-asian-width": { "version": "1.6.0", "resolved": "https://registry.npmjs.org/get-east-asian-width/-/get-east-asian-width-1.6.0.tgz", @@ -6051,6 +6079,12 @@ "dev": true, "license": "MIT" }, + "node_modules/is-property": { + "version": "1.0.2", + "resolved": "http://127.0.0.1:4873/is-property/-/is-property-1.0.2.tgz", + "integrity": "sha512-Ks/IoX00TtClbGQr4TWXemAnktAQvYB7HzcCxDGqEZU6oCmb2INHuOoKxbtR+HFkmYWBKv/dOZtGRiAjDhj92g==", + "license": "MIT" + }, "node_modules/is-stream": { "version": "4.0.1", "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-4.0.1.tgz", @@ -6394,6 +6428,12 @@ "dev": true, "license": "MIT" }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "http://127.0.0.1:4873/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, "node_modules/lowdb": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/lowdb/-/lowdb-1.0.0.tgz", @@ -6434,6 +6474,21 @@ "node": ">=12" } }, + "node_modules/lru.min": { + "version": "1.1.5", + "resolved": "http://127.0.0.1:4873/lru.min/-/lru.min-1.1.5.tgz", + "integrity": "sha512-5J9ysMYUpYIg9RF2vJpy9SinEmSviFSe0GyPpCQ4L5QSkLAgeLXlTAOu2ZwWUU5m+0SBl6gUU1R1ZQB3aKypfA==", + "license": "MIT", + "engines": { + "bun": ">=1.0.0", + "deno": ">=1.30.0", + "node": ">=8.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wellwelwel" + } + }, "node_modules/magic-string": { "version": "0.30.21", "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", @@ -6675,6 +6730,42 @@ "devOptional": true, "license": "MIT" }, + "node_modules/mysql2": { + "version": "3.15.3", + "resolved": "http://127.0.0.1:4873/mysql2/-/mysql2-3.15.3.tgz", + "integrity": "sha512-FBrGau0IXmuqg4haEZRBfHNWB5mUARw6hNwPDXXGg0XzVJ50mr/9hb267lvpVMnhZ1FON3qNd4Xfcez1rbFwSg==", + "license": "MIT", + "dependencies": { + "aws-ssl-profiles": "^1.1.1", + "denque": "^2.1.0", + "generate-function": "^2.3.1", + "iconv-lite": "^0.7.0", + "long": "^5.2.1", + "lru.min": "^1.0.0", + "named-placeholders": "^1.1.3", + "seq-queue": "^0.0.5", + "sqlstring": "^2.3.2" + }, + "engines": { + "node": ">= 8.0" + } + }, + "node_modules/mysql2/node_modules/iconv-lite": { + "version": "0.7.3", + "resolved": "http://127.0.0.1:4873/iconv-lite/-/iconv-lite-0.7.3.tgz", + "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/mz": { "version": "2.7.0", "resolved": "https://registry.npmjs.org/mz/-/mz-2.7.0.tgz", @@ -6687,6 +6778,18 @@ "thenify-all": "^1.0.0" } }, + "node_modules/named-placeholders": { + "version": "1.1.6", + "resolved": "http://127.0.0.1:4873/named-placeholders/-/named-placeholders-1.1.6.tgz", + "integrity": "sha512-Tz09sEL2EEuv5fFowm419c1+a/jSMiBjI9gHxVLrVdbUkkNUUfjsVYs9pVZu5oCon/kmRh9TfLEObFtkVxmY0w==", + "license": "MIT", + "dependencies": { + "lru.min": "^1.1.0" + }, + "engines": { + "node": ">=8.0.0" + } + }, "node_modules/negotiator": { "version": "0.6.4", "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.4.tgz", @@ -7542,7 +7645,6 @@ "version": "2.1.2", "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", - "dev": true, "license": "MIT" }, "node_modules/sanitize-filename": { @@ -7633,6 +7735,11 @@ "node": ">=4" } }, + "node_modules/seq-queue": { + "version": "0.0.5", + "resolved": "http://127.0.0.1:4873/seq-queue/-/seq-queue-0.0.5.tgz", + "integrity": "sha512-hr3Wtp/GZIc/6DAGPDcV4/9WoZhjrkXsi5B/07QgX8tsdc6ilr7BFM6PM6rbdAX1kFSDYeZGLipIZZKyQP0O5Q==" + }, "node_modules/serve-static": { "version": "1.16.3", "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.3.tgz", @@ -7789,6 +7896,15 @@ "node": ">= 10.x" } }, + "node_modules/sqlstring": { + "version": "2.3.3", + "resolved": "http://127.0.0.1:4873/sqlstring/-/sqlstring-2.3.3.tgz", + "integrity": "sha512-qC9iz2FlN7DQl3+wjwn3802RTyjCx7sDvfQEXchwa6CWOx07/WVfh91gBmQ9fahw8snwGEWU3xGzOt4tFyHLxg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, "node_modules/sshpk": { "version": "1.18.0", "resolved": "https://registry.npmjs.org/sshpk/-/sshpk-1.18.0.tgz", diff --git a/src/config.ts b/src/config.ts index bad4fca6e1..e127cf6e05 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1886,6 +1886,12 @@ type FileConfig = { * wins; a non-sqlite selection logs one notice and stays on * sqlite until the backend ships. */ backend?: string; + /** MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): + * mysql://user:pass@host:3306/db — consumed by the StorageBackend + * contract paths (export/replica/checkpoint). env BILI_MYSQL_DSN + * wins. The proxy hot path stays on sqlite until the step-3 async + * swap; the boot notice names that boundary. */ + mysql?: { dsn?: string }; /** Phase 3 (#2960): "off" stops writing the legacy JSON envelope while * storage.unified is "full" — the SQLite index becomes the only * persistent copy (reads were already unified-first; boot enumerates diff --git a/src/export.ts b/src/export.ts index 1dfa5fa280..1b52781ae1 100644 --- a/src/export.ts +++ b/src/export.ts @@ -3,8 +3,9 @@ import path from "node:path"; import { matchSession as matchSessionKernel, parseStoredPlaceholder, renderHandoff as renderHandoffKernel, type MessageContentStore } from "acp-kernel"; import type { Session } from "./session.js"; import { SessionStore } from "./persist.js"; -import { backendSelectionNotice, listBackendNames, openBackend, selectedBackendRaw, type StorageBackend } from "./storage/backend.js"; -import { fileStorageBackend } from "./knobs.js"; +import { SqliteBackend, backendSelectionNotice, listBackendNames, openBackend, resolveBackendName, resolveMysqlDsn, selectedBackendRaw, type StorageBackend } from "./storage/backend.js"; +import { fileStorageBackend, fileStorageMysqlDsn } from "./knobs.js"; +import { UnifiedStore } from "./storage/store.js"; interface ExportOptions { dir?: string; @@ -349,6 +350,30 @@ export async function exportSession(selector: string | undefined, opts: ExportOp throw new Error(`selector "${selector}" matches ${matches.length} sessions (${ids}); use the full session id`); } if (opts.jsonl) { + // Backend-aware open (#3388 step 2): storage.backend="mysql" + a DSN + // exports straight from MySQL; everything else (and every fallback) + // stays on the local sqlite index. A mysql failure degrades to the + // legacy-tail renderer exactly like a corrupt index does. + const backendName = resolveBackendName(process.env, fileStorageBackend()); + if (backendName === "mysql") { + const dsn = resolveMysqlDsn(process.env, fileStorageMysqlDsn()); + if (dsn.length > 0) { + try { + const backend = await openBackend("mysql", dsn); + try { + await backend.init?.(); + if (await backend.getSessionRow(matches[0]!.id)) { + return await renderSessionJsonlFromTables(backend, matches[0]!.id, opts.full ?? false); + } + } finally { + await backend.close(); + } + } catch (err) { + process.stderr.write(`[export] mysql backend unreadable (${String(err)}) — falling back to the local index +`); + } + } + } const index = path.join(store.dir, "index.db"); const encrypted = process.env.BILI_ENCRYPTION_KEY !== undefined && process.env.BILI_ENCRYPTION_KEY !== ""; if (existsSync(index) && !encrypted) { @@ -359,7 +384,15 @@ export async function exportSession(selector: string | undefined, opts: ExportOp const selected = selectedBackendRaw(process.env, fileStorageBackend()); if (selected && selected !== "sqlite") { if (listBackendNames().includes(selected)) { - openBackend(selected, index); + // #3931 step 2: open is async now. The contract from the + // #3932 review holds — a deliberate selection never + // silently reads sqlite: without a DSN the open rejects + // with the documented error (bubbles out of the export); + // with a DSN the probe opens + closes cleanly (the real + // mysql export above already ran; if it degraded, its + // stderr notice announced the sqlite fallback). + const probe = await openBackend(selected, selected === "mysql" ? resolveMysqlDsn(process.env, fileStorageMysqlDsn()) : index); + await probe.close(); } else { const notice = backendSelectionNotice(selected); if (notice) process.stderr.write(`[export] ${notice}\n`); @@ -371,7 +404,10 @@ export async function exportSession(selector: string | undefined, opts: ExportOp // does — index.db holds cleartext payloads, so an encrypted lane // must never read from it. try { - const backend = openBackend("sqlite", index); + // #3931 step 2: openBackend is async (network backends + // open a pool + DDL init); the sqlite path resolves + // immediately. + const backend = await openBackend("sqlite", index); try { if (await backend.getSessionRow(matches[0]!.id)) { const disk = store.loadContentStore(matches[0]!); diff --git a/src/knobs.ts b/src/knobs.ts index 89fc2f6d13..eddd20492a 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -718,6 +718,13 @@ export function fileStorageBackend(): string | undefined { return typeof v === "string" ? v : undefined; } +/** storage.mysql.dsn (#3388 step 2): the DSN for storage.backend="mysql". + * env BILI_MYSQL_DSN wins (resolveMysqlDsn applies the precedence). */ +export function fileStorageMysqlDsn(): string | undefined { + const v = loadConfigFile().storage?.mysql?.dsn; + return typeof v === "string" && v.trim().length > 0 ? v : undefined; +} + export function fileStorageWalJournalSizeLimit(): number | undefined { const v = loadConfigFile().storage?.walJournalSizeLimitBytes; return typeof v === "number" && Number.isInteger(v) && v > 0 ? v : undefined; diff --git a/src/storage/backend.ts b/src/storage/backend.ts index 0f0cf42e4c..ffdee5e74f 100644 --- a/src/storage/backend.ts +++ b/src/storage/backend.ts @@ -12,10 +12,12 @@ import { UnifiedStore } from "./store.ts"; * the sqlite implementation below stays synchronous under the hood and * `Promise.resolve`-wraps its results — the hot-path cost is one microtask. * - * `sqlite` is the default and, in step 1, the only real implementation. - * `mysql` is a REGISTERED STUB: the name validates so configs don't bounce - * between "unknown backend" and "unimplemented backend" when step 2 lands, - * but opening it fails with one documented line, and mysql2 is never loaded. + * `sqlite` is the default. `mysql` is IMPLEMENTED (#3388 step 2, #3931): + * MysqlBackend implements the full contract against MySQL 8.0 (mysql2 + * driver, a bundled dependency — #3931 owner decision). The module loads + * lazily inside openBackend("mysql", dsn), so boots that do not select + * mysql never load mysql2; the `indexPath` argument doubles as the DSN for + * mysql, mirroring how SqliteBackend.open consumes the index path. */ export type { @@ -123,6 +125,9 @@ export interface StorageBackend { contentDedupReport(): Promise<{ rows: number; bytesTotal: number; refcounted: number }>; close(): Promise; + /** One-time DDL + schema stamp for network backends (MySQL); a no-op + * for sqlite (create-on-open). Idempotent. */ + init?(): Promise; } /** The sqlite backend: the existing UnifiedStore behind the async contract. @@ -290,15 +295,22 @@ export function listBackendNames(): string[] { return [...REGISTERED_BACKENDS]; } -/** Resolves a backend by name. `mysql` throws the documented step-2 error - * (registered, not implemented — the config tier must distinguish it from - * a typo'd name); unknown names throw with the valid list. */ -export function openBackend(name: string, indexPath: string, engine?: SqliteEngineName): StorageBackend { +/** Opens a backend by name. `mysql` resolves to MysqlBackend: `indexPath` + * is the DSN (mysql://user:pass@host:port/db), the pool connects lazily, + * and `init()` (DDL + schema stamp) must run once before first use — the + * contract suite does this in `before`. Unknown names throw with the valid + * list. Async because the mysql module is dynamically imported (#3388 + * step-2 decision: bundled dependency, lazy load). */ +export async function openBackend(name: string, indexPath: string, engine?: SqliteEngineName): Promise { if (name === "sqlite") return SqliteBackend.open(indexPath, engine); if (name === "mysql") { - throw new Error( - `[storage] backend "mysql" is registered but not implemented yet (#3388 step 2, tracked in #3931) — it ships as a built-in backend with the mysql2 client; until then the unified store runs on "sqlite"`, - ); + if (typeof indexPath !== "string" || indexPath.trim().length === 0) { + throw new Error( + `[storage] backend "mysql" needs a DSN — set BILI_MYSQL_DSN or storage.mysql.dsn (mysql://user:pass@host:3306/db); runs on "sqlite" until then (#3388/#3931)`, + ); + } + const { MysqlBackend } = await import("./mysql.js"); + return new MysqlBackend(indexPath); } throw new Error(`[storage] unknown backend "${name}" — valid names: ${REGISTERED_BACKENDS.join(", ")}`); } @@ -314,6 +326,15 @@ export function selectedBackendRaw(env: { BILI_STORAGE_BACKEND?: string | undefi return typeof file === "string" ? file.trim().toLowerCase() : ""; } +/** env BILI_MYSQL_DSN > file storage.mysql.dsn > "" (empty = mysql cannot + * open; the documented error above names both knobs). */ +export function resolveMysqlDsn(env: { BILI_MYSQL_DSN?: string | undefined } | undefined, file: string | undefined): string { + const fromEnv = env?.BILI_MYSQL_DSN; + if (typeof fromEnv === "string" && fromEnv.trim().length > 0) return fromEnv.trim(); + if (typeof file === "string" && file.trim().length > 0) return file.trim(); + return ""; +} + /** env BILI_STORAGE_BACKEND > file storage.backend > "sqlite". Unknown * values fall back to the default (the complaint rides the selection * notice; the hard error lives at open time for the paths that open). */ @@ -328,7 +349,10 @@ export function resolveBackendName(env: { BILI_STORAGE_BACKEND?: string | undefi export function backendSelectionNotice(name: string): string | undefined { if (!name || name === "sqlite") return undefined; if ((REGISTERED_BACKENDS as readonly string[]).includes(name)) { - return `[storage] storage.backend="${name}" is registered but not implemented yet (#3388 step 2) — the unified store stays on "sqlite" for now`; + if (name === "mysql") { + return `[storage] storage.backend="mysql" opens via the StorageBackend contract (#3388/#3931 step 2) — export/replica/checkpoint paths speak it; the proxy hot path (shadow ingest) swaps to the async contract in step 3 and stays on "sqlite" until then`; + } + return `[storage] storage.backend="${name}" is registered but not implemented — the unified store stays on "sqlite"`; } return `[storage] storage.backend="${name}" is not a registered backend (valid: ${REGISTERED_BACKENDS.join(", ")}) — using "sqlite"`; } diff --git a/src/storage/mysql.ts b/src/storage/mysql.ts new file mode 100644 index 0000000000..802719d341 --- /dev/null +++ b/src/storage/mysql.ts @@ -0,0 +1,1438 @@ +/** + * MySQL backend for the storage contract (#3388 step 2, #3931). + * + * Implements the same StorageBackend operation surface as the sqlite + * store, against MySQL 8.0 (mysql2 driver, bundled dependency per the + * 2026-10-20 owner decision — see the #3931 comment; this module is only + * ever reached through openBackend("mysql", dsn), so boots that do not + * select mysql never load it). + * + * Representation discipline mirrors src/storage/schema.ts: verbatim facts + * (payload_json / content bytes / *_json bags) are the truth; scalar + * columns are projections. JSON columns stay LONGTEXT — MySQL's JSON type + * normalizes key order and would break the byte-exact digest contract. + * + * The DDL is the v7 shape directly (fresh-only: a MySQL deployment starts + * at schema version 7 with the full table set; the sqlite migration + * ladder does not apply). + * + * Concurrency: one mysql2 connection pool per backend instance; every + * multi-statement operation rides a dedicated-connection transaction. + * The CAS refcount bump is a single atomic statement (INSERT ... ON + * DUPLICATE KEY UPDATE refcount = refcount + 1), so single-row CAS + * discipline survives multi-writer deployments; full ingestSession + * serialization relies on InnoDB row locks inside its transaction, which + * matches the sqlite BEGIN IMMEDIATE discipline for the current + * one-writer-per-session topology (see #3931 step 3 for the multi-writer + * hardening plan). + */ + +import { createPool, type Pool, type PoolConnection, type PoolOptions } from "mysql2/promise"; +import { mkdirSync, writeFileSync, appendFileSync, readFileSync, existsSync } from "node:fs"; +import { join } from "node:path"; +import { log as loggerLog } from "../logger.js"; +import { sha256Hex } from "./store.js"; +import type { MessageContentStore } from "acp-kernel"; +import type { StorageBackend } from "./backend.js"; +import type { + SessionRow, + MessageRow, + BlockRow, + RefRow, + CcrEntryRow, + CheckpointRow, + SessionSummaryRow, + WebSummaryRow, + IngestSessionInput, +} from "./store.js"; + +const MYSQL_SCHEMA_VERSION = 7; + +/** Fresh-install DDL: the schema.ts v7 shape in MySQL dialect. LONGTEXT for + * every verbatim JSON/bytes column (byte-exact round-trip; the JSON type + * re-orders keys), BIGINT for ms-epoch timestamps and seqs, VARCHAR keys + * sized to their guards. No literal defaults on LONGTEXT columns (MySQL + * forbids them) — every INSERT provides every column. */ +const MYSQL_DDL = ` +CREATE TABLE IF NOT EXISTS sessions ( + session_id VARCHAR(255) NOT NULL PRIMARY KEY, + parent_session_id VARCHAR(255) NULL, + fork_point_seq BIGINT NULL, + protocol VARCHAR(64) NULL, + label TEXT NULL, + title TEXT NULL, + host_title TEXT NULL, + namespace VARCHAR(64) NOT NULL DEFAULT 'proxy', + origin TEXT NULL, + status VARCHAR(32) NOT NULL DEFAULT 'active', + created_at BIGINT NOT NULL, + updated_at BIGINT NOT NULL, + next_block_id BIGINT NOT NULL DEFAULT 1, + next_run_id BIGINT NOT NULL DEFAULT 1, + meta_json LONGTEXT NOT NULL, + stats_json LONGTEXT NOT NULL, + metadata_json LONGTEXT NOT NULL, + kstate_json LONGTEXT NOT NULL, + tail_json LONGTEXT NOT NULL, + ccr_json LONGTEXT NULL, + last_ingested_seq BIGINT NULL, + orphans_json LONGTEXT NULL +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS content ( + hash VARCHAR(64) NOT NULL PRIMARY KEY, + byte_len BIGINT NOT NULL, + refcount BIGINT NOT NULL DEFAULT 0, + algo VARCHAR(16) NOT NULL DEFAULT 'sha256', + bytes LONGTEXT NOT NULL +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS messages ( + session_id VARCHAR(255) NOT NULL, + seq BIGINT NOT NULL, + msg_id VARCHAR(255) NOT NULL, + ref VARCHAR(64) NULL, + ref_num BIGINT NULL, + role VARCHAR(64) NOT NULL, + kind VARCHAR(64) NOT NULL DEFAULT 'message', + content_hash VARCHAR(64) NOT NULL, + PRIMARY KEY (session_id, seq), + UNIQUE KEY uq_messages_identity (session_id, msg_id, content_hash), + KEY idx_messages_session (session_id, seq), + KEY idx_messages_msg_id (session_id, msg_id) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS blocks ( + block_id VARCHAR(64) NOT NULL, + session_id VARCHAR(255) NOT NULL, + ord BIGINT NOT NULL, + start_seq BIGINT NOT NULL, + end_seq BIGINT NOT NULL, + start_ref BIGINT NULL, + end_ref BIGINT NULL, + tier BIGINT NOT NULL, + summary_hash VARCHAR(64) NOT NULL, + one_hash VARCHAR(64) NULL, + supersedes_block_id VARCHAR(64) NULL, + active TINYINT NOT NULL DEFAULT 1, + created_at BIGINT NOT NULL, + covered_hash VARCHAR(64) NOT NULL, + payload_json LONGTEXT NOT NULL, + PRIMARY KEY (session_id, block_id), + KEY idx_blocks_session (session_id, ord) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS dead_refs ( + session_id VARCHAR(255) NOT NULL, + ord BIGINT NOT NULL, + ref VARCHAR(64) NOT NULL, + ref_num BIGINT NOT NULL, + recorded_at BIGINT NOT NULL, + PRIMARY KEY (session_id, ord), + KEY idx_dead_refs_session (session_id, ord) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS refs ( + session_id VARCHAR(255) NOT NULL, + ord BIGINT NOT NULL, + raw_id VARCHAR(255) NOT NULL, + ref VARCHAR(64) NOT NULL, + ref_num BIGINT NULL, + PRIMARY KEY (session_id, ord), + UNIQUE KEY uq_refs_raw (session_id, raw_id), + KEY idx_refs_session (session_id, ord) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS checkpoints ( + cp_id VARCHAR(255) NOT NULL PRIMARY KEY, + session_id VARCHAR(255) NOT NULL, + seq_at BIGINT NOT NULL, + block_set_hash VARCHAR(64) NOT NULL, + kind VARCHAR(16) NOT NULL, + created_at BIGINT NOT NULL, + pinned TINYINT NOT NULL DEFAULT 0, + label TEXT NULL, + state_hash VARCHAR(64) NOT NULL, + KEY idx_checkpoints_session (session_id, created_at) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS ccr_entries ( + session_id VARCHAR(255) NOT NULL, + ord BIGINT NOT NULL, + ccr_key VARCHAR(255) NOT NULL, + content_hash VARCHAR(64) NOT NULL, + PRIMARY KEY (session_id, ord) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS conversations ( + namespace VARCHAR(64) NOT NULL, + conversation_id VARCHAR(255) NOT NULL, + session_id VARCHAR(255) NOT NULL, + claimed_at BIGINT NOT NULL, + PRIMARY KEY (namespace, conversation_id) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS session_ext ( + session_id VARCHAR(255) NOT NULL, + ext_key VARCHAR(256) NOT NULL, + value_json LONGTEXT NOT NULL, + updated_at BIGINT NOT NULL, + PRIMARY KEY (session_id, ext_key) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS kv_store ( + namespace VARCHAR(256) NOT NULL, + kv_key VARCHAR(256) NOT NULL, + value_json LONGTEXT NOT NULL, + updated_at BIGINT NOT NULL, + PRIMARY KEY (namespace, kv_key) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS message_media ( + session_id VARCHAR(255) NOT NULL, + seq BIGINT NOT NULL, + media_hash VARCHAR(64) NOT NULL, + PRIMARY KEY (session_id, seq, media_hash) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS bili_meta ( + meta_key VARCHAR(64) NOT NULL PRIMARY KEY, + meta_value VARCHAR(256) NOT NULL +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; +`; + +type Row = Record; + +function num(v: unknown, fallback = 0): number { + return typeof v === "number" && Number.isFinite(v) ? v : fallback; +} + +function nullableNum(v: unknown): number | null { + return v === null || v === undefined ? null : num(v); +} + +function str(v: unknown): string { + return typeof v === "string" ? v : String(v ?? ""); +} + +function nullableStr(v: unknown): string | null { + return v === null || v === undefined ? null : str(v); +} + +export class MysqlBackend implements StorageBackend { + private readonly pool: Pool; + + constructor(dsnOrOpts: string | PoolOptions) { + this.pool = + typeof dsnOrOpts === "string" ? createPool(dsnOrOpts) : createPool(dsnOrOpts); + } + + /** Applies the DDL + schema-version stamp. Idempotent. */ + async init(): Promise { + const conn = await this.pool.getConnection(); + try { + const multi = MYSQL_DDL.trim(); + for (const stmt of multi.split(";")) { + const s = stmt.trim(); + if (s.length === 0) continue; + await conn.query(s); + } + await conn.query( + "INSERT INTO bili_meta (meta_key, meta_value) VALUES ('schema_version', ?) " + + "ON DUPLICATE KEY UPDATE meta_value = meta_value", + [String(MYSQL_SCHEMA_VERSION)], + ); + } finally { + conn.release(); + } + } + + async close(): Promise { + await this.pool.end(); + } + + /** Serialized transaction on a dedicated connection. */ + private async tx(fn: (c: PoolConnection) => Promise): Promise { + const conn = await this.pool.getConnection(); + try { + await conn.beginTransaction(); + const out = await fn(conn); + await conn.commit(); + return out; + } catch (err) { + try { + await conn.rollback(); + } catch { + /* connection-level failure — release below handles it */ + } + throw err; + } finally { + conn.release(); + } + } + + // --- content CAS ------------------------------------------------------- + + /** Atomic refcount-aware CAS insert-or-bump. */ + private async putContent(c: PoolConnection | Pool, content: string, precomputedHash?: string): Promise { + const bytes = Buffer.from(content, "utf8"); + const hash = precomputedHash ?? sha256Hex(bytes); + await c.query( + "INSERT INTO content (hash, byte_len, refcount, algo, bytes) VALUES (?, ?, 1, 'sha256', ?) " + + "ON DUPLICATE KEY UPDATE refcount = refcount + 1", + [hash, bytes.byteLength, content], + ); + return hash; + } + + private async releaseContent(c: PoolConnection | Pool, hash: string | null): Promise { + if (!hash) return; + await c.query("UPDATE content SET refcount = GREATEST(refcount - 1, 0) WHERE hash = ?", [hash]); + } + + private async getContent(c: PoolConnection | Pool, hash: string): Promise { + const [rows] = await c.query("SELECT bytes FROM content WHERE hash = ?", [hash]); + const list = rows as Row[]; + if (list.length === 0) return null; + return str(list[0].bytes); + } + + // --- ingest ------------------------------------------------------------ + + async ingestSession( + input: IngestSessionInput, + delta?: { trustedSeq: number; boundaryMsgId: string; boundaryHash: string; trustedTailSeqs: number[] }, + opts?: { completeLog?: boolean }, + ): Promise { + const s = { ...input.session, tailJson: input.session.tailJson }; + const ok = await this.tx(async (c): Promise => { + let messages: MessageRow[]; + let watermark: number | null; + if (delta) { + const probe = await this.probeIngestBoundaryOn(c, s.sessionId, delta.trustedSeq); + if (!probe || probe.msgId !== delta.boundaryMsgId || probe.contentHash !== delta.boundaryHash) { + return false; + } + for (const m of input.messages) { + if (!m.contentHash) return false; + const [hit] = (await c.query( + "SELECT 1 FROM messages WHERE session_id = ? AND msg_id = ? AND content_hash = ? LIMIT 1", + [s.sessionId, m.msgId, m.contentHash], + )) as unknown as Row[][]; + if (hit.length > 0) return false; + } + messages = input.messages.map((m, i) => ({ ...m, seq: probe.maxSeq + 1 + i })); + const tail = JSON.parse(s.tailJson) as { tailSeqs?: number[] | null }; + const prefixOk = + Array.isArray(tail.tailSeqs) && + tail.tailSeqs.length === delta.trustedTailSeqs.length + input.messages.length && + delta.trustedTailSeqs.every((v, i) => tail.tailSeqs?.[i] === v); + if (!prefixOk) return false; + s.tailJson = JSON.stringify({ + ...tail, + tailSeqs: [...delta.trustedTailSeqs, ...messages.map((m) => m.seq)], + }); + watermark = messages.length > 0 ? messages[messages.length - 1].seq : delta.trustedSeq; + } else { + const full = await this.remapPlanToAbsoluteOn(c, input); + messages = full.messages; + watermark = full.watermark; + if (opts?.completeLog === true && !(await this.hasLaneExtOn(c, s.sessionId))) { + const incomingIds = new Set(); + for (const m of messages) { + incomingIds.add(`${m.msgId}\u0000${m.contentHash ?? sha256Hex(m.content)}`); + } + const staleRows = (await c.query( + "SELECT seq, msg_id, content_hash FROM messages WHERE session_id = ?", + [s.sessionId], + )) as unknown as Row[][]; + for (const row of staleRows[0]) { + if (incomingIds.has(`${row.msg_id}\u0000${row.content_hash}`)) continue; + await this.releaseContent(c, str(row.content_hash)); + const media = (await c.query( + "SELECT media_hash FROM message_media WHERE session_id = ? AND seq = ?", + [s.sessionId, num(row.seq)], + )) as unknown as Row[][]; + for (const mm of media[0]) await this.releaseContent(c, str(mm.media_hash)); + await c.query("DELETE FROM messages WHERE session_id = ? AND seq = ?", [s.sessionId, num(row.seq)]); + await c.query("DELETE FROM message_media WHERE session_id = ? AND seq = ?", [s.sessionId, num(row.seq)]); + } + } + } + await c.query( + `INSERT INTO sessions (session_id, parent_session_id, fork_point_seq, protocol, label, title, host_title, namespace, origin, + status, created_at, updated_at, next_block_id, next_run_id, + meta_json, stats_json, metadata_json, kstate_json, tail_json, ccr_json, last_ingested_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON DUPLICATE KEY UPDATE + parent_session_id = VALUES(parent_session_id), + fork_point_seq = VALUES(fork_point_seq), + protocol = VALUES(protocol), + label = VALUES(label), + title = VALUES(title), + host_title = VALUES(host_title), + namespace = VALUES(namespace), + origin = VALUES(origin), + status = VALUES(status), + updated_at = VALUES(updated_at), + next_block_id = VALUES(next_block_id), + next_run_id = VALUES(next_run_id), + meta_json = VALUES(meta_json), + stats_json = VALUES(stats_json), + metadata_json = VALUES(metadata_json), + kstate_json = VALUES(kstate_json), + tail_json = VALUES(tail_json), + ccr_json = VALUES(ccr_json), + last_ingested_seq = VALUES(last_ingested_seq)`, + [ + s.sessionId, + s.parentSessionId ?? null, + s.forkPointSeq ?? null, + s.protocol ?? null, + s.label ?? null, + s.title ?? null, + s.hostTitle ?? null, + s.namespace ?? "proxy", + s.origin ?? null, + s.status ?? "active", + s.createdAt, + s.updatedAt, + s.nextBlockId, + s.nextRunId, + s.metaJson, + s.statsJson, + s.metadataJson, + s.kstateJson, + s.tailJson, + s.ccrJson ?? null, + watermark, + ], + ); + await c.query( + "INSERT IGNORE INTO conversations (namespace, conversation_id, session_id, claimed_at) VALUES (?, ?, ?, ?)", + [s.namespace ?? "proxy", s.sessionId, s.sessionId, s.updatedAt], + ); + for (const m of messages) { + const hash = await this.putContent(c, m.content, m.contentHash); + const [res] = (await c.query( + "INSERT IGNORE INTO messages (session_id, seq, msg_id, ref, ref_num, role, kind, content_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + [s.sessionId, m.seq, m.msgId, m.ref, m.refNum, m.role, m.kind, hash], + )) as unknown as [{ affectedRows: number }, unknown]; + if (res.affectedRows === 0) { + await this.releaseContent(c, hash); + } else if (m.media && m.media.length > 0) { + for (const md of m.media) { + await this.putContent(c, md.bytes, md.hash); + const [mm] = (await c.query( + "INSERT IGNORE INTO message_media (session_id, seq, media_hash) VALUES (?, ?, ?)", + [s.sessionId, m.seq, md.hash], + )) as unknown as [{ affectedRows: number }, unknown]; + if (mm.affectedRows === 0) await this.releaseContent(c, md.hash); + } + } + } + const incomingBlockIds = new Set(input.blocks.map((b) => b.blockId)); + const staleBlocks = (await c.query( + "SELECT block_id, summary_hash, one_hash FROM blocks WHERE session_id = ?", + [s.sessionId], + )) as unknown as Row[][]; + for (const stale of staleBlocks[0]) { + if (incomingBlockIds.has(str(stale.block_id))) continue; + await this.releaseContent(c, str(stale.summary_hash)); + if (stale.one_hash !== null && stale.one_hash !== undefined) await this.releaseContent(c, str(stale.one_hash)); + await c.query("DELETE FROM blocks WHERE session_id = ? AND block_id = ?", [s.sessionId, stale.block_id]); + } + for (const b of input.blocks) { + const summaryHash = await this.putContent(c, b.summary); + const oneHash = b.one === null ? null : await this.putContent(c, b.one); + const [existingRows] = (await c.query( + "SELECT summary_hash, one_hash FROM blocks WHERE session_id = ? AND block_id = ?", + [s.sessionId, b.blockId], + )) as unknown as Row[][]; + const existing = existingRows[0]; + if (existing) { + if (str(existing.summary_hash) === summaryHash) await this.releaseContent(c, summaryHash); + else await this.releaseContent(c, str(existing.summary_hash)); + if (existing.one_hash !== null && existing.one_hash !== undefined) { + const exOne = str(existing.one_hash); + if (exOne === oneHash && oneHash !== null) await this.releaseContent(c, oneHash); + else await this.releaseContent(c, exOne); + } + } + await c.query( + `INSERT INTO blocks (block_id, session_id, ord, start_seq, end_seq, start_ref, end_ref, tier, + summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON DUPLICATE KEY UPDATE + ord = VALUES(ord), start_seq = VALUES(start_seq), end_seq = VALUES(end_seq), + start_ref = VALUES(start_ref), end_ref = VALUES(end_ref), tier = VALUES(tier), + summary_hash = VALUES(summary_hash), one_hash = VALUES(one_hash), + supersedes_block_id = VALUES(supersedes_block_id), active = VALUES(active), + created_at = VALUES(created_at), covered_hash = VALUES(covered_hash), + payload_json = VALUES(payload_json)`, + [ + b.blockId, + s.sessionId, + b.ord, + b.startSeq, + b.endSeq, + b.startRef, + b.endRef, + b.tier, + summaryHash, + oneHash, + b.supersedesBlockId, + b.active ? 1 : 0, + b.createdAt, + b.coveredHash, + b.payloadJson, + ], + ); + } + await c.query("DELETE FROM dead_refs WHERE session_id = ? AND ord >= ?", [s.sessionId, input.deadRefs.length]); + for (const d of input.deadRefs) { + await c.query( + "INSERT INTO dead_refs (session_id, ord, ref, ref_num, recorded_at) VALUES (?, ?, ?, ?, ?) " + + "ON DUPLICATE KEY UPDATE ref = VALUES(ref), ref_num = VALUES(ref_num), recorded_at = VALUES(recorded_at)", + [s.sessionId, d.ord, d.ref, d.refNum, d.recordedAt], + ); + } + await c.query("DELETE FROM refs WHERE session_id = ?", [s.sessionId]); + for (const r of input.refs) { + await c.query( + "INSERT INTO refs (session_id, ord, raw_id, ref, ref_num) VALUES (?, ?, ?, ?, ?) " + + "ON DUPLICATE KEY UPDATE ord = VALUES(ord), ref = VALUES(ref), ref_num = VALUES(ref_num)", + [s.sessionId, r.ord, r.rawId, r.ref, r.refNum], + ); + } + const oldCcr = (await c.query( + "SELECT content_hash FROM ccr_entries WHERE session_id = ?", + [s.sessionId], + )) as unknown as Row[][]; + for (const row of oldCcr[0]) await this.releaseContent(c, str(row.content_hash)); + await c.query("DELETE FROM ccr_entries WHERE session_id = ?", [s.sessionId]); + for (const e of input.ccrEntries) { + const hash = await this.putContent(c, e.content); + await c.query( + "INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)", + [s.sessionId, e.ord, e.key, hash], + ); + } + return true; + }); + return ok; + } + + private async hasLaneExtOn(c: PoolConnection, sessionId: string): Promise { + const [rows] = (await c.query( + "SELECT 1 FROM session_ext WHERE session_id = ? AND ext_key LIKE 'inc:%' LIMIT 1", + [sessionId], + )) as unknown as Row[][]; + return rows.length > 0; + } + + private async remapPlanToAbsoluteOn( + c: PoolConnection, + input: IngestSessionInput, + ): Promise<{ messages: MessageRow[]; watermark: number | null }> { + const s = input.session; + const [rows] = (await c.query( + "SELECT seq, msg_id, content_hash FROM messages WHERE session_id = ?", + [s.sessionId], + )) as unknown as Row[][]; + const seqByIdentity = new Map(); + let nextSeq = 0; + for (const row of rows) { + seqByIdentity.set(`${row.msg_id}\u0000${row.content_hash}`, num(row.seq)); + if (num(row.seq) + 1 > nextSeq) nextSeq = num(row.seq) + 1; + } + const planToAbs = new Map(); + const messages = input.messages.map((m) => { + const hash = m.contentHash ?? sha256Hex(m.content); + const key = `${m.msgId}\u0000${hash}`; + let abs = seqByIdentity.get(key); + if (abs === undefined) { + abs = nextSeq++; + seqByIdentity.set(key, abs); + } + planToAbs.set(m.seq, abs); + return { ...m, seq: abs }; + }); + const remapSeqs = (seqs: number[] | null | undefined): number[] | null | undefined => + seqs === null || seqs === undefined ? seqs : seqs.map((p) => planToAbs.get(p) ?? p); + const pluginSnapshotSeqs = remapSeqs(input.pluginSnapshotSeqs); + const tailSeqs = remapSeqs(input.tailSeqs); + const watermark = tailSeqs !== null && tailSeqs !== undefined && tailSeqs.length > 0 ? tailSeqs[tailSeqs.length - 1] : null; + s.tailJson = JSON.stringify({ + ...(JSON.parse(s.tailJson) as Record), + ...(pluginSnapshotSeqs !== undefined ? { pluginSnapshotSeqs } : {}), + ...(tailSeqs !== undefined ? { tailSeqs } : {}), + }); + return { messages, watermark }; + } + + private async probeIngestBoundaryOn( + c: PoolConnection, + sessionId: string, + seq: number, + ): Promise<{ msgId: string; contentHash: string; maxSeq: number } | null> { + const [rows] = (await c.query( + "SELECT msg_id, content_hash FROM messages WHERE session_id = ? AND seq = ?", + [sessionId, seq], + )) as unknown as Row[][]; + if (rows.length === 0) return null; + const [maxRows] = (await c.query( + "SELECT MAX(seq) AS m FROM messages WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + const m = maxRows[0]?.m; + return { msgId: str(rows[0].msg_id), contentHash: str(rows[0].content_hash), maxSeq: m === null || m === undefined ? -1 : num(m) }; + } + + async sessionCounters(sessionId: string): Promise<{ requests: number; nextBlockId: number } | null> { + const [rows] = (await this.pool.query( + "SELECT next_block_id, stats_json FROM sessions WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + if (rows.length === 0) return null; + try { + const stats = JSON.parse(str(rows[0].stats_json)) as { requests?: unknown }; + return { + requests: typeof stats.requests === "number" ? stats.requests : 0, + nextBlockId: num(rows[0].next_block_id), + }; + } catch { + return null; + } + } + + async sessionUpdatedAt(sessionId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT updated_at FROM sessions WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + return rows.length > 0 ? num(rows[0].updated_at) : null; + } + + async ingestWatermark(sessionId: string): Promise<{ seq: number; tailSeqs: number[] } | null> { + const [rows] = (await this.pool.query( + "SELECT last_ingested_seq, tail_json FROM sessions WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + if (rows.length === 0) return null; + const seq = rows[0].last_ingested_seq; + if (typeof seq !== "number" || !Number.isSafeInteger(seq) || seq < 0) return null; + const tail = JSON.parse(str(rows[0].tail_json)) as { tailSeqs?: number[] | null }; + const seqs = tail.tailSeqs; + if (!Array.isArray(seqs) || seqs.length === 0 || seqs[seqs.length - 1] !== seq) return null; + return { seq, tailSeqs: seqs }; + } + + async tailSeqs(sessionId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT tail_json FROM sessions WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + if (rows.length === 0) return null; + try { + const tail = JSON.parse(str(rows[0].tail_json)) as { tailSeqs?: number[] | null }; + return Array.isArray(tail.tailSeqs) ? tail.tailSeqs : null; + } catch { + return null; + } + } + + async probeIngestBoundary(sessionId: string, seq: number): Promise<{ msgId: string; contentHash: string; maxSeq: number } | null> { + const conn = await this.pool.getConnection(); + try { + return await this.probeIngestBoundaryOn(conn, sessionId, seq); + } finally { + conn.release(); + } + } + + async clearIngestWatermark(sessionId: string): Promise { + await this.pool.query("UPDATE sessions SET last_ingested_seq = NULL WHERE session_id = ?", [sessionId]); + } + + async resetAllIngestWatermarks(): Promise { + await this.pool.query("UPDATE sessions SET last_ingested_seq = NULL"); + } + + async reapBlocks( + sessionId: string, + reapedIds: Set, + orphanContents: Map, + ): Promise { + if (reapedIds.size === 0 && orphanContents.size === 0) return; + await this.tx(async (c) => { + for (const blockId of reapedIds) { + const [rows] = (await c.query( + "SELECT summary_hash, one_hash FROM blocks WHERE session_id = ? AND block_id = ?", + [sessionId, blockId], + )) as unknown as Row[][]; + if (rows.length === 0) continue; + await this.releaseContent(c, str(rows[0].summary_hash)); + if (rows[0].one_hash !== null && rows[0].one_hash !== undefined) await this.releaseContent(c, str(rows[0].one_hash)); + await c.query("DELETE FROM blocks WHERE session_id = ? AND block_id = ?", [sessionId, blockId]); + } + if (orphanContents.size > 0) { + const [sess] = (await c.query( + "SELECT orphans_json FROM sessions WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + if (sess.length > 0) { + let merged: Record = {}; + try { + const parsed = JSON.parse(str(sess[0].orphans_json ?? "{}")) as unknown; + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) merged = parsed as Record; + } catch { + /* corrupt journal → start fresh */ + } + for (const [blockId, view] of orphanContents) merged[blockId] = view; + await c.query("UPDATE sessions SET orphans_json = ? WHERE session_id = ?", [JSON.stringify(merged), sessionId]); + } + } + }); + } + + async deleteSession(sessionId: string): Promise { + await this.tx(async (c) => { + const refs: string[] = []; + const msgRows = (await c.query("SELECT content_hash FROM messages WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + for (const m of msgRows[0]) refs.push(str(m.content_hash)); + const mediaRows = (await c.query("SELECT media_hash FROM message_media WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + for (const mm of mediaRows[0]) refs.push(str(mm.media_hash)); + const blockRows = (await c.query("SELECT summary_hash, one_hash FROM blocks WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + for (const b of blockRows[0]) { + refs.push(str(b.summary_hash)); + if (b.one_hash !== null && b.one_hash !== undefined) refs.push(str(b.one_hash)); + } + const ccrRows = (await c.query("SELECT content_hash FROM ccr_entries WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + for (const e of ccrRows[0]) refs.push(str(e.content_hash)); + const cpRows = (await c.query("SELECT cp_id, state_hash FROM checkpoints WHERE session_id = ? AND pinned = 0", [sessionId])) as unknown as Row[][]; + for (const cp of cpRows[0]) { + await c.query("DELETE FROM checkpoints WHERE cp_id = ?", [cp.cp_id]); + refs.push(str(cp.state_hash)); + } + await c.query("DELETE FROM messages WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM message_media WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM blocks WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM dead_refs WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM refs WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM ccr_entries WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM conversations WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM session_ext WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM sessions WHERE session_id = ?", [sessionId]); + for (const hash of refs) await this.releaseContent(c, hash); + await c.query("DELETE FROM content WHERE refcount <= 0"); + }); + } + + async listSessionIds(): Promise { + const [rows] = (await this.pool.query("SELECT session_id FROM sessions ORDER BY session_id")) as unknown as Row[][]; + return rows.map((r) => str(r.session_id)); + } + + async lookupConversation(namespace: string, conversationId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT session_id FROM conversations WHERE namespace = ? AND conversation_id = ?", + [namespace, conversationId], + )) as unknown as Row[][]; + return rows.length > 0 ? str(rows[0].session_id) : null; + } + + async listConversationBindings(namespace?: string): Promise<{ namespace: string; conversationId: string; sessionId: string; claimedAt: number }[]> { + const rows = (namespace === undefined + ? await this.pool.query("SELECT namespace, conversation_id, session_id, claimed_at FROM conversations ORDER BY namespace, conversation_id") + : await this.pool.query( + "SELECT namespace, conversation_id, session_id, claimed_at FROM conversations WHERE namespace = ? ORDER BY conversation_id", + [namespace], + )) as unknown as Row[][]; + return rows[0].map((r) => ({ + namespace: str(r.namespace), + conversationId: str(r.conversation_id), + sessionId: str(r.session_id), + claimedAt: num(r.claimed_at), + })); + } + + async getSessionRow(sessionId: string): Promise { + const [rows] = (await this.pool.query( + `SELECT session_id, parent_session_id, fork_point_seq, protocol, label, title, host_title, namespace, origin, status, + created_at, updated_at, next_block_id, next_run_id, + meta_json, stats_json, metadata_json, kstate_json, tail_json, ccr_json, last_ingested_seq, orphans_json + FROM sessions WHERE session_id = ?`, + [sessionId], + )) as unknown as Row[][]; + if (rows.length === 0) return null; + const r = rows[0]; + return { + sessionId: str(r.session_id), + parentSessionId: nullableStr(r.parent_session_id), + forkPointSeq: nullableNum(r.fork_point_seq), + protocol: nullableStr(r.protocol), + label: nullableStr(r.label), + title: nullableStr(r.title), + hostTitle: nullableStr(r.host_title), + namespace: typeof r.namespace === "string" ? r.namespace : "proxy", + origin: nullableStr(r.origin), + status: str(r.status ?? "active"), + createdAt: num(r.created_at), + updatedAt: num(r.updated_at), + nextBlockId: num(r.next_block_id, 1), + nextRunId: num(r.next_run_id, 1), + metaJson: str(r.meta_json ?? "{}"), + statsJson: str(r.stats_json ?? "{}"), + metadataJson: str(r.metadata_json ?? "{}"), + kstateJson: str(r.kstate_json ?? "{}"), + tailJson: str(r.tail_json ?? "{}"), + ccrJson: nullableStr(r.ccr_json), + lastIngestedSeq: nullableNum(r.last_ingested_seq), + orphansJson: nullableStr(r.orphans_json), + }; + } + + async getMessages(sessionId: string): Promise<(MessageRow & { contentHash: string })[]> { + const [rows] = (await this.pool.query( + "SELECT seq, msg_id, ref, ref_num, role, kind, content_hash FROM messages WHERE session_id = ? ORDER BY seq", + [sessionId], + )) as unknown as Row[][]; + const out: (MessageRow & { contentHash: string })[] = []; + for (const r of rows) { + const contentHash = str(r.content_hash); + out.push({ + seq: num(r.seq), + msgId: str(r.msg_id), + ref: nullableStr(r.ref), + refNum: nullableNum(r.ref_num), + role: str(r.role), + kind: str(r.kind), + contentHash, + content: await this.hydrateMediaAsync((await this.getContent(this.pool, contentHash)) ?? ""), + }); + } + return out; + } + + private async hydrateMediaAsync(content: string): Promise { + if (!content.includes('"$media"')) return content; + try { + const parsed = JSON.parse(content) as { imageBase64?: unknown }; + const slot = parsed.imageBase64; + if (slot !== null && typeof slot === "object" && typeof (slot as { $media?: unknown }).$media === "string") { + const bytes = await this.getContent(this.pool, (slot as { $media: string }).$media); + if (typeof bytes === "string" && bytes.length > 0) { + parsed.imageBase64 = bytes; + return JSON.stringify(parsed); + } + } + return content; + } catch { + return content; + } + } + + async getBlocks(sessionId: string): Promise<(BlockRow & { summaryHash: string; oneHash: string | null })[]> { + const [rows] = (await this.pool.query( + `SELECT block_id, ord, start_seq, end_seq, start_ref, end_ref, tier, summary_hash, one_hash, + supersedes_block_id, active, created_at, covered_hash, payload_json + FROM blocks WHERE session_id = ? ORDER BY ord`, + [sessionId], + )) as unknown as Row[][]; + const out: (BlockRow & { summaryHash: string; oneHash: string | null })[] = []; + for (const r of rows) { + out.push({ + blockId: str(r.block_id), + ord: num(r.ord), + startSeq: num(r.start_seq), + endSeq: num(r.end_seq), + startRef: nullableNum(r.start_ref), + endRef: nullableNum(r.end_ref), + tier: num(r.tier), + summaryHash: str(r.summary_hash), + oneHash: nullableStr(r.one_hash), + summary: (await this.getContent(this.pool, str(r.summary_hash))) ?? "", + one: r.one_hash === null || r.one_hash === undefined ? null : ((await this.getContent(this.pool, str(r.one_hash))) ?? null), + supersedesBlockId: nullableStr(r.supersedes_block_id), + active: num(r.active) === 1, + createdAt: num(r.created_at), + coveredHash: str(r.covered_hash), + payloadJson: str(r.payload_json), + }); + } + return out; + } + + async getDeadRefs(sessionId: string): Promise<{ ord: number; ref: string; refNum: number; recordedAt: number }[]> { + const [rows] = (await this.pool.query( + "SELECT ord, ref, ref_num, recorded_at FROM dead_refs WHERE session_id = ? ORDER BY ord", + [sessionId], + )) as unknown as Row[][]; + return rows.map((r) => ({ + ord: num(r.ord), + ref: str(r.ref), + refNum: num(r.ref_num), + recordedAt: num(r.recorded_at), + })); + } + + async getRefs(sessionId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT ord, raw_id, ref, ref_num FROM refs WHERE session_id = ? ORDER BY ord", + [sessionId], + )) as unknown as Row[][]; + return rows.map((r) => ({ + ord: num(r.ord), + rawId: str(r.raw_id), + ref: str(r.ref), + refNum: nullableNum(r.ref_num), + })); + } + + async getCcr(sessionId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT ccr_json FROM sessions WHERE session_id = ?", + [sessionId], + )) as unknown as Row[][]; + if (rows.length === 0 || rows[0].ccr_json === null || rows[0].ccr_json === undefined) return null; + try { + const parsed = JSON.parse(str(rows[0].ccr_json)) as MessageContentStore; + if (parsed && typeof parsed === "object" && parsed.version === 1 + && parsed.byHash && typeof parsed.byHash === "object" + && parsed.byRef && typeof parsed.byRef === "object") { + return parsed; + } + } catch { + /* corrupt mirror → null */ + } + return null; + } + + async getCcrEntries(sessionId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord", + [sessionId], + )) as unknown as Row[][]; + const out: CcrEntryRow[] = []; + for (const r of rows) { + out.push({ ord: num(r.ord), key: str(r.ccr_key), content: (await this.getContent(this.pool, str(r.content_hash))) ?? "" }); + } + return out; + } + + // --- archive / GC ------------------------------------------------------ + + static readonly ARCHIVE_LOG_NAME = "archive.jsonl"; + + async archiveSession(sessionId: string, opts: { archiveDir: string }): Promise<{ ok: boolean; exportPath?: string; reason?: string; contentRowsDeleted?: number }> { + const row = await this.getSessionRow(sessionId); + if (!row) return { ok: false, reason: "not-found" }; + const exportPath = join(opts.archiveDir, MysqlBackend.ARCHIVE_LOG_NAME); + if (row.status === "archived") return { ok: true, exportPath }; + const q = async (sql: string): Promise => ((await this.pool.query(sql, [sessionId])) as unknown as Row[][])[0]; + const messages = await q("SELECT seq, msg_id, ref, ref_num, role, kind, content_hash FROM messages WHERE session_id = ? ORDER BY seq"); + const mediaMaps = await q("SELECT seq, media_hash FROM message_media WHERE session_id = ?"); + const blocks = await q("SELECT block_id, ord, start_seq, end_seq, start_ref, end_ref, tier, summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json FROM blocks WHERE session_id = ? ORDER BY ord"); + const refsRows = await q("SELECT ord, raw_id, ref, ref_num FROM refs WHERE session_id = ? ORDER BY ord"); + const deadRefs = await q("SELECT ord, ref, ref_num, recorded_at FROM dead_refs WHERE session_id = ? ORDER BY ord"); + const ccr = await q("SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord"); + const ext = await q("SELECT ext_key AS `key`, value_json FROM session_ext WHERE session_id = ?"); + const checkpoints = await q("SELECT cp_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash FROM checkpoints WHERE session_id = ?"); + const wanted = new Set(); + for (const m of messages) wanted.add(str(m.content_hash)); + for (const mm of mediaMaps) wanted.add(str(mm.media_hash)); + for (const b of blocks) { + wanted.add(str(b.summary_hash)); + if (b.one_hash !== null && b.one_hash !== undefined) wanted.add(str(b.one_hash)); + } + for (const e of ccr) wanted.add(str(e.content_hash)); + for (const cp of checkpoints) if (num(cp.pinned) === 0) wanted.add(str(cp.state_hash)); + const content: Record = {}; + for (const hash of wanted) { + const c = await this.getContent(this.pool, hash); + if (c !== null) content[hash] = c; + } + mkdirSync(opts.archiveDir, { recursive: true }); + appendFileSync(exportPath, JSON.stringify({ version: 1, sessionId, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints }) + "\n"); + let contentRowsDeleted = 0; + await this.tx(async (c) => { + const rel: string[] = []; + for (const m of messages) rel.push(str(m.content_hash)); + for (const mm of mediaMaps) rel.push(str(mm.media_hash)); + for (const b of blocks) { + rel.push(str(b.summary_hash)); + if (b.one_hash !== null && b.one_hash !== undefined) rel.push(str(b.one_hash)); + } + for (const e of ccr) rel.push(str(e.content_hash)); + for (const cp of checkpoints) { + if (num(cp.pinned) === 0) { + rel.push(str(cp.state_hash)); + await c.query("DELETE FROM checkpoints WHERE cp_id = ?", [cp.cp_id]); + } + } + await c.query("DELETE FROM messages WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM message_media WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM blocks WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM dead_refs WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM refs WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM ccr_entries WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM session_ext WHERE session_id = ?", [sessionId]); + await c.query("UPDATE sessions SET status = 'archived' WHERE session_id = ?", [sessionId]); + for (const hash of rel) await this.releaseContent(c, hash); + const [zero] = (await c.query("SELECT COUNT(*) AS n FROM content WHERE refcount <= 0")) as unknown as Row[][]; + contentRowsDeleted = num(zero[0].n); + await c.query("DELETE FROM content WHERE refcount <= 0"); + }); + loggerLog("info", `[#3931] archived ${sessionId} → ${exportPath} (${contentRowsDeleted} content rows collected)`); + return { ok: true, exportPath, contentRowsDeleted }; + } + + async restoreArchived(sessionId: string, opts: { archiveDir: string }): Promise<{ ok: boolean; reason?: string }> { + const exportPath = join(opts.archiveDir, MysqlBackend.ARCHIVE_LOG_NAME); + if (!existsSync(exportPath)) return { ok: false, reason: "export-missing" }; + let dump: { + session: SessionRow; + content: Record; + messages: Record[]; + blocks: Record[]; + refs: Record[]; + deadRefs: Record[]; + ccrEntries: Record[]; + sessionExt: Record[]; + checkpoints: Record[]; + }; + let lineNo = -1; + let rawLines: string[] = []; + try { + rawLines = readFileSync(exportPath, "utf8").split("\n"); + for (let i = 0; i < rawLines.length; i++) { + if (!rawLines[i].trim()) continue; + let candidate: { sessionId?: string }; + try { + candidate = JSON.parse(rawLines[i]) as { sessionId?: string }; + } catch { + continue; + } + if (candidate.sessionId === sessionId) { + lineNo = i; + break; + } + } + } catch { + return { ok: false, reason: "export-corrupt" }; + } + if (lineNo === -1) return { ok: false, reason: "export-missing" }; + try { + dump = JSON.parse(rawLines[lineNo]) as typeof dump; + } catch { + return { ok: false, reason: "export-corrupt" }; + } + if (!dump.session || dump.session.sessionId !== sessionId) return { ok: false, reason: "export-mismatch" }; + const sess = dump.session; + await this.tx(async (c) => { + for (const [hash, body] of Object.entries(dump.content)) await this.putContent(c, body, hash); + for (const m of dump.messages) { + await c.query( + "INSERT INTO messages (session_id, seq, msg_id, ref, ref_num, role, kind, content_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + [sessionId, m.seq, m.msg_id, m.ref ?? null, m.ref_num ?? null, m.role, m.kind ?? "message", m.content_hash], + ); + } + for (const mm of (dump as { mediaMaps?: Record[] }).mediaMaps ?? []) { + await c.query("INSERT IGNORE INTO message_media (session_id, seq, media_hash) VALUES (?, ?, ?)", [sessionId, mm.seq, mm.media_hash]); + } + for (const b of dump.blocks) { + await c.query( + `INSERT IGNORE INTO blocks (block_id, session_id, ord, start_seq, end_seq, start_ref, end_ref, tier, + summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + [b.block_id, sessionId, b.ord, b.start_seq, b.end_seq, b.start_ref ?? null, b.end_ref ?? null, b.tier, b.summary_hash, b.one_hash ?? null, b.supersedes_block_id ?? null, b.active, b.created_at, b.covered_hash, b.payload_json], + ); + } + for (const r of dump.refs) { + await c.query("INSERT INTO refs (session_id, ord, raw_id, ref, ref_num) VALUES (?, ?, ?, ?, ?)", [sessionId, r.ord, r.raw_id, r.ref, r.ref_num ?? null]); + } + for (const d of dump.deadRefs) { + await c.query("INSERT INTO dead_refs (session_id, ord, ref, ref_num, recorded_at) VALUES (?, ?, ?, ?, ?)", [sessionId, d.ord, d.ref, d.ref_num, d.recorded_at]); + } + for (const e of dump.ccrEntries) { + await c.query("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)", [sessionId, e.ord, e.ccr_key, e.content_hash]); + } + for (const x of dump.sessionExt) { + await c.query( + "INSERT INTO session_ext (session_id, ext_key, value_json, updated_at) VALUES (?, ?, ?, ?) " + + "ON DUPLICATE KEY UPDATE value_json = VALUES(value_json), updated_at = VALUES(updated_at)", + [sessionId, x.key, x.value_json, Date.now()], + ); + } + for (const cp of dump.checkpoints) { + await c.query( + "INSERT IGNORE INTO checkpoints (cp_id, session_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + [cp.cp_id, sessionId, cp.seq_at, cp.block_set_hash, cp.kind, cp.created_at, cp.pinned, cp.label ?? null, cp.state_hash], + ); + } + await c.query( + `UPDATE sessions SET parent_session_id = ?, fork_point_seq = ?, protocol = ?, label = ?, title = ?, host_title = ?, namespace = ?, origin = ?, + created_at = ?, updated_at = ?, next_block_id = ?, next_run_id = ?, meta_json = ?, stats_json = ?, metadata_json = ?, kstate_json = ?, tail_json = ?, ccr_json = ?, last_ingested_seq = ?, orphans_json = ?, status = 'active' WHERE session_id = ?`, + [ + sess.parentSessionId ?? null, sess.forkPointSeq ?? null, sess.protocol ?? null, sess.label ?? null, + sess.title ?? null, sess.hostTitle ?? null, sess.namespace ?? "proxy", sess.origin ?? null, + sess.createdAt, sess.updatedAt, sess.nextBlockId, sess.nextRunId, + sess.metaJson, sess.statsJson, sess.metadataJson, sess.kstateJson, sess.tailJson, sess.ccrJson ?? null, + sess.lastIngestedSeq ?? null, sess.orphansJson ?? null, sessionId, + ], + ); + }); + const kept: string[] = []; + for (const l of rawLines) { + if (!l.trim()) continue; + try { + const head = JSON.parse(l) as { sessionId?: string }; + if (head.sessionId === sessionId) continue; + } catch { + continue; + } + kept.push(l); + } + writeFileSync(exportPath, kept.length > 0 ? kept.join("\n") + "\n" : ""); + loggerLog("info", `[#3931] restored archived session ${sessionId} from ${exportPath} (line removed from the archive log)`); + return { ok: true }; + } + + async gc(opts: { archiveAfterDays: number; archiveDir: string; now?: number; vacuum?: "full" | "incremental" }): Promise<{ archived: number; contentRowsDeleted: number }> { + const now = opts.now ?? Date.now(); + let archived = 0; + let contentRowsDeleted = 0; + if (opts.archiveAfterDays > 0) { + const cutoff = now - opts.archiveAfterDays * 86_400_000; + const [stale] = (await this.pool.query( + "SELECT session_id FROM sessions WHERE status = 'active' AND updated_at < ?", + [cutoff], + )) as unknown as Row[][]; + for (const s of stale) { + const r = await this.archiveSession(str(s.session_id), { archiveDir: opts.archiveDir }); + if (r.ok) archived++; + if (r.contentRowsDeleted) contentRowsDeleted += r.contentRowsDeleted; + } + } + const [zero] = (await this.pool.query("SELECT COUNT(*) AS n FROM content WHERE refcount <= 0")) as unknown as Row[][]; + if (num(zero[0].n) > 0) { + await this.pool.query("DELETE FROM content WHERE refcount <= 0"); + contentRowsDeleted += num(zero[0].n); + } + if (opts.vacuum === "full") { + loggerLog("info", "[#3931] gc: running OPTIMIZE TABLE (mysql counterpart of the sqlite full VACUUM)"); + await this.pool.query("OPTIMIZE TABLE sessions"); + await this.pool.query("OPTIMIZE TABLE messages"); + await this.pool.query("OPTIMIZE TABLE content"); + } + return { archived, contentRowsDeleted }; + } + + async contentRefCountOfBody(needle: string): Promise { + const [rows] = (await this.pool.query( + "SELECT refcount FROM content WHERE bytes LIKE CONCAT('%', ?, '%') LIMIT 1", + [needle], + )) as unknown as Row[][]; + return rows.length > 0 ? num(rows[0].refcount) : 0; + } + + async mediaRefCount(bytes: string): Promise { + const [rows] = (await this.pool.query("SELECT refcount FROM content WHERE bytes = ? LIMIT 1", [bytes])) as unknown as Row[][]; + return rows.length > 0 ? num(rows[0].refcount) : 0; + } + + // --- checkpoints ------------------------------------------------------- + + async insertCheckpoint(cp: { + cpId: string; + sessionId: string; + seqAt: number; + blockSetHash: string; + kind: "auto" | "manual" | "replica" | "fork_base"; + label?: string | null; + pinned: boolean; + createdAt: number; + stateJson: string; + }): Promise { + await this.tx(async (c) => { + const stateHash = await this.putContent(c, cp.stateJson); + await c.query( + "INSERT INTO checkpoints (cp_id, session_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + [cp.cpId, cp.sessionId, cp.seqAt, cp.blockSetHash, cp.kind, cp.createdAt, cp.pinned ? 1 : 0, cp.label ?? null, stateHash], + ); + }); + } + + async listCheckpoints(sessionId?: string): Promise { + const cols = "cp_id, session_id, seq_at, block_set_hash, kind, created_at, pinned, label"; + const rows = (sessionId === undefined + ? await this.pool.query(`SELECT ${cols} FROM checkpoints ORDER BY created_at, cp_id`) + : await this.pool.query(`SELECT ${cols} FROM checkpoints WHERE session_id = ? ORDER BY created_at, cp_id`, [sessionId])) as unknown as Row[][]; + return rows[0].map((r) => ({ + cp_id: str(r.cp_id), + session_id: str(r.session_id), + seq_at: num(r.seq_at), + block_set_hash: str(r.block_set_hash), + kind: str(r.kind) as CheckpointRow["kind"], + created_at: num(r.created_at), + pinned: num(r.pinned), + label: nullableStr(r.label), + })); + } + + async getCheckpoint(cpId: string): Promise { + const [rows] = (await this.pool.query( + "SELECT cp_id, session_id, seq_at, block_set_hash, kind, created_at, pinned, label FROM checkpoints WHERE cp_id = ?", + [cpId], + )) as unknown as Row[][]; + if (rows.length === 0) return undefined; + const r = rows[0]; + return { + cp_id: str(r.cp_id), + session_id: str(r.session_id), + seq_at: num(r.seq_at), + block_set_hash: str(r.block_set_hash), + kind: str(r.kind) as CheckpointRow["kind"], + created_at: num(r.created_at), + pinned: num(r.pinned), + label: nullableStr(r.label), + }; + } + + async checkpointState(cpId: string): Promise<{ stateJson: string; byteLen: number }> { + const [rows] = (await this.pool.query("SELECT state_hash FROM checkpoints WHERE cp_id = ?", [cpId])) as unknown as Row[][]; + if (rows.length === 0) throw new Error(`checkpoint not found: ${cpId}`); + const [blobs] = (await this.pool.query("SELECT bytes, byte_len FROM content WHERE hash = ?", [rows[0].state_hash])) as unknown as Row[][]; + if (blobs.length === 0) throw new Error(`checkpoint blob missing (CAS leak): ${cpId}`); + return { stateJson: str(blobs[0].bytes), byteLen: num(blobs[0].byte_len) }; + } + + async deleteCheckpoint(cpId: string): Promise { + return this.tx(async (c): Promise => { + const [rows] = (await c.query("SELECT state_hash FROM checkpoints WHERE cp_id = ?", [cpId])) as unknown as Row[][]; + if (rows.length === 0) return false; + await c.query("DELETE FROM checkpoints WHERE cp_id = ?", [cpId]); + await this.releaseContent(c, str(rows[0].state_hash)); + await c.query("DELETE FROM content WHERE refcount <= 0"); + return true; + }); + } + + // --- session_ext / kv -------------------------------------------------- + + async setSessionExt(sessionId: string, key: string, value: unknown): Promise { + const valueJson = JSON.stringify(value); + await this.pool.query( + "INSERT INTO session_ext (session_id, ext_key, value_json, updated_at) VALUES (?, ?, ?, ?) " + + "ON DUPLICATE KEY UPDATE value_json = VALUES(value_json), updated_at = VALUES(updated_at)", + [sessionId, key, valueJson, Date.now()], + ); + } + + async getSessionExt(sessionId: string, key: string): Promise { + const [rows] = (await this.pool.query( + "SELECT value_json FROM session_ext WHERE session_id = ? AND ext_key = ?", + [sessionId, key], + )) as unknown as Row[][]; + return rows.length === 0 ? undefined : JSON.parse(str(rows[0].value_json)); + } + + async listSessionExt(sessionId: string): Promise<{ key: string; value: unknown; updatedAt: number }[]> { + const [rows] = (await this.pool.query( + "SELECT ext_key, value_json, updated_at FROM session_ext WHERE session_id = ? ORDER BY ext_key", + [sessionId], + )) as unknown as Row[][]; + return rows.map((r) => ({ key: str(r.ext_key), value: JSON.parse(str(r.value_json)), updatedAt: num(r.updated_at) })); + } + + async deleteSessionExt(sessionId: string, key?: string): Promise { + if (key === undefined) { + const [r] = (await this.pool.query("DELETE FROM session_ext WHERE session_id = ?", [sessionId])) as unknown as [{ affectedRows: number }, unknown]; + return r.affectedRows; + } + const [r] = (await this.pool.query("DELETE FROM session_ext WHERE session_id = ? AND ext_key = ?", [sessionId, key])) as unknown as [{ affectedRows: number }, unknown]; + return r.affectedRows; + } + + async setKv(namespace: string, key: string, value: unknown): Promise { + const valueJson = JSON.stringify(value); + await this.pool.query( + "INSERT INTO kv_store (namespace, kv_key, value_json, updated_at) VALUES (?, ?, ?, ?) " + + "ON DUPLICATE KEY UPDATE value_json = VALUES(value_json), updated_at = VALUES(updated_at)", + [namespace, key, valueJson, Date.now()], + ); + } + + async getKv(namespace: string, key: string): Promise { + const [rows] = (await this.pool.query( + "SELECT value_json FROM kv_store WHERE namespace = ? AND kv_key = ?", + [namespace, key], + )) as unknown as Row[][]; + return rows.length === 0 ? undefined : JSON.parse(str(rows[0].value_json)); + } + + async listKv(namespace?: string, keyPrefix?: string): Promise<{ namespace: string; key: string; value: unknown; updatedAt: number }[]> { + const where: string[] = []; + const args: unknown[] = []; + if (namespace !== undefined) { + where.push("namespace = ?"); + args.push(namespace); + } + if (keyPrefix !== undefined && keyPrefix !== "") { + where.push("kv_key LIKE CONCAT(?, '%')"); + args.push(keyPrefix.replace(/[\\%_]/g, (ch) => "\\" + ch)); + } + const sql = + "SELECT namespace, kv_key, value_json, updated_at FROM kv_store" + + (where.length > 0 ? " WHERE " + where.join(" AND ") : "") + + " ORDER BY namespace, kv_key"; + const [rows] = (await this.pool.query(sql, args)) as unknown as Row[][]; + return rows.map((r) => ({ namespace: str(r.namespace), key: str(r.kv_key), value: JSON.parse(str(r.value_json)), updatedAt: num(r.updated_at) })); + } + + async deleteKv(namespace: string, key?: string): Promise { + if (key === undefined) { + const [r] = (await this.pool.query("DELETE FROM kv_store WHERE namespace = ?", [namespace])) as unknown as [{ affectedRows: number }, unknown]; + return r.affectedRows; + } + const [r] = (await this.pool.query("DELETE FROM kv_store WHERE namespace = ? AND kv_key = ?", [namespace, key])) as unknown as [{ affectedRows: number }, unknown]; + return r.affectedRows; + } + + // --- summaries / diagnostics ------------------------------------------- + + async sessionSummaries(): Promise { + const [rows] = (await this.pool.query( + `SELECT s.session_id AS sessionId, s.host_title AS hostTitle, s.title AS title, s.label AS label, + s.protocol AS protocol, s.origin AS origin, s.namespace AS namespace, s.stats_json AS statsJson, + s.updated_at AS updatedAt, + (SELECT COUNT(*) FROM blocks b WHERE b.session_id = s.session_id) AS blocks, + (SELECT MAX(b.created_at) FROM blocks b WHERE b.session_id = s.session_id) AS latestBlockAt + FROM sessions s`, + )) as unknown as Row[][]; + return rows.map((r) => { + let contextTokens: number | undefined; + try { + const v = (JSON.parse(str(r.statsJson)) as { contextTokens?: unknown }).contextTokens; + if (typeof v === "number" && Number.isFinite(v)) contextTokens = v; + } catch { + /* malformed stats: undefined */ + } + return { + sessionId: str(r.sessionId), + title: r.hostTitle !== null && r.hostTitle !== undefined ? str(r.hostTitle) : r.title !== null && r.title !== undefined ? str(r.title) : undefined, + label: r.label !== null && r.label !== undefined ? str(r.label) : undefined, + protocol: r.protocol !== null && r.protocol !== undefined ? str(r.protocol) : undefined, + origin: r.origin !== null && r.origin !== undefined ? str(r.origin) : undefined, + namespace: str(r.namespace ?? "proxy"), + blocks: num(r.blocks), + latestBlockAt: r.latestBlockAt !== null && r.latestBlockAt !== undefined ? num(r.latestBlockAt) : undefined, + contextTokens, + updatedAt: num(r.updatedAt), + }; + }); + } + + async webSummaryRows(): Promise { + const [rows] = (await this.pool.query( + `SELECT s.session_id AS sessionId, s.title AS title, s.host_title AS hostTitle, s.label AS label, + s.protocol AS protocol, s.origin AS origin, s.updated_at AS updatedAt, + s.stats_json AS statsJson, s.metadata_json AS metadataJson, + (SELECT COUNT(*) FROM blocks b WHERE b.session_id = s.session_id) AS blocks, + (SELECT JSON_UNQUOTE(JSON_EXTRACT(b.payload_json, '$.topic')) FROM blocks b + WHERE b.session_id = s.session_id + AND (JSON_EXTRACT(b.payload_json, '$.topic') IS NOT NULL + OR JSON_EXTRACT(b.payload_json, '$.summary') IS NOT NULL) + ORDER BY b.ord LIMIT 1) AS firstTopic, + (SELECT JSON_UNQUOTE(JSON_EXTRACT(b.payload_json, '$.summary')) FROM blocks b + WHERE b.session_id = s.session_id + AND (JSON_EXTRACT(b.payload_json, '$.topic') IS NOT NULL + OR JSON_EXTRACT(b.payload_json, '$.summary') IS NOT NULL) + ORDER BY b.ord LIMIT 1) AS firstSummary, + (SELECT COALESCE(SUM(CAST(JSON_EXTRACT(b.payload_json, '$.compressedTokens') AS DOUBLE)), 0) + FROM blocks b WHERE b.session_id = s.session_id AND b.active = 1) AS activeCompressed + FROM sessions s`, + )) as unknown as Row[][]; + return rows.map((r) => ({ + sessionId: str(r.sessionId), + title: r.title !== null && r.title !== undefined ? str(r.title) : undefined, + hostTitle: r.hostTitle !== null && r.hostTitle !== undefined ? str(r.hostTitle) : undefined, + label: r.label !== null && r.label !== undefined ? str(r.label) : undefined, + protocol: r.protocol !== null && r.protocol !== undefined ? str(r.protocol) : undefined, + origin: r.origin !== null && r.origin !== undefined ? str(r.origin) : undefined, + updatedAt: num(r.updatedAt), + statsJson: str(r.statsJson), + metadataJson: str(r.metadataJson), + blocks: num(r.blocks), + firstTopic: nullableStr(r.firstTopic), + firstSummary: nullableStr(r.firstSummary), + activeCompressedTokens: num(r.activeCompressed), + })); + } + + async verifyProjections(sessionId: string): Promise<{ ok: true; checks: number } | { ok: false; error: string }> { + const fail = (where: string, msg: string): { ok: false; error: string } => ({ + ok: false, + error: `${sessionId}: ${where}: ${msg}`, + }); + let checks = 0; + const s = await this.getSessionRow(sessionId); + if (!s) return fail("sessions", "row missing"); + const meta = JSON.parse(s.metaJson) as Record; + checks++; + if ((s.protocol ?? null) !== (meta.protocol ?? null)) return fail("sessions.protocol", `${s.protocol} vs ${String(meta.protocol)}`); + if ((s.label ?? null) !== (meta.label ?? null)) return fail("sessions.label", `${s.label} vs ${String(meta.label)}`); + if ((s.title ?? null) !== (meta.title ?? null)) return fail("sessions.title", `${s.title} vs ${String(meta.title)}`); + if ((s.hostTitle ?? null) !== (meta.hostTitle ?? null)) return fail("sessions.hostTitle", `${s.hostTitle} vs ${String(meta.hostTitle)}`); + const kstate = JSON.parse(s.kstateJson) as Record; + checks++; + if (kstate.nextBlockId !== undefined || kstate.nextRunId !== undefined) { + return fail("sessions.kstate_json", "must not carry nextBlockId/nextRunId (columns own them)"); + } + const messages = await this.getMessages(sessionId); + const refByRaw = new Map((await this.getRefs(sessionId)).map((r) => [r.rawId, r.ref])); + for (const m of messages) { + checks++; + if (m.ref !== (refByRaw.get(m.msgId) ?? null)) { + return fail(`messages[${m.seq}].ref`, `${String(m.ref)} vs refs ledger ${String(refByRaw.get(m.msgId))}`); + } + } + for (const m of messages) { + checks++; + const parsed = JSON.parse(m.content) as { id?: string; role?: string; contentType?: string }; + if (parsed.id !== m.msgId) return fail(`messages[${m.seq}].msg_id`, `${m.msgId} vs ${String(parsed.id)}`); + if (parsed.role !== m.role) return fail(`messages[${m.seq}].role`, `${m.role} vs ${String(parsed.role)}`); + if ((parsed.contentType ?? "message") !== m.kind) { + return fail(`messages[${m.seq}].kind`, `${m.kind} vs ${String(parsed.contentType)}`); + } + } + const blocks = await this.getBlocks(sessionId); + for (const b of blocks) { + checks++; + const payload = JSON.parse(b.payloadJson) as Record; + if (payload.blockId !== b.blockId) return fail(`blocks[${b.blockId}].blockId`, "id mismatch"); + if (payload.tier !== b.tier) return fail(`blocks[${b.blockId}].tier`, `${b.tier} vs ${String(payload.tier)}`); + if ((payload.active ?? true) !== b.active) { + return fail(`blocks[${b.blockId}].active`, `${b.active} vs ${String(payload.active)}`); + } + } + return { ok: true, checks }; + } + + async counts(): Promise> { + const out: Record = {}; + for (const t of ["sessions", "messages", "refs", "blocks", "dead_refs", "ccr_entries", "content", "checkpoints"]) { + const [rows] = (await this.pool.query(`SELECT COUNT(*) AS n FROM ${t}`)) as unknown as Row[][]; + out[t] = num(rows[0].n); + } + return out; + } + + async contentDedupReport(): Promise<{ rows: number; bytesTotal: number; refcounted: number }> { + const [rows] = (await this.pool.query( + "SELECT COUNT(*) AS n, COALESCE(SUM(byte_len),0) AS b, COALESCE(SUM(refcount),0) AS r FROM content", + )) as unknown as Row[][]; + return { rows: num(rows[0].n), bytesTotal: num(rows[0].b), refcounted: num(rows[0].r) }; + } +} diff --git a/tests/storage-backend-contract.test.ts b/tests/storage-backend-contract.test.ts index abf138e36f..ba36a1a71b 100644 --- a/tests/storage-backend-contract.test.ts +++ b/tests/storage-backend-contract.test.ts @@ -7,7 +7,28 @@ import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; // The contract suite deliberately imports NOTHING from store.ts: every // assertion goes through the StorageBackend interface only. When the mysql // backend lands (#3388 step 2) this same file runs against it unchanged. -import { openBackend, listBackendNames } from "../src/storage/backend.ts"; +import { openBackend, listBackendNames, type StorageBackend } from "../src/storage/backend.ts"; + +const MYSQL_DSN = process.env.BILI_MYSQL_TEST_DSN ?? ""; + +/** The suite runs against every backend it can reach: sqlite always, mysql + * when BILI_MYSQL_TEST_DSN is exported (CI runs a mysql:8.0 service). The + * mysql backend needs init() (DDL + schema stamp) once per fresh database. */ +const lanes: { name: string; open: (scratchDir: string) => Promise }[] = [ + { name: "sqlite", open: async (dir) => openBackend("sqlite", path.join(dir, "index.db")) }, +]; +if (MYSQL_DSN.length > 0) { + lanes.push({ + name: "mysql", + open: async () => { + const be = await openBackend("mysql", MYSQL_DSN); + const { MysqlBackend } = await import("../src/storage/mysql.ts"); + assert.ok(be instanceof MysqlBackend, "mysql lane opens the real backend"); + await be.init(); + return be; + }, + }); +} /** * #3931 (step 1 of the #3388 pluggable-storage plan): the operation-level @@ -50,7 +71,7 @@ test("registry: sqlite resolves through the registry; names are advertised", asy assert.ok(names.includes("mysql"), "mysql is registered as a step-2 stub so config validation recognizes it"); const dir = scratch(); try { - const backend = openBackend("sqlite", path.join(dir, "index.db")); + const backend = await openBackend("sqlite", path.join(dir, "index.db")); await backend.close(); } finally { fs.rmSync(dir, { recursive: true, force: true }); @@ -60,7 +81,7 @@ test("registry: sqlite resolves through the registry; names are advertised", asy test("contract: ingest → reads round-trip through the interface", async () => { const dir = scratch(); try { - const backend = openBackend("sqlite", path.join(dir, "index.db")); + const backend = await openBackend("sqlite", path.join(dir, "index.db")); try { const msgs = [mkMsg("m1", "hello " + "x".repeat(200)), mkMsg("m2", "world " + "y".repeat(200))]; await backend.ingestSession(planIngest(mkSession("sess-contract", msgs))); @@ -92,7 +113,7 @@ test("contract: ingest → reads round-trip through the interface", async () => test("contract: session_ext + watermark + tailSeqs", async () => { const dir = scratch(); try { - const backend = openBackend("sqlite", path.join(dir, "index.db")); + const backend = await openBackend("sqlite", path.join(dir, "index.db")); try { await backend.ingestSession(planIngest(mkSession("sess-ext", [mkMsg("m1", "payload".repeat(60))]))); await backend.setSessionExt("sess-ext", "inc:canonical-ptrs", [0, 1]); @@ -121,7 +142,7 @@ test("contract: session_ext + watermark + tailSeqs", async () => { test("contract: reap journal orphans through the interface (#3841 semantics)", async () => { const dir = scratch(); try { - const backend = openBackend("sqlite", path.join(dir, "index.db")); + const backend = await openBackend("sqlite", path.join(dir, "index.db")); try { await backend.ingestSession(planIngest(mkSession("sess-reap", [mkMsg("m1", "keep".repeat(80))]))); const orphans = new Map([["b2", { one: null, full: { text: "orphaned summary", count: 0 } }]]); @@ -140,7 +161,7 @@ test("contract: archive → restore round-trip + gc counters", async () => { const dir = scratch(); const archiveDir = path.join(dir, "archive"); try { - const backend = openBackend("sqlite", path.join(dir, "index.db")); + const backend = await openBackend("sqlite", path.join(dir, "index.db")); try { await backend.ingestSession(planIngest(mkSession("sess-gc", [mkMsg("m1", "gc target".repeat(120))]))); const archived = await backend.archiveSession("sess-gc", { archiveDir }); @@ -165,7 +186,7 @@ test("contract: archive → restore round-trip + gc counters", async () => { test("contract: conversation lookup + delete releases content refs", async () => { const dir = scratch(); try { - const backend = openBackend("sqlite", path.join(dir, "index.db")); + const backend = await openBackend("sqlite", path.join(dir, "index.db")); try { const shared = mkMsg("m-shared", "shared body ".repeat(50)); await backend.ingestSession(planIngest(mkSession("sess-del-a", [shared]))); @@ -183,3 +204,89 @@ test("contract: conversation lookup + delete releases content refs", async () => fs.rmSync(dir, { recursive: true, force: true }); } }); + +// --------------------------------------------------------------------------- +// MySQL lane (#3388 step 2): the same contract surface against MySQL 8.0. +// Runs only when BILI_MYSQL_TEST_DSN is exported (CI starts a mysql:8.0 +// service; locally `docker run -e MYSQL_ROOT_PASSWORD=… -p 3307:3306 mysql:8.0`). +// Session ids are suffixed so the shared database stays isolated per run. +// --------------------------------------------------------------------------- + +test("mysql lane: contract surface against MySQL 8.0", { skip: MYSQL_DSN.length === 0 ? "set BILI_MYSQL_TEST_DSN to run" : false }, async (t) => { + const backend = await openBackend("mysql", MYSQL_DSN); + await backend.init?.(); + t.after(async () => { + await backend.close(); + }); + const run = String(Date.now()); + + // ingest → reads round-trip + const sid = `sess-mysql-${run}`; + const msgs = [mkMsg("m1", "hello " + "x".repeat(200)), mkMsg("m2", "world " + "y".repeat(200))]; + assert.ok(await backend.ingestSession(planIngest(mkSession(sid, msgs))), "ingest ok"); + const row = await backend.getSessionRow(sid); + assert.ok(row && row.sessionId === sid && row.label === `label-${sid}`, "session row + label"); + const rows = await backend.getMessages(sid); + assert.equal(rows.length, 2, "both messages landed"); + assert.equal((JSON.parse(rows[1]!.content) as { id?: string }).id, "m2", "verbatim core JSON"); + assert.ok(typeof rows[1]!.contentHash === "string" && rows[1]!.contentHash.length > 0, "content hashes ride the rows"); + const counters = await backend.sessionCounters(sid); + assert.ok(counters && counters.nextBlockId >= 1, "counters readable"); + assert.ok((await backend.listSessionIds()).includes(sid), "enumerated"); + + // ext / watermark / tailSeqs + await backend.setSessionExt(sid, "inc:canonical-ptrs", [0, 1]); + assert.deepEqual(await backend.getSessionExt(sid, "inc:canonical-ptrs"), [0, 1]); + const wm = await backend.ingestWatermark(sid); + assert.ok(wm && typeof wm.seq === "number", "watermark readable"); + assert.ok(Array.isArray(await backend.tailSeqs(sid)), "tailSeqs readable"); + await backend.clearIngestWatermark(sid); + assert.equal(await backend.ingestWatermark(sid), null); + + // re-ingest the same identity is stable (seq preservation via remap) + assert.ok(await backend.ingestSession(planIngest(mkSession(sid, msgs))), "re-ingest ok"); + const rows2 = await backend.getMessages(sid); + assert.equal(rows2.length, 2, "no duplicate rows on identical re-ingest"); + + // reap journal + await backend.reapBlocks(sid, new Set(["b2"]), new Map([["b2", { one: null, full: { text: "orphaned", count: 0 } }]])); + await backend.reapBlocks(sid, new Set(["b2"]), new Map(), ); + + // archive → restore + gc + const archiveDir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-backend-mysql-archive-")); + try { + const archived = await backend.archiveSession(sid, { archiveDir }); + assert.ok(archived.ok && archived.exportPath, `archive ok (${archived.reason ?? ""})`); + assert.ok(fs.existsSync(archived.exportPath!), "export landed on disk (local JSONL even for mysql)"); + assert.equal((await backend.getSessionRow(sid))!.status, "archived", "tombstone keeps the row"); + const restored = await backend.restoreArchived(sid, { archiveDir }); + assert.ok(restored.ok, `restore ok (${restored.reason ?? ""})`); + assert.equal((await backend.getSessionRow(sid))!.status, "active", "restore reactivates"); + const gc = await backend.gc({ archiveAfterDays: 365, archiveDir, now: 9_000_000_000_000 }); + assert.ok(gc.archived >= 0 && gc.contentRowsDeleted >= 0, "gc returns counters"); + } finally { + fs.rmSync(archiveDir, { recursive: true, force: true }); + } + + // conversation lookup + refcounted delete + const sharedA = `sess-mysql-del-a-${run}`; + const sharedB = `sess-mysql-del-b-${run}`; + const shared = mkMsg("m-shared", "shared body ".repeat(50)); + assert.ok(await backend.ingestSession(planIngest(mkSession(sharedA, [shared])))); + assert.ok(await backend.ingestSession(planIngest(mkSession(sharedB, [shared])))); + const before = await backend.contentRefCountOfBody("shared body "); + assert.ok(before >= 2, "shared content refcounted across sessions"); + await backend.deleteSession(sharedA); + const after = await backend.contentRefCountOfBody("shared body "); + assert.equal(after, before - 1, "delete releases exactly this session's refs"); + assert.equal(await backend.getSessionRow(sharedA), null, "row gone"); + await backend.deleteSession(sharedB); + await backend.deleteSession(sid); + + // verifyProjections on a fresh session + const vSid = `sess-mysql-verify-${run}`; + assert.ok(await backend.ingestSession(planIngest(mkSession(vSid, [mkMsg("m1", "verify ".repeat(40))])))); + const verified = await backend.verifyProjections(vSid); + assert.ok(verified.ok, `projections verified (${verified.ok ? verified.checks : verified.error})`); + await backend.deleteSession(vSid); +}); diff --git a/tests/storage-backend-knob.test.ts b/tests/storage-backend-knob.test.ts index 971c1478d8..92f8bbe106 100644 --- a/tests/storage-backend-knob.test.ts +++ b/tests/storage-backend-knob.test.ts @@ -3,19 +3,22 @@ import assert from "node:assert/strict"; import * as fs from "node:fs"; import * as os from "node:os"; import * as path from "node:path"; -import { resolveBackendName, backendSelectionNotice, selectedBackendRaw } from "../src/storage/backend.ts"; -import { openBackend } from "../src/storage/backend.ts"; +import { resolveBackendName, backendSelectionNotice, selectedBackendRaw, resolveMysqlDsn, openBackend } from "../src/storage/backend.ts"; + /** * #3931 (step 1): the `storage.backend` selection surface. * * env BILI_STORAGE_BACKEND > file storage.backend > "sqlite" * - * `mysql` is a REGISTERED STUB — the name validates (so configs don't bounce - * between "unknown backend" and "unimplemented backend" as step 2 lands), - * but opening it fails with one documented, doctor-readable line. Step 1 - * never loads mysql2; the selection notice keeps the main store on sqlite - * with a single warn line instead of crashing boots. + * `mysql` is IMPLEMENTED (#3388 step 2): openBackend("mysql", dsn) opens the + * real MysqlBackend (lazily importing mysql2 — boots that never select it + * never load it). The DSN precedence mirrors the backend name: + * + * env BILI_MYSQL_DSN > file storage.mysql.dsn > "" (documented error) + * + * The selection notice still keeps the proxy hot path on sqlite — the + * shadow ingest rides the step-3 async swap instead of crashing boots. */ test("resolveBackendName: env > file > default sqlite", () => { @@ -36,18 +39,18 @@ test("backendSelectionNotice: silent on sqlite, explicit everywhere else", () => assert.ok(backendSelectionNotice("oracle")!.includes("oracle"), "unknown names also produce a notice"); }); -test("openBackend: sqlite opens; mysql throws the documented step-2 error; unknown lists valid names", () => { +test("openBackend: sqlite opens; mysql needs a DSN; unknown lists valid names", async () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-backend-knob-")); try { - const b = openBackend("sqlite", path.join(dir, "index.db")); - b.close(); + const b = await openBackend("sqlite", path.join(dir, "index.db")); + await b.close(); - assert.throws(() => openBackend("mysql", path.join(dir, "index.db")), (err: unknown) => { + await assert.rejects(() => openBackend("mysql", ""), (err: unknown) => { const msg = String((err as Error).message); - return msg.includes("mysql") && msg.includes("#3388") && msg.includes("sqlite"); - }, "mysql stub error names the backend, the issue, and the working default"); + return msg.includes("mysql") && msg.includes("DSN") && msg.includes("BILI_MYSQL_DSN"); + }, "mysql without a DSN names both knobs and the working default"); - assert.throws(() => openBackend("oracle", path.join(dir, "index.db")), (err: unknown) => { + await assert.rejects(() => openBackend("oracle", path.join(dir, "index.db")), (err: unknown) => { const msg = String((err as Error).message); return msg.includes("oracle") && msg.includes("sqlite") && msg.includes("mysql"); }, "unknown backend error lists the valid names"); @@ -64,3 +67,10 @@ test("lane-boot notice composition: the RAW selection decides — typo'd names a assert.equal(backendSelectionNotice(selectedBackendRaw(undefined, undefined)), undefined, "no selection stays silent"); assert.equal(backendSelectionNotice(selectedBackendRaw({ BILI_STORAGE_BACKEND: " SQLITE " }, "mysql")), undefined, "env wins over file; normalized sqlite stays silent"); }); + +test("resolveMysqlDsn: env > file > empty; whitespace trims", () => { + assert.equal(resolveMysqlDsn({ BILI_MYSQL_DSN: "mysql://e/db" }, "mysql://f/db"), "mysql://e/db", "env wins"); + assert.equal(resolveMysqlDsn(undefined, "mysql://f/db"), "mysql://f/db", "file fills in"); + assert.equal(resolveMysqlDsn({ BILI_MYSQL_DSN: " mysql://e/db " }, undefined), "mysql://e/db", "env trims"); + assert.equal(resolveMysqlDsn(undefined, undefined), "", "empty when neither is set"); +}); diff --git a/website/config-reference/env-only.yaml b/website/config-reference/env-only.yaml index 895332aa7f..ebbe693037 100644 --- a/website/config-reference/env-only.yaml +++ b/website/config-reference/env-only.yaml @@ -16,8 +16,16 @@ entries: kind: env-only audience: user group: envonly - desc_en: "Env tier for storage.backend (#3931 step 1): sqlite (default, only real backend) or mysql (registered stub - one boot notice, stays on sqlite until step 2). Wins over the file tier." - desc_zh: "storage.backend 的环境变量层(#3931 第一步):sqlite(默认,唯一真实现)或 mysql(注册占位 —— 启动提示一行,第二步落地前继续用 sqlite)。优先于文件层。" + desc_en: "Env tier for storage.backend (#3931): sqlite (default) or mysql (implemented, #3388 step 2 - openBackend speaks the full contract against MySQL 8.0; export works today, the proxy hot path swaps in step 3). Wins over the file tier." + desc_zh: "storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已实现,#3388 第二步 —— openBackend 对 MySQL 8.0 说完整契约;export 今天可用,代理热路径第三步接入)。优先于文件层。" + - key: BILI_MYSQL_DSN + type: string + default: "unset" + kind: env-only + audience: user + group: envonly + desc_en: "MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend=\"mysql\" (#3388 step 2, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs." + desc_zh: "storage.backend=\"mysql\" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388 第二步,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。" - key: BILI_STORAGE_ARCHIVE_AFTER_DAYS type: integer default: "365" diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 8c004ae2ef..e21b7f75ed 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -257,8 +257,15 @@ entries: default: "unset (sqlite)" env: BILI_STORAGE_BACKEND group: process - desc_en: "Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot." - desc_zh: "存储后端选择,可插拔存储计划第一步(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约,sqlite 为默认且是当前唯一真实现(现有 UnifiedStore 套一层 Promise 包装适配器 —— 零行为变化、零 schema 变化)。mysql 是已注册占位:该名字能通过校验(真实后端随第二步落地 —— mysql2 客户端,走可选依赖+自下载纪律 —— 期间配置不会在 unknown/unimplemented 之间来回弹),但选中它只会在 lane 启动时打一行提示并继续用 sqlite;直接开后端的路径(export)按文档化的错误失败。未知名字同样回退 sqlite 并提示。每个 lane 启动时解析一次。" + desc_en: "Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is IMPLEMENTED (step 2): openBackend(\"mysql\", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path (shadow ingest) stays on sqlite until the step-3 async swap — the boot notice names that boundary; export speaks mysql today (storage.backend=\"mysql\" + a DSN exports straight from MySQL). Unknown names fall back to sqlite with a notice. Resolved once per lane boot." + desc_zh: "存储后端选择(#3388 / #3931):统一存储消费操作级的异步 StorageBackend 契约。sqlite 为默认(现有 UnifiedStore 套一层 Promise 包装适配器)。mysql 已实现(第二步):openBackend(\"mysql\", dsn) 打开真正的 MysqlBackend,对接 MySQL 8.0 —— 完整契约面(带 CAS 引用计数的 ingest、读取、session_ext、收割台账、归档/恢复/GC、checkpoint、会话绑定),mysql2 作为常规依赖打包并按需懒加载(从不选 mysql 的启动永不加载它)。代理热路径(shadow ingest)在第三步异步化改造前仍走 sqlite —— 启动提示会写明这一边界;export 今天就能说 mysql(storage.backend=\"mysql\" + DSN 直接从 MySQL 导出)。未知名字回退 sqlite 并提示。每个 lane 启动时解析一次。" + - key: storage.mysql.dsn + type: string + default: "unset" + env: BILI_MYSQL_DSN + group: process + desc_en: "MySQL DSN for storage.backend=\"mysql\" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (export today; the proxy hot path swaps in step 3). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default." + desc_zh: "storage.backend=\"mysql\" 的 MySQL DSN(#3388 第二步,#3931):mysql://user:pass@host:3306/db。由 StorageBackend 契约路径消费(今天为 export;代理热路径第三步接入)。环境变量 BILI_MYSQL_DSN 优先于文件键。空/未设置 = mysql 后端无法打开 —— 文档化错误会同时列出两个旋钮和可用的默认值。" - key: storage.sqliteAutoDownload type: boolean default: "true"