Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,39 @@ jobs:
ACP_TEST_CLAUDE_NATIVE: "1"
CI_TEST_SHARD: "${{ matrix.shard }}/${{ matrix.shardTotal }}" # denominator carried per-include by the plan job (#3295)

# #3388 step 2: the MySQL contract lane. The storage-backend contract
# suite runs against a real mysql:8.0 service (BILI_MYSQL_TEST_DSN) in
# addition to the always-on sqlite lane inside the main test job. Cheap
# (two files, ~1min) so it rides the core scope — every PR proves the
# mysql backend still satisfies the operation contract.
mysql-contract:
needs: [plan]
if: ${{ !failure() && !cancelled() && needs.plan.outputs.scope != 'skip' }}
runs-on: ubuntu-latest
services:
mysql:
image: mysql:8.0
env:
MYSQL_ROOT_PASSWORD: bili-test-root
MYSQL_DATABASE: bili
ports:
- 3306:3306
options: >-
--health-cmd="mysqladmin ping -uroot -pbili-test-root"
--health-interval=10s
--health-timeout=5s
--health-retries=10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm install --ignore-scripts --no-fund --no-audit
- run: node --import tsx --test tests/storage-backend-contract.test.ts tests/storage-backend-knob.test.ts
env:
BILI_MYSQL_TEST_DSN: mysql://root:bili-test-root@127.0.0.1:3306/bili

# #2816: typecheck + the in-repo component suites + the bundle build used
# to run sequentially in front of the test job (~90s of critical path per
# leg). They are independent of the proxy test suites, so they run as
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ All notable changes to **billion-context** are documented here.
Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag.

## [Unreleased]

### 存储:MySQL 后端落地(#3388 第二步,#3931)

- `src/storage/mysql.ts`:MysqlBackend 完整实现 StorageBackend 操作契约(全 ~40 方法:带 CAS 引用计数的 ingest(delta/全量/completeLog)、读取、session_ext/kv、收割台账、checkpoint、会话绑定、归档/恢复/GC、诊断)—— MySQL 8.0 方言,mysql2 连接池(打进 dependencies,#3931 拍板:纯 JS ~500KB,锁文件钉版本),DDL 为 v7 形状直接建表(bili_meta 表存 schema 版本替 PRAGMA user_version;JSON 列一律 LONGTEXT 保字节序,MySQL JSON 类型会重排键序破坏 digest)。
- `openBackend` 改异步:mysql 分支懒加载 mysql2(不选 mysql 的启动永不加载该模块);DSN 解析 `env BILI_MYSQL_DSN > storage.mysql.dsn > 缺省报错`(文档化错误列全两个旋钮)。
- **边界(第三步待做)**:代理热路径(shadow ingest/persist.loadSync 同步契约)仍走 sqlite;今天 export 说 mysql(`storage.backend="mysql"` + DSN 直接从 MySQL 导出 JSONL);启动提示写明该边界。
- CI 新增 `mysql-contract` lane:每次 PR 用 mysql:8.0 service 容器跑同一份契约套件(sqlite lane 永远在,mysql lane 有 DSN 就跑)。
- 归档文件系统语义跨后端一致:mysql 后端的归档/恢复同样落本地 `archive/archive.jsonl`(单文件日志,#3927)。
- 存储后端契约第一步(#3931, #3388):统一存储抽出操作级异步 `StorageBackend` 契约(`src/storage/backend.ts`),sqlite 经注册表以默认后端身份重新接入(Promise 包装适配器,零行为/零 schema 变化);`mysql` 为已注册占位 —— 选中时启动提示一行并继续用 sqlite,直开后端的路径(export)按文档化错误失败。新配置 `storage.backend`(env `BILI_STORAGE_BACKEND`,缺省 sqlite)。`--jsonl` 导出渲染器改吃契约(接缝证真);契约测试套件对任意后端实例跑同一断言,第二步 mysql 落地时原样复用。

### 存储收尾九项一车(#3901,修复 #3463/#3762/#3841/#3798/#3633;#3723 落地)
Expand Down
7 changes: 5 additions & 2 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `persist.epermAlertRepeatMs` | number | 0 (no repeats) | BILI_PERSIST_EPERM_ALERT_REPEAT_MS | Repeat interval for the EPERM alert; 0 logs it once. |
| `storage` | object | {} (all defaults) | — | Unified SQLite storage sidecar settings (#2671). |
| `storage.unified` | legacy\|shadow\|full | unset (full) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to FULL (#3818 flip - the shadow dry-run reconciled cleanly since #2815, so the index is the load path; JSON envelopes keep being written, and legacy remains the one-line escape hatch). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. |
| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection, step 1 of the pluggable-storage plan (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract, with sqlite as the default and today the only real implementation (the existing UnifiedStore behind a Promise-wrapped adapter - zero behavior change, no schema change). mysql is a REGISTERED STUB: the name validates so configs do not bounce between unknown/unimplemented as the real backend ships (step 2, mysql2 client via the optional-dependency + self-download discipline), but selecting it logs one notice line at lane boot and the store stays on sqlite; paths that open a backend directly (export) fail with the documented error. Unknown names also fall back to sqlite with a notice. Resolved once per lane boot. |
| `storage.backend` | sqlite\|mysql | unset (sqlite) | BILI_STORAGE_BACKEND | Storage backend selection (#3388 / #3931): the unified store consumes an operation-level async StorageBackend contract. sqlite is the default (the existing UnifiedStore behind a Promise-wrapped adapter). mysql is IMPLEMENTED (step 2): openBackend("mysql", dsn) opens the real MysqlBackend against MySQL 8.0 — the full contract surface (ingest with CAS refcounts, reads, session_ext, reap journal, archive/restore/gc, checkpoints, conversations), with mysql2 bundled as a regular dependency and loaded lazily (boots that never select mysql never load it). The proxy hot path (shadow ingest) stays on sqlite until the step-3 async swap — the boot notice names that boundary; export speaks mysql today (storage.backend="mysql" + a DSN exports straight from MySQL). Unknown names fall back to sqlite with a notice. Resolved once per lane boot. |
| `storage.mysql.dsn` | string | unset | BILI_MYSQL_DSN | MySQL DSN for storage.backend="mysql" (#3388 step 2, #3931): mysql://user:pass@host:3306/db. Consumed by the StorageBackend contract paths (export today; the proxy hot path swaps in step 3). env BILI_MYSQL_DSN wins over the file key. Empty/unset = the mysql backend cannot open — the documented error names both knobs and the working default. |
| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). |
| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). |
| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. |
Expand Down Expand Up @@ -334,7 +335,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| Key | Type | Default | Env | Description |
|-----|------|---------|-----|-------------|
| `BILI_DSH_AUTHORED_COMPACTION` | boolean | true (on, #3901 flip) | — | Let bili ANSWER dsh compaction summarize calls locally with the folded snapshot instead of refusing (#3733): identity header + kernel-rendered block summaries, zero egress; replay recognized via the #2432 rebase. |
| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931 step 1): sqlite (default, only real backend) or mysql (registered stub - one boot notice, stays on sqlite until step 2). Wins over the file tier. |
| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931): sqlite (default) or mysql (implemented, #3388 step 2 - openBackend speaks the full contract against MySQL 8.0; export works today, the proxy hot path swaps in step 3). Wins over the file tier. |
| `BILI_MYSQL_DSN` | string | unset | — | MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend="mysql" (#3388 step 2, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs. |
| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (one JSON line appended to the single archive log (archive/archive.jsonl) + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. |
| `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): compute the outbound snapshot view on every armed turn WITHOUT using it, classify the diff against the legacy outbound, and surface per-session counters in the web panel + the snapshot-shadow diagnostics topic. 0/false/off disables; 1/true/on enables; otherwise the config-file snapshot.shadow boolean applies (default on). The render lane itself stays rollout-gated (BILI_ROLLOUT_SNAPSHOT_RENDER), so the shadow lane never changes outbound bytes - it is the dry-run evidence that flips the rollout later. |
| `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | Override the config file path (path relocation). |
Expand Down Expand Up @@ -1748,6 +1750,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes
| `BILI_MITM` | `mitm.enabled` | true |
| `BILI_MITM_DOMAINS` | `mitm.domains` | [] |
| `BILI_MITM_HANDSHAKE_TIMEOUT_MS` | `mitm.handshakeTimeoutMs` | 10000 |
| `BILI_MYSQL_DSN` | `storage.mysql.dsn` | unset |
| `BILI_NATIVE_ATTACH_EXTERNAL` | `native.attachExternal` | false |
| `BILI_NATIVE_ATTACH_POLICY` | `native.attachPolicy` | own-lane |
| `BILI_NON_HTTP_PROVIDERS` | `compactionOptIn` | false |
Expand Down
Loading
Loading