Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -729,6 +729,7 @@ bili sessions restore <id> # bring an archived session back from the log l
bili sessions foreign-blocks # (#5312) list fold blocks whose covered messages never belonged to this transcript — dry-run; --apply reversibly archives them, --restore <sid>:<bid> brings one back
bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]
# print the sweep plan (sessions past retention, zero-ref content rows, ~pages reclaimable), then run the same gc() the HTTP endpoint calls; --dry-run stops after the plan
bili storage migrate [--dry-run] # (#5599) one-shot bulk import of every legacy JSON session into the unified SQLite index with per-session digest reconciliation (imported / indexed / drifted / failed); read-only over the JSON files — archive/delete them yourself once the report is clean; --dry-run plans without writing
bili storage backup [--keep N] # online-consistent index.db snapshot into <sessions>/backups/ (VACUUM INTO —
# never rewrites the source, no exclusive lock, safe while a proxy runs); the
# snapshot is quick_check-verified before it counts; keeps the newest N (default 3)
Expand All @@ -740,7 +741,7 @@ bili storage repair [--from f.db] # with --from: verify the backup (quick_check
# survives). Both swap paths refuse to run while another live process holds index.db.lock.
```

`archive`/`restore`/`gc` require the unified store (`BILI_STORAGE_UNIFIED=sqlite|shadow|full` or `storage.unified` in the config file — the default is now pure sqlite) — legacy-only installs refuse with a hint. `--days` defaults to `storage.archiveAfterDays` (365). No scheduler: these run on demand. All commands accept `--json`.
`archive`/`restore`/`gc`/`migrate` require the unified store (`BILI_STORAGE_UNIFIED=sqlite|shadow|full` or `storage.unified` in the config file — the default is now pure sqlite) — legacy-only installs refuse with a hint. `--days` defaults to `storage.archiveAfterDays` (365). No scheduler: these run on demand. All commands accept `--json`.

**Backup & repair (#7156, 2026-11-20 production incident).** A 3.2 GB index.db proved corrupt (foreign second writer, #7156) and the only recovery path was a hand-rolled `.recover`-equivalent plus manual file surgery — those two manual moves are now the `backup`/`repair` commands above. Operational discipline that keeps the incident from recurring: run `bili storage backup` on a schedule (the retention flag is the only state it keeps), and keep **exactly one** long-lived bili process per sessions dir — a second live writer is the corruption mode, and every storage command now detects a live foreign writer and refuses rather than contending.

Expand Down
4 changes: 3 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -503,11 +503,13 @@ bili sessions ls # id、label、解码后大小、最后更新
bili sessions rm <id> # 完整删除(释放内容引用计数,含媒体);随后收缩 index.db(--no-vacuum 跳过)
bili sessions archive <id> # 导出到 archive/archive.jsonl + 清行;tombstone 保留 label 与身份
bili sessions restore <id> # 从日志行恢复已归档的会话
bili sessions foreign-blocks # (#5312) 列出折叠块覆盖的消息从未属于本会话的记录 —— dry-run;--apply 可逆地归档它们,--restore <sid>:<bid> 找回其中一条
bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]
# 先打印清扫计划(超期会话、零引用内容行、可回收页数估算),再执行与 HTTP 端点相同的 gc();--dry-run 只打印计划
bili storage migrate [--dry-run] # (#5599) 一次性把全部 legacy JSON 会话批量导入统一 SQLite 索引,逐会话做摘要对账(imported / indexed / drifted / failed);对 JSON 文件只读 —— 报告干净后归档/删除由你自己决定;--dry-run 只打印计划不写入
```

`archive`/`restore`/`gc` 需要统一存储(`BILI_STORAGE_UNIFIED=sqlite|shadow|full` 或配置文件里的 `storage.unified` —— 缺省现在就是纯 sqlite)—— legacy-only 安装会拒绝并给出提示。`--days` 默认取 `storage.archiveAfterDays`(365)。没有调度器:全部按需运行。所有命令都支持 `--json`。
`archive`/`restore`/`gc`/`migrate` 需要统一存储(`BILI_STORAGE_UNIFIED=sqlite|shadow|full` 或配置文件里的 `storage.unified` —— 缺省现在就是纯 sqlite)—— legacy-only 安装会拒绝并给出提示。`--days` 默认取 `storage.archiveAfterDays`(365)。没有调度器:全部按需运行。所有命令都支持 `--json`。

**大库上 full vacuum 是离线操作(#4612)。** `--vacuum full`(默认)会在独占写锁下重写整个 index.db —— O(库大小),多 GB 的库要几分钟到几十分钟,且没有中间进度(SQLite 的 VACUUM 是原子操作,拆不开)。在有代理正从同一会话目录服务时跑它,会把那个代理的写入饿死到其存储层降级为止,所以 `storage gc --vacuum full` **在另一个存活进程持有该数据库时会拒绝启动**(`index.db.lock` pid 纪律,#4175):CLI 以退出码 1 给出可操作提示,面板端点返回 HTTP 409。先停掉另一个实例(或等它退出)再重试;要在线安全地缩库用 `--vacuum incremental`,它以短事务回收空闲页、不拿独占锁。零引用删除清扫无论哪种模式都分批执行并逐批打印进度。

Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. The tarball download walks a builtin mirror list (#3338): the configured registry (BILI_UPDATE_REGISTRY / update.registry) is tried first, then https://registry.npmjs.org, then https://registry.npmmirror.com - the first mirror that yields a sha512-verified tarball wins; a Node < 22.5 with no engine gets an explicit Node upgrade hint in the degrade reason (#3385). Engine artifacts are pinned PER RUNTIME, because a binding cannot be loaded outside the range its package declares: Node >= 22 with N-API >= 10 uses better-sqlite3 v13 (one tarball that carries its own prebuilds), while Node 20 and the Node 22.0-22.12 band - which passes v13's major floor but only ships N-API 9 - use v12.9.0 (a JS-only tarball plus its ABI-keyed prebuild, fetched from the binary mirror and bound through nativeBinding). That keeps a low Node on the unified lane instead of degrading to legacy. A runtime no pinned artifact serves (below every floor, or an ABI no prebuild targets) is refused before any load, with neither the download nor a cached copy registered, since loading such a binding crashes the process instead of failing gracefully (#3341, #3894). A failed download arms a 6-hour retry cooldown persisted as a marker file under the cache dir (billion-context/sqlite/.bootstrap-failed.json); the storage diagnostics blob shows the marker's absolute expiry time and path - delete that file and restart bili to force an immediate retry instead of waiting out the whole window (#3788). |
| `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960) — default flipped this release: UNSET now defaults to OFF for the pure-sqlite selection (storage.unified absent or "sqlite") — the SQLite index is the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Explicit full keeps the historical #3818 dual-write default (envelopes stay written unless this key says otherwise). An explicit on/off at either tier always wins in effective-full mode: under shadow/legacy (or with encryption active) a requested off is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings ("off"/"on"). Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. To restore dual-write: set storage.jsonWriter=on (or BILI_STORAGE_JSON_WRITER=1) — index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session CCR content-store sidecar stops being written as well - in full mode the index's ccr mirror is consulted first on every read, so the mirror is the durable copy. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). |
| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. |
| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore <id> (README 'Storage maintenance CLI' section). Env wins over the file tier. |
| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore <id>, and `bili storage migrate [--dry-run]` (#5599: one-shot bulk import of every legacy JSON session into the unified index with per-session digest reconciliation — read-only over the JSON files; README 'Storage maintenance CLI' section). Env wins over the file tier. |
| `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. |
| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). |
| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. |
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@
| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 tarball 下载内置镜像列表兜底(#3338):先试配置的 registry(BILI_UPDATE_REGISTRY / update.registry),再试 https://registry.npmjs.org、https://registry.npmmirror.com —— 第一个通过 sha512 校验的镜像生效;Node < 22.5 且无引擎时,降级原因会给出明确的 Node 升级提示(#3385)。引擎产物按运行时特性分别钉死,因为 binding 无法在包自身声明的范围之外加载:Node >= 22 且 N-API >= 10 用 better-sqlite3 v13(单个 tarball 自带 prebuilds),而 Node 20 与 Node 22.0–22.12(满足 v13 的大版本下限但只带 N-API 9)用 v12.9.0(纯 JS tarball 加与之 ABI 匹配的 prebuild,从二进制镜像获取并经 nativeBinding 注入)—— 低版本 Node 因此留在统一存储车道上,而不是降级到 legacy。任何钉死产物都不支持的运行时(低于所有下限,或 ABI 无对应 prebuild)会在加载之前被拒绝,下载与缓存副本注册都不进行 —— 在那里加载该 binding 会让进程崩溃,而不是优雅失败(#3341、#3894)。下载失败会置一个 6 小时的重试冷却,以标记文件形式持久化在缓存目录下(billion-context/sqlite/.bootstrap-failed.json);storage 诊断块会显示该标记的绝对到期时间与路径 —— 删除该文件并重启 bili 即可强制立即重试,不必等完整个冷却窗口(#3788)。 |
| `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960)—— 本次发布翻转缺省:UNSET(未设置)现在对纯 SQLite 选择(storage.unified 未设置或 "sqlite")缺省为 OFF —— SQLite 索引成为唯一持久副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。显式 full 保留 #3818 的历史双写缺省(除非本键另说,信封照写)。有效 full 模式下任一层的显式 on/off 总是生效:shadow/legacy(或加密激活)下请求的 off 会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层("off"/"on")。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。恢复双写:设 storage.jsonWriter=on(或 BILI_STORAGE_JSON_WRITER=1)—— 纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 CCR content-store 边车也不再写 —— full 模式下读取时先查索引里的 ccr 镜像,镜像才是持久副本。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 |
| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 |
| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore <id>(README「存储维护 CLI」一节)。环境变量优先于配置文件层。 |
| `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore <id>,还有 `bili storage migrate [--dry-run]`(#5599:一次性把全部 legacy JSON 会话批量导入统一索引并逐会话摘要对账——对 JSON 文件只读;README「存储维护 CLI」一节)。环境变量优先于配置文件层。 |
| `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 |
| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 |
| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 |
Expand Down
Loading
Loading