Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | Env tier for storage.backend (#3931): sqlite (default) or mysql (fully wired, #3388 steps 2+3 - saves ingest through the async shadow queue, sync reads ride the JSON envelope, replicas/ext/GC/fork via the async twins (#4911); export too). Wins over the file tier. |
| `BILI_MYSQL_DSN` | string | unset | — | MySQL DSN (mysql://user:pass@host:3306/db) for storage.backend="mysql" (#3388, #3931). Wins over the file key storage.mysql.dsn. Empty = the mysql backend cannot open; the documented error names both knobs. The pool opens lazily; closeQuietlyAsync drains the ingest queue and closes it. |
| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC age sweep window in days (#3463): sessions idle longer than this are archived (one JSON line appended to the single archive log (archive/archive.jsonl) + tombstone row + content/media refs released). 0 disables the age sweep; the manual entry POST /__bili/storage/gc always runs. Config-file key storage.archiveAfterDays is the persistent form. |
| `BILI_STORAGE_QUICK_CHECK` | string | on | — | Storage integrity self-check at open (#4175): every index.db open runs PRAGMA quick_check BEFORE migrations; if a corruption is PROVEN by THIS BOOT's own scan NOTHING is moved, copied or rebuilt automatically (auto-recovery was withdrawn #4609/#4626-era: with BILI_STORAGE_JSON_WRITER=off the index is the SOLE durable copy, so an automatic move or rebuild cannot guarantee full data recovery) and bili STARTS ANYWAY — serving from the JSON envelopes with a loud, actionable alarm naming the manual fix (back up then move index.db/-wal/-shm aside BY HAND, then start again — a fresh index rebuilds from the JSON envelopes). A successful PASS is cached so later starts skip the rescan; a FAIL verdict is NEVER persisted (2026-10-30 owner policy): runtime/background findings are alarms only (loud logs + runtime-status, the process keeps serving — JSON envelopes remain the recovery source) and every boot decides again from a fresh scan of the CURRENT file set, so a stale FAIL left behind by an older version can never block future boots. There is NO automatic rename/repair AND no legacy-only fallback: continuing legacy-only would fork the JSON envelopes from the dead index forever (never reconcilable). TRANSIENT open failures (no SQLite engine / CANTOPEN / second writer / readonly / backend DSN) make bili REFUSE TO START loudly with an actionable error — no silent legacy fallback for environment problems either (2026-10-30 owner policy); deployments that genuinely want JSON-only serve explicitly (BILI_STORAGE_UNIFIED=legacy or unset). Set 0 only for boot-time-sensitive huge deployments where the check's cost matters. |
| `BILI_STORAGE_QUICK_CHECK` | string | on | — | Storage integrity self-check at open (#4175): every index.db open runs PRAGMA quick_check BEFORE migrations; if a corruption is PROVEN by THIS BOOT's own scan NOTHING is moved, copied or rebuilt automatically (auto-recovery was withdrawn #4609/#4626-era: with BILI_STORAGE_JSON_WRITER=off the index is the SOLE durable copy, so an automatic move or rebuild cannot guarantee full data recovery) and bili STARTS ANYWAY — serving from the JSON envelopes with a loud, actionable alarm naming the manual fix (back up then move index.db/-wal/-shm aside BY HAND, then start again — a fresh index rebuilds from the JSON envelopes). A successful PASS is cached so later starts skip the rescan; a FAIL verdict is NEVER persisted (2026-10-30 owner policy): runtime/background findings are alarms only (loud logs + runtime-status, the process keeps serving — JSON envelopes remain the recovery source) and every boot decides again from a fresh scan of the CURRENT file set, so a stale FAIL left behind by an older version can never block future boots. There is NO automatic rename/repair AND no legacy-only fallback: continuing legacy-only would fork the JSON envelopes from the dead index forever (never reconcilable). TRANSIENT open failures (no SQLite engine / CANTOPEN / readonly / backend DSN) make bili REFUSE TO START loudly with an actionable error — no silent legacy fallback for environment problems either (2026-10-30 owner policy; a live second writer left this refusal list on 2026-10-31 — sharing one local-disk index.db is the supported topology, informational only, never blocking); deployments that genuinely want JSON-only serve explicitly (BILI_STORAGE_UNIFIED=legacy or unset). Set 0 only for boot-time-sensitive huge deployments where the check's cost matters. |
| `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): compute the outbound snapshot view on every armed turn WITHOUT using it, classify the diff against the legacy outbound, and surface per-session counters in the web panel + the snapshot-shadow diagnostics topic. 0/false/off disables; 1/true/on enables; otherwise the config-file snapshot.shadow boolean applies (default on). The render lane itself stays rollout-gated (BILI_ROLLOUT_SNAPSHOT_RENDER), so the shadow lane never changes outbound bytes - it is the dry-run evidence that flips the rollout later. |
| `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | Override the config file path (path relocation). |
| `BILI_SESSIONS_DIR` | string | unset (XDG state path) | — | Where session records are stored (path relocation). |
Expand Down
2 changes: 1 addition & 1 deletion CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@
| `BILI_STORAGE_BACKEND` | sqlite\|mysql | sqlite | — | storage.backend 的环境变量层(#3931):sqlite(默认)或 mysql(已接线,#3388 第二+三步 —— 保存经异步影子队列入 mysql,同步读走 JSON 信封,replicas/ext/GC/fork 走异步孪生(#4911);export 同样支持)。优先于文件层。 |
| `BILI_MYSQL_DSN` | string | unset | — | storage.backend="mysql" 的 MySQL DSN(mysql://user:pass@host:3306/db)(#3388,#3931)。优先于文件键 storage.mysql.dsn。留空 = mysql 后端无法打开;文档化错误会列出两个旋钮。连接池惰性打开;closeQuietlyAsync 排空入队事务并关闭。 |
| `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | integer | 365 | — | GC 年龄清扫窗口(天,#3463):闲置超过该时长的会话被归档(单文件归档日志 archive/archive.jsonl 追加一行 + 墓碑行保留标签 + 释放内容/媒体引用)。0 关闭年龄清扫;手动入口 POST /__bili/storage/gc 始终可用。配置文件键 storage.archiveAfterDays 为持久形式。 |
| `BILI_STORAGE_QUICK_CHECK` | string | on | — | 存储开库自检(#4175):每次 index.db 打开在迁移之前先跑 PRAGMA quick_check;一旦由本次启动自己的扫描证实损坏,文件原样保留在原地——不做任何自动搬移、复制或重建(自动恢复已撤回:BILI_STORAGE_JSON_WRITER=off 时索引是唯一的持久副本,自动操作无法保证完整数据恢复)——bili 照常启动,改用 JSON 信封服务,并给出响亮、可操作的告警指明手工修复步骤(备份后把 index.db/-wal/-shm 手工移开再启动——全新索引会从 JSON 信封重建)。PASS 判定会被缓存,之后的启动跳过重扫;FAIL 判定永不持久化(2026-10-30 owner 策略):运行期/后台发现只是告警(响亮日志 + runtime-status,进程继续服务——JSON 信封始终是恢复来源),每次启动都对当前文件集做新鲜扫描再判定,旧版本遗留的陈旧 FAIL 永远不能阻断后续启动。既没有自动改名/修复,也没有 legacy-only 兜底(继续以 legacy-only 运行会让 JSON 信封与死掉的索引永久分叉、再无对账可能)。瞬态打开失败(无引擎 / CANTOPEN / 第二写者 / 只读 / 后端 DSN)仍响亮地拒绝启动并给出可操作的错误——环境类问题同样不允许静默走 legacy 兜底;真正需要纯 JSON 服务的部署请显式设置 BILI_STORAGE_UNIFIED=legacy(或留空)。仅当超大部署对启动耗时敏感、检测成本构成问题时才设 0 跳过。 |
| `BILI_STORAGE_QUICK_CHECK` | string | on | — | 存储开库自检(#4175):每次 index.db 打开在迁移之前先跑 PRAGMA quick_check;一旦由本次启动自己的扫描证实损坏,文件原样保留在原地——不做任何自动搬移、复制或重建(自动恢复已撤回:BILI_STORAGE_JSON_WRITER=off 时索引是唯一的持久副本,自动操作无法保证完整数据恢复)——bili 照常启动,改用 JSON 信封服务,并给出响亮、可操作的告警指明手工修复步骤(备份后把 index.db/-wal/-shm 手工移开再启动——全新索引会从 JSON 信封重建)。PASS 判定会被缓存,之后的启动跳过重扫;FAIL 判定永不持久化(2026-10-30 owner 策略):运行期/后台发现只是告警(响亮日志 + runtime-status,进程继续服务——JSON 信封始终是恢复来源),每次启动都对当前文件集做新鲜扫描再判定,旧版本遗留的陈旧 FAIL 永远不能阻断后续启动。既没有自动改名/修复,也没有 legacy-only 兜底(继续以 legacy-only 运行会让 JSON 信封与死掉的索引永久分叉、再无对账可能)。瞬态打开失败(无引擎 / CANTOPEN / 只读 / 后端 DSN)仍响亮地拒绝启动并给出可操作的错误——环境类问题同样不允许静默走 legacy 兜底(活的第二写者已于 2026-10-31 移出该拒绝清单:多进程共享一个本地盘 index.db 是受支持拓扑,只提示、永不阻断启动);真正需要纯 JSON 服务的部署请显式设置 BILI_STORAGE_UNIFIED=legacy(或留空)。仅当超大部署对启动耗时敏感、检测成本构成问题时才设 0 跳过。 |
| `BILI_SNAPSHOT_SHADOW` | boolean | true (on) | — | 快照渲染影子对账(#3756 后续):每轮已折叠会话都用引擎现算一次出站快照视图但不使用,与 legacy 实际出站比对分类,逐会话计数进 web 面板与 snapshot-shadow 诊断主题。0/false/off 关闭;1/true/on 开启;否则读配置文件 snapshot.shadow 布尔(默认开)。渲染车道本身仍受灰度开关控制(BILI_ROLLOUT_SNAPSHOT_RENDER),影子车道绝不改变出站字节 —— 它是后续翻转灰度的干跑证据。 |
| `BILI_CONFIG_FILE` | string | unset (XDG config path) | — | 覆盖配置文件路径(路径重定位)。 |
| `BILI_SESSIONS_DIR` | string | unset (XDG state path) | — | 会话记录的存放目录(路径重定位)。 |
Expand Down
28 changes: 16 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -645,20 +645,24 @@ Consequences for anyone tempted to put several instances behind one Service:
single-instance by design.

**Single machine, multiple installs (#5046).** The topology above covers
deployment replicas; one desktop has a sibling hazard: two installs on ONE
deployment replicas; one desktop has a sibling situation: two installs on ONE
machine (e.g. two DSH profiles — desktop + bot, or a global npm copy beside a
host-tree copy) each start their own proxy and, unless isolated, both resolve
the SAME default XDG sessions directory and the same config file. With
`storage.unified` on (shadow/full), two processes then double-write one
`index.db`: the log shows `[instances] another bili instance is running …`
and, for overlapping lanes, `[storage] … two processes on one index.db
corrupt it`; SQLite write-lock contention (`SQLITE_BUSY` / `database is
locked`) can fail whole model turns, and sustained double-writing risks
corruption. The fix is isolation: give each install its own
`BILI_SESSIONS_DIR` (plus `BILI_CONFIG_FILE` when the config should differ
too) through that install's launcher/profile environment — no global env
changes needed. Once isolated, the installs coexist safely and the
`[instances]` notice drops to info level (different dirs ⇒ no shared state).
the SAME default XDG sessions directory and the same config file. Multiple
processes writing ONE local-disk `index.db` is the SUPPORTED topology
(#5082/#5091) — SQLite WAL is multi-process safe on a local filesystem — so
startup NEVER blocks on a live foreign writer: the peer shows up as an
info-level `[storage] process pid … is also using …` line plus the status wire,
nothing more (policy settled 2026-10-31, withdrawing the #5046/B-era
refuse-to-start for exactly this shape). What genuinely remains hazardous is
two installs walking the SAME *session* concurrently — that case alarms
separately (split-brain contest, #3923). If you still want strictly separate
histories and stores per install, give each its own `BILI_SESSIONS_DIR` (plus
`BILI_CONFIG_FILE` when the config should differ too) through that install's
launcher/profile environment — no global env changes needed. Once isolated,
the `[instances]` notice drops off entirely. Note the distinction from the
deployment case above: local disk with SQLite WAL tolerates many writers;
shared RWX/NFS storage does NOT and keeps requiring one writer per volume.

Naming note: `POST /__bili/replicas` creates a session **checkpoint fork**
(from stored checkpoints), not a pod replica — read it as `/__bili/checkpoints`.
Expand Down
17 changes: 5 additions & 12 deletions src/storage-maintenance.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
import { existsSync, statSync } from "node:fs";
import { join } from "node:path";
import { SessionStore } from "./persist.js";
import { StorageSecondWriterError } from "./storage/store.js";
import type { GcPlan } from "./storage/store.js";
import { fileStorageArchiveAfterDays, resolveArchiveAfterDays } from "./knobs.js";

Expand Down Expand Up @@ -278,17 +277,11 @@ export async function runStorageGcCommand(opts: {
try {
store = new SessionStore({ dir: opts.dir, enabled: true });
} catch (err) {
// #4612: under the no-degradation policy a live foreign writer makes the
// unified open REFUSE (constructor throws) before runGc's own guard can
// answer. Surface the same actionable refusal the in-process guard
// produces instead of leaking a stack trace through the CLI.
if (err instanceof StorageSecondWriterError) {
const reason = `refused: another process (pid ${err.pid}) is actively writing index.db — 'storage gc --vacuum full' rewrites the whole database under an exclusive write lock and would starve the other instance's writes until its storage layer degrades. Stop the other process (or wait for it to exit) and retry, or use --vacuum incremental for an online-safe shrink.`;
if (opts.dryRun) return { code: 0, text: `${reason}\ndry-run: nothing executed\n` };
const data: Record<string, unknown> = { refused: true, writerPid: err.pid, vacuum: opts.vacuum ?? "full", dryRun: false };
if (opts.json) return { code: 1, text: `${JSON.stringify(data, null, 2)}\n` };
return { code: 1, text: `${reason}\n` };
}
// #4612: a live foreign writer does NOT make the unified open refuse
// anymore (owner decision 2026-10-31: second-writer conditions are
// informational, never blocking) — the full-VACUUM starvation guard
// lives in runGc itself (store.unifiedActiveWriter()), so no special
// branch is needed here.
throw err;
}
try {
Expand Down
2 changes: 0 additions & 2 deletions src/storage/runtime-status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -208,8 +208,6 @@ function degradeReason(): string {
}
if (state.sidecarOpenError) {
const e = shortenError(state.sidecarOpenError);
// #5046/B: shadow.ts records second-writer refusals pre-formatted (already names the fix + what's lost) — render verbatim instead of wrapping them as "index database failed to open".
if (e.startsWith("second-writer:")) return e;
// The driver's no-engine error surfaces through the same open path —
// say what it actually is instead of "index database failed to open".
if (isNoEngineError(e)) {
Expand Down
17 changes: 3 additions & 14 deletions src/storage/shadow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import { log as loggerLog } from "../logger.js";
import { planIngest, planIngestDelta, loadLegacyView, LegacyViewBuilder, appendBlockContents, mergeOrphanBlockContents, completeBlockContents, type LegacySessionLike } from "./ingest.ts";
import type { BlockView } from "../session.js";
import { resolveStorageMode, resolveStorageWalJournalSizeLimit, type StorageMode } from "../knobs.ts";
import { UnifiedStore, StorageIntegrityError, StorageSecondWriterError, type CheckpointRow, type SessionSummaryRow, type WebSummaryRow } from "./store.ts";
import { UnifiedStore, StorageIntegrityError, type CheckpointRow, type SessionSummaryRow, type WebSummaryRow } from "./store.ts";
import path from "node:path";
import { resolveBackendName, backendSelectionNotice, selectedBackendRaw, openBackend, type StorageBackend } from "./backend.ts";
import { canonicalDigest, canonicalize, sha256 } from "./canonical.ts";
Expand Down Expand Up @@ -254,10 +254,8 @@ export class UnifiedShadow {
let store: UnifiedStore;
try {
try {
store = UnifiedStore.open(`${dir}/index.db`, undefined, { walJournalSizeLimitBytes: walLimit, secondWriterPolicy: "degrade" });
store = UnifiedStore.open(`${dir}/index.db`, undefined, { walJournalSizeLimitBytes: walLimit });
} catch (err) {
// #5046/B: a live foreign writer owns this db — never touch it (the file is NOT corrupt) and never take over its lock; the outer catch REFUSES.
if (err instanceof StorageSecondWriterError) throw err;
// #4175: anything else throwing StorageIntegrityError means quick_check
// proved the file set corrupt — handled below WITHOUT moving any file
// (#4626-era owner decision, 2026-10-30).
Expand Down Expand Up @@ -291,15 +289,6 @@ export class UnifiedShadow {
// per session re-establishes trust from its own view.
store.resetAllIngestWatermarks();
} catch (err) {
// #5046/B (owner decision 2026-10-29): refuse to become a persistent second writer — degrade to legacy-only with ONE actionable error. Availability is preserved (legacy persist keeps compression working); CCR / checkpoints / the unified fast-read path are off until isolated. The status wire carries the same fact via recordSidecarOpenFailure (#3040).
if (err instanceof StorageSecondWriterError) {
loggerLog(
"error",
`[storage] unified storage REFUSED for this instance (pid ${process.pid}): another bili process (pid ${err.pid}) already writes ${err.dbPath} — sharing one sessions dir across lanes is the supported topology (#5082/#5091), but two processes writing the SAME sessions dir can split-brain it (#3923/#3959), so this one refuses to co-write and REFUSES TO START until isolated (#5046 owner chain 2026-10-30: no silent degradation of any kind). Fix: give each install its own sessions dir — set BILI_SESSIONS_DIR (and BILI_CONFIG_FILE if the configs differ) for THIS install, then restart (README "Running multiple instances", #5046/#5068); or force legacy explicitly (BILI_STORAGE_UNIFIED=legacy).`,
);
recordSidecarOpenFailure(`second-writer: another bili process (pid ${err.pid}) owns index.db — isolate via BILI_SESSIONS_DIR(+BILI_CONFIG_FILE); refusing to start until then (#5046)`);
throw err;
}
const errStr = String(err);
// Deleting the index cannot fix a no-engine failure (this Node.js has neither builtin
// node:sqlite nor a loadable better-sqlite3), and it cannot clear an OS-level refusal
Expand All @@ -310,7 +299,7 @@ export class UnifiedShadow {
// above handles them (loud alarm + boot WITHOUT the index).
// #4626-era owner decision chain (2026-10-30): every OTHER
// sidecar-open failure REFUSES TO START instead of degrading —
// no-engine/CANTOPEN/readonly/second-writer: availability at the
// no-engine/CANTOPEN/readonly/OS-level: availability at the
// price of a silently different storage path was withdrawn by
// owner instruction "在任何情况下不降级". Intentional opt-outs
// (legacy mode / encryption opt-out / persistence off) resolve to
Expand Down
Loading
Loading