Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to shadow (JSON stays the sole source of truth; the index is digest-reconciled telemetry - set legacy to opt out). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. |
| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. |
| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). |
| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. |
| `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. |
| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). |
| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. |
Expand Down Expand Up @@ -1712,6 +1713,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes
| `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false |
| `BILI_STORAGE_JSON_WRITER` | `storage.jsonWriter` | on |
| `BILI_STORAGE_UNIFIED` | `storage.unified` | unset (shadow) |
| `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES` | `storage.walJournalSizeLimitBytes` | 8388608 |
| `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol |
| `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) |
| `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true |
Expand Down
2 changes: 2 additions & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@
| `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 shadow(JSON 仍是唯一事实源,索引只做对账遥测 —— 设 legacy 可退出)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 |
| `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 |
| `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 |
| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 |
| `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 |
| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 |
| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 |
Expand Down Expand Up @@ -1711,6 +1712,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报
| `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false |
| `BILI_STORAGE_JSON_WRITER` | `storage.jsonWriter` | on |
| `BILI_STORAGE_UNIFIED` | `storage.unified` | unset (shadow) |
| `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES` | `storage.walJournalSizeLimitBytes` | 8388608 |
| `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol |
| `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) |
| `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true |
Expand Down
11 changes: 11 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1802,6 +1802,17 @@ type FileConfig = {
* unset (JSON writing resumes; index-only sessions keep loading via
* the boot union, and replicas/restore re-materialize envelopes). */
jsonWriter?: string;
/** WAL reclamation cap for index.db-wal (#3250), in bytes: every
* completed checkpoint truncates the WAL file back to this size.
* NOT a per-session data quota — data lands in index.db as it is
* written and the WAL only stages frames between checkpoints, so
* session count/size does not scale this. Higher = rarer truncate
* IO but more steady disk use and a longer crash-replay tail.
* Env BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES wins; default
* 8388608 (8 MiB); a non-positive/non-integer value falls back to
* the file/default tier. Applied when the store opens (lane boot) —
* hot edits apply on the next restart. */
walJournalSizeLimitBytes?: number;
};
/** Network & timing knobs (#2030) — every field was an env-only variable
* before (BILI_*_MS / BILI_REPLAY_* family); env still wins when set.
Expand Down
22 changes: 22 additions & 0 deletions src/knobs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import { stateDir } from "./paths.js";
import { loadConfigFile } from "./config.js";
import { UPSTREAM_TIMEOUT_MS, REPLAY_MAX_ATTEMPTS, REPLAY_BASE_DELAY_MS } from "./fetch-util.js";
import { PROXY_KEEPALIVE_MAX_MS } from "./upstream-proxy.js";
import { WAL_JOURNAL_SIZE_LIMIT } from "./storage/schema.js";

/** Tiered raw number: the env var wins when SET (even empty/garbage — the
* caller applies the historical validation), else a finite number from the
Expand Down Expand Up @@ -498,6 +499,11 @@ export function fileStorageAutoDownload(): boolean | undefined {
return typeof v === "boolean" ? v : undefined;
}

export function fileStorageWalJournalSizeLimit(): number | undefined {
const v = loadConfigFile().storage?.walJournalSizeLimitBytes;
return typeof v === "number" && Number.isInteger(v) && v > 0 ? v : undefined;
}

/** Tiered storage-migration switch (#2671): legacy | shadow | full. Lives here
* per the env-var discipline — env tier BILI_STORAGE_UNIFIED (1/true = shadow,
* full/unified = full, 0/false/legacy/garbage = legacy) wins over the
Expand Down Expand Up @@ -551,3 +557,19 @@ export function resolveSqliteAutoDownload(env: NodeJS.ProcessEnv = process.env,
if (raw === "0" || raw === "false") return false;
return typeof fileValue === "boolean" ? fileValue : true;
}

/** #3250 WAL reclamation cap in bytes. Env
* BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES (positive integer) wins over the
* config-file `storage.walJournalSizeLimitBytes`; a garbage/non-positive env
* value falls through to the file/default tier (same discipline as
* resolveSqliteAutoDownload — never crash boot on a typo'd knob). Pure:
* env + file value in, bytes out (never throws). */
export function resolveStorageWalJournalSizeLimit(env: NodeJS.ProcessEnv = process.env, fileValue?: number): number {
const raw = env.BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES?.trim();
if (raw) {
const n = Number(raw);
if (Number.isInteger(n) && n > 0) return n;
}
if (typeof fileValue === "number" && Number.isInteger(fileValue) && fileValue > 0) return fileValue;
return WAL_JOURNAL_SIZE_LIMIT;
}
3 changes: 2 additions & 1 deletion src/persist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import {
resolveJsonWriterOff as knobResolveJsonWriterOff,
fileStorageUnified,
fileStorageJsonWriter,
fileStorageWalJournalSizeLimit,
} from "./knobs.js";
import { log as loggerLog } from "./logger.js";
import { recordRequestedMode } from "./storage/runtime-status.js";
Expand Down Expand Up @@ -407,7 +408,7 @@ export class SessionStore {
? (UnifiedShadow.maybeOpen(this.sessionsDir, process.env, (rec) => {
const s = buildSession(rec as PersistedSession);
return toLegacyLike(s);
}, fileStorageUnified()) ?? undefined)
}, fileStorageUnified(), fileStorageWalJournalSizeLimit()) ?? undefined)
: undefined;
this.store = new StateStore<PersistedSession>({
dir: this.sessionsDir,
Expand Down
5 changes: 4 additions & 1 deletion src/storage/runtime-status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

import type { BootstrapOutcome, CooldownMarker } from "./engine-bootstrap.js";
import { COOLDOWN_MS, readBootstrapFailureMarker } from "./engine-bootstrap.js";
import { fileStorageAutoDownload, fileStorageJsonWriter, fileStorageUnified, resolveStorageMode } from "../knobs.js";
import { fileStorageAutoDownload, fileStorageJsonWriter, fileStorageUnified, fileStorageWalJournalSizeLimit, resolveStorageMode } from "../knobs.js";
import { MAX_CONSECUTIVE_FAILURES, type UnifiedShadow } from "./shadow.js";
import type { StorageMode } from "./shadow.js";

Expand Down Expand Up @@ -208,6 +208,7 @@ export function storageDiagnosticsLines(ctx: { sessions: number; storageLogLines
const fileUnified = fileStorageUnified();
const fileJsonWriter = fileStorageJsonWriter();
const fileAutoDownload = fileStorageAutoDownload();
const fileWalLimit = fileStorageWalJournalSizeLimit();
const lines: string[] = [
"--- storage ---",
`requested mode: ${st.requested}`,
Expand All @@ -222,6 +223,8 @@ export function storageDiagnosticsLines(ctx: { sessions: number; storageLogLines
`storage.jsonWriter (config file): ${fileJsonWriter ?? "(unset)"}`,
`BILI_SQLITE_AUTO_DOWNLOAD: ${env.BILI_SQLITE_AUTO_DOWNLOAD ?? "(unset)"}`,
`storage.sqliteAutoDownload (config file): ${fileAutoDownload ?? "(unset)"}`,
`BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES: ${env.BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES ?? "(unset)"}`,
`storage.walJournalSizeLimitBytes (config file): ${fileWalLimit ?? "(unset)"}`,
`BILI_ENCRYPTION_KEY set: ${env.BILI_ENCRYPTION_KEY !== undefined && env.BILI_ENCRYPTION_KEY !== "" ? "yes" : "no"}`,
`sessions on disk: ${ctx.sessions}`,
];
Expand Down
32 changes: 26 additions & 6 deletions src/storage/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,32 @@
// build upgrades a v4 file with one idempotent ALTER TABLE.
const STORAGE_SCHEMA_VERSION = 5;

export const STORAGE_PRAGMAS = [
"PRAGMA journal_mode = WAL",
"PRAGMA synchronous = NORMAL",
"PRAGMA busy_timeout = 5000",
"PRAGMA foreign_keys = ON",
] as const;
/** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and
* auto checkpoints move frames into the main db and reset the write offset
* but NEVER shrink the WAL file — without this limit a long-lived process's
* index.db-wal stays at its historical high-water mark forever (field:
* 55 MiB flat). With it, every completed checkpoint truncates the file back
* to the cap. 8 MiB sits well above any single-save burst while keeping
* disk use and crash-recovery replay bounded. Per-connection setting: each
* process that opens the shared index applies its own copy (#2671 topology).
* Tunable since #3250 via storage.walJournalSizeLimitBytes /
* BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES (resolved by src/knobs.ts). */
export const WAL_JOURNAL_SIZE_LIMIT = 8 * 1024 * 1024;

/** Connection pragmas for every index.db opener. The journal cap is
* per-opener (see WAL_JOURNAL_SIZE_LIMIT); the default list below is what
* short-lived openers (export fast path) and tests apply. */
export function storagePragmas(walJournalSizeLimitBytes: number): string[] {
return [
"PRAGMA journal_mode = WAL",
"PRAGMA synchronous = NORMAL",
"PRAGMA busy_timeout = 5000",
"PRAGMA foreign_keys = ON",
`PRAGMA journal_size_limit = ${walJournalSizeLimitBytes}`,
];
}

export const STORAGE_PRAGMAS = storagePragmas(WAL_JOURNAL_SIZE_LIMIT);

const STORAGE_DDL_V1 = `CREATE TABLE IF NOT EXISTS sessions (
session_id TEXT PRIMARY KEY,
Expand Down
11 changes: 7 additions & 4 deletions src/storage/shadow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@

import { log as loggerLog } from "../logger.js";
import { planIngest, planIngestDelta, loadLegacyView, completeBlockContents, type LegacySessionLike } from "./ingest.ts";
import { resolveStorageMode, type StorageMode } from "../knobs.ts";
import { resolveStorageMode, resolveStorageWalJournalSizeLimit, type StorageMode } from "../knobs.ts";
import { UnifiedStore, type CheckpointRow, type SessionSummaryRow, type WebSummaryRow } from "./store.ts";
import { canonicalize, sha256 } from "./canonical.ts";
import { recordSidecarInstance, recordSidecarOpenFailure } from "./runtime-status.ts";
Expand Down Expand Up @@ -77,12 +77,14 @@ export class UnifiedShadow {
env: NodeJS.ProcessEnv = process.env,
legacyLoad?: (record: unknown) => LegacySessionLike | null,
fileUnified?: string,
fileWalLimit?: number,
): UnifiedShadow | null {
const mode = resolveStorageMode(env, fileUnified);
if (mode === "legacy") return null;
const walLimit = resolveStorageWalJournalSizeLimit(env, fileWalLimit);
let store: UnifiedStore;
try {
store = UnifiedStore.open(`${dir}/index.db`);
store = UnifiedStore.open(`${dir}/index.db`, undefined, { walJournalSizeLimitBytes: walLimit });
} catch (err) {
// #2771: a corrupt/unreadable index.db must degrade to legacy-only
// (same shape as the encryption opt-out), never brick SessionStore
Expand All @@ -98,11 +100,12 @@ export class UnifiedShadow {
const shadow = new UnifiedShadow(mode, store, legacyLoad ?? (() => null));
recordSidecarInstance(shadow);
const engineTag = ` [engine: ${store.engine}/${store.engineSource}]`;
const walTag = ` [wal-cap: ${walLimit}]`;
loggerLog(
"info",
mode === "full"
? `[storage] unified FULL mode ENABLED (storage.unified=full or BILI_STORAGE_UNIFIED=full): index.db is the load path (unified-first, legacy fallback); legacy files keep being written — unset the knob to roll back${engineTag}`
: `[storage] unified shadow mode ENABLED (storage.unified=shadow or BILI_STORAGE_UNIFIED): every session write is double-ingested into index.db and digest-reconciled${engineTag}`,
? `[storage] unified FULL mode ENABLED (storage.unified=full or BILI_STORAGE_UNIFIED=full): index.db is the load path (unified-first, legacy fallback); legacy files keep being written — unset the knob to roll back${engineTag}${walTag}`
: `[storage] unified shadow mode ENABLED (storage.unified=shadow or BILI_STORAGE_UNIFIED): every session write is double-ingested into index.db and digest-reconciled${engineTag}${walTag}`,
);
return shadow;
}
Expand Down
Loading
Loading