diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 4318bc65b5..3d9d293b75 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -162,6 +162,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | Tiered storage-migration switch (#2671): shadow = unified SQLite double-write dry-run with digest reconciliation; full = drop-in read path (unified-first loads with organic import, legacy fallback). Absent everywhere now defaults to shadow (JSON stays the sole source of truth; the index is digest-reconciled telemetry - set legacy to opt out). File tier is strict legacy/shadow/full; the env tier also accepts 1/true (=shadow) and unified (=full) and wins over the file. Resolved once per lane boot - hot edits need a restart. Encryption (BILI_ENCRYPTION_KEY) hard-blocks the sidecar. | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | Gate for the one-time SQLite engine self-download (#2833): when no engine is available in-process (no built-in node:sqlite, no host-installed better-sqlite3) and the unified sidecar is wanted on this platform, bili downloads the pinned better-sqlite3 prebuilt tarball from the npm registry once (~11 MB, sha512-verified) into the cache dir and reuses it on every later start. Engine resolution order: built-in node:sqlite (Node >= 22.5) -> host-provided better-sqlite3 -> cached self-download -> self-download (gated by this switch) -> none (legacy-only, the #2825 degrade contract). The switch gates only the network fetch - a cached copy is used even when it is false; unsupported platforms never download. | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | +| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. | @@ -1712,6 +1713,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | | `BILI_STORAGE_JSON_WRITER` | `storage.jsonWriter` | on | | `BILI_STORAGE_UNIFIED` | `storage.unified` | unset (shadow) | +| `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES` | `storage.walJournalSizeLimitBytes` | 8388608 | | `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol | | `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) | | `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index df7067c674..7ab13782e4 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -162,6 +162,7 @@ | `storage.unified` | legacy\|shadow\|full | unset (shadow) | BILI_STORAGE_UNIFIED | 分层存储迁移开关(#2671):shadow = 统一 SQLite 双写干跑 + digest 对账;full = 平替读路径(unified-first 加载 + 有机导入 + legacy 兜底)。两处均未设置时默认 shadow(JSON 仍是唯一事实源,索引只做对账遥测 —— 设 legacy 可退出)。文件层只接受 legacy/shadow/full;环境变量层还接受 1/true(=shadow)与 unified(=full)并优先于文件。每个 lane 启动时解析一次 —— 热改动需重启生效。加密(BILI_ENCRYPTION_KEY)会硬阻断边车。 | | `storage.sqliteAutoDownload` | boolean | true | BILI_SQLITE_AUTO_DOWNLOAD | SQLite 引擎一次性自下载的闸门(#2833):进程内没有可用引擎(无内置 node:sqlite、宿主未装 better-sqlite3)且本平台希望启用统一边车时,bili 从 npm registry 一次性下载钉死的 better-sqlite3 预编译 tarball(约 11 MB,sha512 校验)到缓存目录,之后每次启动复用。引擎解析顺序:内置 node:sqlite(Node >= 22.5)-> 宿主机已装的 better-sqlite3 -> 缓存的自下载副本 -> 自下载(本开关控制)-> 无引擎(仅 legacy,即 #2825 降级契约)。开关只控制网络下载 —— 为 false 时缓存副本仍会使用;不支持的平台永不下载。 | | `storage.jsonWriter` | on\|off | on | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | +| `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 | @@ -1711,6 +1712,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | | `BILI_STORAGE_JSON_WRITER` | `storage.jsonWriter` | on | | `BILI_STORAGE_UNIFIED` | `storage.unified` | unset (shadow) | +| `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES` | `storage.walJournalSizeLimitBytes` | 8388608 | | `BILI_STREAM_ERROR_SHAPE` | `compat.streamErrorShape` | protocol | | `BILI_STREAM_KEEPALIVE_MS` | `network.streamKeepAliveMs` | 15000 (0 disables) | | `BILI_SUBAGENT_SPLIT` | `subagentSplit` | true | diff --git a/src/config.ts b/src/config.ts index 811b6318f1..5a4d7c5be3 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1802,6 +1802,17 @@ type FileConfig = { * unset (JSON writing resumes; index-only sessions keep loading via * the boot union, and replicas/restore re-materialize envelopes). */ jsonWriter?: string; + /** WAL reclamation cap for index.db-wal (#3250), in bytes: every + * completed checkpoint truncates the WAL file back to this size. + * NOT a per-session data quota — data lands in index.db as it is + * written and the WAL only stages frames between checkpoints, so + * session count/size does not scale this. Higher = rarer truncate + * IO but more steady disk use and a longer crash-replay tail. + * Env BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES wins; default + * 8388608 (8 MiB); a non-positive/non-integer value falls back to + * the file/default tier. Applied when the store opens (lane boot) — + * hot edits apply on the next restart. */ + walJournalSizeLimitBytes?: number; }; /** Network & timing knobs (#2030) — every field was an env-only variable * before (BILI_*_MS / BILI_REPLAY_* family); env still wins when set. diff --git a/src/knobs.ts b/src/knobs.ts index ee6bdd0646..e56c338fca 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -25,6 +25,7 @@ import { stateDir } from "./paths.js"; import { loadConfigFile } from "./config.js"; import { UPSTREAM_TIMEOUT_MS, REPLAY_MAX_ATTEMPTS, REPLAY_BASE_DELAY_MS } from "./fetch-util.js"; import { PROXY_KEEPALIVE_MAX_MS } from "./upstream-proxy.js"; +import { WAL_JOURNAL_SIZE_LIMIT } from "./storage/schema.js"; /** Tiered raw number: the env var wins when SET (even empty/garbage — the * caller applies the historical validation), else a finite number from the @@ -498,6 +499,11 @@ export function fileStorageAutoDownload(): boolean | undefined { return typeof v === "boolean" ? v : undefined; } +export function fileStorageWalJournalSizeLimit(): number | undefined { + const v = loadConfigFile().storage?.walJournalSizeLimitBytes; + return typeof v === "number" && Number.isInteger(v) && v > 0 ? v : undefined; +} + /** Tiered storage-migration switch (#2671): legacy | shadow | full. Lives here * per the env-var discipline — env tier BILI_STORAGE_UNIFIED (1/true = shadow, * full/unified = full, 0/false/legacy/garbage = legacy) wins over the @@ -551,3 +557,19 @@ export function resolveSqliteAutoDownload(env: NodeJS.ProcessEnv = process.env, if (raw === "0" || raw === "false") return false; return typeof fileValue === "boolean" ? fileValue : true; } + +/** #3250 WAL reclamation cap in bytes. Env + * BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES (positive integer) wins over the + * config-file `storage.walJournalSizeLimitBytes`; a garbage/non-positive env + * value falls through to the file/default tier (same discipline as + * resolveSqliteAutoDownload — never crash boot on a typo'd knob). Pure: + * env + file value in, bytes out (never throws). */ +export function resolveStorageWalJournalSizeLimit(env: NodeJS.ProcessEnv = process.env, fileValue?: number): number { + const raw = env.BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES?.trim(); + if (raw) { + const n = Number(raw); + if (Number.isInteger(n) && n > 0) return n; + } + if (typeof fileValue === "number" && Number.isInteger(fileValue) && fileValue > 0) return fileValue; + return WAL_JOURNAL_SIZE_LIMIT; +} diff --git a/src/persist.ts b/src/persist.ts index 82271e5966..876c4efd43 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -14,6 +14,7 @@ import { resolveJsonWriterOff as knobResolveJsonWriterOff, fileStorageUnified, fileStorageJsonWriter, + fileStorageWalJournalSizeLimit, } from "./knobs.js"; import { log as loggerLog } from "./logger.js"; import { recordRequestedMode } from "./storage/runtime-status.js"; @@ -407,7 +408,7 @@ export class SessionStore { ? (UnifiedShadow.maybeOpen(this.sessionsDir, process.env, (rec) => { const s = buildSession(rec as PersistedSession); return toLegacyLike(s); - }, fileStorageUnified()) ?? undefined) + }, fileStorageUnified(), fileStorageWalJournalSizeLimit()) ?? undefined) : undefined; this.store = new StateStore({ dir: this.sessionsDir, diff --git a/src/storage/runtime-status.ts b/src/storage/runtime-status.ts index 438b7df7f2..e33846db11 100644 --- a/src/storage/runtime-status.ts +++ b/src/storage/runtime-status.ts @@ -25,7 +25,7 @@ import type { BootstrapOutcome, CooldownMarker } from "./engine-bootstrap.js"; import { COOLDOWN_MS, readBootstrapFailureMarker } from "./engine-bootstrap.js"; -import { fileStorageAutoDownload, fileStorageJsonWriter, fileStorageUnified, resolveStorageMode } from "../knobs.js"; +import { fileStorageAutoDownload, fileStorageJsonWriter, fileStorageUnified, fileStorageWalJournalSizeLimit, resolveStorageMode } from "../knobs.js"; import { MAX_CONSECUTIVE_FAILURES, type UnifiedShadow } from "./shadow.js"; import type { StorageMode } from "./shadow.js"; @@ -208,6 +208,7 @@ export function storageDiagnosticsLines(ctx: { sessions: number; storageLogLines const fileUnified = fileStorageUnified(); const fileJsonWriter = fileStorageJsonWriter(); const fileAutoDownload = fileStorageAutoDownload(); + const fileWalLimit = fileStorageWalJournalSizeLimit(); const lines: string[] = [ "--- storage ---", `requested mode: ${st.requested}`, @@ -222,6 +223,8 @@ export function storageDiagnosticsLines(ctx: { sessions: number; storageLogLines `storage.jsonWriter (config file): ${fileJsonWriter ?? "(unset)"}`, `BILI_SQLITE_AUTO_DOWNLOAD: ${env.BILI_SQLITE_AUTO_DOWNLOAD ?? "(unset)"}`, `storage.sqliteAutoDownload (config file): ${fileAutoDownload ?? "(unset)"}`, + `BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES: ${env.BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES ?? "(unset)"}`, + `storage.walJournalSizeLimitBytes (config file): ${fileWalLimit ?? "(unset)"}`, `BILI_ENCRYPTION_KEY set: ${env.BILI_ENCRYPTION_KEY !== undefined && env.BILI_ENCRYPTION_KEY !== "" ? "yes" : "no"}`, `sessions on disk: ${ctx.sessions}`, ]; diff --git a/src/storage/schema.ts b/src/storage/schema.ts index 27ab713472..3267fd460f 100644 --- a/src/storage/schema.ts +++ b/src/storage/schema.ts @@ -65,12 +65,32 @@ // build upgrades a v4 file with one idempotent ALTER TABLE. const STORAGE_SCHEMA_VERSION = 5; -export const STORAGE_PRAGMAS = [ - "PRAGMA journal_mode = WAL", - "PRAGMA synchronous = NORMAL", - "PRAGMA busy_timeout = 5000", - "PRAGMA foreign_keys = ON", -] as const; +/** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and + * auto checkpoints move frames into the main db and reset the write offset + * but NEVER shrink the WAL file — without this limit a long-lived process's + * index.db-wal stays at its historical high-water mark forever (field: + * 55 MiB flat). With it, every completed checkpoint truncates the file back + * to the cap. 8 MiB sits well above any single-save burst while keeping + * disk use and crash-recovery replay bounded. Per-connection setting: each + * process that opens the shared index applies its own copy (#2671 topology). + * Tunable since #3250 via storage.walJournalSizeLimitBytes / + * BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES (resolved by src/knobs.ts). */ +export const WAL_JOURNAL_SIZE_LIMIT = 8 * 1024 * 1024; + +/** Connection pragmas for every index.db opener. The journal cap is + * per-opener (see WAL_JOURNAL_SIZE_LIMIT); the default list below is what + * short-lived openers (export fast path) and tests apply. */ +export function storagePragmas(walJournalSizeLimitBytes: number): string[] { + return [ + "PRAGMA journal_mode = WAL", + "PRAGMA synchronous = NORMAL", + "PRAGMA busy_timeout = 5000", + "PRAGMA foreign_keys = ON", + `PRAGMA journal_size_limit = ${walJournalSizeLimitBytes}`, + ]; +} + +export const STORAGE_PRAGMAS = storagePragmas(WAL_JOURNAL_SIZE_LIMIT); const STORAGE_DDL_V1 = `CREATE TABLE IF NOT EXISTS sessions ( session_id TEXT PRIMARY KEY, diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 1aeea1eede..3720038ac2 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -23,7 +23,7 @@ import { log as loggerLog } from "../logger.js"; import { planIngest, planIngestDelta, loadLegacyView, completeBlockContents, type LegacySessionLike } from "./ingest.ts"; -import { resolveStorageMode, type StorageMode } from "../knobs.ts"; +import { resolveStorageMode, resolveStorageWalJournalSizeLimit, type StorageMode } from "../knobs.ts"; import { UnifiedStore, type CheckpointRow, type SessionSummaryRow, type WebSummaryRow } from "./store.ts"; import { canonicalize, sha256 } from "./canonical.ts"; import { recordSidecarInstance, recordSidecarOpenFailure } from "./runtime-status.ts"; @@ -77,12 +77,14 @@ export class UnifiedShadow { env: NodeJS.ProcessEnv = process.env, legacyLoad?: (record: unknown) => LegacySessionLike | null, fileUnified?: string, + fileWalLimit?: number, ): UnifiedShadow | null { const mode = resolveStorageMode(env, fileUnified); if (mode === "legacy") return null; + const walLimit = resolveStorageWalJournalSizeLimit(env, fileWalLimit); let store: UnifiedStore; try { - store = UnifiedStore.open(`${dir}/index.db`); + store = UnifiedStore.open(`${dir}/index.db`, undefined, { walJournalSizeLimitBytes: walLimit }); } catch (err) { // #2771: a corrupt/unreadable index.db must degrade to legacy-only // (same shape as the encryption opt-out), never brick SessionStore @@ -98,11 +100,12 @@ export class UnifiedShadow { const shadow = new UnifiedShadow(mode, store, legacyLoad ?? (() => null)); recordSidecarInstance(shadow); const engineTag = ` [engine: ${store.engine}/${store.engineSource}]`; + const walTag = ` [wal-cap: ${walLimit}]`; loggerLog( "info", mode === "full" - ? `[storage] unified FULL mode ENABLED (storage.unified=full or BILI_STORAGE_UNIFIED=full): index.db is the load path (unified-first, legacy fallback); legacy files keep being written — unset the knob to roll back${engineTag}` - : `[storage] unified shadow mode ENABLED (storage.unified=shadow or BILI_STORAGE_UNIFIED): every session write is double-ingested into index.db and digest-reconciled${engineTag}`, + ? `[storage] unified FULL mode ENABLED (storage.unified=full or BILI_STORAGE_UNIFIED=full): index.db is the load path (unified-first, legacy fallback); legacy files keep being written — unset the knob to roll back${engineTag}${walTag}` + : `[storage] unified shadow mode ENABLED (storage.unified=shadow or BILI_STORAGE_UNIFIED): every session write is double-ingested into index.db and digest-reconciled${engineTag}${walTag}`, ); return shadow; } diff --git a/src/storage/store.ts b/src/storage/store.ts index e078860426..4db22f1a4c 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -19,10 +19,20 @@ import { createHash } from "node:crypto"; import { mkdirSync } from "node:fs"; import { dirname } from "node:path"; +import { log as loggerLog } from "../logger.js"; import { openSqliteDatabase, type SqliteDatabase, type SqliteEngineName, type SqliteEngineSource } from "./driver.js"; -import { migrateSchema, STORAGE_PRAGMAS } from "./schema.js"; +import { migrateSchema, storagePragmas, WAL_JOURNAL_SIZE_LIMIT } from "./schema.js"; import type { MessageContentStore } from "acp-kernel"; +/** #3250: run a PASSIVE wal_checkpoint every N successful ingests. The + * built-in autocheckpoint alone cannot guarantee reclamation in a long-lived + * process: while a foreign connection holds a read snapshot (dashboard / + * export / second lane sharing the directory — #2671 topology) WAL backfill + * is blocked at that snapshot and the file grows until the reader lets go. + * PASSIVE never blocks the ingest path; a failed checkpoint is logged and + * forgotten — the next tick retries and journal_size_limit does the rest. */ +const WAL_CHECKPOINT_EVERY_INGESTS = 20; + export function sha256Hex(bytes: string | Buffer): string { return createHash("sha256").update(bytes).digest("hex"); } @@ -219,6 +229,8 @@ export class UnifiedStore { /** Where the engine came from (#2833): builtin / host / self-downloaded — logged at enable so field diagnosis can tell which acquisition path a lane ran on. */ readonly engineSource: SqliteEngineSource; private readonly db: SqliteDatabase; + /** #3250: successful-ingest counter driving the WAL_CHECKPOINT_EVERY_INGESTS cadence. */ + private ingestCount = 0; private constructor(db: SqliteDatabase, engine: SqliteEngineName, engineSource: SqliteEngineSource) { this.db = db; @@ -226,11 +238,16 @@ export class UnifiedStore { this.engineSource = engineSource; } - static open(path: string, engine?: SqliteEngineName): UnifiedStore { + static open( + path: string, + engine?: SqliteEngineName, + opts?: { walJournalSizeLimitBytes?: number }, + ): UnifiedStore { mkdirSync(dirname(path), { recursive: true }); const { db, engine: used, source } = openSqliteDatabase(path, engine); try { - for (const pragma of STORAGE_PRAGMAS) db.exec(pragma); + const limit = opts?.walJournalSizeLimitBytes ?? WAL_JOURNAL_SIZE_LIMIT; + for (const pragma of storagePragmas(limit)) db.exec(pragma); migrateSchema(db); } catch (err) { // The handle we just opened must not leak when the file turns out @@ -241,10 +258,31 @@ export class UnifiedStore { db.close(); throw err; } + // #3250: converge the WAL an unclean exit left behind (SIGKILL / power + // loss never reaches close()). Moving committed frames into the main db + // now shortens the NEXT crash's recovery replay. Isolated from the open + // contract above on purpose: under a multi-process shared directory + // (#2671) a TRUNCATE can come back busy while a foreign connection is + // mid-transaction — that must log, not fail the open; the periodic and + // close-time passes retry the reclamation. + try { + db.prepare("PRAGMA wal_checkpoint(TRUNCATE)").get(); + } catch (err) { + loggerLog("warn", `[storage] WAL checkpoint on open failed for ${path}: ${String(err)}`); + } return new UnifiedStore(db, used, source); } close(): void { + // #3250: converge the WAL before releasing the handle so a clean exit + // leaves nothing for the next boot's recovery to replay. Same isolation + // as open(): a checkpoint hiccup logs and never throws — callers treat + // close() as best-effort shutdown work. + try { + this.db.prepare("PRAGMA wal_checkpoint(TRUNCATE)").get(); + } catch (err) { + loggerLog("warn", `[storage] WAL checkpoint on close failed: ${String(err)}`); + } this.db.close(); } @@ -301,7 +339,7 @@ export class UnifiedStore { delta?: { trustedSeq: number; boundaryMsgId: string; boundaryHash: string; trustedTailSeqs: number[] }, ): boolean { const s = input.session; - return this.db.transaction((): boolean => { + const ok = this.db.transaction((): boolean => { let messages: MessageRow[]; let watermark: number | null; if (delta) { @@ -537,6 +575,27 @@ export class UnifiedStore { } return true; }); + if (ok) this.maybeCheckpointWal(); + return ok; + } + + /** #3250: cadence-tick WAL reclamation (see WAL_CHECKPOINT_EVERY_INGESTS). + * Never throws into the ingest path — a failed checkpoint just leaves the + * WAL longer than ideal until the next tick. */ + private maybeCheckpointWal(): void { + this.ingestCount += 1; + if (this.ingestCount % WAL_CHECKPOINT_EVERY_INGESTS !== 0) return; + try { + // Log the result: operators tuning storage.walJournalSizeLimitBytes + // need to see whether ticks are actually reclaiming frames. + const row = this.db.prepare("PRAGMA wal_checkpoint(PASSIVE)").get() as { busy: number; log: number; checkpointed: number }; + loggerLog( + "debug", + `[storage] periodic WAL checkpoint at ingest ${this.ingestCount}: busy=${row.busy} log=${row.log} checkpointed=${row.checkpointed}`, + ); + } catch (err) { + loggerLog("debug", `[storage] periodic WAL checkpoint failed at ingest ${this.ingestCount}: ${String(err)}`); + } } /** Full-ingest path of ingestSession, extracted for readability: the diff --git a/tests/issue3250-wal-checkpoint.test.ts b/tests/issue3250-wal-checkpoint.test.ts new file mode 100644 index 0000000000..e010cb8a79 --- /dev/null +++ b/tests/issue3250-wal-checkpoint.test.ts @@ -0,0 +1,304 @@ +/** + * #3250: index.db-wal reclamation in long-lived processes. + * + * The unified index runs in WAL mode, but nothing ever shrank the WAL FILE: + * passive/auto checkpoints move frames into the main db and reset the write + * offset, yet they never ftruncate the file — so a long-lived lane process + * (dashboard/export/second-lane readers pin frames, big saves burst past any + * peak) left index.db-wal growing to its historical high-water mark and + * staying there (field observation: 55 MiB flat, `{busy:0, log:2720, + * checkpointed:2720}` — every frame reclaimable, nobody reclaiming). Costs: + * permanent disk occupation proportional to uptime, and crash-recovery + * replay time after SIGKILL/power loss. + * + * Fix under test (src/storage/schema.ts + src/storage/store.ts): + * 1. `PRAGMA journal_size_limit` (default 8 MiB, tunable via + * storage.walJournalSizeLimitBytes / BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES, + * resolved in src/knobs.ts) — every COMPLETED checkpoint now truncates the + * file back to the cap; + * 2. one-shot `wal_checkpoint(TRUNCATE)` on UnifiedStore.open (converges + * residue an unclean exit left behind — shortens the NEXT recovery) + * and before close (clean exit leaves nothing to replay); + * 3. a PASSIVE checkpoint every WAL_CHECKPOINT_EVERY_INGESTS successful + * ingests — reclamation keeps happening under continuous ingest even + * when the autocheckpoint window never appears; never blocks, failures + * are logged and forgotten. + * + * Every test below is a RED/GREEN discriminator: on the pre-fix tree the + * asserted size is the unreclaimed high-water mark; with the fix the file + * converges to the cap (or zero). + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { migrateSchema, STORAGE_PRAGMAS } from "../src/storage/schema.ts"; +import { planIngest, loadLegacyView, type LegacySessionLike } from "../src/storage/ingest.ts"; +import type { CoreMessage } from "acp-kernel"; + +process.env.BILI_PERSIST_ZSTD = "0"; +// Same hermeticity pin as the other storage suites: a developer machine with +// storage.unified in its real config must not leak a tier into these tests. +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-3250-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); + +function tmpdir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function rmBestEffort(p: string): void { + try { + fs.rmSync(p, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + /* leaked temp dir on a Windows runner is harmless */ + } +} + +const indexDb = (dir: string): string => path.join(dir, "index.db"); +const walPath = (dir: string): string => `${indexDb(dir)}-wal`; + +function walSize(dir: string): number { + try { + return fs.statSync(walPath(dir)).size; + } catch { + return 0; + } +} + +/** Raw connection mirroring UnifiedStore.open's setup — used to hold the + * database open (keeping the -wal file alive across another store's + * close()) or to take a pinned read snapshot. */ +function rawOpen(dir: string): { db: ReturnType["db"]; close(): void } { + const { db } = openSqliteDatabase(indexDb(dir)); + for (const pragma of STORAGE_PRAGMAS) db.exec(pragma); + migrateSchema(db); + return { db, close: () => db.close() }; +} + +let msgCounter = 0; +function mkMsg(text: string): CoreMessage { + msgCounter += 1; + return { id: `h_msg${msgCounter}`, role: msgCounter % 2 === 0 ? "user" : "assistant", contentType: "text", text } as CoreMessage; +} + +function mkLegacy(id: string, totalBytes: number, msgsPerSession: number): LegacySessionLike { + const pad = Math.max(1, Math.floor(totalBytes / msgsPerSession)); + const lastMessages = Array.from({ length: msgsPerSession }, (_, k) => mkMsg(`${id}-${k}-`.padEnd(pad, "x"))); + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "anthropic" }, + stats: { requests: 1 }, + metadata: {}, + // Key order matters: loadLegacyView rebuilds state in MERGE_STATE_FIELDS + // order and the shadow digest is JSON.stringify-based (order-sensitive). + state: { + blocks: [], + messageRefs: { byRaw: {}, byRef: {} }, + nextBlockId: 1, + nextRunId: 1, + }, + blockContents: new Map(), + lastMessages, + lastMessagesFolded: false, + }; +} + +function ingest(store: UnifiedStore, id: string, totalBytes: number, msgsPerSession: number): void { + assert.equal(store.ingestSession(planIngest(mkLegacy(id, totalBytes, msgsPerSession))), true, `ingest ${id} must apply`); +} + +const KiB = 1024; +const MiB = 1024 * 1024; +// Contract literals, deliberately NOT imported from src: the assertions must +// hold even if the implementation's constants drift. 8 MiB is the #3250 +// journal_size_limit; 20 is the periodic-checkpoint cadence (ingests). +const WAL_CAP = 8 * MiB; +const CHECKPOINT_EVERY_N = 20; +/** Truncation lands page-aligned; allow one page of measurement slack. */ +const CAP_SLACK = 4 * KiB; + +test("#3250: close() converges the WAL so a clean exit leaves nothing to replay", () => { + const dir = tmpdir("bili-3250-close-"); + let pin: ReturnType | undefined; + let store: UnifiedStore | undefined; + let check: UnifiedStore | undefined; + let storeClosed = false; // the body closes `store`; keep the finally idempotent (double close throws ERR_INVALID_STATE) + try { + // A foreign connection keeps the database (and thus the -wal file) + // alive after `store` closes — otherwise a clean last-connection + // close would delete the WAL file and mask whether close() did the + // converging work itself. + pin = rawOpen(dir); + store = UnifiedStore.open(indexDb(dir)); + ingest(store, "s1", 1 * MiB, 40); + ingest(store, "s2", 1 * MiB, 40); + assert.ok(walSize(dir) > 0, "frames must be sitting in the WAL before close"); + + store.close(); + storeClosed = true; + assert.ok( + walSize(dir) <= CAP_SLACK, + `close() must have truncated the WAL (left ${walSize(dir)} bytes — un-reclaimed high-water mark)`, + ); + + check = UnifiedStore.open(indexDb(dir)); + for (const id of ["s1", "s2"]) { + assert.equal(check.getMessages(id).length, 40, `${id} message rows must survive the checkpoint`); + } + } finally { + check?.close(); + if (!storeClosed) store?.close(); + pin?.close(); + rmBestEffort(dir); + } +}); + +test("#3250: open() converges the WAL an unclean exit left behind, data intact", () => { + const dir = tmpdir("bili-3250-open-"); + let dirty: UnifiedStore | undefined; + let late: UnifiedStore | undefined; + try { + dirty = UnifiedStore.open(indexDb(dir)); + ingest(dirty, "s1", 1 * MiB, 40); + ingest(dirty, "s2", 1 * MiB, 40); + // `dirty` is deliberately NOT closed: the process died (SIGKILL / + // power loss) with its handle open, leaving a live WAL behind. + assert.ok(walSize(dir) > 0, "the unclean exit must have left WAL frames behind"); + + late = UnifiedStore.open(indexDb(dir)); + assert.ok( + walSize(dir) <= CAP_SLACK, + `open() must have checkpointed the residue (left ${walSize(dir)} bytes — next crash would replay it all)`, + ); + for (const id of ["s1", "s2"]) { + assert.equal(late.getMessages(id).length, 40, `${id} message rows must survive the open-time checkpoint`); + } + } finally { + late?.close(); + dirty?.close(); + rmBestEffort(dir); + } +}); + +test("#3250: a save burst past the cap comes back under the cap at the next cadence tick", () => { + const dir = tmpdir("bili-3250-burst-"); + let store: UnifiedStore | undefined; + try { + store = UnifiedStore.open(indexDb(dir)); + // One 12 MiB commit (≈3000 pages). A connection cannot checkpoint the + // frames it just wrote while it is still the last writer (verified: + // same-conn PASSIVE right after the commit reclaims nothing) — so the + // burst residue sitting in the WAL is normal WAL behavior, not the bug. + // The #3250 defect is what happens NEXT: without journal_size_limit, + // every later checkpoint resets the write offset but never shrinks the + // file, so the ~12 MiB high-water mark persists for the life of the + // process. With the limit, the first COMPLETED checkpoint after newer + // writes truncate the file back to the cap — here driven by the + // periodic PASSIVE at the CHECKPOINT_EVERY_N-th ingest. + ingest(store, "big", 12 * MiB, 480); + assert.ok(walSize(dir) > 0, "the burst must have left frames in the WAL"); + for (let i = 2; i <= CHECKPOINT_EVERY_N; i++) ingest(store, `small${i}`, 1 * MiB, 40); + assert.ok( + walSize(dir) <= WAL_CAP + CAP_SLACK, + `WAL must be truncated to the cap by the cadence checkpoint (left ${walSize(dir)} bytes, cap ${WAL_CAP})`, + ); + assert.equal(store.getMessages("big").length, 480, "all burst rows must land"); + assert.equal(store.getMessages(`small${CHECKPOINT_EVERY_N}`).length, 40, "all follow-up rows must land"); + } finally { + store?.close(); + rmBestEffort(dir); + } +}); + +test("#3250: pinned reader lifecycle — growth while pinned, reclamation after release", () => { + const dir = tmpdir("bili-3250-pin-"); + let store: UnifiedStore | undefined; + let reader: ReturnType | undefined; + try { + store = UnifiedStore.open(indexDb(dir)); + ingest(store, "seed1", 1 * MiB, 40); + ingest(store, "seed2", 1 * MiB, 40); + + // A foreign connection takes a read snapshot AFTER the seed frames + // exist and holds it open — the production shape (dashboard/export/ + // second lane sharing the directory, #2671 topology). WAL backfill is + // sequential from frame 1, so the pinned snapshot blocks reclamation + // of everything written after it: the file must grow while pinned. + reader = rawOpen(dir); + reader.db.exec("BEGIN"); + reader.db.prepare("SELECT count(*) FROM sessions").get(); + + for (let i = 0; i < 20; i++) ingest(store, `pin${i}`, 512 * KiB, 20); + const pinnedSize = walSize(dir); + assert.ok(pinnedSize > WAL_CAP, `pinned growth must exceed the cap (${pinnedSize} bytes)`); + + reader.db.exec("ROLLBACK"); + reader.close(); + reader = undefined; + + // Drain enough ingests past the checkpoint cadence: the first + // COMPLETED checkpoint after the release truncates the file back to + // the cap (autocheckpoint or the periodic PASSIVE — either way the + // reclamation now actually happens). + for (let i = 0; i < 2 * CHECKPOINT_EVERY_N; i++) ingest(store, `post${i}`, 100 * KiB, 4); + assert.ok( + walSize(dir) <= WAL_CAP + CAP_SLACK, + `WAL must come back under the cap after the reader released (left ${walSize(dir)} bytes, cap ${WAL_CAP})`, + ); + + const view = loadLegacyView({ + session: (() => { + const row = store.getSessionRow("pin0"); + assert.ok(row, "pin0 row must exist"); + return row; + })(), + messages: store.getMessages("pin0"), + blocks: store.getBlocks("pin0"), + deadRefs: store.getDeadRefs("pin0"), + refs: store.getRefs("pin0"), + }); + assert.equal(view.lastMessages?.length, 20, "pinned-phase rows must round-trip"); + } finally { + reader?.close(); + store?.close(); + rmBestEffort(dir); + } +}); + +test("#3250: a configured cap flows open() → journal_size_limit (custom limit honored end-to-end)", () => { + const dir = tmpdir("bili-3250-customcap-"); + let store: UnifiedStore | undefined; + try { + const customCap = 1 * MiB; + store = UnifiedStore.open(indexDb(dir), undefined, { walJournalSizeLimitBytes: customCap }); + // A 3 MiB burst overshoots the CUSTOM cap in-flight... + ingest(store, "big", 3 * MiB, 120); + assert.ok(walSize(dir) > customCap, "the burst must overshoot the custom cap in-flight"); + // ...and once the WAL crosses the autocheckpoint window (1000 pages), the + // writer's own completed checkpoint truncates back to the CONFIGURED cap, + // not the 8 MiB default: with the default the file would sit at the + // ~4.4 MiB high-water mark here, never touching 1 MiB. (A lone writer's + // cadence-tick PASSIVE can backfill without shrinking the file — its own + // tail read-lock blocks the truncate — so the autocheckpoint plateau is + // the observable proof the configured value reached the pragma.) + for (let i = 2; i <= 10; i++) ingest(store, `small${i}`, 128 * KiB, 8); + assert.ok( + walSize(dir) <= customCap + CAP_SLACK, + `WAL must be truncated to the configured cap at the autocheckpoint (left ${walSize(dir)} bytes, cap ${customCap})`, + ); + assert.equal(store.getMessages("big").length, 120, "all burst rows must land"); + // A clean exit converges to zero whatever the cap. + store.close(); + store = undefined; + assert.ok(walSize(dir) <= CAP_SLACK, `close() must converge the WAL (left ${walSize(dir)} bytes)`); + } finally { + store?.close(); + rmBestEffort(dir); + } +}); diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index 1cdbc15b65..6595b846fb 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -11,6 +11,7 @@ import { canonicalize, sha256 } from "./golden-canonical.ts"; import { planIngest, loadLegacyView, toLegacyLike, type LegacySessionLike } from "../src/storage/ingest.ts"; import { UnifiedStore } from "../src/storage/store.ts"; import { resolveStorageMode } from "../src/storage/shadow.ts"; +import { resolveStorageWalJournalSizeLimit } from "../src/knobs.ts"; import { setPreferredSqliteEngineForTests, type SqliteEngineName } from "../src/storage/driver.ts"; /** @@ -372,6 +373,29 @@ test("resolveStorageMode file tier: storage.unified applies when the env is abse assert.equal(resolveStorageMode({ BILI_STORAGE_UNIFIED: "legacy" }, "full"), "legacy"); }); +test("resolveStorageWalJournalSizeLimit: env wins, garbage falls through, file tier, default (#3250)", () => { + const DEFAULT = 8 * 1024 * 1024; // contract literal, mirrors tests/issue3250-wal-checkpoint.test.ts + assert.equal(resolveStorageWalJournalSizeLimit({}), DEFAULT, "absent everywhere must give the default cap"); + assert.equal( + resolveStorageWalJournalSizeLimit({ BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES: "67108864" }, DEFAULT), + 67108864, + "env tier wins over the file tier", + ); + assert.equal( + resolveStorageWalJournalSizeLimit({ BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES: "1048576" }, 67108864), + 1048576, + "env tier wins even when smaller than the file tier", + ); + for (const bad of ["garbage", "", "0", "-1", "1.5"]) { + assert.equal( + resolveStorageWalJournalSizeLimit({ BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES: bad }, 4096), + 4096, + `env=${JSON.stringify(bad)} must fall through to the file tier (never crash boot)`, + ); + } + assert.equal(resolveStorageWalJournalSizeLimit({}, 2097152), 2097152, "file tier applies when the env is absent"); +}); + test("storage unified FULL mode: read path serves from the index with organic import + GC parity", async () => { const meta = CASE_META["folded"]!; const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-full-")); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 7693dac5bc..06ac46f27c 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -193,6 +193,13 @@ entries: group: process desc_en: "Phase 3 envelope switch (#2960): off stops writing the legacy JSON session envelope while storage.unified is full - the SQLite index becomes the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Only honored in effective-full mode: under shadow/legacy (or with encryption active) the request is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings. Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. Rollback: unset the knob - writing resumes, index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session content-store sidecar keeps being written either way. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes)." desc_zh: "Phase 3 信封开关(#2960):off 时若 storage.unified 为 full 则不再写 legacy JSON 会话信封 —— SQLite 索引成为唯一持久化副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。仅在有效 full 模式下生效:shadow/legacy(或加密激活)下请求会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。回滚:去掉该开关 —— 写入恢复,纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 content-store 边车无论开关如何都照写。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。" + - key: storage.walJournalSizeLimitBytes + type: number + default: "8388608" + env: BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES + group: process + desc_en: "WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob." + desc_zh: "index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。" - key: sessions type: object default: "{} (all defaults)"