Skip to content

fix: 长驻进程里 index.db-wal 永不回收 —— journal_size_limit + open/close TRUNCATE + 周期 PASSIVE (#3250) - #3255

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-15_fix-3250-wal-checkpoint
Oct 15, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-15_fix-3250-wal-checkpoint

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Closes #3250

修什么

长驻进程里 <state>/sessions/index.db-wal 单调增长、永不回收(现场:55 MiB 平直)。STORAGE_PRAGMAS 没有 journal_size_limit,运行时路径上也没有任何显式 wal_checkpoint 调用。

关键机制(本地实测钉死):passive/auto checkpoint 把帧搬进主 db 并重置写偏移,但从不缩小 WAL 文件——没有 journal_size_limit 时文件永远停在历史高水位。另外,连接在仍是最后写者期间无法 checkpoint 自己刚写的帧(commit 后立刻 same-conn PASSIVE 一帧都收不回来),所以回收依赖「后续写入 + 完成一次 checkpoint」的循环。

修法(三层回收)

  1. journal_size_limit(默认 8 MiB)(src/storage/schema.ts STORAGE_PRAGMAS 追加,值可配,见下方 Config surface):每次完成的 checkpoint 把文件截回上限。per-connection 设置,共享目录(Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 拓扑)下每个进程各管自己的副本;只删死尾空间,不动其他进程的活帧。
  2. open 后 / close 前各一次 TRUNCATE(src/storage/store.ts UnifiedStore.open/close):open 收敛非正常退出(SIGKILL/断电永远到不了 close)留下的残留,缩短下一次崩溃的恢复重放;干净退出则什么都不留给下次启动。
  3. 每 20 次成功 ingest 一次 PASSIVE(WAL_CHECKPOINT_EVERY_INGESTS):连续 ingest 下保证回收——外部读快照(dashboard/export/第二 lane)pin 住帧期间 backfill 停在该快照处,只靠 autocheckpoint 不保证回收。

所有 checkpoint 调用独立 try/catch:失败只记日志(open/close warn、周期 debug——周期日志带 busy/log/checkpointed 结果,调参的人能直接看到 tick 是否真的在收帧),绝不影响 ingest/open/close 契约——sidecar「绝不拖垮 legacy」策略不变;多进程共享目录下 TRUNCATE busy 返回行而非抛错,自然跳过。

一个实测细节(影响预期、不影响正确性):单进程长驻里,自己触发的 tick checkpoint 可能只完成 backfill 而缩不了文件(自身尾部读锁挡住 truncate 步骤),实际缩小等下一次完成的 checkpoint(autocheckpoint 窗口)或外部连接参与;单进程最坏稳态高水位 ≈ cap + 一个 autocheckpoint 窗口(~4 MiB)。多进程共享目录(生产形态)与干净退出则立即收敛。

行为变更披露(old → new)

项 旧 新
index.db-wal 大小 随运行时长单调增长、永不缩小(现场 55 MiB 平直) 完成的 checkpoint 之后有界(默认 ≈8 MiB;单进程长驻最坏 ≈ cap + 4 MiB;两次回收之间可暂时超出)
启动 IO 无 checkpoint 一次性 TRUNCATE(ms 级;并发活动 busy 时自动跳过)
干净退出 留全量 WAL,下次崩溃恢复全部重放 WAL 截到零,下次启动无需重放
ingest 热路径 无 每 20 次保存多一次 PASSIVE(非阻塞,busy 即返回)

持久化语义不变:synchronous = NORMAL 未动(#3135 的掉电丢提交 INFO 是另一机制,不在本 PR);schema version 保持 5;无 wire/persistence 格式变更。

Config surface(owner 批准的新增)

owner 决策(#3250 thread,2026-10-15):「直接搞一个参数吧 storage下面?」——多会话宿主可能想要更大的回收底线。新增:

项 值
配置键 storage.walJournalSizeLimitBytes(正整数字节,可选)
env BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES(优先于配置文件)
默认 8388608(8 MiB,即本 PR 原硬编码值;不配置 = 保持本 PR 行为)
生效时机 store open 时(lane 启动);热改需重启(与 unified/jsonWriter/sqliteAutoDownload 同生命周期)
非法值 env 非正整数静默落到配置文件层 → 默认,boot 不崩
诊断 runtime-status 诊断块报两层取值

为何不用既有面:既有面没有任何 per-store SQLite pragma 拨盘;providers 表管 provider 认证/路由、compress 层级管压缩策略,都是错误领域;放 storage 段与 unified/jsonWriter/sqliteAutoDownload 三个兄弟同构。兼容:纯增量可选字段,老配置零迁移,不设 = 默认 8 MiB。

两个常量(WAL_JOURNAL_SIZE_LIMIT 8 MiB、WAL_CHECKPOINT_EVERY_INGESTS 20)均为模块私有(unused-export gate);回归测试用独立字面量钉契约值——任一变红即测试失败。

测试

新增 tests/issue3250-wal-checkpoint.test.ts(5 例;先在 pristine master 上证明 RED——断言全部以具体字节数失败,修复后 GREEN):

  1. close() 把 WAL 截到零(外部连接保活 db,证明收敛是 close 本身做的);
  2. 非正常退出留下脏 WAL 后,open() 收敛且数据完整;
  3. 12 MiB burst 超上限 → 下一个 cadence tick 后截回上限(writer-lock 机制:burst 自身不能自回收,属正常 WAL 行为而非 bug,断言因此放在第 20 次 ingest 后);
  4. pinned reader 生命周期:pin 期间增长必须超上限(生产形态复现),释放后回收;
  5. 配置 cap 端到端:UnifiedStore.open(dir, undefined, { walJournalSizeLimitBytes: 1 MiB }) → burst 超自定义上限后,autocheckpoint 把文件截回 1 MiB 而非 8 MiB 默认(证明 config → knobs → open → pragma 链路生效),close() 再收敛到零。

tests/storage-unified.test.ts 新增 resolveStorageWalJournalSizeLimit 分层用例(env 优先 / 非法 env 穿透 / 文件层 / 默认)。

本地定向验证:新 suite 5/5 绿;storage 区 20 个测试文件 152 pass / 0 fail / 1 skip(既有 skip);typecheck 干净;unused-export gate 干净。全量回归交给 CI。

已 rebase 到当前 master(fc0f6c1,含 #3233 wire-exit 时序相对化修复——上一 head CI 的两个无关红之一正出自该处);分支同时保留 review-fix commit(cadence 常量模块私有化)。

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-15_fix-3250-wal-checkpoint (58e8689)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3759-pr.3255.f80da72

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3255.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 open+closed issue/PR 全量检索(WAL / checkpoint / journal_size_limit / index.db-wal),「单进程 WAL 无文件级回收触发点」这一主题只有本 PR 及其来源 issue #3250。与相邻项逐一比对:

这是 [PR] 镜像(GitHub PR #3255,另一 agent 会话已完成的工作),我不重做——拉取分支独立核对代码、审原 issue 方向、回归清点、测试有效性、CI 状态。审查进行中,结论按结构化八问随后补上。

ranxianglei pushed a commit that referenced this pull request Oct 15, 2026
…x for #3255)

Nothing imports it — the #3250 test suite pins the cadence with a literal
on purpose (assertions must hold even if the constant drifts), so the
export survived only via a comment mention under the textual unused-export
gate. Align it with its sibling WAL_JOURNAL_SIZE_LIMIT, which the previous
commit already made module-private for the same gate reason.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审查结论:可合(待 CI 测试分片转绿)。 查重:不重复(首条回复已给比对)。本 PR 是另一 agent 会话的完成工作,我独立拉分支核对、复现 RED/GREEN、清点回归面,未重做。以下为独立审核的八问。

结构化八问

  1. 是什么问题:长驻 bili 进程里 index.db-wal 随运行时长单调增长、永不缩小(现场 55 MiB 平直),白占磁盘,且 SIGKILL/断电后下次启动要重放全部残留帧。
  2. 什么原因导致的:两层机制叠加——① sqlite 的 passive/auto checkpoint 只把帧搬进主库并前移写偏移,从不对 WAL 文件做 ftruncate,没有 journal_size_limit 时文件永远钉在历史高水位;② 运行时路径上没有任何显式回收触发点(sidecar 只在 SIGTERM/SIGINT 走 close(),非正常退出到不了),且「连接在仍是最后写者期间无法 checkpoint 自己刚写的帧」,所以连 autocheckpoint 都救不了 burst 后的残留。用户报告的「文件增长」是表面现象,根因是回收触发点缺失 + 文件上限缺失,PR 恰好修在这两层,不是治标(没有「启动时删文件」「cron 限盘」这类遮罩)。
  3. 影响范围与严重程度:所有启用 unified shadow 存储的长驻 lane(dsh/opencode/pi 等)。属资源泄漏类——磁盘占用 ∝ uptime + 崩溃恢复重放时间拉长;不影响数据正确性与可用性;不修则每个长会话机器的 index.db-wal 永久膨胀。
  4. 怎么修复的:三层回收——journal_size_limit = 8 MiB(每次完成的 checkpoint 把文件截回上限;per-connection,共享目录拓扑下每进程各管各的副本);open 后/close 前各一次 wal_checkpoint(TRUNCATE)(收敛非正常退出残留 + 干净退出不留重放对象);每 20 次成功 ingest 一次 PASSIVE(外部读快照 pin 住帧期间 backfill 停摆时保证仍有周期性回收尝试)。所有 checkpoint 调用独立 try/catch:失败只记日志(open/close warn、周期 debug),绝不拖垮 open/close/ingest 契约。
  5. 是否有回归问题:无。逐项清点的既有行为:open 契约(成功→可用 store / 失败→关句柄+抛错)两条路径均未变,新 checkpoint 隔离在其后、不可能抛;close 契约——调用方 persist.ts flushAll/closeQuietly 本就是 best-effort 关机路径,close 不比之前多抛任何东西;ingestSession 事务语义原样(delta rollback 路径不计数不 checkpoint;PASSIVE 只在事务提交返回后跑,非阻塞,busy 立即返回行而非抛错);STORAGE_PRAGMAS 仅追加一行,synchronous=NORMAL/busy_timeout/foreign_keys/journal_mode 未动(Storage Subsystem Wrap-up Checklist (#3130 Review): Dropping the last tick upon self-disable; BILI_PERSIST=0 falsely triggering a "Storage Degraded" banner; storageMode stamp mismatched with the actual enabled mode; and 3 more (6 items total) | 存储子系统收尾清单… #3135 掉电丢提交是另一机制,未被触碰);schema version 保持 5,持久化格式零变更;Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 多进程共享目录:活读快照需要的帧会同时挡住 checkpoint 与截断(SQLite 语义),TRUNCATE busy 按既有 busy_timeout=5000 等待后跳过——最坏是回收延迟,不存在损坏路径,测试 4 专门钉了这个形态。
  6. 是否破坏缓存:不涉及。WAL 文件是 SQLite 磁盘层内部件,与 prefix cache、增量游标、去重键等任何失效策略无关。
  7. 是否改了设计方向:否。不动两种压缩模式、wire、持久化语义;纯存储层资源生命周期管理。
  8. 是否引入了新的配置:无新增 env/config/CLI。8 MiB 与 20-ingest 是模块私有常量;测试用字面量钉死且刻意不从 src 导入(常量漂移时断言仍成立),不是用户拨盘。

验证证据(本地定向,全量交给 CI)

  • RED/GREEN 独立复现:pristine master(effd544)上 4 例全部以具体字节数失败——close/open 残留 2,381,392 B、burst 残留 13,192,272 B(上限 8,388,608)、pinned 释放后残留 15,103,952 B;PR 分支上 4/4 绿(~1.5s)。断言有牙,bug 复发必被抓。
  • storage 区定向(20 个文件):151 pass / 2 fail / 1 skip(既有 skip)。2 个 fail 均在 tests/proxy-persist.test.ts(scheduleSave debounces…、flushSync writes immediately…,同为 4 !== 1)——在 pristine master 上逐字复现同样失败 = 本沙箱预存环境问题,与本 PR 无关,以 CI 为准。
  • typecheck 干净;unused-exports gate 干净;PR 描述的行为变更披露表与 diff 逐项核对一致(WAL 无上界→≈8MiB 有界;启动 +1 次 ms 级 TRUNCATE;干净退出 WAL→0;ingest 热路径每 20 次保存 +1 次非阻塞 PASSIVE)。
  • CI(head ae06f4b):static gates / guards(kernel-guard、pi-subagents-guard、config-surface-guard、docs-sync、lock-sync、unused-exports)/ hermetic registry e2e / real-client lanes 全绿;12 个 test 分片在跑。

审查顺手修(已直接提交 PR 分支,独立标注 commit edbfbd6)

WAL_CHECKPOINT_EVERY_INGESTS 去掉了 export:没有任何代码 import 它(测试刻意用字面量),它之所以逃过 unused-exports gate 仅因 gate 是文本计数、测试头注释里提了这个名字——同文件的姊妹常量 WAL_JOURNAL_SIZE_LIMIT 上一个 commit(ae06f4b)已因同一 gate 改为模块私有,这里对齐处理。零行为变更(值与用法不变),已 push,CI 在新 head 重跑中。

Auto-merge 门槛评估(§7.4)

单模块 scoped ✓ / 回归测试 RED→GREEN ✓ / rebased-head 本地 typecheck+build 干净、全量看 CI ✓ / 无 config schema·持久化格式·wire 变更(schema v5 未动)✓ / 纯 fix ✓ / diff 干净(3 文件 330 行,每行相关)✓ / Fixes #3250 ✓ / 不触碰 load-bearing infra ✓ —— 满足 auto-merge 条件;合并仍归人:**#3255 分片转绿后可合)。

awork added 4 commits October 15, 2026 15:09
…x for #3255)

Nothing imports it — the #3250 test suite pins the cadence with a literal
on purpose (assertions must hold even if the constant drifts), so the
export survived only via a comment mention under the textual unused-export
gate. Align it with its sibling WAL_JOURNAL_SIZE_LIMIT, which the previous
commit already made module-private for the same gate reason.
storage.walJournalSizeLimitBytes (env BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES, default 8 MiB) now flows config -> knobs -> UnifiedStore.open -> PRAGMA journal_size_limit, per owner request on #3250 (multi-session hosts may want a larger reclamation floor). Additive optional field: unset configs keep today's behavior.
@ranxianglei
ranxianglei force-pushed the 2026-10-15_fix-3250-wal-checkpoint branch from edbfbd6 to d97bf3e Compare October 15, 2026 07:13
…enerator (#3250)

CONFIGURATION.md / zh-CN / website docs are generated artifacts of website/config-reference/*.yaml (tools/gen-config-docs.mjs); the previous commit hand-edited the artifact and skipped the seed, tripping docs-sync + docs-coverage + config-surface-guard. Adds the seed entry and regenerates.
@ranxianglei
ranxianglei merged commit 42350e8 into master Oct 15, 2026
40 of 42 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.444-master.42350e8 (rolling master tag).

npm install -g billion-context@0.0.444-master.42350e8  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 15, 2026
…arsing in unified storage modes (#3199)

/__bili/overview (and /__bili/sessions) froze the event loop for minutes on
cold instances with large session corpora in unified storage: shadow mode
deliberately bypassed the index fast path (rows may lag unrecorded files),
so the first list hit decoded the entire corpus on the main thread, and
boot parsed the whole corpus before the listener started — feeding the
host-side respawn churn (#3044 family).

Two mechanisms, one failure domain (main-thread cold-corpus cost):

1. Shadow list surfaces now read the index as a BASELINE and reconcile
   against the file tree inside refreshIndex: fresh rows are adopted
   without decoding (file mtime within 5 min of row updated_at — normal
   writes always land after their row), files newer than their row or never
   recorded are decoded individually, ghost rows drop because only files
   produce entries. Full mode stays authoritative (vanished-file sessions
   remain listed); legacy keeps the plain walk. Cold-start cost drops from
   "decode the whole corpus" to one SQL + one stat per file + O(delta)
   decodes.

2. Boot in unified modes (shadow or full) parses only a bounded window of
   the newest files (mtime desc) before listen; label aliases are seeded
   synchronously from index rows and the remaining alias/canonical
   bookkeeping finishes in the background (backgroundScanComplete()).
   Survivor arbitration still uses the original savedAt-desc trim, so the
   #404 freshness semantics hold. Legacy mode keeps the synchronous full
   parse verbatim.

WAL reclamation (issue suggestion 3a) landed independently on master via
#3250/#3255 (open/close TRUNCATE checkpoints + periodic PASSIVE +
configurable cap) — this PR no longer carries its own checkpoint code.

Behavior changes (disclosed per owner policy):
- shadow-mode web lists may lag bypass-written files by up to 5 minutes,
  display-only (detail views always read the file; any activity re-ingests);
- unified-mode boot readiness = warm-window parse only; label aliases for
  UNRECORDED files fill post-boot in the background (recorded labels are
  seeded synchronously from the index).

No new config surface (reuses sessions.warmMax / sessions.warmMaxBytes).
Kernel untouched. Plugin/proxy wire modes unaffected (admin/web surface).

Tests: new tests/issue3199-overview-cold-scan.test.ts pins all paths;
targeted suites green locally (web-sessions, storage-unified, store,
persist-zstd, persist-encryption, bounded-storage, issue404,
identity-registry, search-context-cross-session); full verdict via remote CI.
@github-actions github-actions Bot mentioned this pull request Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant