Repository navigation
fix: 长驻进程里 index.db-wal 永不回收 —— journal_size_limit + open/close TRUNCATE + 周期 PASSIVE (#3250) - #3255
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3759-pr.3255.f80da72This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3255.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…x for #3255) Nothing imports it — the #3250 test suite pins the cadence with a literal on purpose (assertions must hold even if the constant drifts), so the export survived only via a comment mention under the textual unused-export gate. Align it with its sibling WAL_JOURNAL_SIZE_LIMIT, which the previous commit already made module-private for the same gate reason.
(ai: vllm-qwen/qwen3.8-27b) 审查结论:可合(待 CI 测试分片转绿)。 查重:不重复(首条回复已给比对)。本 PR 是另一 agent 会话的完成工作,我独立拉分支核对、复现 RED/GREEN、清点回归面,未重做。以下为独立审核的八问。 结构化八问
验证证据(本地定向,全量交给 CI)
审查顺手修(已直接提交 PR 分支,独立标注 commit edbfbd6)
Auto-merge 门槛评估(§7.4)单模块 scoped ✓ / 回归测试 RED→GREEN ✓ / rebased-head 本地 typecheck+build 干净、全量看 CI ✓ / 无 config schema·持久化格式·wire 变更(schema v5 未动)✓ / 纯 fix ✓ / diff 干净(3 文件 330 行,每行相关)✓ / Fixes #3250 ✓ / 不触碰 load-bearing infra ✓ —— 满足 auto-merge 条件;合并仍归人:**#3255 分片转绿后可合)。 |
…x for #3255) Nothing imports it — the #3250 test suite pins the cadence with a literal on purpose (assertions must hold even if the constant drifts), so the export survived only via a comment mention under the textual unused-export gate. Align it with its sibling WAL_JOURNAL_SIZE_LIMIT, which the previous commit already made module-private for the same gate reason.
storage.walJournalSizeLimitBytes (env BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES, default 8 MiB) now flows config -> knobs -> UnifiedStore.open -> PRAGMA journal_size_limit, per owner request on #3250 (multi-session hosts may want a larger reclamation floor). Additive optional field: unset configs keep today's behavior.
edbfbd6 to
d97bf3e
Compare
…enerator (#3250) CONFIGURATION.md / zh-CN / website docs are generated artifacts of website/config-reference/*.yaml (tools/gen-config-docs.mjs); the previous commit hand-edited the artifact and skipped the seed, tripping docs-sync + docs-coverage + config-surface-guard. Adds the seed entry and regenerates.
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.444-master.42350e8 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…arsing in unified storage modes (#3199) /__bili/overview (and /__bili/sessions) froze the event loop for minutes on cold instances with large session corpora in unified storage: shadow mode deliberately bypassed the index fast path (rows may lag unrecorded files), so the first list hit decoded the entire corpus on the main thread, and boot parsed the whole corpus before the listener started — feeding the host-side respawn churn (#3044 family). Two mechanisms, one failure domain (main-thread cold-corpus cost): 1. Shadow list surfaces now read the index as a BASELINE and reconcile against the file tree inside refreshIndex: fresh rows are adopted without decoding (file mtime within 5 min of row updated_at — normal writes always land after their row), files newer than their row or never recorded are decoded individually, ghost rows drop because only files produce entries. Full mode stays authoritative (vanished-file sessions remain listed); legacy keeps the plain walk. Cold-start cost drops from "decode the whole corpus" to one SQL + one stat per file + O(delta) decodes. 2. Boot in unified modes (shadow or full) parses only a bounded window of the newest files (mtime desc) before listen; label aliases are seeded synchronously from index rows and the remaining alias/canonical bookkeeping finishes in the background (backgroundScanComplete()). Survivor arbitration still uses the original savedAt-desc trim, so the #404 freshness semantics hold. Legacy mode keeps the synchronous full parse verbatim. WAL reclamation (issue suggestion 3a) landed independently on master via #3250/#3255 (open/close TRUNCATE checkpoints + periodic PASSIVE + configurable cap) — this PR no longer carries its own checkpoint code. Behavior changes (disclosed per owner policy): - shadow-mode web lists may lag bypass-written files by up to 5 minutes, display-only (detail views always read the file; any activity re-ingests); - unified-mode boot readiness = warm-window parse only; label aliases for UNRECORDED files fill post-boot in the background (recorded labels are seeded synchronously from the index). No new config surface (reuses sessions.warmMax / sessions.warmMaxBytes). Kernel untouched. Plugin/proxy wire modes unaffected (admin/web surface). Tests: new tests/issue3199-overview-cold-scan.test.ts pins all paths; targeted suites green locally (web-sessions, storage-unified, store, persist-zstd, persist-encryption, bounded-storage, issue404, identity-registry, search-context-cross-session); full verdict via remote CI.
Closes #3250
修什么
长驻进程里
<state>/sessions/index.db-wal单调增长、永不回收(现场:55 MiB 平直)。STORAGE_PRAGMAS没有journal_size_limit,运行时路径上也没有任何显式wal_checkpoint调用。关键机制(本地实测钉死):passive/auto checkpoint 把帧搬进主 db 并重置写偏移,但从不缩小 WAL 文件——没有 journal_size_limit 时文件永远停在历史高水位。另外,连接在仍是最后写者期间无法 checkpoint 自己刚写的帧(commit 后立刻 same-conn PASSIVE 一帧都收不回来),所以回收依赖「后续写入 + 完成一次 checkpoint」的循环。
修法(三层回收)
journal_size_limit(默认 8 MiB)(src/storage/schema.tsSTORAGE_PRAGMAS 追加,值可配,见下方 Config surface):每次完成的 checkpoint 把文件截回上限。per-connection 设置,共享目录(Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 拓扑)下每个进程各管自己的副本;只删死尾空间,不动其他进程的活帧。src/storage/store.tsUnifiedStore.open/close):open 收敛非正常退出(SIGKILL/断电永远到不了 close)留下的残留,缩短下一次崩溃的恢复重放;干净退出则什么都不留给下次启动。WAL_CHECKPOINT_EVERY_INGESTS):连续 ingest 下保证回收——外部读快照(dashboard/export/第二 lane)pin 住帧期间 backfill 停在该快照处,只靠 autocheckpoint 不保证回收。所有 checkpoint 调用独立 try/catch:失败只记日志(open/close warn、周期 debug——周期日志带 busy/log/checkpointed 结果,调参的人能直接看到 tick 是否真的在收帧),绝不影响 ingest/open/close 契约——sidecar「绝不拖垮 legacy」策略不变;多进程共享目录下 TRUNCATE busy 返回行而非抛错,自然跳过。
一个实测细节(影响预期、不影响正确性):单进程长驻里,自己触发的 tick checkpoint 可能只完成 backfill 而缩不了文件(自身尾部读锁挡住 truncate 步骤),实际缩小等下一次完成的 checkpoint(autocheckpoint 窗口)或外部连接参与;单进程最坏稳态高水位 ≈ cap + 一个 autocheckpoint 窗口(~4 MiB)。多进程共享目录(生产形态)与干净退出则立即收敛。
行为变更披露(old → new)
持久化语义不变:
synchronous = NORMAL未动(#3135 的掉电丢提交 INFO 是另一机制,不在本 PR);schema version 保持 5;无 wire/persistence 格式变更。Config surface(owner 批准的新增)
owner 决策(#3250 thread,2026-10-15):「直接搞一个参数吧 storage下面?」——多会话宿主可能想要更大的回收底线。新增:
storage.walJournalSizeLimitBytes(正整数字节,可选)BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES(优先于配置文件)为何不用既有面:既有面没有任何 per-store SQLite pragma 拨盘;providers 表管 provider 认证/路由、compress 层级管压缩策略,都是错误领域;放 storage 段与 unified/jsonWriter/sqliteAutoDownload 三个兄弟同构。兼容:纯增量可选字段,老配置零迁移,不设 = 默认 8 MiB。
两个常量(
WAL_JOURNAL_SIZE_LIMIT8 MiB、WAL_CHECKPOINT_EVERY_INGESTS20)均为模块私有(unused-export gate);回归测试用独立字面量钉契约值——任一变红即测试失败。测试
新增
tests/issue3250-wal-checkpoint.test.ts(5 例;先在 pristine master 上证明 RED——断言全部以具体字节数失败,修复后 GREEN):UnifiedStore.open(dir, undefined, { walJournalSizeLimitBytes: 1 MiB })→ burst 超自定义上限后,autocheckpoint 把文件截回 1 MiB 而非 8 MiB 默认(证明 config → knobs → open → pragma 链路生效),close() 再收敛到零。tests/storage-unified.test.ts新增resolveStorageWalJournalSizeLimit分层用例(env 优先 / 非法 env 穿透 / 文件层 / 默认)。本地定向验证:新 suite 5/5 绿;storage 区 20 个测试文件 152 pass / 0 fail / 1 skip(既有 skip);typecheck 干净;unused-export gate 干净。全量回归交给 CI。