Skip to content

[PR] storage: unified index GC — release rows on file sweep + manual reconcile/reclaim/VACUUM pass - #3466

Closed
ranxianglei wants to merge 5 commits into
masterfrom
2026-10-17_storage-gc-step1
Closed

ranxianglei wants to merge 5 commits into
masterfrom
2026-10-17_storage-gc-step1

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Closes #3463

Scope: step 1 of the issue (refcount reclamation paths + manual GC entry point). Step 2 (archiveAfterDays automation + incremental vacuum) is deliberately NOT here — the issue sequences it explicitly ("提供手工入口先跑一遍,验证收益后再谈自动化"); it waits on the measured benefit + your go. Its design is fully settled in #2671 §7 item 4 and ready to follow.

What the audit found

The refcount machinery already exists and is tested (content.refcount maintained by ingestSession/deleteSession, pinned by tests/storage-engine-proofs.test.ts). What was actually missing:

  1. The file sweep never reached the unified store. gcSessionFiles ([Feature] Default state file cleanup strategy #1082) deleted only the legacy JSON file via rm(file); under shadow/full modes every index.db row of a swept session — and every CAS refcount behind it — survived forever, so index.db grew without bound even while files were being swept away. Fix: the sweep now calls SessionStore.removeUnifiedSession(id) BEFORE rm(file) (index-first-then-file, same order as removeLegacyFile; row removal is idempotent so a failed file rm self-heals on the next sweep).
  2. No operator entry point to reconcile / reclaim / shrink. Added UnifiedStore.runGc() + POST /__bili/storage/gc (loopback-only, like every other __bili route):
    • fix (default true): reconciles each CAS row's booked refcount against the ACTUALLY counted references across all five source columns (messages.content_hash, blocks.summary_hash, blocks.one_hash, ccr_entries.content_hash, checkpoints.state_hash — the exhaustive set per schema v5). Over-booked (leaked) refs become reclaimable; under-booked (premature-delete hazard) refs are corrected BEFORE anything is deleted.
    • reclaim: DELETE FROM content WHERE refcount <= 0 — provably-unreferenced rows only.
    • vacuum (default true): full VACUUM returns freed pages to the OS (exclusive lock + ~2x transient space — an idle-time ops action; periodic/incremental automation belongs to step 2).
    • Response reports session count, reclaimed rows/bytes, drift-class counts, and before/after file size. Every step is safe by construction (fix only moves counts toward reality, reclaim only touches zero-ref rows, VACUUM never touches live data).

Behavior changes (disclosed)

  • NEW endpoint POST /__bili/storage/gc. No new config keys / env vars / CLI flags — config surface unchanged.
  • With BILI_SESSION_GC=1 under unified modes, the [Feature] Default state file cleanup strategy #1082 sweep now ALSO releases the unified rows of each file it deletes (previously it left them behind — that is the bug this fixes). Legacy-only mode is untouched (no sidecar → the call is a no-op).
  • No wire/persistence-format/schema change (STORAGE_SCHEMA_VERSION stays 5). kernel/ untouched.

Verification

  • NEW tests/storage-gc.test.ts (5 cases): over-booked drift detected + fixed; under-booked hazard corrected before reclaim (referenced row survives); orphaned zero-ref garbage reclaimed; VACUUM provably shrinks the file after mass deletion (strict size decrease); end-to-end PARITY — aged legacy file + unified index rows, sweep → file gone AND index row gone AND exclusive CAS row gone, shared CAS row survives.
  • Targeted suites green locally: session-gc/state-gc/proxy-orphan-gc/storage-engine-proofs/storage-ext/storage-replicas (56 pass); storage-unified (the Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 harness)/storage-save-path/storage-json-writer-off/web-storage-banner/storage-ingest-watermark/storage-review-fixes (43 pass, 1 pre-existing environmental skip: better-sqlite3 not installed on this runner). Full-suite verdict = remote CI on this head.
  • Measurement (synthetic corpus — the live ~/.local/state/billion-context/sessions/index.db is not reachable from this sandbox): 300 live sessions + 100 dead sessions whose rows had been removed WITHOUT releasing CAS refs (simulating the pre-fix sweep-leak class) → one runGc({fix:true, vacuum:true}): index.db 78.9 MB → 59.26 MB (−24.9%) in 0.3 s; 100 orphaned refs corrected, 100 rows / 19.54 MB reclaimed; remainder ≈ exactly the live sessions' content. On the reporter's 637 MB machine the first-pass gain depends on how much dead material the sweep covers — the big lever there is turning ON BILI_SESSION_GC=1 (opt-in) so deletions finally release the index, then one GC pass; step 2 automates the rest.

Complementary write-side defect: #3748 (NOT fixed by this PR)

External measurement (@yanghuan, #3748; bili 0.1.201, shadow default, dsh lane, Windows) shows a second, dominant producer of refcount = 0 rows on image-heavy sessions: the host re-mints message ids (h_…_N) and a per-pass wireIndex on every re-projection, and planIngest hashes the WHOLE wrapper (JSON.stringify(m) — volatile fields included, src/storage/ingest.ts:150), so byte-identical payloads never share a CAS row (measured 11.06× redundancy; 88.6% of content bytes are rc=0 orphans produced continuously by re-ingest, ~758 MiB/h growth).

Consequence for reading this PR: the GC endpoint below reclaims exactly those rc=0 rows (one pass ≈ 2.2 GB of the reporter's 2.5 GiB file), but the minting continues until #3748 lands — do not read one green GC run as "growth solved". The write-side fixes proposed in #3748 (normalized CAS key / idempotent upsert / shadow not mirroring raw image bytes) are behavior changes in owner-decision territory and are tracked there; this PR deliberately does not touch the write path, and the CONFIGURATION.md disk-expectation correction rides with #3748 so doc and code land together.

Known residual (noted, out of scope)

Docs

CONFIGURATION.md en+zh: endpoint + disk-expectation text in the BILI_STORAGE_UNIFIED section.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_storage-gc-step1 (851e349)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4430-pr.3466.80e1bdf

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3466.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 本 PR 是 #3463(统一存储只进不出,index.db 637 MB)第一步的唯一实现——检索 open+closed 的 issue/PR(refcount / VACUUM / storage gc / archiveAfterDays)无并行工作;最接近的已合项都不是同一机制(#2745 GC-proof pins = checkpoint 保活,#1189 CCR 文件侧 GC)。

在飞重叠提示(非重复,但合入顺序需注意):#3432(StorageBackend 契约重构)与本 PR 同改 src/storage/store.ts,#3419(storage-races)同改 src/persist.ts——三者在热文件上有语义冲突面,review 里会核对。

开始独立审核:拉分支通读 diff → 对照 #3463 验收标准 → 既有行为回归清点 → 测试有效性 → CI 状态。结论按八问回本线程。

@ranxianglei
ranxianglei force-pushed the 2026-10-17_storage-gc-step1 branch from cc846a0 to be7f18c Compare October 17, 2026 02:37
ranxianglei pushed a commit that referenced this pull request Oct 17, 2026
Review follow-up on the GC route: a body of 'null' crashed the handler with a
TypeError (500 instead of 400), and other non-object bodies (arrays, strings,
numbers) were silently accepted as 'use defaults'. Non-object JSON is now
rejected with the same 400 as wrong-typed options, and a new HTTP-level case
pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged
run -> 200 with fix+vacuum defaults ON.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论:可合并 ✅(CI 全绿 @ 1ea36367d)

结构化八问

  1. 是什么问题:统一存储模式(BILI_STORAGE_UNIFIED=1)下索引库只增不减——会话文件清扫(BILI_SESSION_GC=1)删了 .jsonl/.meta 文件,但 index.db 里对应行(messages/blocks/CAS content)永远留着,dev 机已涨到 637MB。本 PR 是 [storage] unified store only grows: content rows are never deleted or archived — index.db at 637 MB on a single dev machine (GC: refcount content, expire sessions, vacuum) #3463 的 step 1:refcount 对账 + 零引用回收 + 手动 GC 入口。
  2. 什么原因导致的:文件侧 GC 与索引侧生命周期是两套互不相通的系统——gcSessionFiles 只删文件,没有任何路径释放对应索引行。每个被清扫的会话把整个索引足迹永久泄漏。
  3. 影响范围与严重程度:所有开了统一存储 + 会话 GC 的用户;index.db 无界单调增长(磁盘 + 查询延迟)。数据本身完好(文件已删、行只是孤儿),属资源泄漏而非正确性 bug——用得越久越严重,不触发即无感。
  4. 怎么修复的(机制层):①在源头堵漏(疏)——清扫在通过全部既有资格门(含 resident-session 保护)后、rm(file) 之前调 store.removeUnifiedSession(id),不再产生新泄漏;②对账历史漂移——refcountAudit 跨全部 5 个 CAS 引用列核对实际引用数 vs content.refcount,fix=true 事务内复位并分类 orphaned/overBooked/underBooked;③回收零引用内容(DELETE WHERE refcount<=0)+ 可选 VACUUM 归还空间给 OS;④手动入口 POST /__bili/storage/gc(loopback-only,{fix?,vacuum?} 默认 true)。step 2(archiveAfterDays 自动化)按 [storage] unified store only grows: content rows are never deleted or archived — index.db at 637 MB on a single dev machine (GC: refcount content, expire sessions, vacuum) #3463 明确不在本 PR。
  5. 是否有回归问题(逐项过触碰路径的既有行为):
    • 会话文件清扫资格判定:不变——removeUnifiedSession 只在全部既有门通过后执行;shadow.remove 永不抛错(try/catch+warn),统一存储侧故障不会打断文件清扫;legacy-only 模式下为 no-op。
    • refcount 纪律:逐一核对过 store 内全部释放点(deleteSession 单事务释放 5 源、pinned checkpoint 按 Storage Phase 2a: replica/checkpoint REST — freeze, restore, GC-proof pins (#2671) #2745 保留引用;stale message/block 清扫、CCR 整体替换均先 releaseContent 再 delete)——一致。reclaim 在 PRAGMA foreign_keys=ON(每个 opener 都应用)下 fail-fast:under-booked 行会 FK 违例抛错,不会静默删坏数据。
    • schema v5 未变;kernel/ 未动;wire 协议未动;持久化格式未动;src/update.ts 未动。
    • 新端点 loopback-only(与全部 __bili 路由同一 gate),无公网面变化;CONFIGURATION.md en+zh 同步更新。
    • 有意行为变更披露:(a) 新增端点 POST /__bili/storage/gc,loopback-only,默认 fix+vacuum 均开;(b) 统一存储 + 会话 GC 同时开启时,被清扫会话从此同时失去索引行(旧行为:行永久泄漏——即本 bug);(c) 其余无任何阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及——prefix cache、nudge 状态、fold-state reconciliation 均未触碰;GC 只作用于已死会话。
  7. 是否改了设计方向:不涉及——严格按 [storage] unified store only grows: content rows are never deleted or archived — index.db at 637 MB on a single dev machine (GC: refcount content, expire sessions, vacuum) #3463 step 1 执行,step 2 留作后续自动化。
  8. 是否引入新配置:无。端点 body 参数 {fix,vacuum} 是请求级参数非持久配置;未新增 env/config 字段。

审核动作(独立验证,非照读 diff)

  • 穷尽性:CAS_REF_SOURCES 覆盖 schema 中全部 5 个指向 content(hash) 的外键列——audit/reclaim 无漏网引用路径。
  • 测试有效性(④):5 个用例都有牙——over-booked 用裸 SQL 注入脏 refcount 验证检测+修复闭环;under-booked 钉住「fix 先于 reclaim」的顺序保证;reclaim 精确断言只删垃圾行;vacuum 用 ~3MB 灌入证明删除不缩文件(freelist)、VACUUM 严格缩小;PARITY 用例钉的就是 bug 本身(修复前代码必挂)。
  • review 中发现并已修的问题(分支上独立标注 commit):
    • be7f18cf6(原作者,rebase 保留):测试中命名 RefcountAuditReport 以过 unused-exports gate。
    • 3371fcdfd:admin 路由拒绝非对象 JSON body(null/array/string/number → 400)——原代码 null→TypeError→500;并补了该路由缺失的 HTTP 层测试(原 PR 只有 store 层)。
    • 1ea36367d:http 用例 teardown 先关 store handle 再 rmrf——startServer 服务的是 persist.ts 模块单例 getStore(),proxy.close() 不关它,Windows 上 index.db 句柄未释放导致 EBUSY unlink(POSIX 容忍)。
  • rebase:分支落后 master 50 个提交,已 rebase 到 fa41483d4(干净,CONFIGURATION.md 与 master 新增文本合并连贯)。
  • 本地定向验证(本地测试纪律:只跑相关面):typecheck/build/unused-exports 全绿;storage-gc 6/6;12 个相关套件定向跑 100 tests / 99 pass / 1 skip(skip 为 gated e2e)。

CI 与遗留

  • CI 全绿 @ 1ea36367d:全部静态 gate + 12 个 test shard(ubuntu/windows × node22/24)+ e2e 套件。期间两次运行各有一个无关既有测试在不同 Windows shard 抖动变红(不同 shard、不同测试,均在他处通过、重跑转绿,master 事发前 35 分钟全绿)——已立 Two wall-clock-sensitive tests flake under loaded Windows CI shards (auto-fold queue-drop, dsh self-lock ladder) #3478 跟踪两个子缺陷(self-lock ladder 绝对毫秒窗口断言 + auto-fold queue-drop 池饱和墙钟窗口),与本 PR 无关。
  • 遗留(属 step 2,非缺陷):本修复上线之前已被清扫文件的「幽灵会话」(文件没了、索引行内部自洽)step 1 够不到——reconcile 视其为活会话;只能等 step 2(archiveAfterDays)或手工 SQL。另:sidecar 自禁用期间被清扫文件的行仍会重新泄漏(禁用窗口的既有特性,非本 PR 引入)。

判定:可合并。 建议人工合并 #3466 (agent 不合并 PR)。

ranxianglei pushed a commit that referenced this pull request Oct 17, 2026
Review follow-up on the GC route: a body of 'null' crashed the handler with a
TypeError (500 instead of 400), and other non-object bodies (arrays, strings,
numbers) were silently accepted as 'use defaults'. Non-object JSON is now
rejected with the same 400 as wrong-typed options, and a new HTTP-level case
pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged
run -> 200 with fix+vacuum defaults ON.
@ranxianglei
ranxianglei force-pushed the 2026-10-17_storage-gc-step1 branch from 1ea3636 to 2cbf0a6 Compare October 17, 2026 04:10
ranxianglei pushed a commit that referenced this pull request Oct 17, 2026
Review follow-up on the GC route: a body of 'null' crashed the handler with a
TypeError (500 instead of 400), and other non-object bodies (arrays, strings,
numbers) were silently accepted as 'use defaults'. Non-object JSON is now
rejected with the same 400 as wrong-typed options, and a new HTTP-level case
pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged
run -> 200 with fix+vacuum defaults ON.
@ranxianglei
ranxianglei force-pushed the 2026-10-17_storage-gc-step1 branch from 2cbf0a6 to 22257be Compare October 17, 2026 04:47
@ranxianglei ranxianglei changed the title storage: unified index GC — release rows on file sweep + manual reconcile/reclaim/VACUUM pass [PR] storage: unified index GC — release rows on file sweep + manual reconcile/reclaim/VACUUM pass Oct 18, 2026
awork added 5 commits October 19, 2026 04:50
The #1082 file sweep deleted only the legacy JSON file; under unified
modes the session's index.db rows (and every CAS refcount behind them)
survived forever, so index.db grew without bound even as files were
swept away. gcSessionFiles now calls SessionStore.removeUnifiedSession
(index-first-then-file, same order as removeLegacyFile: a failed file rm
self-heals on the next sweep because row removal is idempotent).
…VACUUM (#3463)

UnifiedStore.runGc() + POST /__bili/storage/gc (loopback-only):
reconcile each content-CAS row's booked refcount against the actually
counted references across the five source columns (messages, block
summary/one-fold, ccr_entries, checkpoint state blobs), reclaim
provably-unreferenced rows, then optionally VACUUM so freed pages return
to the OS. fix/vacuum default true at the HTTP layer: safe by
construction — fix only moves counts toward reality, reclaim only touches
zero-ref rows, VACUUM never touches live data. Response reports sessions,
reclaimed rows/bytes, drift-class counts, before/after file size.

No new config keys/env vars; schema version unchanged (v5); kernel
untouched. CONFIGURATION.md en+zh document the endpoint.
…ate)

CI unused-exports flagged the report type as INTERNAL_ONLY: the test
exercised refcountAudit() but never named its return type. Pin the shape
with an explicit annotation instead of waiving.
Review follow-up on the GC route: a body of 'null' crashed the handler with a
TypeError (500 instead of 400), and other non-object bodies (arrays, strings,
numbers) were silently accepted as 'use defaults'. Non-object JSON is now
rejected with the same 400 as wrong-typed options, and a new HTTP-level case
pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged
run -> 200 with fix+vacuum defaults ON.
…(Windows EBUSY)

CI windows-latest shard: EBUSY unlinking <tmp>/index.db — startServer serves
from the persist singleton, whose unified sidecar stays open after
proxy.close(); POSIX tolerates unlinking open files, Windows does not. Close
the store (and reset the test singleton) before removing the dir.
@ranxianglei

Copy link
Copy Markdown
Owner Author

Superseded by #3911 (merged): the unified GC landed as store.gc() + POST /__bili/storage/gc (age-sweep archive to the single archive.jsonl log, zero-ref CAS release incl. media refs, ?vacuum=full / incremental auto_vacuum) with POST /__bili/storage/restore as the recovery path, plus the follow-up #3927. The manual reconcile/reclaim pass here pre-dates the refcount release paths that now run on every delete/archive/sweep. Closing as superseded — nothing here is still missing from master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant