Repository navigation
[PR] storage: unified index GC — release rows on file sweep + manual reconcile/reclaim/VACUUM pass - #3466
[PR] storage: unified index GC — release rows on file sweep + manual reconcile/reclaim/VACUUM pass#3466ranxianglei wants to merge 5 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4430-pr.3466.80e1bdfThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3466.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 本 PR 是 #3463(统一存储只进不出,index.db 637 MB)第一步的唯一实现——检索 open+closed 的 issue/PR(refcount / VACUUM / storage gc / archiveAfterDays)无并行工作;最接近的已合项都不是同一机制(#2745 GC-proof pins = checkpoint 保活,#1189 CCR 文件侧 GC)。 在飞重叠提示(非重复,但合入顺序需注意):#3432(StorageBackend 契约重构)与本 PR 同改 开始独立审核:拉分支通读 diff → 对照 #3463 验收标准 → 既有行为回归清点 → 测试有效性 → CI 状态。结论按八问回本线程。 |
cc846a0 to
be7f18c
Compare
Review follow-up on the GC route: a body of 'null' crashed the handler with a TypeError (500 instead of 400), and other non-object bodies (arrays, strings, numbers) were silently accepted as 'use defaults'. Non-object JSON is now rejected with the same 400 as wrong-typed options, and a new HTTP-level case pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged run -> 200 with fix+vacuum defaults ON.
(ai: vllm-qwen/qwen3.8-27b) 审核结论:可合并 ✅(CI 全绿 @
|
Review follow-up on the GC route: a body of 'null' crashed the handler with a TypeError (500 instead of 400), and other non-object bodies (arrays, strings, numbers) were silently accepted as 'use defaults'. Non-object JSON is now rejected with the same 400 as wrong-typed options, and a new HTTP-level case pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged run -> 200 with fix+vacuum defaults ON.
1ea3636 to
2cbf0a6
Compare
Review follow-up on the GC route: a body of 'null' crashed the handler with a TypeError (500 instead of 400), and other non-object bodies (arrays, strings, numbers) were silently accepted as 'use defaults'. Non-object JSON is now rejected with the same 400 as wrong-typed options, and a new HTTP-level case pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged run -> 200 with fix+vacuum defaults ON.
2cbf0a6 to
22257be
Compare
The #1082 file sweep deleted only the legacy JSON file; under unified modes the session's index.db rows (and every CAS refcount behind them) survived forever, so index.db grew without bound even as files were swept away. gcSessionFiles now calls SessionStore.removeUnifiedSession (index-first-then-file, same order as removeLegacyFile: a failed file rm self-heals on the next sweep because row removal is idempotent).
…VACUUM (#3463) UnifiedStore.runGc() + POST /__bili/storage/gc (loopback-only): reconcile each content-CAS row's booked refcount against the actually counted references across the five source columns (messages, block summary/one-fold, ccr_entries, checkpoint state blobs), reclaim provably-unreferenced rows, then optionally VACUUM so freed pages return to the OS. fix/vacuum default true at the HTTP layer: safe by construction — fix only moves counts toward reality, reclaim only touches zero-ref rows, VACUUM never touches live data. Response reports sessions, reclaimed rows/bytes, drift-class counts, before/after file size. No new config keys/env vars; schema version unchanged (v5); kernel untouched. CONFIGURATION.md en+zh document the endpoint.
…ate) CI unused-exports flagged the report type as INTERNAL_ONLY: the test exercised refcountAudit() but never named its return type. Pin the shape with an explicit annotation instead of waiving.
Review follow-up on the GC route: a body of 'null' crashed the handler with a TypeError (500 instead of 400), and other non-object bodies (arrays, strings, numbers) were silently accepted as 'use defaults'. Non-object JSON is now rejected with the same 400 as wrong-typed options, and a new HTTP-level case pins the route: non-object bodies -> 400, wrong-typed option -> 400, unflagged run -> 200 with fix+vacuum defaults ON.
…(Windows EBUSY) CI windows-latest shard: EBUSY unlinking <tmp>/index.db — startServer serves from the persist singleton, whose unified sidecar stays open after proxy.close(); POSIX tolerates unlinking open files, Windows does not. Close the store (and reset the test singleton) before removing the dir.
22257be to
851e349
Compare
|
Superseded by #3911 (merged): the unified GC landed as |
Closes #3463
Scope: step 1 of the issue (refcount reclamation paths + manual GC entry point). Step 2 (
archiveAfterDaysautomation + incremental vacuum) is deliberately NOT here — the issue sequences it explicitly ("提供手工入口先跑一遍,验证收益后再谈自动化"); it waits on the measured benefit + your go. Its design is fully settled in #2671 §7 item 4 and ready to follow.What the audit found
The refcount machinery already exists and is tested (
content.refcountmaintained byingestSession/deleteSession, pinned bytests/storage-engine-proofs.test.ts). What was actually missing:gcSessionFiles([Feature] Default state file cleanup strategy #1082) deleted only the legacy JSON file viarm(file); under shadow/full modes everyindex.dbrow of a swept session — and every CAS refcount behind it — survived forever, soindex.dbgrew without bound even while files were being swept away. Fix: the sweep now callsSessionStore.removeUnifiedSession(id)BEFORErm(file)(index-first-then-file, same order asremoveLegacyFile; row removal is idempotent so a failed file rm self-heals on the next sweep).UnifiedStore.runGc()+POST /__bili/storage/gc(loopback-only, like every other__biliroute):true): reconciles each CAS row's booked refcount against the ACTUALLY counted references across all five source columns (messages.content_hash,blocks.summary_hash,blocks.one_hash,ccr_entries.content_hash,checkpoints.state_hash— the exhaustive set per schema v5). Over-booked (leaked) refs become reclaimable; under-booked (premature-delete hazard) refs are corrected BEFORE anything is deleted.DELETE FROM content WHERE refcount <= 0— provably-unreferenced rows only.true): fullVACUUMreturns freed pages to the OS (exclusive lock + ~2x transient space — an idle-time ops action; periodic/incremental automation belongs to step 2).Behavior changes (disclosed)
POST /__bili/storage/gc. No new config keys / env vars / CLI flags — config surface unchanged.BILI_SESSION_GC=1under unified modes, the [Feature] Default state file cleanup strategy #1082 sweep now ALSO releases the unified rows of each file it deletes (previously it left them behind — that is the bug this fixes). Legacy-only mode is untouched (no sidecar → the call is a no-op).STORAGE_SCHEMA_VERSIONstays 5).kernel/untouched.Verification
tests/storage-gc.test.ts(5 cases): over-booked drift detected + fixed; under-booked hazard corrected before reclaim (referenced row survives); orphaned zero-ref garbage reclaimed; VACUUM provably shrinks the file after mass deletion (strict size decrease); end-to-end PARITY — aged legacy file + unified index rows, sweep → file gone AND index row gone AND exclusive CAS row gone, shared CAS row survives.session-gc/state-gc/proxy-orphan-gc/storage-engine-proofs/storage-ext/storage-replicas(56 pass);storage-unified(the Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 harness)/storage-save-path/storage-json-writer-off/web-storage-banner/storage-ingest-watermark/storage-review-fixes(43 pass, 1 pre-existing environmental skip: better-sqlite3 not installed on this runner). Full-suite verdict = remote CI on this head.~/.local/state/billion-context/sessions/index.dbis not reachable from this sandbox): 300 live sessions + 100 dead sessions whose rows had been removed WITHOUT releasing CAS refs (simulating the pre-fix sweep-leak class) → onerunGc({fix:true, vacuum:true}): index.db 78.9 MB → 59.26 MB (−24.9%) in 0.3 s; 100 orphaned refs corrected, 100 rows / 19.54 MB reclaimed; remainder ≈ exactly the live sessions' content. On the reporter's 637 MB machine the first-pass gain depends on how much dead material the sweep covers — the big lever there is turning ONBILI_SESSION_GC=1(opt-in) so deletions finally release the index, then one GC pass; step 2 automates the rest.Complementary write-side defect: #3748 (NOT fixed by this PR)
External measurement (@yanghuan, #3748; bili 0.1.201, shadow default, dsh lane, Windows) shows a second, dominant producer of
refcount = 0rows on image-heavy sessions: the host re-mints message ids (h_…_N) and a per-passwireIndexon every re-projection, andplanIngesthashes the WHOLE wrapper (JSON.stringify(m)— volatile fields included, src/storage/ingest.ts:150), so byte-identical payloads never share a CAS row (measured 11.06× redundancy; 88.6% of content bytes are rc=0 orphans produced continuously by re-ingest, ~758 MiB/h growth).Consequence for reading this PR: the GC endpoint below reclaims exactly those rc=0 rows (one pass ≈ 2.2 GB of the reporter's 2.5 GiB file), but the minting continues until #3748 lands — do not read one green GC run as "growth solved". The write-side fixes proposed in #3748 (normalized CAS key / idempotent upsert / shadow not mirroring raw image bytes) are behavior changes in owner-decision territory and are tracked there; this PR deliberately does not touch the write path, and the CONFIGURATION.md disk-expectation correction rides with #3748 so doc and code land together.
Known residual (noted, out of scope)
inc:) sessions skip the [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 stale-message sweep until the next boot watermark reset — stale message rows hold CAS refs meanwhile. Self-heals at boot.archiveAfterDays(default 365d per Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 §7), slimming archive (drop originals, keep ledger + summaries + tombstone), incremental-vacuum scheduling.Docs
CONFIGURATION.md en+zh: endpoint + disk-expectation text in the
BILI_STORAGE_UNIFIEDsection.