Repository navigation
[PR] feat: add supported session cleanup (bili sessions ls/rm/prune + VACUUM) - #3860
ranxianglei wants to merge 6 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4586-pr.3860.e8d194aThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3860.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…UM) (#3830) The sessions dir only had write paths: JSON envelopes are kept forever and, since unified storage defaults to shadow double-write (#2671), every session also lives in index.db rows — with no auto-VACUUM and no user-facing deletion entry. Manually rm'ing files orphans the index.db rows; in full+jsonWriter=off lanes the DB is the sole persistent copy. New standalone commands (no proxy required, no config surface added): - bili sessions ls [--with-sizes] [--json] — enumerate persisted sessions (unified rows ∪ JSON tree; ghost index-only rows included) with per-session JSON envelope bytes and honest index.db attribution (inline JSON columns + refcount-1 exclusive CAS bytes; shared blobs counted but never double-attributed) - bili sessions rm <id> [--force] [--no-vacuum] — delete through the existing supported path (deleteSession cascades every table and releases CAS refcounts, then every on-disk artifact including namespaced envelopes and CCR content-store files), then VACUUM + wal_checkpoint(TRUNCATE) to physically shrink index.db - bili sessions prune --older-than <dur> [--force] [--no-vacuum] — batch rm by last saved time (s/m/h/d/w); one VACUUM at the end Safety semantics: resident sessions (live flag reported by any live instance's /__bili/sessions) and sessions seen within the last 10 minutes are refused without --force; sessions holding pinned replica checkpoints are refused without --force (pinned checkpoints hold CAS refcounts and outlive deletion by design — they survive even with --force). Unreachable live instances warn, never fail. Legacy-only lanes degrade to JSON-file deletion automatically. Supporting changes: - UnifiedStore.vacuum(): VACUUM then wal_checkpoint(TRUNCATE) — in WAL mode the compacted image lands in the WAL until checkpointed (verified empirically: checkpoint-first leaves the main file size unchanged) - UnifiedStore.sessionStorageStats(id): per-session row counts / inline bytes / CAS split - SessionStore.removeSession(id): public single-session deletion reusing the boot-rekey path; removeLegacyFile now shares findSessionFiles/removeSessionFiles, a strict superset of the old candidate list (adds a namespaced-tree walk for standalone openers that never ran a directory scan, plus .content-store.json siblings) - UnifiedShadow passthroughs (vacuum, sessionStorageStats) - Docs: CONFIGURATION.md zh/en + README zh/en
… gate) CleanupSessionInfo/RemoveOutcome/removeOneSession/SessionsCommandOptions have zero references outside their declaring file (the suite imports only parseDuration + runSessionsCommand); the unused-exports gate classifies all four INTERNAL_ONLY.
…ttribution
sessionStorageStats counted checkpoints.state_hash blobs of PINNED replicas as exclusiveBytes ('freed when deleted'), but deleteSession keeps pinned checkpoints by design (#2671 Phase 2) — over-attribution, worst for MB-scale pinned session records. Mirror deleteSession's exact release set: AND pinned = 0.
findSessionFiles derived .content-store.json removal only from matched parent envelopes, so an orphaned content-store (envelope already gone: crash mid-delete, manual rm) survived despite the strict-superset claim over the old removeLegacyFile, which deleted the two known candidates unconditionally. Add them back as explicit existsSync-filtered candidates.
…n from ls attribution Covers the stated acceptance 'shadow & legacy parity' gap: full+jsonWriter=off lanes keep index-only sessions, rm empties every table with no envelope to remove. Also asserts pinned checkpoint state blobs are excluded from casExclusiveBytes (the F2 regression teeth).
…nup sections #3818 flipped the unified default shadow->full (JSON still written by default); 'default shadow double-write' was already stale at branch cut. Wording now names the actual default without re-pinning a mode.
aecec2f to
de652b2
Compare
(ai: vllm-qwen/qwen3.8-27b) 审核结论:代码可合入,等 #3871 清掉继承的 gate 红即可已把分支 rebase 到当前 master(干净、零冲突)并推送 5 个审核修复 commit( 八问
行为变更披露(Deliverables #4)对既有代码路径无行为变更。唯一可观测的数字变化: CI 状态(两个红 gate 与本 PR 无关,已取证)
测试有效性(gate ④)原 9 个测试全部真实集成(真实 SQLite 引擎、真实 writeNow 持久化路径、本地 HTTP + 实例标记文件伪造驻留实例);断言有牙(每 id 七张表全空 + content 表清空 + VACUUM 严格缩容 + 负向断言);各 gate 双向钉住。缺口:声明验收「shadow & legacy parity」没覆盖 推送的 commit(审核修复,各自独立标注)
判定:代码可合入 —— 本 PR 自身的问题已全部修复并被测试钉住;剩余红 gate 属 master 继承(#3871 在途)。#3871 合并后重跑转绿,人工合并即关 #3830。 |
…ow full (ranxianglei#3870) The hand-written long env description of BILI_STORAGE_UNIFIED still ended 'Default off.' / '默认关闭。' — stale since the ranxianglei#3818 storage-full flip that made unset default to full (the generated rows already say 'unset (full)'). Same root cause as the dedup above, flagged during the ranxianglei#3860 audit. Correct the trailing sentence to state the real default.
|
Half-covered by #3911 (merged): the heavy lifting landed as |
Closes #3830
查重: 无(#1082 相关但机制不同:BILI_SESSION_GC 是 opt-in 自动清扫,只删从未压缩且 ≤1M token 的 JSON 文件、从不碰 index.db;本 PR 是面向用户的显式清理命令,覆盖 JSON 信封 + index.db 行 + CAS 引用计数释放 + VACUUM 物理缩容。)
What
The sessions dir had write paths only: JSON envelopes are kept forever and — since unified storage defaults to shadow double-write (#2671) — every session also lives in index.db rows, with no auto-VACUUM and no user-facing deletion entry (
removeLegacyFilewas private, boot-rekey-only). Manually rm'ing files orphans the DB rows; infull+jsonWriter=offlanes the DB is the sole persistent copy.New standalone commands (no proxy required):
--with-sizesadds per-session JSON envelope bytes plus honest index.db attribution: inline JSON-column bytes + refcount-1 exclusive CAS bytes (shared blobs counted but never double-attributed), row counts, and totals vs actual index.db/WAL file size.UnifiedStore.deleteSessioncascades every table and releases CAS refcounts, then every on-disk artifact (namespaced envelopes, flat names, CCR content-store files). Default-on VACUUM physically shrinks the file.Safety semantics:
/__bili/sessions) and sessions seen within the last 10 minutes → refused without--force;--force; pinned checkpoints hold CAS refcounts and outlive deletion by design — they survive even with--force(and are reported as kept);warnActivePluginSessions, 会话卡死自愈机制:卡死模式检测与代理侧主动补救 #2155);full+jsonWriter=offlanes delete rows-only.Supporting changes
UnifiedStore.vacuum(): VACUUM thenwal_checkpoint(TRUNCATE)— in WAL mode the compacted image lands in the WAL until checkpointed (verified empirically: checkpoint-first leaves the main file size unchanged); retries on SQLITE_BUSY.UnifiedStore.sessionStorageStats(id): per-session row counts / inline bytes / CAS split.SessionStore.removeSession(id): public single-session deletion reusing the boot-rekey path.removeLegacyFilenow sharesfindSessionFiles/removeSessionFiles— a strict superset of the old candidate list (adds a namespaced-tree walk for standalone openers that never ran a directory scan, plus.content-store.jsonsiblings).Config surface
None added — pure CLI capability reusing existing deletion primitives; no env vars, no config fields, no defaults changed (per the issue's non-goals).
Behavior changes
None to existing behavior. The only touched pre-existing code path is
removeLegacyFile(boot rekey/dedup), whose behavior is a strict superset of before: same explicit candidates, plus the tree walk and content-store sibling removal that were missing for standalone-style opens.Both compression modes (§6)
No wire, message-shape, nudge, preflight, or tool-injection change — proxy mode and plugin mode are untouched. The change is persistence/CLI-layer only; reasoning per mode: proxy-mode sessions and plugin-mode sessions share the same on-disk layout and the same
deleteSessionprimitive, so one code path covers both; the residency probe reads/__bili/sessions, which both lanes serve identically.Tests
tests/sessions-cleanup.test.ts(9 tests, all green locally): parseDuration; ls sizes (shadow); rm integration — JSON envelope gone, every index.db table empty for the id, CAS content table emptied (refcounts really released), VACUUM shrinks the file; active-session refusal +--forceoverride (fake live instance serving/__bili/sessions); pinned-checkpoint refusal +--forcekeeps the pinned row; unknown id rejects; legacy mode degrades to JSON-only; prune cutoff filter; prune refusal exit codes. Regression suites run locally for the touched area: storage-unified 14/14, fix-401-storage + session-gc 27/27. Full matrix = CI.