Skip to content

[PR] feat: add supported session cleanup (bili sessions ls/rm/prune + VACUUM) - #3860

Closed
ranxianglei wants to merge 6 commits into
masterfrom
2026-10-19_sessions-cleanup
Closed

ranxianglei wants to merge 6 commits into
masterfrom
2026-10-19_sessions-cleanup

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3830

查重: 无(#1082 相关但机制不同:BILI_SESSION_GC 是 opt-in 自动清扫,只删从未压缩且 ≤1M token 的 JSON 文件、从不碰 index.db;本 PR 是面向用户的显式清理命令,覆盖 JSON 信封 + index.db 行 + CAS 引用计数释放 + VACUUM 物理缩容。)

What

The sessions dir had write paths only: JSON envelopes are kept forever and — since unified storage defaults to shadow double-write (#2671) — every session also lives in index.db rows, with no auto-VACUUM and no user-facing deletion entry (removeLegacyFile was private, boot-rekey-only). Manually rm'ing files orphans the DB rows; in full + jsonWriter=off lanes the DB is the sole persistent copy.

New standalone commands (no proxy required):

bili sessions ls [--with-sizes] [--json]
bili sessions rm <id> [--force] [--no-vacuum]
bili sessions prune --older-than <dur> [--force] [--no-vacuum]   # s|m|h|d|w
  • ls enumerates unified rows ∪ the JSON tree (ghost index-only rows included so you can see what space they hold); --with-sizes adds per-session JSON envelope bytes plus honest index.db attribution: inline JSON-column bytes + refcount-1 exclusive CAS bytes (shared blobs counted but never double-attributed), row counts, and totals vs actual index.db/WAL file size.
  • rm deletes through the existing supported path: UnifiedStore.deleteSession cascades every table and releases CAS refcounts, then every on-disk artifact (namespaced envelopes, flat names, CCR content-store files). Default-on VACUUM physically shrinks the file.
  • prune batch-rms by last saved time; per-session refusals reported individually; one VACUUM at the end.

Safety semantics:

  • resident sessions (live flag from any live instance's /__bili/sessions) and sessions seen within the last 10 minutes → refused without --force;
  • sessions holding pinned replica checkpoints → refused without --force; pinned checkpoints hold CAS refcounts and outlive deletion by design — they survive even with --force (and are reported as kept);
  • unreachable live instances produce a warning line, never a failure (same best-effort posture as warnActivePluginSessions, 会话卡死自愈机制:卡死模式检测与代理侧主动补救 #2155);
  • legacy-only lanes degrade to JSON-file deletion automatically; full+jsonWriter=off lanes delete rows-only.

Supporting changes

  • UnifiedStore.vacuum(): VACUUM then wal_checkpoint(TRUNCATE) — in WAL mode the compacted image lands in the WAL until checkpointed (verified empirically: checkpoint-first leaves the main file size unchanged); retries on SQLITE_BUSY.
  • UnifiedStore.sessionStorageStats(id): per-session row counts / inline bytes / CAS split.
  • SessionStore.removeSession(id): public single-session deletion reusing the boot-rekey path. removeLegacyFile now shares findSessionFiles/removeSessionFiles — a strict superset of the old candidate list (adds a namespaced-tree walk for standalone openers that never ran a directory scan, plus .content-store.json siblings).
  • Docs: CONFIGURATION.md zh/en + README zh/en (CLI table + dedicated section).

Config surface

None added — pure CLI capability reusing existing deletion primitives; no env vars, no config fields, no defaults changed (per the issue's non-goals).

Behavior changes

None to existing behavior. The only touched pre-existing code path is removeLegacyFile (boot rekey/dedup), whose behavior is a strict superset of before: same explicit candidates, plus the tree walk and content-store sibling removal that were missing for standalone-style opens.

Both compression modes (§6)

No wire, message-shape, nudge, preflight, or tool-injection change — proxy mode and plugin mode are untouched. The change is persistence/CLI-layer only; reasoning per mode: proxy-mode sessions and plugin-mode sessions share the same on-disk layout and the same deleteSession primitive, so one code path covers both; the residency probe reads /__bili/sessions, which both lanes serve identically.

Tests

tests/sessions-cleanup.test.ts (9 tests, all green locally): parseDuration; ls sizes (shadow); rm integration — JSON envelope gone, every index.db table empty for the id, CAS content table emptied (refcounts really released), VACUUM shrinks the file; active-session refusal + --force override (fake live instance serving /__bili/sessions); pinned-checkpoint refusal + --force keeps the pinned row; unknown id rejects; legacy mode degrades to JSON-only; prune cutoff filter; prune refusal exit codes. Regression suites run locally for the touched area: storage-unified 14/14, fix-401-storage + session-gc 27/27. Full matrix = CI.

@ranxianglei ranxianglei changed the title feat: add supported session cleanup (bili sessions ls/rm/prune + VACUUM) [PR] feat: add supported session cleanup (bili sessions ls/rm/prune + VACUUM) Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-19_sessions-cleanup (de652b2)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4586-pr.3860.e8d194a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3860.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 是 #3830 的唯一在途实现(v1 全范围);open+closed 检索 session cleanup / prune / VACUUM 无并行工作。两个相关条目:

标签: scope/storage + p2(持久化层/会话文件与 index.db 布局;沿用 #3830 既有定级;功能类不打 bug,客户端无关不打客户端标)。

状态:审核进行中。CI 首跑两个红 gate(unused-exports、docs-sync),连带 test/static-gates lane 被跳过未跑——正在甄别是本 PR 引入还是 master 既有红,随后进入代码本体审查。结论按八问体例跟帖。

awork added 6 commits October 20, 2026 03:11
…UM) (#3830)

The sessions dir only had write paths: JSON envelopes are kept forever and, since unified storage defaults to shadow double-write (#2671), every session also lives in index.db rows — with no auto-VACUUM and no user-facing deletion entry. Manually rm'ing files orphans the index.db rows; in full+jsonWriter=off lanes the DB is the sole persistent copy.

New standalone commands (no proxy required, no config surface added):

- bili sessions ls [--with-sizes] [--json] — enumerate persisted sessions (unified rows ∪ JSON tree; ghost index-only rows included) with per-session JSON envelope bytes and honest index.db attribution (inline JSON columns + refcount-1 exclusive CAS bytes; shared blobs counted but never double-attributed)

- bili sessions rm <id> [--force] [--no-vacuum] — delete through the existing supported path (deleteSession cascades every table and releases CAS refcounts, then every on-disk artifact including namespaced envelopes and CCR content-store files), then VACUUM + wal_checkpoint(TRUNCATE) to physically shrink index.db

- bili sessions prune --older-than <dur> [--force] [--no-vacuum] — batch rm by last saved time (s/m/h/d/w); one VACUUM at the end

Safety semantics: resident sessions (live flag reported by any live instance's /__bili/sessions) and sessions seen within the last 10 minutes are refused without --force; sessions holding pinned replica checkpoints are refused without --force (pinned checkpoints hold CAS refcounts and outlive deletion by design — they survive even with --force). Unreachable live instances warn, never fail. Legacy-only lanes degrade to JSON-file deletion automatically.

Supporting changes:

- UnifiedStore.vacuum(): VACUUM then wal_checkpoint(TRUNCATE) — in WAL mode the compacted image lands in the WAL until checkpointed (verified empirically: checkpoint-first leaves the main file size unchanged)

- UnifiedStore.sessionStorageStats(id): per-session row counts / inline bytes / CAS split

- SessionStore.removeSession(id): public single-session deletion reusing the boot-rekey path; removeLegacyFile now shares findSessionFiles/removeSessionFiles, a strict superset of the old candidate list (adds a namespaced-tree walk for standalone openers that never ran a directory scan, plus .content-store.json siblings)

- UnifiedShadow passthroughs (vacuum, sessionStorageStats)

- Docs: CONFIGURATION.md zh/en + README zh/en
… gate)

CleanupSessionInfo/RemoveOutcome/removeOneSession/SessionsCommandOptions have zero references outside their declaring file (the suite imports only parseDuration + runSessionsCommand); the unused-exports gate classifies all four INTERNAL_ONLY.
…ttribution

sessionStorageStats counted checkpoints.state_hash blobs of PINNED replicas as exclusiveBytes ('freed when deleted'), but deleteSession keeps pinned checkpoints by design (#2671 Phase 2) — over-attribution, worst for MB-scale pinned session records. Mirror deleteSession's exact release set: AND pinned = 0.
findSessionFiles derived .content-store.json removal only from matched parent envelopes, so an orphaned content-store (envelope already gone: crash mid-delete, manual rm) survived despite the strict-superset claim over the old removeLegacyFile, which deleted the two known candidates unconditionally. Add them back as explicit existsSync-filtered candidates.
…n from ls attribution

Covers the stated acceptance 'shadow & legacy parity' gap: full+jsonWriter=off lanes keep index-only sessions, rm empties every table with no envelope to remove. Also asserts pinned checkpoint state blobs are excluded from casExclusiveBytes (the F2 regression teeth).
…nup sections

#3818 flipped the unified default shadow->full (JSON still written by default); 'default shadow double-write' was already stale at branch cut. Wording now names the actual default without re-pinning a mode.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论:代码可合入,等 #3871 清掉继承的 gate 红即可

已把分支 rebase 到当前 master(干净、零冲突)并推送 5 个审核修复 commit(aecec2ffd → de652b25a)。本地定向验证全绿:typecheck clean、tests/sessions-cleanup.test.ts 10/10、storage-unified + fix-401-storage + session-gc 41 pass/1 skip。全量矩阵交给 CI。

八问

  1. 是什么问题:sessions 目录只有写路径、没有面向用户的清理入口——JSON 信封永久保留,统一存储双写进 index.db 且无自动 VACUUM;手工 rm 文件会孤儿化 DB 行,full+jsonWriter=off lane 里 DB 还是唯一副本。
  2. 什么原因导致的:设计缺口而非某条路径的 bug——受支持的删除原语一直存在(UnifiedStore.deleteSession),但从未暴露给用户入口(removeLegacyFile 是私有的、boot-rekey 专用)。
  3. 影响范围与严重程度:所有长会话用户的 sessions 目录无界增长(存储膨胀/体验);无数据丢失、无功能损坏 → p2。
  4. 怎么修复的:新增独立 CLI bili sessions ls/rm/prune(不需要代理在跑):走既有 deleteSession 级联(全表级联删 + 释放 CAS 引用计数)+ 磁盘产物全覆盖(显式候选 + 命名空间树遍历 + content-store 兄弟文件)+ 默认开 VACUUM(实测确认必须先 VACUUM 再 wal_checkpoint(TRUNCATE),WAL 模式下顺序反了主文件不缩容)。安全语义:驻留/最近 10 分钟活跃/pinned checkpoint 无 --force 拒绝;pinned 即使 --force 也保留并如实报告;不可达实例只警告不失败(会话卡死自愈机制:卡死模式检测与代理侧主动补救 #2155 姿态);legacy 自动降级为仅删 JSON。
  5. 是否有回归问题(diff 触碰的既有行为逐项审过):
    • removeLegacyFile(boot rekey/dedup,唯一被改的既有路径):PR 声称「严格超集」,我查出缺口——孤立 content-store(信封已不在)新代码不再删(旧代码无条件删)→ 本次已修(两个已知候选加回显式列表),现在才是真严格超集;
    • deleteSession 本体:未动;
    • ls --with-sizes 归属计算:发现 pinned checkpoint 的 state blob 被计入 exclusiveBytes(「删除时会释放」)——pinned 按设计存活,属过度归因(MB 级 pinned 记录会虚高)→ 本次已修(AND pinned = 0,与 deleteSession 释放集精确对齐)+ 测试钉住;
    • 两种压缩模式:均不受影响(纯持久化/CLI 层,零 wire 变更);
    • 高危模式逐项过:无 fail-fast 变吞错(VACUUM 失败仅告警、行已提交——有意的 fail-soft);无阈值/时序/输出格式漂移;无配置面变更。
  6. 是否破坏缓存:不涉及(不碰前缀缓存/压缩状态;CAS 引用计数只在真实删除时释放,pinned 部分保留)。
  7. 是否改了设计方向:不涉及(复用既有删除原语,无新架构;按 feat: add supported session cleanup (bili sessions rm/prune + VACUUM) for the sessions dir / index.db #3830 非目标不引入自动保留/TTL——那是 [storage] unified store only grows: content rows are never deleted or archived — index.db at 637 MB on a single dev machine (GC: refcount content, expire sessions, vacuum) #3463 第二步,已在该 issue 留互链与收窄建议)。
  8. 是否引入新配置:无(纯 CLI 能力;不加 env/字段/默认值)。

行为变更披露(Deliverables #4)

对既有代码路径无行为变更。唯一可观测的数字变化:ls --with-sizes 不再把 pinned checkpoint 的 blob 计为「可删」字节(旧值虚高 → 新值准确),display-only。文档措辞修正:「默认 shadow 双写」在 #3818 翻转后已过时(现默认 full + jsonWriter on),en/zh 均已改准;PR 正文里的同句是切分支时的历史表述,仓库文件无需再动。

CI 状态(两个红 gate 与本 PR 无关,已取证)

测试有效性(gate ④)

原 9 个测试全部真实集成(真实 SQLite 引擎、真实 writeNow 持久化路径、本地 HTTP + 实例标记文件伪造驻留实例);断言有牙(每 id 七张表全空 + content 表清空 + VACUUM 严格缩容 + 负向断言);各 gate 双向钉住。缺口:声明验收「shadow & legacy parity」没覆盖 full+jsonWriter=off 的 index-only lane → 本次补 1 个测试,另给 pinned 归属修复加了回归断言。现 10/10。

推送的 commit(审核修复,各自独立标注)

  1. fix: de-export internal-only sessions-cleanup symbols (unused-exports gate)
  2. fix: exclude pinned checkpoint state blobs from ls --with-sizes CAS attribution
  3. fix: keep orphaned content-store files covered by session deletion
  4. test: pin full-mode index-only deletion parity + pinned-blob exclusion from ls attribution
  5. docs: correct post-#3818 default-double-write wording in session cleanup sections

判定:代码可合入 —— 本 PR 自身的问题已全部修复并被测试钉住;剩余红 gate 属 master 继承(#3871 在途)。#3871 合并后重跑转绿,人工合并即关 #3830。

ranxianglei pushed a commit to ReineLan/billion-context that referenced this pull request Oct 20, 2026
…ow full (ranxianglei#3870)

The hand-written long env description of BILI_STORAGE_UNIFIED still ended
'Default off.' / '默认关闭。' — stale since the ranxianglei#3818 storage-full flip that
made unset default to full (the generated rows already say 'unset (full)').
Same root cause as the dedup above, flagged during the ranxianglei#3860 audit. Correct
the trailing sentence to state the real default.
@ranxianglei

Copy link
Copy Markdown
Owner Author

Half-covered by #3911 (merged): the heavy lifting landed as store.gc() + POST /__bili/storage/gc (age-sweep archive → single archive.jsonl, zero-ref CAS release, ?vacuum=full/incremental auto_vacuum) and POST /__bili/storage/restore, plus archive/GC knobs (storage.archiveAfterDays, default 365). What this PR still uniquely offered — the bili sessions ls/rm/prune CLI entry — is being re-filed as a small follow-up issue instead of carrying this conflicted branch. Closing as superseded/re-scoped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add supported session cleanup (bili sessions rm/prune + VACUUM) for the sessions dir / index.db

1 participant