Skip to content

[PR] feat: CLI surface for storage maintenance — bili sessions ls/rm/archive/restore + bili storage gc (#3928) - #3958

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-20_storage-maintenance-cli
Oct 20, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-20_storage-maintenance-cli

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3928

Re-scoped follow-up of the closed #3860 (its store layer was superseded by #3911; this picks up the CLI entry that #3911 deliberately left out). Thin command layer over the same store APIs the HTTP endpoints use — no new store semantics, no scheduler.

Commands

bili sessions ls                  # id, label, size (decoded bytes), last-updated, storage status (active/archived)
bili sessions rm <id> [--no-vacuum]
                                  # full delete via the existing deleteSession discipline
                                  # (content refcount release incl. media); VACUUMs afterwards
bili sessions archive <id>        # export to the single append-only archive/archive.jsonl + sweep rows
bili sessions restore <id>        # bring an archived session back from its log line
bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]
                                  # print the sweep plan (age-sweep candidates, zero-ref content
                                  # rows, ~pages reclaimable), then run the SAME unifiedGc() the
                                  # HTTP endpoint calls; --dry-run stops after the plan

All commands accept --json. archive/restore/gc require the unified store (BILI_STORAGE_UNIFIED=shadow|full or storage.unified) — legacy-only lanes refuse with the same hint as the 501 on the HTTP side. --days defaults to storage.archiveAfterDays (365) through the same resolveArchiveAfterDays chain as the HTTP path.

What changed where

  • src/storage-maintenance.ts (new): runSessionsCommand / runStorageGcCommand — all store access for the CLI lives here; cli.ts only parses and dispatches.
  • src/cli.ts: two commands, six flags/positionals, HELP lines; standalone dispatch before server start (same shape as export/doctor).
  • src/storage/store.ts (additive only): gcPlan() read-only prediction (same cutoff query as gc()), contentZeroRefReport(), sessionStorageStats() (per-session index.db footprint, ported from the [PR] feat: add supported session cleanup (bili sessions ls/rm/prune + VACUUM) #3860 work and adapted for the v7 message_media CAS), vacuum(mode) (full = VACUUM + wal_checkpoint(TRUNCATE) with SQLITE_BUSY retry), and an additive status column in sessionSummaries()/webSummaryRows().
  • src/persist.ts: facade wrappers (unifiedArchiveSession, unifiedSessionStorageStats, unifiedVacuum, unifiedGcPlan), removeSession(id) (+findSessionFiles made public, removeLegacyFile refactored onto removeSessionFiles), decodedBytesOf(file) (the Bound startup session loading and preserve historical recovery #2803 decoded-bytes idiom).
  • src/storage/shadow.ts: four pass-throughs.
  • Tests: tests/storage-maintenance-cli.test.ts (new, 7 tests) — every command demonstrated against scratch stores; gc e2e numbers pinned equal to direct store.gc() on an equivalent store; legacy-lane refusals; parseArgs usage errors exit 2.
  • Docs: README en/zh "Storage maintenance CLI" section + CONFIGURATION en/zh mention the CLI forms next to the HTTP ones.

Behavior changes (owner policy 2026-09-24 disclosure)

None intended. Two additive notes:

  1. sessionSummaries()/webSummaryRows() gain a status field — anything rendering those rows (incl. the /__bili/sessions JSON) sees ONE extra key per row; no existing field changed.
  2. removeLegacyFile now delegates to the shared removeSessionFiles helper — identical candidate set (relPathFor×2 + flatFileNameFor + content-store pairs), verified by the existing persist suites still passing.

Verification

  • npm run typecheck: clean.
  • Targeted local runs: tests/storage-maintenance-cli.test.ts 7/7, tests/storage-archive-gc.test.ts 5/5, tests/cli-export.test.ts 2/2. Full-suite verdict = remote CI on this head.
  • No kernel/ changes (kernel-change: n/a); no config surface changes (no new fields/env vars — --days maps onto the existing storage.archiveAfterDays).

Prior art: the closed #3860 branch (2026-10-19_sessions-cleanup) contributed the safety semantics and the stats/vacuum SQL, adapted here to the post-#3911/#3927 single-log store layer.

@ranxianglei ranxianglei changed the title feat: CLI surface for storage maintenance — bili sessions ls/rm/archive/restore + bili storage gc (#3928) [PR] feat: CLI surface for storage maintenance — bili sessions ls/rm/archive/restore + bili storage gc (#3928) Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_storage-maintenance-cli (9d5a00d)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4748-pr.3958.e54e542

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3958.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本 PR 是 #3928 唯一在途实现;#3860(旧 sessions cleanup CLI)已关闭、其 store 层被 #3911 取代,store/GC/archive 的 HTTP 面(#3911)与单日志 archive(#3927)均已合入 master,本 PR 只补 CLI 入口,与在途的 #3932/#3946(StorageBackend 契约/MySQL,动 store 抽象层)无重叠改动。

标签: pr scope/storage p3(功能类 PR,存储域,非缺陷;p3 = 最小影响面)。

开始按 PR Review 规范审:方向(#3928 scope/acceptance 对照)→ 回归清点 → 代码 → 测试有效性 → diff 整洁度。CI 当前还在排队(gate/kernel-guard/pi-subagents-guard 已过),结论会等 CI 全绿后给八问。

ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…ase busy (#3928)

Review finding on #3958: runRm's bare catch reported EVERY unifiedVacuum failure as 'database busy' (e.g. ENOSPC or corruption). Only SQLITE_BUSY/'database is locked' now renders as 'database busy'; anything else carries its real message. The deletion-committed semantics are unchanged.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the sessions/storage help entries used a 3-space indent while every sibling entry uses 2, and the pre-existing 'acp-cache diff' entry was re-indented without moving its continuation lines (header description col 36 vs continuations col 35). Restores the acp-cache block byte-for-byte and aligns the new entries to the sibling 2-space indent.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_storage-maintenance-cli branch from 494edb6 to 2a7de54 Compare October 20, 2026 12:00
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…ase busy (#3928)

Review finding on #3958: runRm's bare catch reported EVERY unifiedVacuum failure as 'database busy' (e.g. ENOSPC or corruption). Only SQLITE_BUSY/'database is locked' now renders as 'database busy'; anything else carries its real message. The deletion-committed semantics are unchanged.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the sessions/storage help entries used a 3-space indent while every sibling entry uses 2, and the pre-existing 'acp-cache diff' entry was re-indented without moving its continuation lines (header description col 36 vs continuations col 35). Restores the acp-cache block byte-for-byte and aligns the new entries to the sibling 2-space indent.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the CLI sentence was hand-edited into CONFIGURATION.md/zh only, so the #2107 generator (single source = website/config-reference/*.yaml) stripped it and docs-sync went red. The note now lives in the process.yaml seed (backtick-free per the generator rule) and the docs are the regenerated artifact.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…rts gate (#3928)

Review finding on #3958: StorageCommandResult and GcPlanCandidate were exported but imported nowhere — the gate goes red on merged heads. Both are module-internal now; GcPlan (still exported, consumed by the persist facade signature) is unchanged.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_storage-maintenance-cli branch from b152008 to a299f37 Compare October 20, 2026 12:37
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…ase busy (#3928)

Review finding on #3958: runRm's bare catch reported EVERY unifiedVacuum failure as 'database busy' (e.g. ENOSPC or corruption). Only SQLITE_BUSY/'database is locked' now renders as 'database busy'; anything else carries its real message. The deletion-committed semantics are unchanged.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the sessions/storage help entries used a 3-space indent while every sibling entry uses 2, and the pre-existing 'acp-cache diff' entry was re-indented without moving its continuation lines (header description col 36 vs continuations col 35). Restores the acp-cache block byte-for-byte and aligns the new entries to the sibling 2-space indent.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the CLI sentence was hand-edited into CONFIGURATION.md/zh only, so the #2107 generator (single source = website/config-reference/*.yaml) stripped it and docs-sync went red. The note now lives in the process.yaml seed (backtick-free per the generator rule) and the docs are the regenerated artifact.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…rts gate (#3928)

Review finding on #3958: StorageCommandResult and GcPlanCandidate were exported but imported nowhere — the gate goes red on merged heads. Both are module-internal now; GcPlan (still exported, consumed by the persist facade signature) is unchanged.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_storage-maintenance-cli branch from a299f37 to 95e1016 Compare October 20, 2026 12:56
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…ase busy (#3928)

Review finding on #3958: runRm's bare catch reported EVERY unifiedVacuum failure as 'database busy' (e.g. ENOSPC or corruption). Only SQLITE_BUSY/'database is locked' now renders as 'database busy'; anything else carries its real message. The deletion-committed semantics are unchanged.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the sessions/storage help entries used a 3-space indent while every sibling entry uses 2, and the pre-existing 'acp-cache diff' entry was re-indented without moving its continuation lines (header description col 36 vs continuations col 35). Restores the acp-cache block byte-for-byte and aligns the new entries to the sibling 2-space indent.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
)

Review finding on #3958: the CLI sentence was hand-edited into CONFIGURATION.md/zh only, so the #2107 generator (single source = website/config-reference/*.yaml) stripped it and docs-sync went red. The note now lives in the process.yaml seed (backtick-free per the generator rule) and the docs are the regenerated artifact.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…rts gate (#3928)

Review finding on #3958: StorageCommandResult and GcPlanCandidate were exported but imported nowhere — the gate goes red on merged heads. Both are module-internal now; GcPlan (still exported, consumed by the persist facade signature) is unchanged.
awork added 6 commits October 20, 2026 21:02
…ve/restore + bili storage gc (#3928)

Thin command layer over the store APIs #3911 exposed over HTTP (POST
/__bili/storage/gc, POST /__bili/storage/restore): no new store semantics,
no scheduler.

- sessions ls: id, label, decoded size, last-updated, storage status
  (active/archived); DB rows unioned with the JSON tree like bili export.
- sessions rm <id>: full delete via the existing deleteSession discipline
  (content refcount release incl. media), VACUUM afterwards (--no-vacuum).
- sessions archive/restore <id>: wrap archiveSession/restoreArchived on
  the single append-only archive.jsonl log; unified-store-only commands
  refuse legacy lanes with the BILI_STORAGE_UNIFIED hint.
- storage gc [--days N] [--vacuum full|incremental] [--dry-run]: prints
  the sweep plan (age-sweep candidates, zero-ref content rows, ~pages)
  from read-only queries, then runs the same unifiedGc() the HTTP
  endpoint calls; --dry-run stops after the plan.

Additive store surface backing the plan/statistics: GcPlan/gcPlan
(read-only prediction), contentZeroRefReport, sessionStorageStats,
vacuum(mode), status column in sessionSummaries/webSummaryRows.

Tests: tests/storage-maintenance-cli.test.ts demonstrates every command
against scratch stores and pins gc parity with direct store.gc().
Docs: README en/zh 'Storage maintenance CLI' + CONFIGURATION en/zh.
…ase busy (#3928)

Review finding on #3958: runRm's bare catch reported EVERY unifiedVacuum failure as 'database busy' (e.g. ENOSPC or corruption). Only SQLITE_BUSY/'database is locked' now renders as 'database busy'; anything else carries its real message. The deletion-committed semantics are unchanged.
)

Review finding on #3958: the sessions/storage help entries used a 3-space indent while every sibling entry uses 2, and the pre-existing 'acp-cache diff' entry was re-indented without moving its continuation lines (header description col 36 vs continuations col 35). Restores the acp-cache block byte-for-byte and aligns the new entries to the sibling 2-space indent.
)

Review finding on #3958: the CLI sentence was hand-edited into CONFIGURATION.md/zh only, so the #2107 generator (single source = website/config-reference/*.yaml) stripped it and docs-sync went red. The note now lives in the process.yaml seed (backtick-free per the generator rule) and the docs are the regenerated artifact.
…rts gate (#3928)

Review finding on #3958: StorageCommandResult and GcPlanCandidate were exported but imported nowhere — the gate goes red on merged heads. Both are module-internal now; GcPlan (still exported, consumed by the persist facade signature) is unchanged.
… + carry status into mysql summary rows

Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_storage-maintenance-cli branch from 95e1016 to 9d5a00d Compare October 20, 2026 13:02
@ranxianglei
ranxianglei merged commit 99f1b93 into master Oct 20, 2026
37 of 42 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.679-master.99f1b93 (rolling master tag).

npm install -g billion-context@0.0.679-master.99f1b93  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI surface for storage maintenance: bili sessions ls/rm/archive/restore + bili storage gc (re-scoped from #3860)

1 participant