Skip to content

[PR] storage gc: batch orphan-content collection + refuse full VACUUM while another process holds the db (#4612, #4796) - #5193

Merged
ranxianglei merged 12 commits into
masterfrom
2026-10-30_gc-combined-4612-4796
Oct 30, 2026
Merged

ranxianglei merged 12 commits into
masterfrom
2026-10-30_gc-combined-4612-4796

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What this fixes

Two independent failures of bili storage gc in the field, both reported by users:

  1. gc never finishes (billion-context database cleanup records GC bug | billion-context 数据库清理记录 gc bug #4796): on a 21.6 GB db where 14.4 GB of the content table (658,125 of 766,125 rows) are unreferenced, bili storage gc --vacuum full ran 8 hours / 28,131 s of CPU, deleted zero rows, and the WAL grew to 221 MB before vacuum could even start. Cause: no index on content(refcount) made candidate selection a full-table scan, and the unbounded DELETE ran as one giant transaction holding the exclusive write lock the whole time.
  2. gc starves a live peer (storage gc full VACUUM holds the write lock across the whole O(db) rewrite with no progress — online proxy shadow disabled mid-run | storage gc 全量 VACUUM 全程持写锁且无进度:在线代理影子层被降级 #4612): full VACUUM is an O(db-size) atomic rewrite under the exclusive write lock with no intermediate progress. Field report: 14 min 25 s of silence on a 14 GB db; an online agent sharing the same database hit database is locked during the run and its unified-storage shadow layer tripped to legacy-only for the rest of the process life. Nothing refused or warned before the contention began.

Both fixes were drafted separately (#4800, #4628). They edit the same lines of UnifiedStore.gc() and cannot be merged independently — this single PR carries both, conflict-resolved.

Changes

Batched orphan-content collection (from #4800)

  • SQLite schema v8 → v9: adds idx_content_refcount; candidate selection becomes an index range scan instead of a full-table scan. Existing v8 dbs gain the index on next open.
  • The gc sweep deletes zero-ref content in fixed 2000-row batches: each batch commits and takes a best-effort PRAGMA wal_checkpoint(TRUNCATE) between batches (bounds WAL growth, releases the write lock between batches). Progress line every 10th batch and always on the final batch: [#4796] gc: collected N zero-ref content rows so far (batch k). A sweep that dies mid-run converges on the next gc — refcount <= 0 rows are garbage by definition.
  • Session lifecycle ops (deleteSession, archiveSession, deleteCheckpoint) no longer trigger whole-table sweeps: they collect only within the hashes their own operation released (chunks of 500).
  • MySQL backend mirrors all of the above.

Online guard against full VACUUM (from #4628)

  • New UnifiedStore.activeForeignWriter(): union of the lockfile holder captured at open (pid liveness re-checked) and a fresh re-read of <db>.lock, so peers that started after this store opened are also caught. Returns a live pid or null.
  • gc({ vacuum: "full" }) now refuses up front with an actionable error naming the pid instead of silently starving the peer for the whole rewrite. Incremental / no-vacuum sweeps remain online-safe (short transactions covered by the existing busy_timeout handling).
  • Surfaced end-to-end: CLI exits 1 with a --vacuum incremental hint (src/storage-maintenance.ts); panel gc endpoint returns HTTP 409 with writerPid (src/server/admin.ts); exposed through SessionStore.unifiedActiveWriter() and UnifiedShadow.activeForeignWriter().
  • Vacuum start/finish logs now report main-file (+WAL) sizes before/after plus wall time, so the deliberately-offline full vacuum is self-documenting in logs.
  • README (en + zh): full vacuum documented as an offline operation.

MySQL schema-stamp gap found during integration

Neither draft bumped MYSQL_SCHEMA_VERSION; while they developed, master moved 6 → 8. An existing MySQL install stamped 8 would never enter the upgrade block and would be missing the new index forever. This branch bumps the stamp to 9 and adds an open-time information_schema probe that adds the index once for already-stamped dbs.

How the conflicts were resolved

Behavior changes

  • bili storage gc --vacuum full fails fast (exit 1, actionable message naming the peer pid) while another live process holds the database. Previously it would quietly hold the exclusive lock for the entire rewrite.
  • Panel gc with vacuum: "full" returns HTTP 409 in the same situation.
  • Sweep and lifecycle cleanup emit the progress lines noted above (previously silent for hours).

Verification (local, node --import tsx --test)

  • New: tests/issue4796-gc-batched-sweep.test.ts — 7 cases (fresh install + v8→v9 migration gain the index and selection uses it; multi-MB blob sweep over one batch; exact multiple of the batch size; mid-sweep crash convergence; e2e v8 wireIndex twin release; lifecycle ops scoped to their own hashes; deleteCheckpoint scoping).
  • New: tests/issue4612-gc-online-guard.test.ts — 5 cases (live peer → full gc refuses while incremental proceeds; dead holder stops blocking; later-started peer caught by fresh lockfile read; stale lockfile entry ignored; multi-batch sweep with progress lines and zero surviving garbage).
  • Re-ran untouched suites touching the same files: storage-unified, storage-ext, storage-ingest-watermark, issue4677-v8-migration-streaming, storage-backend-contract, storage-backend-knob — 65 passed / 0 failed total across these six-plus-two files.
  • npm run typecheck (tsc --noEmit after the kernel build step) clean.

Fixes #4612
Fixes #4796

This supersedes #4800 and #4628 — please close those two once this merges.

ci-check and others added 12 commits October 25, 2026 07:06
…the db (#4612)

Field report (bili 0.1.204, 14 GB index.db): an online 'storage gc'
(vacuum=full) held the exclusive write lock across the whole O(db-size)
VACUUM rewrite for 14 minutes with zero progress output, starving the
running proxy until its unified shadow layer degraded (3 consecutive
'database is locked' failures -> legacy-only for the process lifetime).

- UnifiedStore.activeForeignWriter(): union of the open-time captured
  #4175 lock holder (revalidated for liveness) and a fresh <db>.lock read
  (catches an opener that started after us); fail-open on unreadable file.
- UnifiedStore.gc(): vacuum:'full' refuses up front when a live foreign
  writer is present (actionable error naming the pid); incremental /
  no-vacuum sweeps are short transactions and stay allowed.
- Zero-ref sweep now commits in bounded batches (rowid subselect; SQLite
  DELETE takes no LIMIT) with per-batch progress lines; batch grows with
  the table so log volume stays bounded. Crash-safe: rows are garbage by
  definition, leftovers are collected by the next sweep.
- Full-vacuum sizing logs: main/WAL size before, elapsed + size delta
  after.
- CLI lane surfaces the refusal as exit 1 + actionable text; panel
  endpoint returns 409. Dry-run stays read-only.
Dummy live process (spawned node child) stands in for the running
proxy: open-time holder capture, later-opener lockfile detection, liveness
re-validation on exit, incremental/dry-run exemptions, batched zero-ref
sweep outcome + progress lines, and the CLI refusal surface (code 1,
actionable text, json fields).
README en+zh: 'storage gc --vacuum full' refuses to start while another
live process holds the database (CLI exit 1 / panel HTTP 409); use
--vacuum incremental for online-safe shrink.
The new gc online-guard test spawns an idle node child as its writer
fixture; the run-ci-tests SPAWN_RE tripwire requires every such suite in
LIVE_FILES or it aborts all shards before any test runs (#4604 class).
…ted red)

Unused-exports gate fails on every PR head rebased onto master tip
146b34f with the same line master itself fails on:
INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck
(#4545 train left the type exported though nothing outside the file
imports it; missed by the 4d34abf de-export sweep). One-word fix,
identical to #4606 commit 1; drop this commit when rebasing after
#4606 merges.
…4796)

The gc() sweep deleted zero-ref content rows with one uncommitted
statement over an unindexed scan. On bloated stores (field report:
766K rows / ~20 GB, 658K orphans) it never committed - invisible to
every other connection for hours while holding the exclusive write
lock and starving live proxies - so orphan content accumulated
forever (v8 identity backfill releases refs without delete-on-zero
and relies on gc to collect).

- schema v9: idx_content_refcount on content(refcount); candidate
  selection becomes an index range scan; one-time build cost is what
  open-time quick_check already pays on every open
- sqlite gc(): autocommit batches of 2000 (select candidates by hash,
  delete re-filtered under each statement's own snapshot - no explicit
  transaction, which would pin the read snapshot and risk deleting a
  concurrently re-referenced row), best-effort wal_checkpoint(TRUNCATE)
  between batches, progress logs
- mysql backend mirrored: DDL KEY + probe/ALTER in init(), DELETE ...
  LIMIT ? batches in gc()
- regression suite incl. an end-to-end repro of the v8-backfill
  residue path; version-pin ripples in four existing suites
…hashes (#4796)

PR #4800 review item 1: deleteSession/archiveSession/deleteCheckpoint
(sqlite + mysql, 6 sites) each finished with a full-table
DELETE FROM content WHERE refcount <= 0 inside their lifecycle
transaction — the same giant-commit pathology as the old gc sweep on
a bloated store. Their own releases already go through
releaseContent's delete-on-zero, so the trailing sweep only ever
collected OTHER sessions' pre-existing orphans incidentally.

Replace it with collectZeroRefIn(): a chunked DELETE re-filtering
refcount <= 0 over just the operation's released-hash set — bounded
work, transaction atomicity kept. Pre-existing orphans now persist
until the next gc run (disclosed semantics change; gc remains the
collector of record). archiveSession.contentRowsDeleted now counts
this operation's scoped deletions only.
# Conflicts:
#	src/storage/mysql.ts
#	src/storage/schema.ts
Conflict resolution notes:
- store.ts constructor: keep master's signature (dbPath last param, #4852
  integrity work); #4628's extra path param drops in favour of this.dbPath.
- gc() zero-ref sweep: keep #4800's index-driven batched collection
  (collectZeroRefContent, fixed 2000-row batches) instead of #4628's
  rowid-IN-subquery adaptive batches — with idx_content_refcount present,
  candidate selection is an index range scan and per-batch TRUNCATE
  checkpoints bound WAL growth. #4612's test asserts the actual log format.
- live-files.mjs: union of both sides' entries.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-30_gc-combined-4612-4796 (428d87d)

Latest build — install this exact version

npm install -g billion-context@0.0.6934-pr.5193.846e6b9

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
428d87d billion-context@0.0.6934-pr.5193.846e6b9 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5193.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title storage gc: batch orphan-content collection + refuse full VACUUM while another process holds the db (#4612, #4796) [PR] storage gc: batch orphan-content collection + refuse full VACUUM while another process holds the db (#4612, #4796) Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.924-master.6db5906 (rolling master tag).

npm install -g billion-context@0.0.924-master.6db5906  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
…p plateau pin

Schema v9 (#4796/#5193) adds idx_content_refcount, whose index-maintenance
frames ride every content insert and shift where the 1000-page autocheckpoint
window lands inside this fixed ingest sequence. The post-reset tail grew from
34.2 KiB (pre-v9, 1083592) to 54.3 KiB locally / 66.4 KiB on CI (1116552;
SQLite build page packing moves the crossing point ~±12 KiB between runner
images), tripping the 64 KiB budget that #4506 set for the same drift class.

The truncate-at-checkpoint invariant is intact — every reset lands exactly on
the configured cap (measured) — so follow the in-file precedent and widen
CUSTOM_CAP_SLACK 64 -> 128 KiB. Still discriminating: a cap that failed to
reach the pragma would leave the file at its ~4.4 MiB high-water mark, far
above the bound. Verified: pre-v9 tree 1083592, v9 tree w/o index 1083592,
v9 tree 1104192 local; suite 5/5 green.

Follows #4796 #5193 #4506
ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
Schema v9 (#4796/#5193, additive refcount index + version bump) invalidated
this suite's stale user_version == 8 pins; the sibling
issue4677-v8-migration-streaming suite was already updated to 9 by the schema
change. Bump the five assertions/comments to 9. The mysql GET_LOCK lock name
(bili_v8_upgrade_) is unchanged in src/storage/mysql.ts, so the contention
simulation stays valid. Verified: 2 pass / 3 skip (DSN-gated) / 0 fail.

Follows #4796 #5193
Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Oct 30, 2026
ranxianglei#4538: sampled read-back verification in bounded batches of 32 (headers
separately, blocks in persisted order, message hydration only at the
verifier's pointer seqs); sync and async share one code path; SQLite and
MySQL get bounded LIMIT/OFFSET + seq-IN reads.

Conflict resolutions (p4540 base is pre-ranxianglei#5193/ranxianglei#4765/ranxianglei#4750; our side is the
superset in every region):
- src/storage/backend.ts: keep ours — p4540 added no content beyond ours
  (its sessionViewRows/readSnapshot/StorageReadView surface already present
  from ranxianglei#4765/ranxianglei#4750 merges); verified diff shows zero theirs-only lines.
- src/storage/store.ts: union — p4540's getBlocks 'ORDER BY ord, block_id'
  total ordering (stable pagination across generations) onto our gc()
  batch-sweep constants, viewPointerSeqs, readSnapshot/readSnapshotAsync,
  activeForeignWriter, formatDbSize; comment kept ours' pluginSnapshotSeqs
  wording (documents the live field shape).
- src/storage/mysql.ts: union — p4540's dedicated-pool reads with
  hydrateMediaAsync(conn) parameterization onto our v9 stamp +
  idx_content_refcount probe block + collectZeroRefIn lifecycle scoping;
  theirs-only lines audited line-by-line: all stale base versions of code
  our branch already carries (ranxianglei#5193 master state).
- tests/plugin-fork.test.ts: keep ours' longer ranxianglei#4599 comment,
  replicasAvailable() gate (supersedes theirs' storageMode==="full"), and
  ours' ranxianglei#5140 raw-tail cold-restart test semantics (200, not 409); p4540's
  storageMode loop + env save/restore already on our side via ranxianglei#4765.
- tests/storage-orphan-recovery.test.ts: take p4540's extra assert
  (alignedRef === "m00001") — pure addition, variable already defined.
ranxianglei pushed a commit that referenced this pull request Nov 2, 2026
…ap case (#5539)

The six FK-child indexes added by schema v11 append index-maintenance frames to every insert on messages/blocks/ccr_entries/message_media/checkpoints. In the #3250 custom-cap sequence the post-reset tail therefore grew from 54.3 KiB locally / 66.4 KiB on CI (pre-v11) to 130.8-146.9 KiB (1182472-1198952; identical 1198952 on ubuntu-24 shard 3), past the 128 KiB budget. Same drift class as #4506 and #4796/#5193 (documented in-test): every checkpoint reset still truncates exactly to the configured cap (verified per-ingest on both trees), so the inter-reset budget widens 128 -> 256 KiB with provenance; the bound stays far below the ~4.4 MiB default-limit plateau the assertion discriminates against.
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…way combo

Connection recipe (same /bili/ prefix pattern as magpie) plus the four
load-bearing caveats from #5193: stickiness keys off body
metadata.conversation_id only (no header fallback), namespaced model
names need providers window declarations, Qoder lane reports zero token
usage upstream, mid-conversation account switches are invisible to
KEY SWITCHES attribution. en + zh kept in sync.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant