Repository navigation
[PR] storage gc: batch orphan-content collection + refuse full VACUUM while another process holds the db (#4612, #4796) - #5193
Merged
Conversation
…the db (#4612) Field report (bili 0.1.204, 14 GB index.db): an online 'storage gc' (vacuum=full) held the exclusive write lock across the whole O(db-size) VACUUM rewrite for 14 minutes with zero progress output, starving the running proxy until its unified shadow layer degraded (3 consecutive 'database is locked' failures -> legacy-only for the process lifetime). - UnifiedStore.activeForeignWriter(): union of the open-time captured #4175 lock holder (revalidated for liveness) and a fresh <db>.lock read (catches an opener that started after us); fail-open on unreadable file. - UnifiedStore.gc(): vacuum:'full' refuses up front when a live foreign writer is present (actionable error naming the pid); incremental / no-vacuum sweeps are short transactions and stay allowed. - Zero-ref sweep now commits in bounded batches (rowid subselect; SQLite DELETE takes no LIMIT) with per-batch progress lines; batch grows with the table so log volume stays bounded. Crash-safe: rows are garbage by definition, leftovers are collected by the next sweep. - Full-vacuum sizing logs: main/WAL size before, elapsed + size delta after. - CLI lane surfaces the refusal as exit 1 + actionable text; panel endpoint returns 409. Dry-run stays read-only.
Dummy live process (spawned node child) stands in for the running proxy: open-time holder capture, later-opener lockfile detection, liveness re-validation on exit, incremental/dry-run exemptions, batched zero-ref sweep outcome + progress lines, and the CLI refusal surface (code 1, actionable text, json fields).
README en+zh: 'storage gc --vacuum full' refuses to start while another live process holds the database (CLI exit 1 / panel HTTP 409); use --vacuum incremental for online-safe shrink.
The new gc online-guard test spawns an idle node child as its writer fixture; the run-ci-tests SPAWN_RE tripwire requires every such suite in LIVE_FILES or it aborts all shards before any test runs (#4604 class).
…ted red) Unused-exports gate fails on every PR head rebased onto master tip 146b34f with the same line master itself fails on: INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck (#4545 train left the type exported though nothing outside the file imports it; missed by the 4d34abf de-export sweep). One-word fix, identical to #4606 commit 1; drop this commit when rebasing after #4606 merges.
…4796) The gc() sweep deleted zero-ref content rows with one uncommitted statement over an unindexed scan. On bloated stores (field report: 766K rows / ~20 GB, 658K orphans) it never committed - invisible to every other connection for hours while holding the exclusive write lock and starving live proxies - so orphan content accumulated forever (v8 identity backfill releases refs without delete-on-zero and relies on gc to collect). - schema v9: idx_content_refcount on content(refcount); candidate selection becomes an index range scan; one-time build cost is what open-time quick_check already pays on every open - sqlite gc(): autocommit batches of 2000 (select candidates by hash, delete re-filtered under each statement's own snapshot - no explicit transaction, which would pin the read snapshot and risk deleting a concurrently re-referenced row), best-effort wal_checkpoint(TRUNCATE) between batches, progress logs - mysql backend mirrored: DDL KEY + probe/ALTER in init(), DELETE ... LIMIT ? batches in gc() - regression suite incl. an end-to-end repro of the v8-backfill residue path; version-pin ripples in four existing suites
…hashes (#4796) PR #4800 review item 1: deleteSession/archiveSession/deleteCheckpoint (sqlite + mysql, 6 sites) each finished with a full-table DELETE FROM content WHERE refcount <= 0 inside their lifecycle transaction — the same giant-commit pathology as the old gc sweep on a bloated store. Their own releases already go through releaseContent's delete-on-zero, so the trailing sweep only ever collected OTHER sessions' pre-existing orphans incidentally. Replace it with collectZeroRefIn(): a chunked DELETE re-filtering refcount <= 0 over just the operation's released-hash set — bounded work, transaction atomicity kept. Pre-existing orphans now persist until the next gc run (disclosed semantics change; gc remains the collector of record). archiveSession.contentRowsDeleted now counts this operation's scoped deletions only.
# Conflicts: # src/storage/mysql.ts # src/storage/schema.ts
Conflict resolution notes: - store.ts constructor: keep master's signature (dbPath last param, #4852 integrity work); #4628's extra path param drops in favour of this.dbPath. - gc() zero-ref sweep: keep #4800's index-driven batched collection (collectZeroRefContent, fixed 2000-row batches) instead of #4628's rowid-IN-subquery adaptive batches — with idx_content_refcount present, candidate selection is an index range scan and per-batch TRUNCATE checkpoints bound WAL growth. #4612's test asserts the actual log format. - live-files.mjs: union of both sides' entries.
This was referenced Oct 30, 2026
Contributor
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6934-pr.5193.846e6b9This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5193.tgz
npm install -g packageInstall instructions are refreshed on each push. |
Contributor
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.924-master.6db5906 # this exact build
npm install -g billion-context@master # always the newest merged build
|
ranxianglei
pushed a commit
that referenced
this pull request
Oct 30, 2026
…p plateau pin Schema v9 (#4796/#5193) adds idx_content_refcount, whose index-maintenance frames ride every content insert and shift where the 1000-page autocheckpoint window lands inside this fixed ingest sequence. The post-reset tail grew from 34.2 KiB (pre-v9, 1083592) to 54.3 KiB locally / 66.4 KiB on CI (1116552; SQLite build page packing moves the crossing point ~±12 KiB between runner images), tripping the 64 KiB budget that #4506 set for the same drift class. The truncate-at-checkpoint invariant is intact — every reset lands exactly on the configured cap (measured) — so follow the in-file precedent and widen CUSTOM_CAP_SLACK 64 -> 128 KiB. Still discriminating: a cap that failed to reach the pragma would leave the file at its ~4.4 MiB high-water mark, far above the bound. Verified: pre-v9 tree 1083592, v9 tree w/o index 1083592, v9 tree 1104192 local; suite 5/5 green. Follows #4796 #5193 #4506
ranxianglei
pushed a commit
that referenced
this pull request
Oct 30, 2026
Schema v9 (#4796/#5193, additive refcount index + version bump) invalidated this suite's stale user_version == 8 pins; the sibling issue4677-v8-migration-streaming suite was already updated to 9 by the schema change. Bump the five assertions/comments to 9. The mysql GET_LOCK lock name (bili_v8_upgrade_) is unchanged in src/storage/mysql.ts, so the contention simulation stays valid. Verified: 2 pass / 3 skip (DSN-gated) / 0 fail. Follows #4796 #5193
Louie0806
pushed a commit
to Louie0806/billion-context
that referenced
this pull request
Oct 30, 2026
ranxianglei#4538: sampled read-back verification in bounded batches of 32 (headers separately, blocks in persisted order, message hydration only at the verifier's pointer seqs); sync and async share one code path; SQLite and MySQL get bounded LIMIT/OFFSET + seq-IN reads. Conflict resolutions (p4540 base is pre-ranxianglei#5193/ranxianglei#4765/ranxianglei#4750; our side is the superset in every region): - src/storage/backend.ts: keep ours — p4540 added no content beyond ours (its sessionViewRows/readSnapshot/StorageReadView surface already present from ranxianglei#4765/ranxianglei#4750 merges); verified diff shows zero theirs-only lines. - src/storage/store.ts: union — p4540's getBlocks 'ORDER BY ord, block_id' total ordering (stable pagination across generations) onto our gc() batch-sweep constants, viewPointerSeqs, readSnapshot/readSnapshotAsync, activeForeignWriter, formatDbSize; comment kept ours' pluginSnapshotSeqs wording (documents the live field shape). - src/storage/mysql.ts: union — p4540's dedicated-pool reads with hydrateMediaAsync(conn) parameterization onto our v9 stamp + idx_content_refcount probe block + collectZeroRefIn lifecycle scoping; theirs-only lines audited line-by-line: all stale base versions of code our branch already carries (ranxianglei#5193 master state). - tests/plugin-fork.test.ts: keep ours' longer ranxianglei#4599 comment, replicasAvailable() gate (supersedes theirs' storageMode==="full"), and ours' ranxianglei#5140 raw-tail cold-restart test semantics (200, not 409); p4540's storageMode loop + env save/restore already on our side via ranxianglei#4765. - tests/storage-orphan-recovery.test.ts: take p4540's extra assert (alignedRef === "m00001") — pure addition, variable already defined.
ranxianglei
pushed a commit
that referenced
this pull request
Nov 2, 2026
…ap case (#5539) The six FK-child indexes added by schema v11 append index-maintenance frames to every insert on messages/blocks/ccr_entries/message_media/checkpoints. In the #3250 custom-cap sequence the post-reset tail therefore grew from 54.3 KiB locally / 66.4 KiB on CI (pre-v11) to 130.8-146.9 KiB (1182472-1198952; identical 1198952 on ubuntu-24 shard 3), past the 128 KiB budget. Same drift class as #4506 and #4796/#5193 (documented in-test): every checkpoint reset still truncates exactly to the configured cap (verified per-ingest on both trees), so the inter-reset budget widens 128 -> 256 KiB with provenance; the bound stays far below the ~4.4 MiB default-limit plateau the assertion discriminates against.
ranxianglei
pushed a commit
that referenced
this pull request
Nov 11, 2026
…way combo Connection recipe (same /bili/ prefix pattern as magpie) plus the four load-bearing caveats from #5193: stickiness keys off body metadata.conversation_id only (no header fallback), namespaced model names need providers window declarations, Qoder lane reports zero token usage upstream, mid-conversation account switches are invisible to KEY SWITCHES attribution. en + zh kept in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Two independent failures of
bili storage gcin the field, both reported by users:contenttable (658,125 of 766,125 rows) are unreferenced,bili storage gc --vacuum fullran 8 hours / 28,131 s of CPU, deleted zero rows, and the WAL grew to 221 MB before vacuum could even start. Cause: no index oncontent(refcount)made candidate selection a full-table scan, and the unbounded DELETE ran as one giant transaction holding the exclusive write lock the whole time.database is lockedduring the run and its unified-storage shadow layer tripped to legacy-only for the rest of the process life. Nothing refused or warned before the contention began.Both fixes were drafted separately (#4800, #4628). They edit the same lines of
UnifiedStore.gc()and cannot be merged independently — this single PR carries both, conflict-resolved.Changes
Batched orphan-content collection (from #4800)
idx_content_refcount; candidate selection becomes an index range scan instead of a full-table scan. Existing v8 dbs gain the index on next open.PRAGMA wal_checkpoint(TRUNCATE)between batches (bounds WAL growth, releases the write lock between batches). Progress line every 10th batch and always on the final batch:[#4796] gc: collected N zero-ref content rows so far (batch k). A sweep that dies mid-run converges on the next gc —refcount <= 0rows are garbage by definition.deleteSession,archiveSession,deleteCheckpoint) no longer trigger whole-table sweeps: they collect only within the hashes their own operation released (chunks of 500).Online guard against full VACUUM (from #4628)
UnifiedStore.activeForeignWriter(): union of the lockfile holder captured at open (pid liveness re-checked) and a fresh re-read of<db>.lock, so peers that started after this store opened are also caught. Returns a live pid or null.gc({ vacuum: "full" })now refuses up front with an actionable error naming the pid instead of silently starving the peer for the whole rewrite. Incremental / no-vacuum sweeps remain online-safe (short transactions covered by the existing busy_timeout handling).--vacuum incrementalhint (src/storage-maintenance.ts); panel gc endpoint returns HTTP 409 withwriterPid(src/server/admin.ts); exposed throughSessionStore.unifiedActiveWriter()andUnifiedShadow.activeForeignWriter().MySQL schema-stamp gap found during integration
Neither draft bumped
MYSQL_SCHEMA_VERSION; while they developed, master moved 6 → 8. An existing MySQL install stamped 8 would never enter the upgrade block and would be missing the new index forever. This branch bumps the stamp to 9 and adds an open-timeinformation_schemaprobe that adds the index once for already-stamped dbs.How the conflicts were resolved
deleted/total (%)progress format is superseded by thebatch kcadence line.pathconstructor parameter drops out in favor of master'sdbPathfield (added by the [PR] fix(storage): DB open self-check no longer blocks startup — large-DB background check + verdict cache (#4609) | fix(storage): 开库自检不再阻塞启动——大库后台查 + 判定缓存(#4609) #4852 quick-check work after both drafts branched).tests/issue4612-gc-online-guard.test.ts's sweep case was retargeted to assert the final log format and batch cadence.Behavior changes
bili storage gc --vacuum fullfails fast (exit 1, actionable message naming the peer pid) while another live process holds the database. Previously it would quietly hold the exclusive lock for the entire rewrite.vacuum: "full"returns HTTP 409 in the same situation.Verification (local,
node --import tsx --test)tests/issue4796-gc-batched-sweep.test.ts— 7 cases (fresh install + v8→v9 migration gain the index and selection uses it; multi-MB blob sweep over one batch; exact multiple of the batch size; mid-sweep crash convergence; e2e v8 wireIndex twin release; lifecycle ops scoped to their own hashes; deleteCheckpoint scoping).tests/issue4612-gc-online-guard.test.ts— 5 cases (live peer → full gc refuses while incremental proceeds; dead holder stops blocking; later-started peer caught by fresh lockfile read; stale lockfile entry ignored; multi-batch sweep with progress lines and zero surviving garbage).npm run typecheck(tsc --noEmit after the kernel build step) clean.Fixes #4612
Fixes #4796
This supersedes #4800 and #4628 — please close those two once this merges.