Skip to content

[PR] perf(storage): index session GC companion suffix lookups - #6611

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_index-companion-lookups
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_index-companion-lookups

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6609

Problem

The content-store co-deletion guard and orphan sweep scan all session basenames for each companion. A directory with 5,000 live sessions and 5,000 old but still-referenced companions therefore performs millions of repeated basename/suffix checks on every GC sweep, despite retaining every file.

Change

Build a per-sweep reversed-basename index lazily when a companion actually needs matching. Binary search finds the existing arbitrary suffix matches; matching paths are restored to the original filesystem-walk order before the existing twin checks run. This preserves legacy names and cross-namespace fallback without assuming a hash length or an underscore separator.

The index is built once in O(N log N) time and uses O(N) entries plus reversed filename strings. A lookup costs O(log N + K log K) for K matches, including restoration of probe order. Sweeps that never query companions do not build the index. Existing deleted-session filtering, readable-own-store checks, age/token gates, resident-session protection, and deletion/audit behavior remain in place.

Verification

  • Ten real-filesystem compatibility cases cover arbitrary suffixes and Unicode, cross-namespace references, similar nonmatching names, unreadable twin stores, deletion of both twins in one sweep, original twin-probe order, fresh orphan preservation, and a changed filesystem on the next sweep. All pass on the original implementation and this patch.
  • The new cases plus the existing session-GC suite: 36 passed on both baseline and candidate, with BILI_STORAGE_JSON_WRITER=1 BILI_PERSIST_ZSTD=0 to exercise the file-persistence contract expected by that suite.
  • Under the unpinned ambient storage defaults, the same two existing tests fail on both baseline and candidate: the pending-save case and the rawInputTokens JSON-file persistence case. Both pass with the file writer enabled; no unrelated persistence-policy change is included.
  • Typecheck, full build, unused-export and test-import-graph guards, and git diff --check pass.
  • Independent review checked suffix equivalence, preservation of original probe order, and the live deleted-session filter.

Reproducible measurements

The included benchmark creates real session and content-store files, warms up once, and times complete gcSessionFiles calls. Setup and SHA-256 verification are outside the timed region. Every filename and file-content byte is hashed before and after the sweeps, and the full GC result is checked.

Seven fresh-process baseline/candidate pairs per workload, alternating order, on Node 24.22.1 / Linux x86_64:

Workload Sweeps per sample Baseline median Candidate median
5,000 sessions + 5,000 referenced companions 3 12,828.30 ms 1,075.58 ms
5,000 sessions, no companions (control) 3 476.37 ms 527.40 ms
16 sessions + 16 companions (control) 10 35.56 ms 37.39 ms

The large companion case improves in all seven pairs (7.81–15.65× paired wall-time ratio). Median CPU time is 12,390.16 → 1,006.53 ms; wall-time ranges are 11,706.59–14,870.34 ms and 842.60–1,499.01 ms. All fourteen input/output hashes match within each workload, with every expected path retained.

Both control medians increase. Their ranges overlap; the no-companion candidate is slower in six of seven pairs, while the small control has mixed results. These controls do not establish a speedup or a no-regression guarantee. The measured benefit is specifically the many-companion matching workload. This is not a claim about general request latency, all GC layouts, or overall memory use.

node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-session-gc-companions.ts 5000 5000 3
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-session-gc-companions.ts 5000 0 3
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-session-gc-companions.ts 16 16 10

Run the same driver on baseline e1f31ad9d5b47500362699a9867819b861c7de96 and the candidate. No dependency, configuration surface, schema, or kernel changes.

@ranxianglei ranxianglei changed the title perf(storage): index session GC companion suffix lookups [PR] perf(storage): index session GC companion suffix lookups Nov 13, 2026
@ranxianglei
ranxianglei merged commit f6a4c01 into ranxianglei:master Nov 14, 2026
44 of 48 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit f6a4c010ed8 (Merge Afloat16/2026-11-13_index-companion-lookups (PR #6611)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6609 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session GC repeatedly scans all session names for each content-store companion

2 participants