Repository navigation
[PR] fix(storage): keep retired-message reactivation linear - #6604
Merged
ranxianglei merged 1 commit intoNov 14, 2026
Merged
Conversation
Owner
|
Superseded by #6677 (merged as Landed into master via the real merge commit The fix for #6601 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6601
Problem
Full ingestion collects retired absolute message sequences and then scans the entire presented message array once for every retired row. With 30,000 stored messages and a 4,000-message visible window, a steady-state save performs 26,000 × 4,000 sequence comparisons even when no retired message is returning. This remains after #5772, which removed repeated retirement SQL work and deliberately left reactivation unchanged.
Change
Build the presented absolute-sequence set once when retired rows exist, in both SQLite and MySQL. The existing retired-row loop and SQL update order stay the same. Membership work becomes O(retired rows + presented messages), with O(presented messages) temporary space. Sessions without retired rows do not allocate the set.
Identity remapping, sequence allocation, tombstone policy, media references, watermark construction, and transaction boundaries retain their existing behavior. The separate non-complete-log edited-generation proposal in #4888 is not included.
Validation
npm run typecheck,npm run build, unused-export and test-import-graph guards, andgit diff --checkpassed.BILI_MYSQL_TEST_DSNis set. No MySQL service was available locally; the local runtime measurements use real SQLite only.Reproducible measurement
The included
scripts/bench-storage-reactivation.tscreates a real database, retires the prefix through normalcompleteLogingestion, performs two warmups, and times fullplanIngest+ingestSessioncalls. Setup and read-back verification are outside the timed region. It emits input and logical read-back SHA-256 hashes, wall time, CPU time, and peak process RSS.Seven fresh-process pairs per workload, alternating baseline/candidate order, on Node 24.22.1 / Linux x86_64. All fourteen input hashes and all fourteen read-back hashes match within each workload. The read-back includes message sequences/content hashes/retirement state, session data, watermarks, row counts, and CAS accounting; setup-generated retirement timestamps are normalized to retired/live state.
The large retired-history case improved in all seven pairs (1.20–1.85× paired wall-time ratio); median CPU time was 4,942.50 → 3,438.53 ms. Its wall-time ranges were 4,387.42–6,379.64 ms and 2,887.67–4,366.74 ms. The controls have overlapping ranges and mixed paired results, so they do not establish a speedup; the small control's median increased. Peak RSS ranges also overlap. These are controlled local ingest measurements, not a claim about overall model/proxy latency or all workloads.
Reproduce one sample on each checkout, copying the unchanged benchmark driver to the baseline checkout:
Baseline:
e1f31ad9d5b47500362699a9867819b861c7de96. No configuration surface, schema, dependency, or kernel changes.