Skip to content

[PR] fix(storage): keep retired-message reactivation linear - #6604

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_linear-reactivation
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_linear-reactivation

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6601

Problem

Full ingestion collects retired absolute message sequences and then scans the entire presented message array once for every retired row. With 30,000 stored messages and a 4,000-message visible window, a steady-state save performs 26,000 × 4,000 sequence comparisons even when no retired message is returning. This remains after #5772, which removed repeated retirement SQL work and deliberately left reactivation unchanged.

Change

Build the presented absolute-sequence set once when retired rows exist, in both SQLite and MySQL. The existing retired-row loop and SQL update order stay the same. Membership work becomes O(retired rows + presented messages), with O(presented messages) temporary space. Sessions without retired rows do not allocate the set.

Identity remapping, sequence allocation, tombstone policy, media references, watermark construction, and transaction boundaries retain their existing behavior. The separate non-complete-log edited-generation proposal in #4888 is not included.

Validation

  • Real storage backend contract coverage checks sparse out-of-order reactivation under both complete-log modes, duplicate presented messages, original absolute sequence reuse, unchanged unrelated tombstones, image hydration, full-log edit/undo, empty windows, and unrelated sessions.
  • The new semantic controls pass on the original implementation and the candidate: 11 passed, 1 MySQL test skipped in the backend contract file. This is a performance correction, so the regression evidence is the measured work below rather than an artificial timing assertion in CI.
  • Six related storage suites: 40 passed, 1 skipped. They include media CAS, remap collision repair, ingestion watermarks, and the earlier retirement optimization.
  • npm run typecheck, npm run build, unused-export and test-import-graph guards, and git diff --check passed.
  • The shared contract runs against real MySQL when BILI_MYSQL_TEST_DSN is set. No MySQL service was available locally; the local runtime measurements use real SQLite only.

Reproducible measurement

The included scripts/bench-storage-reactivation.ts creates a real database, retires the prefix through normal completeLog ingestion, performs two warmups, and times full planIngest + ingestSession calls. Setup and read-back verification are outside the timed region. It emits input and logical read-back SHA-256 hashes, wall time, CPU time, and peak process RSS.

Seven fresh-process pairs per workload, alternating baseline/candidate order, on Node 24.22.1 / Linux x86_64. All fourteen input hashes and all fourteen read-back hashes match within each workload. The read-back includes message sequences/content hashes/retirement state, session data, watermarks, row counts, and CAS accounting; setup-generated retirement timestamps are normalized to retired/live state.

Workload Calls per sample Baseline median wall time Candidate median wall time
30,000 historical messages, 4,000 visible 10 4,835.46 ms 3,446.51 ms
3,000 messages, none retired (control) 10 1,889.72 ms 1,498.97 ms
32 messages, 8 visible (control) 100 120.36 ms 138.05 ms

The large retired-history case improved in all seven pairs (1.20–1.85× paired wall-time ratio); median CPU time was 4,942.50 → 3,438.53 ms. Its wall-time ranges were 4,387.42–6,379.64 ms and 2,887.67–4,366.74 ms. The controls have overlapping ranges and mixed paired results, so they do not establish a speedup; the small control's median increased. Peak RSS ranges also overlap. These are controlled local ingest measurements, not a claim about overall model/proxy latency or all workloads.

Reproduce one sample on each checkout, copying the unchanged benchmark driver to the baseline checkout:

node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-storage-reactivation.ts 30000 4000 10
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-storage-reactivation.ts 3000 3000 10
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-storage-reactivation.ts 32 8 100

Baseline: e1f31ad9d5b47500362699a9867819b861c7de96. No configuration surface, schema, dependency, or kernel changes.

@ranxianglei ranxianglei changed the title fix(storage): keep retired-message reactivation linear [PR] fix(storage): keep retired-message reactivation linear Nov 13, 2026
@ranxianglei
ranxianglei merged commit 4bec778 into ranxianglei:master Nov 14, 2026
43 of 48 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit 4bec77821c2 (Merge Afloat16/2026-11-13_linear-reactivation (PR #6604)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6601 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full ingest reactivation repeatedly scans the presented window for every retired row

2 participants