Skip to content

[PR] perf(storage): join block content in SQLite reads - #6618

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_join-block-content
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_join-block-content

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #6616.

SQLite block reads first select the block rows and then execute up to two separate content-addressed lookups per row. A session with 5,000 compression blocks therefore makes up to 10,000 extra statement executions for one complete read. Both the block-list path and the paginated block-content path incur this cost.

Fetch the two content blobs with LEFT JOINs in the same query. For bounded reads, select and order the requested page before joining its content, so skipped rows do not hydrate their blobs. Preserve the final ord, block_id ordering, existing Buffer UTF-8 decoding, missing-content fallbacks, nullable one-line views and row-over-orphan precedence. The content hash is unique, so the joins cannot multiply block rows.

Measured effect

Seven alternating fresh-process baseline/candidate pairs for each workload on Node 24.22.1 with the real SQLite store. The included driver ingests real messages and compression blocks, then times complete public reads after one separate warmup per method. Setup, output hashing and teardown are outside the timer.

Each number below is the median total elapsed time for the stated repetitions, not a per-read measurement.

Workload Repetitions Complete block list, before / after All 128-row content pages, before / after Decoded content map, before / after
5,000 blocks 5 725.33 / 207.80 ms 824.50 / 352.76 ms 668.60 / 240.82 ms
8 blocks 50 10.77 / 5.66 ms 10.21 / 4.29 ms 12.13 / 8.72 ms
Empty block list 50 0.76 / 1.73 ms 0.83 / 1.76 ms 1.52 / 2.64 ms

All seven large-workload pairs improve for each method. The small paginated control improves in six of seven pairs; the other small methods improve in all seven. Empty-store reads are slower in every pair because the joined query has extra fixed work: their median total penalty is about 0.9–1.1 ms over 50 reads. One candidate empty run had a 32.7 ms block-list outlier. This is a tradeoff for populated sessions, not a claim that every workload improves.

Every baseline/candidate process has an identical logical input hash and identical output hashes for all three methods in its workload. Decoded maps also include one deliberately retained orphan, including the empty-block control. No request-wide latency, memory or network-backend improvement is claimed.

Reproduce at both revisions with the same driver:

node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-block-content-reads.ts 5000 5
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-block-content-reads.ts 8 50
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-block-content-reads.ts 0 50

Validation

  • Five new real-database compatibility cases cover complete and paginated field/order equality, duplicate block IDs across sessions, orphan precedence, missing CAS rows, and zero-byte/invalid-UTF-8 blob decoding. They pass on both the original and optimized implementation. Optional better-sqlite3 copies skip when that driver is unavailable.
  • New cases plus bounded-load/read-snapshot suites: 14 passed, 7 skipped on both revisions.
  • Broader storage, unified-mode and bounded-session suites on the candidate: 35 passed, 8 skipped.
  • Typecheck, complete build, unused-export guard, test-import-graph guard and git diff --check pass.
  • Independent review checked unique-key join cardinality, LEFT JOIN fallback semantics, byte decoding and pagination-before-hydration ordering.

Scope

Only SQLite block-read queries change. No schema, format/version, configuration, kernel, wire-shape or MySQL implementation changes. Current-head remote CI is a separate check after publication.

@ranxianglei ranxianglei changed the title perf(storage): join block content in SQLite reads [PR] perf(storage): join block content in SQLite reads Nov 13, 2026
@ranxianglei
ranxianglei merged commit d343336 into ranxianglei:master Nov 14, 2026
44 of 48 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit d34333666be (Merge Afloat16/2026-11-13_join-block-content (PR #6618)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6616 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite block readers issue separate CAS queries for every block view

2 participants