Repository navigation
[PR] perf(storage): join block content in SQLite reads - #6618
Merged
ranxianglei merged 1 commit intoNov 14, 2026
Merged
Conversation
This was referenced Nov 13, 2026
Owner
|
Superseded by #6677 (merged as Landed into master via the real merge commit The fix for #6616 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix. |
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6616.
SQLite block reads first select the block rows and then execute up to two separate content-addressed lookups per row. A session with 5,000 compression blocks therefore makes up to 10,000 extra statement executions for one complete read. Both the block-list path and the paginated block-content path incur this cost.
Fetch the two content blobs with
LEFT JOINs in the same query. For bounded reads, select and order the requested page before joining its content, so skipped rows do not hydrate their blobs. Preserve the finalord, block_idordering, existingBufferUTF-8 decoding, missing-content fallbacks, nullable one-line views and row-over-orphan precedence. The content hash is unique, so the joins cannot multiply block rows.Measured effect
Seven alternating fresh-process baseline/candidate pairs for each workload on Node 24.22.1 with the real SQLite store. The included driver ingests real messages and compression blocks, then times complete public reads after one separate warmup per method. Setup, output hashing and teardown are outside the timer.
Each number below is the median total elapsed time for the stated repetitions, not a per-read measurement.
All seven large-workload pairs improve for each method. The small paginated control improves in six of seven pairs; the other small methods improve in all seven. Empty-store reads are slower in every pair because the joined query has extra fixed work: their median total penalty is about 0.9–1.1 ms over 50 reads. One candidate empty run had a 32.7 ms block-list outlier. This is a tradeoff for populated sessions, not a claim that every workload improves.
Every baseline/candidate process has an identical logical input hash and identical output hashes for all three methods in its workload. Decoded maps also include one deliberately retained orphan, including the empty-block control. No request-wide latency, memory or network-backend improvement is claimed.
Reproduce at both revisions with the same driver:
Validation
better-sqlite3copies skip when that driver is unavailable.git diff --checkpass.LEFT JOINfallback semantics, byte decoding and pagination-before-hydration ordering.Scope
Only SQLite block-read queries change. No schema, format/version, configuration, kernel, wire-shape or MySQL implementation changes. Current-head remote CI is a separate check after publication.