Repository navigation
Conversation
tamirms
added this pull request to stack #1089
October 6, 2026 09:22
tamirms
removed this pull request from stack #1089
October 8, 2026 08:18
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 8, 2026 08:18
c4d415b to
07d6a19
Compare
tamirms
changed the base branch from
tamirms/events-cold-read-concurrency
to
tamirms/events-split-terms
October 8, 2026 08:18
tamirms
added this pull request to stack #1102
October 8, 2026 08:19
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
3 times, most recently
from
October 9, 2026 06:02
b2642ed to
2bc993a
Compare
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 15, 2026 19:15
2bc993a to
1364aef
Compare
Store.LoadSorted writes an ascending sequence of entries as one SST file under the store's own loading directory and moves it into the column family with IngestExternalFile. Readers see all of the entries or none, the entries survive a crash once the call returns, and they bypass the WAL and the memtable. The keys must not overlap keys the column family already holds: the file then lands below every other file and never needs compacting, and an overlap fails the load. The store's lifecycle lock is held for the whole load, so Close waits for it, and a read-write open removes the files a stopped process left behind. The per-column-family options are now kept by name, which the file writer needs to give a file the column family's own block size, filter and compression. CFOptions gains CacheIndexAndFilterBlocks, which keeps an SST's index and filter blocks in the block cache instead of on the heap, so a column family made of many loaded files cannot grow the heap with them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiCL19kUrno3Z4SAYgpAfh
tamirms
force-pushed
the
tamirms/rocksdb-bulk-load
branch
from
October 19, 2026 13:00
1364aef to
49a4f2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The hot events index was written one RocksDB key per posting into every ledger's commit batch, about 6,000 keys per ledger on pubnet, and each key passed through the write-ahead log, the memtable and later compaction. Those keys were most of each commit. The next PR in this stack stops doing that and instead writes a whole slab of the index at once, hundreds of thousands of entries, which needs a way to add them to a column family without paying those three costs.
The idea
RocksDB can adopt a file you wrote yourself. Write the entries in sorted order as one SST file, RocksDB's own file format, and hand it over with
IngestExternalFile: RocksDB links the file into the column family without passing a byte through the write-ahead log or the memtable. If the file's keys do not overlap any key the column family already holds, it goes straight to the bottom level and is never compacted. Readers see all of its entries at once, and they are durable when the call returns.What the store adds
Store.LoadSorted(cf, entries)writes the file under the store's ownloading/directory with the column family's own block size, filter and compression, ingests it, and fails if the keys overlap rather than creating compaction work. It holds the store's lifecycle lock soClosewaits for it, and a read-write open removes files a stopped process left behind. The per-column-family options are now kept by name so the file can be given them.CFOptions.CacheIndexAndFilterBlocksmoves each file's index and filter blocks from the heap into the block cache. Without it a column family made of many loaded files grows the heap by about 100 KB per file; with it the memory bound the next PR relies on holds.Measured
On the stress stream of #1097, 109 loaded files all sat at the bottom level with level 0 empty. With the index and filter blocks in the cache, the table readers' heap went from 10.5 MB to 0.23 MB with lookups unchanged. The ingest call took 15 to 17 ms at p50 on macOS, which is the fsync.
🤖 Generated with Claude Code