Repository navigation
Conversation
tamirms
added this pull request to stack #1089
October 6, 2026 09:22
tamirms
removed this pull request from stack #1089
October 8, 2026 08:18
tamirms
force-pushed
the
tamirms/events-cold-index-runs
branch
from
October 8, 2026 08:18
39f9553 to
65cedca
Compare
tamirms
added this pull request to stack #1102
October 8, 2026 08:19
tamirms
force-pushed
the
tamirms/events-cold-index-runs
branch
from
October 8, 2026 08:34
65cedca to
ecd07ab
Compare
tamirms
force-pushed
the
tamirms/events-cold-index-runs
branch
2 times, most recently
from
October 9, 2026 06:02
038ba1a to
17e0837
Compare
The cold events index was built from a map of every term of the chunk to its bitmap, held in memory for the whole backfill or freeze of the chunk. Its memory grew with the number of distinct terms in the chunk, once per chunk being built. ColdIndexBuilder keeps one slab of postings in memory, in a slab buffer of the kind the hot index fills, keyed by the routed term key so that a slab's terms come out in the order the MPHF builder wants. The slab is written as a run file, in a directory beside the index files, when the next slab starts or when the buffer fills, and Write merges the runs three times: to count the terms, to build index.hash with the sorted streamhash builder, and to write index.pack. Terms arrive in MPHF block order and a block's slots are contiguous, so index.pack is written in slot order holding back at most one block of entries. The runs are removed once the index is written, or when the build is abandoned. A build that crashes leaves them beside the chunk's "freezing" key, so the catalog sweep removes the directory, as a scratch path of the kind, when it destroys the key; a rebuild reads only the runs it wrote itself. The MPHF is built with the sorted builder, so streamhash no longer writes partition files under the temp directory. Bitmaps, the whole-chunk map, is now a test fixture. The design documents describe the build and the scratch directory. Measured on the same real pubnet chunk (10,000 ledgers, 10.19M events) built twice each way from the bulk backend: the chunk took 75.7 and 74.9 s before, 77.1 and 77.2 s after; peak RSS 1,486 and 1,523 MB before, 730 and 727 MB after. On a chunk of 60M synthetic events the live heap at 28M events was 875 MB, of which the index held 10.5 MB, and the peak RSS of 5.1 GB was reached before the index was written. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiCL19kUrno3Z4SAYgpAfh
tamirms
force-pushed
the
tamirms/events-cold-index-runs
branch
from
October 15, 2026 19:15
17e0837 to
c004318
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The cold events index is built at every freeze and in every backfill worker, and the build kept every term's bitmap for the whole chunk in memory until the chunk was done: about half a gigabyte on a pubnet chunk, 24 GB on an adversarial one, multiplied by the number of workers. The minimal perfect hash builder then wrote its own partition files under the temp directory.
The idea
Make the build an external sort. Postings go into the same fixed slab buffer the hot index uses. When it fills, its terms are written to disk as one sorted run. At the end the runs are merged in a single streaming pass, and the index files are written as the merged terms come out. Memory is one slab whatever the chunk holds. Because the runs are sorted by the term's routed key, the merged order is exactly the order the index files need, so the writer never holds back more than one block of entries.
How it works
ColdIndexBuilderspills a run per full slab into a directory beside the index files (#1099 moves it to the local disk).Writemerges the runs three times: to count the terms, to buildindex.hashwith the sorted streamhash builder, and to writeindex.pack. The runs are deleted once the index is written. If the build crashes they are left beside the chunk's "freezing" key, and the catalog sweep that destroys the key removes them as that key's scratch directory, so no new cleanup code was needed. The whole-chunk mapBitmapsis now only a test fixture.Building the cold index from the hot database's sealed slabs instead would serve the freeze but not the backfill, which has no hot database, so both take their postings from the raw ledgers.
Measured
A real pubnet chunk (10,000 ledgers, 10.19 million events) built twice each way from the bulk backend:
On a chunk of 60 million synthetic events, the live heap at 28 million events was 875 MB, of which the index held 10.5 MB. That run's 5.1 GB peak was reached before the index was written and belongs to the tx-hash cold accumulator, the next item.
🤖 Generated with Claude Code