Conversation
tamirms
added this pull request to stack #1044
September 25, 2026 19:41
A getEvents query was answered by intersecting each filter's term bitmaps across the whole chunk, unioning the results across filters, and only then restricting to the window. Its cost tracked the terms' size across the chunk, not the window the client asked for or the page it would receive. The engine now walks the window one slab at a time: 65,536 consecutive event ids, roaring's container size, in the requested direction. Each filter becomes plans that are plain ANDs of terms. On each slab, every plan is one FastAnd over the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs it appears in. Work scales with the window and the page, memory per query with one container, and both directions share one path. The first batch is the caller's page size, so a full page arrives in one fetch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
tamirms
force-pushed
the
tamirms/events-slab-matcher
branch
from
September 25, 2026 20:35
847cf12 to
809e74f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rewrites how the events store matches a getEvents query. It now walks the query's window one slab at a time (65,536 consecutive event ids, roaring's container size) instead of combining whole-chunk bitmaps before the first fetch.
Tests:
TestMatches_FetchesOnlyTrueCandidatesrequires the fetched ids to be exactly the query's matches; the matrices inslab_match_test.gocompare against an oracle computed without the index;TestSlabStepperSkipsCandidateFreeSlabspins which slabs the walk opens;roaring_contract_test.gopins, against roaring alone and under-race, the properties the shared index bitmaps rely on;TestMatches_ConcurrentIngestBorrowSafetyholds index snapshots across a walk while ingest keeps writing.Why
A query is a list of filters, each an AND of terms (contract id, topics, event type), and the candidates are the ids that satisfy any filter inside the window. Today the engine intersects each filter's term bitmaps across the whole chunk, unions the results, and only then restricts to the window, all before the first event is fetched. Its cost tracks the terms' size across the chunk rather than the window or the page: a page of 1,000 events from a popular contract with a topic constraint first intersects that contract's millions of ids.
This PR turns each filter into plans, each a plain AND of terms (a topic-count range becomes one plan per count), and walks the window slab by slab in the requested direction. On each slab, every plan is one
FastAndover the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id (NextValue/PreviousValue) and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs where it has events. Work scales with the window and the page, memory per query with one container, and both directions share one path. The first batch is the caller's page size, so a full page arrives in one fetch.Known limitations
🤖 Generated with Claude Code