Skip to content

events: match queries one slab at a time - #1041

Draft
tamirms wants to merge 1 commit into
tamirms/roaring-2.26-mphf-mmapfrom
tamirms/events-slab-matcher
Draft

tamirms wants to merge 1 commit into
tamirms/roaring-2.26-mphf-mmapfrom
tamirms/events-slab-matcher

Conversation

@tamirms

@tamirms tamirms commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

What

Rewrites how the events store matches a getEvents query. It now walks the query's window one slab at a time (65,536 consecutive event ids, roaring's container size) instead of combining whole-chunk bitmaps before the first fetch.

Tests: TestMatches_FetchesOnlyTrueCandidates requires the fetched ids to be exactly the query's matches; the matrices in slab_match_test.go compare against an oracle computed without the index; TestSlabStepperSkipsCandidateFreeSlabs pins which slabs the walk opens; roaring_contract_test.go pins, against roaring alone and under -race, the properties the shared index bitmaps rely on; TestMatches_ConcurrentIngestBorrowSafety holds index snapshots across a walk while ingest keeps writing.

Why

A query is a list of filters, each an AND of terms (contract id, topics, event type), and the candidates are the ids that satisfy any filter inside the window. Today the engine intersects each filter's term bitmaps across the whole chunk, unions the results, and only then restricts to the window, all before the first event is fetched. Its cost tracks the terms' size across the chunk rather than the window or the page: a page of 1,000 events from a popular contract with a topic constraint first intersects that contract's millions of ids.

This PR turns each filter into plans, each a plain AND of terms (a topic-count range becomes one plan per count), and walks the window slab by slab in the requested direction. On each slab, every plan is one FastAnd over the slab's ids in the window and the plan's terms, and the plans' results are ORed and handed to the fetch. Before opening a slab, the walk asks each term for its next id (NextValue / PreviousValue) and jumps to the first slab that can hold a candidate, so a rare term opens only the slabs where it has events. Work scales with the window and the page, memory per query with one container, and both directions share one path. The first batch is the caller's page size, so a full page arrives in one fetch.

Known limitations

🤖 Generated with Claude Code

A getEvents query was answered by intersecting each filter's term bitmaps
across the whole chunk, unioning the results across filters, and only then
restricting to the window. Its cost tracked the terms' size across the
chunk, not the window the client asked for or the page it would receive.

The engine now walks the window one slab at a time: 65,536 consecutive
event ids, roaring's container size, in the requested direction. Each
filter becomes plans that are plain ANDs of terms. On each slab, every plan
is one FastAnd over the slab's ids in the window and the plan's terms, and
the plans' results are ORed and handed to the fetch. Before opening a slab,
the walk asks each term for its next id and jumps to the first slab that
can hold a candidate, so a rare term opens only the slabs it appears in.
Work scales with the window and the page, memory per query with one
container, and both directions share one path. The first batch is the
caller's page size, so a full page arrives in one fetch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
@tamirms
tamirms force-pushed the tamirms/events-slab-matcher branch from 847cf12 to 809e74f Compare September 25, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant