rpcv2: getEvents read-path performance across both tiers

[tamirms]

getEvents read-path performance on both tiers: the hot RocksDB store, the cold packfile store, and the match engine they share. It also fixes a latent race in the cold fetch arena.

Match engine

What the engine does. A getEvents query is a list of filters. Each filter names constraints: a contract ID, a topic at a position, an event type, a topic count. The index maps every term to a bitmap of the event IDs in the chunk that carry it. The candidates for a query are the IDs that satisfy at least one filter, and a filter is satisfied when all of its constraints hold: an OR across filters of an AND within each filter, restricted to the window: the range of event IDs inside the chunk that the client's ledger range covers. Candidates are then fetched and post-filtered, and the client receives at most a page of them, its limit.

How the base branch does it. All at once, over whole-chunk bitmaps: intersect each filter's term bitmaps across the entire chunk, union the results across filters, and intersect with the window last. Everything is computed before the first event is fetched. For any query with more than one term or filter, the cost tracks the size of the terms across the whole chunk rather than the window the client asked for or the page it will receive. A page of 1,000 events from a popular contract with a topic constraint pays to intersect that contract's millions of IDs first.

How this PR does it. Roaring bitmaps store IDs in containers of 65,536 consecutive IDs, and every bitmap operation works one container at a time. The engine follows that grain. A filter is first turned into plans, each a plain AND of single terms: a topic-count range, the one constraint that is an OR of terms, becomes one plan per bucket, since A and (b or c) is (A and b) or (A and c), and a plan that repeats an earlier one is dropped. The walk then visits the window one container-sized slab at a time, in the requested direction, and evaluates every plan on that slab alone, one FastAnd over the slab's IDs inside the window and the plan's terms, ORs the results across plans, and hands the union to the fetch. Every intermediate value is a single container, so the work per slab is bounded, and a page that fills within a couple of slabs never touches the rest of the chunk. Ascending and descending share one code path; only the walk order differs.

Skipping slabs. A rare term over a wide window would still visit every slab, so before evaluating one the walk asks each term for its next ID at or after the cursor, NextValue on a roaring bitmap, which is cheap. Within a plan every term must hold, so nothing can match before the term whose next ID is furthest away. Across plans any may match, so the earliest wins. The walk jumps straight to the slab holding that ID. Each plan's bound is kept until the cursor passes it, and a plan with no IDs left is retired. Descending uses PreviousValue and the mirror-image rules. A query for a rare term opens only the slabs where that term has events.

Term lookups. Both tiers serve the walk through Reader.LookupKeys, which returns each term's bitmap once at query start. The walk holds those bitmaps for its whole duration and never mutates them, so a query sees one consistent image of the index while ingest continues, the same guarantee the pinned window already gives. On the hot tier they are snapshots shared between readers; on the cold tier they are fresh copies.

Why it is better. Work scales with the window and the page instead of the chunk: the walk opens only the slabs the window holds and stops once the page is full, so a limit-1 request that finds a match in its first slab pays for one slab whatever the size of the chunk, the ten-thousand-ledger unit of storage. Memory per query is bounded by a container instead of by chunk-sized intermediates. One engine serves both directions, replacing two paths.

roaring moves from v2.18.2 to v2.26.0. At that version FastAnd intersects pairwise, so a plan that matches nothing on a slab still allocates one intermediate there. RoaringBitmap/roaring#566 and the FastAnd change built on it count a slab's result before allocating it, which makes an empty plan cost no container at all; that bump is a follow-up, and its allocation pin for roaring_contract_test.go is written.

Hot tier

• DataCF blocks are 8 KiB (dataCFBlockSize), the unit getEvents decompresses per fetched event. Existing databases read fine, since SSTs self-describe their block size, and converge through compaction. Ingest throughput is unchanged; disk grows 0.8 %.
• The fetch path builds the batched read's options once at open, carves the whole key list out of one buffer, reads each pinned value once, and copies the values into a per-batch arena. It spends two heap allocations per event ID, both grocksdb's own, pinned by TestHotStore_FetchEventsAllocationBudget.
• The first match batch is the caller's validated page size, honored in full (1,000 on v2, up to 10,000 on v1), so a full page arrives in one fetch on either tier.

Cold tier

• Cold event reads fan out over the packfiles with a concurrency of 8 (defaultColdEventReadConcurrency). The fetch arena is now locked against that fan-out. Before this PR, any cold read concurrency above 1 produced SIGSEGVs or silently short results; the wired value of 1 kept the race latent.
• The MPHF is mmapped, so the page cache is the cross-request index cache. This removes an 18.5 MB read and heap copy per request at high term cardinality.
• The packfile open path pools its offsets table, decode scratch and open buffers, and returns them on every exit, failed opens included. A buffer dropped by a pool's capacity cap is counted and exported as pooled_buffer_cap_skips_total; a nonzero count means a cap wants raising.

Tests

• TestMatches_FetchesOnlyTrueCandidates traces every fetch and requires the fetched ordinals to be exactly the query's matches, in emission order. Output equality alone cannot see an over-wide candidate set, because the post-filter hides it as wasted I/O.
• The shaped, whole-stream and randomized matrices in slab_match_test.go compare against an oracle computed without the index, at several slab widths.
• TestSlabStepperSkipsCandidateFreeSlabs pins the exact slab list the walk opens per shape; TestMatches_RareTermsSpanSlabs runs the skip-dominated shapes at widths that put hundreds of empty slabs between candidates.
• roaring_contract_test.go pins, against roaring alone, the properties the shared index bitmaps rest on: FastAnd leaves copy-on-write arguments byte-identical, returns containers that share no storage with them, and equals the pairwise And chain, for every subset of the container kinds over whole and partial slab ranges; a range built with AddRange is stored as run containers; and NextValue / PreviousValue are inclusive and read-only, with eight concurrent readers under -race. It is the first thing to run on a roaring bump.
• TestMatches_ConcurrentIngestBorrowSafety holds index snapshots across a walk while ingest publishes new term states on the same keys.
• Cold: pool reuse, the close handshake and the cap-skip counter in packfile/pools_test.go; the fan-out test compares whole payloads.

Measured impact

Service layer, 50 rps, K-corpus at limit=1000; sac = the phase-3 stress chunk, pubnet = chunk 6410 (9.1 M events). Baseline is the branch base (32 KiB blocks, cold concurrency 1); after is this PR.

Hot

	p50	p99
sac	7.41 → 3.68 ms	21.1 → 7.2 ms
pubnet	5.25 → 2.23 ms	12.0 → 4.18 ms

Cold

	p50	p99	alloc/request
sac	18.3 → 4.3 ms	98.7 → 15.2 ms	31.4 → 3.2 MB
pubnet	5.2 → 3.0 ms	12.9 → 5.9 ms	—

Item counts are identical on both sides of every A/B cell (2,800,552 / 1,694,968).

Filter lists. Engine alone on one 8.4 M-event chunk (128 slabs), Xeon 8375C, roaring v2.26.0, limit 1 unless noted; before is this branch before the per-plan engine, after is this head. Wall time and allocation per request.

256 filters	before	after
identical	1.00 ms, 4.1 MB	0.05 ms, 0.0 MB
16 distinct, sparse, limit 1000	12.6 ms, 5.9 MB	1.0 ms, 0.6 MB
distinct dense pairs that never co-occur	173 ms, 529 MB	111 ms, 337 MB
dense triples, pairwise overlapping, jointly empty	340 ms, 671 MB	85 ms, 168 MB
v1 topic-count ranges that never match	352 ms, 888 MB	51 ms, 114 MB

Matching rows are unchanged. With the count-first roaring the three never-matching rows fall to 5 to 16 ms and 1 to 4 MB.

🤖 Generated with Claude Code

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	golang/​github.com/​RoaringBitmap/​roaring/​v2@​v2.18.2 ⏵ v2.26.0	+1

View full report

[Copilot]

Pull request overview

Optimizes getEvents across hot and cold storage while fixing concurrent cold-fetch safety.

Changes:

• Adds lazy ascending event matching and broader fetch batching.
• Reduces allocations through arenas, pooled packfile buffers, and mmap-backed indexes.
• Enables concurrent cold reads and upgrades RoaringBitmap.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
go.mod	Upgrades bitmap dependencies.
go.sum	Updates dependency checksums.
event/arena.go	Adds payload byte arena.
event/arena_test.go	Tests arena stability.
event/cold_format.go	Opens MPHF indexes via mmap.
event/cold_reader.go	Makes concurrent arena use safe.
event/cold_reader_fanout_test.go	Tests concurrent cold fetching.
event/concurrent_bitmaps.go	Exposes immutable postings views.
event/hot_store.go	Uses postings fast path and smaller blocks.
event/match.go	Splits ascending and descending matching.
event/match_iter.go	Implements lazy iterator-tree matching.
event/match_iter_test.go	Tests and benchmarks iterator matching.
event/matches_differential_test.go	Adds randomized differential coverage.
packfile/index.go	Pools index decoding allocations.
packfile/pools.go	Adds packfile allocation pools.
packfile/pools_test.go	Tests pooling and close/read interaction.
packfile/reader.go	Pools open state and guards offset recycling.
query/resolve.go	Enables eight-way cold-read fan-out.
rocksdb/rocksdb.go	Consolidates batch value copies.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/stellar-rpc/internal/rpcv2/query/resolve.go:183

• The production override described here is not reachable: coldEventReadConcurrency is an unexported field, neither OpenRegistry nor NewRegistry accepts it, and startup constructs the registry through query.OpenRegistry. Consequently every deployment is forced to fan out eight workers per cold page, despite the comment above noting that this multiplies per-request goroutines and 1 MiB buffers. Please thread an operator setting through startup/registry construction (with validation), or remove the claimed override and choose a globally safe bound.

		conc := a.coldEventReadConcurrency
		if conc == 0 {
			conc = defaultColdEventReadConcurrency

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

cmd/stellar-rpc/internal/rpcv2/query/registry.go:45

• This override is not reachable by a deployment: Registry is constructed through OpenRegistry, the field is unexported, and there is no constructor option or setter for it. Consequently production is still fixed at 8 despite the comments saying operators can raise it. Thread the setting through the public registry construction/configuration path, or describe this as a test-only seam instead.

	// coldEventReadConcurrency overrides the packfile read fan-out one cold
	// events read gets. Zero means defaultColdEventReadConcurrency, resolved
	// where the reader is opened (ReadView.Events). Held here, like
	// maxScanLedgers, so a deployment with I/O headroom to spend — or a test
	// pinning the serialized path — sets it on its Registry rather than
	// rebuilding.
	coldEventReadConcurrency int

cmd/stellar-rpc/internal/rpcv2/stores/event/arena.go:24

• The arena’s first non-empty copy always allocates at least 64 KiB. getEvents accepts a limit of 1, and the pager passes room+1 to Matches, so a small cold page can fetch only two ~250-byte payloads yet now allocates 64 KiB instead of roughly the bytes returned. This regresses allocation volume for low-limit requests; size the initial chunk from the requested batch (or grow from a smaller first chunk) and reserve 64 KiB chunks for batches that actually reach that size.

func (a *byteArena) copy(b []byte) []byte {
	if len(b) > cap(a.buf)-len(a.buf) {
		a.buf = make([]byte, 0, max(arenaChunkSize, len(b)))
	}

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cmd/stellar-rpc/internal/rpcv2/stores/event/match.go:230

• Every positive hint becomes the initial slice capacity in both filtered and descending match-all streams. v1 only bounds the request by the operator-configured maximum, which may be raised as high as MaxInt32; a selective request under such a setting can therefore allocate billions of entries before discovering that only a few events match. Preserve the 10,000-item fast path, but impose a hard internal cap or chunk oversized first fetches—the hint changes I/O shape, not result semantics.

	if hint > 0 {
		first = hint
	}

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.

Suppressed comments (2)

cmd/stellar-rpc/internal/rpcv2/packfile/pools_test.go:85

• This does not actually verify that Close withheld the offsets from the pool: the sole item has already been located and decoded before this callback runs, and no subsequent open overwrites a pooled array, so an implementation that always recycled r.offsets would still pass the byte check. Assert here, after Close has returned while the callback still keeps inflight nonzero, that r.offsets remains non-nil.

			<-unpark

cmd/stellar-rpc/internal/rpcv2/query/resolve.go:155

• This hard-codes eight 1 MiB packfile worker buffers per in-flight cold read for every deployment, even though the comment notes that the optimum depends on storage and scales with concurrent pages. Deployments on constrained disks or memory limits cannot tune the new fan-out without rebuilding. Please carry a registry/configured concurrency value into each ReadView (with zero selecting this default), as is done for other per-view serving bounds.

const defaultColdEventReadConcurrency = 8

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cmd/stellar-rpc/internal/rpcv2/stores/event/match.go:234

• This makes the initial allocation and fetch unbounded. v1 has no upper validation for max_items_per_response and passes configured limits through up to math.MaxInt32, so an operator-raised cap can make streamSlabs allocate a multi-gigabyte ID slice (and descending match-all allocate an equally unbounded Match block) before serving anything. Bound the first batch independently of the page limit; subsequent batches can still complete larger pages.

	if hint > 0 {
		first = hint

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cmd/stellar-rpc/internal/rpcv2/stores/event/match.go:234

• A positive hint is now used as an allocation capacity with no internal ceiling (streamSlabs allocates ids at that capacity, and descending streamRange does the same for block). This is not bounded by validation for v1: max_items_per_response may be raised arbitrarily, so an accepted request can force a multi-gigabyte allocation/OOM before the window is even clipped. Keep the first fetch optimization, but cap the internal batch to a bounded multiple of matchBatchSize and let subsequent batches serve the rest of the page.

	if hint > 0 {
		first = hint

[tamirms]

Split into smaller PRs. Everything in this one now lives in one of them:

• events: 8 KiB blocks for the hot store's event data #1045: 8 KiB DataCF blocks for the hot store
• packfile: reuse the open path's buffers #1046: packfile reuses the open path's buffers
• events: batch fetched payload copies into arenas, and fan cold reads out #1047: fetch arenas, the arena lock, and cold read fan-out at concurrency 8
• events, deps: memory-map the cold MPHF, and roaring v2.26.0 #1040: memory-map the cold MPHF, and roaring v2.26.0
• events: match queries one slab at a time #1041: the slab-by-slab match engine
• events: ask the index for a window, and match in two stages #1042: carries the hot store's LookupKeys snapshot doc, alongside the windowed lookup

#1040–#1043 are stacked on #1034. #1045–#1047 are independent.