Skip to content

Hot ingestion: minimum optimization to fit the Phase 2 budget #974

Description

@marwen-abid

Goal

Phase 2 (1 s block time) gives ingestion 405 ms p99 per ledger. Hot ingestion on feature/full-history must stay under that for a whole 10,000-ledger chunk. Currently we're able to fit within that budget on small 500-1000 ledgers run, but fail when we run on the full chunk.

Problem

Full-length Phase 2 runs on the sac-5000 profile miss the budget. Runs capped at 1,000 hot ledgers pass, so the pass was an artifact of the cap.

hot leg ingest p99 apply p99 run
1,000 ledgers 163 ms 28 ms phase2-rps-query-pr961-c110e623-20260829T233734Z
10,000 ledgers 508 ms 344 ms phase2-pr961-full-c110e623-20260831T141126Z

Same commit, box type and dataset. The other five ingest phases do not change. The growth is in apply: ConcurrentBitmaps.AddTo clones each dense term's roaring bitmap on every write, and the clone cost grows with the number of containers as the chunk fills. Every published run since July shows the same dependence on hot-leg length.

Work

Out of scope

Memory growth of the in-memory index over a chunk (peak RSS 10.6 GiB on sac at full length). #902 addresses it with a new index and row format.

Acceptance

Full-length Phase 2 hot ingest p99 under 405 ms on all three profiles, on the merged branch, in a published run.

Metadata

Metadata

Assignees

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions