Goal
Phase 2 (1 s block time) gives ingestion 405 ms p99 per ledger. Hot ingestion on feature/full-history must stay under that for a whole 10,000-ledger chunk. Currently we're able to fit within that budget on small 500-1000 ledgers run, but fail when we run on the full chunk.
Problem
Full-length Phase 2 runs on the sac-5000 profile miss the budget. Runs capped at 1,000 hot ledgers pass, so the pass was an artifact of the cap.
Same commit, box type and dataset. The other five ingest phases do not change. The growth is in apply: ConcurrentBitmaps.AddTo clones each dense term's roaring bitmap on every write, and the clone cost grows with the number of containers as the chunk fills. Every published run since July shows the same dependence on hot-leg length.
Work
Out of scope
Memory growth of the in-memory index over a chunk (peak RSS 10.6 GiB on sac at full length). #902 addresses it with a new index and row format.
Acceptance
Full-length Phase 2 hot ingest p99 under 405 ms on all three profiles, on the merged branch, in a published run.
Goal
Phase 2 (1 s block time) gives ingestion 405 ms p99 per ledger. Hot ingestion on
feature/full-historymust stay under that for a whole 10,000-ledger chunk. Currently we're able to fit within that budget on small 500-1000 ledgers run, but fail when we run on the full chunk.Problem
Full-length Phase 2 runs on the
sac-5000profile miss the budget. Runs capped at 1,000 hot ledgers pass, so the pass was an artifact of the cap.Same commit, box type and dataset. The other five ingest phases do not change. The growth is in
apply:ConcurrentBitmaps.AddToclones each dense term's roaring bitmap on every write, and the clone cost grows with the number of containers as the chunk fills. Every published run since July shows the same dependence on hot-leg length.Work
hot_num_ledgers=0as the default for phase verdict runs. Capped runs stay for query-only work and smoke tests.Out of scope
Memory growth of the in-memory index over a chunk (peak RSS 10.6 GiB on sac at full length). #902 addresses it with a new index and row format.
Acceptance
Full-length Phase 2 hot ingest p99 under 405 ms on all three profiles, on the merged branch, in a published run.