TL;DR
The synthetic apply-load ingestion load test (#760) runs the O3 target transaction profiles (sac, oz, soroswap) and surfaced several problems. This issue is to investigate and fix all of them.
To investigate and fix
Ingestion latency tail
For the SAC payments profile, total ingestion duration shows a large gap between the median and the tail: p50 is 261 ms but p99 is 1062 ms, roughly four times higher. This is surprising because every block in the synthetic dataset carries the same load, so per-ledger ingestion times should be uniform. We need to find what causes the tail to spike and reduce the p99. One hypothesis to test: the load test ingests ledgers back-to-back with no pause between them, so the system never gets the roughly five-second gap it would have in production to run compaction and garbage collection. Re-running with a realistic per-ledger cadence may bring the tail down.
Memory usage and OOM
The current ingest loads all ledgers into memory at once instead of buffering and flushing incrementally, which caused out-of-memory failures on smaller machines (temporarily worked around by adding swap on the NVMe disk). Make ingestion memory-efficient so it no longer holds the whole dataset in memory.
RocksDB configuration
RocksDB settings were inconsistent across benchmark runs: the earlier proof-of-concept runs used compression on and auto-compaction off, while recent runs had compaction on and compression off, with no configuration chosen deliberately. Settle on the correct, consistent configuration (whether to disable auto-compaction is an open question) and re-run the benchmarks so the results are comparable.
Other bottlenecks
Investigate and fix any other ingestion bottlenecks the O3 load-test results reveal.
TL;DR
The synthetic apply-load ingestion load test (#760) runs the O3 target transaction profiles (sac, oz, soroswap) and surfaced several problems. This issue is to investigate and fix all of them.
To investigate and fix
Ingestion latency tail
For the SAC payments profile, total ingestion duration shows a large gap between the median and the tail: p50 is 261 ms but p99 is 1062 ms, roughly four times higher. This is surprising because every block in the synthetic dataset carries the same load, so per-ledger ingestion times should be uniform. We need to find what causes the tail to spike and reduce the p99. One hypothesis to test: the load test ingests ledgers back-to-back with no pause between them, so the system never gets the roughly five-second gap it would have in production to run compaction and garbage collection. Re-running with a realistic per-ledger cadence may bring the tail down.
Memory usage and OOM
The current ingest loads all ledgers into memory at once instead of buffering and flushing incrementally, which caused out-of-memory failures on smaller machines (temporarily worked around by adding swap on the NVMe disk). Make ingestion memory-efficient so it no longer holds the whole dataset in memory.
RocksDB configuration
RocksDB settings were inconsistent across benchmark runs: the earlier proof-of-concept runs used compression on and auto-compaction off, while recent runs had compaction on and compression off, with no configuration chosen deliberately. Settle on the correct, consistent configuration (whether to disable auto-compaction is an open question) and re-run the benchmarks so the results are comparable.
Other bottlenecks
Investigate and fix any other ingestion bottlenecks the O3 load-test results reveal.