RPC v2 EBS Benchmarking #1100

#1099 describes running rpc v2 with its storage split across two disks. The EBS volume holds the catalog and every file that is written once and never changed: ledger packs, event packs, event indexes, and completed tx-hash indexes. The instance's local NVMe disk holds everything that is rewritten or temporary: the RocksDB databases for recent ledgers, the tx-hash index of the current window, build scratch files, the raw tx-hash .bin files, and captive core. We have not tested this setup yet.

The main concern is ingestion latency. Requests for old data consume the EBS volume's IOPS and throughput quota, and a freeze and the tx-hash index build write to the same volume at the same time. We need to know whether ingestion latency suffers when all of this happens together. This issue is to run the setup under load and answer three questions.

1. Does the EBS volume change ingestion latency while we serve requests and freezes run?

Run the same code twice at each load level: once with everything on the local disk as the control, and once on this setup. In both runs, queries for both recent and old data run at the same time as ingestion, which crosses one chunk boundary. Do this for two boundaries: one with enough raw tx-hash .bin files already in the window that the tx-hash index build at that boundary is near its worst case, and one where the window completes, so the finished index is written to EBS and the .bin files are removed. Step the load up through several levels, ending with the SAC 6000 stress profile used for Phase 3. For each level and phase (steady ingestion, during a freeze, during the tx-hash index build) report ingestion latency, freeze duration, and request latency for recent and old data, at p50 and p99, along with the EBS volume's IOPS and throughput. The goal is that the EBS volume does not change ingestion latency, and that the Phase 3 level stays within the 105 ms ingestion target.

2. Should we provision the EBS volume above the gp3 baseline of 3,000 IOPS and 125 MiB/s, which reads and writes share? If so, what do we recommend?

Run 1 at the baseline first. If ingestion or request latency fails or is marginal, repeat it at higher IOPS and throughput levels and give the metrics and a verdict for each level. Also measure a backfill: how long it takes and how much of the volume's IOPS and throughput it uses, at the baseline and at higher levels if the baseline is the bottleneck. Each endpoint has an SLA, and higher request rates consume more of the volume's quota, so also measure request latency at increasing request rates and report the provisioning each rate needs. The recommendation should be one provisioning level that covers both serving and a large backfill and meets every endpoint's SLA comfortably at the request rates we plan to support.

3. Do we need more code changes to make the EBS deployment work well?

List what the measurements show is needed, if anything.