storage: put the tx-hash index being rebuilt and build scratch under the hot root

[tamirms]

Motivation

rpc v2 is meant to run with its data split across storage tiers: a small local NVMe disk for what is recent or rewritten, and a large durable volume (EBS today, possibly object storage later) for everything else. The [storage] section already lets each store sit on its own disk. What makes the split work is that the cold stores are write-once: a chunk's ledger pack, events pack, events index and terminal tx-hash index are each written once, at freeze, and only read after that. The slow volume sees its reads plus one sequential write per chunk, and its throughput goes to serving queries.

Two things did not fit that model.

The tx-hash index of the window still being filled. A window is 1000 chunks, and its index is rebuilt from scratch at every chunk boundary until the window is complete: the rebuild reads every .bin input of the window and writes a new .idx that grows with the window, then the previous .idx is swept. That is a sequential rewrite landing on the cold volume at every boundary, for the life of every window. On EBS the IOPS and bandwidth quotas are shared between reads and writes, so for as long as the rewrite runs, getTransaction and getEvents cold reads queue behind it. The gp3 baseline is 125 MiB/s and 3,000 IOPS, and a sequential rewrite of a growing index takes the bandwidth quota for as long as it runs.

The events index build's spilled runs. Since #1098 the build keeps one slab of postings in memory and spills the rest as sorted runs on disk, reading them back three times at finalize. Those runs were written beside the index files, so every build, including the parallel backfill builds, put its transient I/O on the cold volume.

What changes

Both now live with the hot tier, which is where everything else that is rewritten already lives.

• Layout.TxHashIndexDir takes the coverage, not the window id. A coverage that is not terminal is written under the hot root, in txhash-index-{window}/ beside the chunk DBs. A terminal coverage is a cold artifact under txhash_index/{window}/. The build, the sweep and the reader all resolve the path from the coverage, so the terminal build lands in the cold root and the same commit batch demotes the last hot-tier coverage, which the sweep then removes from under hot. No catalog key or value changes: whether a coverage is terminal was already derived from the key.
• Layout owns the tx-hash window arithmetic (WithTxHashIndex, TxHashIndex). catalog.Open reads it from the layout instead of taking a second parameter, so the two cannot disagree.
• The events index build spills under {hot}/scratch/{chunk}-index.runs/ instead of the index root. The cold roots hold only artifacts, and moving hot moves every build's transient writes with it; there is no second key to forget.

A two-tier deployment

[storage]
# The durable volume holds the catalog and every write-once artifact:
# ledger packs, events packs and indexes, terminal tx-hash indexes.
default_data_dir = "/mnt/ebs/rpc-v2"
# The local NVMe disk holds what is rewritten or transient.
hot = "/mnt/nvme/rpc-v2/hot"
txhash_raw = "/mnt/nvme/rpc-v2/txhash-raw"

[ingestion]
captive_core_storage_path = "/mnt/nvme/rpc-v2/captive-core"

The rule behind the split is write pattern, not age. hot takes every ledger's writes (the chunk DBs), the per-boundary rewrite of the window index, and the builds' scratch. txhash_raw holds the .bin files the per-boundary rebuild reads in full each time; they are written once per chunk but read at every boundary until the window terminates. Captive core churns its own database. Everything on the NVMe disk is derived data: the chunk DBs from the ledger backend, the window index and the .bin files from the ledger packs. The catalog is the one thing nothing can rebuild from the files, so it stays on the durable volume with the artifacts it describes. The remaining stores default under default_data_dir. events_index can also go on the NVMe disk when filtered event queries over wide ranges matter more than the space, since the indexes are small and probed at random.

🤖 Generated with Claude Code