Synthetic ledger datasets for Phases 1–3 ("golden packs") #867

TL;DR

Generation is complete. Six synthetic ledger datasets — one per (tx model, TPL) pair in the Phase 1–3 table — were generated with stellar-core apply-load and uploaded, byte-verified, to GCS. Benchmarks replay them (--source=pack); generation is one-time, campaigns never regenerate. Open work on this issue: freeze the raw streams into ledger packs, and regenerate sac-6000 at full length.

Design: datasets are keyed by (tx model, TPL)

A dataset bakes in only the transaction model and the transactions-per-ledger count. Block time is applied at replay (--close-interval), so phases that share a TPL share a dataset. Six datasets cover all nine phase cells:

The set

Location: gs://rpc-full-history/synthetic-ledgers/2026-07-18-apply-load-20k/<model>-<tpl>/. Each dataset directory holds METADATA.md (exact config, build, validation stats), apply-load.cfg, apply-load.log, and meta.xdr.zst. The set is immutable once referenced by a campaign; frozen packs land under <dataset>/packs/cold/.

Totals: ≈111 k ledgers, ≈384 M Soroban transactions, 1.14 TiB raw meta → 83.3 GiB compressed.

Build: stellar-core 27.1.1-3420.8612ca32f (~perftests, master of 2026-07-17), run on the benchmarking devbox (#866). The streams carry the pubnet network passphrase, so the stored transaction result hashes are correct as-is. This set supersedes the 2026-06-04 set (generated on a pre-fix build) for all benchmark use.

Validation per dataset: final ledger sequence matches the target count, stream growth monitored through generation, and every upload byte-verified against the local file before local deletion.

Remaining work

• Freeze each dataset into ledger packs at <dataset>/packs/cold/ — the form the campaigns consume (--source=pack).
• Regenerate sac-6000 at full length (20,000 ledgers). SAC generation mints a fresh destination address per transaction and all live Soroban state is held in process memory, so peak RSS reaches ≈52 GiB at full length (≈0.41 GiB per million transactions) — the run needs a ≥96 GB RAM box. Until then, campaigns use chunk 1 (10,000 ledgers) of the partial.
• Publish the generation runbook (build → generate → validate → upload → freeze), so anyone on the team can regenerate a dataset.

Consumers

• Run the Phase 1 benchmark campaign — ingest and reads at 2 s blocks #872 / Run the Phase 2 benchmark campaign — ingest and reads at 1 s blocks #873 / Run the Phase 3 benchmark campaign — ingest and reads at 600 ms blocks #874 — the Phase 1/2/3 benchmark campaigns (replay this set at 2 s / 1 s / 600 ms)
• Full-history benchmark commands — ingest across all data types #771 / Full-history query benchmark commands — reads across all data types  #856 — the ingest and query harnesses that read the packs

Done when

All six dataset directories carry packs/cold/, sac-6000 is full-length, and the campaigns consume packs from this set.