Config-driven benchmark campaign runner for the full-history bench subcommands #896

TL;DR

Add tooling that runs a full benchmark campaign from one config file with one command. The runner builds a pinned binary, prepares every dataset, runs the configured ingest and query loops across datasets, chunks, and repetitions, records provenance, bundles the results, and optionally publishes them. A companion script bootstraps the bench box; another safeguards the bundle to object storage.

Why

The bench subcommands run one leg at a time — one ingest or query pass against one source. A campaign runs many legs across several datasets and chunks, repeated for statistics, and the results only compare if every leg used the identical binary, dataset preparation, and flags. Doing that by hand is slow and error-prone. This tooling makes a campaign reproducible: a config file names the binary, the datasets, and the loops, and the runner does the rest the same way every time.

The runner — campaign.sh

One command runs the whole campaign:

BENCH=/mnt/nvme/bench ./scripts/bench-devbox/campaign.sh my-campaign.cfg [--dry-run]

Steps, in order:

1. Validate. The config is a sourced bash fragment. The runner rejects any unknown key and validates every value (enums, integers, Go durations, chunk lists, dataset entries) before it builds or runs anything.
2. Build the binary under test. REF is built into a versioned binary stellar-rpc-<sha>, then the original checkout is restored. REF selects only the binary — nothing else. A dirty work tree is refused when REF is set, so the built commit is exact. The build is skipped if that binary already exists.
3. Prepare each dataset to a local cold pack root. Four source kinds converge on the same on-disk shape (ledgers/, events/, txhash/):
   • packs-local — a local cold pack root, used in place.
   • packs-gs — a gs:// prefix, fetched once into $BENCH/golden/<name>/ and reused.
   • bsb-s3 — an S3 bucket path; an untimed cold backfill materializes the golden root.
   • fixture — a generated fixture pack plus an untimed cold ingest materialize the golden root.
     Golden datasets are fetched or built once and reused across runs. Removing $BENCH/golden/<name> forces a re-fetch.
4. Run the configured loops. Per (dataset, chunk), RUNS repetitions, each a fresh process with its own --out directory:
   • INGEST=cold|both → bench-ingest cold (scratch wiped before each run).
   • INGEST=hot|both → bench-ingest hot, optionally paced at CLOSE_INTERVAL. The last run's hot DB is kept for the hot query suite.
   • QUERY=yes → bench-query cold against each frozen pack root, and bench-query hot against the kept hot DB (hot query needs a hot ingest, so it runs only when INGEST is hot or both).
5. Record provenance and bundle. Results land in $BENCH/results/<run_id>/, where run_id = <NAME>-<sha>-<stamp>, alongside the config, binary.txt, machine-metadata.txt, and metadata.json. The directory is tarred to durable storage so the bundle survives an instance stop.
6. Publish (optional). When PUBLISH_URI is set, publish.sh uploads the bundle. A publish failure is not a benchmark failure — the data is already safe locally, so it exits with the exact retry command instead of corrupting the "campaign done" signal.

--dry-run prints every command with resolved paths and flags, and builds, downloads, and runs nothing.

Config format

The config sets only documented keys:

Box bootstrap — bootstrap.sh

Idempotent setup for an EC2 box with a local NVMe instance store (m6id class, Ubuntu 24.04). It formats and mounts the instance store, runs an fsync honesty probe and warns if the disk absorbs fsync (hot-commit numbers would be fiction on such a disk — the reason this box class exists), installs the toolchain and native libs, builds the daemon, and race-tests the bench package. It is safe to re-run after a stop/start, which wipes the instance store; the runner re-fetches golden packs.

Publishing — publish.sh

Safeguards a finished bundle to <dest-root>/<run_id>/. It dispatches by scheme (gs:// via gcloud storage rsync, s3:// via aws s3 sync). Published runs are immutable: it refuses a destination that already holds objects unless --force is given. --dry-run prints the cloud commands without running them.

What this delivers

• One config file plus one command runs a full ingest and query campaign end to end.
• Strict config validation and a --dry-run that prints the exact command plan.
• Ref-pinned, versioned binary so results are reproducible and attributable to a commit.
• Four dataset source kinds behind one interface, with fetch-once golden reuse.
• Fresh process and output directory per run; scratch and hot DBs reset between runs.
• Self-describing bundle: config, binary identity, machine metadata (with fsync probe), and a machine-readable metadata.json (run id, config, datasets, hardware).
• Idempotent box bootstrap and an immutable, scheme-agnostic publish step.

Related

• Full-history benchmark commands — ingest across all data types #771 / Full-history query benchmark commands — reads across all data types  #856 — the bench-ingest and bench-query subcommands this runner orchestrates.
• e2e benchmark campaigns workflow — dispatch, run on EC2, publish results #868 — scheduled campaigns; the CI/dispatch layer that drives this runner on ephemeral EC2.
• Run the Phase 1 benchmark campaign — ingest and reads at 2 s blocks #872 / Run the Phase 2 benchmark campaign — ingest and reads at 1 s blocks #873 / Run the Phase 3 benchmark campaign — ingest and reads at 600 ms blocks #874 — the phase campaign runs, executed through this runner.
• Benchmarking infra: stellar-rpc-bench S3 bucket + benchmarking devbox #866 — results bucket the bundles publish to.
• Benchmark results explorer — browsable reports for campaign runs #871 — results explorer; consumes the published bundles and their metadata.json.