stellar-rpc-benchmarks

Benchmark reports — as data — for stellar-rpc's feature/full-history branch (RocksDB hot tier + immutable packfile cold tier). Each benchmark run is committed as a plain JSON file and rendered by a static, dependency-free viewer. No database, no build step, no server: the numbers live in git and the site is just HTML/JS reading them.

Live site: https://stellar-experimental.github.io/stellar-rpc-benchmarks

What this is

The bench suite (stellar-rpc bench ingest cold|hot and bench query --data-dir; bench query cold|hot before stellar-rpc#1070, bench-ingest / bench-query before #1105) runs in campaigns on an AWS NVMe devbox (m6id.2xlarge), driven by the config-driven runner in runner/ (see "Run a campaign" below). Each campaign is several configurations × 5 fresh-process runs; every run writes CSVs (stage,n,n_items,total_ns,p50_ns,p90_ns,p99_ns,max_ns) into its own directory, and the results are mirrored to GCS under gs://rpc-full-history/results/.

converter/convert.py (Python 3, standard library only) turns one results directory into docs/runs/<run-id>.json (schema v1) and updates the manifest docs/runs/index.json. The static viewer in docs/ renders any committed run via a dropdown or ?run=<id>. Appending &view=hot (or the toolbar toggle) shows a focused, stakeholder-facing hot-ingestion-only view.

Campaign runs also carry the phase performance targets (phases 1, 2, 3, 3b) as data (see SCHEMA.md). The converter matches the run's phase from its recorded close interval (2 s → Phase 1, 1 s → Phase 2, 600 ms → Phase 3, 900 ms → Phase 3b) and the viewer judges hot ingestion against that phase by default — budget lines, keep-up verdicts, and the ingest-slice p99 target. Append &phase=N (or use the selector next to the target table) to view the same measurements against another phase's targets; a caveat states the run's actual pace when it differs. The pacing-lag figure always uses the run's actual close interval.

How deploys work: there is no build step — the viewer is static vanilla JS and the run JSONs are committed alongside it. GitHub Pages serves the gh-pages branch, and the deploy-pages.yml workflow syncs main:/docs to it on every push to main, so deploying is still just committing to main. The indirection exists for one reason: PR previews. pr-preview.yml deploys each PR's docs/ to https://stellar-experimental.github.io/stellar-rpc-benchmarks/pr-preview/pr-<number>/ and comments the link on the PR (removed automatically when the PR closes).

View locally

make serve   # python3 -m http.server 8000 -d docs

Then open http://localhost:8000. The viewer fetches runs/index.json and the run files over HTTP, so opening docs/index.html as a file:// URL will not work — use make serve.

To smoke-test the viewer headlessly (loads each run in a jsdom DOM, asserts zero JS errors and the expected figure/section counts and sanity values), run make smoke (needs Node; installs jsdom under tests/smoke/ on first run).

Run a campaign

Campaigns run on the benchmark devbox via the campaign CLI in runner/ — this repo's operations side. The runner treats stellar-rpc as a black box: it maintains a build clone of it under $BENCH_ROOT/src, builds the configured ref, and drives the bench subcommands — no standalone stellar-rpc checkout is needed anywhere. See runner/README.md for the full CLI and config reference, the bundle layout it produces, and the minimum stellar-rpc ref it requires (the compatibility floor).

# 0. One-time on a fresh devbox (and again after every instance stop/start,
#    which wipes the NVMe instance store): provision the machine.
./runner/bootstrap.sh

# Everything below runs from runner/ (the devbox has Go; bootstrap installs it).
cd runner

# 1. Write a campaign config (copy example-campaign.toml, adjust the keys) and
#    sanity-check the full command plan. --dry-run builds, downloads, and runs
#    nothing — it works on any machine, e.g. a laptop:
go run ./cmd/campaign run my-campaign.toml --dry-run

# 2. Run it (in tmux — campaigns run for hours). Results land in
#    $BENCH_ROOT/results/<name>-<sha>-<stamp>/, tarred to /tmp so the bundle
#    survives an instance stop.
go run ./cmd/campaign run my-campaign.toml

# 3. Publish the bundle to GCS. This happens automatically when the config
#    sets publish_uri; run it by hand otherwise (or to retry a failed upload —
#    an upload that died partway leaves objects at the destination, so the
#    retry needs --force, which is a MERGE: files already there that this
#    bundle lacks survive it. Against a destination that really is empty,
#    --force only skips the emptiness check, so it is safe on a first publish):
go run ./cmd/campaign publish /mnt/nvme/bench/results/<run-id> \
  gs://rpc-full-history/results --force

The published bundle is exactly what the next section ingests into a committed run JSON — closing the loop: campaign config → run → publish → ingest → viewer.

Add a run

scripts/ingest.sh is the one path from a published bundle to a committed run: it fetches the bundle, converts it, and commits the site update. The same script backs make ingest and the automated flow below, so a run ingested from a laptop and one ingested from CI are byte-identical, commit message included.

# On a laptop with AWS credentials for the results bucket. The s3:// path is
# recorded as campaign.source_uri in the run JSON (the viewer's "Source data" link).
make ingest \
  BUNDLE=s3://stellar-rpc-bench/results/<run-id> \
  KIND=synthetic

BUNDLE is auto-detected and may be a gs:// or s3:// bundle URI, a local bundle directory, or a bench-results-<run_id>.tgz tarball — the shapes the campaign CLI's run and publish leave behind. KIND is the one thing you have to state, because it is the one fact the bundle doesn't record: datasets[].kind in the manifest is the dataset's transport (packs-gs, bsb-s3, …), not pubnet-vs-synthetic.

Everything else is derived from the bundle's own metadata.json (see SCHEMA.md § Inputs):

Derived	From
Run id, and so the file	run_id → docs/runs/<run_id>.json
Run date	started_at
Run name	campaign.name
campaign.source_uri	the s3:// (or gs://) URI you passed (recorded as provenance)
Commit body	campaign config, close interval, datasets, hardware

Three modes, least to most committal:

Mode	Effect
--dry-run	Converts into a temp directory — never docs/runs/ — and prints the converter output, the derived run id, the would-be commit body, the would-be branch, and the exact commands --push-main would run. Executes none of them. A remote bundle isn't fetched either; the fetch command is printed instead, so a dry run works offline.
--local	(default) Converts into docs/runs/, creates run/<run_id> off HEAD, commits the two changed files. No push.
--push-main	Converts into docs/runs/, runs make test + make smoke as a push gate, commits on HEAD (which must be at origin/main), and pushes it to origin main. Prints viewer: <url>. Pushing to main is the deploy (deploy-pages.yml). Concurrent ingests are safe: a push rejected because main moved is recovered by resetting to the new origin/main and redoing convert + gate + commit + push, up to INGEST_PUSH_ATTEMPTS (default 5) times.

make ingest passes --local on purpose: it stops at the commit, so you can read the diff and make serve the result before anything leaves the machine. Call the script directly for the other two modes:

# Look before you leap — converts to a temp dir, touches nothing:
scripts/ingest.sh s3://stellar-rpc-bench/results/<run-id> --dataset-kind synthetic --dry-run

# Publish: convert, gate on the tests, commit on main, push (the deploy):
scripts/ingest.sh s3://stellar-rpc-bench/results/<run-id> --dataset-kind synthetic --push-main

Two rails keep the flow from surprising you. The script refuses to run against a working tree with uncommitted tracked changes, and refuses to overwrite an already-ingested run — an existing docs/runs/<run_id>.json is an error until you pass --force. A --force re-ingest reuses the existing run/<run_id> branch instead of failing on it, and exits quietly when the reconverted JSON turns out byte-identical to the committed one.

Anything after a literal -- is passed straight through to convert.py, which is how you override a derived field without leaving the flow:

scripts/ingest.sh gs://rpc-full-history/results/<run-id> --dataset-kind synthetic -- \
  --run-name "Phase 3 — c6id.8xlarge rerun" \
  --unit-facts converter/facts/synthetic-2026-07-15.json

make convert — the layer underneath

make convert calls converter/convert.py and nothing else: no fetch, no branch, no commit. Reach for it when the bundle can't identify itself — the legacy pubnet and synthetic layouts predate metadata.json, so --run-id/--run-name/--run-date have no defaults to fall back on — or when you want to name every field by hand. Worked example against the archived pubnet run (docs/runs/archive/pubnet-2026-07-13.json):

# 1. Pull the results directory from GCS. (This is the exact path recorded as that
#    run's provenance.)
gcloud storage cp -r \
  gs://rpc-full-history/benchmarks/2026-07-13-user-dev-063a \
  ./results-in

# 2. Convert it into docs/runs/pubnet-2026-07-13.json and update docs/runs/index.json.
make convert \
  RESULTS=./results-in/2026-07-13-user-dev-063a \
  RUN_ID=pubnet-2026-07-13 \
  RUN_NAME="Pubnet — 4 sampled chunks (user-dev-063a)" \
  KIND=pubnet \
  RUN_DATE=2026-07-13 \
  GCS=gs://rpc-full-history/benchmarks/2026-07-13-user-dev-063a

# 3. Review the diff, then commit on a branch and open a PR — the same place
#    `scripts/ingest.sh` would have left you.
git add docs/runs
git commit -m "Add run pubnet-2026-07-13"

make convert variables:

Variable	Required	Meaning
RESULTS	yes	Path to the downloaded results directory
RUN_ID	yes	Slug; the file becomes docs/runs/<RUN_ID>.json
RUN_NAME	yes	Human-readable run name
KIND	yes	pubnet or synthetic
RUN_DATE	yes	YYYY-MM-DD
FACTS	no	Path to a --unit-facts sidecar JSON (synthetic dataset meta)
URI	no	Source s3://… (or gs://…) path, recorded in the run for provenance
GCS	no	Legacy: source gs://… path as campaign.source_gcs; prefer URI

Omitting any required variable fails with a message naming the missing one. results-in/ is git-ignored.

Transaction-submission results (interim page)

docs/tx-submission.html reports a one-off sendTransaction benchmark (stellar/stellar-rpc#869): the time the stellar-rpc process spends inside one submission request, across three transaction profiles on two networks. Those results don't fit the schema-v1 run JSON (single run, exact per-request percentiles — not 5-rep CSV campaigns), so the harvest summaries produced by stellar-rpc-blaster's scripts/tx-submission/harvest.py are committed verbatim under docs/txsub/ and rendered by docs/txsub.js. This is deliberate and interim — a unifying refactor happens when the other benchmark families land.

To add a future run: upload the bundle to GCS, copy its summary-<mode>.json files byte-identical into docs/txsub/<bundle-id>/, and add an entry to docs/txsub/index.json (network, role, date, rps × duration, rpc_version, instance, GCS path, summary paths).

Automated ingest (stellar-rpc's bench-campaign.yml)

Campaigns dispatched from stellar-rpc's bench-campaign.yml workflow ingest their own results: after a passing campaign, its notify job downloads the results tarball the benchmark box uploaded, clones this repo, and runs

scripts/ingest.sh bench-results-<run_id>.tgz --dataset-kind synthetic --push-main \
  -- --source-uri s3://stellar-rpc-bench/results/<run_id>

— the same script as the local flow above, which is the point: CI is not a second implementation that can drift. --push-main gates the push on make test + make smoke, commits on main, and pushes; deploy-pages.yml then publishes the site, and the campaign's summary and Slack message carry the https://stellar-experimental.github.io/stellar-rpc-benchmarks/?run=<run_id> link.

The push authenticates with a write-capable token for this repo, stored as the BENCHMARKS_PUSH_TOKEN secret on stellar/stellar-rpc. No AWS credentials live in this repo: the tarball travels through the campaign's own S3 prefix, which stellar-rpc's existing CI role can read. A failed ingest never fails the campaign — the notification falls back to the raw s3:// results URI, and the run can be ingested later from a laptop with the same command.

Data model

One JSON file per run under docs/runs/, listed in docs/runs/index.json. The full schema (value shapes, sections, manifest) is documented in SCHEMA.md.

Statistical conventions

• Every reported value is the median across the 5 runs; spread is min–max; the raw 5-run array is kept alongside.
• Percentiles are per-run values aggregated the same way — never averaged across runs.
• Derived rates (throughputs) are computed per run from raw fields, then aggregated — never derived from already-aggregated medians.
• Nothing is interpolated or smoothed; every rendered number traces to a raw CSV field.

See SCHEMA.md for the authoritative statement.

Repo layout

stellar-rpc-benchmarks/
├── Makefile                 # ingest / convert / test / smoke / serve / help
├── README.md
├── SCHEMA.md                # run JSON schema v1 (the data contract)
├── .github/
│   └── workflows/
│       ├── ingest.yml       # workflow_dispatch: GCS bundle → run PR (delegates to scripts/ingest.sh)
│       ├── deploy-pages.yml # sync main:/docs to the gh-pages branch Pages serves
│       ├── pr-preview.yml   # publish each PR's docs/ under gh-pages:/pr-preview/pr-<n>/
│       ├── runner-go.yml    # go vet + go test for the campaign runner
│       └── shellcheck.yml   # lint the remaining shell scripts on every PR that touches them
├── runner/                  # benchmark operations: the devbox side producing result bundles
│   ├── bootstrap.sh         # provision the devbox (idempotent, no builds)
│   ├── cmd/campaign/        # campaign CLI: run · plan · preflight · publish (see runner/README.md)
│   ├── internal/            # config · plan · run · preflight · publish · bundle
│   └── example-campaign.toml # annotated config to copy from
├── scripts/
│   └── ingest.sh            # bundle → converted run → run/<id> branch → PR (make ingest)
├── converter/
│   ├── convert.py           # results dir → docs/runs/<id>.json (+ manifest), stdlib only
│   ├── facts/               # per-unit sidecar facts (e.g. synthetic model/tps/pack)
│   └── tests/               # make test → python3 -m unittest discover converter/tests
├── tests/
│   └── smoke/               # make smoke → jsdom viewer smoke test (Node, dev-only)
└── docs/                    # GitHub Pages root (static vanilla-JS viewer)
    ├── index.html           # the viewer shell (dropdown / ?run=<id>)
    ├── app.js               # renderers (per dataset.kind) + charts
    ├── styles.css           # design system (light + dark), shared by every page
    ├── summary.html         # stakeholder summary page (summary.js + summary.css)
    ├── latency-model.html   # end-to-end latency model against the phase targets
    ├── tx-submission.html   # transaction-submission report (txsub.js)
    ├── targets.json         # phase performance targets — single source of truth
    ├── dataset-sizes.json   # measured sizes of the synthetic dataset profiles
    ├── txsub/               # tx-submission harvest summaries, verbatim (+ index.json)
    └── runs/
        ├── index.json       # manifest of runs (oldest date first)
        ├── <run-id>.json    # one file per run (schema v1)
        └── archive/         # retired pre-campaign runs, with their own manifest

Future work

• Nightly scheduled ingestion / auto-discovery of new GCS runs (vs. today's manual dispatch).
• bench-compare A/B views in the viewer.
• Cross-run diffing / trend charts across campaigns over time.
• Migration to a stellar-org repo from stellar-experimental.
• The WIF setup itself (the pending dev-hubble pool/provider + service account that unblocks the GitHub Action flow).