e2e benchmark campaigns workflow — dispatch, run on EC2, publish results #868

TL;DR

Queue a benchmark campaign with one workflow dispatch: a fresh EC2 NVMe box runs from stellar-rpc-benchmarks, publishes the result bundle to s3://stellar-rpc-bench/results/<run_id>/ then sends the results to Slack.

Design

GitHub-hosted jobs cap at 6 h; campaigns run 5–17 h (pacing-bound). The workflow uses a poll-relay chain: validate (<30 s: reject bad combos, estimate wall clock, render the campaign TOML) → launch tagged instance → sequential ~5.5 h poller jobs that hand off → notify → always() cleanup. Orphan protection is three independent layers: boot-scheduled self-terminate, the cleanup job, and a reaper cron that kills any tagged instance past its deadline tag.

Dispatch inputs: ingest (cold|hot|both|none), query (yes|no), phase (1|2|3 → datasets + close interval), runs (default 1), workers, machine (2x|8x), ref, hot_num_ledgers (smoke-run shortener).

Progress

• S3 bucket + IAM (stellar/terraform#6624, applied)
• Datasets migrated GCS → S3, byte-verified (six profiles, ~185 GB)
• Go campaign runner (contract: align SCHEMA.md and the converter with the runner bundle manifests stellar-experimental/stellar-rpc-benchmarks#20 through runner: operator docs and the example campaign config stellar-experimental/stellar-rpc-benchmarks#27)
• Runner: packs-s3 dataset kind + unattended bootstrap (runner: packs-s3 dataset kind + unattended bootstrap stellar-experimental/stellar-rpc-benchmarks#29)
• bench-campaign.yml + leg script + relay poller + reaper — bench(full-history): run benchmark campaigns from GitHub Actions (#868) #921 (draft; registration stub ci: registration stub for the bench-campaign dispatch workflow #922 merged)
• Secrets: SLACK_BENCH_WEBHOOK_URL
• Retire the standing benchmark devboxes after the first automated campaign (from Benchmarking infra: stellar-rpc-bench S3 bucket + benchmarking devbox #866)
• Acceptance: smoke campaign (~1 h), failure drills, full phase-3 then phase-1 run

Done when

A team member dispatches a campaign and comes back to a Slack message linking the published bundle; the box is gone. Weekly cron is out of scope for v1.

Related

• Run full-history benchmark commands in CI to catch perf regressions #775 — CI regression gate (sibling, unchanged scope)
• Benchmarking infra: stellar-rpc-bench S3 bucket + benchmarking devbox #866 — S3 bucket + devbox infra (closed; remaining items folded in here)
• Config-driven benchmark campaign runner for the full-history bench subcommands #896 — bash campaign runner tooling (closed)