Skip to content

Run every benchmark through one reusable bench workflow #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dbworku merged 1 commit into from
Oct 14, 2026
Merged

Run every benchmark through one reusable bench workflow #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actionlint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Runner labels outside actionlint's built-in set. The StarSling labels and the ubuntu-26.04-{2,8}
# larger runners are registered at the org level; ubuntu-26.04 is a real GitHub-hosted label that
# actionlint 1.7.12 predates (drop it once the pinned actionlint knows it). Keep this list in sync
# with the labels used across .github/workflows/.
# with the matrix labels in .github/workflows/bench.yml.
self-hosted-runner:
labels:
- starsling-ubuntu-26.04-2
Expand Down
63 changes: 63 additions & 0 deletions .github/actions/rust-spotify-player/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: "Suite: Rust build (spotify-player)"
description: >-
aome510/spotify-player's CI, after runs-on-demo/benchmark-public's rust suite. Steps named
"phase: <name>" are measured, each opened by `mark`.
runs:
using: composite
steps:
- name: Environment
shell: bash
run: |
{
echo "CARGO_TERM_COLOR=always"
echo "RUST_FEATURES=rodio-backend,media-control,image,notify,fzf"
} >> "$GITHUB_ENV"
- shell: bash
run: mark checkout
- name: "phase: checkout"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: aome510/spotify-player
ref: 7dc9d17a7bf5671eb6129fdf9127729f659637fc
path: work
persist-credentials: false
- shell: bash
run: mark "system packages"
- name: "phase: system packages"
shell: bash
run: |
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq \
build-essential pkg-config libssl-dev libasound2-dev libdbus-1-dev \
libxcb-shape0-dev libxcb-xfixes0-dev
- shell: bash
run: mark toolchain
- name: "phase: toolchain"
uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master, 2026-10-09
with:
toolchain: 1.98.1
components: clippy, rustfmt
- shell: bash
run: mark "cargo fmt"
- name: "phase: cargo fmt"
shell: bash
working-directory: work
run: cargo fmt --all -- --check
- shell: bash
run: mark "cargo test"
- name: "phase: cargo test"
shell: bash
working-directory: work
run: cargo test --no-default-features --features "$RUST_FEATURES"
- shell: bash
run: mark "cargo clippy"
- name: "phase: cargo clippy"
shell: bash
working-directory: work
run: cargo clippy --no-default-features --features "$RUST_FEATURES" -- -D warnings
- shell: bash
run: mark "cargo clippy (no features)"
- name: "phase: cargo clippy (no features)"
shell: bash
working-directory: work
run: cargo clippy --no-default-features -- -D warnings
15 changes: 12 additions & 3 deletions .github/actions/summarize/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
name: Summarize
description: Tabulate every bench job's phase timings into the run summary and a results artifact.
inputs:
replicas:
description: Jobs each runner was asked to run (the denominator of the ok column)
required: true
runs:
using: composite
steps:
Expand All @@ -11,12 +15,17 @@ runs:
# lines, then bench-finish's `end <µs> <job status>`). A phase runs to the next boundary. The
# table has one row per runner label, with p50 / p90 over that label's successful replicas.
- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
REPLICAS: ${{ inputs.replicas }}
with:
script: |
const fs = require("node:fs");
const dir = `${process.env.RUNNER_TEMP}/bench`;
const read = (job, file) => fs.readFileSync(`${dir}/${job}/${file}`, "utf8");
const jobs = (fs.existsSync(dir) ? fs.readdirSync(dir) : []).map((job) => {
// A job whose host-info step failed left no host.json; it only counts against its runner's ok
// column, through the replicas denominator.
const artifacts = fs.existsSync(dir) ? fs.readdirSync(dir).filter((job) => fs.existsSync(`${dir}/${job}/host.json`)) : [];
const jobs = artifacts.map((job) => {
const rows = read(job, "phases.tsv").trim().split("\n").map((line) => line.split("\t"));
const phases = rows.slice(0, -1).flatMap(([name, start], i) => (name === "-" ? [] : [{ name, seconds: (rows[i + 1][1] - start) / 1e6 }]));
const totalSeconds = phases.reduce((sum, phase) => sum + phase.seconds, 0);
Expand All @@ -41,14 +50,14 @@ runs:
cells.sort((a, b) => a.advertisedVcpu - b.advertisedVcpu || a.p50 - b.p50);
await core.summary
.addHeading(context.workflow)
.addRaw("p50 / p90 seconds over each runner's successful replicas.", true)
.addRaw("p50 / p90 seconds over each runner's successful replicas; ok is successful / requested replicas.", true)
.addTable([
["vCPU", "runner", "RAM GiB", "ok", ...names, "total"].map((data) => ({ data, header: true })),
...cells.map((cell) => [
`${cell.vcpu}`,
cell.label,
`${cell.memoryGiB}`,
`${cell.ok.length}/${cell.runs.length}`,
`${cell.ok.length}/${process.env.REPLICAS}`,
...names.map((name) => stat(cell.ok.flatMap((job) => job.phases.filter((phase) => phase.name === name).map((phase) => phase.seconds)))),
stat(cell.ok.map((job) => job.totalSeconds)),
]),
Expand Down
78 changes: 78 additions & 0 deletions .github/actions/typescript-mastra/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: "Suite: TypeScript build (mastra)"
description: >-
mastra-ai/mastra's checkout, cold install, lint and build. Mirrors the realworld-mastra profile in
hpc-sandbox-benchmarks (same pin, same commands, same cold-store rules). Steps named
"phase: <name>" are measured, each opened by `mark`; the rest is setup and never timed.
runs:
using: composite
steps:
- name: Environment
shell: bash
# pnpm's store is what makes an install cold or warm. pnpm 11 resolves it from PNPM_HOME
# first, then <XDG_DATA_HOME>/pnpm/store; images that set PNPM_HOME or carry a store in $HOME
# would otherwise turn the cold install into a warm one. Pin every knob into an empty
# directory under RUNNER_TEMP. A fresh checkout has no turbo cache, but TURBO_FORCE never lets
# a cache hit stand in for a build.
run: |
root="$RUNNER_TEMP/cold"
mkdir -p "$root/pnpm-home" "$root/xdg-data" "$root/xdg-cache" "$root/corepack"
{
echo "CI=true"
echo "COREPACK_ENABLE_DOWNLOAD_PROMPT=0"
echo "DO_NOT_TRACK=1"
echo "TURBO_TELEMETRY_DISABLED=1"
echo "TURBO_FORCE=true"
echo "PNPM_HOME=$root/pnpm-home"
echo "XDG_DATA_HOME=$root/xdg-data"
echo "XDG_CACHE_HOME=$root/xdg-cache"
echo "COREPACK_HOME=$root/corepack"
} >> "$GITHUB_ENV"
- shell: bash
run: mark checkout
- name: "phase: checkout"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: mastra-ai/mastra
ref: b6eeb9d08ebeaf27bc6fd16b6b88087040aaf767
path: work
persist-credentials: false
- shell: bash
run: mark -
- name: Setup Node
uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version: "22"
- name: Setup pnpm
shell: bash
working-directory: work
# corepack fetches the pnpm that mastra's packageManager field pins (sha512-verified) here, so
# the download never lands inside the cold install.
run: |
corepack enable
corepack install
pnpm --version
# Refuse to time a warm install: the store must not exist yet.
store="$(pnpm store path)"
if [ -e "$store" ]; then
echo "::error::pnpm store already exists at $store; the install would not be cold"
exit 1
fi
- shell: bash
run: mark "cold install"
- name: "phase: cold install"
shell: bash
working-directory: work
run: pnpm install --frozen-lockfile
- shell: bash
run: mark lint
- name: "phase: lint"
shell: bash
working-directory: work
run: pnpm lint:format
- shell: bash
run: mark build
- name: "phase: build"
shell: bash
working-directory: work
# Scoped to packages/core (and its workspace deps), as in the realworld-mastra profile.
run: pnpm build:core
103 changes: 103 additions & 0 deletions .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
name: Bench

# The pipeline every benchmark shares: plan the replicas, run the suite once per runner and replica,
# then summarize. Each entry workflow (typescript-mastra.yml, rust-spotify-player.yml) calls this
# with its suite name; the suite's phases live in the composite action .github/actions/<suite>.

on:
workflow_call:
inputs:
suite:
description: Suite to run (a composite action under .github/actions/)
type: string
required: true
replicas:
description: Jobs per runner (1-40)
type: number
required: true

permissions:
contents: read

jobs:
plan:
runs-on: starsling-ubuntu-26.04-2
outputs:
replicas: ${{ steps.replicas.outputs.list }}
steps:
# Expressions have no range(): turn the count into the [1..N] list the matrix fans out over.
# 40 keeps 6 runners x N under GitHub's 256-job matrix limit.
- id: replicas
env:
SUITE: ${{ inputs.suite }}
REPLICAS: ${{ inputs.replicas }}
run: |
# Keep in sync with the suite steps of the bench job.
case "$SUITE" in
typescript-mastra | rust-spotify-player) ;;
*) echo "::error::unknown suite '$SUITE'"; exit 1 ;;
esac
if ! [[ "$REPLICAS" =~ ^[0-9]+$ ]] || (( REPLICAS < 1 || REPLICAS > 40 )); then
echo "::error::replicas must be 1-40, got '$REPLICAS'"
exit 1
fi
echo "list=[$(seq 1 "$REPLICAS" | paste -sd, -)]" >> "$GITHUB_OUTPUT"

bench:
needs: plan
name: ${{ matrix.vcpu }} vCPU · ${{ matrix.label }} · ${{ matrix.replica }}
strategy:
fail-fast: false
matrix:
vcpu: [2, 4, 8]
provider: [starsling, github]
replica: ${{ fromJSON(needs.plan.outputs.replicas) }}
include:
- { vcpu: 2, provider: starsling, label: starsling-ubuntu-26.04-2 }
- { vcpu: 2, provider: github, label: ubuntu-26.04-2 }
- { vcpu: 4, provider: starsling, label: starsling-ubuntu-26.04 }
# GitHub's standard runner, 4 vCPU in public repositories (2 and 8 are the org's larger
# runners).
- { vcpu: 4, provider: github, label: ubuntu-26.04 }
- { vcpu: 8, provider: starsling, label: starsling-ubuntu-26.04-8 }
- { vcpu: 8, provider: github, label: ubuntu-26.04-8 }
runs-on: ${{ matrix.label }}
timeout-minutes: 45
steps:
- id: harness
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: .harness
persist-credentials: false
# Puts bin/mark on PATH for the suite's phase boundaries.
- run: echo "$GITHUB_WORKSPACE/.harness/bin" >> "$GITHUB_PATH"
- name: Host info
uses: ./.harness/.github/actions/host-info
with:
label: ${{ matrix.label }}
vcpu: ${{ matrix.vcpu }}
# `uses:` can't take an expression, so each suite is its own step.
- if: inputs.suite == 'typescript-mastra'
uses: ./.harness/.github/actions/typescript-mastra
- if: inputs.suite == 'rust-spotify-player'
uses: ./.harness/.github/actions/rust-spotify-player
# Needs the harness for the action itself; without it the job left no timings, and the summary
# counts it as a failed replica.
- name: Bench finish
if: always() && steps.harness.outcome == 'success'
uses: ./.harness/.github/actions/bench-finish
with:
key: ${{ matrix.label }}-${{ matrix.replica }}

# Runs even when bench jobs fail or are cancelled, but not when plan rejected the inputs.
summary:
needs: [plan, bench]
if: always() && needs.plan.result == 'success'
runs-on: starsling-ubuntu-26.04-2
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/summarize
with:
replicas: ${{ inputs.replicas }}
Loading