Skip to content

Run every benchmark through one reusable bench workflow - #3

Merged
dbworku merged 1 commit into
mainfrom
consolidate-bench
Oct 14, 2026
Merged

dbworku merged 1 commit into
mainfrom
consolidate-bench

Conversation

@dbworku

@dbworku dbworku commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Both workflows duplicated the plan job, the runner matrix, host info, bench-finish and the summary job. This moves all of it into one place.

  • .github/workflows/bench.yml (workflow_call, inputs suite and replicas): the shared pipeline. The plan job validates the suite and replicas and builds the replica list; bench runs the suite once per runner and replica; summary renders the p50 / p90 table. The runner matrix is defined only here.
  • .github/actions/typescript-mastra, .github/actions/rust-spotify-player: each suite's phases as a composite action. The commands, pins and phase names are unchanged.
  • Entry workflows: each is now just its replicas input and a call to bench.yml.

A single composite action can't hold a job matrix or separate jobs, so the shared part is a reusable workflow. uses: can't take an expression, so bench.yml picks the suite with one if:-gated step per suite.

Docs: the README gains a layout table and steps for adding a runner or a benchmark, and its benchmark table links each suite action. Comments in bin/mark, mise.toml (which still said "Non-Bun") and .github/actionlint.yaml now point at bench.yml.

Failure-path fixes (pre-existing, found in review):

  • bench-finish only runs when the harness checkout succeeded, since it is loaded from the harness.
  • The summary skips artifacts without host.json instead of crashing the whole table, and counts ok against the requested replicas, so a job that never reported counts as a failure instead of disappearing.
  • The summary no longer runs when plan rejects the inputs.

actionlint, zizmor and shellcheck are clean, and the summary was tested locally against fixtures for these failure paths. Not yet run in Actions.

The plan job, runner matrix, replicas, host info, bench-finish and summary job
were duplicated across both workflows. They now live once in
.github/workflows/bench.yml (workflow_call with suite and replicas inputs).
Each suite's phases move into a composite action (.github/actions/<suite>),
and the entry workflows shrink to their replicas input and a call to
bench.yml. bench.yml puts bin/ on PATH so suites call mark directly.

Failure paths: bench-finish only runs when the harness checkout succeeded,
the summary skips artifacts without host.json instead of crashing and counts
ok against the requested replicas (so a job that never reported is a
failure, not absent), and the summary no longer runs when plan rejects the
inputs.

Docs: README gains a layout table and how to add a runner or a benchmark,
and the matrix, mark, mise.toml and actionlint comments point at the new
layout.
@dbworku
dbworku merged commit 74e5e8c into main Oct 14, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant