Repository navigation
Run every benchmark through one reusable bench workflow - #3
Merged
Merged
Conversation
dbworku
force-pushed
the
consolidate-bench
branch
from
October 13, 2026 23:37
bdd74e9 to
8260d44
Compare
The plan job, runner matrix, replicas, host info, bench-finish and summary job were duplicated across both workflows. They now live once in .github/workflows/bench.yml (workflow_call with suite and replicas inputs). Each suite's phases move into a composite action (.github/actions/<suite>), and the entry workflows shrink to their replicas input and a call to bench.yml. bench.yml puts bin/ on PATH so suites call mark directly. Failure paths: bench-finish only runs when the harness checkout succeeded, the summary skips artifacts without host.json instead of crashing and counts ok against the requested replicas (so a job that never reported is a failure, not absent), and the summary no longer runs when plan rejects the inputs. Docs: README gains a layout table and how to add a runner or a benchmark, and the matrix, mark, mise.toml and actionlint comments point at the new layout.
dbworku
force-pushed
the
consolidate-bench
branch
from
October 13, 2026 23:48
8260d44 to
22761bd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both workflows duplicated the plan job, the runner matrix, host info, bench-finish and the summary job. This moves all of it into one place.
.github/workflows/bench.yml(workflow_call, inputssuiteandreplicas): the shared pipeline. Theplanjob validates the suite and replicas and builds the replica list;benchruns the suite once per runner and replica;summaryrenders the p50 / p90 table. The runner matrix is defined only here..github/actions/typescript-mastra,.github/actions/rust-spotify-player: each suite's phases as a composite action. The commands, pins and phase names are unchanged.replicasinput and a call tobench.yml.A single composite action can't hold a job matrix or separate jobs, so the shared part is a reusable workflow.
uses:can't take an expression, sobench.ymlpicks the suite with oneif:-gated step per suite.Docs: the README gains a layout table and steps for adding a runner or a benchmark, and its benchmark table links each suite action. Comments in
bin/mark,mise.toml(which still said "Non-Bun") and.github/actionlint.yamlnow point atbench.yml.Failure-path fixes (pre-existing, found in review):
host.jsoninstead of crashing the whole table, and countsokagainst the requested replicas, so a job that never reported counts as a failure instead of disappearing.planrejects the inputs.actionlint, zizmor and shellcheck are clean, and the summary was tested locally against fixtures for these failure paths. Not yet run in Actions.