ci: opt-in benchmark workflow, micro and end-to-end - #250
Open
BenjaminDEMAILLE wants to merge 2 commits into
Open
ci: opt-in benchmark workflow, micro and end-to-end#250BenjaminDEMAILLE wants to merge 2 commits into
BenjaminDEMAILLE wants to merge 2 commits into
Conversation
Several open dependency questions (#162, #202, #205, #208) all start with "measure first", and each was about to invent its own measurement. This is the shared one. Divan rather than criterion: the questions that prompted a harness compare peak memory as much as speed, and divan reports allocation counts next to wall time without extra setup; a full run also takes seconds, so it stays usable in a pull request rather than only in a nightly job. Criterion is the better choice if the CI integration in #245 needs its report format, and swapping is a benches/ change rather than an API one. Three groups, chosen as the functions the open questions would replace: seed-extension scanning (`find_stop`, what a portable-SIMD crate would swap), the gene-overlap query and the annotation build (what an interval crate would swap). All are pure functions with no genome index to build, so a run takes seconds and a regression is attributable to one function. `find_stop` and the GTF record type are private to the crate; rather than widen the published API for a benchmark, both are exposed under a `bench` feature that only benches/ enables. Closes #204.
Two jobs, both opt-in the way the large-dataset job is (manual dispatch or a `benchmark` label), because a wall time from a busy shared runner is worse than no wall time. - micro: runs the divan benches, optionally against a baseline ref, and summarises them as a table with anything past 10% flagged. Handles a baseline older than the harness by saying so rather than failing. - end-to-end: times indexing and alignment separately for both aligners on the nf-core fixture, reporting wall time and peak RSS. Deliberately without thresholds: it is a data point, and only comparable within one machine. test/bench_report.py parses divan's tree output into the table. It was checked both ways: it recovers the group/argument names from real output, and with one baseline row perturbed it flags that row at +41% and leaves the others at 0%. Closes #245.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #245. Stacked on #248 (the divan harness), which it uses.
Two jobs, both opt-in the way #249's large-dataset job is (manual dispatch, or a
benchmarklabel on a pull request), because a wall time from a busy shared runner is worse than no wall time.microRuns
cargo bench --bench hot_paths, optionally against a baseline ref, and writes a table into the job summary:A baseline older than the harness has no benches to run; the job says so rather than failing.
end-to-endTimes indexing and alignment separately for both aligners on the nf-core fixture, with peak RSS, because they are different questions: indexing is dominated by suffix-array construction, alignment by the seed and stitch loops. Deliberately without thresholds — it is a data point, and only comparable within one machine.
The parser was checked both ways
test/bench_report.pyturns divan's box-drawing tree into the table. My first version silently lost the group names and reported rows called100and2000; the fix keys them asgene_overlap_query/100. It is also checked against a perturbed baseline: with one row moved from 98.7 ns to 70 ns in the baseline file, that row is flagged at +41.0% and every other row reads +0.0%.What I could not do from here
The issue asks to reuse the scverse benchmarking setup on scverse AWS resources. The runner label is an input with
ubuntu-latestas the default, so pointing this at a runner group is a one-word change once that group exists — but I cannot create it, and I did not guess at its name.🤖 Generated with Claude Code