test(perf): scale timing bounds by a measured machine-speed factor#760
Open
haakon-e wants to merge 1 commit into
Open
test(perf): scale timing bounds by a measured machine-speed factor#760haakon-e wants to merge 1 commit into
haakon-e wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #760 +/- ##
=======================================
Coverage 92.95% 92.95%
=======================================
Files 56 56
Lines 2880 2880
=======================================
Hits 2677 2677
Misses 203 203
🚀 New features to boost your workflow:
|
haakon-e
force-pushed
the
he/perf-thresholds
branch
from
July 9, 2026 05:59
c93f6dc to
b3ecafb
Compare
Author timing bounds for the reference machine (a linux GPU node) and scale them at runtime by a machine-speed factor measured once per suite from a fixed calibration workload, so strict reference bounds hold on slower GitHub-hosted runners without loosening them on the machines where regressions must be caught.
haakon-e
force-pushed
the
he/perf-thresholds
branch
from
July 9, 2026 06:19
b3ecafb to
3ea7055
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Remove the recurring false failures in
test/performance_tests.jlon slow CI runners (notably the Julia 1.12 GitHub-hosted runners, which run the microphysics benchmarks several times slower than the reference machine the hardcoded bounds were tuned on).What
Measure a machine-speed factor once per suite by benchmarking a fixed transcendental workload (
calibration_workload) against a reference minimum runtime (REFERENCE_CALIBRATION_NS), clamped to[1, MAX_SPEED_FACTOR]. Every timing assertion scales its bound by this factor:min_run_timebecomesmin_run_time * SPEED_FACTOR[].Why
The timing bounds are absolute nanosecond limits tuned on one machine. On a slower runner every bound is proportionally too tight, so type-stable, non-regressed code fails purely on wall-clock time. Scaling the bounds by a measured per-machine factor keeps the same regression sensitivity on fast machines while removing the false failures on slow ones. The memory and allocation assertions are machine-independent and unchanged.