diff --git a/CHANGELOG.md b/CHANGELOG.md index 2752c7a7..590d1029 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ Notable changes to house-rules. Versions follow [semantic versioning](https://se ## Unreleased +- `hr-benchmarking` now gives a default stopping rule for performance claims, in place of "no + universal minimum". Run 3 alternating runs per arm, and add 2 more per arm only when the arms' + ranges overlap or the gap is within the variation criterion, for a cap of 5. If 5 runs per arm + leave the comparison unresolved, it is reported as inconclusive. Another count is used only when + the user asks for it or a run is too long to repeat 3 times, and is declared with its reason + before the first run. Results always carry the run count and range, never a bare average. The + expected wall-clock time is stated before the first run. The JavaScript reference no longer asks + for at least 10 runs when timing startup. + - The `hr-code-review` panel recommendations gain an entry for Muse Wonder Full 1.0 on the Bug Hunt board: two runs each at xhigh (41.5 of 105) and max (42.0), no published price, and reachable only through a Muse Code subscription plan. The existing Opus 5.5 and Sonnet 5.5 entries were checked diff --git a/skills/hr-benchmarking/SKILL.md b/skills/hr-benchmarking/SKILL.md index 19967252..7d1dbbcc 100644 --- a/skills/hr-benchmarking/SKILL.md +++ b/skills/hr-benchmarking/SKILL.md @@ -30,7 +30,8 @@ Before running, list the selected engines, workloads, metrics, controls and repetitions. Count the planned runs explicitly, including controls and stages; state which decision each retained dimension helps make. Choose combinations for that decision rather than automatically running every combination. Use a -broader matrix when its required coverage is stated. +broader matrix when its required coverage is stated. State the benchmark's +expected wall-clock time, waits included, before the first run. Name the measurement unit: request, operation, batch or whole command. State whether setup, startup, compilation, input loading, retries and cleanup count. @@ -118,18 +119,25 @@ Before exploratory runs, define the decision-relevant signal for selecting comparisons for confirmation. Choose comparisons that meet that signal, plus decision-critical comparisons, for repeated confirmation; keep necessary controls. Set the confirmation plan before its runs and update the explicit run count -when the scope changes. Repetition counts depend on the -decision and observed variation; there is no universal minimum. +when the scope changes. -For a performance claim, add alternating runs to assess run-to-run variation. +The default stopping rule for a performance claim is 3 runs per arm. Add 2 more +per arm only when the arms' ranges overlap or the effect is near the noise, +meaning the gap is within the variation criterion set below. 5 per arm is the +cap. If 5 runs per arm leave the comparison unresolved, report it as +inconclusive rather than adding runs. Use another count only when the user asks +for it or a run is too long to repeat 3 times, and declare it before the first +run, with the reason. Alternate the arms in every run so drift reaches each arm. + +For a performance claim, assess run-to-run variation from those alternating runs. Treat variation as already known only when earlier repeated runs of the same variant, build and input used the same workload, host, run conditions and variation statistic, and met the repetition count or stopping rule declared for this claim. Otherwise include an A/A arm, which runs the same variant twice in the same alternation. Its spread is that comparison's own noise floor, and it can supply the preset variation statistic. -Before the first run used in that claim, set a repetition count or stopping -rule and a reproducible variation criterion (statistic and threshold). +Before the first run used in that claim, set a reproducible variation criterion +(statistic and threshold), and the stopping rule if it differs from the default. For short or tail metrics, such as the longest stall in a run, give the threshold an absolute floor or use a percentile or count metric; a percentage @@ -139,7 +147,8 @@ claim's sample. Capture raw results, exit status, elapsed time, CPU and memory observations. Report monitoring overhead separately from the measured workload; batch compatible probes when they dominate elapsed time. Report the selected repetition count or stopping rule, variation statistic and -threshold, sample count, median and range. Apply the preset +threshold, sample count, median and range. Always give the run count and the +range with any central value, never a bare average. Apply the preset criterion and treat a gap within run-to-run variation as inconclusive. Profile separately from the runs used to claim a speed difference. diff --git a/skills/hr-benchmarking/references/javascript.md b/skills/hr-benchmarking/references/javascript.md index 50246712..118a7e7b 100644 --- a/skills/hr-benchmarking/references/javascript.md +++ b/skills/hr-benchmarking/references/javascript.md @@ -88,8 +88,9 @@ workload and host, not a benchmark. kept and dropped, keeping the owner reachable to the end. One researcher's run, not repeated, kept only the two fields that were read and went from 61.45 MiB to 2.10 MiB over 30 sessions of 10 turns of 200 KiB. -- **Time startup on the built artifact, interleaved.** Alternate arms over at - least 10 runs on one build. An environment variable can stand in for an API +- **Time startup on the built artifact, interleaved.** Alternate arms on + one build, using the default run count in [the skill](../SKILL.md). An + environment variable can stand in for an API call, but note what it changes (it also reaches child processes). A V8 compile cache took a CLI's `--help` from a 554 ms median to 381 ms over 11 runs (M). - **Treat another researcher's numbers as leads.** Of three top candidates from a