From 70d90d6cdda8f75e817bdb6cc379b04f02b2bdde Mon Sep 17 00:00:00 2001 From: bompus Date: Wed, 14 Oct 2026 18:58:17 -0600 Subject: [PATCH 1/4] docs(benchmarking): default to 3 runs per arm, up to 5 Add 2 runs per arm only when the ranges overlap or the effect is near the noise. Report the run count and range, and state the expected wall-clock time before the first run. --- CHANGELOG.md | 6 ++++++ skills/hr-benchmarking/SKILL.md | 21 ++++++++++++------- .../hr-benchmarking/references/javascript.md | 5 +++-- 3 files changed, 23 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2752c7a7..d8d318a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ Notable changes to house-rules. Versions follow [semantic versioning](https://se ## Unreleased +- `hr-benchmarking` now gives a default stopping rule for performance claims, in place of "no + universal minimum". Run 3 alternating runs per arm, and add 2 more per arm only when the arms' + ranges overlap or the effect is near the noise, for a cap of 5. Results always carry the run count + and range, never a bare average. The expected wall-clock time is stated before the first run. The + JavaScript reference no longer asks for at least 10 runs when timing startup. + - The `hr-code-review` panel recommendations gain an entry for Muse Wonder Full 1.0 on the Bug Hunt board: two runs each at xhigh (41.5 of 105) and max (42.0), no published price, and reachable only through a Muse Code subscription plan. The existing Opus 5.5 and Sonnet 5.5 entries were checked diff --git a/skills/hr-benchmarking/SKILL.md b/skills/hr-benchmarking/SKILL.md index 19967252..828146cf 100644 --- a/skills/hr-benchmarking/SKILL.md +++ b/skills/hr-benchmarking/SKILL.md @@ -30,7 +30,8 @@ Before running, list the selected engines, workloads, metrics, controls and repetitions. Count the planned runs explicitly, including controls and stages; state which decision each retained dimension helps make. Choose combinations for that decision rather than automatically running every combination. Use a -broader matrix when its required coverage is stated. +broader matrix when its required coverage is stated. State the benchmark's +expected wall-clock time, waits included, before the first run. Name the measurement unit: request, operation, batch or whole command. State whether setup, startup, compilation, input loading, retries and cleanup count. @@ -118,18 +119,23 @@ Before exploratory runs, define the decision-relevant signal for selecting comparisons for confirmation. Choose comparisons that meet that signal, plus decision-critical comparisons, for repeated confirmation; keep necessary controls. Set the confirmation plan before its runs and update the explicit run count -when the scope changes. Repetition counts depend on the -decision and observed variation; there is no universal minimum. +when the scope changes. -For a performance claim, add alternating runs to assess run-to-run variation. +The default stopping rule for a performance claim is 3 runs per arm. Add 2 more +per arm only when the arms' ranges overlap or the effect is near the noise, so +5 per arm is the cap. If 5 leave the comparison unresolved, report it as +inconclusive rather than adding runs. Declare any other count before the first +run, with the reason. Alternate the arms in every run so drift reaches each arm. + +For a performance claim, assess run-to-run variation from those alternating runs. Treat variation as already known only when earlier repeated runs of the same variant, build and input used the same workload, host, run conditions and variation statistic, and met the repetition count or stopping rule declared for this claim. Otherwise include an A/A arm, which runs the same variant twice in the same alternation. Its spread is that comparison's own noise floor, and it can supply the preset variation statistic. -Before the first run used in that claim, set a repetition count or stopping -rule and a reproducible variation criterion (statistic and threshold). +Before the first run used in that claim, set a reproducible variation criterion +(statistic and threshold), and the stopping rule if it differs from the default. For short or tail metrics, such as the longest stall in a run, give the threshold an absolute floor or use a percentile or count metric; a percentage @@ -139,7 +145,8 @@ claim's sample. Capture raw results, exit status, elapsed time, CPU and memory observations. Report monitoring overhead separately from the measured workload; batch compatible probes when they dominate elapsed time. Report the selected repetition count or stopping rule, variation statistic and -threshold, sample count, median and range. Apply the preset +threshold, sample count, median and range. Always give the run count and the +range with any central value, never a bare average. Apply the preset criterion and treat a gap within run-to-run variation as inconclusive. Profile separately from the runs used to claim a speed difference. diff --git a/skills/hr-benchmarking/references/javascript.md b/skills/hr-benchmarking/references/javascript.md index 50246712..118a7e7b 100644 --- a/skills/hr-benchmarking/references/javascript.md +++ b/skills/hr-benchmarking/references/javascript.md @@ -88,8 +88,9 @@ workload and host, not a benchmark. kept and dropped, keeping the owner reachable to the end. One researcher's run, not repeated, kept only the two fields that were read and went from 61.45 MiB to 2.10 MiB over 30 sessions of 10 turns of 200 KiB. -- **Time startup on the built artifact, interleaved.** Alternate arms over at - least 10 runs on one build. An environment variable can stand in for an API +- **Time startup on the built artifact, interleaved.** Alternate arms on + one build, using the default run count in [the skill](../SKILL.md). An + environment variable can stand in for an API call, but note what it changes (it also reaches child processes). A V8 compile cache took a CLI's `--help` from a 554 ms median to 381 ms over 11 runs (M). - **Treat another researcher's numbers as leads.** Of three top candidates from a From 1675df1ed033adf7fcaff898748664fa6ff91376 Mon Sep 17 00:00:00 2001 From: bompus Date: Wed, 14 Oct 2026 19:02:11 -0600 Subject: [PATCH 2/4] docs(benchmarking): tie 'near the noise' to the variation criterion --- CHANGELOG.md | 2 +- skills/hr-benchmarking/SKILL.md | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d8d318a7..285c3e29 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ Notable changes to house-rules. Versions follow [semantic versioning](https://se - `hr-benchmarking` now gives a default stopping rule for performance claims, in place of "no universal minimum". Run 3 alternating runs per arm, and add 2 more per arm only when the arms' - ranges overlap or the effect is near the noise, for a cap of 5. Results always carry the run count + ranges overlap or the gap is within the variation criterion, for a cap of 5. Results always carry the run count and range, never a bare average. The expected wall-clock time is stated before the first run. The JavaScript reference no longer asks for at least 10 runs when timing startup. diff --git a/skills/hr-benchmarking/SKILL.md b/skills/hr-benchmarking/SKILL.md index 828146cf..1eae2956 100644 --- a/skills/hr-benchmarking/SKILL.md +++ b/skills/hr-benchmarking/SKILL.md @@ -122,8 +122,9 @@ Set the confirmation plan before its runs and update the explicit run count when the scope changes. The default stopping rule for a performance claim is 3 runs per arm. Add 2 more -per arm only when the arms' ranges overlap or the effect is near the noise, so -5 per arm is the cap. If 5 leave the comparison unresolved, report it as +per arm only when the arms' ranges overlap or the effect is near the noise, +meaning the gap is within the variation criterion set below. 5 per arm is the +cap. If 5 leave the comparison unresolved, report it as inconclusive rather than adding runs. Declare any other count before the first run, with the reason. Alternate the arms in every run so drift reaches each arm. From 5147acd412e8f00f9af7362e57cc04f740244e22 Mon Sep 17 00:00:00 2001 From: bompus Date: Wed, 14 Oct 2026 19:11:43 -0600 Subject: [PATCH 3/4] docs(benchmarking): limit other run counts to a user request or a too-long run --- skills/hr-benchmarking/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/skills/hr-benchmarking/SKILL.md b/skills/hr-benchmarking/SKILL.md index 1eae2956..7d1dbbcc 100644 --- a/skills/hr-benchmarking/SKILL.md +++ b/skills/hr-benchmarking/SKILL.md @@ -124,8 +124,9 @@ when the scope changes. The default stopping rule for a performance claim is 3 runs per arm. Add 2 more per arm only when the arms' ranges overlap or the effect is near the noise, meaning the gap is within the variation criterion set below. 5 per arm is the -cap. If 5 leave the comparison unresolved, report it as -inconclusive rather than adding runs. Declare any other count before the first +cap. If 5 runs per arm leave the comparison unresolved, report it as +inconclusive rather than adding runs. Use another count only when the user asks +for it or a run is too long to repeat 3 times, and declare it before the first run, with the reason. Alternate the arms in every run so drift reaches each arm. For a performance claim, assess run-to-run variation from those alternating runs. From d768b0e90e4d4e6afec23610f23ab24f061d3ffc Mon Sep 17 00:00:00 2001 From: bompus Date: Wed, 14 Oct 2026 19:18:55 -0600 Subject: [PATCH 4/4] docs(changelog): list the inconclusive outcome and the other-count rule --- CHANGELOG.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 285c3e29..590d1029 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,12 @@ Notable changes to house-rules. Versions follow [semantic versioning](https://se - `hr-benchmarking` now gives a default stopping rule for performance claims, in place of "no universal minimum". Run 3 alternating runs per arm, and add 2 more per arm only when the arms' - ranges overlap or the gap is within the variation criterion, for a cap of 5. Results always carry the run count - and range, never a bare average. The expected wall-clock time is stated before the first run. The - JavaScript reference no longer asks for at least 10 runs when timing startup. + ranges overlap or the gap is within the variation criterion, for a cap of 5. If 5 runs per arm + leave the comparison unresolved, it is reported as inconclusive. Another count is used only when + the user asks for it or a run is too long to repeat 3 times, and is declared with its reason + before the first run. Results always carry the run count and range, never a bare average. The + expected wall-clock time is stated before the first run. The JavaScript reference no longer asks + for at least 10 runs when timing startup. - The `hr-code-review` panel recommendations gain an entry for Muse Wonder Full 1.0 on the Bug Hunt board: two runs each at xhigh (41.5 of 105) and max (42.0), no published price, and reachable only