bug(bench): bench-compare base-then-head ordering fabricates large false regressions under thermal load

[ohdearquant]

Summary

make bench-compare (scripts/bench-compare.sh) builds and benches BASE first, then HEAD second, in the same process on the same machine. The second half runs on a CPU already thermally saturated by the first half's full Criterion sweep, so it throttles and every HEAD bench inflates uniformly. This fabricates large one-sided "regressions" that are pure artifacts of run order, not code.

Reproduction (null A/B — identical source both sides)

scripts/bench-compare.sh origin/main origin/main   # same commit vs itself, zero diff

Result on an idle M2 Max (all other jobs quiesced), 2026-07-08:

• 56 FAIL / 12 WARN, worst simd_cosine_similarity/simd/384 +378.87% (CI [+371.58%, +386.27%]).
• 42 co-reported "confirmed improvements" from the same zero-diff comparison.

Since BASE and HEAD are the byte-identical commit, all 56 "regressions" are instrument artifacts. For reference, a real PR (#713, whose diff is entirely #[cfg(feature = "bench-internals")]-gated and cannot affect these benches) reported fewer — 35 FAIL — i.e. a genuine no-op change scores better than the tool comparing main to itself.

Impact

bench-compare output is currently uninterpretable as a perf gate on this machine: the false-positive floor (56 FAIL from nothing) swamps any real signal. Every perf PR that pastes bench-compare output is affected, and reviewers cannot distinguish real regressions from thermal noise.

Fix directions

1. Interleave the two refs at the per-group level (A/B/A/B) instead of all-BASE-then-all-HEAD, so thermal drift lands symmetrically (this is the same mitigation the GDN perf(inference): chunked GDN prefill scan (B=64 to 128, simdgroup_matrix) #175 harness adopted with its --interleaved mode).
2. Insert a cooldown between the BASE and HEAD halves and/or a longer warmup on the second half.
3. Randomize/alternate which ref runs first and report order-sensitivity.

Until fixed, treat bench-compare deltas as advisory only; disprove any flagged regression with a same-ref null A/B or a same-binary re-run.

[ohdearquant]

New quantitative sample of this noise class, from PR #781's review cycle — the first time we have TWO same-source, same-compiler quick-mode runs on identical refs to compare directly.

Setup: RUSTUP_TOOLCHAIN=1.94.1 make bench-compare (quick mode), both refs forced to the same toolchain, identical benched code both runs (the diff between refs touches no benched code path), runs minutes-to-tens-of-minutes apart under varying background load:

• Run A: 19 FAIL / 39 WARN / 6 improvement
• Run B: 4 FAIL / 16 WARN / 53 improvement

Localization identical in both: every FAIL/WARN confined to lattice-embed SIMD/cosine/euclidean groups (plus a single one-run-only gelu/4096 +7.57% WARN); inference groups otherwise clean in both. Dozens of embed entries flip sign between runs — run-A regressions re-measure as run-B improvements on identical binaries-modulo-noise.

Implication for this issue: quick-mode variance in those embed SIMD groups dominates any real signal, so quick-mode FAIL counts there carry near-zero information. Two candidate fixes this data seeds: (a) widen quick-mode confidence intervals (or raise the FAIL threshold) for the known-noisy embed SIMD groups, or (b) exclude those groups from quick-mode gating and only judge them in --full runs. Full outputs are in PR #781's description (same-toolchain runs section).