Skip to content

feat(benchmarks): full-roster cold re-run, and enforce the cold protocol in the driver - #3058

Merged
Xore merged 1 commit into
mainfrom
feat/cold-full-roster
Sep 6, 2026
Merged

Xore merged 1 commit into
mainfrom
feat/cold-full-roster

Conversation

@Xore

@Xore Xore commented Sep 5, 2026

Copy link
Copy Markdown
Owner

The matrix mixes two regimes, and #1947 rule 2 says two vintages must never share
a table:

models regime
phases 1+2 89 contended (hp-llm-worker live), N=2 + escalation
phase 4 5 cold, workers stopped, N=3, every cell ±0

The #3023 cold probe showed contention did not move scores — a fully resident
control reproduced [63,63] exactly, and an escalated spilling cell resolved to
the majority the protocol had already chosen. So this is not a correction of wrong
numbers.

It makes the regime uniform, which matters because the top of the field is
separated by one point across sixteen models
. A promotion decided on that margin
cannot rest on a mixed protocol.

N=2, not N=3

Part 4 found "22.1 of the 37.4 model-wall hours re-derived identical bytes" and
concluded repeats belong on the axes that are not fixed — seed, quant, prompt.
#3036 now escalates 2 → 3 → 5 automatically wherever runs actually disagree, so
genuine variance is still caught without paying a third run on every deterministic
cell. About a third cheaper for the same information.

sweep_extra.sh gains two things it should always have had

STOP_WORKERS stops hp-llm-worker and ghidra-revdeck-1 for the run and
restores them via a trap on EXIT/INT/TERM. #3023's finding was not that
contention broke the scores — it was that the protocol depended on an operator
remembering
. This moves it into the driver. The trap covers INT/TERM because
this script is routinely killed between models.

KEEP_WEIGHTS_ABOVE_GB gates the post-model ollama rm on free space. That
removal was correct at 92% full and is actively harmful with terabytes free: it
destroyed all ten of the requant plan's source models, and it is precisely why a
cold re-run of 89 models has to re-download 80 of them.

Results go to 1947cold/, not over 1947full/

The contended numbers are not garbage. Contended-vs-cold across 89 models is the
largest evidence anyone will ever have on whether the regime matters
— discarding
it to save disk would repeat the mistake that lost the weights in the first place.

The chaining avoids a GPU race

chain_cold.sh waits on a positive condition — every tag in
models_requant.txt has both tier files — rather than "the GPU is idle".
chain_phase3.sh is already armed on the same card; two idle-checks would fire in
the same poll window and double-book it. coldrun.sh keeps its own running-sweep
guard as a second line, because the failure mode here is a wasted day of GPU
rather than an error message.

Verified live

Refs #1947, #3023, #2245, #3031, #3036, #2641

…col in the driver

The matrix mixes two regimes, and #1947 rule 2 says two vintages must never
share a table:

  phases 1+2   89 models   contended (hp-llm-worker live), N=2 + escalation
  phase 4       5 models   cold, workers stopped, N=3, every cell +-0

The #3023 cold probe showed contention did not move scores -- a fully resident
control reproduced [63,63] exactly, and an escalated spilling cell resolved to
the majority the protocol had already chosen. So this is not a correction of
wrong numbers. It makes the regime uniform, which matters because the top of the
field is separated by ONE point across sixteen models, and a promotion decided on
that margin cannot rest on a mixed protocol.

N=2, not N=3. Part 4 found "22.1 of the 37.4 model-wall hours re-derived
identical bytes" and concluded repeats belong on the axes that are NOT fixed.
#3036 escalates 2 -> 3 -> 5 automatically wherever runs actually disagree, so
genuine variance is caught without a third run on every deterministic cell --
about a third cheaper for the same information.

sweep_extra.sh gains two things it should always have had:

- STOP_WORKERS stops hp-llm-worker and ghidra-revdeck-1 for the run and restores
  them via a trap on EXIT/INT/TERM. #3023's finding was not that contention broke
  the scores, but that the protocol depended on an operator remembering; this
  moves it into the driver. The trap covers INT/TERM because this script is
  routinely killed between models.
- KEEP_WEIGHTS_ABOVE_GB gates the post-model `ollama rm` on free space. That
  removal was correct at 92% full and is actively harmful with terabytes free: it
  destroyed all ten of the requant plan's source models, and it is the reason a
  cold re-run of 89 models has to re-download 80 of them.

Results go to 1947cold/, not over 1947full/. The contended numbers are not
garbage: contended-vs-cold across 89 models is the largest evidence anyone will
have on whether the regime matters, and discarding it to save disk would repeat
the mistake that lost the weights.

chain_cold.sh waits on a POSITIVE condition -- every tag in models_requant.txt
has both tier files -- rather than "the GPU is idle". chain_phase3.sh is already
armed on the same card; two idle-checks would fire in the same poll window and
double-book it. coldrun.sh keeps its own running-sweep guard as a second line,
because the failure mode is a wasted day rather than an error message.

Guards verified live: coldrun aborts while phase 2 runs, chain_cold aborts at
0/7 ladder tags, and the combined roster de-duplicates to 96 unique entries
across the phase 1, phase 2 and ladder lists.

Refs #1947, #3023, #2245, #3031, #3036, #2641
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@Xore
Xore merged commit f4fde14 into main Sep 6, 2026
105 checks passed
@Xore
Xore deleted the feat/cold-full-roster branch September 6, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant