feat(benchmarks): full-roster cold re-run, and enforce the cold protocol in the driver - #3058
Merged
Merged
Conversation
…col in the driver The matrix mixes two regimes, and #1947 rule 2 says two vintages must never share a table: phases 1+2 89 models contended (hp-llm-worker live), N=2 + escalation phase 4 5 models cold, workers stopped, N=3, every cell +-0 The #3023 cold probe showed contention did not move scores -- a fully resident control reproduced [63,63] exactly, and an escalated spilling cell resolved to the majority the protocol had already chosen. So this is not a correction of wrong numbers. It makes the regime uniform, which matters because the top of the field is separated by ONE point across sixteen models, and a promotion decided on that margin cannot rest on a mixed protocol. N=2, not N=3. Part 4 found "22.1 of the 37.4 model-wall hours re-derived identical bytes" and concluded repeats belong on the axes that are NOT fixed. #3036 escalates 2 -> 3 -> 5 automatically wherever runs actually disagree, so genuine variance is caught without a third run on every deterministic cell -- about a third cheaper for the same information. sweep_extra.sh gains two things it should always have had: - STOP_WORKERS stops hp-llm-worker and ghidra-revdeck-1 for the run and restores them via a trap on EXIT/INT/TERM. #3023's finding was not that contention broke the scores, but that the protocol depended on an operator remembering; this moves it into the driver. The trap covers INT/TERM because this script is routinely killed between models. - KEEP_WEIGHTS_ABOVE_GB gates the post-model `ollama rm` on free space. That removal was correct at 92% full and is actively harmful with terabytes free: it destroyed all ten of the requant plan's source models, and it is the reason a cold re-run of 89 models has to re-download 80 of them. Results go to 1947cold/, not over 1947full/. The contended numbers are not garbage: contended-vs-cold across 89 models is the largest evidence anyone will have on whether the regime matters, and discarding it to save disk would repeat the mistake that lost the weights. chain_cold.sh waits on a POSITIVE condition -- every tag in models_requant.txt has both tier files -- rather than "the GPU is idle". chain_phase3.sh is already armed on the same card; two idle-checks would fire in the same poll window and double-book it. coldrun.sh keeps its own running-sweep guard as a second line, because the failure mode is a wasted day rather than an error message. Guards verified live: coldrun aborts while phase 2 runs, chain_cold aborts at 0/7 ladder tags, and the combined roster de-duplicates to 96 unique entries across the phase 1, phase 2 and ladder lists. Refs #1947, #3023, #2245, #3031, #3036, #2641
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The matrix mixes two regimes, and #1947 rule 2 says two vintages must never share
a table:
hp-llm-workerlive), N=2 + escalation±0The #3023 cold probe showed contention did not move scores — a fully resident
control reproduced
[63,63]exactly, and an escalated spilling cell resolved tothe majority the protocol had already chosen. So this is not a correction of wrong
numbers.
It makes the regime uniform, which matters because the top of the field is
separated by one point across sixteen models. A promotion decided on that margin
cannot rest on a mixed protocol.
N=2, not N=3
Part 4 found "22.1 of the 37.4 model-wall hours re-derived identical bytes" and
concluded repeats belong on the axes that are not fixed — seed, quant, prompt.
#3036 now escalates 2 → 3 → 5 automatically wherever runs actually disagree, so
genuine variance is still caught without paying a third run on every deterministic
cell. About a third cheaper for the same information.
sweep_extra.shgains two things it should always have hadSTOP_WORKERSstopshp-llm-workerandghidra-revdeck-1for the run andrestores them via a trap on
EXIT/INT/TERM. #3023's finding was not thatcontention broke the scores — it was that the protocol depended on an operator
remembering. This moves it into the driver. The trap covers
INT/TERMbecausethis script is routinely killed between models.
KEEP_WEIGHTS_ABOVE_GBgates the post-modelollama rmon free space. Thatremoval was correct at 92% full and is actively harmful with terabytes free: it
destroyed all ten of the requant plan's source models, and it is precisely why a
cold re-run of 89 models has to re-download 80 of them.
Results go to
1947cold/, not over1947full/The contended numbers are not garbage. Contended-vs-cold across 89 models is the
largest evidence anyone will ever have on whether the regime matters — discarding
it to save disk would repeat the mistake that lost the weights in the first place.
The chaining avoids a GPU race
chain_cold.shwaits on a positive condition — every tag inmodels_requant.txthas both tier files — rather than "the GPU is idle".chain_phase3.shis already armed on the same card; two idle-checks would fire inthe same poll window and double-book it.
coldrun.shkeeps its own running-sweepguard as a second line, because the failure mode here is a wasted day of GPU
rather than an error message.
Verified live
coldrun.shaborts while phase 2 is runningchain_cold.shaborts at 0/7 ladder tagsphase 2 and ladder lists (case-insensitively, since Ollama resolves names that
way and the rosters spell some quant tags differently — Two roster models are unpullable: Ollama rejects a model-name segment longer than 80 characters #2738)
bash -nclean on all threeRefs #1947, #3023, #2245, #3031, #3036, #2641