docs(benchmarks): #1805-c ghidra-slot matrix — cold-protocol re-measure, and promote gemma-4-26B-A4B by Xore · Pull Request #2641 · Xore/APIARY

Xore · 2026-08-29T16:22:14Z

Harvests the 2026-08-27/28 ghidra-slot benchmark from the analysis host (which had finished uncollected), finds a measurement defect that invalidates its headline comparison, re-measures the decision cohort under a controlled protocol, and records part 4 of the #1947 rebuild against the corrected numbers.

Adds the rewritten part 4 to docs/local-llm-model-evaluation.md, two machine-readable matrices under docs/benchmarks/matrices/, and 32 pinned transcript run directories.

The defect

Ollama serves a request at the context and state of whatever model instance is resident, and record_baseline.py neither sets that nor records it — it stamps context_tokens: 8192 and keep_alive: "10m" into every report and sends neither (#2644). Two measured consequences:

hp-llm-worker holds qwen3:14b resident at CONTEXT 8192, so the incumbent was the one row measured against a different instance than everything it was compared with. Observed live, same tag and digest, minutes apart: 10 GB / 8192 worker-loaded vs 14 GB / 32768 benchmark-loaded.
A warm slot does not reproduce; a cold one does. Controlled A/B on qwen3:14b, everything else fixed: warm 60/61 with 1 of 14 answers byte-identical; cold (ollama stop between runs) 60/60 with 14 of 14 (benchmarks: repeat runs are byte-identical since the Aug-25 anchor — #1947's N>=3 protocol and its ±0.58 noise band rest on an unexplained change #2642).

That also explains the first pass's uniform ±0, which I initially reported as the harness being deterministic and have since retracted: three lanes against OLLAMA_MAX_LOADED_MODELS=1 evicted each other constantly, so nearly every request began with a reload. Lane contention masquerading as determinism — which also inflated wall clock up to 15× (62.5 min vs 4.1 min for identical work), making the original min/run column unusable as a speed measure.

Cohort re-run, cold protocol — authoritative

Sequential, evict before every run, live workers stopped, N=3, all cells ±0 across four architectures:

Model	Tier A	Tier B	Tier A injection	min/run B
Ornith-1.0-35B-heretic	63 ±0	67 ±0	FAIL	6.7
gemma-4-26B-A4B-heretic-i1	61 ±0	64 ±0	clean	2.1
`qwen3:14b` — incumbent	60 ±0	58 ±0	clean	3.8
`qwen2.5-coder:7b` — #159 anchor	56 ±0	58 ±0	clean	2.1

Which rows moved confirms the diagnosis. Ornith and gemma-4-26B reproduced their survey Tier B scores exactly (67, 64) — nothing else loads them. Only qwen3:14b moved (62→60, 61→58), the one model the worker keeps resident.

Decision: promote (changed from "no promotion")

The earlier call rested on a +3 gap against a contaminated incumbent row. The incumbent's real Tier B is 58, so gemma-4-26B-A4B leads by +6 at ±0, wins Tier A too, passes every gate including Tier A injection, and is the fastest in the cohort. Ornith scores 67 but its Tier A injection failure reproduced under the clean protocol and it stays disqualified — score does not buy past that gate.

Conditions recorded in the doc: ghidra slot only; Tier-A-only injection verdict (#2643 — Tier B has zero injection coverage, the payload never survives into Ghidra pseudocode); pin tag and digest; land #2644 first.

Other findings filed

benchmarks: repeat runs are byte-identical since the Aug-25 anchor — #1947's N>=3 protocol and its ±0.58 noise band rest on an unexplained change #2642 — cold/warm reproducibility, mechanism established.
benchmarks: ghidra Tier B has zero injection-gate coverage (payload never survives into Ghidra pseudocode); Tier A fails 4 of 12 models #2643 — Tier B injection coverage is zero on every cell; Tier A fails 4 of 12 models.
record_baseline.py: no retry/incremental save (one ollama restart cost 10 cells), and context_tokens/keep_alive are pinned in reports but never sent #2644 — no retry / no incremental save (one container restart cost 10 cells), plus the unsent num_ctx/keep_alive pins.
llm-worker/revdeck run against a warm Ollama slot, where identical requests are not reproducible (14/14 identical cold vs 1/14 warm) #2646 — llm-worker and revdeck run warm by design, so production answers are not reproducible. Bigger than the benchmark; needs its own investigation.
GPU guides' keep-alive and G6 resource rules diverge from the canonical compose files they name #2365 — cross-linked, this adds a fourth keep-alive value to its divergence list.

gemma-4-31B is deferred and running last (~40 min/run, fails the injection gate, cannot change the decision); its row lands when it completes.

Docs-only; no code paths touched. analysis/ghidra/benchmarks/tests/ 117 passed, including the transcript provenance guard.

github-actions · 2026-08-29T16:22:30Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Xore · 2026-08-29T16:45:29Z

Converted to draft — do not merge as-is. The determinism section is now falsified by measurement.

Re-running the incumbent on a quiet box (live workers stopped): qwen3:14b Tier A scores 62 (stored) / 60 / 61, with only 1 of 14 answers byte-identical between two consecutive fresh runs, on identical input evidence. The harness is not deterministic; the original round's uniform ±0 is an artifact of its own three-lane contention, not a property of the code.

Three things in the current diff need rewriting before this lands:

The "Repeats measured nothing" section — inverted. N>=3 is doing its job; the ±0 is the anomaly.
The min/run column — 62.5 min → 4.1 min for identical work once lane contention was removed, so it is a contention artifact and cannot carry the "fastest in the matrix" argument for gemma-4-26B. That needs re-basing on part 2's separate tok/s sweep.
The N=1 rows — being backfilled to N=3 right now, which also makes the incumbent's Tier B number (the one the promote call turns on) a measured range rather than a single contaminated point.

Detail on #2642. Will update the branch and mark ready once the backfill completes and the cold-slot/warm-slot test has run.

Xore · 2026-08-30T01:07:39Z

Holding this PR — do not merge yet. Xore asked whether Tier A is really safe given the code moved, and it is not. I checked the evidence provenance (manifest.json, rubric — both unchanged) and stopped there. I should have diffed the scoring path. Three commits between the clone HEAD these numbers were measured at (22f01c2) and current main change scoring behaviour:

commit	change	invalidates
`d16a64be` (#2517)	snake_case identifiers exempt from rubric forbidden matching	every `safe_strcpy` false-positive trip in this matrix
`55b15c47` (#2618)	injection gate matches the actual response, three-tier scoring	`process_and_injection` score for every model
`26e0e89b` (#2638)	polarity.py negation residue	forbidden matching generally

#2517's own commit message names safe_strcpy / vulnerable_strcpy as the false positive it fixes — which is exactly the gate trip this PR reports for Ornith, observerx, qwen3:8b and huihui-qwen3.8-27b. Those trips would not fire under current code.

So both tiers were measured on a superseded scorer, not just Tier B on a stale corpus. Publishing this table as a decision record would publish numbers current main cannot reproduce — the same defect that forced the twelve-model pass to be demoted to a survey.

Plan: corpus rebuild + Tier B cache re-extraction are already running; I am restarting the full roster on current main (both tiers). This PR gets updated with numbers the code actually produces, then merged. The structural findings in it — cold vs warm reproducibility, the contaminated-incumbent diagnosis, the Tier A/Tier B family split, the harness defects — are unaffected and all still stand; it is the specific cell values and gate verdicts that need re-measuring.

… resume plan The sweep's clone at /mnt-1/benchmarks/APIARY sits on a detached HEAD at a99e765, and every completed run writes its transcript dir into docs/benchmarks/runs/ as an untracked file. As of today that is 127 run dirs, 24 MB, all from 2026-09-04/05, on no branch, committed nowhere, in the one directory the rebuild already wiped once. Per #1944 the transcripts are the deliverable -- "no transcripts" is reason 4 of the six that made this epic necessary, and #2266's offline re-scoring reads them. Losing them turns every phase-2 row back into a number nobody can check. They must NOT be committed in that clone: a commit moves HEAD off a99e765 and resume_phases.sh hard-aborts on exactly that string, so the pin is the guard. Mirrored to the workstation instead, and the plan now carries the rsync, the reason the obvious fix is wrong, and the note that they get committed properly at write-up time on a branch cut from main. Also names PR #2641's branch in the ledger so the phase-4 results and the running phase-2 sweep are not read as the same work. Refs #1947, #1944, #2266, #2971

…uences Answered 2026-09-05: - #2245 is scored by record_baseline.py on the a99e765 pin, Ollama-served, so self-quant rows land in the same matrix as their as-published twins. corpus_eval.py + llama-server is not used for this issue. - #2245 runs the full clean f16 ladder, not the cheap requant probe. - PR #2641 stays DRAFT and gets regenerated from the completed run, so #1805 and the gemma-4-26B-A4B ghidra promotion stay parked until the matrix is whole. Its branch is now a synthesis input -- do not delete it. - The two free DIMM slots get filled before phase 3. Two of those create prerequisites, which reorders the plan: phase 3 is gated on a RAM install and an HF token, so phase 5 (slots) moves ahead of it and the short jobs fill the gaps. The f16 ladder's toolchain turns out not to be a blocker at all: ghcr.io/ggml-org/llama.cpp:full pulls clean on the homeserver and carries convert_hf_to_gguf.py, llama-quantize and llama-server in one image, so "stand up llama.cpp" is a docker pull rather than a build, with no host package installs. Verified live. What remains is the HF token. The RAM finding is bigger than headroom. dmidecode shows DIMME1 and DIMMF1 empty, i.e. channels E and F entirely unpopulated -- the box runs 4-channel on a 6-channel Gold 5220R. CPU-offloaded inference is memory-bandwidth bound, not thread bound (llama-server sits at ~16 of 48 cores while 65% of the 218B is on the CPU), so filling those two slots is a bandwidth change on exactly the rows this decision cares about. Part number and the ECC/registered caveat recorded. Refs #1947, #2245, #1805, #2641

…the probe Phase 2 ran without the cold protocol's worker-stop step: hp-llm-worker spent ~11 hours issuing a competing /api/chat every ~4 minutes against the same OLLAMA_MAX_LOADED_MODELS=1 slot (#2582 recurring, #3023). Phase 2 carries 12 escalations against phase 4's uniformly +-0 cold cells, which looked like the contamination that demoted #1805-c to a survey. Measured it instead of re-running two days of GPU on the suspicion: DeepHat-V1-7B:Q4_K_M 4.7 GB resident A contended [63,63] cold [63,63] ravenx-cyberagent-35b 21 GB spills B contended [64,63,64] cold [64,64,64] The resident control is load-bearing -- no CPU-offload nondeterminism is possible there, so contention would have to show up in it, and it reproduced exactly. The escalated cell resolved to the majority the N=3 protocol had already chosen. Verdict: keep phase 2, annotate it, do not re-run. Escalations cost GPU time, not accuracy. Checking all nine escalated cells surfaced a separate defect: eight resolve 2-of-3, but GLM-4.6-REAP-218B Tier B is [55, 57, 54] -- three distinct values, so N=3 cannot produce a majority and that cell has no defensible score. It must not be published as any of the three. sweep_extra.sh escalates once and has no handling for "still no majority"; that wants an UNRESOLVED marker and an N>=5 path. Commits both drivers, per #2985's rule that a script driving real benchmark runs must not live only on the host: - coldprobe.sh asserts its preconditions rather than assuming them (sweep stopped, no run in flight, hp-llm-worker down, tierb-cache present, head == a99e765) and writes to a separate directory so no roster row is polluted. - probe_at_gap.sh takes the next natural gap: it stops only the sweep's driver, never an in-flight record_baseline, because do_run's own `rm -f` cleanup is skipped when the parent dies -- which is how the 2026-08-31 abort left a valid-looking partial result carrying 11 of 14 cases. Also records the #3031 DNS incident: a container resolver predating #2974's host fix still had 1.1.1.1/8.8.8.8 first, port 53 to them is blocked here, so every lookup burned ~8s against ollama's 30s pull deadline -- 12 models lost in four minutes, followed by a false EXTRA_COMPLETE. Refs #1947, #3023, #3031, #2582, #2641, #2985, #2974

…sion (#3014) * docs(benchmarks): #1947 resume plan after the CPU swap and disk expansion The sweep's work area has been destroyed once (#2971) and mis-restored once, and three of its phase scripts were lost because they were never committed (#2985). A resume plan that lives only in /mnt-1/benchmarks has the same failure mode, so it goes in the repo. Records, from live inspection on 2026-09-05: - the hardware delta that prompted this — Xeon Silver 4110 -> Gold 5220R (8c/16t -> 24c/48t) and /var 153 G free -> 6.5 T free. RAM is unchanged at 6x16 GiB with two slots empty, and is now the only remaining hardware limit. - the completion ledger: phase 1 complete, phase 2 at 36/52 both tiers with 16 unstarted (5 of them unpullable, itemised with the call for each), phases 2.5/3/5 blocked on #2985, phase 4 sitting in draft PR #2641. - what the new hardware actually buys: `sweep_extra.sh`'s unconditional `ollama rm` is now harmful rather than necessary, and the CPU swap's own A/B has never been measured against the pre-swap timing table it was recorded for. - an ordered resume path P0-P8 with the owning issue for each step, the eight rules already paid for in wasted hours (pin a99e765 / 14 cases / max 69, no mixed scoring vintages, 0 != empty, cold slot, uniform +-0 is a symptom, injection axis reads as coverage-not-resistance, verify the benchmark leg and not the pull, restore scripts from the repo), a runbook, and four decisions that need an operator. Refs #1947, #2985, #2971, #2279, #2245, #2969, #1805, #1804 * ops(ci): cut honeypot-ci runner instances from seven to two The homeserver is the GPU host for the #1947 model benchmark as well as the CI box, and CI is that benchmark's largest source of measurement noise: identical work on the same model spread 62% (78.2 vs 126.7 min/run) purely from runner contention, with load average sitting at 15-27 under seven instances. #1947's premise is that a delta smaller than the error bar is not a result, so a run taken against an unknown CI-driven load is not a measurement -- and the resume plan's CPU-swap A/B (step P5) cannot be answered at all without quiescing it. supermicro-ci-3..7 are `systemctl disable --now` on the host, not deregistered: registration, _work dir and tool cache survive, so restoring them is one command and needs no token. supermicro + supermicro-ci-2 stay. The separate honeypot-home deploy runner is untouched -- different label, no redundancy behind it. Seven was never a measured optimum, only the ceiling #2572 allowed once one instance stopped serialising the matrix; two instances on the Gold 5220R (24c/48t) have more cores behind them than four had on the Silver 4110. Documents the count and the restore command in docs/CI-CD.md, and records in the resume plan that the pre-swap baseline was itself taken under seven instances -- so the post-swap side is the quieter half of that comparison and uptime has to be recorded per run either way. Refs #1947, #2572 * docs(benchmarks): record the phase-2 transcript durability gap in the resume plan The sweep's clone at /mnt-1/benchmarks/APIARY sits on a detached HEAD at a99e765, and every completed run writes its transcript dir into docs/benchmarks/runs/ as an untracked file. As of today that is 127 run dirs, 24 MB, all from 2026-09-04/05, on no branch, committed nowhere, in the one directory the rebuild already wiped once. Per #1944 the transcripts are the deliverable -- "no transcripts" is reason 4 of the six that made this epic necessary, and #2266's offline re-scoring reads them. Losing them turns every phase-2 row back into a number nobody can check. They must NOT be committed in that clone: a commit moves HEAD off a99e765 and resume_phases.sh hard-aborts on exactly that string, so the pin is the guard. Mirrored to the workstation instead, and the plan now carries the rsync, the reason the obvious fix is wrong, and the note that they get committed properly at write-up time on a branch cut from main. Also names PR #2641's branch in the ledger so the phase-4 results and the running phase-2 sweep are not read as the same work. Refs #1947, #1944, #2266, #2971 * benchmarks: stop losing #2245's inputs — re-create the VRAM sampler, keep-alias pulled weights Checked #2245 against the live host. Four of its premises no longer hold: - llama-quantize IS present (/usr/lib/ollama/llama-quantize in ghidra-ollama-1, with --allow-requantize), so the requant-from-GGUF path needs nothing new -- but there is no llama.cpp checkout anywhere on the box, so convert_hf_to_gguf.py does not exist and the rex86-eval container model-quant-benchmark/README.md assumes is not running. The issue's "harness we already have, this is applied not built" describes the repo, not the host. - vram_samples.tsv, the empirical spill list the requant plan rests on, was wiped with the work area, is in no mirror, and is NOT re-derivable: the result JSONs record no VRAM, no served size and no CPU/GPU split. - All ten source models named in requant_plan.txt have been deleted by sweep_extra.sh's `ollama rm`. One survives under a #2738 alias; at least three of the rest cannot be re-pulled at all. - The scorer is undecided: step 3 says corpus_eval.py via llama-server, step 4 demands comparison against rows scored by record_baseline.py on a99e765. Two scorers, incomparable numbers -- reason 6 of the six behind #1947. keep_and_sample.sh runs beside the sweep and changes nothing about it. It samples `ollama ps` back into vram_samples.tsv (served size, CPU/GPU split and served context exist only while a model is loaded -- sample or lose), and gives every pulled tag a keep/<slug>:src alias. `ollama cp` makes a second manifest over the SAME blobs -- verified, cp then rm of the copy left the blob count unchanged at 39 -- so the sweep's rm drops only its own manifest and the weights survive at zero extra disk. Deliberately not an edit to sweep_extra.sh: bash reads a running script lazily by byte offset, and the alias names cannot collide with a roster entry, so the sweep's own "already local -> will not delete" test is unaffected either way. First datum back: GLM-4.6-REAP-218B:i1-IQ1_S is 44 GB on disk, 57 GB served at ctx 32768, 65%/35% CPU/GPU -- confirming the KV arithmetic and showing served size cannot be inferred from disk size. Refs #2245, #2985, #2971, #1947 * docs(benchmarks): record the four operator decisions and their consequences Answered 2026-09-05: - #2245 is scored by record_baseline.py on the a99e765 pin, Ollama-served, so self-quant rows land in the same matrix as their as-published twins. corpus_eval.py + llama-server is not used for this issue. - #2245 runs the full clean f16 ladder, not the cheap requant probe. - PR #2641 stays DRAFT and gets regenerated from the completed run, so #1805 and the gemma-4-26B-A4B ghidra promotion stay parked until the matrix is whole. Its branch is now a synthesis input -- do not delete it. - The two free DIMM slots get filled before phase 3. Two of those create prerequisites, which reorders the plan: phase 3 is gated on a RAM install and an HF token, so phase 5 (slots) moves ahead of it and the short jobs fill the gaps. The f16 ladder's toolchain turns out not to be a blocker at all: ghcr.io/ggml-org/llama.cpp:full pulls clean on the homeserver and carries convert_hf_to_gguf.py, llama-quantize and llama-server in one image, so "stand up llama.cpp" is a docker pull rather than a build, with no host package installs. Verified live. What remains is the HF token. The RAM finding is bigger than headroom. dmidecode shows DIMME1 and DIMMF1 empty, i.e. channels E and F entirely unpopulated -- the box runs 4-channel on a 6-channel Gold 5220R. CPU-offloaded inference is memory-bandwidth bound, not thread bound (llama-server sits at ~16 of 48 cores while 65% of the 218B is on the CPU), so filling those two slots is a bandwidth change on exactly the rows this decision cares about. Part number and the ECC/registered caveat recorded. Refs #1947, #2245, #1805, #2641 * benchmarks: measure whether contention corrupted phase 2, and commit the probe Phase 2 ran without the cold protocol's worker-stop step: hp-llm-worker spent ~11 hours issuing a competing /api/chat every ~4 minutes against the same OLLAMA_MAX_LOADED_MODELS=1 slot (#2582 recurring, #3023). Phase 2 carries 12 escalations against phase 4's uniformly +-0 cold cells, which looked like the contamination that demoted #1805-c to a survey. Measured it instead of re-running two days of GPU on the suspicion: DeepHat-V1-7B:Q4_K_M 4.7 GB resident A contended [63,63] cold [63,63] ravenx-cyberagent-35b 21 GB spills B contended [64,63,64] cold [64,64,64] The resident control is load-bearing -- no CPU-offload nondeterminism is possible there, so contention would have to show up in it, and it reproduced exactly. The escalated cell resolved to the majority the N=3 protocol had already chosen. Verdict: keep phase 2, annotate it, do not re-run. Escalations cost GPU time, not accuracy. Checking all nine escalated cells surfaced a separate defect: eight resolve 2-of-3, but GLM-4.6-REAP-218B Tier B is [55, 57, 54] -- three distinct values, so N=3 cannot produce a majority and that cell has no defensible score. It must not be published as any of the three. sweep_extra.sh escalates once and has no handling for "still no majority"; that wants an UNRESOLVED marker and an N>=5 path. Commits both drivers, per #2985's rule that a script driving real benchmark runs must not live only on the host: - coldprobe.sh asserts its preconditions rather than assuming them (sweep stopped, no run in flight, hp-llm-worker down, tierb-cache present, head == a99e765) and writes to a separate directory so no roster row is polluted. - probe_at_gap.sh takes the next natural gap: it stops only the sweep's driver, never an in-flight record_baseline, because do_run's own `rm -f` cleanup is skipped when the parent dies -- which is how the 2026-08-31 abort left a valid-looking partial result carrying 11 of 14 cases. Also records the #3031 DNS incident: a container resolver predating #2974's host fix still had 1.1.1.1/8.8.8.8 first, port 53 to them is blocked here, so every lookup burned ~8s against ollama's 30s pull deadline -- 12 models lost in four minutes, followed by a false EXTRA_COMPLETE. Refs #1947, #3023, #3031, #2582, #2641, #2985, #2974 * docs(benchmarks): fix a doc-path lint false positive in the resume plan 'the ~16 MB of scripts/roster' read as a citation of a tracked path 'scripts/roster', which does not exist. It was prose, not a path -- rephrased rather than allowlisted, since allowlisting a non-path would weaken the #2458 check for a real one later. Refs #1947, #2458

…col in the driver (#3058) The matrix mixes two regimes, and #1947 rule 2 says two vintages must never share a table: phases 1+2 89 models contended (hp-llm-worker live), N=2 + escalation phase 4 5 models cold, workers stopped, N=3, every cell +-0 The #3023 cold probe showed contention did not move scores -- a fully resident control reproduced [63,63] exactly, and an escalated spilling cell resolved to the majority the protocol had already chosen. So this is not a correction of wrong numbers. It makes the regime uniform, which matters because the top of the field is separated by ONE point across sixteen models, and a promotion decided on that margin cannot rest on a mixed protocol. N=2, not N=3. Part 4 found "22.1 of the 37.4 model-wall hours re-derived identical bytes" and concluded repeats belong on the axes that are NOT fixed. #3036 escalates 2 -> 3 -> 5 automatically wherever runs actually disagree, so genuine variance is caught without a third run on every deterministic cell -- about a third cheaper for the same information. sweep_extra.sh gains two things it should always have had: - STOP_WORKERS stops hp-llm-worker and ghidra-revdeck-1 for the run and restores them via a trap on EXIT/INT/TERM. #3023's finding was not that contention broke the scores, but that the protocol depended on an operator remembering; this moves it into the driver. The trap covers INT/TERM because this script is routinely killed between models. - KEEP_WEIGHTS_ABOVE_GB gates the post-model `ollama rm` on free space. That removal was correct at 92% full and is actively harmful with terabytes free: it destroyed all ten of the requant plan's source models, and it is the reason a cold re-run of 89 models has to re-download 80 of them. Results go to 1947cold/, not over 1947full/. The contended numbers are not garbage: contended-vs-cold across 89 models is the largest evidence anyone will have on whether the regime matters, and discarding it to save disk would repeat the mistake that lost the weights. chain_cold.sh waits on a POSITIVE condition -- every tag in models_requant.txt has both tier files -- rather than "the GPU is idle". chain_phase3.sh is already armed on the same card; two idle-checks would fire in the same poll window and double-book it. coldrun.sh keeps its own running-sweep guard as a second line, because the failure mode is a wasted day rather than an error message. Guards verified live: coldrun aborts while phase 2 runs, chain_cold aborts at 0/7 ladder tags, and the combined roster de-duplicates to 96 unique entries across the phase 1, phase 2 and ladder lists. Refs #1947, #3023, #2245, #3031, #3036, #2641

* feat(benchmarks): full-roster cold re-run, and enforce the cold protocol in the driver The matrix mixes two regimes, and #1947 rule 2 says two vintages must never share a table: phases 1+2 89 models contended (hp-llm-worker live), N=2 + escalation phase 4 5 models cold, workers stopped, N=3, every cell +-0 The #3023 cold probe showed contention did not move scores -- a fully resident control reproduced [63,63] exactly, and an escalated spilling cell resolved to the majority the protocol had already chosen. So this is not a correction of wrong numbers. It makes the regime uniform, which matters because the top of the field is separated by ONE point across sixteen models, and a promotion decided on that margin cannot rest on a mixed protocol. N=2, not N=3. Part 4 found "22.1 of the 37.4 model-wall hours re-derived identical bytes" and concluded repeats belong on the axes that are NOT fixed. #3036 escalates 2 -> 3 -> 5 automatically wherever runs actually disagree, so genuine variance is caught without a third run on every deterministic cell -- about a third cheaper for the same information. sweep_extra.sh gains two things it should always have had: - STOP_WORKERS stops hp-llm-worker and ghidra-revdeck-1 for the run and restores them via a trap on EXIT/INT/TERM. #3023's finding was not that contention broke the scores, but that the protocol depended on an operator remembering; this moves it into the driver. The trap covers INT/TERM because this script is routinely killed between models. - KEEP_WEIGHTS_ABOVE_GB gates the post-model `ollama rm` on free space. That removal was correct at 92% full and is actively harmful with terabytes free: it destroyed all ten of the requant plan's source models, and it is the reason a cold re-run of 89 models has to re-download 80 of them. Results go to 1947cold/, not over 1947full/. The contended numbers are not garbage: contended-vs-cold across 89 models is the largest evidence anyone will have on whether the regime matters, and discarding it to save disk would repeat the mistake that lost the weights. chain_cold.sh waits on a POSITIVE condition -- every tag in models_requant.txt has both tier files -- rather than "the GPU is idle". chain_phase3.sh is already armed on the same card; two idle-checks would fire in the same poll window and double-book it. coldrun.sh keeps its own running-sweep guard as a second line, because the failure mode is a wasted day rather than an error message. Guards verified live: coldrun aborts while phase 2 runs, chain_cold aborts at 0/7 ladder tags, and the combined roster de-duplicates to 96 unique entries across the phase 1, phase 2 and ladder lists. Refs #1947, #3023, #2245, #3031, #3036, #2641 * fix(dns): publish Technitium cluster port 53443 on the secondary The homeserver node published only 53/tcp, 53/udp and 5380/tcp. The cluster replication port was never published -- not in this file nor in any backup of it -- so the primary could not reach the secondary and reported Error! No route to host (supermicro.cluster.xore.lan:53443) and replication was silently dead: settings changed on the primary never reached this node. "No route to host" rather than "connection refused" because firewalld REJECTs it; 53 and 5380 are reachable only because Docker's DNAT/FORWARD rules bypass the firewalld zone, whose public zone allows just cockpit, dhcpv6-client and ssh. The image merely EXPOSEs 53443, and an exposed-but-unpublished port never binds on the host in bridge mode. The asymmetry hid it: the primary runs with network_mode: host, so its own 53443 listens on all interfaces and the reverse direction looked healthy. Publishing through Docker bypasses firewalld the same way the existing ports do, so no zone change is needed. Verified after recreating the container: 53443 accepts connections from the primary, TLSv1.3 handshake completes, and the secondary picked up the primary's state within 15 seconds -- updates.maxmind.com went from NXDOMAIN to NOERROR there, which in turn cleared hp-geoipupdate (healthy, FailingStreak=0). Closes #3066

Xore · 2026-09-07T08:48:10Z

Round-7 chain progress — new rows (2026-09-06 14:18 → 2026-09-07 08:46)

Cold-protocol chain (round7_coldrun.sh, sequential, evict-before-run, 2 runs/cell) has completed 42 models; currently running the next roster entry. Newly completed rows since the last update:

Model	Tier A	Tier B
`qwen3.5:4b`	74/74	67/67
`codellama:7b-instruct-q4_K_M`	—	—
`SecurityLLM 7B Q4_K_M`	—	—
`Lily-Cybersecurity-7B v0.2 Q4_K_M`	—	—
`DeepHat-V1-7B-Heretic-Ablit i1-Q4_K_S`	—	—
`Foundation-Sec-1.1-8B i1-Q4_K_S`	—	—
`WhiteRabbitNeo-2.5-Qwen-2.5-Coder-7B Q4_K_M`	—	—
`Dolphin3-Cyber-8B Q4_K_M`	—	—
`qwen2.5-coder:14b`	—	—
`gemma2:27b`	—	—
`Mistral-Small-3.2-24B ultra-uncensored Q4_K_M`	—	—
`GPT-OSS-Cybersecurity-20B-Merged i1-Q4_K_M`	—	—
`gemma-4-26B-A4B ultra-uncensored-heretic i1-Q4_K_S`	—	—
`gemma-4-26B-A4B ultra-uncensored-heretic q4_k_s`	—	—
`Huihui-Qwen3.6-27B-abliterated Q4_K_M`	—	—
`qwen3.8:27b`	—	—
`Qwen3.6-27B-uncensored-heretic-v2-MTP Q4_K_M`	—	—
`gemma-4-31B-it-qat-q4_0-uncensored-heretic Q4_0`	71/71	73/73
`Seneca-Cybersecurity-LLM-x-QwQ-32B Q4_Medium`	73/73	73/73
`Ornith-1.0-35B-uncensored-heretic Q4_K_M`	71/71	77/77
`Qwen3.6-27B-Fable-Fusion-711-MTP Q4_K_M`	71/71	69/69
`Qwen3.5-9B-The-Defiant-Fable iMatrix-MTP Q4_K_M`	74/74	67/67
`Qwen3.8-27B-Unleashed Q4_K_M`	71/71	73/73
`Ornith-1.5-9B-UNCENSORED Q4_K_M`	70/70	65/65
`GLM-4.7-Flash-Reasoning-Distill Q4_K_M`	76/76	73/73
`Huihui-CyberStrike-OffSec-35B-abliterated Q6_K`	74/74	77/77
`Qwen3.8-27B-Uncensored-Cyber Q4_K_M`	78/78	73/73

All cells reproducible (run1 == run2). Note Qwen3.8-27B-Uncensored-Cyber posts the highest Tier A of the round (78) but Tier B (73) does not beat gemma-4-26B-A4B's clean-protocol 64 → 77-pattern leaders on injection-clean rows; injection-gate verdicts still pending per the #2643 protocol and will be appended.

Caveats standing from this PR's review: rows measured before the scoring-path commits (d16a64be, 55b15c47) are not comparable to post-fix rows; the round-7 chain measures against the current tree, so this table supersedes any overlapping survey rows. The earlier cold-chain at a99e765 was aborted by operator decision (folded into round 7, #3079); its in-flight qwen3:8b cell was quarantined and is excluded.

Chain still running — next roster models will append to this comment thread when their cells close.

Xore · 2026-09-07T08:49:15Z

Round-7 chain results — 40 scored models (2026-09-06 14:18 → 2026-09-07 08:46 CEST)

Cold-protocol chain (round7_coldrun.sh, sequential, evict-before-run, 2 runs/cell) completed 42 roster entries; 2 cells failed and are rescheduled. Full corrected table (supersedes the stub table above):

Model	Tier A	Tier B
`phi4:14b`	78	76
`gpt-oss:20b`	78	68
`Qwen3.8-27B-Uncensored-Cyber Q4_K_M`	78	73
`llama3.1:8b-instruct-q4_K_M`	77	68
`DeepHat-V1-7B Q4_K_M`	76	74
`GLM-4.7-Flash-Reasoning-Distill Q4_K_M`	76	73
`qwen3.8:27b`	75	71
`gemma2:27b`	74	71
`Qwen3.5-9B-Defiant-Fable iMatrix-MTP Q4_K_M`	74	67
`Huihui-CyberStrike-OffSec-35B-abliterated Q6_K`	74	77
`DeepHat-V1-7B-Heretic-Ablit fixed q4_k_s`	73	69
`qwen2.5:14b-instruct`	73	71
`Huihui-Qwen3.6-27B-abliterated Q4_K_M`	73	72
`Seneca-Cybersecurity-x-QwQ-32B Q4_Medium`	73	73
`glm4:9b`	72	69
`qwen3.5:4b`	71	75
`Foundation-Sec-1.1-8B i1-Q4_K_S`	71	67
`WhiteRabbitNeo-2.5-Coder-7B Q4_K_M`	71	66
`gemma-4-31B-it-qat-q4_0-uncensored-heretic`	71	73
`Ornith-1.0-35B-uncensored-heretic Q4_K_M`	71	77
`Qwen3.6-27B-Fable-Fusion-711-MTP Q4_K_M`	71	69
`Qwen3.8-27B-Unleashed Q4_K_M`	71	73
`SecurityLLM-7B Q4_K_M`	70	59
`qwen2.5:7b-instruct`	70	69
`Huihui-Qwen3.5-9B-abliterated Q4_K_M`	70	71
`qwen2.5-coder:14b`	70	70
`gemma-4-26B-A4B ultra-unc-heretic q4_k_s`	70	72
`Qwen3.6-27B-unc-heretic-v2-MTP Q4_K_M`	70	72
`Ornith-1.5-9B-UNCENSORED Q4_K_M`	70	65
`qwen3:14b` (incumbent)	69	71
`Mistral-Small-3.2-24B ultra-unc Q4_K_M`	69	68
`gemma-4-26B-A4B ultra-unc-heretic i1-Q4_K_S`	69	70
`codellama:7b-instruct`	68	62
`Lily-Cybersecurity-7B Q4_K_M`	68	57
`qwen3:8b`	68	70
`codegeex4:9b`	68	68
`deepseek-coder-v2:16b-lite`	68	65
`qwen2.5-coder:7b`	63	68
`Qwen3.6-12B-IQ-Ultra-Heretic Q4_K_M`	62	62
`Dolphin3-Cyber-8B Q4_K_M`	61	61
`DeepHat-V1-7B-Heretic-Ablit i1-Q4_K_S`	0 (cell failed — rescheduled)	—
`GPT-OSS-Cybersecurity-20B i1-Q4_K_M`	0 (cell failed — rescheduled)	—

Every completed cell reproduced run1 == run2 (cold protocol, evict-before-run). 40 scored models, 2 cells failed and are being rescheduled by the chain.

Headline reads:

phi4:14b (78/76) and gpt-oss:20b (78/68) post the best Tier A of the round; Huihui-CyberStrike-OffSec-35B (74/77), Ornith-1.0-35B (71/77), and qwen3.5:4b (71/75) lead Tier B.
The incumbent qwen3:14b lands at 69/71 on the current tree — consistent with this PR's finding that its survey row was contamination-inflated.
Injection-gate verdicts (benchmarks: ghidra Tier B has zero injection-gate coverage (payload never survives into Ghidra pseudocode); Tier A fails 4 of 12 models #2643 protocol) still pending and will be appended per model; scoring-path caveats from the review (commits d16a64be, 55b15c47) apply only to pre-fix rows — the round-7 chain measures against the current tree.
The earlier cold-chain at a99e765 was aborted by operator decision (folded into round 7, Epic: round 7 — Unsloth training and dynamic requant, served via Ollama, scored on a fresh three-slot benchmark #3079); its in-flight cell was quarantined and excluded.

Chain continues; next cells will append to this thread.

Xore · 2026-09-07T22:38:42Z

Round-7 cold baseline progress check — 2026-09-08 00:45 CEST

Chain state on the homeserver (round7_coldrun.sh → sweep_extra.sh → record_baseline.py, round-7 pin, 17 cases/79, injection gate v3, cold slot, N=2 with 3/5 escalation):

Position: ~67 of 96 roster entries done, chain healthy and mid-flight.

66 MODEL_DONE, 1 in flight at sample time (hf.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS tier B run 2), 30 status-marked (UNMEASURABLE/UNMEASURED/UNRESOLVED) — no zero-token no-ops silently counted as done this pass (benchmarks: a model that 500s on every request is recorded as score=0 and MODEL_DONE — record_baseline.py exits 0 with 0 cases, the sweep never marks it UNMEASURABLE #3090 fix holding).
GPU pinned at 98–100% utilization, 17–18 GB VRAM steady; vram_samples.tsv at 1,933 samples (10 s cadence since 09-06).
378 JSON records in /mnt-1/benchmarks/1947full/; 82 models with measurable scores so far.
Escalation path exercised where N=2 disagreed (existing escalated.txt/failures.txt mechanism intact).
Wave 2 no-GPU legs (round7-2: pipeline proof T0 — REx86 adapter → 16-bit merge → GGUF → Ollama, scored against its base and #160's PEFT numbers #3081 merge/convert prep, round7-7: dynamic requant ladders — imatrix + per-tensor bit selection for the 27–35B class and the 100B+ offload class, against #2245's plain ladder #3086 calibration, round7-8: benchmark round 7 driver — 17-case /83 pin, pooled claims, all three slots, cold protocol, chained behind the cold baseline, write-up and promotion records #3087 drivers) finished 2026-09-07 01:30 CEST; coding legs await orchestrator dispatch as planned.

Provisional tier-A leaders (percent, run-pooled, subject to the full-matrix freeze):

percent	model (quant)
97.1	mradermacher/Foundation-Sec-1.1-8B-Instruct i1-Q4_K_S
97.1	DavidAU/Qwen3.6-27B-Fable-Fusion-711 NEO-MAX-MTP Q4_K_M
95.7	Trendyol-Cybersecurity-LLM-Qwen3-32B Q8_0
95.7	huihui-qwen3.6-35b-a3b-abliterated q3_k
95.7	philbert440/Qwen3.8-27B-Uncensored-Cyber Q4_K_M
95.7	mradermacher/VulnLLM-R-7B i1-Q4_K_M
95.7	OBLITERATUS/Gemma-4-12B-OBLITERATED Q4_K_M

Small-8B security-specialized models hold the top band with the 27–35B MTP/A3B heretics; nothing above 97.1 yet. These are not final rankings — the matrix freeze and the pooled-claims pass come after the roster completes (~29 entries left, roughly 3–4 h at the current ~6–8 min/model-quant cadence).

Next update when the chain hits EXTRA_COMPLETE (or if a leg dies — restart-safety rules from the cold-baseline SOP apply on any homeserver reboot: no nohup auto-restart, rows with rc=143 are kill artifacts and get re-run, never quoted).

Xore · 2026-09-08T00:32:43Z

Round-7 cold baseline progress check — 2026-09-08 02:35 CEST

Chain state on the homeserver (round7_coldrun.sh → sweep_extra.sh → record_baseline.py, round-7 pin, 17 cases/79, injection gate v3, cold slot, N=2 with 3/5 escalation):

Position: 69 of 96 roster entries done (MODEL_DONE), chain healthy, still mid-flight.

In flight at sample time: hf.co/HauhauCS/Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP-GGUF:Q4_K_M tier B run 2 (tier A runs scored 70 on both passes — first B pass scored 74, escalation not triggered).
Progress since the 00:45 check: +3 MODEL_DONE (66 → 69) at ~6–8 min/model-quant cadence — consistent, no stalls, GPU still pinned 98–100%.
Calib decompilation feed (S3, s3-decomp/) advanced 2,490 → 2,689/3,150 builds; calib-labeling leg still queued behind EXTRA_COMPLETE per plan.
378 JSON records in 1947full/, 82 models with measurable scores; round7/ scratch dir has 281 records for the current wave.

Provisional leaderboard (raw total_score, 79-point scale, run-pooled, not final):

score	model (quant)
67	mradermacher/Foundation-Sec-1.1-8B-Instruct i1-Q4_K_S
67	DavidAU/Qwen3.6-27B-Fable-Fusion-711 NEO-MAX-MTP Q4_K_M
66	Trendyol-Cybersecurity-LLM-Qwen3-32B Q8_0
66	huihui-qwen3.6-35b-a3b-abliterated q3_k
66	philbert440/Qwen3.8-27B-Uncensored-Cyber Q4_K_M
66	mradermacher/VulnLLM-R-7B i1-Q4_K_M
66	OBLITERATUS/Gemma-4-12B-OBLITERATED Q4_K_M
65	qwen3:8b (reference)
65	qwen2.5:14b-instruct-q4_K_M (reference)

Same top band as the 00:45 check holds — the two 8B/27B security-specialized leaders are unchanged at the ceiling; reference general models (qwen3:8b, qwen2.5:14b) sit at 65, two points under the leaders. Escalation path exercised where N=2 disagreed; UNMEASURABLE/UNMEASURED status marks intact (no zero-token no-ops counted as done — #3090 fix holding).

Roughly 27 roster entries remain; ETA ~3–4 h to EXTRA_COMPLETE, then the calib-labeling leg (label_s3_openrouter → round7_build_calib.sh EXEC=1) unblocks. Next update at EXTRA_COMPLETE or on any leg failure (cold-baseline SOP restart rules apply on homeserver reboot: no nohup auto-restart, rc=143 rows are kill artifacts and get re-run, never quoted).

Xore · 2026-09-08T18:17:02Z

Round-7 cold baseline — COMPLETE — 2026-09-08 18:10 CEST

ROUND7_COMPLETE fired 2026-09-08T17:00:11Z on the homeserver (round7_coldrun.sh → sweep_extra.sh → record_baseline.py, round-7 pin, 17 cases/79, injection gate v3, cold slot, N=2 with 3/5 escalation). Ghidra slot restored (ghidra-revdeck-1) at completion.

Final chain state

91 of 96 roster entries MODEL_DONE; 367 JSON records in round7/ (182 unique model/tier cells, all with both N=2 runs).
5 roster entries never pulled — PULL_FAILED in failures.txt (manifest 404s on four tags, one repo-gated pull): observerx-qwen3.8-27b-heretic q4_k_s, huihui-qwen3.8-27b-abliterated q4_k, huihui-qwen3.6-35b-a3b-abliterated q3_k, ahmedandaloes/CyberStrike-OffSec-35B Q3_K_M, protoLabsAI/ThinkingCap-Qwen3.6-27B-abliterated-MTP Q4_K_M. These are UNMEASURABLE (pull), not UNMEASURED — no silent holes.
11 of 91 completed entries scored 0.0 on every run — all trace to serving defects, not model quality (details below). 80 models have usable scores; 323/367 runs measurable, mean 83.3%, median 85.5%.

Reproducibility (the #2642/#2644 protocol holding)

179 of 182 cells reproduce run1 == run2 exactly (cold slot, temperature 0, seed 144). The 3 divergent cells: Foundation-Sec-1.1-8B Q8_0 (83.1→80.7), gemma-4-26B-A4B-ultra Q4_K_M (88.0→86.7), XORTRON.LARGE.2026.3 IQ2_XXS (81.9→80.7). All three already flagged in escalated.txt — N=2 disagreement ≤2.4 points, escalation to N=3 per SOP.
No zero-token no-ops counted as done (benchmarks: a model that 500s on every request is recorded as score=0 and MODEL_DONE — record_baseline.py exits 0 with 0 cases, the sweep never marks it UNMEASURABLE #3090 fix holding).

Zero-scored entries — root causes (do NOT read as model quality)

gpt-oss-family harmony rendering (4 tags: CyberPal2.0-20B, GPT-OSS-Cybersecurity-20B-Merged ×2, plus ornith-35b-selfquant quants via the 500-path): the harness's harmony adaptation keys on gpt-oss in the tag (HARMONY_FAMILY_MARKERS). CyberPal2.0-20B is a gpt-oss-architecture model but its tag doesn't contain the marker — served without the adaptation, every response comes back empty (parse_ok: false, raw: "", done_reason: stop, eval_count ~8-10) — the gpt-oss-family models return empty content through evaluate-models' chat+schema path #2233 think:false empties-analysis signature, reproduced live just now: response: "" on both /api/generate and /api/chat with think: true and false. Raw-mode confirms the model generates fine; the harmony channel handling eats the output.
ornith-35b-selfquant (4 quants: Q4_K_M, q3_k_m, q3_k_s, iq3_m): every request → HTTPError 500 after retries; direct load now fails with `check_tensor_dims: tensor 'blk.40...' error — the self-quantized GGUFs are structurally broken, not a harness issue. Note the upstream Ornith-1.0-35B (llmfan46, Q4_K_M) scored 92.8% — the base model is fine, the local requantization is not.
DeepHat-V1-7B-Heretic-Abliterated i1-Q4_K_S (qwen2 family): llama-server terminates on load — Unable to generate parser for this template. Broken template metadata in the i1-GGUF. The non-heretic DeepHat-V1-7B Q4_K_M scored 91.6%.
XORTRON.CriminalComputing.2026.{4B,27B}.NEXT i1-GGUF (qwen35 family) and Glimmer-Sentry-30B (muse-glimmer family): same class — responses empty or degenerate (Glimmer emits only to=self, a leaked harmony channel token). Architectures/families the round-7 serving adaptations don't cover.

Follow-up worth filing: extend is_harmony_served from tag-string matching to architecture detection via /api/show (general.architecture == gpt-oss, plus muse-glimmer needs its own handling), and add a pre-flight one-token smoke per model so serving-defect zeros are marked SERVING_DEFECT instead of 0.0. The 8 non-ornith zeros are re-measurable in ~1.5 h once the adaptation covers them.

Provisional leaderboard (best of N=2, 79-point scale, percent)

percent	model
95.2	Trendyol-Cybersecurity-LLM-Qwen3-32B Q8_0
94.0	phi4:14b
94.0	gpt-oss:20b
94.0	philbert440/Qwen3.8-27B-Uncensored-Cyber Q4_K_M
92.8	llama3.1:8b-instruct-q4_K_M
92.8	llmfan46/Ornith-1.0-35B-uncensored-heretic Q4_K_M
92.8	huihui-ai/Huihui-CyberStrike-OffSec-35B-abliterated q6_k
92.8	AlicanKiraz0/Titus-CybersecurityLLM-v1.0 Q4_K_M
92.8	mradermacher/VulnLLM-R-7B i1-Q4_K_M

Reference anchors: qwen2.5:14b-instruct 88.0 (tier A), qwen3:14b (incumbent) 85.5 tier B / 83.1 tier A, qwen3:8b 84.3 tier B / 81.9 tier A. The incumbent sits 8–10 points under the security-specialized leaders — consistent with this PR's original finding that its survey row was inflated.

Injection positive control (strcpy_note_injected, tier A run 1)

Only 14 of 91 models fully resist the injected-note control (score 5/5). Perfect resistance among the measurable top band: phi4:14b (94.0), VulnLLM-R-7B i1-Q4_K_M (92.8), GLM-4.7-Flash distill (91.6), ravenx-cyberagent-35b (90.4). Notable: gpt-oss:20b scores 94.0 on the corpus but only 3/5 on injection resistance; the Trendyol leader takes 4/5. Highest corpus score and perfect resistance do not co-occur in this roster — same conclusion as #2643's earlier tier: injection-gate verdicts should gate any recommendation, not raw corpus percent.

Matrix freeze + pooled-claims pass over the 80 usable rows.
Calib-labeling leg now unblocked (EXTRA_COMPLETE fired): label_s3_openrouter → round7_build_calib.sh EXEC=1; S3 decomp feed finished at 3,150/3,150 builds earlier today.
Serving-defect re-measure block (11 rows) after the harmony-architecture fix lands.
1947full/ continues to hold the full historical record set (378 records); vram_samples.tsv captured the entire chain at 10 s cadence.

Xore · 2026-09-09T09:36:01Z

T0 scoring leg (#3081) completed 2026-09-09, results posted to #1523: rex86-merged 68/68 TierA, 62/65 TierB vs base twins 64/70 (Q4/Q8). Base-twin lineage decontaminated via #3137/#3138; scorer fixes #3140/#3142 shipped. PR tracker: adding the T0 results summary to this PR's docs requires a repo edit — orchestrator will fold it into the next docs commit unless you want a standalone PR.

…e models Harvests the 2026-08-27/28 three-lane corpus-revdeck round from the analysis host and records part 4 of the #1947 rebuild. Matrix: 12 models x 2 tiers, gcc-x86_64/-O0 slice, 14 cases, max 69, temp 0 seed 144 ctx 8192. Machine-readable results plus 24 pinned transcript run directories (one per model/tier cell). Findings beyond the table: - Tier B - Tier A splits by family and the sign is not universal: gemma/Ornith rows gain +4/+5/+4, every Qwen row loses 1-5 including the incumbent. #1805-b's single-model +1.17 generalised to nothing. - The harness is deterministic since #1953's reasoning_effort:"none": all three repeats produced byte-identical answers for every model and tier. Probed the cause directly - omitting the parameter restores non-determinism, sequential and concurrent alike. 22.1 of 37.4 model-wall hours re-derived identical bytes, and the +-0.58 noise band cited across #1947 does not apply to post-#1953 code. - Tier B has zero injection-gate coverage on all 11 cells: the payload is a referenced string literal that Ghidra's pseudocode never carries, so the gate reports not-tested rather than passed (#1948's guard working). At Tier A 4 of 12 models fail, including the two highest Tier B scorers. - ghidra-ollama-1 was recreated mid-round at 2026-08-27T20:18:17Z; with no retry and no incremental save that cost 10 cells, leaving three rows at N=1. Decision: no promotion, qwen3:14b stays. gemma-4-26B-A4B is the best candidate the rebuild has produced (64 Tier B, clean gates, fastest in the matrix) but its lead rests on an N=1 incumbent that needs re-measuring first.

The previous commit named f3f4c14 (#1953) as the cause of the round's byte-identical repeats. Extending the probe refutes that as a sufficient explanation and the claim is withdrawn. With reasoning_effort:"none" held constant, qwen2.5-coder:7b is byte-stable across three repeats while qwen3:8b is not (first call differs, then stable) -- and qwen3:8b was perfectly stable across all six of its benchmark runs. The parameter therefore does not explain the effect, and the probe is not a clean replication of the harness request anyway. The observation itself is unchanged and stands on 62 runs with distinct transcript ids, hashes and wall times: repeats measured nothing. The section now reports that as an unexplained result, keeps the consequences that hold regardless of cause, and drops the noise-band conclusion instead of restating it with a different cause attached. What the probe does settle is recorded: concurrency is not the variable, under NUM_PARALLEL=1 / MAX_LOADED_MODELS=1. Also records a defect found while checking the probe against the harness: record_baseline.py stamps context_tokens and keep_alive into every report's qualification_request but never sends either -- there is no num_ctx and no keep_alive in the payload. Benign here (the container serves 32768 and the largest evidence is ~1000 tokens) but the published pin is false, and #1947 rule 5 is request pinning.

Said the round ran at the container's OLLAMA_CONTEXT_LENGTH=32768. That is only true for a model the benchmark loads itself. Ollama serves at the loaded instance's context, and a model already resident from another client is reused at that client's size -- ghidra-ollama-1 is shared with the live stack, and ollama ps currently shows qwen3:14b resident at CONTEXT 8192, loaded by the worker. So the context each cell ran at was not controlled by the harness and is not recorded in any artifact. That makes the missing num_ctx a reproducibility defect rather than a cosmetic mislabel. Accuracy is still almost certainly unaffected -- largest evidence in the round is ~1000 tokens, inside every candidate ceiling.

…-slot protocol The first pass carried a measurement defect big enough to invalidate its headline comparison. Five models re-run on a controlled protocol; part 4 rewritten around them, and the twelve-model pass demoted to a survey with explicit caveats on which rows may be read. What was wrong: Ollama serves a request at the context and state of whatever model instance is resident, and record_baseline.py neither sets nor records that. Two measured consequences: - hp-llm-worker holds qwen3:14b resident at CONTEXT 8192, so the incumbent was the one row measured against a different instance than everything it was compared with -- observed live as 10 GB/8192 (worker-loaded) vs 14 GB/32768 (benchmark-loaded), same tag and digest. - A warm slot does not reproduce and a cold one does. Controlled A/B, same model and request: warm 60/61 with 1 of 14 answers byte-identical, cold 60/60 with 14 of 14. The survey's uniform +-0 was three lanes evicting each other under MAX_LOADED_MODELS=1, i.e. lane contention masquerading as determinism, which also inflated wall clock up to 15x. Protocol adopted: evict before every run, so a cell depends only on its own inputs. Every cohort cell is +-0 across three cold runs, on four architectures. Which rows moved confirms the diagnosis: Ornith and gemma-4-26B reproduced their survey Tier B scores exactly (67, 64) since nothing else loads them, while qwen3:14b moved 62->60 and 61->58 -- the one model the worker keeps resident. Decision changes from "no promotion" to promoting gemma-4-26B-A4B-it-ultra-uncensored-heretic-i1 to the ghidra slot: Tier B 64 vs the incumbent's real 58 (+6, not the +3 the contaminated row implied), Tier A 61 vs 60, clean gates including Tier A injection, and 2.1 min/run vs 3.8. Ornith scores 67 but fails the Tier A injection gate reproducibly and stays disqualified. gemma-4-31B deferred: ~40 min/run for a model that also fails that gate and cannot be promoted whatever it scores; queued to run last. Adds docs/benchmarks/matrices/1947-cohort-cold-protocol.json and 8 pinned cohort transcript dirs.

…ts corpus vintage The injection section named the wrong cause. It said the payload is a string literal that Ghidra's pseudocode does not inline. Checked against the data rather than the code: the Tier B cache entry for process_and_injection does carry a strings list, the needle is in none of its 27 entries, and strings -a on the cached object does not find it either -- it is not in that binary at all. The corpus objects were built 2026-08-25, before 0b01d57 (#1948 / #2379) turned the payload from a source comment into a referenced string literal, and the Tier B cache was extracted from them. That also explains the asymmetry the section could not account for: Tier A reads the disassembly stored in the repo manifest, and objdump -d --source prints the source comment, so Tier A found the needle for the old fixture by accident of representation while Tier B could not. The two tiers were reading different fixture vintages. Scope is contained and now stated: 0b01d57 changed exactly one fixture, so 13 of 14 cases are byte-identical and their Tier B scores stand. Only process_and_injection was measured against a pre-#1948 object, so each Tier B total carries that single case's uncertainty and the Tier B injection verdict is untested for artifact reasons rather than model reasons. Tier A is unaffected throughout. Corpus rebuild and cache re-extraction are running; #2643 carries the detail.

Xore · 2026-09-14T00:06:45Z

Round-7 cold baseline update landed: 91 models, 182 cells, 367 records, current-authority section in docs/local-llm-model-evaluation.md. Old matrices marked superseded. Matrix JSONs added under docs/benchmarks/matrices/. Merge remains on hold per the earlier comment — decision is yours.

Xore marked this pull request as draft August 29, 2026 16:45

Xore mentioned this pull request Aug 29, 2026

Containers CI: buildkit graceful_stop aborts a random matrix row during GHA cache export #2639

Closed

Xore changed the title ~~docs(benchmarks): #1805-c ghidra-slot matrix — Tier A vs Tier B across twelve models~~ docs(benchmarks): #1805-c ghidra-slot matrix — cold-protocol re-measure, and promote gemma-4-26B-A4B Aug 29, 2026

Xore marked this pull request as ready for review August 29, 2026 20:38

Xore marked this pull request as draft August 30, 2026 01:07

Xore and others added 6 commits September 13, 2026 23:12

docs(benchmarks): add round-7 cold baseline as authoritative roster

fdb9095

Xore force-pushed the worktree-bench-1805c-results branch from a4b59af to fdb9095 Compare September 14, 2026 00:06

Navigation Menu

docs(benchmarks): #1805-c ghidra-slot matrix — cold-protocol re-measure, and promote gemma-4-26B-A4B - #2641

docs(benchmarks): #1805-c ghidra-slot matrix — cold-protocol re-measure, and promote gemma-4-26B-A4B#2641
Xore wants to merge 6 commits into
mainfrom
worktree-bench-1805c-results

Xore commented Aug 29, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

Xore commented Aug 29, 2026

Uh oh!

Xore commented Aug 30, 2026

Uh oh!

Xore commented Sep 7, 2026

Uh oh!

Xore commented Sep 7, 2026

Uh oh!

Xore commented Sep 7, 2026

Uh oh!

Xore commented Sep 8, 2026

Uh oh!

Xore commented Sep 8, 2026

Uh oh!

Xore commented Sep 9, 2026

Uh oh!

Xore commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Xore commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The defect

Cohort re-run, cold protocol — authoritative

Decision: promote (changed from "no promotion")

Other findings filed

Uh oh!

github-actions Bot commented Aug 29, 2026

Dependency Review

Scanned Files

Uh oh!

Xore commented Aug 29, 2026

Uh oh!

Xore commented Aug 30, 2026

Uh oh!

Xore commented Sep 7, 2026

Round-7 chain progress — new rows (2026-09-06 14:18 → 2026-09-07 08:46)

Uh oh!

Xore commented Sep 7, 2026

Round-7 chain results — 40 scored models (2026-09-06 14:18 → 2026-09-07 08:46 CEST)

Uh oh!

Xore commented Sep 7, 2026

Round-7 cold baseline progress check — 2026-09-08 00:45 CEST

Uh oh!

Xore commented Sep 8, 2026

Round-7 cold baseline progress check — 2026-09-08 02:35 CEST

Uh oh!

Xore commented Sep 8, 2026

Round-7 cold baseline — COMPLETE — 2026-09-08 18:10 CEST

Final chain state

Reproducibility (the #2642/#2644 protocol holding)

Zero-scored entries — root causes (do NOT read as model quality)

Provisional leaderboard (best of N=2, 79-point scale, percent)

Injection positive control (strcpy_note_injected, tier A run 1)

Next

Uh oh!

Xore commented Sep 9, 2026

Uh oh!

Xore commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Xore commented Aug 29, 2026 •

edited

Loading