benchmarks: repeat runs are byte-identical since the Aug-25 anchor — #1947's N>=3 protocol and its ±0.58 noise band rest on an unexplained change

[Xore]

Found while harvesting the #1805-c ghidra-slot round (PR #2641).

The observation

In that round, all 12 models on both tiers scored identically on all three
repeats — and every one of the 14 answer strings was byte-identical across
repeats. Only wall_seconds moved.

This is not an aggregation artifact. The 62 runs each have their own
transcript_run_id, their own transcripts_sha256, and their own wall times;
three independent lanes logged each start and finish; the resume guard keys on
the output filename and each repeat wrote a new one. They are genuinely
separate executions that produced identical bytes.

Why it matters

1. N = 3 bought nothing. The round burned 37.4 h of model-wall, of which
   22.1 h (59%) re-derived identical bytes. Parts 1 and 2 of One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947 show the
   same ±0-dominant spread and very likely paid the same tax undiagnosed.
2. One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947's protocol rests on a noise band that is not a measured property of
   the current code. Rule 3 mandates N >= 3 on the strength of Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805-b's
   2026-08-25 measurement: four runs of one model at temp 0 / seed 144 /
   identical digest scoring 56, 56, 55, 55, with 7 of 14 cases moving and 0 of
   14 answers byte-identical. Two days later, nothing moves at all. ±0.58
   must not be used to dismiss small margins in post-fix(benchmarks): send thinking:false, and stop paying for an empty answer #1953 matrices until
   this is understood.
3. It changes what a single number is worth, in both directions, and the
   promote/no-promote calls in One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947 depend on that.

What is NOT established: the cause

The obvious candidate is f3f4c14 (#1953), merged between the two
measurements, which added reasoning_effort: "none" to every request. A
direct probe refutes that as a sufficient explanation. Ollama 0.32.13 on the
analysis host, identical prompt, temperature 0, seed 144, three repeats each
way:

model	reasoning_effort: "none"	parameter omitted
qwen2.5-coder:7b-instruct-q4_K_M	identical x3, sequential and concurrent	2 distinct outputs, both ways
qwen3:8b	first call differs, then stable	first call differs, then stable

qwen3:8b is not byte-stable under the probe, yet it was perfectly stable
across all six of its benchmark runs. So the parameter is not the explanation,
and the probe is not a clean replication either — it repeats one short prompt,
where the harness sends 14 distinct long ones.

What the probe does settle: concurrency is not the variable. The
container runs OLLAMA_NUM_PARALLEL=1 / OLLAMA_MAX_LOADED_MODELS=1, requests
to a model are serialised and never batched, and the deterministic arm stays
byte-stable under three simultaneous requests.

Remaining hypotheses worth testing, none of them checked: cold-load vs warm
state (the probe's first-call-differs pattern points here), KV-cache prefix
reuse across repeated identical prompts, prompt length, and model-eviction
behaviour under MAX_LOADED_MODELS=1 when lanes interleave.

What to do

1. Establish the cause with a replication that matches the harness request
   shape — long corpus prompts, 14 distinct cases per run, same eviction
   pattern — rather than a synthetic short prompt.
2. Then set an N policy explicitly in the One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947 protocol instead of
   inheriting N >= 3. If repeat-identical requests are genuinely
   deterministic, repeats belong on the axes that are not fixed (seed,
   quant, prompt phrasing), not on the one that is. That is a strictly better
   use of the same GPU hours.
3. Until then, treat run-to-run spread in the Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805-c matrix as
   uninformative, not as evidence the harness is noise-free — and do not
   quote ±0.58 as its noise floor.

Note this does not invalidate the #1805-c scores; it invalidates the error bars
around them, which are all exactly zero for a reason nobody has yet pinned.

[Xore]

The premise of this issue is now falsified, and the direction is the opposite
of what I filed. Measured while re-running the three N=1 rows on a quiet box
(hp-llm-worker and ghidra-revdeck-1 stopped, GPU otherwise idle).

qwen3:14b, Tier A, same corpus, same commit 22f01c2, same request:

run	conditions	score	min
run1 (stored, from the original round)	3 lanes, model loaded by the live worker at CONTEXT 8192	62	62.5
run2 (backfill)	sequential, benchmark-loaded at CONTEXT 32768	60	4.1
run3 (backfill)	sequential, same resident instance as run2	61	4.0

Answers byte-identical: run1 vs run2 → 2/14. run2 vs run3 → 1/14.
evidence_chars is identical case-for-case, so the input never changed.

So: the harness is not deterministic, and it never was. Two consecutive
runs of the same model against the same warm instance, back to back, disagree
on 13 of 14 answers and land 60 vs 61. The "byte-identical repeats" that this
issue was opened about are a property of the original round's conditions, not
of the code — and the noise band is real after all, roughly matching #1805-b's
±0.58 (60, 61, and a contaminated 62).

This retracts the framing in the issue body, which argued from the
byte-identical repeats that N>=3 was wasted GPU time. It is not wasted. On this
evidence N>=3 is doing exactly the job #1947 rule 3 designed it for, and the
#1805-c matrix's uniform ±0 is the anomaly to explain, not the baseline.

Candidate mechanism — explicitly NOT yet tested

The original round ran three lanes against OLLAMA_MAX_LOADED_MODELS=1, so
each lane's request evicted the other two lanes' models. Every request would
have hit a freshly (re)loaded model — a clean slot. The backfill is sequential,
so the model stays resident across all 14 cases and across runs
(OLLAMA_KEEP_ALIVE=30m) and slot/KV state accumulates.

That would make it: cold-slot → reproducible, warm-slot → drifts. It fits
both observations, and it is a guess. The run ordering is not the difference —
I checked, lane1805c.sh also runs 1/2/3 of a (model, tier) back to back, the
same loop the backfill uses.

Decisive test, cheap now that a run is 4 minutes: repeat one cell twice with an
explicit ollama stop <tag> between runs. If the two become byte-identical,
the mechanism is confirmed. Queued for immediately after the backfill finishes,
so as not to perturb it.

Why this matters beyond bookkeeping

If it is the warm/cold split, then the warm case is the production-shaped
one — llm-worker and revdeck both hold a resident model — and the
original matrix measured every model under an artifact of the benchmark's own
lane contention. That would make the cold numbers the unrepresentative ones,
and it would mean per-case scores depend on what the same slot processed
before, which is a cross-case contamination problem inside a single run, not
just across runs.

Also worth noting the timing: 62.5 min → 4.1 min for the same work, 15x,
purely from removing lane contention. The min/run column in the #1805-c
matrix is a contention artifact and must not be read as model speed.

[Xore]

Cause established: cold slot reproduces, warm slot drifts

The decisive test from my previous comment has run. qwen3:14b, Tier A, same
corpus, same commit 22f01c2, same request, quiet box, four runs — the only
variable is whether the model slot was evicted between them.

arm	protocol	scores	answers byte-identical
warm	two runs back to back, model stays resident	60, 61	1 / 14
cold	ollama stop <tag> before each run	60, 60	14 / 14

Cross-check: warm1 vs cold1 are also 14/14 identical — consistent, since
warm1 was itself the first run after an eviction, i.e. a cold run by
construction. Only warm2, the one run that inherited a used slot, diverges.

A freshly loaded Ollama slot produces byte-identical output. A warm resident
slot does not. That is the whole effect, and it explains both observations
this issue was opened about, with no appeal to reasoning_effort (the
hypothesis I filed and then withdrew):

• The original Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805-c round ran three lanes against
  OLLAMA_MAX_LOADED_MODELS=1. Every lane switch evicted the other lanes'
  models, so effectively every run began with a reload — cold — and all three
  repeats came back byte-identical. The uniform ±0 was lane contention
  masquerading as determinism.
• The sequential backfill keeps one model resident across runs
  (OLLAMA_KEEP_ALIVE=30m), so runs 2 and 3 inherit a used slot and drift:
  qwen3:14b Tier B measured 60, 62, 61 — a real spread of 2.

Consequences

1. N >= 3 is justified, and the Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805-c matrix's ±0 is an artifact. The
   spreads in this harness are ~2–3 points, not ±0 and not ±0.58. Any margin
   under ~3 points in the published matrix is inside the noise.
2. The benchmark should force a cold slot per run. It makes a cell
   independent of whatever the slot processed before it, which is what a
   ranking comparison needs, and it is reproducible to the byte. The
   decision-cohort re-run now in flight does exactly this
   (ollama stop before every run) and is the protocol I would put in the
   contract.
3. Production runs warm, and this says something about production. Both
   llm-worker and revdeck hold a resident model, so their answers are not
   reproducible and depend on what the same slot processed earlier. That is a
   cross-request coupling nobody has characterised — it deserves its own issue
   rather than a line in a benchmark doc, because it affects analyst-facing
   output, not just scores.
4. It also means a benchmark run in the cold protocol does not predict
   warm/production behaviour case-for-case. Both are worth measuring; they
   are different questions and must be labelled as such.

Closing note on process: this is the second causal claim on this issue. The
first (#1953's reasoning_effort) was refuted by probe and withdrawn. This one
is a controlled A/B with the mechanism isolated and a consistent cross-check,
so it is on much firmer ground — but it is one model on one tier, and the
cohort re-run will show whether 14/14 cold reproducibility holds across
architectures.

[Xore]

Picked up for automated fix under v2 workflow. Branch: orchestrator/issue-2642 (base main).

[Xore]

PR #2662 merged. CI: 108 passing, 0 failing, mergeState=CLEAN. Closes #2642 confirmed.