record_baseline.py: no retry/incremental save (one ollama restart cost 10 cells), and context_tokens/keep_alive are pinned in reports but never sent

[Xore]

Two defects in analysis/ghidra/benchmarks/corpus/record_baseline.py, both
found while harvesting the #1805-c ghidra-slot round (PR #2641). Filed together
because both are in the same file and both are about the request the harness
actually makes versus the one it claims to make.

1. No retry and no incremental save — one container restart cost 10 cells

ghidra-ollama-1 was recreated at 2026-08-27T20:18:17Z, mid-round. All
three benchmark lanes died within one second of each other:

20:18:16 done A qwen3:14b run2 exit:1                    <- 53 min of work, discarded
20:18:17 done A qwen3:14b run3 exit:1                    <- instant, container still restarting
20:18:17 done B qwen3:14b run1 exit:1                    <- instant
20:18:17 done B qwen3:14b run2 exit:1                    <- instant

identically for qwen2.5-coder:14b-instruct-q4_K_M (lane 2) and
Seneca-Cybersecurity-LLM-x-QwQ-32B (lane 3). The failure is
http.client.RemoteDisconnected out of urlopen(req, timeout=600) at
record_baseline.py:251, with no handler above it.

Two separate costs:

• No retry. The container was back within seconds — the very next run
  (B run3, started 20:18:17) completed normally an hour later. Three of the
  four failures per lane were runs that fired into a two-second restart window
  and died instantly. A bounded retry with backoff on connection errors would
  have lost nothing at all.
• No incremental save. The A run2 failure had already scored 11 of 14
  cases across 53 minutes. Every one of those answers was thrown away because
  the report is only written at the end. Cases are independent; there is no
  reason a run cannot checkpoint per case and resume.

Net effect: 10 of 72 cells lost, leaving three models at N=1 in a matrix
whose whole purpose was N>=3 with error bars, and directly blocking the
gemma-4-26B-A4B promotion call in #1805-c because the incumbent's Tier B
number is one of them.

Fix: catch connection-class exceptions in ask_model, retry with backoff (the
model is resident and keep_alive is generous, so a retry is cheap), and write
per-case results incrementally so an aborted run resumes instead of restarting.
The lane script's resume guard already keys on the output filename, so a
partial-report format is the missing half of a mechanism that already exists.

Related but distinct: #2032 records the same class of defect (no incremental
save, no HTTP retry) in claims.py. This is the same lesson in a second file —
worth checking whether any other benchmark entry point has it too.

2. context_tokens and keep_alive are recorded but never sent

Every report stamps a qualification_request block that is quoted verbatim as
the round's pins:

{"concurrency": 1, "context_tokens": 8192, "keep_alive": "10m",
 "output_tokens": 512, "seed": 144, "temperature": 0, "thinking": false}

The payload ask_model actually builds carries model, messages,
temperature, max_tokens, seed, stream and (since #1953)
reasoning_effort. There is no num_ctx and no keep_alive anywhere in the
file — grep -n "num_ctx\|keep_alive\|context_tokens" returns nothing but
unrelated analysis_options hits.

On the analysis host the effect is benign and was checked rather than assumed:
the container sets OLLAMA_CONTEXT_LENGTH=32768 and the largest evidence in
the round is 4002 characters (~1000 tokens), so nothing was truncated — the
served context was four times the pin that was published.

It is still a false pin, and it is a hole in #1947 rule 5 (tag + digest +
request pinning), which exists precisely so a round can be reproduced. A report
that states a context length it did not set will silently produce different
numbers on a host with a different OLLAMA_CONTEXT_LENGTH, and nothing in the
artifact would show it.

Fix: send options.num_ctx and keep_alive for real, or drop them from
qualification_request and record the server-side context instead. Sending
them is better — the pin is the point. Either way the recorded block and the
transmitted request must agree, and a test should assert that they do.

[Xore]

Correction and escalation on defect 2 — the original text said the missing
num_ctx was benign because the container sets OLLAMA_CONTEXT_LENGTH=32768.
That is only true for a model the benchmark loads itself, and it understates
the problem.

Ollama serves at the loaded instance's context, not the request's. A model
loaded with no num_ctx takes OLLAMA_CONTEXT_LENGTH; a model already
resident, loaded by a different client at a different size, is reused at that
client's size. ghidra-ollama-1 is shared with the live stack, and ollama ps
on the host right now shows:

NAME         ID              SIZE     PROCESSOR    CONTEXT    UNTIL
qwen3:14b    bdbd181c33f2    10 GB    100% GPU     8192       9 minutes from now

8192, loaded by the worker — not 32768, and not by the benchmark.

So the context each cell of #1805-c actually ran at depended on who loaded
the model first, was not controlled by the harness, and is not recorded in any
artifact. With OLLAMA_MAX_LOADED_MODELS=1 and three lanes rotating models
against a live worker, that is genuinely non-deterministic across cells of the
same round.

This makes it a reproducibility defect, not a cosmetic mislabel, and raises
its priority relative to defect 1. Accuracy in this round is still almost
certainly unaffected — the largest evidence is 4002 characters (~1000 tokens),
inside every ceiling in play — so the numbers stand; what does not stand is the
claim that they can be reproduced from the recorded pin.

It also means the fix is not optional: sending options.num_ctx explicitly
is the only way to stop a benchmark cell from inheriting whatever the live
worker happened to load. Dropping the field from qualification_request would
leave the harness silently sharing context configuration with production
traffic. Send it, and assert in a test that the recorded block and the
transmitted payload agree.

Worth checking whether evaluate-models.py has the same hole, since parts 1–3
of the #1947 matrix ran through it.

[Xore]

Cross-links: the keep_alive half of defect 2 is a fourth data point for #2365 (guides say 10m, compose and the live container say 30m, benchmark reports publish 10m while inheriting 30m) — detail posted there. #2409 is the same approval chain from the runtime-image side. Whatever value #2365 settles on, record_baseline.py must either send it or stop printing it, or the divergence reappears inside the benchmark artifacts.

[Xore]

Direct live evidence for defect 2, captured while launching the #1805-c backfill.

Same host, same model tag, same benchmark, minutes apart. Before — the instance
hp-llm-worker had loaded (its env sets LLM_CONTEXT_LENGTH=8192,
LLM_KEEP_ALIVE=10m):

NAME         ID              SIZE     PROCESSOR    CONTEXT    UNTIL
qwen3:14b    bdbd181c33f2    10 GB    100% GPU     8192       9 minutes from now

After stopping the worker, evicting with ollama stop qwen3:14b, and letting
record_baseline.py load it itself:

NAME         ID              SIZE     PROCESSOR    CONTEXT    UNTIL
qwen3:14b    bdbd181c33f2    14 GB    100% GPU     32768      29 minutes from now

Same digest, 8192 vs 32768 context, and 10 GB vs 14 GB of VRAM — decided
entirely by which process loaded the model first. The benchmark has no say in
it and records neither.

This is worse than "the pin is cosmetic" in two ways:

1. VRAM figures are affected too. Benchmark round: new derestricted HF candidates (XORTRON, GLM-4.6-Derestricted, gemma-4 heretic) within the 20GB VRAM budget #1795-c part 2 publishes a sole-resident
   VRAM MiB column. A row measured against a worker-loaded instance and a row
   measured against a benchmark-loaded one are 4 GB apart for reasons that have
   nothing to do with the model.
2. It is model-dependent in practice, so it hits some rows and not others.
   hp-llm-worker and ghidra-revdeck-1 both use qwen3:14b continuously.
   Nothing on the box uses Ornith-35B, the gemma-4 heretics, Seneca or the
   Huihui tunes — those are only ever loaded by the benchmark, at 32768. So in
   the Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805-c matrix the incumbent is the row most likely to have been
   measured at a different context from everything it is being compared
   against, and it is exactly the row the promote/no-promote call turns on.

The backfill now running re-measures qwen3:14b at a known 32768 with the
worker stopped, so this is about to be a measured question rather than a
suspicion: if the new Tier A runs disagree with the stored run1 score of 62,
context contamination changed a published number.

Raises priority again. Sending options.num_ctx explicitly is the fix; a test
asserting the recorded qualification_request matches the transmitted payload
is what stops it recurring.

[Xore]

Picked up for automated fix under v2 workflow. Branch: orchestrator/issue-2644 (base main).

[Xore]

PR #2656 merged. CI: 108 passing, 0 failing, mergeState=CLEAN. Closes #2644 confirmed.