Next benchmark round: re-run gpt-oss cells under the harmony request shape before comparing across rounds

[Xore]

Follow-up to #2233 (fixed by #2277).

Why this needs a line in the round bookkeeping

Pre-#2277 matrix rows for gpt-oss-family models (gpt-oss:20b, GPT-OSS-Cybersecurity-20B-Merged-heretic) were produced by requests whose content came back empty — the scores on those cells are null-field artifacts, not model behavior. #2277 fixes the serving shape, but it also means:

• Round-to-round comparability for these tags breaks across the fix. Any score from before the harmony dispatch is not comparable to anything after it. Under One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947's record-keeping, affected historical cells should be treated as unmeasured (or annotated), not as regression baselines.
• The new wire shape still differs mechanically from the Qwen family (grammar off, thinking channel on, wider anti-repetition window). It is recorded per-request in transcripts, but matrix footnotes should say so for these two tags.

What to do at the next operator benchmark window

1. Re-run sessions + revdeck (and ghidra/triage when Tier B evidence lands) for both family tags through the fixed harness.
2. Confirm no analysis-channel spirals remain at the rescue sampler settings; if one appears, widen repeat_last_n/repeat_penalty rather than touching prompts — prompts must stay byte-identical across families.
3. Annotate the matrix so pre-fix gpt-oss cells are visibly not baselines.

Partial verification already done live (homeserver ollama 0.32.13): grammar-ignored discovery, loop isolation to SESSION_SUFFIX MITRE rules, sampler-rescue JSON parse on case 1 through the real harness — see #2277 body. Full breadth was deferred because an unrelated GPU benchmark held the host.

[Xore]

Reproduced in record_baseline.py — #2277's fix never reached this harness

The full-roster ghidra-slot sweep (#1805-c / #1947 part 4, cold-slot protocol)
just measured gpt-oss:20b and the #2233 signature is intact:

tier	score	empty answers	answer length min/max
A	11 / 69	11 of 14	0 / 1556 chars
B	3 / 69	13 of 14	0 / 310 chars

These are null-field artifacts, not model behaviour, exactly as this issue
predicted. #2277 fixed evaluate-models.py only. record_baseline.py
builds its own payload and has no harmony branch — it sends
reasoning_effort: "none" and max_tokens: 512 to /v1/chat/completions for
every model, which is the OpenAI-compatible equivalent of the think: false
that #2277 documents as emptying the answer for this family rather than
suppressing analysis.

Scope: four rows in the current sweep, not one

Every gpt-oss-architecture tag in the roster is affected:

• gpt-oss:20b — measured above
• hf.co/mradermacher/GPT-OSS-Cybersecurity-20B-Merged-heretic-i1-GGUF:i1-Q4_K_M
• hf.co/mradermacher/GPT-OSS-Cybersecurity-20B-Merged-heretic-GGUF:Q4_K_M
• hf.co/mradermacher/CyberPal2.0-20B-GGUF:Q4_K_M — arch is GptOssForCausalLM,
  so it belongs to this family despite the name not saying so

That last one matters for issue hygiene: is_harmony_served() matches on the
string "gpt-oss" in the tag, so CyberPal2.0-20B would slip past the fix
even in evaluate-models.py. Tag-name matching is not a reliable family
test; the architecture field is.

What I am doing about it, and what I am deliberately not

Not patching record_baseline.py mid-sweep. Each cell is a fresh interpreter
invocation reading the file at run time, so editing it now would silently split
the matrix across two code vintages — the exact defect #1947 rule 6 exists to
prevent, and the reason the #1805-c survey had to be demoted to a survey.

Plan, after the sweep completes:

1. Port fix(benchmarks): serve gpt-oss family through a harmony-aware request shape (#2233) #2277's harmony branch into record_baseline.py's ask_model —
   reasoning_effort raised rather than none, max_tokens to 4096 (the
   analysis channel spends the same budget), and the anti-repetition widening
   (repeat_last_n 256 / repeat_penalty 1.3) mapped onto the
   OpenAI-compatible request.
2. Detect the family by architecture, not tag substring, so CyberPal-class
   tags are covered.
3. Re-run only the four rows above, and record them as an explicitly annotated
   group — per this issue's step 3, pre-fix gpt-oss cells are visibly not
   baselines, and post-fix cells are a different wire shape from every other
   row in the matrix.

The 11/69 and 3/69 numbers above are being carried into the matrix as
unmeasured, not as scores.

[Xore]

Picking this up as part of round 6 batch D.

[Xore]

Step 1 of the issue's own plan is done and already merged — PR #2679 (2026-08-30) ported the harmony branch into record_baseline.py with architecture-based family detection. What's left is step 3, the four-model re-run, and that's GPU-contended with this batch's #1947 resume (same reasoning as #2969's parking note) — also already queued as priority 3 in xore's own in-flight #2641/#1805 work. Parking with the exact model list and the confirmed-merged fix; no further code change needed, just the re-run once the GPU frees up.

[Xore]

Review verdict: APPROVE (honest park) — PR #2679 confirmed MERGED, and stronger than the handover claimed: /mnt-1/benchmarks/APIARY is pinned at a99e765, which is #2679's commit, so the harmony fix is provably in the harness and the re-run needs no code change or re-pin. The residual note is the valuable part — the pre-fix 11/69 and 3/69 cells must be written up as unmeasured, not as scores. Note the GPU contention this parks behind is currently unproductive (see #1947); this short re-run is a good candidate for the gap once that sweep is stopped.

[Xore]

Reconciliation dispatch ended INCOMPLETE despite exit 0. It performed local setup only and misread nonempty recorded tool results as empty, then exited before its promised next read. No model inference, worker changes, or new benchmark results occurred. No provider/API error was recorded. The gpt-oss/Harmony and Ornith comparisons remain pending; no identical retry or acceptance waiver has been applied. A separate read-only check confirmed the historical scripts exist on the benchmark host, so the local path lookup failure does not establish a missing host installation. Recovery needs a corrected execution leg, not another evidence-summary claim.