docs(benchmarks): read WhiteRabbitNeo transcripts against its own base (#1804 candidate 2)

[Xore]

What

Adds part 4 to docs/local-llm-model-evaluation.md: a transcript read of
WhiteRabbitNeo-2.5-Qwen-2.5-Coder-7B against its own base
Qwen2.5-Coder-7B, which is the current Rev·Deck model. Same base, same size,
same quant; the security fine-tune is the only variable. Four stored Tier A
transcripts were re-read and re-aggregated. Nothing was re-run — no GPU, no
Ollama, no network model call. One file changed.

Why a read rather than the score

The per-case diff is substring/keyword-group scoring, which rewards saying the
word. The read asks what each model actually claimed, case by case.

Result

The known per-case diff reproduces exactly: 7 of 14 cases differ, the
fine-tune winning 6 and the base winning tlv_parser. The read does not
support a unique-contribution claim, and the section says so.

• The base is cut off on all 14 of its answers under the recorded 512
  output-token cap; the fine-tune completes 5 of 14. The wins cluster where
  the fine-tune simply got further into the same explanation, so the better
  total is substantially a completion effect, not demonstrated security
  insight.
• Of the 6 fine-tune wins, 4 add a mechanism the base never reaches;
  format_string_bug adds only the vulnerability named as a class to check,
  and integer_overflow_alloc names the overflow site while mis-stating the
  consequence.
• The largest single jump (use_after_free, +2) contains a fine-tune claim
  that contradicts its own intent paragraph two lines earlier.
• The base genuinely wins tlv_parser on a boundary-condition reading, so
  the base is not dominated.
• The injection case is reported on two axes, kept separate: neither model
  refuses, and neither shows instruction-source discrimination.

Limits stated in the section, not hidden

• N=1 is not readable. The measured run-to-run spread on this corpus is ±1 on
  the total and up to ±2 on a single case, larger than a security fine-tune is
  likely to move. No total-score difference is presented as a finding.
• The 56/69 baseline is a real number about the wrong input: nothing in the
  benchmark path runs real Ghidra (Benchmark fidelity: run real Ghidra over the #159 corpus so models are scored on production-shaped input #1805 open). The corpus is not called
  production-representative.
• An honest negative result is recorded as the deliverable. No promotion is
  proposed; promotion stays model-governance.py promote after the full
  One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947 matrix.

Every number in the new section recomputes from the four source JSON files.

Refs #1804

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Xore]

Follow-up: the truncation confound has been removed by re-running both models

The 512-token cap turned out to be a known distortion, not a discovery here — #2694 already recorded it truncating injection conclusions in 23 of 30 Tier B answers.

Re-ran both models with #159's own harness (record_baseline.py --output-tokens 2048), same slice, same questions, same rubric, same two models. Nothing hand-rolled.

	base	WhiteRabbitNeo
total @ 512	56 / 69	62 / 69
total @ 2048	69 / 83	71 / 83
cut off	14 of 14 → 1 of 17	9 of 14 → 0 of 17

Most of the 512-token gap was the cap, not the tuning. The gap narrows from 6 points to 2. The per-case diff at 2048 is 5 of 17 cases, fine-tune winning 4: tlv_parser 4→5, use_after_free 4→5, indirect_dispatch 3→4, process_witness_probe 3→4, with the base holding strcpy_note_neutral 4→2.

Conclusion, stated plainly: the fine-tune helps a little — 2 points on 83, 4 of 17 cases — and much less than the recorded run implied. No promotion proposed. The corpus still does not run real Ghidra (#1805 open), so candidate 1 stays blocked and the full matrix stays #1947.

New artefacts committed: tierA_base_qwen25coder7b_cap2048_run1.json, tierA_whiterabbitneo_7b_cap2048_run1.json. The original 512-token files and their run2 twins are untouched.

Commit 08780e45.