docs(vision): OCRBench quant ladder for gemma4:31b (q4/q8/bf16), and a paired summarizer

docs/maxusai/adr/0036-gemma4-image-chunk-decodes-in-one-batch.md

@@ -56,6 +56,18 @@ still wins; embedding loads and the constrained-CUDA-without-flash-attention pat
 - The generation batch for gemma4 vision at the 8192 rung is 2048, with `generationBatchSurcharge`'s 2 GiB
   budgeted at admission; on a card without that headroom the batch steps down and the split returns, logged as
   "generation batch below the image chunk".
+- **Measured 2026-09-19 (`../ocrbench-gemma4-quant-ladder.md`): the floor is denied at every
+  quantization on gfx1151, so this ADR is currently inert on the fork's own production hardware.**
+  `gemma4:31b` logged `num_batch=1024 image_chunk_batch=2048` (q4_K_M, `-np 1`), `512` (q8_0 and
+  bf16), and `512` at `-np 2`. Not for want of memory: the scheduler logs `available="95.4 GiB"` of
+  GPU while `availableMemoryForLoad` returns the 31 GiB system figure instead, because its iGPU
+  branch prefers live system memory whenever it is smaller than shared GPU free. That branch's
+  premise — iGPU free memory is "a static or slowly refreshed device baseline" — does not hold on a
+  Strix Halo carve-out, where the 96 GiB is real and not shared with the 31 GiB the host sees. Until
+  that sizing is revisited, the 1.5–1.9× prefill gain above is unavailable on gfx1151 and the split
+  is what ships there. Consequence worth stating explicitly, because it was briefly assumed
+  otherwise: the chunk still exceeds `n_ubatch`, so this ADR does **not** mask the HIP MMQ race and
+  `llama/compat/906-revert-hip-integrated-flag.patch` remains load-bearing for gemma4.
 - gemma4 e2b and e4b decode images causally since llama.cpp #28335, so the split never hurt them; they get the same
   batch because the family and the vision capability are what the scheduler can see, and their compute buffers are small.
 - `nemotron_h_omni`'s ceiling (3328) exceeds the batch ladder; it stays on the context rung, out of scope here.

docs/maxusai/ocrbench-gemma4-quant-ladder.md

@@ -0,0 +1,166 @@
+# OCRBench on ROCm/GGUF: gemma4:31b across q4_K_M, q8_0 and bf16
+
+**Date:** 2026-09-19 · **Host:** gfx1151 (Strix Halo, 96 GiB GPU carve-out, 31 GiB system RAM)
+**Build:** `0.34.1-dynres-16649e8c` — 0.34.1 + `906-revert-hip-integrated-flag` + `OLLAMA_IGPU_DIRECT_IO` + ADR 0036
+
+Run to answer two questions at once:
+
+1. **Cross-backend.** MLX-Metal reported OCRBench `correct: 175, accuracy: 0.875` on
+   `gemma4:31b-nvfp4` with the post-#3912 kernel (200 scored, 0 errors, 0 empty). Where does
+   the ROCm GGUF path land on the same slice?
+2. **The unquantized control.** [ADR 0024](adr/0024-locate-faults-before-fixing-them.md) and the
+   2026-09-18 learnings entry ("a tier move is not evidence about a kernel until the unquantized
+   arm is measured") say a quantized score alone cannot tell a model's answer from a
+   quantization artefact. The GGUF path had never had that control measured. bf16 supplies it.
+
+## Result
+
+Tables emitted by `vision-suite/summarize_extbench.py` (SPEC `vision-harness-reuse.md` H7) —
+pasted verbatim, not retyped:
+
+```
+| model | scored | errors | empty | correct | accuracy | think | endpoint |
+|---|---|---|---|---|---|---|---|
+| `gemma4:31b-it-q4_K_M` | 200 | 0 | 0 | 171 | **0.855** | false | generate |
+| `gemma4:31b-it-q8_0` | 200 | 0 | 0 | 169 | **0.845** | false | generate |
+| `gemma4:31b-it-bf16` | 200 | 0 | 0 | 169 | **0.845** | false | generate |
+
+ocrbench — `echo840/OCRBench` [test], rows 0..200.
+
+host: pre-H11 run (not recorded) · build: pre-H11 run (not recorded)
+
+| pair | both ✓ | both ✗ | A only | B only | McNemar exact p |
+|---|---|---|---|---|---|
+| ocr1np_q4_k_m vs ocr1np_q8_0 | 168 | 28 | 3 | 1 | 0.625 |
+| ocr1np_q4_k_m vs ocr1np_bf16 | 168 | 28 | 3 | 1 | 0.625 |
+| ocr1np_q8_0 vs ocr1np_bf16 | 169 | 31 | 0 | 0 | 1.000 |
+```
+
+## What the paired columns say that the accuracies cannot
+
+**`q8_0` and `bf16` are identical on all 200 items.** Not "close" — zero discordant pairs. Every
+item either both got right or both got wrong. On this benchmark, 8-bit quantization of
+`gemma4:31b` costs nothing at all, and `q8_0` is a usable stand-in for the unquantized arm.
+
+**`q4_K_M` differs from both by 3 items won and 1 lost, exact two-sided p = 0.625.** That is a
+coin-flip disagreement, not a quality gap. The 0.855 / 0.845 split in the accuracy column is two
+items wide and would be read as a real ordering by anyone comparing the aggregates alone.
+
+This is the measurement the accuracy column *cannot* make. At n = 200 the 95% interval on 0.855
+is about ±0.049, so every arm here — and MLX-Metal's 0.875 — falls inside every other arm's
+interval. Only the shared row set makes the comparison sharp, and only for arms we ran ourselves.
+
+## Against MLX-Metal
+
+| backend | build | quant | correct | accuracy |
+|---|---|---|---|---|
+| MLX-Metal | 0.34.0, post-#3912 kernel | nvfp4 | 175/200 | 0.875 |
+| ROCm GGUF | `0.34.1-dynres-16649e8c` | q4_K_M | 171/200 | 0.855 |
+| ROCm GGUF | `0.34.1-dynres-16649e8c` | q8_0 | 169/200 | 0.845 |
+| ROCm GGUF | `0.34.1-dynres-16649e8c` | bf16 | 169/200 | 0.845 |
+
+**This table is not a ranking and must not be published as one.** A 4–6 item difference at
+n = 200 is inside noise, and we cannot do better without MLX-Metal's per-item `ext_*_ocrbench.json`:
+McNemar needs the discordant items, and an aggregate cannot supply them. The honest reading is
+that the two backends are indistinguishable on this slice at this sample size.
+
+Worth noting for its own sake: the ROCm ladder is *flat* while the MLX ladder was not. On Metal,
+`31b-nvfp4` and `31b-mlx-bf16` differed by a 9 px fine-text tier once the kernel was fixed. Here
+`bf16` and `q8_0` agree item for item and `q4_K_M` is within noise of both. Different
+quantization schemes, different vision towers, different kernels — but it means the GGUF path has
+no quantization artefact of the kind that misled the Metal investigation.
+
+## Placement, and one confound
+
+Every arm: `-c 16384 -np 1`, `offloaded 61/61 layers`, think off, `/api/generate`, median
+`prompt_eval_count` **1113** in all three — so the image pipeline delivered identical token counts
+and the budget-fill path is not a variable here.
+
+The generation batch was **not** identical, and this is a confound between `q4_K_M` and the other
+two:
+
+| arm | `-b` / `-ub` | median s/req | wall |
+|---|---|---|---|
+| q4_K_M | 1024 | 7.7 | 26 min |
+| q8_0 | 512 | 8.1 | 27 min |
+| bf16 | 512 | 8.1 | 28 min |
+
+So `q4_K_M`'s 3-vs-1 advantage could be quantization **or** batch size; these data cannot separate
+them. `q8_0` vs `bf16` is clean — same batch, same everything, zero discordant items.
+
+## Why the batch differs, and why ADR 0036 never fires here
+
+All three arms logged ADR 0036's own diagnostic on every load:
+
+```
+msg="generation batch below the image chunk, images decode in pieces" num_batch=1024 image_chunk_batch=2048
+```
+
+The 2048 floor is requested and denied at every quantization. The cause is
+`availableMemoryForLoad` (`server/sched.go`), which for an integrated GPU deliberately prefers the
+live system-memory figure over the device baseline:
+
+```go
+if systemInfo.FreeMemory > 0 && sharedGPUFree > 0 && systemInfo.FreeMemory < sharedGPUFree {
+    return discreteGPUFree + systemInfo.FreeMemory, gpuFree, true
+}
+```
+
+On this host the scheduler logs `gpu memory … available="95.4 GiB"` and `system memory total="31.0 GiB"`,
+so `availableMemory` resolves to **31 GiB**, not 95.4. Every rung decision then follows from 31 GiB:
+
+| arm | predicted | 2048 needs ≤ 60% (18.6 GiB) | 1024 needs ≤ 75% (23.3 GiB) | result |
+|---|---|---|---|---|
+| q4_K_M | ~19 GiB | no | yes | 1024 |
+| q8_0 | ~33 GiB | no | no (over the 80% threshold) | 512 |
+| bf16 | ~58 GiB | no | no | 512 |
+
+ADR 0036 already anticipates the step-down ("on a card without that headroom the batch steps down
+and the split returns"). What is new is that **gfx1151 is not such a card** — it has 95.4 GiB free
+and is being sized against 31 GiB of system RAM. The heuristic's comment justifies itself on the
+grounds that iGPU free memory "can be a static or slowly refreshed device baseline"; on Strix Halo
+with a fixed carve-out that assumption does not hold, and the cost is that the fork's own flagship
+vision optimization never engages on the fork's own production hardware.
+
+Two consequences:
+
+- **ADR 0036 is inert on gfx1151 at every quantization**, `-np 1` and `-np 2` alike (512 at
+  `-np 2`, 1024 at `-np 1` for q4). Its measured 1.5–1.9× prefill gain is not available here.
+- **Compat 906 stays load-bearing for gemma4.** The image chunk is 1113 tokens against `n_ubatch`
+  of 1024 or 512, so the decode still exceeds `n_ubatch` — the trigger condition for the HIP MMQ
+  race. ADR 0036 does not incidentally mask the defect, which was a live worry when #320 landed.
+
+## What the misses look like
+
+29 wrong at q4_K_M, all near-misses on scene text and handwriting — the shape of a model reading a
+blurry crop, not of a corrupted decode:
+
+```
+THREADS  → THREAD        CORNERS → CORNER S      GOOD → GOOOD
+Scottynn → Scottlynn     CORONAD → CORONA        both → booth
+```
+
+For contrast, the HIP defect produced whole-image failures and `0.000` scene IoU, not off-by-one
+transcriptions. Nothing in these records resembles it.
+
+## Provenance caveat
+
+The three score files carry **no `host` / `server_version`**: `extbench.py` did not persist them
+until the change that accompanies this document, so the footer renders
+`pre-H11 run (not recorded)` and the generator refuses to let a recorded file vouch for an
+unrecorded one. The build in the heading is attested from the runner logs captured in the run log,
+which is **inferred provenance, not recorded provenance** ([ADR 0024](adr/0024-locate-faults-before-fixing-them.md)).
+Re-running the three arms purely to stamp them would cost ~5 hours of GPU and ~90 GiB of pulls;
+future extbench runs record it for free.
+
+## Reproducing
+
+```sh
+cd docs/maxusai/vision-suite
+LIMIT=200 OFFSET=0 THINK=false \
+  python3 extbench.py http://127.0.0.1:11499 ocr1np_q8_0 gemma4:31b-it-q8_0 ocrbench
+python3 summarize_extbench.py ocrbench ocr1np_q4_k_m ocr1np_q8_0 ocr1np_bf16 --paired
+```
+
+The container must run `OLLAMA_NUM_PARALLEL=1` for the placement above; at `-np 2` the effective
+context doubles to 32768 and q4's batch drops from 1024 to 512.

docs/maxusai/spec/vision-harness-reuse.md

@@ -381,6 +381,7 @@ something previously hidden:
 | H4b | `arm_done` in `vision_suite.py` importing `was_capped` (H5); `test_summarizers.py::TestResumeNeverSkipsCapped` asserts capped, error, and missing blocks all re-run; `::TestWasCappedPrefersDoneReason` asserts the `done_reason` verdict outranks the arithmetic and absence falls back to it |
 | H5, H6 | `summarize_reps.py`, `summarize_geometry.py`, `summarize_matrix.py` and `preflight/checks.py` all import `was_capped` (`test_summarizers.py::TestCappedDiscipline`, `test_verdicts.py::TestQualityCappedExcluded`); the ladder decisions are `capped_arms` / `ceiling_standing` / `mark_not_converged` in the same module, driven by `run_engine_compare.sh` via argv subcommands (`::TestCappedArms`, `::TestLadderCeilingMarker`) |
 | H7 | ADR 0012 rules 1 and 8; request examples via `emit_request.py` (payload captured from `client.py`, never re-derived) |
+| H7, H11, H13 (external benchmarks) | `extbench.py` runs render through `summarize_extbench.py`, never by hand; `extbench.py` persists the `host` / `server_version` that `client.generate()` already stamps on every response, collected as SETS so a mid-run container restart cannot let one build vouch for another's rows; `test_summarizers.py::TestExtbenchSummary` asserts the unrecorded-file, mixed-campaign and clean-footer cases, and that the paired test stays EXACT (at 3-vs-1 discordant the uncorrected chi-square reads p≈0.317 against the true 0.625) |
 | H13 (footers) | `test_summarizers.py::TestProvenanceFooter` — clean / all-pre-H11 / mixed-recording / two-host cases against the rendered footer |
 | H13 (capped rendering) | `cap_or` in `summarize_head_to_head.py` importing `was_capped` (H5); `test_summarizers.py::TestT2CappedCells` asserts a capped scene hides score and latency but keeps tok/s; `q()`/`multi_cell` in `summarize_engine_compare.py` guard every T1 quality cell — `::TestT1CappedQualityCells` |
 | H14 | `token_split.py`'s acceptance gate (`--write` refuses without it, and skips-by-name irreconcilable cells); `test_summarizers.py::TestT1ThinkTokColumn` asserts stamped-count-or-dash, never an estimate; `finetext_probe.py` persists as `finetext_probe` (one producer per persist name) |

docs/maxusai/vision-learnings-log.md

@@ -525,3 +525,39 @@ same binary and window.
   revert a kernel that is wrong on 232722/294912 elements at this shape. Both
   caught only because the bf16 checkpoint was already in the store and took
   ~4 minutes to measure.
+
+### 2026-09-19 — Two accuracies cannot tell "these differ" from "they disagree on coin-flips"
+Aggregate scores on a shared row set throw away the only information that can
+separate a real difference from sampling noise: **which** items each arm got
+right. Reporting 0.855 against 0.845 invites an ordering that the data does not
+support; the paired discordant counts settle it in one line.
+
+- **Evidence** — OCRBench 200 rows, `gemma4:31b` on ROCm/GGUF, one binary
+  `0.34.1-dynres-16649e8c`, think off, `-c 16384 -np 1`
+  (`ocrbench-gemma4-quant-ladder.md`):
+
+  | pair | both ✓ | both ✗ | A only | B only | McNemar exact p |
+  |---|---|---|---|---|---|
+  | q4_K_M vs q8_0 | 168 | 28 | 3 | 1 | 0.625 |
+  | q8_0 vs bf16 | 169 | 31 | 0 | 0 | 1.000 |
+
+  `q8_0` and `bf16` are identical on **every one of the 200 items** — zero
+  discordant pairs — so 8-bit costs nothing here and `q8_0` is a usable stand-in
+  for the unquantized arm on this benchmark. The q4 "lead" is 3 items won and 1
+  lost. At n=200 the 95% interval on 0.855 is ±0.049, so all three arms *and*
+  MLX-Metal's 0.875 sit inside each other's intervals; only the shared row set
+  is sharp, and only for arms we ran ourselves.
+- **The GGUF ladder is flat where the MLX ladder was not.** The 2026-09-18 entry
+  above found nvfp4 and bf16 differing by a 9 px tier on Metal once the kernel
+  was fixed. Same model family, same benchmark family, opposite result — so
+  "quantization is costing us a tier" is a finding about a specific
+  quantization and kernel, never a general property of the checkpoint.
+- **Enforced by** — `summarize_extbench.py --paired` (H7 generator) computes the
+  discordant counts and an **exact** McNemar. Exact, not chi-square: at 3-vs-1
+  the continuity-corrected approximation gives p≈0.317 against the true 0.625,
+  and 0.317 is the kind of number that gets reported as "trending".
+  `test_summarizers.py::TestExtbenchSummary` pins both the exact values and the
+  H13 footer behaviour.
+- **Cost** — none this time, because the paired columns were computed before
+  the numbers were written up. The near-miss is that the first draft of the
+  status report quoted 171 vs 175 across backends as if the gap meant something.

docs/maxusai/vision-suite/README.md

@@ -108,6 +108,14 @@ first is image accounting and the second is generation length.
   `refcoco` mode reports the winning coordinate dialect and JSON key per item, so it doubles
   as a dialect probe. See [../vision-benchmark-survey.md](../vision-benchmark-survey.md) for
   why the external harnesses' own grounding scorers cannot be trusted with our models.
+- `summarize_extbench.py [--dir D] [--paired] <bench> <tag…>` — renders `ext_*.json` as one
+  markdown table with an H13 provenance footer. **Tables from external benchmarks are pasted
+  from this, never retyped** (SPEC H7). `--paired` adds the per-pair discordant counts and an
+  exact McNemar over the items both arms scored: two accuracies on a shared row set cannot
+  distinguish a real difference from a coin-flip disagreement, and the discordant columns can.
+  Measured 2026-09-19: `gemma4:31b` q8_0 and bf16 came back with **zero** discordant items over
+  200 OCRBench rows while their accuracy column read 0.845 for both and q4_K_M read 0.855
+  ([../ocrbench-gemma4-quant-ladder.md](../ocrbench-gemma4-quant-ladder.md)).
 - `run_grid.sh <host> <tag-prefix>` — model × think-mode grid against one host, with an
   optional restart hook between runs (see below). Budgets are **per think-mode** and set by
   the runner: think-off `num_predict` 4000, think-on `num_ctx - CTX_PROMPT_RESERVE` (8192 at

docs/maxusai/vision-suite/extbench.py

@@ -308,6 +308,10 @@ def main():
           f"endpoint={os.environ.get('ENDPOINT', 'generate')}")
 
     records, correct, ious, dialects, empty = [], 0, [], {}, 0
+    # H11: collected as SETS, not scalars. A container restart mid-run
+    # (the vision suite has had them) would otherwise let one build
+    # vouch for rows another build served.
+    hosts, builds = set(), set()
     for i, row in enumerate(rows):
         src = row["image"]["src"] if isinstance(row["image"], dict) else row["image"]
         path = cache_image(BENCH, offset + i, src)
@@ -323,6 +327,8 @@ def main():
             print(f"{offset + i:>4}  ERROR {e}")
             continue
         dt = round(time.time() - t0, 1)
+        hosts.add(r.get("_host"))
+        builds.add(r.get("_server_version"))
         if not pred.strip():
             empty += 1
 
@@ -362,6 +368,10 @@ def main():
         # only one of these cannot distinguish those.
         "think_env": think, "think_on": think == "on",
         "endpoint": os.environ.get("ENDPOINT", "generate"),
+        # H11: where it ran and which build served it. client.generate()
+        # already stamps both on every response; this only persists them.
+        "host": sorted(h for h in hosts if h) or None,
+        "server_version": sorted(b for b in builds if b) or None,
         "correct": correct, "accuracy": round(correct / n, 4) if n else None,
     }
     if BENCH == "refcoco":