diff --git a/docs/maxusai/adr/0036-gemma4-image-chunk-decodes-in-one-batch.md b/docs/maxusai/adr/0036-gemma4-image-chunk-decodes-in-one-batch.md index d2a507461cd..35c303a0433 100644 --- a/docs/maxusai/adr/0036-gemma4-image-chunk-decodes-in-one-batch.md +++ b/docs/maxusai/adr/0036-gemma4-image-chunk-decodes-in-one-batch.md @@ -56,6 +56,18 @@ still wins; embedding loads and the constrained-CUDA-without-flash-attention pat - The generation batch for gemma4 vision at the 8192 rung is 2048, with `generationBatchSurcharge`'s 2 GiB budgeted at admission; on a card without that headroom the batch steps down and the split returns, logged as "generation batch below the image chunk". +- **Measured 2026-09-19 (`../ocrbench-gemma4-quant-ladder.md`): the floor is denied at every + quantization on gfx1151, so this ADR is currently inert on the fork's own production hardware.** + `gemma4:31b` logged `num_batch=1024 image_chunk_batch=2048` (q4_K_M, `-np 1`), `512` (q8_0 and + bf16), and `512` at `-np 2`. Not for want of memory: the scheduler logs `available="95.4 GiB"` of + GPU while `availableMemoryForLoad` returns the 31 GiB system figure instead, because its iGPU + branch prefers live system memory whenever it is smaller than shared GPU free. That branch's + premise — iGPU free memory is "a static or slowly refreshed device baseline" — does not hold on a + Strix Halo carve-out, where the 96 GiB is real and not shared with the 31 GiB the host sees. Until + that sizing is revisited, the 1.5–1.9× prefill gain above is unavailable on gfx1151 and the split + is what ships there. Consequence worth stating explicitly, because it was briefly assumed + otherwise: the chunk still exceeds `n_ubatch`, so this ADR does **not** mask the HIP MMQ race and + `llama/compat/906-revert-hip-integrated-flag.patch` remains load-bearing for gemma4. - gemma4 e2b and e4b decode images causally since llama.cpp #28335, so the split never hurt them; they get the same batch because the family and the vision capability are what the scheduler can see, and their compute buffers are small. - `nemotron_h_omni`'s ceiling (3328) exceeds the batch ladder; it stays on the context rung, out of scope here. diff --git a/docs/maxusai/ocrbench-gemma4-quant-ladder.md b/docs/maxusai/ocrbench-gemma4-quant-ladder.md new file mode 100644 index 00000000000..744f07d8534 --- /dev/null +++ b/docs/maxusai/ocrbench-gemma4-quant-ladder.md @@ -0,0 +1,166 @@ +# OCRBench on ROCm/GGUF: gemma4:31b across q4_K_M, q8_0 and bf16 + +**Date:** 2026-09-19 · **Host:** gfx1151 (Strix Halo, 96 GiB GPU carve-out, 31 GiB system RAM) +**Build:** `0.34.1-dynres-16649e8c` — 0.34.1 + `906-revert-hip-integrated-flag` + `OLLAMA_IGPU_DIRECT_IO` + ADR 0036 + +Run to answer two questions at once: + +1. **Cross-backend.** MLX-Metal reported OCRBench `correct: 175, accuracy: 0.875` on + `gemma4:31b-nvfp4` with the post-#3912 kernel (200 scored, 0 errors, 0 empty). Where does + the ROCm GGUF path land on the same slice? +2. **The unquantized control.** [ADR 0024](adr/0024-locate-faults-before-fixing-them.md) and the + 2026-09-18 learnings entry ("a tier move is not evidence about a kernel until the unquantized + arm is measured") say a quantized score alone cannot tell a model's answer from a + quantization artefact. The GGUF path had never had that control measured. bf16 supplies it. + +## Result + +Tables emitted by `vision-suite/summarize_extbench.py` (SPEC `vision-harness-reuse.md` H7) — +pasted verbatim, not retyped: + +``` +| model | scored | errors | empty | correct | accuracy | think | endpoint | +|---|---|---|---|---|---|---|---| +| `gemma4:31b-it-q4_K_M` | 200 | 0 | 0 | 171 | **0.855** | false | generate | +| `gemma4:31b-it-q8_0` | 200 | 0 | 0 | 169 | **0.845** | false | generate | +| `gemma4:31b-it-bf16` | 200 | 0 | 0 | 169 | **0.845** | false | generate | + +ocrbench — `echo840/OCRBench` [test], rows 0..200. + +host: pre-H11 run (not recorded) · build: pre-H11 run (not recorded) + +| pair | both ✓ | both ✗ | A only | B only | McNemar exact p | +|---|---|---|---|---|---| +| ocr1np_q4_k_m vs ocr1np_q8_0 | 168 | 28 | 3 | 1 | 0.625 | +| ocr1np_q4_k_m vs ocr1np_bf16 | 168 | 28 | 3 | 1 | 0.625 | +| ocr1np_q8_0 vs ocr1np_bf16 | 169 | 31 | 0 | 0 | 1.000 | +``` + +## What the paired columns say that the accuracies cannot + +**`q8_0` and `bf16` are identical on all 200 items.** Not "close" — zero discordant pairs. Every +item either both got right or both got wrong. On this benchmark, 8-bit quantization of +`gemma4:31b` costs nothing at all, and `q8_0` is a usable stand-in for the unquantized arm. + +**`q4_K_M` differs from both by 3 items won and 1 lost, exact two-sided p = 0.625.** That is a +coin-flip disagreement, not a quality gap. The 0.855 / 0.845 split in the accuracy column is two +items wide and would be read as a real ordering by anyone comparing the aggregates alone. + +This is the measurement the accuracy column *cannot* make. At n = 200 the 95% interval on 0.855 +is about ±0.049, so every arm here — and MLX-Metal's 0.875 — falls inside every other arm's +interval. Only the shared row set makes the comparison sharp, and only for arms we ran ourselves. + +## Against MLX-Metal + +| backend | build | quant | correct | accuracy | +|---|---|---|---|---| +| MLX-Metal | 0.34.0, post-#3912 kernel | nvfp4 | 175/200 | 0.875 | +| ROCm GGUF | `0.34.1-dynres-16649e8c` | q4_K_M | 171/200 | 0.855 | +| ROCm GGUF | `0.34.1-dynres-16649e8c` | q8_0 | 169/200 | 0.845 | +| ROCm GGUF | `0.34.1-dynres-16649e8c` | bf16 | 169/200 | 0.845 | + +**This table is not a ranking and must not be published as one.** A 4–6 item difference at +n = 200 is inside noise, and we cannot do better without MLX-Metal's per-item `ext_*_ocrbench.json`: +McNemar needs the discordant items, and an aggregate cannot supply them. The honest reading is +that the two backends are indistinguishable on this slice at this sample size. + +Worth noting for its own sake: the ROCm ladder is *flat* while the MLX ladder was not. On Metal, +`31b-nvfp4` and `31b-mlx-bf16` differed by a 9 px fine-text tier once the kernel was fixed. Here +`bf16` and `q8_0` agree item for item and `q4_K_M` is within noise of both. Different +quantization schemes, different vision towers, different kernels — but it means the GGUF path has +no quantization artefact of the kind that misled the Metal investigation. + +## Placement, and one confound + +Every arm: `-c 16384 -np 1`, `offloaded 61/61 layers`, think off, `/api/generate`, median +`prompt_eval_count` **1113** in all three — so the image pipeline delivered identical token counts +and the budget-fill path is not a variable here. + +The generation batch was **not** identical, and this is a confound between `q4_K_M` and the other +two: + +| arm | `-b` / `-ub` | median s/req | wall | +|---|---|---|---| +| q4_K_M | 1024 | 7.7 | 26 min | +| q8_0 | 512 | 8.1 | 27 min | +| bf16 | 512 | 8.1 | 28 min | + +So `q4_K_M`'s 3-vs-1 advantage could be quantization **or** batch size; these data cannot separate +them. `q8_0` vs `bf16` is clean — same batch, same everything, zero discordant items. + +## Why the batch differs, and why ADR 0036 never fires here + +All three arms logged ADR 0036's own diagnostic on every load: + +``` +msg="generation batch below the image chunk, images decode in pieces" num_batch=1024 image_chunk_batch=2048 +``` + +The 2048 floor is requested and denied at every quantization. The cause is +`availableMemoryForLoad` (`server/sched.go`), which for an integrated GPU deliberately prefers the +live system-memory figure over the device baseline: + +```go +if systemInfo.FreeMemory > 0 && sharedGPUFree > 0 && systemInfo.FreeMemory < sharedGPUFree { + return discreteGPUFree + systemInfo.FreeMemory, gpuFree, true +} +``` + +On this host the scheduler logs `gpu memory … available="95.4 GiB"` and `system memory total="31.0 GiB"`, +so `availableMemory` resolves to **31 GiB**, not 95.4. Every rung decision then follows from 31 GiB: + +| arm | predicted | 2048 needs ≤ 60% (18.6 GiB) | 1024 needs ≤ 75% (23.3 GiB) | result | +|---|---|---|---|---| +| q4_K_M | ~19 GiB | no | yes | 1024 | +| q8_0 | ~33 GiB | no | no (over the 80% threshold) | 512 | +| bf16 | ~58 GiB | no | no | 512 | + +ADR 0036 already anticipates the step-down ("on a card without that headroom the batch steps down +and the split returns"). What is new is that **gfx1151 is not such a card** — it has 95.4 GiB free +and is being sized against 31 GiB of system RAM. The heuristic's comment justifies itself on the +grounds that iGPU free memory "can be a static or slowly refreshed device baseline"; on Strix Halo +with a fixed carve-out that assumption does not hold, and the cost is that the fork's own flagship +vision optimization never engages on the fork's own production hardware. + +Two consequences: + +- **ADR 0036 is inert on gfx1151 at every quantization**, `-np 1` and `-np 2` alike (512 at + `-np 2`, 1024 at `-np 1` for q4). Its measured 1.5–1.9× prefill gain is not available here. +- **Compat 906 stays load-bearing for gemma4.** The image chunk is 1113 tokens against `n_ubatch` + of 1024 or 512, so the decode still exceeds `n_ubatch` — the trigger condition for the HIP MMQ + race. ADR 0036 does not incidentally mask the defect, which was a live worry when #320 landed. + +## What the misses look like + +29 wrong at q4_K_M, all near-misses on scene text and handwriting — the shape of a model reading a +blurry crop, not of a corrupted decode: + +``` +THREADS → THREAD CORNERS → CORNER S GOOD → GOOOD +Scottynn → Scottlynn CORONAD → CORONA both → booth +``` + +For contrast, the HIP defect produced whole-image failures and `0.000` scene IoU, not off-by-one +transcriptions. Nothing in these records resembles it. + +## Provenance caveat + +The three score files carry **no `host` / `server_version`**: `extbench.py` did not persist them +until the change that accompanies this document, so the footer renders +`pre-H11 run (not recorded)` and the generator refuses to let a recorded file vouch for an +unrecorded one. The build in the heading is attested from the runner logs captured in the run log, +which is **inferred provenance, not recorded provenance** ([ADR 0024](adr/0024-locate-faults-before-fixing-them.md)). +Re-running the three arms purely to stamp them would cost ~5 hours of GPU and ~90 GiB of pulls; +future extbench runs record it for free. + +## Reproducing + +```sh +cd docs/maxusai/vision-suite +LIMIT=200 OFFSET=0 THINK=false \ + python3 extbench.py http://127.0.0.1:11499 ocr1np_q8_0 gemma4:31b-it-q8_0 ocrbench +python3 summarize_extbench.py ocrbench ocr1np_q4_k_m ocr1np_q8_0 ocr1np_bf16 --paired +``` + +The container must run `OLLAMA_NUM_PARALLEL=1` for the placement above; at `-np 2` the effective +context doubles to 32768 and q4's batch drops from 1024 to 512. diff --git a/docs/maxusai/spec/vision-harness-reuse.md b/docs/maxusai/spec/vision-harness-reuse.md index 5bb77c62df6..fd9589cda9c 100644 --- a/docs/maxusai/spec/vision-harness-reuse.md +++ b/docs/maxusai/spec/vision-harness-reuse.md @@ -381,6 +381,7 @@ something previously hidden: | H4b | `arm_done` in `vision_suite.py` importing `was_capped` (H5); `test_summarizers.py::TestResumeNeverSkipsCapped` asserts capped, error, and missing blocks all re-run; `::TestWasCappedPrefersDoneReason` asserts the `done_reason` verdict outranks the arithmetic and absence falls back to it | | H5, H6 | `summarize_reps.py`, `summarize_geometry.py`, `summarize_matrix.py` and `preflight/checks.py` all import `was_capped` (`test_summarizers.py::TestCappedDiscipline`, `test_verdicts.py::TestQualityCappedExcluded`); the ladder decisions are `capped_arms` / `ceiling_standing` / `mark_not_converged` in the same module, driven by `run_engine_compare.sh` via argv subcommands (`::TestCappedArms`, `::TestLadderCeilingMarker`) | | H7 | ADR 0012 rules 1 and 8; request examples via `emit_request.py` (payload captured from `client.py`, never re-derived) | +| H7, H11, H13 (external benchmarks) | `extbench.py` runs render through `summarize_extbench.py`, never by hand; `extbench.py` persists the `host` / `server_version` that `client.generate()` already stamps on every response, collected as SETS so a mid-run container restart cannot let one build vouch for another's rows; `test_summarizers.py::TestExtbenchSummary` asserts the unrecorded-file, mixed-campaign and clean-footer cases, and that the paired test stays EXACT (at 3-vs-1 discordant the uncorrected chi-square reads p≈0.317 against the true 0.625) | | H13 (footers) | `test_summarizers.py::TestProvenanceFooter` — clean / all-pre-H11 / mixed-recording / two-host cases against the rendered footer | | H13 (capped rendering) | `cap_or` in `summarize_head_to_head.py` importing `was_capped` (H5); `test_summarizers.py::TestT2CappedCells` asserts a capped scene hides score and latency but keeps tok/s; `q()`/`multi_cell` in `summarize_engine_compare.py` guard every T1 quality cell — `::TestT1CappedQualityCells` | | H14 | `token_split.py`'s acceptance gate (`--write` refuses without it, and skips-by-name irreconcilable cells); `test_summarizers.py::TestT1ThinkTokColumn` asserts stamped-count-or-dash, never an estimate; `finetext_probe.py` persists as `finetext_probe` (one producer per persist name) | diff --git a/docs/maxusai/vision-learnings-log.md b/docs/maxusai/vision-learnings-log.md index 39df9d33b28..a74ddad0ce3 100644 --- a/docs/maxusai/vision-learnings-log.md +++ b/docs/maxusai/vision-learnings-log.md @@ -525,3 +525,39 @@ same binary and window. revert a kernel that is wrong on 232722/294912 elements at this shape. Both caught only because the bf16 checkpoint was already in the store and took ~4 minutes to measure. + +### 2026-09-19 — Two accuracies cannot tell "these differ" from "they disagree on coin-flips" +Aggregate scores on a shared row set throw away the only information that can +separate a real difference from sampling noise: **which** items each arm got +right. Reporting 0.855 against 0.845 invites an ordering that the data does not +support; the paired discordant counts settle it in one line. + +- **Evidence** — OCRBench 200 rows, `gemma4:31b` on ROCm/GGUF, one binary + `0.34.1-dynres-16649e8c`, think off, `-c 16384 -np 1` + (`ocrbench-gemma4-quant-ladder.md`): + + | pair | both ✓ | both ✗ | A only | B only | McNemar exact p | + |---|---|---|---|---|---| + | q4_K_M vs q8_0 | 168 | 28 | 3 | 1 | 0.625 | + | q8_0 vs bf16 | 169 | 31 | 0 | 0 | 1.000 | + + `q8_0` and `bf16` are identical on **every one of the 200 items** — zero + discordant pairs — so 8-bit costs nothing here and `q8_0` is a usable stand-in + for the unquantized arm on this benchmark. The q4 "lead" is 3 items won and 1 + lost. At n=200 the 95% interval on 0.855 is ±0.049, so all three arms *and* + MLX-Metal's 0.875 sit inside each other's intervals; only the shared row set + is sharp, and only for arms we ran ourselves. +- **The GGUF ladder is flat where the MLX ladder was not.** The 2026-09-18 entry + above found nvfp4 and bf16 differing by a 9 px tier on Metal once the kernel + was fixed. Same model family, same benchmark family, opposite result — so + "quantization is costing us a tier" is a finding about a specific + quantization and kernel, never a general property of the checkpoint. +- **Enforced by** — `summarize_extbench.py --paired` (H7 generator) computes the + discordant counts and an **exact** McNemar. Exact, not chi-square: at 3-vs-1 + the continuity-corrected approximation gives p≈0.317 against the true 0.625, + and 0.317 is the kind of number that gets reported as "trending". + `test_summarizers.py::TestExtbenchSummary` pins both the exact values and the + H13 footer behaviour. +- **Cost** — none this time, because the paired columns were computed before + the numbers were written up. The near-miss is that the first draft of the + status report quoted 171 vs 175 across backends as if the gap meant something. diff --git a/docs/maxusai/vision-suite/README.md b/docs/maxusai/vision-suite/README.md index 4264d7ece2e..4386406ebda 100644 --- a/docs/maxusai/vision-suite/README.md +++ b/docs/maxusai/vision-suite/README.md @@ -108,6 +108,14 @@ first is image accounting and the second is generation length. `refcoco` mode reports the winning coordinate dialect and JSON key per item, so it doubles as a dialect probe. See [../vision-benchmark-survey.md](../vision-benchmark-survey.md) for why the external harnesses' own grounding scorers cannot be trusted with our models. +- `summarize_extbench.py [--dir D] [--paired] ` — renders `ext_*.json` as one + markdown table with an H13 provenance footer. **Tables from external benchmarks are pasted + from this, never retyped** (SPEC H7). `--paired` adds the per-pair discordant counts and an + exact McNemar over the items both arms scored: two accuracies on a shared row set cannot + distinguish a real difference from a coin-flip disagreement, and the discordant columns can. + Measured 2026-09-19: `gemma4:31b` q8_0 and bf16 came back with **zero** discordant items over + 200 OCRBench rows while their accuracy column read 0.845 for both and q4_K_M read 0.855 + ([../ocrbench-gemma4-quant-ladder.md](../ocrbench-gemma4-quant-ladder.md)). - `run_grid.sh ` — model × think-mode grid against one host, with an optional restart hook between runs (see below). Budgets are **per think-mode** and set by the runner: think-off `num_predict` 4000, think-on `num_ctx - CTX_PROMPT_RESERVE` (8192 at diff --git a/docs/maxusai/vision-suite/extbench.py b/docs/maxusai/vision-suite/extbench.py index cd4b6312816..363f730e6c3 100644 --- a/docs/maxusai/vision-suite/extbench.py +++ b/docs/maxusai/vision-suite/extbench.py @@ -308,6 +308,10 @@ def main(): f"endpoint={os.environ.get('ENDPOINT', 'generate')}") records, correct, ious, dialects, empty = [], 0, [], {}, 0 + # H11: collected as SETS, not scalars. A container restart mid-run + # (the vision suite has had them) would otherwise let one build + # vouch for rows another build served. + hosts, builds = set(), set() for i, row in enumerate(rows): src = row["image"]["src"] if isinstance(row["image"], dict) else row["image"] path = cache_image(BENCH, offset + i, src) @@ -323,6 +327,8 @@ def main(): print(f"{offset + i:>4} ERROR {e}") continue dt = round(time.time() - t0, 1) + hosts.add(r.get("_host")) + builds.add(r.get("_server_version")) if not pred.strip(): empty += 1 @@ -362,6 +368,10 @@ def main(): # only one of these cannot distinguish those. "think_env": think, "think_on": think == "on", "endpoint": os.environ.get("ENDPOINT", "generate"), + # H11: where it ran and which build served it. client.generate() + # already stamps both on every response; this only persists them. + "host": sorted(h for h in hosts if h) or None, + "server_version": sorted(b for b in builds if b) or None, "correct": correct, "accuracy": round(correct / n, 4) if n else None, } if BENCH == "refcoco": diff --git a/docs/maxusai/vision-suite/summarize_extbench.py b/docs/maxusai/vision-suite/summarize_extbench.py new file mode 100644 index 00000000000..5dc3ea13837 --- /dev/null +++ b/docs/maxusai/vision-suite/summarize_extbench.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Render `extbench.py` runs as one markdown table, with an H13 provenance footer. + +Usage: summarize_extbench.py [--dir D] [--paired] [tag ...] +e.g. summarize_extbench.py ocrbench ocr1np_q4_k_m ocr1np_q8_0 ocr1np_bf16 + +Exists because SPEC `vision-harness-reuse.md` H7 requires tables to be emitted by a +generator and pasted verbatim: an external-benchmark score that is hand-typed into a +document asserts a number nothing checks. `summarize_engine_compare.py` renders the +in-house suite; nothing rendered `ext_*.json` until this. + +H5: `load` and `fmt_bool` are imported from `summarize_engine_compare.py`, never +redefined. + +H13: the footer is built from the `host` / `server_version` fields of every file +rendered. Files written before extbench recorded them show as **pre-H11 run (not +recorded)** rather than silently inheriting a sibling's provenance, and a mix of +hosts, builds or recording states renders the MIXED banner that ADR 0012 +convention 10 makes non-publishable. + +With `--paired`, also runs McNemar's exact test between every pair of arms over the +items both scored. Aggregate accuracies cannot distinguish "these backends differ" +from "they disagree on a handful of coin-flips"; on a shared row set the discordant +pairs can. +""" +import json +import math +import os +import sys + +from summarize_engine_compare import fmt_bool, load # noqa: F401 (H5) + +DIR = os.path.dirname(os.path.abspath(__file__)) + +NOT_RECORDED = "pre-H11 run (not recorded)" + + +def path_for(bench, tag, rundir=DIR): + return os.path.join(rundir, f"ext_{tag}_{bench}.json") + + +def provenance(summary): + """(hosts, builds) as displayable strings. Absent fields are NOT guessed.""" + def one(key): + v = summary.get(key) + if not v: + return NOT_RECORDED + return ", ".join(v) if isinstance(v, list) else str(v) + return one("host"), one("server_version") + + +def mcnemar_exact(b, c): + """Two-sided exact binomial p for b discordant one way, c the other. + + Exact rather than chi-square because the discordant counts here are single + digits. The uncorrected chi-square is anticonservative there -- 3-vs-1 reads + 0.317 against the true 0.625 -- and while the continuity correction recovers + most of it (0.617), the exact test needs no such caveat and costs nothing at + these counts. + """ + n = b + c + if n == 0: + return 1.0 + k = min(b, c) + tail = sum(math.comb(n, i) for i in range(k + 1)) / (2 ** n) + return min(1.0, 2 * tail) + + +def main(): + argv = list(sys.argv[1:]) + paired = "--paired" in argv + argv = [a for a in argv if a != "--paired"] + rundir = DIR + if "--dir" in argv: + k = argv.index("--dir") + rundir = argv[k + 1] + del argv[k:k + 2] + if len(argv) < 2: + sys.exit(__doc__) + bench, tags = argv[0], argv[1:] + + rows, hosts, builds, per_item = [], set(), set(), {} + for tag in tags: + d = load(path_for(bench, tag, rundir)) + if d is None: + print(f"") + continue + s = d["summary"] + h, b = provenance(s) + hosts.add(h) + builds.add(b) + rows.append(s) + per_item[tag] = {r["i"]: bool(r.get("ok")) for r in d["records"] if "error" not in r} + + if not rows: + sys.exit("no score files loaded") + + acc_key = "acc@0.5" if "acc@0.5" in rows[0] else "accuracy" + print(f"| model | scored | errors | empty | correct | {acc_key} | think | endpoint |") + print("|---|---|---|---|---|---|---|---|") + for s in rows: + print(f"| `{s['model']}` | {s['scored']} | {s['errors']} | {s['empty_responses']} | " + f"{s['correct']} | **{s[acc_key]}** | {s['think_env']} | {s['endpoint']} |") + + r0 = rows[0] + print(f"\n{r0['benchmark']} — `{r0['dataset']}` [{r0['split']}], " + f"rows {r0['offset']}..{r0['offset'] + r0['requested']}.") + + if len(hosts) > 1 or len(builds) > 1: + print("\n⚠ **MIXED — rows are not one campaign** " + f"(hosts: {sorted(hosts)}; builds: {sorted(builds)})") + else: + print(f"\nhost: {hosts.pop()} · build: {builds.pop()}") + + if paired and len(tags) > 1: + print(f"\n| pair | both ✓ | both ✗ | A only | B only | McNemar exact p |") + print("|---|---|---|---|---|---|") + for i in range(len(tags)): + for j in range(i + 1, len(tags)): + a, b = tags[i], tags[j] + shared = sorted(set(per_item[a]) & set(per_item[b])) + both = sum(1 for k in shared if per_item[a][k] and per_item[b][k]) + neither = sum(1 for k in shared if not per_item[a][k] and not per_item[b][k]) + aonly = sum(1 for k in shared if per_item[a][k] and not per_item[b][k]) + bonly = sum(1 for k in shared if not per_item[a][k] and per_item[b][k]) + p = mcnemar_exact(aonly, bonly) + print(f"| {a} vs {b} | {both} | {neither} | {aonly} | {bonly} | {p:.3f} |") + + +if __name__ == "__main__": + main() diff --git a/docs/maxusai/vision-suite/test_summarizers.py b/docs/maxusai/vision-suite/test_summarizers.py index 917c243eaeb..c306abb31ae 100644 --- a/docs/maxusai/vision-suite/test_summarizers.py +++ b/docs/maxusai/vision-suite/test_summarizers.py @@ -32,6 +32,7 @@ import summarize_head_to_head as shh # noqa: E402 import summarize_contract_matrix as scm # noqa: E402 import summarize_reps as reps # noqa: E402 +import summarize_extbench as sxb # noqa: E402 import vision_suite as vs # noqa: E402 # One model, two think cells. Numbers chosen so every assertion below can name @@ -1951,5 +1952,84 @@ def test_a_missing_campaign_is_named_not_silently_empty(self): out = self._run(["a_1_", "gone_1_", "alpha:1b"]) self.assertIn("missing `scores_gone_1_alpha_1b_thinkfalse.json`", out) + +class TestExtbenchSummary(unittest.TestCase): + """`summarize_extbench.py` renders external-benchmark runs (H7). + + Two things here are not obvious from reading the table. The footer must not + let an H11-recording file vouch for one that records nothing — the same + defect H13 was written for, reached by a different summarizer. And the + paired test must stay exact: at these discordant counts the chi-square + approximation is anticonservative, which is the direction that manufactures + a difference between two quantizations that agree. + """ + + def setUp(self): + self.dir = tempfile.mkdtemp() + + def _write(self, tag, oks, host=None, ver=None, bench="ocrbench"): + recs = [{"i": i, "ok": ok} for i, ok in enumerate(oks)] + summary = { + "tag": tag, "model": f"m:{tag}", "benchmark": bench, + "dataset": "echo840/OCRBench", "split": "test", + "offset": 0, "requested": len(oks), "scored": len(oks), + "errors": 0, "empty_responses": 0, + "think_env": "false", "think_on": False, "endpoint": "generate", + "correct": sum(oks), "accuracy": round(sum(oks) / len(oks), 4), + } + if host: + summary["host"] = [host] + if ver: + summary["server_version"] = [ver] + path = os.path.join(self.dir, f"ext_{tag}_{bench}.json") + with open(path, "w") as f: + json.dump({"summary": summary, "records": recs}, f) + + def _run(self, *tags, paired=False): + argv = ["summarize_extbench.py", "--dir", self.dir, "ocrbench", *tags] + if paired: + argv.append("--paired") + out = io.StringIO() + with mock_argv(argv), contextlib.redirect_stdout(out): + sxb.main() + return out.getvalue() + + def test_a_file_without_h11_fields_is_named_not_silently_dropped(self): + self._write("noprov", [1, 1, 0]) + self.assertIn(sxb.NOT_RECORDED, self._run("noprov")) + + def test_a_recorded_file_never_vouches_for_an_unrecorded_one(self): + self._write("good", [1, 1, 0], host="http://h:1", ver="0.34.1-x") + self._write("noprov", [1, 0, 0]) + out = self._run("good", "noprov") + self.assertIn("MIXED", out) + self.assertIn(sxb.NOT_RECORDED, out) + + def test_one_campaign_renders_a_clean_footer(self): + self._write("a", [1, 1, 0], host="http://h:1", ver="0.34.1-x") + self._write("b", [1, 0, 0], host="http://h:1", ver="0.34.1-x") + out = self._run("a", "b") + self.assertNotIn("MIXED", out) + self.assertIn("build: 0.34.1-x", out) + + def test_identical_arms_report_zero_discordant_pairs(self): + self._write("a", [1, 1, 0, 1], host="h", ver="v") + self._write("b", [1, 1, 0, 1], host="h", ver="v") + row = [ln for ln in self._run("a", "b", paired=True).splitlines() + if ln.startswith("| a vs b")][0] + self.assertIn("| 3 | 1 | 0 | 0 | 1.000 |", row) + + def test_mcnemar_is_exact_at_small_discordant_counts(self): + # 3 vs 1 is the observed q4-vs-q8 split. The exact two-sided p is + # 2 * (C(4,0) + C(4,1)) / 2**4 = 0.625. The UNCORRECTED chi-square + # gives 0.317 -- the number that would read as a near-significant + # difference between two quantizations. (With continuity correction it + # is 0.617, close to exact here; the correction is what most references + # mean by "McNemar's test", and dropping it is the easy mistake.) + self.assertAlmostEqual(sxb.mcnemar_exact(3, 1), 0.625, places=6) + self.assertEqual(sxb.mcnemar_exact(0, 0), 1.0) + self.assertAlmostEqual(sxb.mcnemar_exact(5, 0), 0.0625, places=6) + + if __name__ == "__main__": unittest.main(verbosity=2)