glennneuber · Sep 19, 2026 · Sep 18, 2026 · Sep 18, 2026 · Sep 18, 2026 · Sep 19, 2026
diff --git a/README.md b/README.md
@@ -29,6 +29,11 @@
 > **Deployed:** the same tag, stamped `0.34.1-dynres-0-g8a7ba94`, on the CUDA host since
 > 2026-09-18, with `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0` on the container (ADR 0033; see the fold's
 > memory finding in [`docs/maxusai/tasks/upstream-sync-0.34.1.md`](docs/maxusai/tasks/upstream-sync-0.34.1.md)).
+> On the Apple Silicon host the same commit, stamped `0.34.0-maxusai-8a7ba949` by `build-macos.sh`, has
+> served the mlx-metal surface on `:11435` since 2026-09-18, promoted on 2026-09-19 with the MLX #3912
+> kernel fix kept ([ADR 0037](docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md)). Its launchd
+> environment does **not** yet set `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0`, so drafting under a grammar is on
+> there, unlike the CUDA container.
 
 > Fork builds are stamped `<upstream-version>-dynres-<n>-g<sha>`; `dynres`
 > names the change that started the fork, not the company that runs it.
@@ -43,13 +48,22 @@
            --version <fold-version> docs/maxusai/vision-suite/preflight/runs/<full-run>.json
      Feed it the FULL run only: the generator takes the newest run per surface,
      so a later smoke (which deliberately skips probes) would overwrite green
-     cells with "skipped". The release notes carry the same generated matrix. -->
+     cells with "skipped". The release notes carry the same generated matrix.
+
+     For v0.34.1 the rows come from TWO hosts' runs, each generated separately
+     and pasted verbatim: the cuda row from the CUDA host's post-deploy run
+     (not committed), the mlx-metal row from
+       release_matrix.py runs/preflight-mlx-metal-0340-8a7ba949.json
+     (committed). The Metal build stamps 0.34.0-maxusai-<sha> rather than
+     0.34.1-dynres-<n>-g<sha> (ADR 0037, Consequences), so a single
+     `--version 0.34.1-dynres` run would drop it. Regenerating both rows from one
+     command needs the CUDA run committed and the stamps reconciled. -->
 
 | surface | Build identity | Image size ladder | Pinned image budget | thinking on/off | Output quality | fp16 overflow canary | Runner isolation | measured on |
 |---|---|---|---|---|---|---|---|---|
 | **cuda** | green | green | green | green | skipped | green | green | `0.34.1-dynres-0-g8a7ba94` |
 | **mlx-cuda** | not run | not run | not run | not run | not run | not run | not run | — |
-| **mlx-metal** | not run | not run | not run | not run | not run | not run | not run | — |
+| **mlx-metal** | skipped | green | skipped | green | not run | skipped | green | `0.34.0-maxusai-8a7ba949` |
 | **apple-silicon-mlx** | not run | not run | not run | not run | not run | not run | not run | — |
 | **rocm** | not run | not run | not run | not run | not run | not run | not run | — |
 | **cpu** | not run | not run | not run | not run | not run | not run | not run | — |

diff --git a/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md b/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md
@@ -0,0 +1,121 @@
+# ADR 0037: keep the MLX #3912 kernel fix — decided on correctness, with OCR accuracy measured neutral
+
+- **Status:** accepted 2026-09-19 (Glenn: "keep 0.34.x and promote to production").
+  Sits beside [ADR 0036](0036-gemma4-image-chunk-decodes-in-one-batch.md), which made the
+  same trade on the GGUF path for the same cell.
+- **Date:** 2026-09-19
+- **Deciders:** MaxusAI fork maintainers
+
+## Context
+
+The v0.34.1 fold moved the MLX pin to `d9add9d1`, which carries upstream
+[ml-explore/mlx#3912](https://github.com/ml-explore/mlx/pull/3912): Metal's `fp_qmm_t` ran its K
+loop past `K_eff` when **K mod 32 == 16**, reading 16 columns of the next row's packed weights and
+scales into the accumulator. The gemma4 vision tower's `mlp.down_proj` is nvfp4 `[1152 × 4304]`,
+K = 134 × 32 + 16, in the 26b and 31b checkpoints (27 layers each). 12b has no `vision_tower.encoder`
+at all, only a `vision_embedder`, and never met the defect.
+
+On Metal the fold moved one scored cell: `gemma4:31b-nvfp4`'s 9 px fine-text tier, 4 → 3. That raised
+the question this ADR answers — **should the fork revert the pin, or carry a patch restoring the old
+kernel behaviour, to get the tier back?**
+
+The evidence, all measured on Apple Silicon `10.8.0.3` against the deployed build
+`0.34.0-maxusai-8a7ba949` and the archived `0.33.2-maxusai-2b95b4a5`:
+
+**The fixed kernel is arithmetically correct and the old one was not.** The guard test
+(`x/mlxrunner/mlx/fp_qmm_t_kmod32_test.go`, #315) compares `QuantizedMatmul` against
+dequantize-then-matmul at M = 256, N = 1152, group 16:
+
+```
+MLX ce916dbb (pre-fix)   K=4288  max 0.000152588   0/294912 elements off by > 1.0
+                         K=4304  max 24.0494       232722/294912
+MLX d9add9d1 (post-fix)  K=4288  max 0.000152588   0/294912
+                         K=4304  max 0.137939      0/294912
+```
+
+`TestVisionGoldenParity`'s max sampled element delta — the fused quantized matmul against
+dequantize-then-matmul on real weights — improves `0.1406 → 0.1094` on 31b across the fold and to
+`0.0898` with the separate global-scale round trip removed, **exactly** MLX-CUDA's `0.0898`, where the
+defective kernel never existed ([upstream-sync-0.34.1.md](../tasks/upstream-sync-0.34.1.md)
+§ "The MLX pin move and the vision encoder", #316).
+
+**The 9 px tier is not a readout of encoder correctness.** Ten reps per quantization, think-off, one
+binary and window (`bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json`):
+
+| checkpoint | vision `down_proj` | rest of tower | LM | 9 px at 4 |
+|---|---|---|---|---|
+| `31b-mlx-bf16` | bf16 | bf16 | bf16 | **10/10** |
+| `31b-mxfp8` | bf16 | 8-bit | 8-bit | **0/10** |
+| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | **2/10** |
+
+`31b-mxfp8` carries a bf16 `down_proj` — the exact tensor #3912 corrupts — and still scores below the
+all-4-bit checkpoint. The cell does not order by numerical precision, and it is bimodal: across both
+of the suite's finetext arms it read 4 on 14 of 14 pre-fix captures and 1 of 16 post-fix, with one
+0.34.0 pair of captures disagreeing at byte-identical provenance by a single character (`RNK-0391-DW18`
+against `RMK-0391-DW18`, ground truth `RNK`).
+
+It is also the cell ADR 0036 already moved. On the GGUF path, decoding a gemma4 image chunk in one
+batch — the bidirectionally faithful path — takes 31b's 9 px tier 4 → 3 deterministically, and the
+fork accepted that as the cost of the faithful computation. **Two unrelated correctness fixes, on two
+engines, move this cell the same way.**
+
+**On real OCR the defect is undetectable.** All 1000 OCRBench v1 items, the same local checkpoint
+under both binaries (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json`):
+
+```
+0.34.0-maxusai-8a7ba949 (MLX d9add9d1, fixed)    835/1000 = 0.8350
+0.33.2-maxusai-2b95b4a5 (MLX c793734e, defect)   833/1000 = 0.8330
+
+both right 826 | both wrong 158 | fixed only 9 | defect only 7
+discordant 16   exact McNemar two-tailed p = 0.8036
+```
+
+## Decision
+
+1. **Keep the MLX pin at `d9add9d1`, with #3912.** No revert of the pin, and no compat patch
+   reintroducing the pre-fix `fp_qmm_t` loop bound.
+2. **The decision rests on correctness, not accuracy.** There is no accuracy argument on either side —
+   OCRBench cannot tell the two kernels apart — and there is a correctness argument on one. A fork that
+   carries a known-wrong kernel because one glyph scored higher would be optimising the benchmark,
+   not the model.
+3. **A fine-text size tier does not gate a pin, kernel or quantization decision on its own.** A tier
+   move across a build is a prompt to measure (the unquantized arm on the same binary and window, a
+   rate rather than one capture, and a many-item external benchmark), not a verdict. The measurement
+   rules are [SPEC vision-harness-reuse](../spec/vision-harness-reuse.md) H15–H18.
+4. **The mlx-metal surface is promoted on this fold.** `0.34.0-maxusai-8a7ba949` serves `:11435`; its
+   preflight run (`preflight/runs/preflight-mlx-metal-0340-8a7ba949.json`, profile `mlx-metal-0-34-0`,
+   PASS 19 / SKIP 12) is recorded, and the README's release matrix and deploy line carry it.
+
+## Options considered
+
+- **Revert the MLX pin to the pre-fix commit.** Restores the tier on one checkpoint. Rejected: it
+  reinstates a kernel wrong on 79 % of that matmul's outputs at the served shape, for a cell that
+  OCRBench shows carries no accuracy, and it drops every other fix in the pin range.
+- **Carry a compat patch restoring the old loop bound.** Same outcome as reverting with more
+  maintenance, and a patch whose whole purpose is to reintroduce an upstream-fixed memory over-read.
+- **Keep the fix but gate the fold on the 9 px tier until it returns to 4.** Rejected: the tier does not
+  order by precision (the bf16-`down_proj` checkpoint scores 0/10), and ADR 0036 already accepted the
+  same move on the other engine.
+- **Keep the fix (chosen).**
+
+## Consequences
+
+- **Every Metal 26b/31b vision number taken before `8a7ba949` went through the defective kernel** and is
+  superseded by [the 2026-09-18 campaign](../vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md).
+- **The registry has drifted from the checkpoints measured here.** Per the MLX-CUDA session's registry
+  audit (#312), the library's current `gemma4:31b-nvfp4` ships a bf16 vision tower where the checkpoint
+  measured here has a 4-bit one — 194 layers differ. A bf16-tower checkpoint never meets #3912 on any
+  build. The two share a config digest, which is expected and means nothing: the config digest is the
+  architecture's `config.json`, shared even by our local nvfp4 and bf16 checkpoints. The measured
+  checkpoints are identified by manifest digest (`31b-nvfp4` `637cc0ff…`, `31b-mxfp8` `1434769c…`,
+  `31b-mlx-bf16` `fb3f25b3…`) and preserved under tower-qualified tags
+  (`gemma4:31b-nvfp4-tower-nvfp4`, `-mxfp8-tower-mxfp8`, `-mlx-bf16-tower-bf16`), whose manifest digests
+  match the originals, so a pull cannot silently replace what was measured.
+- **The build stamps disagree with ADR 0032.** The Metal build script stamps `0.34.0-maxusai-<sha>`; the
+  fold and the CUDA image stamp `0.34.1-dynres-0-g<sha>` for the same commit. `release_matrix.py
+  --version 0.34.1-dynres` would therefore drop the Metal run. Not fixed here; the mlx-metal matrix row
+  is generated from its own run and recorded alongside the CUDA row.
+- **Still open:** the nvfp4 global-scale representation (raw `m` vs `× 2688 / 2688`), which blocks #287
+  and is independent of this decision — the round trip measurably moves the encoder by one f32 ulp on 17
+  of 31b's 191 vision scales and provably does not move the 9 px tier.
+- **Retirement:** none. This is upstream's fix, carried by the pin, not a fork change.
diff --git a/docs/maxusai/spec/vision-harness-reuse.md b/docs/maxusai/spec/vision-harness-reuse.md
@@ -2,7 +2,9 @@
 
 MaxusAI-fork specification. Status: **implemented** — `run_engine_compare.sh`
 carries `REPEATS` / `TAG_PREFIX` / `ONLY_TESTS`, and `summarize_reps.py`
-imports its helpers from `summarize_engine_compare.py`. Written 2026-08-17.
+imports its helpers from `summarize_engine_compare.py`. Written 2026-08-17;
+H15–H18 added 2026-09-19 from the 0.34 fold's kernel investigation
+([ADR 0037](../adr/0037-keep-the-mlx-3912-kernel-fix.md)).
 
 Normative rules for adding to `docs/maxusai/vision-suite/`. The decision and its
 evidence are [ADR 0028](../adr/0028-one-runner-one-set-of-helpers.md); report
@@ -231,6 +233,52 @@ identity. The two generations also scored differently (recall_9px 1/4
 against 2/4), which is the measured instance behind ADR 0012 conv. 4's
 think-on non-reproducibility note.
 
+**H15 — One capture of a quality cell is one draw, not a value.** A score or
+size tier that differs between two builds is not attributed to anything until
+its RATE is measured on each: at least ten reps per build, same binary, window
+and power mode, reported as `k/N`. `n = 1` stays right for a campaign's
+survey of quality ([ADR 0012](../adr/0012-benchmark-report-templates.md)); it
+is not enough to conclude that a build moved a cell.
+
+> Measured 2026-09-18: `gemma4:31b-nvfp4`'s 9 px tier read 3 on every 0.34.0
+> capture of one finetext arm, and the move was published as a clean split by
+> build. Ten reps put it at 2/10 reading 4, and the other arm already held a 4.
+> At greedy, temperature 0.
+
+**H16 — Both finetext arms count, and a tally that reads one is half the
+evidence.** Every campaign cell carries two INDEPENDENT fine-text generations:
+`vision_suite.py`'s folded arm (`scores_<tag>.json["finetext"]`) and
+`finetext_probe.py` (`ft_<tag>.json`). Any count over captures reads both, and
+reports disagreement between them as a result rather than resolving it.
+
+The rendered table does NOT show both: `summarize_engine_compare.py` takes
+`scores.get("finetext")` and falls back to `ft_<tag>.json` only when the suite
+arm is absent, so where both exist the second draw is never rendered. H14's
+note recorded the arms disagreeing at think-on; 2026-09-18 extends it to
+think-off at temperature 0. On `0.34.0` one `31b-nvfp4` cell's two arms had
+byte-identical provenance — same `prompt_sha`, `images_sha`, window,
+`eval_count=263` and `answer_chars=351` — and differed by one character,
+`RNK-0391-DW18` against `RMK-0391-DW18`, scoring 4 and 3. In the 2026-09-18
+campaign 2 of 7 cells' arms disagree on a tier. A count of "twelve runs split
+cleanly by build" was published from the `ft_` arm alone and was wrong.
+
+**H18 — A cross-build accuracy claim needs a paired test on many items, and a
+tier move needs the unquantized arm.** Two builds compared on a benchmark are
+compared item by item on the SAME items with an exact McNemar test on the
+discordant pairs — never by diffing two headline accuracies: at 200 items near 85 % the
+standard error of the difference is about 3.4 points, so a real gap needs to be
+roughly seven points before two headline numbers can show it. A unanimous split cannot reach
+significance below six discordant pairs, so a 200-item slice routinely cannot
+answer the question at all. Before a fine-text or recall tier is read as
+evidence about a kernel, pin or quantization, the checkpoint with no
+quantization in the path (`*-mlx-bf16`) is measured on the same binary and
+window.
+
+> Measured 2026-09-18/19: a 200-item OCRBench slice gave 4–0 to the fixed
+> kernel; all 1000 items gave 9–7, p = 0.80. The slice was also
+> unrepresentative (per-chunk accuracy 0.875 … 0.665), so its headline 87.5 %
+> was not the checkpoint's 83.5 %.
+
 ## 3. Before writing anything
 
 **H8 — Check the inventory first.** `vision-suite/README.md` §Files lists every
@@ -371,6 +419,32 @@ something previously hidden:
 - A cell with no `prompt_sha` predates 2026-08-20. Its workload is whatever its
   campaign document says, which is why campaign docs quote the arm and the date.
 
+**H17 — A checkpoint is its manifest digest, not its tag and not its config
+digest.** Registry tags are mutable: the library's current `gemma4:31b-nvfp4`
+carries a bf16 vision tower, while the checkpoint every 0.34 Metal number was
+measured on carries a 4-bit one — 194 vision layers differ (the MLX-CUDA
+session's registry audit, #312), and a pull replaces one with the other under an
+unchanged name.
+
+The **config** digest cannot tell them apart. It is the architecture's
+`config.json`, identical across quantizations of one model: locally
+`31b-nvfp4` and `31b-mlx-bf16` both carry `b72c5344…`. What identifies a
+checkpoint is the sha256 of its **manifest**, which changes when any layer does
+(`31b-nvfp4` `637cc0ff…`, `31b-mxfp8` `1434769c…`, `31b-mlx-bf16`
+`fb3f25b3…`). A campaign document records the manifest digest of every
+checkpoint it measured, and a checkpoint that must stay reproducible is
+preserved under a tag naming its composition before any pull — the copy's
+manifest digest must equal the original's (`gemma4:31b-nvfp4-tower-nvfp4` is
+`637cc0ff…`).
+
+Two checkpoints share a component only when their FULL layer sets match by
+digest. One matching tensor proves one tensor: `31b-mxfp8` and `31b-mlx-bf16`
+share a byte-identical vision `down_proj` blob (digest `4b80a6f9…`), and the
+claim that they share a bf16 tower was published from that tensor alone — 162 of
+the 356 vision layers are 8-bit in `31b-mxfp8`. `quant_dims.py` lists a
+checkpoint's quantized weights; diff the manifests by `name` and `digest`
+before asserting what two checkpoints have in common.
+
 ## 4. Conformance
 
 | requirement | enforced by |
@@ -388,4 +462,8 @@ something previously hidden:
 | H12 | `prompt_sha` / `images_sha` / `prompt_parts` on every score block, written by `client.generate()`; absence marks a pre-2026-08-20 cell |
 | H11 | `host` / `server_version` on every score block, written unconditionally by `client.generate()`; absence marks a pre-2026-08-20 cell |
 | H10 | `client.RETRY_BACKOFF` = 5/15/30s with `_retries` recorded per cell; `test_client.py::TestTransportRetry` asserts a 400 calls `urlopen` exactly once while a 503 retries. `client.evict_others()` polls `/api/ps` until the eviction is observable and returns what it could not evict; `run_engine_compare.sh` calls it before each model when `RESTART_CMD` is absent, `COLD_START=0` opts out |
+| H15 | **Nothing enforces this.** The 2026-09-18 replication (`bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json`) is the worked example of reporting a rate |
+| H16 | **Not enforced, and the renderer works against it:** `summarize_engine_compare.py` renders the suite arm and shows the probe arm only as a fallback. A render that reported both arms, or flagged their disagreement, would enforce it |
+| H17 | Partly: `quant_dims.py` (#316) lists a checkpoint's quantized weights and their K. Nothing records the manifest digest on a score block, and nothing diffs two checkpoints' layer sets; the config digest a reader might reach for instead identifies the architecture, not the checkpoint |
+| H18 | **Nothing enforces this.** `extbench.py` produces per-item records that a paired test can use; `bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json` carries the paired items and the exact McNemar result |
 | H8 | **Nothing enforces this.** It is a reading habit, and it is the one that would have prevented all three incidents |