From 9ead1b6be4215ecab23b9ea7f561fc533c5aaed9 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sun, 11 Oct 2026 10:43:09 +1000 Subject: [PATCH 1/3] tasks: claim the muse-glimmer onboarding on Metal; the suite's think-on card for it The plan for testing Meta's muse-glimmer on the Metal host, MLX and GGUF tags with and without their DFlash drafter, against production's 0.40.2 release build on :11437. Prepared, not run: the registry facts, what the code leads us to expect, five phases and their pass criteria, and the no-GPU gates' results. sampling.py gains a muse-glimmer think-on entry (temperature from THINK_TEMPERATURE, top_p/top_k from the packaged params), without which think-on cells would run at the packaged temperature 1. Co-Authored-By: Claude Opus 5.5 --- docs/maxusai/tasks/muse-glimmer-onboard.md | 163 +++++++++++++++++++++ docs/maxusai/vision-suite/sampling.py | 5 + 2 files changed, 168 insertions(+) create mode 100644 docs/maxusai/tasks/muse-glimmer-onboard.md diff --git a/docs/maxusai/tasks/muse-glimmer-onboard.md b/docs/maxusai/tasks/muse-glimmer-onboard.md new file mode 100644 index 00000000000..0397422b91f --- /dev/null +++ b/docs/maxusai/tasks/muse-glimmer-onboard.md @@ -0,0 +1,163 @@ +# TASK: onboard Meta's muse-glimmer on Metal (MLX and GGUF, with and without its DFlash drafter) + +Opened by the Metal host on 2026-10-11, while the maintainer pulls the tags. Status: **prepared, not run.** Nothing below +has touched a GPU yet; the no-GPU gates have run (section "Phase 0"). + +## The model, as the registry has it + +`muse-glimmer` is a 30B (27.9B) text+image model with thinking and tools, 128K context, Apache 2.0 +(ollama.com/library/muse-glimmer). Read from the registry's manifests and config blobs, 2026-10-11: + +| tag (as pulled) | runner | weights | drafter | size | note | +|---|---|---|---|---|---| +| `30b-nvfp4` | MLX | NVFP4 (modelopt mixed) | none | 17.3 GB | | +| `30b-mlx` | MLX | NVFP4 | DFlash, 58 `draft.*` tensors | 19.1 GB | **the same manifest as `30b-nvfp4-dflash`**, not an 8-bit build | +| `30b-mxfp8` | MLX | MXFP8 | none | 32.6 GB | the 8-bit MLX build | +| `30b-mxfp8-dflash` | MLX | MXFP8 | DFlash | 35.2 GB | | +| `30b` | llama.cpp | Q4_K_M | none | 18.2 GB | the same manifest as `30b-q4_K_M` and `latest` | +| `30b-q4_K_M-dflash` | llama.cpp | Q4_K_M | DFlash GGUF, 1.6 GB `image.draft` layer | 19.8 GB | | +| `30b-q8_0` | llama.cpp | Q8_0 | none | ~31 GB | | +| `30b-q8_0-dflash` | llama.cpp | Q8_0 | DFlash GGUF | ~33 GB | | + +- **Each (plain, dflash) pair differs only by the drafter.** All 1437 MLX base tensor digests are identical between + `30b-nvfp4` and `30b-nvfp4-dflash`, and between `30b-mxfp8` and `30b-mxfp8-dflash`; each GGUF pair shares its one + model blob. Their quantization metadata JSON differs (different modelopt producer versions; the plain NVFP4 build + lists 300 `vision_encoder` layers the dflash build does not), but that JSON is read only by `ollama create` + (`create/create.go`), not by the runner. The no-draft control in phase 1 checks the claim. +- **Packaged parameters:** temperature 1, top_k 64, top_p 0.95 on every tag. The MLX dflash tags add + `draft_num_predict: 15`, the GGUF ones `draft_num_predict: 3`. +- **Drafter:** `MuseGlimmerAssistantModel`, 5 sliding-window layers, block size 16. It reads target layers + `[1, 13, 25, 37, 49]` (MLX `dflash_config.target_layer_ids`); the GGUF drafter's `dflash.target_layers` is + `[2, 14, 26, 38, 50]`. That is presumably the same layers with a different index base; not verified. +- **Vision:** patch 14, downsample 2, so stride 28. +- The 8 pulled tags hold 102.8 GB of unique blobs. 603 GB was free when the pulls started. + +## What the code says (origin/main 5623192cc = production's 0.40.2-dynres-0-g5623192) + +The fork carries upstream's glimmer support unchanged, except for MLX image budgets (ADR 0021), the +draft-under-grammar knob, and the ADR 0039 scale identity in the DFlash drafter. + +- **Thinking:** think accepts `false`, `"low"`, `"medium"`, `"high"` and `"max"`. Omitted or `true` means `"high"` + (`model/renderers/glimmer.go:66-77`). Any other value is a 400 (`api/types.go:1255`). `false` renders + "Reasoning strength: none." and the parser drops any thinking that is produced anyway. +- **Images, both runners:** aspect-preserving grid, never upscaled, capped at 4096 tokens. + - MLX: `mlxrunner/model/glimmer/media.go:21,125`. + - llama.cpp b11351: `PROJECTOR_TYPE_MUSE_GLIMMER`, `set_limit_image_tokens(1, 4096)`. + - None of the fork's clip patches touch this projector. +- **Request image-token options:** + - MLX honours `image_max_tokens` through `ResolveImageBudget`. Expected trap: a value of 1120 + (`api.DefaultImageMaxTokens`) reads as unset, so 4096. + - GGUF ignores them: `muse-glimmer` is not in `llm/llama_server.go` `visionServerArgs`. +- **GGUF context fit:** + - `MaxImageTokens` and `ImageTokensForSize` have no glimmer case, so the fit check charges 768 per image + (`llm/llama_server.go:1297-1340`), against a real cost of up to about 4098. + - Many large images against a small `num_ctx` is the case to probe. +- **Drafting:** + - MLX drafts whenever the manifest has a drafter, at an adaptive depth. It does not draft for logprobs requests + (`mlxrunner/speculate.go:148`) or, in production's environment, under a format/grammar (`DRAFT_UNDER_GRAMMAR=0`). + It does not read `draft_num_predict`. + - GGUF passes `--spec-type draft-dflash --spec-draft-n-max --spec-draft-model ` + (`llm/llama_server.go:862-890`). llama-server logs `draft acceptance = …`. + +## Phases + +Everything runs on production's own release build, `~/.ollama/binaries/stage-0.40.2-dynres-0-g5623192`, which is +pinned by `EXPECT_STAMP`. It runs: +- on `127.0.0.1:11437`; +- in production's environment: TWO_PASS=1, KV f16, DRAFT_UNDER_GRAMMAR=0, and MAX_LOADED_MODELS=1; +- against an APFS clone of production's store, `~/.ollama/models-mlx-glimmer`. + +Every GPU request waits until production has been idle 15 minutes with powermode 2, and is cancelled and re-run if +production wakes. The drivers live in the worktree's `.campaign/`: + +- `env.sh`: the build, store, port, tags and pairs. +- `stage-gl.sh`: the stage, which is also the restart hook. +- `gate.sh` and `leg.sh`: the per-step gate. +- `drv/drive-gl.sh`: the per-request gate, from `drive-0402.sh` with its port and paths repointed. + +| phase | what | how | verdict | +|---|---|---|---| +| 0 | no-GPU gates | the glimmer Go tests (below); `step-inventory.sh`: manifests byte-identical to the registry, blobs complete, drafters where expected; clone the store | pass/fail | +| 1 | functional smoke, all 8 tags | `leg.sh smoke` → `glimmer_probe.py` → `runs/smoke.json` | PASS/FAIL + INFO | +| 2 | preflight baseline rows | `leg.sh ladder` → `measure_ladder.py`, arch `muse-glimmer`, stride 28: `30b-nvfp4` on `mlx-metal-0-40-2`, `30b` on a new `metal-0-40-2` | rows to paste, measured | +| 3 | vision suite, both think modes | `campaign-gl.sh` (3a off, 3b on at `"high"`): 27 cases + finetext per cell, every tag | scores (ADR 0012 tables) | +| 4 | OCRBench v1 rows 0-199, think off | `campaign-gl.sh` | accuracy per tag | +| 5 | what the drafter buys | `leg.sh tps` → `gl_tps.py`: ABBA × 5 per pair, prose think-off and a think-high reasoning run | tok/s, acceptance | + +### Phase 1, the probe's checks (`glimmer_probe.py`) + +Each check below is PASS/FAIL against a known right answer: +- `show`: the capabilities include completion, vision, tools and thinking. +- `text_think_off`: 17 × 23 = 391, with no thinking. +- `think_levels`: false, low, medium, high, max and omitted, on the bat-and-ball question (0.05). There must be + thinking exactly when it is not `false`. It records tokens to finish at each level, uncapped (num_predict 24576). +- `think_invalid_level`: "extreme" must return 400. +- `tools_round_trip`: think off and default. It must call `get_weather(city=Paris)` and then use the tool's 18 °C. +- `format_schema`: think off and default. The output must be valid JSON for the schema, with thinking exactly when + on (TWO_PASS=1). +- `image_read`: the invoice number `INV-2026-0801` off `visimgs/document.png`. + +Each check below is INFO, measured with no expectation yet: +- `route`: which runner served the tag, the llama-server flags, and whether a drafter loaded. +- `image_token_ladder`: image tokens for 448², 768², 1024×768, 1920×480, 1568², 2048² and 2560×1440 test cards. + Each count is the difference against the same text-only prompt, with a fresh system nonce per request so the + prompt cache cannot shrink it. +- `image_token_options`: `image_max_tokens` set to unset, 1024, 1119, 1120, 1121 and 2048 on a 2048² card, plus + `image_min_tokens` 2048 on a 448² card. This is where the 1120 trap and the GGUF no-op show. +- `ctx_fit_3x2048_at_8k`: three 2048² cards at num_ctx 8192. An error is the honest answer; a 200 with a short + `prompt_eval_count` is silent truncation. +- `longgen`: 600 greedy tokens, run twice, plain and with `logprobs`, which is MLX's no-draft control. It records + tok/s, the drafting lines from the log, and whether the text equals the plain tag's. + +### What would count as a finding + +- **Any phase-1 FAIL.** +- **Drafting changes greedy text.** A drafting tag's greedy text differs from its plain tag's and, on MLX, the + logprobs control matches. That is the drafter itself changing greedy output. Verification is not exact on Metal (drafted text depends on draft depth, which is + wall-clock driven; the `drv/drive-gl.sh` header). +- **The logprobs control differs too.** That means the two tags do not load identically despite identical tensors. +- **GGUF silently truncates images** in `ctx_fit_3x2048_at_8k`. +- **MLX and GGUF image-token counts disagree** at equal sizes. They are meant to run one algorithm. + +## Phase 0, run 2026-10-11 + +On the release worktree at `v0.40.2-dynres`, which is production's build: + +``` +ok github.com/ollama/ollama/model/renderers 0.746s +ok github.com/ollama/ollama/model/parsers 1.255s +ok github.com/ollama/ollama/create 0.530s +ok github.com/ollama/ollama/server 0.918s +ok github.com/ollama/ollama/mlxrunner 0.690s +ok github.com/ollama/ollama/mlxrunner/model 0.471s +ok github.com/ollama/ollama/mlxrunner/model/glimmer 0.508s +``` + +The commands were: +- `go test ./model/renderers ./model/parsers ./create -run Glimmer` +- `./server -run TestGlimmer` +- `./mlxrunner -run 'TestDraftUnderGrammarFromEnv|TestDraftingEnabled|TestDFlash|TestDecodeBlockDraft'` +- `./mlxrunner/model -run TestResolveImageBudget` +- `./mlxrunner/model/glimmer -run 'TestParseConfig|TestParseOfficialHFConfig|TestValidateTokenizerEOS|TestComputeImageSize|TestPrepareMedia'` + +`step-inventory.sh` runs once the pulls finish. + +## Changes in this branch + +- `docs/maxusai/vision-suite/sampling.py`: a `muse-glimmer` think-on entry. Temperature comes from + `THINK_TEMPERATURE` (0), and top_p 0.95 / top_k 64 from the packaged params, which are the only published values. + Without it, think-on cells would send no sampling keys and run at the packaged temperature 1. +- **After phase 2:** the measured `[expect.*.muse-glimmer]` rows. `muse-glimmer` joins `mlx-metal-0-40-2`'s arches, + and a `metal-0-40-2` profile is added with its version-pattern test. The newest llama.cpp-path Metal profile is + `metal-0-32-14`, so 0.40.2's GGUF path has no profile today. + +## Not covered + +- **The bf16 tags** (`30b-bf16(-dflash)`, `30b-mlx-bf16(-dflash)`) were not pulled. With no bf16 arm, a quality gap + between NVFP4/MXFP8/Q4/Q8 cannot be attributed to quantization (the tier-move rule). +- **Think levels in the suite.** The suite measures think-on at the default `"high"` only; levels appear only in the + phase-1 probe. +- **Audio** (glimmer declares none) and **tools beyond one round trip.** +- **CUDA and ROCm.** This is the Metal host's run. + +`macbook-pro-m5-max-128GB/mlx-metal` diff --git a/docs/maxusai/vision-suite/sampling.py b/docs/maxusai/vision-suite/sampling.py index b078edc2b49..3cb4952e644 100644 --- a/docs/maxusai/vision-suite/sampling.py +++ b/docs/maxusai/vision-suite/sampling.py @@ -110,6 +110,11 @@ # no distinction between thinking and non-thinking, and specifies no # penalty; measured convergence needs none. "gemma4": {"temperature": THINK_TEMPERATURE, "top_p": 0.95, "top_k": 64}, + # muse-glimmer: the ollama.com library's packaged params, the only published + # values (the library page states none): every tag carries temperature 1 / + # top_k 64 / top_p 0.95 (registry params blobs 56380ca2ab89, and 0b6cd7fba9a7 + # and 2365fbb6d97b on the -dflash tags, which add draft_num_predict). + "muse-glimmer": {"temperature": THINK_TEMPERATURE, "top_p": 0.95, "top_k": 64}, # nemotron3: the NVIDIA card is gated (HTTP 401), so no values are recorded # here. It is the one family that converges under greedy decoding anyway, so # the fallback below is not currently costing us a measurement. From 8b9662d20bd798db6d2912b5d54c8e9dad2f0db1 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sun, 11 Oct 2026 14:13:03 +1000 Subject: [PATCH 2/3] tasks: muse-glimmer interim results -- phases 0-2, the fixes they produced, and glimmer against gemma4:31b think-off Phases 0-2 done (inventory, smoke on all eight tags, preflight ladders); phase 3 in progress. The findings and their draft PRs (#464-#468), and summarize_head_to_head.py's table of glimmer against gemma4:31b, thinking off, pasted as printed with its MIXED-build provenance. Several glimmer cells sit on the format + think-off path #464 fixes; the rglfix_ re-run is the comparison to read. Also corrects the record's unknown-think-level line (it falls back to the default; upstream #18473) and the probe check that follows from it. Co-Authored-By: Claude Opus 5.5 --- docs/maxusai/tasks/muse-glimmer-onboard.md | 61 ++++++++++++++++++++-- 1 file changed, 57 insertions(+), 4 deletions(-) diff --git a/docs/maxusai/tasks/muse-glimmer-onboard.md b/docs/maxusai/tasks/muse-glimmer-onboard.md index 0397422b91f..e6524dad3db 100644 --- a/docs/maxusai/tasks/muse-glimmer-onboard.md +++ b/docs/maxusai/tasks/muse-glimmer-onboard.md @@ -1,7 +1,7 @@ # TASK: onboard Meta's muse-glimmer on Metal (MLX and GGUF, with and without its DFlash drafter) -Opened by the Metal host on 2026-10-11, while the maintainer pulls the tags. Status: **prepared, not run.** Nothing below -has touched a GPU yet; the no-GPU gates have run (section "Phase 0"). +Opened by the Metal host on 2026-10-11, while the maintainer pulls the tags. Status: **running.** Phases 0-2 are done; +phases 3-5 are in progress. Interim results, and the fixes they produced, are in "Interim results" below. ## The model, as the registry has it @@ -38,7 +38,7 @@ The fork carries upstream's glimmer support unchanged, except for MLX image budg draft-under-grammar knob, and the ADR 0039 scale identity in the DFlash drafter. - **Thinking:** think accepts `false`, `"low"`, `"medium"`, `"high"` and `"max"`. Omitted or `true` means `"high"` - (`model/renderers/glimmer.go:66-77`). Any other value is a 400 (`api/types.go:1255`). `false` renders + (`model/renderers/glimmer.go:66-77`). Any other string falls back to that default with no error (upstream #18473, `model/renderers/thinking.go:30-45`); the 400 in `api/types.go:1255` applies only to models without thinking metadata. `false` renders "Reasoning strength: none." and the parser drops any thinking that is produced anyway. - **Images, both runners:** aspect-preserving grid, never upscaled, capped at 4096 tokens. - MLX: `mlxrunner/model/glimmer/media.go:21,125`. @@ -91,7 +91,7 @@ Each check below is PASS/FAIL against a known right answer: - `text_think_off`: 17 × 23 = 391, with no thinking. - `think_levels`: false, low, medium, high, max and omitted, on the bat-and-ball question (0.05). There must be thinking exactly when it is not `false`. It records tokens to finish at each level, uncapped (num_predict 24576). -- `think_invalid_level`: "extreme" must return 400. +- `think_unknown_level`: "extreme" falls back to the default ("high"), as upstream #18473 specifies: 200, with thinking. - `tools_round_trip`: think off and default. It must call `get_weather(city=Paris)` and then use the tool's 18 °C. - `format_schema`: think off and default. The output must be valid JSON for the schema, with thinking exactly when on (TWO_PASS=1). @@ -142,6 +142,59 @@ The commands were: `step-inventory.sh` runs once the pulls finish. +## Interim results, 2026-10-11 + +### Phases 0-2 +- **Phase 0:** all eight tags match the registry and their blobs are complete, and every `-dflash` tag carries its drafter. +- **Phase 1:** the smoke probe ran on every tag. Every check passed except one: `format` with thinking off, which the + four GGUF tags failed on the release build (`{...}<|eot|>`). It found five issues, four of them now fixed in drafts: + +| # | finding | PR | +|---|---|---| +| 1 | `format` + `think: false` bound the grammar before glimmer's message header, in single pass and two-pass alike | #464 | +| 2 | `draft_num_predict` ignored on MLX | #465 | +| 3 | MLX admission priced glimmer as weights only (kvsize could not parse its config) | #466 | +| 4 | `image_max_tokens: 1120` read as unset on MLX, so 4096 | #468 (ADR 0048) | +| 5 | MLX does not enforce `num_ctx` on prompt length (upstream ollama#18125, intentional upstream) | open question | + +- **Phase 2:** both runners measured the same ladder, `[47, 182, 779, 2995, 4082]` (#467). + +### Against gemma4:31b, thinking off (phase 3, in progress) +glimmer's cells are this campaign's (`rgl_`), on the release build. gemma4's are the v0.40.2 fold's gate 6 +(`r0402fold_`). Both used the same suite, thinking off, `num_ctx` 16384, `num_predict` 2200, greedy. 23 of 27 cases had +byte-identical prompts and images (`prompt_sha`, `images_sha`). The four `bboxm_*_pos` cases ask each family for its +own box format and are not in this table. `summarize_head_to_head.py`, as printed: + +| test | metric | rgl_1_muse-glimmer_30b-nvfp4_thinkfalse | r0402fold_1_gemma4_31b-nvfp4_thinkfalse | rgl_1_muse-glimmer_30b_thinkfalse | r0402fold_1_gemma4_31b-it-q4_K_M_thinkfalse | +|---|---|---|---|---|---| +| scene | bbox IoU | 0.827 (16384) | 0.962 (16384) | 0.711 (16384) | 0.964 (16384) | +| scene | labels / serial | 6/6, ✅ | 6/6, ✅ | 6/6, ✅ | 6/6, ✅ | +| document | items / qty+price / total / invoice | 5/5, 5/5, ✅, ✅ | 5/5, 5/5, ✅, ✅ | 5/5, 5/5, ✅, ✅ | 5/5, 5/5, ✅, ✅ | +| document | name_bbox IoU | 0.774 (16384) | 0.750 (16384) | 0.819 (16384) | 0.751 (16384) | +| fine text | 22/16/12/9/7 px | 4/4/4/4/0 (16384) | 4/4/4/3/3 (16384) | 0/0/0/0/0 (16384) | 4/4/4/4/3 (16384) | +| multi (3 img) | q1 / q2 / q4-bbox / chart | capped (131072) | ✅ ✅ ✅ 5/5 (16384) | ❌ ❌ ❌ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | +| multi (3 img, anchored) | q1 / q2 / q4-bbox / chart | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | +| throughput | gen tok/s | 20 | 19 | 23 | 12 | +| throughput | prefill tok/s | 5138 | 1649 | 4915 | 2089 | +| latency | s/req (unique image) | 18.4 | 30.0 | 16.4 | 45.5 | +| latency | req/h (serial) | 196 | 120 | 220 | 79 | + +Provenance (from score files): host(s) http://127.0.0.1:11436, http://127.0.0.1:11437 · build(s) 0.35.0-dynres-27-g948ef3a, 0.40.2-dynres-0-g5623192 ⚠ MIXED — columns are not one campaign + +**Reading it:** +- **Document extraction:** a tie, and glimmer boxes the document's names slightly better. +- **Scene boxes:** glimmer's clear weakness so far, at 0.83 (MLX) and 0.71 (GGUF) against gemma4's 0.96. +- **Fine text on MLX:** glimmer reads 9 px, which gemma4 misses, but nothing at 7 px. +- **Speed:** glimmer is much faster to first answer, with about 3x the prefill rate and 18 s against 30 s per request on MLX. These are single runs (n=1): a lead for phase 5, not a result. + +**Not glimmer's own score:** several of glimmer's cells take the format + think-off path that #464 fixes. +- **GGUF fine text:** the reply was `{"codes": []}<|eot|>`, an empty answer forced by the grammar binding before the header. +- **MLX `multi_3img`:** the model wrote its header inside the JSON and then degenerated. +- **What to compare instead:** the think-off suite re-run on #464's build (`rglfix_`) gives the comparison to read against gemma4. It is queued behind phases 3-5. + +**Columns:** the generator flags them as MIXED. gemma4's cells ran on the fold's interim build (`0.35.0-dynres-27-g948ef3a`, the +fold's tree), glimmer's on the release build. + ## Changes in this branch - `docs/maxusai/vision-suite/sampling.py`: a `muse-glimmer` think-on entry. Temperature comes from From 4c5ffea11361177fd323d77cc7d2d2053bb37736 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Tue, 13 Oct 2026 23:29:20 +1000 Subject: [PATCH 3/3] tasks: the muse-glimmer onboarding is done -- final results for phases 1-5, the fixes it produced, and the follow-ups Phases 0-5 ran on all eight tags. The record's outcome section lists the five fixes now in v0.40.2-dynres.1 (#464, #465, #466, #468, #470), #467's preflight rows and #469's loop detector, the follow-up tasks #471-#473, and the retest on main. Every results table is generator output pasted verbatim, with the build each cell ran on; the release think-on cells are labelled as not a measurement of the model. Co-Authored-By: Claude Opus 5.5 --- docs/maxusai/tasks/muse-glimmer-onboard.md | 555 ++++++++++++++------- 1 file changed, 388 insertions(+), 167 deletions(-) diff --git a/docs/maxusai/tasks/muse-glimmer-onboard.md b/docs/maxusai/tasks/muse-glimmer-onboard.md index e6524dad3db..693e9bbbf83 100644 --- a/docs/maxusai/tasks/muse-glimmer-onboard.md +++ b/docs/maxusai/tasks/muse-glimmer-onboard.md @@ -1,7 +1,31 @@ # TASK: onboard Meta's muse-glimmer on Metal (MLX and GGUF, with and without its DFlash drafter) -Opened by the Metal host on 2026-10-11, while the maintainer pulls the tags. Status: **running.** Phases 0-2 are done; -phases 3-5 are in progress. Interim results, and the fixes they produced, are in "Interim results" below. +Opened by the Metal host on 2026-10-11. Status: **done** (2026-10-13). Phases 0-5 ran on all eight pulled tags. The +results are in "Results" below; what is still open is in "Not covered". + +## Outcome + +**Five fixes came out of this onboarding. All five are merged on main and are in the `v0.40.2-dynres.1` point tag:** +- **#464, glimmer's format path:** + - With thinking off, a format waits for glimmer's message header instead of binding before it. This was the GGUF `{...}<|eot|>` reply, in single pass and two-pass alike. + - With thinking on, two-pass pass two opens the header before binding the format. Without it, the model echoed the prompt as JSON. +- **#465:** MLX reads `draft_num_predict` per request: 0 turns drafting off and N caps the depth. +- **#466:** MLX admission prices glimmer's KV cache. The estimator now reads the list form of `sliding_window_pattern`; before, it priced weights only. +- **#468:** an explicit `image_max_tokens: 1120` is honoured, and an unset budget reaches the runner as 0 (ADR 0048). +- **#470:** JSON whitespace is bounded at llama.cpp's limit on both runners. Glimmer in single pass padded MLX objects up to the old bound. + +**Also from this campaign:** +- #469, the vision suite's loop detector (ADR 0049): merged. +- #467, the measured preflight rows for both Metal stacks: merged 2026-10-13, after a re-run on `v0.40.2-dynres.1`. + +**Follow-ups, opened as task records for other agents (unclaimed):** +- #471: why Q8_0 grounds worse with thinking on, and the `30b-q8_0-dflash` think-on cell that never ran on a fixed build. +- #472: why no tag reads the 7 px fine text, and whether a larger image budget could help. +- #473: why a glimmer suite cell is 1.6-3× slower than a gemma4:31b cell. + +**Retest on main.** `muse-glimmer:30b-q4_K_M-dflash` was re-run with thinking off on main `601b8a65c`, which is +code-identical to `v0.40.2-dynres.1` outside `docs/`. It reads fine text 4/4/4/4/0, with 0 errors and a scene IoU of +0.726, the same as on #464's build. The table is under phase 3a. ## The model, as the registry has it @@ -19,198 +43,395 @@ phases 3-5 are in progress. Interim results, and the fixes they produced, are in | `30b-q8_0` | llama.cpp | Q8_0 | none | ~31 GB | | | `30b-q8_0-dflash` | llama.cpp | Q8_0 | DFlash GGUF | ~33 GB | | -- **Each (plain, dflash) pair differs only by the drafter.** All 1437 MLX base tensor digests are identical between - `30b-nvfp4` and `30b-nvfp4-dflash`, and between `30b-mxfp8` and `30b-mxfp8-dflash`; each GGUF pair shares its one - model blob. Their quantization metadata JSON differs (different modelopt producer versions; the plain NVFP4 build - lists 300 `vision_encoder` layers the dflash build does not), but that JSON is read only by `ollama create` - (`create/create.go`), not by the runner. The no-draft control in phase 1 checks the claim. +- **Each (plain, dflash) pair differs only by the drafter.** The plain and dflash tags of each MLX pair share all + 1437 base tensor digests, and each GGUF pair shares its one model blob. Their modelopt quantization JSON differs, but only + `ollama create` reads it. - **Packaged parameters:** temperature 1, top_k 64, top_p 0.95 on every tag. The MLX dflash tags add `draft_num_predict: 15`, the GGUF ones `draft_num_predict: 3`. -- **Drafter:** `MuseGlimmerAssistantModel`, 5 sliding-window layers, block size 16. It reads target layers - `[1, 13, 25, 37, 49]` (MLX `dflash_config.target_layer_ids`); the GGUF drafter's `dflash.target_layers` is - `[2, 14, 26, 38, 50]`. That is presumably the same layers with a different index base; not verified. +- **Drafter:** `MuseGlimmerAssistantModel`, 5 sliding-window layers, block size 16. + - MLX target layers `[1, 13, 25, 37, 49]`, GGUF `[2, 14, 26, 38, 50]`. + - Presumably the same layers with a different index base; not verified. - **Vision:** patch 14, downsample 2, so stride 28. -- The 8 pulled tags hold 102.8 GB of unique blobs. 603 GB was free when the pulls started. - -## What the code says (origin/main 5623192cc = production's 0.40.2-dynres-0-g5623192) - -The fork carries upstream's glimmer support unchanged, except for MLX image budgets (ADR 0021), the -draft-under-grammar knob, and the ADR 0039 scale identity in the DFlash drafter. - -- **Thinking:** think accepts `false`, `"low"`, `"medium"`, `"high"` and `"max"`. Omitted or `true` means `"high"` - (`model/renderers/glimmer.go:66-77`). Any other string falls back to that default with no error (upstream #18473, `model/renderers/thinking.go:30-45`); the 400 in `api/types.go:1255` applies only to models without thinking metadata. `false` renders - "Reasoning strength: none." and the parser drops any thinking that is produced anyway. -- **Images, both runners:** aspect-preserving grid, never upscaled, capped at 4096 tokens. - - MLX: `mlxrunner/model/glimmer/media.go:21,125`. - - llama.cpp b11351: `PROJECTOR_TYPE_MUSE_GLIMMER`, `set_limit_image_tokens(1, 4096)`. - - None of the fork's clip patches touch this projector. -- **Request image-token options:** - - MLX honours `image_max_tokens` through `ResolveImageBudget`. Expected trap: a value of 1120 - (`api.DefaultImageMaxTokens`) reads as unset, so 4096. - - GGUF ignores them: `muse-glimmer` is not in `llm/llama_server.go` `visionServerArgs`. -- **GGUF context fit:** - - `MaxImageTokens` and `ImageTokensForSize` have no glimmer case, so the fit check charges 768 per image - (`llm/llama_server.go:1297-1340`), against a real cost of up to about 4098. - - Many large images against a small `num_ctx` is the case to probe. -- **Drafting:** - - MLX drafts whenever the manifest has a drafter, at an adaptive depth. It does not draft for logprobs requests - (`mlxrunner/speculate.go:148`) or, in production's environment, under a format/grammar (`DRAFT_UNDER_GRAMMAR=0`). - It does not read `draft_num_predict`. - - GGUF passes `--spec-type draft-dflash --spec-draft-n-max --spec-draft-model ` - (`llm/llama_server.go:862-890`). llama-server logs `draft acceptance = …`. -## Phases - -Everything runs on production's own release build, `~/.ollama/binaries/stage-0.40.2-dynres-0-g5623192`, which is -pinned by `EXPECT_STAMP`. It runs: -- on `127.0.0.1:11437`; -- in production's environment: TWO_PASS=1, KV f16, DRAFT_UNDER_GRAMMAR=0, and MAX_LOADED_MODELS=1; -- against an APFS clone of production's store, `~/.ollama/models-mlx-glimmer`. - -Every GPU request waits until production has been idle 15 minutes with powermode 2, and is cancelled and re-run if -production wakes. The drivers live in the worktree's `.campaign/`: - -- `env.sh`: the build, store, port, tags and pairs. -- `stage-gl.sh`: the stage, which is also the restart hook. -- `gate.sh` and `leg.sh`: the per-step gate. -- `drv/drive-gl.sh`: the per-request gate, from `drive-0402.sh` with its port and paths repointed. - -| phase | what | how | verdict | -|---|---|---|---| -| 0 | no-GPU gates | the glimmer Go tests (below); `step-inventory.sh`: manifests byte-identical to the registry, blobs complete, drafters where expected; clone the store | pass/fail | -| 1 | functional smoke, all 8 tags | `leg.sh smoke` → `glimmer_probe.py` → `runs/smoke.json` | PASS/FAIL + INFO | -| 2 | preflight baseline rows | `leg.sh ladder` → `measure_ladder.py`, arch `muse-glimmer`, stride 28: `30b-nvfp4` on `mlx-metal-0-40-2`, `30b` on a new `metal-0-40-2` | rows to paste, measured | -| 3 | vision suite, both think modes | `campaign-gl.sh` (3a off, 3b on at `"high"`): 27 cases + finetext per cell, every tag | scores (ADR 0012 tables) | -| 4 | OCRBench v1 rows 0-199, think off | `campaign-gl.sh` | accuracy per tag | -| 5 | what the drafter buys | `leg.sh tps` → `gl_tps.py`: ABBA × 5 per pair, prose think-off and a think-high reasoning run | tok/s, acceptance | - -### Phase 1, the probe's checks (`glimmer_probe.py`) - -Each check below is PASS/FAIL against a known right answer: -- `show`: the capabilities include completion, vision, tools and thinking. -- `text_think_off`: 17 × 23 = 391, with no thinking. -- `think_levels`: false, low, medium, high, max and omitted, on the bat-and-ball question (0.05). There must be - thinking exactly when it is not `false`. It records tokens to finish at each level, uncapped (num_predict 24576). -- `think_unknown_level`: "extreme" falls back to the default ("high"), as upstream #18473 specifies: 200, with thinking. -- `tools_round_trip`: think off and default. It must call `get_weather(city=Paris)` and then use the tool's 18 °C. -- `format_schema`: think off and default. The output must be valid JSON for the schema, with thinking exactly when - on (TWO_PASS=1). -- `image_read`: the invoice number `INV-2026-0801` off `visimgs/document.png`. - -Each check below is INFO, measured with no expectation yet: -- `route`: which runner served the tag, the llama-server flags, and whether a drafter loaded. -- `image_token_ladder`: image tokens for 448², 768², 1024×768, 1920×480, 1568², 2048² and 2560×1440 test cards. - Each count is the difference against the same text-only prompt, with a fresh system nonce per request so the - prompt cache cannot shrink it. -- `image_token_options`: `image_max_tokens` set to unset, 1024, 1119, 1120, 1121 and 2048 on a 2048² card, plus - `image_min_tokens` 2048 on a 448² card. This is where the 1120 trap and the GGUF no-op show. -- `ctx_fit_3x2048_at_8k`: three 2048² cards at num_ctx 8192. An error is the honest answer; a 200 with a short - `prompt_eval_count` is silent truncation. -- `longgen`: 600 greedy tokens, run twice, plain and with `logprobs`, which is MLX's no-draft control. It records - tok/s, the drafting lines from the log, and whether the text equals the plain tag's. - -### What would count as a finding - -- **Any phase-1 FAIL.** -- **Drafting changes greedy text.** A drafting tag's greedy text differs from its plain tag's and, on MLX, the - logprobs control matches. That is the drafter itself changing greedy output. Verification is not exact on Metal (drafted text depends on draft depth, which is - wall-clock driven; the `drv/drive-gl.sh` header). -- **The logprobs control differs too.** That means the two tags do not load identically despite identical tensors. -- **GGUF silently truncates images** in `ctx_fit_3x2048_at_8k`. -- **MLX and GGUF image-token counts disagree** at equal sizes. They are meant to run one algorithm. +## What the code said before the campaign (origin/main 5623192cc = the release build) -## Phase 0, run 2026-10-11 +The fork carried upstream's glimmer support unchanged, apart from MLX image budgets (ADR 0021), the +draft-under-grammar knob and the ADR 0039 drafter scale identity. +- **Thinking:** `false`, `"low"` … `"max"`; omitted or `true` is `"high"` (`model/renderers/glimmer.go:66-77`). +- **Images:** an aspect-preserving 28-pixel grid, never upscaled, capped at 4096 tokens, on both runners. GGUF + ignores the image-token options, and its context fit charges 768 per glimmer image (`llm/llama_server.go:1297-1340`). +- **The expected traps, both now fixed:** MLX read `image_max_tokens: 1120` as unset (#468), and ignored + `draft_num_predict` (#465). -On the release worktree at `v0.40.2-dynres`, which is production's build: - -``` -ok github.com/ollama/ollama/model/renderers 0.746s -ok github.com/ollama/ollama/model/parsers 1.255s -ok github.com/ollama/ollama/create 0.530s -ok github.com/ollama/ollama/server 0.918s -ok github.com/ollama/ollama/mlxrunner 0.690s -ok github.com/ollama/ollama/mlxrunner/model 0.471s -ok github.com/ollama/ollama/mlxrunner/model/glimmer 0.508s -``` - -The commands were: -- `go test ./model/renderers ./model/parsers ./create -run Glimmer` -- `./server -run TestGlimmer` -- `./mlxrunner -run 'TestDraftUnderGrammarFromEnv|TestDraftingEnabled|TestDFlash|TestDecodeBlockDraft'` -- `./mlxrunner/model -run TestResolveImageBudget` -- `./mlxrunner/model/glimmer -run 'TestParseConfig|TestParseOfficialHFConfig|TestValidateTokenizerEOS|TestComputeImageSize|TestPrepareMedia'` +## Phases -`step-inventory.sh` runs once the pulls finish. +Every phase ran under production's environment: TWO_PASS=1, KV f16, DRAFT_UNDER_GRAMMAR=0 and MAX_LOADED_MODELS=1. +- **Where:** on `127.0.0.1:11437`, against an APFS clone of production's store (`~/.ollama/models-mlx-glimmer`). +- **Gate:** every GPU request waited for production to be idle 15 minutes, with powermode 2. A request was cancelled and re-run if production woke. +- **Drivers:** the onboarding worktree's `.campaign/` (not in git): `env.sh`, `stage-gl.sh`, `gate.sh`/`leg.sh`, + `drv/drive-gl.sh`, and `campaign-gl{,-fix,-stock}.sh`. -## Interim results, 2026-10-11 +**Builds, and which cells ran on each.** The version stamp is read from every score file's `server_version`: -### Phases 0-2 -- **Phase 0:** all eight tags match the registry and their blobs are complete, and every `-dflash` tag carries its drafter. -- **Phase 1:** the smoke probe ran on every tag. Every check passed except one: `format` with thinking off, which the - four GGUF tags failed on the release build (`{...}<|eot|>`). It found five issues, four of them now fixed in drafts: +| build | stamp | what ran on it | +|---|---|---| +| release (production then) | `0.40.2-dynres-0-g5623192` | phases 0-2, 4 and 5; phase 3 in both modes on all 8 tags (prefix `rgl_`) | +| #464, first commit `4cfad4ab8` | `0.40.2-dynres-1-g4cfad4a` | phase 3 think-off, all 8 tags (`rglfix_`) | +| #464, second commit `4b7c3b7c9` | `0.40.2-dynres-2-g4b7c3b7` | phase 3 think-on, the 4 plain tags (`rglfix_`) | +| stock ollama 0.40.2 | `0.40.2` | smoke on all 8 tags; phases 3 and 4 on `30b-nvfp4` and `30b` (`rglstock_`, `ocrglstock_`) | +| main `601b8a65c` | `0.40.2-dynres-35-g601b8a6` | phase 3 think-off, `30b-q4_K_M-dflash` only (the retest above) | -| # | finding | PR | +| phase | what | how | |---|---|---| -| 1 | `format` + `think: false` bound the grammar before glimmer's message header, in single pass and two-pass alike | #464 | -| 2 | `draft_num_predict` ignored on MLX | #465 | -| 3 | MLX admission priced glimmer as weights only (kvsize could not parse its config) | #466 | -| 4 | `image_max_tokens: 1120` read as unset on MLX, so 4096 | #468 (ADR 0048) | -| 5 | MLX does not enforce `num_ctx` on prompt length (upstream ollama#18125, intentional upstream) | open question | +| 0 | no-GPU gates | the glimmer Go tests; `step-inventory.sh`: manifests byte-identical to the registry, blobs complete, drafters where expected | +| 1 | functional smoke, all 8 tags | `leg.sh smoke` → `glimmer_probe.py` → `runs/smoke.json` | +| 2 | preflight baseline rows | `leg.sh ladder` → `measure_ladder.py`, arch `muse-glimmer`, stride 28 | +| 3 | vision suite, both think modes | 27 cases + finetext per cell, cold start, greedy. Think off: `num_ctx` 16384, `num_predict` 2200. Think on: at `"high"`, temperature 0 with the packaged top_p/top_k, `num_ctx` ladder from 16384 | +| 4 | OCRBench v1 rows 0-199, think off | `/api/generate`, `num_ctx` 16384, `num_predict` 512, temperature 0 | +| 5 | what the drafter buys | `gl_tps.py`: ABBA × 5 per pair, a 512-token prose answer with thinking off, and a think-high reasoning run | -- **Phase 2:** both runners measured the same ladder, `[47, 182, 779, 2995, 4082]` (#467). +**The release think-on cells are not a measurement of the model.** In that build, two-pass pass two lacked glimmer's header, so format requests echoed the prompt or looped. The think-on +cells to read are #464's. The release think-off cells take the format + think-off path #464's first commit fixes. +They are the "before"; #464's think-off cells are the measurement. -### Against gemma4:31b, thinking off (phase 3, in progress) -glimmer's cells are this campaign's (`rgl_`), on the release build. gemma4's are the v0.40.2 fold's gate 6 -(`r0402fold_`). Both used the same suite, thinking off, `num_ctx` 16384, `num_predict` 2200, greedy. 23 of 27 cases had -byte-identical prompts and images (`prompt_sha`, `images_sha`). The four `bboxm_*_pos` cases ask each family for its -own box format and are not in this table. `summarize_head_to_head.py`, as printed: +## Phase 0, run 2026-10-11 -| test | metric | rgl_1_muse-glimmer_30b-nvfp4_thinkfalse | r0402fold_1_gemma4_31b-nvfp4_thinkfalse | rgl_1_muse-glimmer_30b_thinkfalse | r0402fold_1_gemma4_31b-it-q4_K_M_thinkfalse | +- **Go tests:** the glimmer tests in `model/renderers`, `model/parsers`, `create`, `server`, `mlxrunner`, + `mlxrunner/model` and `mlxrunner/model/glimmer` pass at `v0.40.2-dynres`. +- **The inventory:** all eight tags match the registry, their blobs are complete, and every `-dflash` tag carries its drafter. + +## Results + +All suite and OCRBench cells are **n=1**. A difference between two cells is a lead, not a result, unless it +repeats. Throughput columns are single runs, and a surprising number is a trigger for a targeted re-run, not a +conclusion. The score files are gitignored, so they are not in git. They live on the Metal host, in the onboarding worktree's +`docs/maxusai/vision-suite/` (`scores_rgl*_1_muse-glimmer_*`, `ft_*`, `ext_ocrgl*_*`). Every table below is +`summarize_engine_compare.py`, `summarize_head_to_head.py`, `summarize_extbench.py` or `gl_tps.py` output, pasted as +printed. + +### Phases 1-2 + +**Phase 1:** every smoke check passed on every tag except one. On the release build, `format_schema` with thinking off +failed on the four GGUF tags, returning `{...}<|eot|>`. +- **What the smoke found:** the four issues #464, #465, #466 and #468 fix (see "Outcome"). +- **One open question:** MLX does not enforce `num_ctx` on prompt length. This is upstream ollama#18125, and it is + intentional upstream. +- **Stock 0.40.2 fails more.** Its smoke fails `format_schema` on all eight tags: + - GGUF fails with thinking off, as the release build did. + - MLX fails with thinking on: stock has single pass only, and the answer is `{` followed by whitespace. #470 bounds that whitespace. +- **#464's build passes `format_schema`:** + - in single pass on `30b-nvfp4`, `30b-mxfp8` and `30b` (`runs/1pass-464.json`); + - in two-pass on `30b` and `30b-q4_K_M-dflash` (`runs/smoke-fix.json`). + +**Phase 2:** both runners measured the same ladder, `[47, 182, 779, 2995, 4082]`. The rows are in #467, merged. + +### Phase 3a, thinking off + +**The measurement: #464's first commit (`0.40.2-dynres-1-g4cfad4a`), all eight tags.** Rendered with +`summarize_engine_compare.py --dir . --think false --prefix rglfix_1_ <8 tags>`. This is the same output as `.campaign/runs/quality-off-glimmer.md`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| muse-glimmer:30b | GGUF | 16384 | 0.726 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-mlx | **MLX** | 16384 | **0.823** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | **0.823** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-mxfp8 | **MLX** | 16384 | **0.681** | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-mxfp8-dflash | **MLX** | 16384 | **0.681** | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-q4_K_M-dflash | GGUF | 16384 | 0.726 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-q8_0 | GGUF | 16384 | 0.712 | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 3/5 | +| muse-glimmer:30b-q8_0-dflash | GGUF | 16384 | 0.712 | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 3/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Answer tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| muse-glimmer:30b | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 756 | 12 | 139 | 85.8 | 42 | +| muse-glimmer:30b-mlx | **MLX** | 16384 | 4 | 4 | 4 | 3 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 735 | 20 | 3229 | 37.0 | 97 | +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 735 | 22 | 3541 | 34.4 | 105 | +| muse-glimmer:30b-mxfp8 | **MLX** | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1127 | 11 | 4663 | 107.5 | 33 | +| muse-glimmer:30b-mxfp8-dflash | **MLX** | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1127 | 11 | 2622 | 106.9 | 34 | +| muse-glimmer:30b-q4_K_M-dflash | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 756 | 26 | 214 | 44.1 | 82 | +| muse-glimmer:30b-q8_0 | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1367 | 11 | 198 | 146.4 | 25 | +| muse-glimmer:30b-q8_0-dflash | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1367 | 22 | 238 | 76.2 | 47 | + +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2-dynres-1-g4cfad4a · think=false + +- **Document extraction is perfect on every tag.** Every tag gets 5/5 items, 5/5 quantity and price, the total and the serial. +- **Scene boxes** are glimmer's weakness against gemma4 (below). nvfp4 scores 0.823 and the 8-bit MLX build 0.681. With n=1 and no bf16 arm, that + gap cannot be put on quantization. +- **Fine text** is 4/4 at 22, 16 and 12 px and 3-4/4 at 9 px, and **0/4 at 7 px on every tag** (#472). +- **The drafter changed no score.** Each (plain, dflash) pair has identical quality columns and the same answer token count. + Each pair was one run per tag, greedy. + +**Against gemma4:31b, thinking off.** gemma4's cells are the v0.40.2 fold's gate 6 (`r0402fold_`, a different build +and port). The generator marks the table MIXED for that reason. Rendered with `summarize_head_to_head.py --tags`, from one directory +holding both sets of score files: + +| test | metric | rglfix_1_muse-glimmer_30b-nvfp4_thinkfalse | r0402fold_1_gemma4_31b-nvfp4_thinkfalse | rglfix_1_muse-glimmer_30b_thinkfalse | r0402fold_1_gemma4_31b-it-q4_K_M_thinkfalse | |---|---|---|---|---|---| -| scene | bbox IoU | 0.827 (16384) | 0.962 (16384) | 0.711 (16384) | 0.964 (16384) | +| scene | bbox IoU | 0.823 (16384) | 0.962 (16384) | 0.726 (16384) | 0.964 (16384) | | scene | labels / serial | 6/6, ✅ | 6/6, ✅ | 6/6, ✅ | 6/6, ✅ | | document | items / qty+price / total / invoice | 5/5, 5/5, ✅, ✅ | 5/5, 5/5, ✅, ✅ | 5/5, 5/5, ✅, ✅ | 5/5, 5/5, ✅, ✅ | -| document | name_bbox IoU | 0.774 (16384) | 0.750 (16384) | 0.819 (16384) | 0.751 (16384) | -| fine text | 22/16/12/9/7 px | 4/4/4/4/0 (16384) | 4/4/4/3/3 (16384) | 0/0/0/0/0 (16384) | 4/4/4/4/3 (16384) | -| multi (3 img) | q1 / q2 / q4-bbox / chart | capped (131072) | ✅ ✅ ✅ 5/5 (16384) | ❌ ❌ ❌ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | +| document | name_bbox IoU | 0.626 (16384) | 0.750 (16384) | 0.697 (16384) | 0.751 (16384) | +| fine text | 22/16/12/9/7 px | 4/4/4/3/0 (16384) | 4/4/4/3/3 (16384) | 4/4/4/4/0 (16384) | 4/4/4/4/3 (16384) | +| multi (3 img) | q1 / q2 / q4-bbox / chart | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | | multi (3 img, anchored) | q1 / q2 / q4-bbox / chart | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | ✅ ✅ ✅ 5/5 (16384) | -| throughput | gen tok/s | 20 | 19 | 23 | 12 | -| throughput | prefill tok/s | 5138 | 1649 | 4915 | 2089 | -| latency | s/req (unique image) | 18.4 | 30.0 | 16.4 | 45.5 | -| latency | req/h (serial) | 196 | 120 | 220 | 79 | +| throughput | gen tok/s | 22 | 19 | 12 | 12 | +| throughput | prefill tok/s | 3541 | 1649 | 139 | 2089 | +| latency | s/req (unique image) | 34.4 | 30.0 | 85.8 | 45.5 | +| latency | req/h (serial) | 105 | 120 | 42 | 79 | + +Provenance (from score files): host(s) http://127.0.0.1:11436, http://127.0.0.1:11437 · build(s) 0.35.0-dynres-27-g948ef3a, 0.40.2-dynres-1-g4cfad4a ⚠ MIXED — columns are not one campaign + +- **Document fields:** a tie. +- **gemma4 is ahead on:** scene boxes (0.82 MLX / 0.73 GGUF against 0.96), name boxes, and 7 px text (0/4 against 3/4). +- **The interim record's speed lead does not hold.** It said glimmer was faster, but that came from release cells that stopped early on the format bug. On #464's build, glimmer + takes 34 s against 30 s per request on MLX and 86 s against 46 s on GGUF (n=1; #473). + +**Retest on main `601b8a65c` (`0.40.2-dynres-35-g601b8a6`), `30b-q4_K_M-dflash` only.** The bundle is committed on +branch `bench/0402-metal`, at +`docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-10-13-metal-0402-glimmer-main.json`. Rendered with +`summarize_engine_compare.py --bundle --think false --prefix m0402fix_1_ muse-glimmer:30b-q4_K_M-dflash`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| muse-glimmer:30b-q4_K_M-dflash | GGUF | 8192/16384 ⚠ | 0.726 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Answer tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| muse-glimmer:30b-q4_K_M-dflash | GGUF | 8192/16384 ⚠ | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 756 | 34 | 282 | 33.8 | 106 | + +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2-dynres-35-g601b8a6 · think=false + +- **The quality columns match #464's cell for this tag.** It ran at that campaign's GGUF settings, and the generator flags its + `num_ctx` (8192/16384) as mixed, so its speed columns do not compare with the cells above. + +**The "before": the release build (`0.40.2-dynres-0-g5623192`), all eight tags. These are not glimmer's score.** They +take the format + think-off path #464 fixes, so they include: +- fine text 0/0/0/0/0 on every GGUF and MXFP8 tag; +- capped cells, with `multi_3img` on nvfp4 climbing to 131072; +- three GGUF HTTP 500s ("token repeat limit"): two on `30b`, one on `30b-q4_K_M-dflash`. + +#464 has the before and after. To render this table, run +`summarize_engine_compare.py --dir . --think false --prefix rgl_1_ <8 tags>` on the Metal host. + +### Phase 3b, thinking on + +**The measurement: #464's second commit (`0.40.2-dynres-2-g4b7c3b7`), the four plain tags.** Rendered with +`summarize_engine_compare.py --dir . --think on --prefix rglfix_1_ <4 plain tags>`. This is the same output as `.campaign/runs/quality-on-glimmer-fix.md`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| muse-glimmer:30b | GGUF | 16384 | 0.670 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | **0.724** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-mxfp8 | **MLX** | 16384 | **0.705** | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-q8_0 | GGUF | 16384 | 0.537 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| muse-glimmer:30b | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1702 | 18 | 251 | 106.8 | 34 | +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1590 | 22 | 942 | 75.8 | 48 | +| muse-glimmer:30b-mxfp8 | **MLX** | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1432 | 12 | 886 | 118.5 | 30 | +| muse-glimmer:30b-q8_0 | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 2681 | 14 | 275 | 201.0 | 18 | + +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2-dynres-2-g4b7c3b7 · think=on + +- **All 27 cases finished at `num_ctx` 16384 on every tag,** with no loop and no cap. +- **Thinking does not help glimmer here.** Scene IoU is lower than thinking off on three of four tags. Q8_0's + 0.537 is the lowest (#471). +- **Fine text** is unchanged from thinking off, including 0/4 at 7 px. + +**gemma4:31b-nvfp4, thinking on, for reference.** This is the regression A/B's release cell (`rrg_`, same build and port). Rendered with +`summarize_engine_compare.py --dir . --think on --prefix rrg_1_ gemma4:31b-nvfp4`. This is the same output as `.campaign/runs/quality-on-gemma.md`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| gemma4:31b-nvfp4 | **MLX** | 16384 | **0.966** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput -Provenance (from score files): host(s) http://127.0.0.1:11436, http://127.0.0.1:11437 · build(s) 0.35.0-dynres-27-g948ef3a, 0.40.2-dynres-0-g5623192 ⚠ MIXED — columns are not one campaign +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| gemma4:31b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 3 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 2961 | 26 | 212 | 122.3 | 29 | -**Reading it:** -- **Document extraction:** a tie, and glimmer boxes the document's names slightly better. -- **Scene boxes:** glimmer's clear weakness so far, at 0.83 (MLX) and 0.71 (GGUF) against gemma4's 0.96. -- **Fine text on MLX:** glimmer reads 9 px, which gemma4 misses, but nothing at 7 px. -- **Speed:** glimmer is much faster to first answer, with about 3x the prefill rate and 18 s against 30 s per request on MLX. These are single runs (n=1): a lead for phase 5, not a result. +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2-dynres-0-g5623192 · think=on -**Not glimmer's own score:** several of glimmer's cells take the format + think-off path that #464 fixes. -- **GGUF fine text:** the reply was `{"codes": []}<|eot|>`, an empty answer forced by the grammar binding before the header. -- **MLX `multi_3img`:** the model wrote its header inside the JSON and then degenerated. -- **What to compare instead:** the think-off suite re-run on #464's build (`rglfix_`) gives the comparison to read against gemma4. It is queued behind phases 3-5. +**The release build, all eight tags. This is not a measurement of the model.** Two-pass pass two lacked glimmer's header. The +results: +- `30b-mxfp8` echoed the prompt as `{"user": "You are a ..."}`. +- The loop detector (ADR 0049) ended 2-3 cases per cell on `30b-mxfp8(-dflash)`, `30b-q4_K_M-dflash` and `30b-q8_0(-dflash)`, and 1 on `30b`. +- The scene zeros on `30b`, `30b-q8_0` and `30b-q8_0-dflash` are on the same path: valid JSON, but no labelled objects in it. + +**These are the only think-on cells for the four `-dflash` tags.** None was re-run on a fixed build. Rendered with +`summarize_engine_compare.py --dir . --think on --prefix rgl_1_ <8 tags>`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| muse-glimmer:30b | GGUF | 16384 | 0.000 | 0/0 · 0/6 · 0/6 | ✅ | 0/5 · 0/5 · ❌ | 0/5 | +| muse-glimmer:30b-mlx | **MLX** | 16384 | **0.733** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 3/5 | +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | **0.733** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 3/5 | +| muse-glimmer:30b-mxfp8 | **MLX** | 16384 | **0.724** | 6/6 · 6/6 · 6/6 | ✅ | 0/5 · 0/5 · ❌ | 0/5 | +| muse-glimmer:30b-mxfp8-dflash | **MLX** | 16384 | **0.743** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-q4_K_M-dflash | GGUF | 16384 | 0.657 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-q8_0 | GGUF | 16384 | 0.000 | 0/0 · 0/6 · 0/6 | ❌ | 5/5 · 5/5 · ✅ | 4/5 | +| muse-glimmer:30b-q8_0-dflash | GGUF | 16384 | 0.000 | 0/0 · 0/6 · 0/6 | ❌ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| muse-glimmer:30b | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ❌ q1_right, q2_right, q4_bbox_hit | — | 2464 | 16 | 221 | 170.4 | 21 | +| muse-glimmer:30b-mlx | **MLX** | 16384 | 4 | 4 | 4 | 3 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1336 | 27 | 953 | 52.8 | 68 | +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 0 | ❌ q4_bbox_hit | ✅ q1 + q2 + q4-bbox | — | 1398 | 20 | 888 | 72.1 | 50 | +| muse-glimmer:30b-mxfp8 | **MLX** | 16384 | 4 | 4 | 4 | 4 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ✅ q1 + q2 + q4-bbox | — | 1137 | 12 | 998 | 101.8 | 35 | +| muse-glimmer:30b-mxfp8-dflash | **MLX** | 16384 | 4 | 4 | 4 | 4 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ✅ q1 + q2 + q4-bbox | — | 2012 | 19 | 686 | 112.6 | 32 | +| muse-glimmer:30b-q4_K_M-dflash | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ❌ q1_right, q2_right, q4_bbox_hit | — | 1784 | 25 | 228 | 85.4 | 42 | +| muse-glimmer:30b-q8_0 | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ❌ q1_right, q2_right, q4_bbox_hit | — | 2409 | 12 | 301 | 218.3 | 16 | +| muse-glimmer:30b-q8_0-dflash | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ❌ q1_right, q2_right, q4_bbox_hit | — | 2429 | 23 | 225 | 120.8 | 30 | + +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2-dynres-0-g5623192 · think=on + +### Phase 3, stock ollama 0.40.2 + +**The upstream comparison, `30b-nvfp4` and `30b`.** Stock has no two-pass, so think-on is single pass. + +**Thinking off** fails the same way the release build did, which makes it upstream's bug (ollama#18687 covers its single-pass +half): +- nvfp4's document comes back 0/5; +- `multi_3img` is capped at 131072; +- the GGUF fine text is 0/0/0/0/0, with two HTTP 500s. + +Rendered with `summarize_engine_compare.py --dir . --think false --prefix rglstock_1_ muse-glimmer:30b-nvfp4 muse-glimmer:30b`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| muse-glimmer:30b-nvfp4 | **MLX** | 16384/131072 ⚠ | **0.839** | 6/6 · 6/6 · 6/6 | ✅ | 0/5 · 0/5 · ❌ | 0/5 | +| muse-glimmer:30b | GGUF | 16384 | 0.711 | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Answer tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| muse-glimmer:30b-nvfp4 | **MLX** | 16384/131072 ⚠ | 4 | 4 | 4 | 4 | 0 | capped | ✅ q1 + q2 + q4-bbox | — | 357 | 13 | 4474 | 28.8 | 125 | +| muse-glimmer:30b | GGUF | 16384 | 0 | 0 | 0 | 0 | 0 | ❌ q1_right, q2_right, q4_bbox_hit | ✅ q1 + q2 + q4-bbox | — | 357 | 25 | 5364 | 15.1 | 238 | + +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2 · think=false + +**Thinking on** is clean, and close to #464's build: scene 0.732 against 0.724 on MLX, and 0.670 against 0.670 on GGUF (n=1). +Rendered with `summarize_engine_compare.py --dir . --think on --prefix rglstock_1_ muse-glimmer:30b-nvfp4 muse-glimmer:30b`: + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | **0.732** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 3/5 | +| muse-glimmer:30b | GGUF | 16384 | 0.670 | 5/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| muse-glimmer:30b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1322 | 21 | 3522 | 64.4 | 56 | +| muse-glimmer:30b | GGUF | 16384 | 4 | 4 | 4 | 4 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 1701 | 12 | 160 | 159.3 | 23 | + +Provenance (from score files): host(s) http://127.0.0.1:11437 · build(s) 0.40.2 · think=on + +### Phase 4, OCRBench v1 rows 0-199, thinking off + +**The release build, all eight tags.** OCRBench sends no format, so it does not take #464's path and was not re-run. Rendered with +`summarize_extbench.py ocrbench ocrgl_muse-glimmer_ ...`: + +| model | scored | errors | empty | correct | accuracy | think | endpoint | +|---|---|---|---|---|---|---|---| +| `muse-glimmer:30b-nvfp4` | 200 | 0 | 0 | 179 | **0.895** | false | generate | +| `muse-glimmer:30b-mlx` | 200 | 0 | 0 | 180 | **0.9** | false | generate | +| `muse-glimmer:30b-mxfp8` | 200 | 0 | 0 | 183 | **0.915** | false | generate | +| `muse-glimmer:30b-mxfp8-dflash` | 200 | 0 | 0 | 183 | **0.915** | false | generate | +| `muse-glimmer:30b` | 200 | 0 | 0 | 182 | **0.91** | false | generate | +| `muse-glimmer:30b-q4_K_M-dflash` | 200 | 0 | 0 | 182 | **0.91** | false | generate | +| `muse-glimmer:30b-q8_0` | 200 | 0 | 0 | 182 | **0.91** | false | generate | +| `muse-glimmer:30b-q8_0-dflash` | 200 | 0 | 0 | 182 | **0.91** | false | generate | + +ocrbench — `echo840/OCRBench` [test], rows 0..200. + +host: http://127.0.0.1:11437 · build: 0.40.2-dynres-0-g5623192 · prompt_cache_ram: 0 + +**Stock 0.40.2.** Rendered with `summarize_extbench.py ocrbench ocrglstock_muse-glimmer_30b-nvfp4 ocrglstock_muse-glimmer_30b`: + +| model | scored | errors | empty | correct | accuracy | think | endpoint | +|---|---|---|---|---|---|---|---| +| `muse-glimmer:30b-nvfp4` | 200 | 0 | 0 | 179 | **0.895** | false | generate | +| `muse-glimmer:30b` | 200 | 0 | 0 | 182 | **0.91** | false | generate | + +ocrbench — `echo840/OCRBench` [test], rows 0..200. + +host: http://127.0.0.1:11437 · build: 0.40.2 · prompt_cache_ram: 0 + +- **Every tag scores 0.895-0.915.** On one 200-item slice the standard error is about 2 points, so the tags are not told apart. +- **The drafter and the fork change almost no items.** `summarize_extbench.py --paired`, run pair by pair, finds: + - `30b-nvfp4` against `30b-mlx`: 1 discordant item; + - each of the other three (plain, dflash) pairs: 0; + - release against stock, on `30b-nvfp4` and on `30b`: 0. + +### Phase 5, what the drafter buys + +**The release build, 2026-10-12.** The run was 5 reps per arm in ABBA order, temperature 0: +- prose: thinking off, 512 tokens; +- reason: think `"high"`, the bat-and-ball question, run to completion. + +The median decode tok/s per (plain, dflash) pair, as `gl_tps.py` prints it. It was re-rendered from `runs/tps.json` with `--reps 0`, which sends no requests: + +``` +== median decode tok/s +prose muse-glimmer:30b-nvfp4 17.88 muse-glimmer:30b-mlx 22.75 x1.27 +reason muse-glimmer:30b-nvfp4 18.27 muse-glimmer:30b-mlx 28.64 x1.57 +prose muse-glimmer:30b-mxfp8 10.25 muse-glimmer:30b-mxfp8-dflash 14.40 x1.40 +reason muse-glimmer:30b-mxfp8 10.53 muse-glimmer:30b-mxfp8-dflash 16.24 x1.54 +prose muse-glimmer:30b 15.93 muse-glimmer:30b-q4_K_M-dflash 15.61 x0.98 +reason muse-glimmer:30b 15.57 muse-glimmer:30b-q4_K_M-dflash 22.52 x1.45 +prose muse-glimmer:30b-q8_0 10.69 muse-glimmer:30b-q8_0-dflash 13.26 x1.24 +reason muse-glimmer:30b-q8_0 10.89 muse-glimmer:30b-q8_0-dflash 18.82 x1.73 +``` -**Columns:** the generator flags them as MIXED. gemma4's cells ran on the fold's interim build (`0.35.0-dynres-27-g948ef3a`, the -fold's tree), glimmer's on the release build. +- **Drafting speeds decode 1.24-1.73×** on every pair and workload but one. The exception is GGUF Q4_K_M prose, at 0.98×. +- **Acceptance per request** is in `runs/tps.json`. MLX reports it as `speculative decode stats`, llama-server as `draft acceptance`. +- **These are 5 reps of one prompt each,** run on the release build and before #465. Treat them as a lead for #473, not a ranking of the tags. ## Changes in this branch -- `docs/maxusai/vision-suite/sampling.py`: a `muse-glimmer` think-on entry. Temperature comes from - `THINK_TEMPERATURE` (0), and top_p 0.95 / top_k 64 from the packaged params, which are the only published values. - Without it, think-on cells would send no sampling keys and run at the packaged temperature 1. -- **After phase 2:** the measured `[expect.*.muse-glimmer]` rows. `muse-glimmer` joins `mlx-metal-0-40-2`'s arches, - and a `metal-0-40-2` profile is added with its version-pattern test. The newest llama.cpp-path Metal profile is - `metal-0-32-14`, so 0.40.2's GGUF path has no profile today. +- `docs/maxusai/vision-suite/sampling.py`: a `muse-glimmer` think-on entry. + - Temperature comes from `THINK_TEMPERATURE` (0). + - top_p 0.95 and top_k 64 come from the packaged params, the only published values. + - Without it, think-on cells would send no sampling keys and run at the packaged temperature 1. +- **This record.** The measured `[expect.*.muse-glimmer]` rows and the `metal-0-40-2` profile are in #467. The loop + detector landed through #469. ## Not covered -- **The bf16 tags** (`30b-bf16(-dflash)`, `30b-mlx-bf16(-dflash)`) were not pulled. With no bf16 arm, a quality gap - between NVFP4/MXFP8/Q4/Q8 cannot be attributed to quantization (the tier-move rule). -- **Think levels in the suite.** The suite measures think-on at the default `"high"` only; levels appear only in the - phase-1 probe. -- **Audio** (glimmer declares none) and **tools beyond one round trip.** -- **CUDA and ROCm.** This is the Metal host's run. +- **Think-on on a fixed build for the four `-dflash` tags.** Their only think-on cells are the release ones above, which are not a + measurement. #471 asks for `30b-q8_0-dflash`; `30b-mlx`, `30b-mxfp8-dflash` and `30b-q4_K_M-dflash` are not + claimed by anyone. +- **OCRBench with thinking on,** and OCRBench on #464's build. The OCRBench path sends no format, so #464 should not change it. +- **The bf16 tags** were not pulled. Without a bf16 arm, a quality gap between NVFP4, MXFP8, Q4 and Q8 cannot be put on + quantization (the tier-move rule). +- **Repeats.** Every suite and OCRBench cell is n=1. +- **Not in the suite:** think levels other than `"high"` (the phase-1 probe covers them), audio, and tools beyond one round trip. +- **CUDA and ROCm.** This was the Metal host's run. `macbook-pro-m5-max-128GB/mlx-metal`