From c77958c1a142ee0c9004ea734071a98a0c4a0e95 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Tue, 13 Oct 2026 09:38:13 +1000 Subject: [PATCH] tasks: open glimmer-decode-speed for another agent (muse-glimmer follow-up to #463) Co-Authored-By: Claude Opus 5.5 --- docs/maxusai/tasks/glimmer-decode-speed.md | 67 ++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 docs/maxusai/tasks/glimmer-decode-speed.md diff --git a/docs/maxusai/tasks/glimmer-decode-speed.md b/docs/maxusai/tasks/glimmer-decode-speed.md new file mode 100644 index 00000000000..9fdb669be8b --- /dev/null +++ b/docs/maxusai/tasks/glimmer-decode-speed.md @@ -0,0 +1,67 @@ +# TASK: why is a muse-glimmer suite cell 1.6–3× slower than gemma4:31b's? Explain each part, with sources + +Opened by the Metal host (`macbook-pro-m5-max-128GB/mlx-metal`) on 2026-10-13, for another agent to pick up. +Status: **open, unclaimed.** Context: #463 (onboarding). + +## What was seen + +The vision suite, thinking off, 27 tests per cell, `num_ctx` 16384, greedy. Per request, averaged from the score +files' `prompt_eval_*` / `eval_*`: +- glimmer: #464's build `0.40.2-dynres-1-g4cfad4a`; +- gemma4: the 0.40.2 fold build. + +| model | prompt tok | prefill min | prefill tok/s | gen tok | decode min | decode tok/s | answer chars | cell min | +|---|---|---|---|---|---|---|---|---| +| muse-glimmer_30b-mlx | 4691 | 2.6 | 800 | 815 | 18.9 | 19.4 | 1089 | 21.7 | +| muse-glimmer_30b-mxfp8-dflash | 4691 | 2.8 | 752 | 890 | 38.3 | 10.5 | 1120 | 41.3 | +| muse-glimmer_30b-mxfp8 | 4691 | 2.9 | 731 | 890 | 39.0 | 10.3 | 1120 | 42.1 | +| muse-glimmer_30b-nvfp4 | 4691 | 2.5 | 849 | 815 | 17.7 | 20.7 | 1089 | 20.4 | +| muse-glimmer_30b-q4_K_M-dflash | 4691 | 5.2 | 405 | 853 | 15.8 | 24.3 | 1096 | 21.1 | +| muse-glimmer_30b-q8_0-dflash | 4691 | 5.2 | 409 | 884 | 18.1 | 21.9 | 1133 | 23.5 | +| muse-glimmer_30b-q8_0 | 4691 | 5.6 | 375 | 884 | 39.5 | 10.1 | 1133 | 45.3 | +| muse-glimmer_30b | 4691 | 6.0 | 350 | 837 | 28.6 | 13.2 | 1087 | 34.7 | +| gemma4_31b-it-q4_K_M | 2214 | 2.0 | 490 | 455 | 13.6 | 15.1 | 953 | 15.7 | +| gemma4_31b-nvfp4 | 2214 | 1.7 | 578 | 463 | 10.9 | 19.1 | 959 | 12.7 | + +What the numbers already say. Each point is a hypothesis to confirm and explain: +1. **4-bit decode speed is not the gap.** Glimmer's nvfp4 decodes at 20.7 tok/s, gemma4's at 19.1. +2. **Glimmer generates ~1.8× the tokens for ~1.15× the answer characters**: 815–890 against ~460 tokens. Find where + the extra ~350 tokens per request go. Candidates: + - hidden tokens with `think: false` (the onboarding measured ~120 on a short prompt); + - the message-header protocol (`<|start|>assistant to=user<|message|>`); + - whitespace padding under the JSON grammar (padded to the bound on MLX; see #470); + - a tokenizer that spends more tokens per character. +3. **Glimmer's prompts are ~2.1× longer**: 4691 against 2214 tokens. Glimmer images go up to 4096 tokens at native + resolution; gemma4 caps at 1120. +4. **The 8-bit tags decode at ~10 tok/s, half the 4-bit tags: memory bandwidth, nearly settled.** The weights are + ~1.9× the size and decode is ~2.0× slower. Weights × tok/s is near constant on MLX: + - `nvfp4`: 17.2 GB × 20.7 ≈ 356 GB/s; + - `mxfp8`: 32.6 GB × 10.3 ≈ 336 GB/s; + - GGUF `q8_0`: 31 GB × 10.1 ≈ 313 GB/s. + + These are upper bounds, since the sizes include the vision encoder, which decode does not read. A constant figure is + the signature of a bandwidth-bound dense model. Left to confirm: + - **Dense or MoE:** confirm glimmer is dense, as this implies. + - **The outlier:** GGUF `Q4_K_M` (18.1 GB × 13.2 ≈ 239 GB/s) falls short of the bound. That is likely K-quant + dequantization cost on Metal, and its drafter hides it (`-dflash`: 24.3 tok/s). +5. **GGUF prefill on glimmer is about half MLX's** (350–409 against 731–849 tok/s). gemma4 is 490 on GGUF against 578 + on MLX. + +## What to do + +1. **Search online** for each point: the model card (ollama.com/library/muse-glimmer and the original release), its + architecture (dense or MoE, layers, sliding window, vision encoder size and sparse attention), and llama.cpp and + MLX issues or PRs on muse-glimmer performance. Is any of this documented or known? Cite each source. +2. **Count the hidden tokens.** On one cell, compare `eval_count` against the tokenized answer. Use a raw, + unparsed decode (`/api/generate` `raw`) to see what the extra tokens are. +3. **Measure decode against the bandwidth bound**: weights in bytes per token against the machine's memory + bandwidth, for the q8_0, mxfp8 and nvfp4 tags. +4. If a fork-side cause turns up (for example padding the grammar allows), open a fix PR. If it is the model, say so + in this record with sources. + +Rules: GPU work under the host's gate (production idle 15 min, powermode 2), on :11437 against an APFS store clone. +Never touch :11435. Pin `NUM_BATCH` for GGUF A/Bs. + +## Done when + +Each of the five points is confirmed or refuted, with a measurement or a cited source, in this record.