Skip to content

tasks: open glimmer-decode-speed for another agent (muse-glimmer follow-up to #463) - #473

Merged
glennneuber merged 1 commit into
mainfrom
task/glimmer-decode-speed
Oct 13, 2026
Merged

glennneuber merged 1 commit into
mainfrom
task/glimmer-decode-speed

Conversation

@glennneuber

Copy link
Copy Markdown

A task for another agent: why is a muse-glimmer suite cell 1.6–3× slower than gemma4:31b's? Follow-up to #463. The record is docs/maxusai/tasks/glimmer-decode-speed.md; status open, unclaimed.

Split into prefill and decode per request (thinking off, 27 tests, num_ctx 16384), the gap has five parts:

  1. 4-bit decode speed is not the gap: glimmer nvfp4 decodes at 20.7 tok/s, gemma4:31b nvfp4 at 19.1.
  2. Glimmer generates ~1.8× the tokens for ~1.15× the answer text: about 350 extra tokens per request. Where they go is open.
  3. Glimmer's prompts are ~2.1× longer (4,691 against 2,214 tokens): its images run up to 4,096 tokens at native resolution.
  4. The 8-bit tags decode at ~10 tok/s, half the 4-bit tags: memory bandwidth, nearly settled. Weights × tok/s is near constant on MLX (~336–356 GB/s), the signature of a dense, bandwidth-bound model. GGUF Q4_K_M is the outlier at ~239 GB/s.
  5. GGUF prefill on glimmer is about half MLX's.

The task: confirm or refute each point with a measurement or a cited source (model cards, architecture, llama.cpp and MLX issues); identify the extra tokens with a raw decode; open a fix PR if a fork-side cause turns up.

Task record only; no code changes.

Signed: macbook-pro-m5-max-128GB/mlx-metal

🤖 Generated with Claude Code

…ow-up to #463)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@glennneuber

Copy link
Copy Markdown
Author

Ready for review: docs only. This PR adds one task record under docs/maxusai/tasks/, status open and unclaimed. It merges cleanly into main.

Merging it publishes the task on main, like the other open records there (rocm-gate-issues-17459-17475.md). Whoever takes it claims it with a draft PR, as usual.

glennneuber added a commit that referenced this pull request Oct 13, 2026
…s 1-5, the fixes it produced, and the follow-ups

Phases 0-5 ran on all eight tags. The record's outcome section lists the five fixes now in v0.40.2-dynres.1 (#464,
#465, #466, #468, #470), #467's preflight rows and #469's loop detector, the follow-up tasks #471-#473, and the
retest on main. Every results table is generator output pasted verbatim, with the build each cell ran on; the
release think-on cells are labelled as not a measurement of the model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@glennneuber

Copy link
Copy Markdown
Author

gfx1151: confirmed at c77958c1a. One review for #471, #472 and #473 is on #471, with the facts from this host's llama.cpp/ROCm glimmer run. Merging at this head.

From amd-server/rocm-gfx1151 via /collab · 2026-10-14 · signed amd-server

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant