Eval bug: CUDA MoE weighted-reduction fusion (#25952) breaks speculative-decoding exactness — MTP draft acceptance 0.82 → 0.48 and MTP becomes a net slowdown (bisected to b10751) #29168

Name and Version

First bad build: b10751. Last good: b10750. Still present at b11057 (today's head).

$ llama-server --version
version: 0.1.0-dev (build 10750, ...)   # last good
version: 0.1.0-dev (build 10751, ...)   # first bad

Operating systems

Windows

GGML backends

CUDA

Hardware

RTX 2070 SUPER (sm_75, 8 GB), CUDA 12.4 prebuilt Windows release binaries (llama-<b>-bin-win-cuda-12.4-x64.zip), i7 8-core, 32 GB RAM. Partial offload: -ngl 31 --n-cpu-moe 21.

Models

• Target: unsloth/gemma-4-26B-A4B-it-qat-GGUF → gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (MoE, 30 blocks)
• Draft: the repo's own MTP sidecar, MTP/mtp-gemma-4-26B-A4B-it-Q4_0.gguf, via --spec-type draft-mtp

Problem description & steps to reproduce

#25952 (cuda: fuse MoE weighted expert reduction) makes speculative decoding non-exact on a CUDA MoE model, and turns MTP speculative decoding into a net slowdown.

Speculative decoding is supposed to be a pure speed optimisation: with greedy sampling, the output must be identical to decoding without a draft. From b10751 onwards it is not.

I bisected across the 318 releases between a known-good and a known-bad build, using draft acceptance as the signal (it is sharply bimodal, so every step was unambiguous). b10750 → b10751 is exactly one commit, 3466812, which is #25952 and touches only ggml/src/ggml-cuda/*.

Two things to note:

1. Without a draft the builds are equivalent (38.10 vs 37.02–38.33), so this is not a general decode regression — it is confined to the speculative path.
2. On the affected builds the draft is a net loss: 35.36 tok/s with the draft vs 37.02 without on b10964; 36.73 vs 38.33 on b11057. Users are better off disabling MTP.

b10964 (the build v0.4.1 designates), b11056 and b11057 all produce byte-identical output to each other, so the behaviour is stable and unchanged since it landed.

Suspected mechanism

#25952 does "weighting and ordered expert reduction in one CUDA kernel" instead of writing weighted expert outputs to global memory and reducing them separately. That changes the order of floating-point accumulation across experts. Small differences in the target model's logits are enough to flip the argmax on some tokens, so the draft's proposals stop matching the target and are rejected — which explains both the halved acceptance and the divergence from greedy output. It is consistent with the fusion itself being numerically reasonable but not bit-identical to the unfused path, which speculative decoding's exactness relies on.

Possibly related: #28113 (MoE models produce garbage output on RDNA3.5 since #27621) reported a correctness problem from the sibling MoE fusion PR.

Steps to reproduce

Run the same prompt twice on one build — once with the draft, once without — with greedy sampling, and compare the output text:

# A: with the MTP draft
llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
  --model-draft MTP/mtp-gemma-4-26B-A4B-it-Q4_0.gguf --spec-type draft-mtp --spec-draft-n-max 2 \
  -ngl 31 --n-cpu-moe 21 -c 32768 -ctk q8_0 -ctv q8_0 -fa on -b 512 -ub 512 --port 8080

# B: identical, minus the three draft flags

curl -s localhost:8080/completion -d \
  '{"prompt":"Write one vivid paragraph about the sea.","n_predict":128,"temperature":0,"top_k":1,"seed":1,"cache_prompt":false}' \
  | jq -r .content | sha256sum

• On b10750, A and B give the same sha256.
• On b10751 and later, they differ.

draft acceptance in the slot-timing line drops from 0.823 to 0.481 across the same boundary.

Method notes, in the interest of being honest about what was and was not tested:

• Each build was measured with 1 warm-up plus 3 runs; the order was reversed once to rule out thermal or ordering effects, and the numbers reproduced.
• I used the raw /completion endpoint with no chat template, so the generated text is not meaningful prose on any build. That does not affect the result: the comparison is always within a single build, between drafted and non-drafted decoding of the same prompt with the same settings.
• Greedy output also differs between builds, which changes to kernels and fusion can legitimately explain. I make no claim there.
• I have not tested other MoE architectures, other GPUs, or non-MTP draft types. -np was left at the server default.

First Bad Commit

3466812 — cuda: fuse MoE weighted expert reduction (#25952), first released in b10751.

Bisected over the 318 releases between b10437 and b10964, 8 builds probed:

b10723 GOOD 0.8229   b10736 GOOD 0.8229   b10740 GOOD 0.8229
b10749 GOOD 0.8229   b10750 GOOD 0.8229
b10751 BAD  0.4806   b10775 BAD  0.4806   b10837 BAD  0.4806

(b10741–b10748 do not start at all on this setup — 0xC0000409, apparently the gemma4-assistant window broken by #28159 and fixed by #28183 in b10749. The bisect stepped past them; it does not affect the boundary above.)

Relevant log output

# b10750 — last good
slot print_timing: id  3 | task 0 | draft acceptance = 0.82292 (   79 accepted /    96 generated), mean len =  2.65
  -> 46.35 tok/s

# b10751 and later (b10964, b11056, b11057 all identical)
slot print_timing: id  3 | task 0 | draft acceptance = 0.48062 (   62 accepted /   129 generated), mean len =  1.95
  -> 35.36 tok/s   (vs 37.02 tok/s with no draft at all on the same build)