[Annieren]

MMVDQ change summary

What it adds: a dequantize-to-float matvec path (mmvdq) for K-quant weights
during decode (n=1), as an alternative to the existing mul_mat_vec_q (mmvq)
path.

Why: mmvq quantizes the activation vector to q8_1 first (a quantize_q8_1
pass) so it can do int8 dot products. On RDNA3.5 (gfx1151) int8 has no
throughput advantage over f16, and these matvecs are DRAM-bandwidth-bound on
the weights, so the q8_1 quantize pass is pure overhead. mmvdq dequantizes
the weight blocks straight to float and does the dot in float, skipping the
quantize pass entirely.

Scope (files):

• ggml/src/ggml-cuda/mmvdq.cu (+579), mmvdq.cuh (+27) — new kernel +
  launchers
• ggml/src/ggml-cuda/ggml-cuda.cu (+33) — two routing hooks

Kernel design (mmvdq.cu): one warp per output block, 16 threads process one
super-block, activations loaded once (as float4/float2) and reused across the
block. Covers Q4_K, Q5_K, Q6_K. Includes a fused gate+up SwiGLU variant
(mul_mat_vec_dq_glu) that computes silu(gate)*up in one dispatch sharing the
activation load.

Routing / gating (ggml-cuda.cu):

• Non-fused path in ggml_cuda_mul_mat: triggers when ne11==1, type is
  Q4_K/Q5_K (or Q6_K if enabled), all tensors contiguous, ne02/03/12/13==1, and
  ne[0] % QK_K == 0.
• Fused SwiGLU path in ggml_cuda_try_fuse: requires ids==nullptr,
  GGML_GLU_OP_SWIGLU, matching gate/up shapes+type, single row.

Enablement (env overrides):

• GGML_CUDA_DQ_MMV — master switch: unset = arch default (on for RDNA3.5), 0
  = force off, non-zero = force on
• GGML_CUDA_DQ_Q6K — Q6_K gate, same semantics (Q6_K is bandwidth-bound and
  ~neutral vs mmvq)

Default is on only for RDNA3.5; every other arch keeps mmvq.

Full decode comparison (38 models)

Model	Base t/s	MMVDQ t/s	Δ%
Gemma-4-E2B-IT_Q4_K_M_GGUF_128	95.15	105.09	+10.4%
Gemma-4-E2B-IT_Q4_K_M_GGUF_3968	89.86	98.22	+9.3%
Gemma-2B_Q4_K_M_GGUF_128	106.68	115.00	+7.8%
Janus-Pro-1B_VLM	148.35	158.28	+6.7%
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_128	143.74	153.18	+6.6%
Qwen3-1.7B_Q4_K_M_GGUF_128	139.69	147.69	+5.7%
Gemma-4-31B-IT_VLM	10.43	10.95	+4.9%
Qwen3-4B_Q4_K_M_GGUF_128	71.07	74.46	+4.8%
Qwen3-VL-4B-Instruct_VLM	66.35	69.31	+4.5%
Gemma-3-4B-IT_VLM	67.67	70.45	+4.1%
Qwen3.5-4B_Q4_K_M_GGUF_128	60.36	62.84	+4.1%
Qwen3-1.7B_Q4_K_M_GGUF_3968	106.89	111.27	+4.1%
Qwen3-4B_Q4_K_M_GGUF_3968	59.31	61.70	+4.0%
Llama-2-7B_Q4_K_M_GGUF_128	46.14	47.84	+3.7%
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_128	44.86	46.41	+3.5%
Janus-Pro-7B_VLM	43.50	44.81	+3.0%
Gemma-3-12B-IT_Q4_K_M_GGUF_VLM	25.73	26.49	+3.0%
Llama-2-7B_Q4_K_M_GGUF_1920	33.86	34.84	+2.9%
Gemma-3-12B-IT_Q4_K_M_GGUF_4096 *	23.71	24.37	+2.8%
Qwen3-8B_Q4_K_M_GGUF_3968	36.27	37.28	+2.8%
Qwen3-30B-A3B-Instruct-2507_Q4_K_M_GGUF_128	74.51	76.35	+2.5%
Gemma-2B_Q4_K_M_GGUF_8000	104.26	106.59	+2.2%
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_128	87.44	89.27	+2.1%
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_3968	82.55	84.09	+1.9%
Qwen3.5-9B_Q4_K_M_GGUF_128	36.51	37.11	+1.6%
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_8000	36.19	36.77	+1.6%
Qwen3-8B_Q4_K_M_GGUF_128	41.46	42.12	+1.6%
Llama-3.1-8B-Instruct_Q4_K_M_GGUF_128	42.48	43.13	+1.5%
Qwen2.5-VL-3B-Instruct_VLM	84.45	85.56	+1.3%
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_3968	43.26	43.64	+0.9%
MiniCPM-V-2_6_VLM	45.17	45.47	+0.7%
Qwen2.5-VL-7B-Instruct_VLM	44.63	44.90	+0.6%
MiniCPM-o-2_6_VLM	45.21	45.47	+0.6%
Qwen3.5-35B-A3B_Q4_K_M_GGUF_128 *	54.58	54.69	+0.2%
Qwen3.6-27B_Q4_K_M_GGUF_4096 *	12.22	12.24	+0.2%
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_128	318.81	318.94	+0.0%
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_3968	291.28	290.39	-0.3%
Gemma-4-26B-A4B-IT_VLM	47.30	47.07	-0.5%

Mean +3.09%, median +2.78% (n=38). * = controlled DQ-toggle run (DQ-off/mmvq
vs DQ-on/mmvdq) substituted for the three thermally-throttled full-suite
rows.

[Annieren]

Wrong base branch.