DFlash draft acceptance rate collapses to zero when using quantized --cache-type-k-draft (or v)

Name and Version

Versions tested (subset - many builds were tested):
On Windows:
version: 10015 (12127de)
built with MSVC 19.51.36248.0 for x64

On WSL/ubuntu26.04:
version: 10015 (12127de)
built with GNU 15.2.0 for Linux x86_64

I also tested with these releases, but I no longer have them installed:
https://github.com/ggml-org/llama.cpp/releases/download/b10015/llama-b10015-bin-ubuntu-vulkan-x64.tar.gz
https://github.com/microsoft/winget-pkgs/tree/master/manifests/g/ggml/llamacpp/b9982

Operating systems

Linux, Windows

GGML backends

CUDA, Vulkan

Hardware

CPU: Ryzen 7600X
GPU: RTX 4090 24GB

Models

main model: bartowski/deepreinforce-ai_Ornith-1.0-35B-GGUF:IQ4_XS
https://huggingface.co/bartowski/deepreinforce-ai_Ornith-1.0-35B-GGUF

dflash draft model: williamliao/Qwen3.6-35B-A3B-DFlash-GGUF:Q8_0
https://huggingface.co/williamliao/Qwen3.6-35B-A3B-DFlash-GGUF

Problem description & steps to reproduce

TL;DR:
--spec-type draft-dflash --cache-type-k-draft f16 --cache-type-v-draft f16 : up to 99% draft acceptance
--spec-type draft-dflash --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 : 0-2% draft acceptance
LLAMA_ATTN_ROT_DISABLE=1 --spec-type draft-dflash --cache-type-k-draft q8_0 : restores 99% draft acceptance

When using DFlash speculative decoding, quantizing the DFlash draft-context KV cache keys or values to q8_0 (and perhaps other quants) (via --cache-type-k-draft/--cache-type-v-draft) collapses draft acceptance from ~80-99% down to ~0-2%. Using either f16 or bf16 for both caches (or omitting the flags (default = f16)) gives the expected high acceptance rate. I bisected my configs to determine that setting either k or v quant to q8_0 independently causes the failure.

This reproduces across Windows (winget/Clang/Vulkan and self-compiled MSVC/CUDA) and Ubuntu/WSL (release/Vulkan and self-compiled GCC/CUDA), so it's backend- and compiler-independent.

The only variable that triggers the failure is the draft cache type. Normal inference and MTP speculative decoding are unaffected (they don't use the cache, so of course); this is specific to DFlash's draft-context KV cache. Full CLI args for both the passing (f16/bf16) and failing (q8_0) runs are in the attached logs.

AI use: LLMs were completely unable to identify the source of this bug on their own. This was human testing and discovery. I did ask LLMs for assistance with troubleshooting, research, github issue searches. The only reason I found the bug was because I got lazy when porting my server launch script and left off the draft cache quant params in wsl - I initially thought it was a Windows-specific failure, but the key insight was that Ubuntu lacked the draft cache quant params entirely. Eventually I tried to automate A/B testing with a parameter sweep and my local LLM complained that the win/wsl configs did not actually match, so the tests weren't really comparable. I equalized them by simplifying and suddenly Windows draft acceptance became acceptable.

First Bad Commit

This has probably been an issue since the first DFlash merge. I have tested with builds from around 9970-10000+, but have not kept track during my testing.

Relevant log output

Logs were not particularly helpful in debugging this issue - the only thing I saw was a low draft acceptance. The draft acceptance is around 80%+ with thinking off and quant off or 0-2% with quantization enabled.

fail.log

pass.log

@ruixiang63

cc @ruixiang63

我跟你一样的现象, 确实草稿的kv用f16后问题得到解决

I’m not sure whether this is a bug or whether DFlash is particularly sensitive to the KV cache quant type.
My intuition is that DFlash may be highly sensitive because it relies heavily on KV-cache injection from the target model, so even small numerical mismatches could cause issues.

One thing to try is using the same KV cache quantization type for both the target and draft models and observing whether the behavior changes.

I suspect this may be a limitation of DFlash rather than an bug.

I suspect this may be a limitation of DFlash rather than an bug.

If so, I think emitting a warning (or even an error) when these incompatible flags are combined is warranted. The --help description describes a plethora of draft cache quants to use; most users will not expect that the majority of them are a trap.

@ruixiang63

Going to near 0% acceptance seems strange - I think it is unlikely to be purely due to the quantization. @ruixiang63 Do you reproduce it on your end?

Not yet. I am trying to reproduce that.

TL;DR: this comment adds no new information, just further evidence under wsl.

I did most of my testing on Windows, but I imagine others might test on Linux primarily, so I pulled the latest release, rebuilt, and ran another test demonstrating the failure under wsl, as I had previously only retained logs from Windows testing.

Build settings

cmake -B build-linux \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="89-real" \
  -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_NCCL=OFF \
  -DGGML_NATIVE=ON \
  -DGGML_OPENMP=ON

cmake --build build-linux --config RelWithDebInfo -j"$(nproc)"

llama-server version

$ build-linux/bin/llama-server --version
version: 10032 (3b5321936)
built with GNU 15.2.0 for Linux x86_64

Server launch params

HF_HUB_CACHE=/mnt/d/model_cache build-linux/bin/llama-server \
  --hf-repo bartowski/deepreinforce-ai_Ornith-1.0-35B-GGUF:IQ4_XS \
  --n-gpu-layers all --flash-attn on --ctx-size 8192 \
  --parallel 1 -t 1 --seed 42 --temp 0 --top-k 1 \
  --host 0.0.0.0 --port 8080 \
  --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 \
  --gpu-layers-draft all --spec-type draft-dflash --spec-draft-n-max 4 \
  --hf-repo-draft williamliao/Qwen3.6-35B-A3B-DFlash-GGUF:Q8_0 \
  --reasoning off --alias bartowski_Ornith:IQ4_XS:off:wsl \
  -v > ./wsl_log.txt 2>&1

Prompt: the first few questions from humaneval

Draft acceptance: 0.00987

Log: wsl_log.txt

Does the bug also happen if you set draft KV to BF16?

Does the bug also happen if you set draft KV to BF16?

bf16 and f16 were identical in my testing. Same draft acceptance rate, token generation rate, wall time, and pass@1 score on the benchmark.

Try with LLAMA_ATTN_ROT_DISABLE=1

Try with LLAMA_ATTN_ROT_DISABLE=1

That helps narrow down the issue for sure. Disabling that feature does restore the draft acceptance rate at Q8. I wouldn't run with that configuration, but it's good to help diagnose the issue.

Draft acceptance: 0.97159

Log: wsl_log.txt

@am17an

Yes, nice catch by @am17an. This means we are missing the attention rot in the dflash graph, correct?

Yes, nice catch by @am17an. This means we are missing the attention rot in the dflash graph, correct?

~~But this only happens in draft kv cache quantization, I am also wondering why this helps.~~ I found this PR #21038, it seems rotate only happens in K/V cache quant.

Yes, the rotations in the KV cache are applied only for quantized types. So they are not applied for F16 and BF16.

I think the explanation is that the target model KV cache applies the rotation, but we don't apply it when injecting it to the DFlash KV cache. So the DFlash cache ends up being not rotated, while the target model KV cache is rotated. Simply applying the rotation during injection should fix the problem.

Yes, the rotations in the KV cache are applied only for quantized types. So they are not applied for F16 and BF16.

I think the explanation is that the target model KV cache applies the rotation, but we don't apply it when injecting it to the DFlash KV cache. So the DFlash cache ends up being not rotated, while the target model KV cache is rotated. Simply applying the rotation during injection should fix the problem.

Thanks! The PR #25823 should resolve this issue. Tested locally and works as expected.

@HermiG

can you confrim if the PR #25823 fix the issue from your side? @HermiG

Confirmed fixed. I launched the server with LLAMA_ATTN_ROT_DISABLE=0 and --cache-type-k-draft q8_0 --cache-type-v-draft q5_1. Acceptance rate appears to match f16/bf16 levels.

Draft acceptance: 0.97159

dflash: rotate injected K/V cache when using K/V quantization

Navigation Menu

Repository navigation

DFlash draft acceptance rate collapses to zero when using quantized --cache-type-k-draft (or v) #25725

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Build settings

llama-server version

Server launch params

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DFlash draft acceptance rate collapses to zero when using quantized --cache-type-k-draft (or v) #25725

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

ggerganov commented on Jul 15, 2026

tmiqpl commented on Jul 15, 2026

ruixiang63 commented on Jul 15, 2026

HermiG commented on Jul 15, 2026

ggerganov commented on Jul 15, 2026

ruixiang63 commented on Jul 15, 2026

HermiG commented on Jul 15, 2026

Build settings

llama-server version

Server launch params

Diablo-D3 commented on Jul 15, 2026

HermiG commented on Jul 15, 2026

am17an commented on Jul 16, 2026

HermiG commented on Jul 16, 2026

ggerganov commented on Jul 16, 2026

ruixiang63 commented on Jul 16, 2026

ggerganov commented on Jul 17, 2026

ruixiang63 commented on Jul 17, 2026

ruixiang63 commented on Jul 17, 2026

HermiG commented on Jul 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions