Eval bug: DFlash Speculative Decoding Crash: GGML_ASSERT(buffer) Failure #25191

Name and Version

./llama-cli --version
WARNING: radv is not a conformant Vulkan implementation, testing use only.
version: 9849 (799fcc0)
built with GNU 11.4.0 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

AMD Radeon RX 9070

Models

unsloth/Qwen3.6-27B-Q4_K_M and williamliao/Qwen3.6-27B-DFlash-IQ4_XS

Problem description & steps to reproduce

When using DFlash speculative decoding (--spec-type draft-dflash) with context sizes non default with --cache-type-k and --cache-type-v q8_0 the server crash

#0  0x00007f3c9fa9fff2 in ?? () from /usr/lib/libc.so.6
#1  0x00007f3c9fa9403c in ?? () from /usr/lib/libc.so.6
#2  0x00007f3c9fa94084 in ?? () from /usr/lib/libc.so.6
#3  0x00007f3c9fb0494f in wait4 () from /usr/lib/libc.so.6
#4  0x00007f3ca0151c3b in ggml_print_backtrace () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#5  0x00007f3ca0151dd2 in ggml_abort () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#6  0x00007f3ca0169670 in ggml_backend_buffer_get_type () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#7  0x00007f3ca01696fd in ggml_backend_buffer_is_host () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#8  0x00007f3c9f12e1d7 in llama_kv_cache::set_input_k_rot(ggml_tensor*) const () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#9  0x00007f3c9f117d9e in llm_graph_input_attn_kv_iswa::set_input(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#10 0x00007f3c9f11bd00 in llm_graph_result::set_inputs(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#11 0x00007f3c9f0e7941 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#12 0x00007f3c9f0ee60a in llama_context::decode(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#13 0x00007f3c9f0efe30 in llama_decode () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#14 0x00007f3c9f6d7ce7 in common_speculative_impl_draft_dflash::process(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#15 0x00007f3c9f6cbdbd in common_speculative_process(common_speculative*, llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#16 0x00007f3ca039381b in server_context_impl::decode(int&, int, llama_batch&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#17 0x00007f3ca0395b1d in server_context_impl::update_slots() () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#18 0x00007f3ca033e2f1 in server_queue::start_loop(long) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#19 0x00007f3ca02ddc12 in llama_server(int, char**) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#20 0x00007f3c9fa27741 in ?? () from /usr/lib/libc.so.6
#21 0x00007f3c9fa27879 in __libc_start_main () from /usr/lib/libc.so.6
#22 0x00005621fd94e295 in _start ()

I've try to analyze the crash with the support of a local LLM without using dflash. This is the result analysis written by the LLM used:

The Call Chain

1. common_speculative_impl_draft_dflash::process(batch) calls llama_decode(ctx_dft, batch) on the draft model context
2. llama_context::process_ubatch() builds and executes the computation graph
3. llm_graph_input_attn_kv_iswa::set_input() sets up graph inputs for attention
4. Inside set_input(), the code checks if (self_k_rot) and calls mctx->get_base()->set_input_k_rot(self_k_rot)
5. set_input_k_rot() calls ggml_backend_buffer_is_host(self_k_rot->buffer) — buffer is NULL

Why the Buffer is NULL

The self_k_rot tensor is built in src/llama-graph.cpp:2510:

inp->self_k_rot = mctx_cur->get_base()->build_input_k_rot(ctx0);

In src/llama-kv-cache.cpp:1314-1333:

ggml_tensor * llama_kv_cache::build_input_k_rot(ggml_context * ctx) const {
    ggml_tensor * res = nullptr;

    if (attn_rot_k) {
        int nrot = 64;
        do {
            nrot *= 2;
        } while (n_embd_head_k_all % nrot == 0);
        nrot /= 2;

        res = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, nrot, nrot);
        ggml_set_input(res);
        ggml_set_name(res, "attn_inp_k_rot");
    }

    return res;
}

ggml_new_tensor_2d() creates a tensor with buffer == NULL. It is marked as an input (ggml_set_input) but is never allocated. In normal decoding, the graph allocation step happens before set_input_k_rot() is called, so the buffer is valid. In the DFlash path, this allocation is missing.

Why attn_rot_k is True

The attn_rot_k flag is set during KV cache initialization (llama-kv-cache.cpp:290-294):

attn_rot_k =
    !attn_rot_disable &&
    n_embd_head_k_all > 0 &&
    ggml_is_quantized(type_k) &&
    hparams.n_embd_head_k() % 64 == 0;

For the Qwen3.6-27B model with --cache-type-k q8_0:

• ggml_is_quantized(q8_0) → true
• n_embd_head_k() = 128, 128 % 64 == 0 → true
• attn_rot_disable → false (default)
• Result: attn_rot_k = true → tensor is created but not allocated

Why It Only Crashes at Context > 4096

At small contexts, the graph building and execution phases overlap within the same llama_decode() call, so the tensor happens to be allocated before set_input_k_rot() runs. At larger contexts (>4096), the graph execution path diverges — set_input_k_rot() is called before the tensor allocation step completes, exposing the NULL buffer.

Why Only DFlash (Not EAGLE/EAGLE3)

DFlash creates a separate ctx_dft (draft model context) with its own llama_kv_cache. The graph building happens during initial setup, but set_input_k_rot() is invoked later during llama_decode(ctx_dft) — between these phases, the tensor allocation is not guaranteed. EAGLE and EAGLE3 use different graph paths that don't trigger llm_graph_input_attn_kv_iswa::set_input().

I try to dig in order to understand what are the possible workaround (beside removing the kv parameter) and I can confirm that if set the environment variable LLAMA_ATTN_ROT_DISABLE=1 before launching llama-server resolve temporary the crash (paying the fact that now the kv does not have the Walsh-Hadamard rotation). I can confirm that also doing this resolve the crash.

I then try to understand what are the possible fix on the codebase to try to resolve, again with the help of the LLM. This is the output:

Option A: Ensure Proper Allocation (Preferred)

In common/speculative.cpp, in common_speculative_impl_draft_dflash::process(), ensure the graph is allocated before the first llama_decode(ctx_dft):

// Before calling llama_decode(ctx_dft, ...), ensure graph inputs are allocated
// This may require calling a graph finalization step on ctx_dft

Option B: Defensive Null Check (Quick Fix)

In src/llama-kv-cache.cpp:1673-1674, add a null guard:

void llama_kv_cache::set_input_k_rot(ggml_tensor * dst) const {
    if (dst->buffer == nullptr) return;  // skip if not yet allocated
    GGML_ASSERT(ggml_backend_buffer_is_host(dst->buffer));
    ...
}

This is less ideal because it silently skips the rotation data copy, but it prevents the crash.

At this point I would like a confirmation from someone else about this finding and the resolution fix. If is confirmed I'm willing to try to propose a patch but I would like to understand where is the limit about using the LLM for the patch, like what can and what can not be used for since my knowledge in cpp is very little.

First Bad Commit

No response

Relevant log output

0x00007f3c9fa9fff2 in ?? () from /usr/lib/libc.so.6
#0  0x00007f3c9fa9fff2 in ?? () from /usr/lib/libc.so.6
#1  0x00007f3c9fa9403c in ?? () from /usr/lib/libc.so.6
#2  0x00007f3c9fa94084 in ?? () from /usr/lib/libc.so.6
#3  0x00007f3c9fb0494f in wait4 () from /usr/lib/libc.so.6
#4  0x00007f3ca0151c3b in ggml_print_backtrace () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#5  0x00007f3ca0151dd2 in ggml_abort () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#6  0x00007f3ca0169670 in ggml_backend_buffer_get_type () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#7  0x00007f3ca01696fd in ggml_backend_buffer_is_host () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#8  0x00007f3c9f12e1d7 in llama_kv_cache::set_input_k_rot(ggml_tensor*) const () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#9  0x00007f3c9f117d9e in llm_graph_input_attn_kv_iswa::set_input(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#10 0x00007f3c9f11bd00 in llm_graph_result::set_inputs(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#11 0x00007f3c9f0e7941 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#12 0x00007f3c9f0ee60a in llama_context::decode(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#13 0x00007f3c9f0efe30 in llama_decode () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#14 0x00007f3c9f6d7ce7 in common_speculative_impl_draft_dflash::process(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#15 0x00007f3c9f6cbdbd in common_speculative_process(common_speculative*, llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#16 0x00007f3ca039381b in server_context_impl::decode(int&, int, llama_batch&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#17 0x00007f3ca0395b1d in server_context_impl::update_slots() () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#18 0x00007f3ca033e2f1 in server_queue::start_loop(long) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#19 0x00007f3ca02ddc12 in llama_server(int, char**) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#20 0x00007f3c9fa27741 in ?? () from /usr/lib/libc.so.6
#21 0x00007f3c9fa27879 in __libc_start_main () from /usr/lib/libc.so.6
#22 0x00005621fd94e295 in _start ()