Eval bug: QWEN3.6 27b DFlash draft model fails to load

[waldirsp11]

Name and Version

llama.cpp commit: b3fed31 (master branch)
Relevant DFlash commits:

• d1b3425 spec : add DFlash support ([Speculative decoding] feat: add DFlash support #22105) -- initial DFlash implementation
• fa72bc6 dflash: refactor draft model conversion (dflash: refactor draft model conversion #25110) -- recent refactor

Branch: master (latest commit b3fed31 "jinja, chat: add --reasoning-preserve flag")
Built via Docker using cuda.Dockerfile

Operating systems

Linux

GGML backends

CUDA

Hardware

NVidia RTX 3090
NVidia RTX 3080
NVidia RTX 3060
AMD RX 7900 XT
AMD RX 7900 XTX

Models

qwen3.6 27b

Problem description & steps to reproduce

DFlash draft model fails to load due to architecture name mismatch and nested GGUF metadata keys not recognized.

What happens

Loading a DFlash draft model (Qwen3.6-27B-DFlash-Q4_K_M.gguf) as speculative decoding draft model fails with:

error loading model hyperparameters: DFlash model requires 'target_layers' in GGUF metadata

Root cause - two issues

1. Architecture name mismatch: The GGUF file has general.architecture = "dflash-draft", but llama.cpp's architecture registry maps LLM_ARCH_DFLASH to "dflash" (without the -draft suffix). The loader returns LLM_ARCH_UNKNOWN and refuses to load the model.

2. Nested GGUF metadata key scheme: The DFlash-specific keys in the GGUF use a nested naming under {arch}.dflash.*:

   • dflash-draft.dflash.target_layer_ids (array of int32)
   • dflash-draft.dflash.block_size (uint32)
   • dflash-draft.dflash.mask_token_id (int32)
   • dflash-draft.dflash.n_target_features (uint32)

   But llama.cpp expects flat keys:

   • dflash.target_layers (via LLM_KV_TARGET_LAYERS)
   • dflash.block_size (hardcoded string in common/speculative.cpp)

GGUF metadata dump (relevant keys)

general.architecture = dflash-draft
dflash-draft.block_count = ...
dflash-draft.context_length = ...
dflash-draft.embedding_length = ...
dflash-draft.feed_forward_length = ...
dflash-draft.attention.head_count = ...
dflash-draft.dflash.block_size = 16
dflash-draft.dflash.mask_token_id = ...
dflash-draft.dflash.target_layer_ids = [32, 56, 80]
dflash-draft.dflash.n_target_features = ...
dflash-draft.attention.sliding_window = ...
dflash-draft.attention.sliding_window_pattern = [...]

Steps to reproduce

1. Convert a Qwen3.6 model to GGUF with DFlash draft support using convert_hf_to_gguf.py
2. Attempt to load the resulting GGUF as speculative decoding draft model:
   ./llama-cli -m model.gguf -c 4096 --speculative model-dflash.gguf -ngl 1
3. Observe architecture mismatch error and missing metadata keys

Expected behavior

The DFlash model should load correctly as a speculative decoding draft model, recognizing both the architecture name "dflash-draft" and the nested GGUF key scheme.

First Bad Commit

The initial DFlash support was introduced in commit d1b3425 (spec : add DFlash support #22105). The architecture name mismatch between "dflash" (used in llama.cpp) and "dflash-draft" (used by the GGUF converter) likely existed since this initial implementation.

A recent refactor commit fa72bc6 (dflash: refactor draft model conversion #25110) also touched the DFlash conversion code but did not address this naming mismatch.

Testing was done with Qwen3.6-27B Unsloth variant, converted from https://huggingface.co/Qwen/Qwen3.6-27B using llama.cpp's Python GGUF converter script.

Did not perform git bisect - the issue is a clear mismatch between the architecture name expected by llama.cpp ("dflash") and the one written by the GGUF converter ("dflash-draft").

Relevant log output

Logs

# docker logs -f llm-model
warn: LLAMA_ARG_KV_UNIFIED environment variable is set, but will be overwritten by command line argument --kv-unified
0.00.865.768 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.870.924 I srv    load_model: loading model '/models/Qwen3.6-27B-Q4_K_M-MTP.gguf'
0.01.158.628 E llama_model_load: error loading model: unknown model architecture: 'dflash-draft'
0.01.158.653 E llama_model_load_from_file_impl: failed to load model
0.01.158.754 W srv    load_model: [spec] failed to measure draft model memory: failed to load model
0.02.062.197 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.31.165.722 W cmn  common_init_: KV cache shifting is not supported for this context, disabling KV cache shifting
0.32.183.103 E llama_model_load: error loading model: unknown model architecture: 'dflash-draft'
0.32.183.111 E llama_model_load_from_file_impl: failed to load model
0.32.183.121 E srv    load_model: failed to load draft model, '/models/Qwen3.6-27B-DFlash-Q4_K_M.gguf'
0.32.183.136 I srv    operator(): operator(): cleaning up before exit...
0.32.188.350 E srv  llama_server: exiting due to model loading error

[ulyssesrr]

The unknown model architecture: 'dflash-draft' error means you are using an DFlash model meant for other llama.cpp forks, for this repo try this one:
https://huggingface.co/jojohai/Qwen3.6-27B-DFlash-GGUF

[ruixiang63]

Please convert the DFlash model to GGUF by using convert script, following this

TARGET_MODEL_HF="${MODELS_DIR}/Qwen3-8B"
TARGET_MODEL_GGUF="${MODELS_DIR}/Qwen3-8B.gguf"
DFLASH_MODEL_HF="${MODELS_DIR}/Qwen3-8B-DFlash-b16"
DFLASH_MODEL_GGUF="${MODELS_DIR}/Qwen3-8B-DFlash-b16.gguf"

python convert_hf_to_gguf.py \
    "${TARGET_MODEL_HF}" \
    --outtype bf16 \
    --outfile "${TARGET_MODEL_GGUF}"

python convert_hf_to_gguf.py \
    "${DFLASH_MODEL_HF}" \
    --outtype bf16 \
    --target-model-dir "${TARGET_MODEL_HF}" \
    --outfile "${DFLASH_MODEL_GGUF}"