Eval bug: Gemma 4 31B MTP (draft-mtp) crashes on Vulkan backend, pre-allocated tensor cannot run operation NONE

[kostich]

Name and Version

version: 9601 (4c65955)
built with GNU 16.1.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

CPU: AMD Ryzen 9 5900X
GPU: AMD Radeon RX 7900 XTX (RDNA3, gfx1100), 24GB VRAM
ROCm 7.1.1, Mesa RADV NAVI31 (Vulkan)

Models

Main: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF/blob/main/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
Draft: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF/blob/main/MTP/gemma-4-31B-it-Q4_0-MTP.gguf

Problem description & steps to reproduce

Gemma 4 31B with --spec-type draft-mtp crashes immediately on context
initialization when using the Vulkan backend. The crash does not occur with
ROCm (--device ROCm0), which works correctly at 45-60 tok/s with MTP.

The crash happens during draft model KV cache setup, specifically when
llama.cpp tries to schedule a pre-allocated shared KV tensor (cache_k_l58)
on Vulkan0. The Vulkan backend scheduler aborts because it cannot run the
NONE operation on a pre-allocated tensor in its buffer.

The KV sharing warnings immediately precede the crash:
llama_kv_cache: layer 3: sharing with layer 59
llama_kv_cache: layer 0: sharing with layer 58
llama_kv_cache: layer 1: sharing with layer 58
llama_kv_cache: layer 2: sharing with layer 58

This is Gemma 4's MTP draft KV sharing pattern (layers 0-3 of the draft
sharing with layers 58-59 of the main model). The Vulkan backend scheduler
does not support this cross-context pre-allocated tensor assignment.

Tested with both f16 and q8_0 KV cache — same crash either way.
--fit off is already set; this is not the fit-path crash from issue #24117.

Build command:
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)"
cmake -S . -B build -DGGML_HIP=ON -DGGML_VULKAN=ON
-DAMDGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release
-DGGML_HIP_ROCWMMA_FATTN=ON -DLLAMA_CURL=ON

Reproduce with:
llama-server
--model gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
--spec-draft-model gemma-4-31B-it-Q4_0-MTP.gguf
--device Vulkan0
--spec-type draft-mtp
--spec-draft-n-max 4
--ctx-size 16384
--n-gpu-layers 99
--parallel 1
--fit off
--flash-attn on
--cache-type-k f16
--cache-type-v f16
--reasoning on
--jinja

First Bad Commit

Not bisected. PR #23398 (Gemma 4 MTP support) merged recently, this may be
a pre-existing Vulkan limitation rather than a regression. ROCm was not
tested before #23398 so cannot confirm if this is a new regression.

Relevant log output

Logs

0.00.069.844 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.069.846 I device_info:
0.00.069.874 I   - ROCm0   : AMD Radeon RX 7900 XTX (24560 MiB, 24502 MiB free)
0.00.069.986 I   - Vulkan0 : AMD Radeon RX 7900 XTX (RADV NAVI31) (24576 MiB, 22496 MiB free)
0.00.069.990 I   - CPU     : AMD Ryzen 9 5900X 12-Core Processor (128703 MiB, 128703 MiB free)
0.00.070.038 I system_info: n_threads = 22 (n_threads_batch = 22) / 24 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.070.072 I srv          init: running without SSL
0.00.070.108 I srv          init: using 23 threads for HTTP server
0.00.070.208 W srv  llama_server: -----------------
0.00.070.209 W srv  llama_server: CORS proxy is enabled, do not expose server to untrusted environments
0.00.070.210 W srv  llama_server: This feature is EXPERIMENTAL and may be removed or changed in future versions
0.00.070.210 W srv  llama_server: -----------------
0.00.070.219 W srv  llama_server: -----------------
0.00.070.219 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.070.220 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.070.220 W srv  llama_server: -----------------
0.00.070.225 I srv         start: binding port with default address family
0.00.071.425 I srv  llama_server: loading model
0.00.071.435 I srv    load_model: loading model '/home/marko/AI/models/gemma-4-31b-it/ud_qr_k_xl_qat.gguf'
0.00.455.063 W load: override 'tokenizer.ggml.add_bos_token' to 'true' for Gemma4
0.00.508.593 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.508.930 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.528.480 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.04.957.722 W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.05.044.384 W common_init_from_params: KV cache shifting is not supported for this context, disabling KV cache shifting
0.05.044.392 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.05.121.077 I srv    load_model: loading draft model '/home/marko/AI/models/gemma-4-31b-it/q4_0_mtp.gguf'
0.05.543.412 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.05.543.764 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.05.564.629 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.05.784.913 W llama_context: n_ctx_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
0.05.788.435 W llama_kv_cache: layer   3: sharing with layer 59. k = 0x8001000, v = 0xc001000
0.05.788.449 W llama_kv_cache: layer   0: sharing with layer 58. k = 0x9c01000, v = 0xa801000
0.05.788.450 W llama_kv_cache: layer   1: sharing with layer 58. k = 0x9c01000, v = 0xa801000
0.05.788.451 W llama_kv_cache: layer   2: sharing with layer 58. k = 0x9c01000, v = 0xa801000
/home/marko/AI/llama.cpp/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_k_l58) in a buffer (Vulkan0) that cannot run the operation (NONE)
[New LWP 111011]
[New LWP 110984]
[New LWP 110983]
[New LWP 110982]
[New LWP 110981]
[New LWP 110980]
[New LWP 110979]
[New LWP 110978]
[New LWP 110977]
[New LWP 110976]
[New LWP 110975]
[New LWP 110974]
[New LWP 110973]
[New LWP 110972]
[New LWP 110971]
[New LWP 110970]
[New LWP 110969]
[New LWP 110968]
[New LWP 110967]
[New LWP 110966]
[New LWP 110965]
[New LWP 110964]
[New LWP 110963]
[New LWP 110962]
[New LWP 110961]
[New LWP 110960]
[New LWP 110959]
[New LWP 110957]
[New LWP 110956]

This GDB supports auto-downloading debuginfo from the following URLs:
  <ima:enforcing>
  <https://debuginfod.fedoraproject.org/>
  <ima:ignore>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007feeb4082412 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007feeb4082412 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007feeb407662c in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007feeb4076674 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007feeb40e624f in wait4 () from /lib64/libc.so.6
#4  0x00007feebed5801b in ggml_print_backtrace () from /home/marko/AI/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007feebed5818d in ggml_abort () from /home/marko/AI/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007feebed702e1 in ggml_backend_sched_backend_id_from_cur(ggml_backend_sched*, ggml_tensor*) () from /home/marko/AI/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007feebed721d4 in ggml_backend_sched_split_graph () from /home/marko/AI/llama.cpp/build/bin/libggml-base.so.0
#8  0x00007feebd848d8d in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /home/marko/AI/llama.cpp/build/bin/libllama.so.0
#9  0x00007feebd849e42 in llama_context::sched_reserve() () from /home/marko/AI/llama.cpp/build/bin/libllama.so.0
#10 0x00007feebd8501cd in llama_context::llama_context(llama_model const&, llama_context_params) () from /home/marko/AI/llama.cpp/build/bin/libllama.so.0
#11 0x00007feebd8516a4 in llama_init_from_model () from /home/marko/AI/llama.cpp/build/bin/libllama.so.0
#12 0x00007feebe305af5 in server_context_impl::load_model(common_params&) () from /home/marko/AI/llama.cpp/build/bin/libllama-server-impl.so
#13 0x00007feebe24e0de in llama_server(int, char**) () from /home/marko/AI/llama.cpp/build/bin/libllama-server-impl.so
#14 0x00007feeb400a681 in __libc_start_call_main () from /lib64/libc.so.6
#15 0x00007feeb400a798 in __libc_start_main_impl () from /lib64/libc.so.6
#16 0x00000000004003b5 in _start ()
[Inferior 1 (process 110954) detached]

[LifesLight]

Possibly related to #24366

[Kononnable]

@kostich I can reproduce it only when I'm building binary with both backends active at the same time.

You should be able to work around it by setting -DGGML_HIP=ON or -DGGML_VULKAN=ON during compilation (not both) and having two separate sets of binaries if you need both backends.

[kostich]

I can confirm that it works once I turn OFF all HIP/ROCm related flags. :( I also do not see performance improvement with Gemma on Vulkan, like I do with Qwen, interesting. And it still breaks if I try MTP on multiple GPUs with split mode set to tensor. Closing this one.

[Kononnable]

@kostich You can keep both flags during compilation - just add --spec-draft-device Vulkan0 to the server run parameters.

@am17an With Gemma MTP architecture is there a scenario when someone might use different devices for the main model and the draft?

Currently if someone compiles llama.cpp with multiple backends they can end up with not so obvious error. If they pass --device and forget to pass --spec-draft-device llama_prepare_model_devices can choos wrong backend.
llama.cpp/ggml/src/ggml-backend.cpp:929: pre-allocated tensor (cache_k_l46) in a buffer (Vulkan0) that cannot run the operation (NONE)

I'm thinking if it would be worth to add some sort of assert to notify the user of the problem, or use --device value (if specified) for --spec-draft-device default.

Log with wrong device chosen

0.00.059.685 I srv  llama_server: loading model
0.00.059.694 I srv    load_model: loading model 'gemma-4-12B-it-qat-UD-Q4_K_XL.gguf'
0.00.143.677 I llama_model_loader: loaded meta data with 48 key-value pairs and 667 tensors from gemma-4-12B-it-qat-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
0.00.143.701 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[...]
0.00.296.033 I print_info: file format = GGUF V3 (latest)
0.00.296.034 I print_info: file type   = Q4_0
0.00.296.037 I print_info: file size   = 6.24 GiB (4.50 BPW) 
0.00.296.228 I llama_prepare_model_devices: using device Vulkan0 (AMD Radeon RX 6600 (RADV NAVI23)) (0000:09:00.0) - 7278 MiB free
0.00.520.194 I load: 0 unused tokens
[...]
0.04.028.639 I srv    load_model: loading draft model 'gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf'
0.04.106.057 I llama_model_loader: loaded meta data with 52 key-value pairs and 49 tensors from gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf (version GGUF V3 (latest))
0.04.106.063 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[...]
0.04.256.402 I print_info: file format = GGUF V3 (latest)
0.04.256.404 I print_info: file type   = Q8_0
0.04.256.407 I print_info: file size   = 428.53 MiB (8.50 BPW) 
0.04.256.586 I llama_prepare_model_devices: skipping device Vulkan0 (AMD Radeon RX 6600 (RADV NAVI23)) with id 0000:09:00.0 - already using device ROCm0 (AMD Radeon RX 6600) with the same id
0.04.256.590 I llama_prepare_model_devices: using device ROCm0 (AMD Radeon RX 6600) (0000:09:00.0) - 8120 MiB free
0.04.483.293 I load: 0 unused tokens
[...]
0.04.923.037 I llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
0.04.923.037 I llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
0.04.923.061 I sched_reserve: reserving ...
0.04.923.179 I sched_reserve: resolving fused Gated Delta Net support:
llama.cpp/ggml/src/ggml-backend.cpp:929: pre-allocated tensor (cache_k_l46) in a buffer (Vulkan0) that cannot run the operation (NONE)
[New LWP 1348474]
[New LWP 1348473]
[New LWP 1348472]
[New LWP 1348471]
[New LWP 1348470]
[New LWP 1348469]
[New LWP 1348468]
[New LWP 1348467]

[kostich]

Having a more obvious error message would indeed be helpful and somewhat of a solution for this issue so I'll re-open it.

[Dixon-Cider]

RDNA4 data point — the working-hardware reports here are RDNA3 (#24596 is a 7800 XT). Gemma 4 MTP works on RDNA4 / gfx1201 (R9700, 32 GB) via ROCm/HIP — same pattern others report: crashes on Vulkan, runs fine on ROCm/HIP. Hadn't seen gfx1201 confirmed, so:

Build — tag b9553 (later builds fail to load the drafter, #24795), HIP-only, ROCm HIP SDK 7.1.1. (On this SDK + MSVC ≥14.40 you need the one-line cmath header reorder to compile — #22570.)

Run

llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -md gemma-4-31B-it-Q8_0-MTP.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -np 1 --jinja

Results (single GPU, gfx1201):

• 28.6 → ~48–50 tok/s with MTP (~1.6×, lossless).
• Draft acceptance: 55% @ n_max 2, 38% @ n_max 4 (QAT); 64% on a Q5_K build.
• -fa on + KV quant (q8_0/q4_0) both work. q4_0 reaches the full 256K ctx on 32 GB (with -ub 256 to avoid a compute-buffer spill); q8_0 ~150K.
• q4_0 KV showed no measurable recall/reasoning loss vs q8_0 — a multi-hop reasoning probe scored 8/8 at every length to 246K, matching the q8_0 control.
• Tool-calling (--jinja) works concurrently with MTP active.

Two notes:

1. Gemma4Assistant requires ctx_other to be set during memory fitting is benign here (MTP still runs) — not the silent-disable in Eval bug: Gemma4 MTP is silently disabled in case of insufficient VRAM #24758.
2. Mask extra GPUs (HIP_VISIBLE_DEVICES=0) — with an iGPU or 2nd card also visible, gfx1201 can segfault at startup (cf. llama.cpp rocm segfaults with R9700 (gfx1201) if integrated GPU (gfx1036) also detected lemonade-sdk/llamacpp-rocm#96); it also sidesteps the multi-device drafter-placement failure discussed in this thread.

So: add RDNA4 / gfx1201 to the "ROCm/HIP works" column for Gemma 4 MTP. Fuller writeup (context sweep + KV-quant A/B): https://github.com/Dixon-Cider/gemma4-mtp-rocm-windows-r9700

[vlascik]

This also happens on 2x5070ti Blackwell CUDA with split-mode tensor.

sched_reserve: resolving fused Gated Delta Net support:
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_k_l58) in a buffer (Meta()) that cannot run the operation (NONE)

[fatrmiloon-hub]

I'm seeing similar behavior here. It appears this "pre-allocated tensor" crash isn't isolated to Vulkan, as it's now being reported on RDNA4 via ROCm/HIP and even on Blackwell CUDA when using split-mode tensor. I agree that improving the error message would be a helpful first step to clarify these backend and multi-GPU limitations for Gemma 4 MTP.

…

On Tue, Jun 30, 2026, 2:45 PM vlascik ***@***.***> wrote: *vlascik* left a comment (ggml-org/llama.cpp#24492) <#24492 (comment)> This also happens on 2x5070ti Blackwell CUDA with split-mode tensor. sched_reserve: resolving fused Gated Delta Net support: D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_k_l58) in a buffer (Meta()) that cannot run the operation (NONE) — Reply to this email directly, view it on GitHub <#24492?email_source=notifications&email_token=CBDVLWQJ5DDPQDVTNWHLXD35CQKHDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUG4ZTIMRVGA42M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4847342509>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CBDVLWTR2S6LNPVHCCES5235CQKHDAVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TINRUGQZTAMBTHEZ2C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CBDVLWUZMIVHYSSFCAO4HDD5CQKHDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUG4ZTIMRVGA42M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/CBDVLWXRBMHHFVD5YGSES3L5CQKHDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUG4ZTIMRVGA42M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[liminfei-amd]

Reproduced on a dual-backend build (-DGGML_HIP=ON -DGGML_VULKAN=ON) with a draft-mtp model.

Root cause (as @Kononnable noted): with --spec-draft-device omitted, the draft lands on a different backend than the target, so its shared pre-allocated KV tensor can't be scheduled (target Vulkan0, draft ROCm1) → the opaque pre-allocated tensor ... (NONE) abort. Workaround: --spec-draft-device <dev> matching --device.

Opened a PR that fails with a clear message (naming both backends + pointing to --spec-draft-device) instead of aborting. Put the guard at KV-cache construction rather than llama_prepare_model_devices so it can name the exact buffers — happy to move it earlier if you prefer. Doesn't enable cross-backend sharing, just makes it diagnosable.

PR: #25232

[RandoTTTest]

Opened a PR that fails with a clear message (naming both backends + pointing to --spec-draft-device) instead of aborting. Put the guard at KV-cache construction rather than llama_prepare_model_devices so it can name the exact buffers — happy to move it earlier if you prefer. Doesn't enable cross-backend sharing, just makes it diagnosable.

PR: #25232

@liminfei-amd if you want to live on the edge, -nkvo disables KV offload and lets you run on cross-backend config with no error. Tested on 8060S with Vulkan0/ROCm0 and ROCm0/Vulkan0 successfully