Eval bug: gemma4-assistant MTP draft model fails to load — "invalid vector subscript" (regression: works on b9553, broken on b9702/b9717)

[Metodv1]

Name and Version

Affected (BROKEN):
version: 9717 (8141e73) built with Clang 20.1.8 for Windows x86_64
version: 9702 (968c438)

Last known good (WORKS):
version: 9553 (9e3b928) built with Clang 19.1.5 for Windows x86_64

Official Windows CUDA 13.3 release builds.

Operating systems

Windows

GGML backends

CUDA

Hardware

Ryzen 7 5800X + 4070Ti Super 16gb

Models

Speculative decoding (MTP draft) setup:

• Target: gemma4-v2-Q8_0.gguf (arch: gemma4)
• Draft (MTP): gemma-4-12B-it-MTP-Q8_0.gguf (arch: gemma4-assistant, nextn_predict_layers=4, block_count=4)

Both from: https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF

The draft is the unsloth-converted GGUF of Google's official gemma-4-12B-it-assistant draft head.
Gemma 4 MTP support landed in mainline via PR #23398 (merged 2026-06-07). The model author confirmed
the files are correct and not built against any private fork. Also reproduced with the -MTP-BF16 variant.

Problem description & steps to reproduce

Loading a gemma4-assistant MTP draft model for speculative decoding (--spec-type draft-mtp) crashes
during model load with "invalid vector subscript". The main/target model loads fine — only the draft
load crashes, so generation never starts.

This is a REGRESSION. The exact same models + command load cleanly and work on b9553 (~2.5x speedup,
lossless), but crash on b9702 and b9717. Something in the gemma4-assistant loader / context-init path
regressed between b9553 and b9702.

Command used (crashes on b9702/b9717, works on b9553 with the older flag names):

b9702/b9717 (new flag names):
llama-server -m gemma4-v2-Q8_0.gguf
--spec-type draft-mtp --spec-draft-model gemma-4-12B-it-MTP-Q8_0.gguf
--spec-draft-ngl 99 --ctx-size 65536 -ngl 99 --flash-attn on --jinja

b9553 (old flag names, WORKS):
llama-server -m gemma4-v2-Q8_0.gguf
--model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp
--spec-draft-n-max 4 -ngl 99 -ngld 99 -fa on --jinja

Key detail: the "invalid vector subscript" fires the instant load_tensors: loading model tensors...
begins for the draft — before any tensor data is read. So the out-of-bounds is in the gemma4-assistant
tensor-layout / model-build code, not in reading tensor data. Likely an out-of-range index over a
per-layer array (block_count=4), e.g. attention.head_count_kv=[8,8,8,1] or
sliding_window_pattern=[true,true,true,false], or nextn_predict_layers / embedding_length_out=3840 handling.

The crash occurs AFTER the memory-fitting step (which harmlessly aborts with "n_gpu_layers already set
by user to 99, abort"), at the actual draft load — so -fit off does not change it. The
"Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)" line also
appears on the working b9553 build and is harmless; the fatal regression is the "invalid vector subscript".

Tried — all crash identically on b9702/b9717, both quants:
b9702 + Q8_0 -> crash
b9717 + Q8_0 -> crash
b9717 + BF16 -> crash
b9553 + Q8_0 -> WORKS

So it is neither a file/quant issue nor a user-config issue. It is in the model loader and reproduces
with any binary that loads this draft.

First Bad Commit

Not bisected. Good: b9553 (9e3b928). Bad: b9702 (968c438) and b9717 (8141e73).
The regression was introduced somewhere in the b9553 -> b9702 range.

Relevant log output

Logs

 srv    load_model: loading draft model '...gemma-4-12B-it-MTP-Q8_0.gguf'
  load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
  llama_model_load: error loading model: invalid vector subscript
  llama_model_load_from_file_impl: failed to load model
  srv    load_model: failed to load draft model, '...gemma-4-12B-it-MTP-Q8_0.gguf'
  srv  llama_server: exiting due to model loading error
  A pod blokiem console (poza nim) wklej link do nowego gista:
  Full verbose (-lv 5) log from b9717: <https://gist.github.com/Metodv1/e6e06c9058fdd0b32881699e4ae00e24>

[HelloItMeMort]

Confirming regression as well

[poog26]

I'm experiencing this as well.

[Kononnable]

0.09.272.097 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4070 Ti SUPER) (0000:08:00.0) - 0 MiB free
Looks like a lack of resources, not a regression. You don't seem to have enough resources to run this quant with specified settings.

Why did it work before? Probably other programs were using less VRAM or some changes in llama.cpp increased the memory consumption which put your configuration over the line (increasing memory usage does not necessarily mean regression). You may want to try after closing all unnecessary applications(to free vram), or bisect if you still think this is a llama.cpp issue.

[Snegovik]

I have too. LLAMA_CPP_VERSION=b9672

Error: 500 Internal Server Error: llama-server process has terminated: llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)

llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting): llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)

llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)

[samos667]

llama-server -m gemma4-v2-Q8_0.gguf
--spec-type draft-mtp --spec-draft-model gemma-4-12B-it-MTP-Q8_0.gguf
--spec-draft-ngl 99 --ctx-size 65536 -ngl 99 --flash-attn on --jinja

Using b10091 the model load but use 18Go of VRAM

[fishchev]

Not bisected. Good: b9553 (9e3b928).

Repeatable in the good commit.
This seems to be just insufficient VRAM during MTP .gguf loading.
With -lv 4 you'll see:

0.26.594.938 I common_speculative_init_result: loading draft model 'mtp-gemma-4-31B-it.gguf'
...
0.26.925.530 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:01:00.0) - 0 MiB free

---

I'm not familiar with the codebase, but assuming a single GPU, here's control flow in load_tensors:

free is 0, but total is reported. splits[0] is set to 0:

llama.cpp/src/llama-model.cpp

Lines 1330 to 1338 in 8b86400

	ggml_backend_dev_memory(dev, &free, &total);
	// devices can return 0 bytes for free and total memory if they do not
	// have any to report. in this case, we will use the host memory as a fallback
	// fixes: https://github.com/ggml-org/llama.cpp/issues/18577
	if (free == 0 && total == 0) {
	ggml_backend_dev_memory(cpu_dev, &free, &total);
	}
	splits[i] = free;

then splits[0] /= 0, so NaN:

llama.cpp/src/llama-model.cpp

Lines 1345 to 1352 in 8b86400

	float split_sum = 0.0f;
	for (size_t i = 0; i < n_devices(); ++i) {
	split_sum += splits[i];
	splits[i] = split_sum;
	}
	for (size_t i = 0; i < n_devices(); ++i) {
	splits[i] /= split_sum;
	}

layer_gpu, when provided with { NaN }, evaluates to 1, which is an invalid index for devices:

llama.cpp/src/llama-model.cpp

Lines 1362 to 1363 in 8b86400

	const int layer_gpu = std::upper_bound(splits.begin(), splits.begin() + n_devices(), float(il - i_gpu_start)/act_gpu_layers) - splits.begin();
	auto * dev = devices.at(layer_gpu).dev;

---

I guess it's due to lack of MTP memory fitting implementation: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting. MTP gets moved without any fit check to GPU if main model was fully offloaded.

Since MTP is fairly small (1 GB at 131K?), that works okay. But if you have just enough memory for the model, but not MTP -- you get this bug.

On Windows WDDM free-memory shenanigans don't help and this issue easier to stumble into:

# no matter the actual VRAM usage, llama.cpp sees the same amount of free VRAM at startup:
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:01:00.0) - 23300 MiB free

# on the second free memory query / log (after the main model is loaded, right before loading MTP .gguf):
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:01:00.0) - 1412 MiB free
# 1412 MB is more than enough for MTP, so it'll succeed

# but if there's VRAM used by other processes, you could randomly get this
# (despite the same 23300 MB always being reported as free at startup):
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:01:00.0) - 0 MiB free

So it seems that even if MTP memory fitting was fixed, the issue would remain (on Windows), as WDDM lies about free memory at process startup and then dynamically walks back on available free memory.
So, to trigger this on Windows, you could have nowhere near enough free GPU memory for the model itself. First allocation for the model itself would succeed. But whent it's time to load MTP, WDDM would report 0 free.

I guess it's not worth fixing. But adding clear error about 0 VRAM would be great. Right now it's only an info log, which you wouldn't see without -lv 4.