Eval bug: Model type gemma4_assistant not supported

[zelenukhin]

Name and Version

llama-server --version
version: 9181 (18675b6)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

Core i7 6700k, RTX 3060 12 Gb

Models

Model: gemma-4-31B-it-UD-IQ3_XXS.gguf

Draft gemma-4 31b Model:
https://huggingface.co/AtomicChat/gemma-4-31B-it-assistant-GGUF/

Problem description & steps to reproduce

Try:
llama-server.exe -m c:\gemma-4\gemma-4-31B-it-UD-IQ3_XXS.gguf -md c:\gemma-4\gemma-4-31B-it-assistant.Q4_K_S.gguf --spec-type draft-mtp --host 192.168.1.231 --port 8080 -c 4096 -ngl 38 --mlock -fa on -ctk q8_0 -ctv q8_0 -t 4 -np 1 --chat-template-kwargs "{"enable_thinking": false}"

Error:
Model type gemma4_assistant not supported

First Bad Commit

No response

Relevant log output

Logs

c:\llama.cpp>llama-server.exe -m c:\gemma-4\gemma-4-31B-it-UD-IQ3_XXS.gguf -md c:\gemma-4\gemma-4-31B-it-assistant.Q4_K_S.gguf --spec-type draft-mtp --host 192.168.1.231 --port 8080 -c 4096 -ngl 38 --mlock -fa on -ctk q8_0 -ctv q8_0 -t 4 -np 1 --chat-template-kwargs "{\"enable_thinking\": false}"
�[34m0.01.145.891�[0m �[35mW Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead.
�[0m�[34m0.01.146.965�[0m �[32mI �[0mlog_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
�[34m0.01.146.967�[0m �[32mI �[0mdevice_info:
�[34m0.01.300.746�[0m �[32mI �[0m  - CUDA0   : NVIDIA GeForce RTX 3060 (12287 MiB, 11245 MiB free)
�[34m0.01.300.758�[0m �[32mI �[0m  - CPU     : Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (16308 MiB, 7964 MiB free)
�[34m0.01.300.983�[0m �[32mI �[0msystem_info: n_threads = 4 (n_threads_batch = 4) / 8 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
�[34m0.01.301.719�[0m �[32mI �[0msrv          init: running without SSL
�[34m0.01.302.322�[0m �[32mI �[0msrv          init: using 7 threads for HTTP server
�[34m0.01.303.784�[0m �[32mI �[0msrv         start: binding port with default address family
�[34m0.01.318.308�[0m �[32mI �[0msrv          main: loading model
�[34m0.01.318.552�[0m �[32mI �[0msrv    load_model: loading model 'c:\gemma-4\gemma-4-31B-it-UD-IQ3_XXS.gguf'
�[34m0.01.319.043�[0m �[32mI �[0mcommon_init_result: fitting params to device memory ...
�[34m0.01.319.045�[0m �[32mI �[0mcommon_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
�[34m0.02.983.373�[0m �[35mW load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
�[0m�[34m0.02.983.784�[0m �[35mW load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
�[0m�[34m0.02.984.338�[0m �[35mW load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
�[0m�[34m0.03.034.845�[0m �[35mW load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
�[0m�[34m0.23.835.356�[0m �[35mW llama_context: n_ctx_seq (4096) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
�[0m�[34m0.24.089.583�[0m �[35mW common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
�[0m�[34m0.25.947.661�[0m �[32mI �[0msrv    load_model: loading draft model 'c:\gemma-4\gemma-4-31B-it-assistant.Q4_K_S.gguf'
�[34m0.26.443.958�[0m �[31mE llama_model_load: error loading model: unknown model architecture: 'gemma4_assistant'
�[0m�[34m0.26.443.963�[0m �[31mE llama_model_load_from_file_impl: failed to load model
�[0m�[34m0.26.443.966�[0m �[31mE srv    load_model: failed to load draft model, 'c:\gemma-4\gemma-4-31B-it-assistant.Q4_K_S.gguf'
�[0m�[34m0.26.444.916�[0m �[32mI �[0msrv    operator(): operator(): cleaning up before exit...
�[34m0.26.452.513�[0m �[31mE srv          main: exiting due to model loading error
�[0m
c:\llama.cpp>

[m8rr]

https://huggingface.co/Radamanthys11/Gemma-4-26B-A4B-it-assistant-GGUF

version: 9189 (64b38b561)
built with Clang 19.1.5 for Windows x86_64

-md D:\AI\llama.cpp\models\gemma-4-26B-A4B-it-assistant-Q8_0.gguf --spec-draft-n-max 3 --spec-type draft-mtp --spec-default

0.14.824.453 E llama_model_load: error loading model: unknown model architecture: 'gemma4_mtp'
0.14.824.466 E llama_model_load_from_file_impl: failed to load model
0.14.824.470 E srv    load_model: failed to load draft model, 'D:\AI\llama.cpp\models\gemma-4-26B-A4B-it-assistant-Q8_0.gguf'
0.14.824.492 I srv    operator(): operator(): cleaning up before exit...
0.14.828.569 E srv          main: exiting due to model loading error

[MorenoCLR]

Happened to me too in b9190.

Launch param:

numactl -C0,1,2,3,4,5,6,7 /usr/bin/llama-server \
    -m /home/mclr/Llama-cpp/models/gemma-4-imatrix/google_gemma-4-E4B-it-IQ4_NL.gguf \
    --mmproj /home/mclr/Llama-cpp/models/gemma-4-imatrix/mmproj-google_gemma-4-E4B-it-bf16.gguf \
    --spec-draft-model /home/mclr/Llama-cpp/models/MTP/gemma-4-E4B-it-assistant.Q4_K_M.gguf \
    --spec-type draft-mtp \
    --spec-draft-n-max 3 \
    --alias Gemma-4-E4B-it \
    --tools all \
    -ngl all \
    -b 512 -ub 512 \
    --image-min-tokens 560 --image-max-tokens 2240 \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    -ctk q8_0 -ctv q8_0 -fa on \
    --host 127.0.0.1 --port 8001 \
    --reasoning on \
    --cont-batching \
    --no-mmap --mlock \
    --no-host --metrics \
    --cache-ram 20480 \
    --threads 8 \
    --numa numactl \
    -np 1 --kv-unified

Logs:

0.01.723.981 I srv    load_model: loading draft model '/home/mclr/Llama-cpp/models/MTP/gemma-4-E4B-it-assistant.Q4_K_M.gguf'
0.01.843.250 E llama_model_load: error loading model: unknown model architecture: 'gemma4_assistant'
0.01.843.256 E llama_model_load_from_file_impl: failed to load model
0.01.843.258 E srv    load_model: failed to load draft model, '/home/mclr/Llama-cpp/models/MTP/gemma-4-E4B-it-assistant.Q4_K_M.gguf'
0.01.843.262 I srv    operator(): operator(): cleaning up before exit...
0.01.843.920 E srv          main: exiting due to model loading error

[aldehir]

Gemma 4 MTP is not yet supported.

[jbsolomon]

@aldehir is there any planned support for Gemma 4 MTP or an issue to track support for it? I noticed there's a forked llama.cpp that claims to support it:

https://huggingface.co/AtomicChat/gemma-4-31B-it-assistant-GGUF

[aldehir]

@jbsolomon it is being worked on in #23398

[reasondk]

@aldehir Huggingface download gemma 4 assistant gguf can not load？0.24.688.038 E llama_model_load: error loading model: unknown model architecture: 'gemma4_mtp'
0.24.697.809 E llama_model_load_from_file_impl: failed to load model
0.24.697.815 E srv load_model: failed to load draft model, '/Users/qrh/Downloads/gemma-4-E4B-it-assistant-Q4_K_M.gguf'
0.24.697.831 I srv operator(): operator(): cleaning up before exit...
0.24.698.299 E srv llama_server: exiting due to model loading error

[aldehir]

@reasondk please see the PR #23398 for more information, am17an created a GGUF containing the MTP layers. You must also compile the PR yourself for now, as it is a work in progress.

[Erquint]

The PR has been merged. Somebody should report it working here and close the issue, perhaps.

[mohamed-em2m]

i tried it but same bug repeated llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant'

[m8rr]

@mohamed-em2m You have to use the one that was converted for llama. Ones converted from different forks will not work.

[Novacane01]

@mohamed-em2m You have to use the one that was converted for llama. Ones converted from different forks will not work.

ok but do you have a link to an example?

[m8rr]

@Novacane01 His problem was that he used an older version of Llama. The GGUFs from Unsloth or any GGUFs released after the Llama update should work just fine.

work: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF
not work:
https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF
https://huggingface.co/Radamanthys11/Gemma-4-26B-A4B-it-assistant-GGUF

[Erquint]

I used G4 MTP GGUF sidecar in KCPP, which is downstream of LCPP.
It was as ineffective as speculative decoding always is, but it worked.