Repository navigation
Eval bug: Qwen3.5 gibberish output with -nkvo (Vulkan) #23827
Description
Name and Version
version: 9396 (445b7ce)
built with GNU 15.2.1 for Linux x86_64
Operating systems
Linux
GGML backends
Vulkan
Hardware
RX 7900 XT, Ryzen 3900x
Models
I used Qwen3.5 4B (quant by unsloth) for the test, but Qwen3.6 and the MOEs also display this behavior.
Problem description & steps to reproduce
Using -nkvo makes the output completely wrong:
#!/bin/bash
LLAMA_DIR=/home/ja/llm/llama.cpp/
$LLAMA_DIR/build/bin/llama-completion --version
printf "Result without nkvo:\n"
llama-completion --device Vulkan0 -fit off --simple-io -lv 0 -no-cnv -st \
-m ~/Downloads/Qwen3.5-4B-UD-Q4_K_XL.gguf -p "Letters of the alphabet in order:" --n-predict 10
printf "Result with nkvo:\n"
llama-completion --device Vulkan0 -fit off --simple-io -lv 0 -no-cnv -st \
-m ~/Downloads/Qwen3.5-4B-UD-Q4_K_XL.gguf -p "Letters of the alphabet in order:" --n-predict 10 \
-nkvoOutput (b9396):
version: 9396 (445b7cef6)
built with GNU 15.2.1 for Linux x86_64
Result without nkvo:
Letters of the alphabet in order:
A B C D E F G H I
Result with nkvo:
Letters of the alphabet in order:"),0%<;@+:
This is consistent across runs. The second one is always some form of gibberish, but the first one outputs letters or at least tries to begin a thinking process.
Output (b8183, appears to be the last working build):
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
version: 8183 (66d65ec29)
built with GNU 15.2.1 for Linux x86_64
Result without nkvo:
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Letters of the alphabet in order:
<think>
Thinking Process:
1.
Result with nkvo:
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Letters of the alphabet in order: A B C D E F G H I J
First Bad Commit
i'm not sure, but b8184 is the first build that doesn't work, with Commit 3191462 (vulkan: improve partial offloading performance on AMD (#19976))
Edit: Apparently, somewhere near b8248 it started working, but now it's broken again. I'll be trying to bisect it better.
Results:
- 8210 seems to have fixed it for a moment, the letters are right while they're not on 8209
- 8248 works too, as mentioned, but i didn't check every release to rule out any intermittent fixes
- 8323 broke it again
Activity
inforithmics commentedon May 30, 2026
I have the same issue with an igpu AMD 780m and Qwen 3.6 and b9415
this command line outputs giberish too
llama-cli -m c:\models\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -nkvo -p "Write a hello world programm"
This is on windows, after a certain time the driver crashes and windows restarts.
suka2137 commentedon May 30, 2026
Not great, but what's the point? The IGPU is supposed to share the system memory, so unless there's something I'm missing, I'm pretty sure that there's no benefit or even defined behavior to using nkvo here, because it's already in regular RAM.
suka2137 commentedon May 30, 2026
Turns out it works with ROCm in this case.
inforithmics commentedon May 30, 2026
Intrestingly it works with a Radeon VII graphic card.
Concerning the usefulness on AMD iGpu there are two memories dedicated vram, shared vram and system memory and the fastest is dedicated vram even if it is all physically the same ram.
So maybe this is Driver related.
aqrig1 commentedon May 31, 2026
0cc4m commentedon May 31, 2026
That is not this issue, but one of the MoltenVK AMD ones.
0cc4m commentedon Jun 1, 2026
There should not be a difference, it's the same RAM. Are you sure you tested with the optimal configuration? There are some things like IOMMU that will slow it down if enabled.
I can reproduce this issue, it is not driver-related. It is some kind of synchronization bug. You can work around it by running with
GGML_VK_DISABLE_ASYNC=1. The reason is theggml_vk_synchronizecall at the end of graph_compute that this enables.It also goes away if async host->GPU copies are disabled. I'm still trying to find the actual problem. (@jeffbolznv let me know if you have an idea)
jeffbolznv commentedon Jun 1, 2026
I debugged this some with codex. It looks like after a call to cpy_tensor_async, ggml-backend can reuse the source (host) buffer without synchronizing. ggml-cuda doesn't have this problem because it rejects cpy_tensor_async when the source is a host buffer.
As usual, there is basically no documentation on how this backend interface is supposed to behave, so I don't know whether we should change ggml-vulkan or ggml-backend.
github-actions commentedon Jul 17, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
inforithmics commentedon Aug 22, 2026
Is fixed now I think #26040 fixed it