Repository navigation
Eval bug: upper image content is lost while Vulkan/CPU are correct #27419
Description
Name and Version
version: 0.1.2-dev (build 402, commit 990e3bf)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
HIP
Hardware
AMD Ryzen AI Max+ 395, Radeon 8060S, gfx1151, 128 GB unified memory with 96 GB dedicated to the iGPU.
Models
Model: Qwen3.8-27B-UD-Q4_K_XL.gguf
Projector: mmproj-BF16.gguf
Problem description & steps to reproduce
The diagnostic image has ten horizontal rows:
1 RED APPLE
2 BLUE CAR
3 GREEN TREE
4 YELLOW BANANA
5 BLACK CAT
6 PURPLE GRAPES
7 BROWN DOG
8 PINK FLOWER
9 GRAY PLANE
10 ORANGE ORANGE
Run the server:
./build-hip/bin/llama-server \
-m Qwen3.8-27B-UD-Q4_K_XL.gguf \
--mmproj mmproj-BF16.gguf \
-ngl 999 \
-c 262144 \
--parallel 1 \
-t 16
Ask the model to "Ignore the positions and ordering of everything in the image. List every distinct numbered label, color, object word, and pictured object that you can actually see anywhere in the image. Do not infer or reconstruct anything that is not visibly present."
On HIP, Qwen consistently sees only approximately the bottom 40%:
7
8
9
10
PINK
GRAY
ORANGE
FLOWER
PLANE
ORANGE
dog
flower
plane
orange
On Vulkan with the same commit 990e3bf, same model, same mmproj, same image, same request, it correctly identifies all rows 1–10, all ten colors, all ten object labels, and all ten pictured objects.
On CPU, it also correctly identifies all rows 1–10.
I managed to reproduce the same original failure with both the stock Unsloth Qwen3.8 and the HauhauCS Q5_K_P checkpoint. Removing MTP made no difference. The issue doesn't appear to be tied to the uncensored checkpoint, quantization variant, or MTP.
First Bad Commit
No response
Relevant log output
Logs
$ ./build/bin/llama-server --version
version: 0.1.2-dev (build 402, commit 990e3bf)
built with GNU 13.3.0 for Linux x86_64
$ ./build-hip/bin/llama-server --version
version: 0.1.2-dev (build 402, commit 990e3bf)
built with GNU 13.3.0 for Linux x86_64
$ git rev-parse --short HEAD
990e3bf
=== HIP backend, -ngl 999 ===
Model:
Qwen3.8-27B-UD-Q4_K_XL.gguf
Projector:
mmproj-BF16.gguf
Expected image contents:
1 RED APPLE
2 BLUE CAR
3 GREEN TREE
4 YELLOW BANANA
5 BLACK CAT
6 PURPLE GRAPES
7 BROWN DOG
8 PINK FLOWER
9 GRAY PLANE
10 ORANGE ORANGE
Response:
- 7
- 8
- 9
- 10
- PINK
- GRAY
- ORANGE
- FLOWER
- PLANE
- ORANGE
- dog
- flower
- plane
- orange
HIP therefore appears to lose all visual content from approximately rows 1-6.
=== Vulkan backend, -ngl 999 ===
Same model, projector, image, prompt and llama.cpp commit.
Response identifies:
Numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Colors: RED, BLUE, GREEN, YELLOW, BLACK, PURPLE, BROWN, PINK, GRAY, ORANGE
Objects: APPLE, CAR, TREE, BANANA, CAT, GRAPES, DOG, FLOWER, PLANE, ORANGE
All 10 pictured objects are also identified correctly.
=== CPU baseline, -ngl 0 --no-mmproj-offload ===
Same model, projector, image and prompt.
Response correctly identifies:
1 RED APPLE
2 BLUE CAR
3 GREEN TREE
4 YELLOW BANANA
5 BLACK CAT
6 PURPLE GRAPES
7 BROWN DOG
8 PINK FLOWER
9 GRAY PLANE
10 ORANGE ORANGE
=== Additional HIP isolation tests ===
HIP + --no-mmproj-offload:
Still sees only rows 7-10.
HIP + GGML_CUDA_DISABLE_GRAPHS=1:
Still sees only rows 7-10.
Activity
grimwiz commentedon Aug 20, 2026
Independent reproduction on the same silicon, via a different stack (Ollama's
bundled llama.cpp rather than a self-built
llama-server) and a differentmodel source (Ollama library weights, not the Unsloth GGUF). Same three-way
outcome: HIP wrong, Vulkan correct, CPU correct.
Environment
Reproducer (synthetic, deterministic ground truth)
Rather than a photograph, a 128x98 PNG generated in-process containing three
lines of six digits in a 5x7 bitmap font at 3x scale, black on white, 10 px
margins:
Prompt: "Read all three lines of digits in this image. Reply with the three
numbers separated by commas, nothing else." Sent to
/api/chatwith"think": falseand nonum_predictcap.The digit strings cannot be guessed, so any correct answer is evidence the
pixels arrived. I had the generated PNG independently inspected to confirm it
renders as intended and that no line is clipped.
Results
Backend confirmed from the Ollama journal in every case, not inferred:
clip_ctx: CLIP using ROCm0 backendclip_ctx: CLIP using Vulkan0 backendsize_vram=0.0GBHIP output,
think=false, three consecutive runs:Vulkan, same request, three consecutive runs:
A control on unrelated NVIDIA/CUDA hardware, same model digest, same image:
correct 3/3.
Additional data points
Coarse image content is read correctly on HIP. A 210x60 PNG of seven solid
vertical colour bands is answered correctly on HIP ("7 stripes, leftmost
orange"), repeatedly. On this image the HIP failure is specific to fine detail.
thinkchanges what the failure looks like. Withthink=false, all threelines are wrong. With thinking enabled, the lower two lines come back correct
and only the top line is wrong, e.g.
'195139, 605218, 927364'— the wrongvalue differing on each run. Anyone reproducing this with a thinking-enabled
model may therefore see a partially-correct answer.
Two different projectors both fail on HIP. Besides the library model above,
a locally-built tag using a separately supplied
Qwen3.8-27B-mmproj.gguf(1M-context GGUF with YaRN in the header) fails the same way on HIP.
Varied without effect on HIP (all still wrong):
OLLAMA_KV_CACHE_TYPE=q8_0vs default f16;
OLLAMA_FLASH_ATTENTION=1vs unset; context 32768 / 262144 /1048576; with and without YaRN rope scaling; output token cap present or absent.
Notes / limits of this report
does not report the upstream revision, and I have not built llama.cpp directly
on this machine.
test-backend-opson this host.OLLAMA_IGPU_ENABLE=1as well asOLLAMA_VULKAN=1; without it Ollama logsdropping integrated GPU; to enable, set OLLAMA_IGPU_ENABLE=1and falls backto ROCm (or to CPU if HIP is also hidden). Worth knowing for anyone trying to
compare the two backends on Strix Halo.
nasal commentedon Sep 21, 2026
This has now been root-caused and fixed upstream.
Issue #28211 identified the failure as silent HIP corruption on gfx1151 when the integrated/UMA host-buffer path is enabled and a decode exceeds n_ubatch. PR #28604 reverted the problematic prop.integrated handling and was merged as d4389a4.
I independently reran this issue’s original image/model/projector reproducer on the original 990e3bf build:
That workaround masks the corruption by serializing ubatches, but #28604 is the proper upstream fix. Current master includes it, and downstream testing reported restored vision quality across multiple models on gfx1151.
Closing as fixed by #28604; see #28211 for the detailed root-cause analysis.