Repository navigation
gfx1151 vision regression root-caused: upstream prop.integrated on HIP — cross-platform verification wanted (and a lead on #17475) #313
Description
Status: root cause found and fixed on ROCm/gfx1151. Posting because the same defect class may reach CUDA unified-memory devices, and because it bears directly on gate issue ollama#17475 — which we have never been able to reproduce.
What happened
ollama 0.34.1 (llama.cpp b10864) silently destroys vision quality on gfx1151. Measured against the 2026-09-17 baseline (#303), five-model campaign, same runner and digests:
| model | 0.32.1 scene IoU | 0.34.1 | 0.34.1 +906 |
|---|---|---|---|
qwen3.8:27b-q4_K_M |
0.991 | 0.065 | 1.000 |
gemma4:31b-it-q4_K_M |
0.961 | 0.922 (3/6 objects) | 0.960 (6/6) |
nemotron3:33b-q4_K_M |
0.857 | 0.161 | 0.862 |
Invoice extraction went 5/5 → 0/5 → 5/5. Fine-text OCR 4/4/4 → 0/0/0 → restored.
Root cause
Upstream c7d8722922a (#24233) set info.devices[id].integrated = prop.integrated on HIP builds. On an integrated/UMA device that changes which buffer types the backend claims, and upstream #28211 root-causes the consequence as an MMQ tile-barrier race on gfx115x that produces wrong output with no crash and no warning, once a decode exceeds n_ubatch.
Upstream reverted it in #28604, merged 2026-09-08T14:19:53Z. Our payload b10864 is 5d806aa25, 2026-09-08T13:01:03Z. We missed the fix by 78 minutes.
Carried as llama/compat/906-revert-hip-integrated-flag.patch — four lines.
Two strangers reported our exact symptom on our exact silicon before we started: #27419 ("upper image content is lost while Vulkan/CPU are correct", Ryzen AI Max+ 395 / 8060S / gfx1151, Qwen3.8-27B) and #28211.
Why it looked like an image-budget bug
The failure tracks the image-token budget exactly — name_bbox_mean_iou on gemma4:31b with the budget pinned per cell:
| budget | 0.32.1 | 0.34.1 |
|---|---|---|
| 280 | 0.612 | 0.691 |
| 560 | 0.713 | 0.714 |
| 1120 | 0.728 | 0.000 |
Because 1120 image tokens push the decode past n_ubatch while 280 and 560 do not. prompt_eval_count is identical between builds throughout, so the grid and the pixels reaching the model never changed.
What CUDA could verify for us
This is the ask that matters most, and it is not hypothetical.
The reverted line is inside #if defined(GGML_USE_HIP), so a CUDA build does not compile it. But the integrated flag and the host-buffer path it selects exist for CUDA too, and gate issue #17475 — the cross-request content leak that half our AMD gate rests on — was reported on a DGX Spark GB10, a unified-memory CUDA device. We ran 436 victim extractions against it on gfx1151 and could not reproduce it (#304).
A silent, deterministic corruption that only appears once a decode exceeds n_ubatch would look a great deal like a "leak" to someone who did not know to pin the batch size.
Concretely, if you have a unified-memory CUDA device (Jetson/Orin/DGX Spark/GB10):
- Does
ggml_cuda_init()reportintegrated = trueon it? One log line settles it. - Run the budget ladder —
run_budget_sweep.shwithBUDGETS="280 560 1120"ongemma4:31b,min == maxpinned. We are looking for a cliff at the rung where image tokens exceedn_ubatch, not a gradual decline. Checkprompt_eval_countmoves with the budget or the cell is meaningless (SPEC B4). - If there is a cliff, re-run Cross-request content leakage on shared slot (-np 1, context-shift): concurrent vision requests + client aborts leak one image's data into another request's output (0.32.5) ollama/ollama#17475's protocol with
num_batchraised so the image fits one sub-batch. If the "leak" disappears, Cross-request content leakage on shared slot (-np 1, context-shift): concurrent vision requests + client aborts leak one image's data into another request's output (0.32.5) ollama/ollama#17475 is this bug wearing a disguise and the gate's second pillar falls.
On a discrete CUDA card we would expect all three to come back clean — which is still worth knowing, as it bounds the blast radius.
What Metal/MLX could verify for us
Metal does not use ggml-cuda at all, so it cannot carry this defect. Two things are still useful:
- The budget ladder as a control. Same sweep on Apple GGML. If there is no cliff at 1120, that confirms the ladder itself is sound and the cliff is backend-specific. Note per docs(compat): 004 is not redundant at b10864, and why #309 that MLX does not use patch 004 and gets its geometry from
llm.BudgetFillSize— so this also checks the two paths have not silently diverged. - The greedy non-reproducibility in docs(vision): SOLVED — the 0.34.0 gemma4 vision shift is upstream MLX #3912 #310. You measured
evalvarying across runs attemperature 0, top_k 1on 0.34.0; we see the same thing on ROCm and had written it off as warm/cold noise. We will test whether 906 fixes ours. If it does and yours persists, they are distinct bugs and your5844a6cd5/ scoped-lifetimes hypotheses stand on their own. If ours persists too, it is a third thing and neither of us should attribute it to the fold's memory work yet.
Already refuted here — please do not re-run these
Each killed by experiment, not argument:
- fp16 accumulation in the vision tower —
GGML_CUDA_CUBLAS_COMPUTE_TYPEf32 / f16 / unset all equally broken, env var verified to reach thellama-serverprocess - multi-sub-batch image decode — at
num_batch 2048the image decodes in a singledecoding image batch 1/1, n_tokens_batch = 2040and is still broken - the b10864 resize-algo flip (
RESIZE_ALGO_BILINEAR→BICUBICfor GEMMA4V/QWEN3VL, #27594) — reverted as a compat patch, verified present in-binary by output-hash diff at identicalprompt_eval/eval, fixes nothing - the gemma4 token-budget bump (#28335) — it creates the 1120 rung that exposes the bug, but is not itself the defect
GGML_CUDA_NO_PINNED=1— not a workaround; the runner aborts on load (signal: aborted, core dumped)- direct I/O, GPU layer placement, host drift, and our compat series (stock
ollama/ollama:0.34.1-rocmreproduces it identically)
Caveats
- Two of five models still running in the confirmation campaign at time of writing.
gemma4:31bname_bboxreturns to 0.663–0.72 against the baseline's 0.728 — restored, but repeats are needed before calling it equivalent, especially given the non-determinism above.- Everything here is gfx1151 / ROCm 7.2.x. The CUDA and Metal questions above are open precisely because we cannot answer them.
🤖 Generated with Claude Code
Activity
glennneuber commentedon Sep 18, 2026
CUDA host (discrete: RTX PRO 6000 Blackwell, sm_120, 96 GB; no unified-memory CUDA device on this estate, so the GB10/Jetson half of the ask stays open). Answers to the three items, in order, all on the deployed
0.34.1-dynres-0-g8a7ba94image, llama.cpp b10864, the same payload commit as yours.1.
integratedon this cardDiscrete, so
cudaDeviceProp.integratedis 0 and the host-buffer path is not selected. The observable that matters is where the buffers land: the runner log puts model, KV and compute buffers all on the device (CUDA0 model buffer size = 17790 MiB,CUDA0 KV buffer size,CUDA0 compute buffer size), nothing onCUDA_Hostbeyond the usual token-embedding/output residue. ollama's owninference computediscovery line in this build carries nointegratedfield, andggml_cuda_init's device lines do not reach the ollama log at INFO, so there is no single log line to quote; the buffer placement is the evidence.2. The budget ladder,
min == maxpinned, gemma4:31bYour protocol exactly:
run_budget_sweep.sh,BUDGETS="280 560 1120",min == maxpinned per cell,scene_single+document_single, think-off, num_ctx 16384 (so the automatic batch is 1024 and the 1120 rung decodes pastn_ubatch, which is the condition your cliff sits on). Generator output (summarize_head_to_head.py --tags):No cliff. Scene IoU rises with the budget and
name_bboxis flat where yours went to 0.000. SPEC B4 holds and, more usefully, the geometry is byte-identical to yours:prompt_eval_counthere is 848 / 614 at 280, 1111 / 887 at 560, 1684 / 1447 at 1120 (scene / document), which are your b10864 figures digit for digit. Same pixels reach the model on both hosts, and the runner log shows the 1120 cells decoding as1/2 n_tokens_batch = 1024+2/2of 76 and 65 tokens, so the "decode exceedsn_ubatch" condition was met and was harmless here. That bounds the blast radius the way you hoped: the defect needs the HIPintegratedpath, not the batch geometry.3.
num_batchand the split image decode, measured directlyYour refuted list says the image decoding in a single sub-batch at
num_batch 2048is still broken on gfx1151, so the split was never your defect. Here is what the split itself does on discrete CUDA, because the fork's task doc had it staged as an open question ("gemma4 image chunk vs. batch"): the two gemma4 GGUF models that decode images non-causally, think-off, num_ctx pinned at 8192, two repeats per arm, one container, generator tables (preflight-runs/batchab-0341-render-thinkfalse.md).The runner log shows the mechanism, not an inference: at the scheduler's automatic batch for that rung (n_batch = n_ubatch = 1024) every image chunk on these fixtures (1064–1100 tokens) is
decoding image batch 1/2, n_tokens_batch = 1024then2/2of 40–76 tokens; withnum_batch: 2048per request it is1/1.Every scored cell is identical between repeats on both arms, so this path is deterministic here. Three readings:
ggml0341_1_1_), and the split at that rung is exactly what production has been serving.For ollama#17475's protocol: not run here, and this card is not the reported device class (unified memory). If a GB10/Spark ever lands on this estate the three steps above are the first thing it gets.
🤖 Generated with Claude Code
task: #313's pinned budget ladder on discrete CUDA — no cliff at 1120…
glennneuber commentedon Sep 20, 2026
Status update: the ROCm half is closed upstream — compat 906 has been retired
Since this was opened, the fix landed in our payload and the carry-patch is gone.
llama/compat/906was retired in the 0.34.2 fold (3dade569, "b10969 ships upstream's own revert"), so gfx1151 now gets upstream #28604 directly rather than through a patch we maintain.Verified on the payload without the patch — same five-model campaign, same runner, think-off,
num_ctx16384. Generator output (summarize_head_to_head.py --tags):qwen3.8:27bnemotron3:33bgemma4:31bAgainst the 0.065 / 0.161 collapse this issue documents, all three are at or above baseline with no patch carried. Invoice extraction 5/5 and scene labels 6/6 on every arm. Clause 4 passes with zero degenerate rows on both the 7.2.4 and the ROCm 10.0.0 artifacts.
What remains open — and it is the half nobody here can run
The one comment on this issue answered the discrete-CUDA side (RTX PRO 6000 Blackwell,
integrated= 0, buffers on device, no reproduction) and flagged that "the GB10/Jetson half of the ask stays open" because there is no unified-memory CUDA device on this estate.That is still true, and it is the half that matters for the original question — whether this defect class reaches CUDA unified memory, and therefore whether ollama#17475 is this bug in disguise. A discrete card cannot answer it:
prop.integratedis 0, so the code path under suspicion is never selected.So this issue should stay open, but its scope has narrowed to one thing: someone with a GB10, Jetson, or any CUDA device where
cudaDeviceProp.integratedis 1, running the budget-ladder protocol in the body withmin == maxpinned at the 1120 rung so the decode exceedsn_ubatch. Everything else asked for here has been answered.Worth noting for whoever picks that up: on our side the cliff was silent — no crash, no warning, correct-looking output with wrong boxes. If you run this, score the output; do not read the logs and conclude it is fine.