Repository navigation
HIP backend silently corrupts Qwen3.5-27B (Gated DeltaNet) inference on gfx1151 - oldest context lost; Vulkan correct at identical commit #27556
Description
Summary
On gfx1151 (Strix Halo / Ryzen AI Max+ 395, Windows 11) the HIP backend silently corrupts inference of Qwen3.5-27B (hybrid Gated-DeltaNet architecture, tested as VLM with mmproj): the oldest part of the context is consistently lost while output stays fluent and deterministic. The Vulkan backend at the identical commit is correct. We ran an extensive exclusion campaign (details below); every individually testable component passes, so the defect appears to live at graph-composition level of the GDN block on HIP.
Environment
- HW: AMD Ryzen AI Max+ 395, Radeon 8060S iGPU (gfx1151), 128 GB unified memory
- OS: Windows 11 Pro 26200, Adrenalin driver 32.0.31035.1003
- ROCm runtime: TheRock
therock-dist-windows-gfx1151-7.14.0rc3(also reproduced with a different ROCm 7.x runtime DLL set) - llama.cpp: official release zips
llama-b10453-bin-win-rocm-7.14-x64.zipandllama-b10582-bin-win-rocm-7.14-x64.zip, and a self-built b10582 (e85caa8) with the TheRock clang toolchain — all three affected identically - Model: Qwen3.8-27B Q8_0 (unsloth GGUF) + mmproj BF16
- Counter-checks: Vulkan builds of the very same commits (b10453 and b10582) are correct; Qwen3-VL-30B-A3B (pure attention, no GDN) is correct on the same HIP builds
Reproduction (deterministic, temp 0)
Generate a synthetic 4-panel test image (no external assets needed):
from PIL import Image, ImageDraw, ImageFont
W,H=660,640; img=Image.new('RGB',(W,H),'white'); d=ImageDraw.Draw(img)
try: fb=ImageFont.truetype('arialbd.ttf',20); fs=ImageFont.truetype('arial.ttf',14)
except: fb=fs=ImageFont.load_default()
for x,y,t,c in [(0,0,'Alpha Risk','Risk from parallel shifts of the reference rate.'),
(330,0,'Beta Risk','Risk from non-parallel curve changes.'),
(0,320,'Gamma Risk','Risk from embedded options, e.g. early repayment.'),
(330,320,'Delta Risk','Risk from hedging with different rate indices.')]:
d.rectangle([x+2,y+2,x+328,y+318],outline='black',width=2)
d.ellipse([x+12,y+16,x+22,y+26],fill='black'); d.text((x+30,y+12),t,fill='black',font=fb)
d.rectangle([x+30,y+50,x+290,y+200],outline='gray')
if 'Gamma' in t:
for i,bh in enumerate([60,90,130,100,80]): d.rectangle([x+50+i*45,y+195-bh,x+80+i*45,y+195],fill='gray')
else: d.line([(x+40+i*25,y+190-int(12*(i**1.2))) for i in range(10)],fill='black',width=2)
d.text((x+30,y+215),c,fill='black',font=fs)
img.save('synthetic_4panel.png')llama-mtmd-cli -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-Qwen3.8-27B-BF16.gguf \
-ngl 99 -c 4096 --temp 0 --seed 42 --image synthetic_4panel.png \
-p "List ALL bold bullet headings in this image, top-left to bottom-right. Only the list."
- HIP (gfx1151):
['Beta Risk', 'Gamma Risk', 'Delta Risk']— top-left "Alpha Risk" (the oldest image tokens in raster order) is dropped - Vulkan (same commit, same machine): all four headings
- With a denser real-world 4-panel document scan the effect is stronger: HIP loses the entire upper half (0/4 headings) while Vulkan stays 4/4; flipping the image vertically "heals" it (the previously-top content becomes the newest tokens), which points at recency-position rather than image-coordinates.
Exclusion campaign (everything below was tested on the same machine)
| Hypothesis | Test | Result |
|---|---|---|
| Runtime-DLL mismatch | official b10453 binary + clean matching TheRock 7.14 runtime | still wrong |
| Code regression / fix between builds | b10453 and b10582 built for Vulkan (same commits) | both correct → code exonerated |
| Toolchain | self-built HIP b10582 (TheRock clang) | reproduces official zips exactly |
| mtmd preprocessing | top-half crop alone is read fine; full preprocessing chain reviewed | exonerated |
| hipGraphs | GGML_CUDA_DISABLE_GRAPHS=1 |
still wrong |
| ubatch state handoff | -ub 128 / 512 / 8192 (whole prompt in one ubatch) |
still wrong |
| GATED_DELTA_NET kernel | test-backend-ops -o GATED_DELTA_NET, including added prefill-scale cases (n_seq_tokens 64/256/1024 at the real 32x128 geometry, v_repeat 2, and K=4/8 snapshot slots — upstream currently tests max n_seq_tokens=4 and K>1 only with tiny shapes) |
43/43 pass at NMSE 1e-7 |
| Any other op in the graph | per-op sweep: SSM_CONV, CUMSUM, L2_NORM, RMS_NORM, EXP, SIGMOID, SOFT_MAX, ADD, MUL, SCALE, CPY, SET_ROWS, GET_ROWS, ROPE, FLASH_ATTN_EXT, GLU, SUM_ROWS, TRI, MUL_MAT_ID, MUL_MAT (>6400 cases total) | all pass |
Test-case additions we used (happy to open a PR):
test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 1, 1));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 64, 1));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 256, 1));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 1024, 1));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 256, 1, 2));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 256, 1, 1, false, false, 4));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 1024, 1, 1, false, false, 4));
+ test_cases.emplace_back(new test_gated_delta_net(GGML_TYPE_F32, 32, 128, 1024, 1, 2, false, false, 8));Possibly related observations (same machine, can file separately)
test-backend-ops -o SOLVE_TRIcrashes on this HIP setup withrocBLAS error from hip error code: 'hipErrorInvalidImage'/ "device kernel image is invalid" after 6 passing cases — this also makes the unfused gated-delta-net graph path unusable on HIP, so we could not A/B fused vs. unfused. (Filed separately.)llama-mtmd-clidies silently (no error output after "llama threadpool init") whenever CPU layers participate (-ngl < 99and-ngl 0), on our HIP and Vulkan self-builds — this blocked partial-offload bisection.
Ask
Since every op passes in isolation at 1e-7 but the composed model degrades deterministically (and only on HIP), we suspect the HIP-specific composition of the GDN block (fused-graph variant selection, recurrent-state handling, or an interaction invisible to single-op tests). Happy to run any diagnostics/patches on this hardware — the repro is fully deterministic and takes ~40 s per run.
Activity
jgxgmxcom commentedon Aug 22, 2026
The SOLVE_TRI crash mentioned under 'Possibly related observations' is filed as #27557.
nabe2030 commentedon Aug 25, 2026
We did a systematic check of this on a different setup (Qwen3.8-27B, gfx1151, llama.cpp
master b10588). Plain generation, single slot: clean up to ~32k tokens on both HIP and Vulkan —
we couldn't reproduce your symptom that way. But we since found (via a tool-call-array
reproducer from a related report) that single-slot HIP does corrupt on a different kind of
load — a long prompt carrying a large tool/function array, failing consistently past ~29k
tokens even with
--parallel 1. Full detail and the mechanism we now believe explains it(a
process_ubatchinput-write race, amplified by an APU-specific code path from #24233):#27579 (comment) and
#27579 (comment).
Your report used a VLM with
mmprojand multimodal input rather than a tool array, so it may ormay not be the same defect — but if you still have a repro setup, it'd help to know: what was
--parallelset to on the server, and was the affected request the first on that server processor could a prior multi-slot session have left something in a bad state? That would help tell
whether this is the same root cause at a different trigger threshold.
DeMaulwurfn commentedon Aug 25, 2026
I think i am hitting the same thing and it looks like a known scheduler race, not a GDN kernel problem.
Same hardware class here (Strix Halo gfx1151, Framework Desktop 128 GiB, Fedora 44, rocm container). I run Qwen3.5-122B-A10B and Qwen3.6 27B, so also gated deltanet hybrids. My symptom since end of july was exactly what you describe, output stays fluent but the oldest context is gone. A needle test only finds a marker if it sits in the last ~1024 tokens before the prompt end.
On the offical b10454 docker image (commit 4df29be, one build after your b10453) the needle test fails 13 of 30 cells, and all fails are markers more then 1024 tokens away from the prompt end. Single op tests never show it, it needs the full server graph with the default n_ubatch 512 < n_batch 2048.
The cause is a write after read race on graph input tensors when the backend computes directly from host memory on integrated GPUs. There are two open fixes: PR #27311 (scheduler UMA ring buffer) and PR #25863 (avoids the host compute path). I built #27311 at commit b16cc7d362ec69da7122e0d638ffdd5db3df9bbc on the same base and the needle test goes to 30 of 30. The sanitizer included there (GGML_SCHED_SANITIZE=1) reports write-after-read conflicts on the unpatched build and 0 race(s) with the ring. Two more gfx1151 users confirmed it in the PR thread, one measured perplexity going from ~872006 to 63.59 (vulkan reference 64.10).
That would also explain why your vulkan build is correct at the identical commit, the race only exists on the HIP host buffer path.
@nabe2030 plain generation wont show it, the output stays fluent. You need a retrieval probe: put a unique marker early in the prompt and ask for it back verbatim. On my box everything more then ~1024 tokens before the prompt end is gone while the text still reads fine.
Maybe worth linking this issue to #27311.
nabe2030 commentedon Aug 26, 2026
@DeMaulwurfn — thank you, this corrects our test design and I want to be explicit rather
than bury it.
Our "single slot clean up to ~32k on both HIP and Vulkan" was measured on plain
generation, which is exactly the probe you're saying cannot see this. Fluent output was
our pass criterion, so a run that had silently dropped the head of the context scored
clean. That claim should be read as "we did not test for this", not as a non-reproduction.
What we did have that behaves like your needle test is the tool-array reproducer in
#27579: a large tool array early in the prompt, and the model asked to name its own tools.
That one does fail single-slot on HIP — 0/8 at 29,355 tokens with
--parallel 1,16/16 on Vulkan at the same commit — and the failure drops entries from the head of the
array while describing the tail correctly. Same shape as your ~1024-token recall horizon,
reached from a different angle.
Agreed on linking this to #27311, and thanks for the pointer. We're building it now and
will post results there rather than here.
One cell worth flagging to that PR, and it is this issue specifically.
@Superfish1000's measurements in #27311 give the rule
n_ubatch < n_batch ⇒ corrupt, withthe single-ubatch case as the only passing configuration, 8/8. But the exclusion table in
the original report here has
-ub 8192— whole prompt in one ubatch — still failing.That is the one data point that does not fit the rule.
The difference is that this report is a VLM run with
--mmproj, where the image encode isa separate graph execution from the text graphs. If the write-after-read also spans
consecutive
decode()calls, and not only consecutive ubatches within one, then asingle-ubatch VLM run still races and the rule holds with a wider scope. It would also
explain why the tokens lost here are the oldest image tokens in raster order, and why
flipping the image vertically heals it.
@jgxgmxcom — if you still have the setup, running your repro on #27311 with
GGML_SCHED_UMA_RING=2vs=1at-ub 8192would settle this directly, and it is a cellthe PR does not currently cover. Your repro is deterministic and takes ~40 s, so it is a
much cheaper test on your box than on ours. We can run it too if you'd rather not, but
we'd be reconstructing your setup from the report.
github-actions commentedon Oct 10, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.