Repository navigation
Eval bug: [ROCm] Severe PPL explosion starting from b10040 #27506
Description
Name and Version
./build/bin/llama-cli --version
version: 0.1.0-dev (build 10440, commit 6b4344e)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
HIP
Hardware
Ryzen AI Max+ 395 (iGPU Radeon 8060S)
Models
Llama-3.2-3B
Qwen3.6-35B-A3B-UD-IQ4_XS
Laguna-S-2.1
Problem description & steps to reproduce
Summary
When built with the ROCm backend, model output quality drastically degrades starting from commit b10040 (and remains broken on the latest master), resulting in a severe perplexity (PPL) explosion.
Details
When using the ROCm build, context comprehension significantly degrades, and tool calling frequently fails.
Using the exact same model and arguments with a Vulkan build runs without any issues (completing tool calls perfectly), indicating that this issue is specific to the ROCm implementation.
Reproduction Command
llama-perplexity -m /models/llama-3.2-3b-q4_0.gguf -f /wikitext-2-raw/wiki.test.raw -c 32768 -ngl 999 --chunks 1
Bisect / PPL Test Results Across Commits
- b10038:
PPL = 7.7241 - 5839ba3:
PPL = 7.7241 - b10040:
PPL = 3024.3587(Broken / PPL Explosion) - bb4caa7 (Latest
master/ v0.2.0):PPL = 2177.3497(Still Broken)
Environment Details
- Hardware: Ryzen AI MAX+ 395 (EVO-X2)
- OS: Ubuntu 26.04 (Host)
- Docker Base Image:
rocm/dev-ubuntu-24.04:7.2.4 - Model tested: Llama-3.2-3B-Instruct (Q4_0)
Additional Context
I initially noticed this issue while using newer models like Qwen 3.6 35B and Laguna S2.1, and I confirmed/bisected the issue using Llama-3.2-3B for faster verification.
I am not proficient in C++ to submit a direct code fix, but I have verified that checking out 5839ba3 works as expected, whereas checking out b10040 (and latest master) introduces this bug.
Note: This is my very first GitHub Issue. I can't write C++ or speak English fluently, but I verified the bisect results and actual PPL values on my local system using Gemini.
The verification was undoubtedly carried out by me.
First Bad Commit
b10040
- Last working commit: 5839ba3 (b10038)
- First broken commit: b10040
Relevant log output
Logs
# create script
cat > /app/bisect.sh <<'EOF'
#!/bin/bash
cd /app/llama.cpp
cmake -B build-bs -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache >/dev/null 2>&1 || exit 125
cmake --build build-bs --target llama-perplexity -j$(nproc) >/dev/null 2>&1 || exit 125
PPL=$(./build-bs/bin/llama-perplexity -m /opt/models/llama-3.2-3b-q4_0.gguf \
-f /opt/models/wikitext-2-raw/wiki.test.raw -c 32768 -ngl 999 --chunks 1 2>&1 \
| grep -oP 'Final estimate: PPL = \K[0-9.]+')
echo "PPL=$PPL"
[ -z "$PPL" ] && exit 125
awk -v p="$PPL" 'BEGIN { exit (p < 100) ? 0 : 1 }'
EOF
chmod +x /app/bisect.sh
# Verification begins.
/app/llama.cpp# git checkout b9850
/app/llama.cpp# /app/bisect.sh
PPL=7.7197
/app/llama.cpp# git checkout b10217
/app/llama.cpp# /app/bisect.sh
PPL=3008.9946
/app/llama.cpp# git checkout b10007
/app/llama.cpp# /app/bisect.sh
PPL=7.7241
/app/llama.cpp# git checkout b10075
/app/llama.cpp# /app/bisect.sh
PPL=3016.7824
/app/llama.cpp# git checkout b10040
/app/llama.cpp# /app/bisect.sh
PPL=3024.3587
/app/llama.cpp# git checkout b10038
/app/llama.cpp# /app/bisect.sh
PPL=7.7241
/app/llama.cpp# git checkout 5839ba352471b2a7b45e7ba401619a6896f10f8b
/app/llama.cpp# /app/bisect.sh
PPL=7.7241
/app/llama.cpp# git switch master
/app/llama.cpp# git pull
/app/llama.cpp# git log
commit bb4caa7540188872173c44d161602d9271386413 (HEAD -> master, origin/master, origin/HEAD)
Author: Georgi Gerganov <ggerganov@gmail.com>
Date: Fri Aug 21 15:01:24 2026 +0300
llama.cpp : bump version to 0.2.0 (#27498)
...
/app/llama.cpp# /app/bisect.sh
PPL=2177.3497
Activity
Rcat999 commentedon Aug 21, 2026
Here is an additional observation regarding context size (
-c).It appears that the issue only manifests when the context length is increased. When testing with
-c 512, the perplexity remains normal (PPL = 6.1261), but starting from-c 2048, the perplexity explodes:sjvrensburg commentedon Aug 22, 2026
Confirming this on a second gfx1151 machine, and I can add three data points that narrow it down: it is not quant-specific, not model-specific, and
rocWMMAis not an escape hatch.Environment
rocm-core-7.1.1-4.fc44), built withGGML_HIP_NO_VMM=ONmesa-vulkan-drivers-26.1.7-1.fc44666f8898a)Results
Same machine, same model files, same corpus (
wikitext-2-raw/wiki.test.raw),-ngl 999:build-vulkanbuild-rocmbuild-rocm-wmmaThat is 214x and 388x. Vulkan on the same machine is clean.
What this adds to the original report
GGML_HIP_ROCWMMA_FATTNbuilds are equally affected.build-rocm-wmmagives 14520.94 againstbuild-rocm's 14468.78 -- statistically indistinguishable, so rocWMMA flash attention neither causes nor avoids it.NO_VMM = 1is present insystem_infoon the affected runs, soGGML_HIP_NO_VMM=ONdoes not avoid it either.Degradation is immediate -- chunk
[1]is already 3705.7453 on the Q8_0 run against 10.5117 on Vulkan -- so this is not an accumulating drift, and it should be cheap to bisect against b10038/b10040 with a small model and a handful of chunks.Happy to run further tests on this hardware if it would help narrow the commit.
Rcat999 commentedon Aug 22, 2026
Thank you so much for the thorough verification and detailed data!
It is very encouraging to see this reproduced on another gfx1151 machine.
Root Cause Identified
I have been investigating commit
b10040further today, and I have just identified the exact location causing this behavior.In
b10040, the following check was enabled for HIP:ggml/src/ggml-cuda/ggml-cuda.cu
The fact that this branching point is related to HIP is consistent with the issue occurring in ROCm but not in Vulkan.
Workaround / Fix Verification
I confirmed that PPL stabilizes when running the latest
masterbranch with the changes from b10040 reverted.However, I can only revert the changes. The reason is that I do not understand why the code causing this issue was inserted in the first place.
nabe2030 commentedon Aug 25, 2026
Wanted to close the loop here — thanks for pinning this to b10040 and later, and especially to
@Rcat999 for independently identifying the exact same commit (c7d8722, #24233) from your
own bisection days before we got there from a different symptom (multi-slot output corruption,
not PPL). Our automated
git bisectbetween b10039 and b10588 converged on the identical commitand
#ifdefblock you found manually — good independent confirmation from two different angles.Since then we've gone further: the corruption we were chasing (and, we now think, yours too)
traces to a race in
llama_context::process_ubatch()— an H2D input-write racing a still-runninggraph compute, ungated on single-GPU setups. c7d8722's
integrated=truerestoration on HIPenables a host-buffer code path that appears to widen that race window enough to hit routinely
on APU/UMA hardware. We tested the actual sync fix (not just the
integrated=falseworkaroundyou and we both independently landed on) and it resolves the corruption too. Full writeup and
test matrix: #27579 (comment) and
#27579 (comment).
If your PPL runs batch multiple sequences/chunks per pass on HIP, that would line up with what
we're seeing — the
integrated=falserevert you verified is the same workaround we converged onindependently, and should keep working as a stopgap; the durable fix is the sync correction
described above.
Rcat999 commentedon Aug 25, 2026
@nabe2030
Thank you so much for sharing the detailed findings, and for providing a permanent fix!
I am currently unable to test the fix immediately, but I will build and verify it on my end within a few days. Once I confirm the results, I will update this thread and close the issue.
thedelph commentedon Aug 25, 2026
Independent confirmation on the same hardware, with a numeric Vulkan control.
Hardware: Ryzen AI Max+ 395, Radeon 8060S (gfx1151), running in a Proxmox
LXC container.
Model: Qwen3.6-35B-A3B, UD-Q4_K_M, 20.6 GiB.
Perplexity over the same file and the same 20 chunks, the two builds run back
to back:
build - Vulkan (build-vulkan/bin/llama-server)
PPL - 2.1254
build - ROCm/HIP
PPL- 2007.97
A factor of about 945. These are a different model and file from the ones in
the opening report, so the absolute numbers are not comparable to the 7.72 /
3024 pair — the ratio and the direction are the point.
The part that may be new: the ROCm chunk series is already four figures at
chunk 1 and stays there. It does not degrade with context, and it is not
triggered by prompt size, by concurrency, or by generation length. It is wrong
from the first token, and the visible symptom — requests returning an unbroken
wall of a single repeated character — is only where that surfaces. On our box
a small greedy-completion canary ran clean for forty minutes against this
backend while it was numerically broken throughout, so "output looks sane" is
a very weak signal here; only perplexity against a control caught it.
Consistent with the bisect in this thread: our own checkout has the re-enabled
host-buffer path at ggml/src/ggml-cuda/ggml-cuda.cu:305, and the #else
branch immediately below it records that upstream had already disabled the
same path on CUDA "due to issues with corrupted output".
The speed trade was not one. Switching to Vulkan cost 5–9% on prefill and
gained 24% on decode (49.5 → 61.5 tok/s) on this chip, so there is no
throughput reason to stay on the ROCm build while this is open.
Happy to re-run either arm with a specific model, file or flag set if that
would help move this off bug-unconfirmed.
**correction to my own caveat above — same model, ours is UD-Q4_K_M against the report's UD-IQ4_XS, so the quantisation differs but the model does not. **
nabe2030 commentedon Aug 26, 2026
@thedelph — thank you, the Vulkan-controlled numbers are what this thread needed. And
the point about the greedy canary running clean for forty minutes while the backend was
numerically broken throughout deserves more attention than it will get: we hit the same
trap from the other side, and reported HIP throughput figures (
-np 16, 42 t/s) for aday before output verification showed the generations were degenerate. tok/s and HTTP 200
are not evidence of correctness on this backend.
On your "wrong from the first token, not triggered by prompt size, by concurrency, or by
generation length" — that now has an explanation, and you're right that it isn't depth.
@Superfish1000 established the rule in #27311 (Aug 25) on the same hardware class:
corruption tracks
n_ubatch < n_batch, 8/8. Holding-b 4096and varying-ub:-c-b-uband with
ub == bit is clean at 512, 1024 and 2048 alike. The only passing configurationis the one where the whole prompt lands in a single ubatch. So it is wrong from chunk 1
because a single chunk larger than
n_ubatchis already two graph executions — nothinghas to accumulate across chunks.
That also puts @Rcat999's
-c 512clean /-c 2048broken boundary earlier in thisthread exactly on the rule, with
--chunks 1and the default-ub 512. And@sjvrensburg's runs, which used default flags, would have had
ub < bthroughout.Superfish1000 also confirmed the mechanism directly rather than by bisect: making
integratedruntime-switchable on an otherwise unmodified tree gives PPL 850,121 with iton and 72.80 with it off, from the same binary. That closes the loop on the
#ifdefblock@Rcat999 identified manually and we hit independently by bisect.
@Rcat999 — on closing this. The durable fix is #27311, which measures PPL 63.59
against a Vulkan reference of 64.10 where the base gives 872,006, at no measurable
throughput cost. #25863 fixes both symptoms equally well (63.67) and is far smaller,
if a targeted mitigation is preferred over the general scheduler change. Neither is merged
— #27311 needs two approving reviews and has four pending. Your
integrated = falserevert stays correct as a stopgap in the meantime, but I'd suggest leaving this issue open
until one of them lands: the revert removes a path that #27311 is trying to make correct
rather than delete, so "fixed by reverting" and "fixed" are different outcomes here.
One thing your report adds that the #27311 thread does not have, @thedelph: your Vulkan
switch cost 5–9% on prefill and gained 24% on decode (49.5 → 61.5 t/s). Worth stating
plainly for anyone still weighing whether to wait — on this chip there is currently no
throughput argument for staying on the ROCm build.
We're testing #27311 on our Strix Halo now against the multi-slot and tool-array failure
modes we reported, and will post results to that PR.
Rcat999 commentedon Aug 26, 2026
@nabe2030 Thank you so much for the thorough summary!
I especially want to thank you for letting me know when it’s appropriate to close this issue. As I'm still new to this, I was almost about to close it prematurely!
Also, thank you for pointing out the specific PRs (#27311 and #25863) to follow. To be honest, I don't fully grasp all the technical code details myself, but I do adjust batch parameters in my day-to-day use, so learning about the
n_ubatch < n_batchbehavior was super helpful.I’ll keep this issue open and follow PR #27311 until it gets merged. Thanks again to everyone for the great help!
IMbackK commentedon Oct 5, 2026
This is currently worked around in the code and there is no consensus on how to structure a change to the Scheduler so the workaround will have to suffice for now.