Repository navigation
Eval bug: HIP/ROCm on gfx1151, wrong logits (not a crash), triggered by prompts longer than n_ubatch. #28211
Description
Name and Version
version: 0.3.0-dev (build 10729, commit 458681e)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
HIP
Hardware
CPU AMD Ryzen AI MAX+ 395 w/ Radeon 8060S (Strix Halo APU)
GPU Radeon 8060S Graphics, gfx1151 (RDNA 3.5, integrated)
VRAM 98304 MiB carved out to the GPU
ROCm 7.2.1
HIP 7.2.53211-e1a6bc5663
compiler AMD clang version 22.0.0git (roc-7.2.1 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e)
kernel Linux 7.0.0-30-generic x86_64
distro Ubuntu 24.04.4 LTS
build:
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build
HIP compile flags actually used:
-O3 -DNDEBUG -std=gnu++17 --offload-arch=gfx1151 -fPIC
Vulkan build used for comparison (same machine, same GPU):
cmake -B build-vk -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
Models
https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF (Qwen3.5-0.8B-Q4_K_M.gguf, 508 MB)
https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf (gemma-4-E2B_q4_0-it.gguf, 3.3 GB)
Also observed, same machine, same HIP build:
- gemma-4-E2B-it BF16, Q8_0, Q4_K_M and Q4_0, all four from
https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF - gemma-4-31B-it Q4_K_M (a local file; its size does not match the copy
currently published by unsloth, so I am not citing a URL for it)
The BF16 case is the useful one for narrowing: BF16 involves no
dequantisation, so this is not a quantised-matmul kernel. Perplexity from
llama-perplexity on WikiText-2, same 2-item set, default n_ubatch, HIP
against Vulkan:
BF16 84.7476 vs 13.8167
Q8_0 110.2795 vs 14.4350
Q4_K_M 92.7499 vs 12.9609
Q4_0 112.7851 vs 12.1284
The 31B case is how this was first noticed: it read ppl 93.86 on HIP against
7.40 on Vulkan, which looks like a weak model rather than a broken backend.
Problem description & steps to reproduce
Any prompt longer than n_ubatch produces badly wrong logits. No error, no
warning, no crash: the numbers look plausible. Default n_ubatch is 512, so in
practice almost every real prompt is affected.
repro.cpp
// Minimal reproducer, public llama.h only.
//
// Decodes one prompt twice with two contexts that differ in exactly one
// parameter: n_ubatch. Everything else is identical, including the token ids.
// A correct backend must return the same distribution both times.
//
// ./download.sh
// ./repro /tmp/Qwen3.5-0.8B-Q4_K_M.gguf 99 qwen35 architecture
// ./repro /tmp/gemma-4-E2B_q4_0-it.gguf 99 gemma4 architecture
//
// Pass 0 as the second argument to run on CPU.
//
// Compares the logits at every position and prints how often the two runs
// disagree about the most likely next token, the mean and worst KL divergence
// per position, and the largest absolute logit difference.
#include "llama.h"
#include <cmath>
#include <cstdio>
#include <cstring>
#include <algorithm>
#include <string>
#include <vector>
static std::vector<llama_token> make_prompt(const llama_vocab * vocab,
int want) {
std::string text = "The quick brown fox jumps over the lazy dog. ";
while ((int) text.size() < want * 6) {
text += "Numbers and words follow in a long unremarkable sentence. ";
}
std::vector<llama_token> out(want * 4);
const int n = llama_tokenize(vocab, text.c_str(), (int32_t) text.size(),
out.data(), (int32_t) out.size(), true, false);
out.resize(n > want ? want : (n < 0 ? 0 : n));
return out;
}
static std::vector<float> run(llama_model * model,
const std::vector<llama_token> & toks,
uint32_t n_batch, uint32_t n_ubatch, int ngl) {
llama_context_params cp = llama_context_default_params();
cp.n_ctx = 4096;
cp.n_batch = n_batch;
cp.n_ubatch = n_ubatch;
cp.n_threads = 8;
(void) ngl;
llama_context * ctx = llama_init_from_model(model, cp);
std::vector<float> last;
if (ctx != nullptr) {
const int n = (int) toks.size();
llama_batch batch = llama_batch_init(n, 0, 1);
batch.n_tokens = n;
for (int i = 0; i < n; ++i) {
batch.token [i] = toks[i];
batch.pos [i] = i;
batch.n_seq_id[i] = 1;
batch.seq_id [i][0] = 0;
batch.logits [i] = 1;
}
const int rc = llama_decode(ctx, batch);
if (rc == 0) {
const llama_vocab * v = llama_model_get_vocab(model);
const int n_vocab = llama_vocab_n_tokens(v);
last.resize((size_t) n * (size_t) n_vocab);
for (int i = 0; i < n; ++i) {
const float * lg = llama_get_logits_ith(ctx, i);
if (lg != nullptr) {
std::copy(lg, lg + n_vocab,
last.begin() + (size_t) i * n_vocab);
}
}
} else {
fprintf(stderr, "llama_decode failed: %d\n", rc);
}
llama_batch_free(batch);
llama_free(ctx);
}
return last;
}
static double kl_div(const std::vector<float> & p, const std::vector<float> & q) {
double mp = p[0];
double mq = q[0];
for (size_t i = 1; i < p.size(); ++i) {
if (p[i] > mp) { mp = p[i]; }
if (q[i] > mq) { mq = q[i]; }
}
double sp = 0.0;
double sq = 0.0;
for (size_t i = 0; i < p.size(); ++i) {
sp += std::exp((double) p[i] - mp);
sq += std::exp((double) q[i] - mq);
}
const double lp = mp + std::log(sp);
const double lq = mq + std::log(sq);
double kl = 0.0;
for (size_t i = 0; i < p.size(); ++i) {
const double a = (double) p[i] - lp;
const double b = (double) q[i] - lq;
kl += std::exp(a) * (a - b);
}
return kl;
}
static int argmax(const std::vector<float> & v) {
int best = 0;
for (size_t i = 1; i < v.size(); ++i) {
if (v[i] > v[best]) { best = (int) i; }
}
return best;
}
int main(int argc, char ** argv) {
int status = 1;
if (argc < 2) {
fprintf(stderr, "usage: %s <model.gguf> [n_gpu_layers]\n", argv[0]);
} else {
const int ngl = argc > 2 ? atoi(argv[2]) : 99;
llama_backend_init();
llama_model_params mp = llama_model_default_params();
mp.n_gpu_layers = ngl;
llama_model * model = llama_model_load_from_file(argv[1], mp);
if (model == nullptr) {
fprintf(stderr, "failed to load %s\n", argv[1]);
} else {
const llama_vocab * vocab = llama_model_get_vocab(model);
const std::vector<llama_token> toks = make_prompt(vocab, 768);
printf("model %s\n", argv[1]);
printf("n_gpu_layers %d\n", ngl);
printf("prompt %zu tokens\n", toks.size());
const std::vector<float> a = run(model, toks, 2048, 2048, ngl);
const std::vector<float> b = run(model, toks, 2048, 512, ngl);
if (a.size() == b.size() && !a.empty()) {
const int nv = llama_vocab_n_tokens(vocab);
const int np = (int) (a.size() / (size_t) nv);
double worst = 0.0;
double kl = 0.0;
double klmax = 0.0;
int diff = 0;
for (int t = 0; t < np; ++t) {
const std::vector<float> pa(a.begin() + (size_t) t * nv,
a.begin() + (size_t) (t + 1) * nv);
const std::vector<float> pb(b.begin() + (size_t) t * nv,
b.begin() + (size_t) (t + 1) * nv);
for (int i = 0; i < nv; ++i) {
const double d = std::fabs((double) pa[i] - (double) pb[i]);
if (d > worst) { worst = d; }
}
const double k = kl_div(pa, pb);
kl += k;
if (k > klmax) { klmax = k; }
if (argmax(pa) != argmax(pb)) { diff++; }
}
kl /= (double) np;
const int ia = 0;
const int ib = 0;
(void) ia; (void) ib;
printf("positions compared %d\n", np);
printf("argmax differs at %d of %d positions (%.2f%%)\n",
diff, np, 100.0 * diff / np);
printf("mean KL per position %.8f\n", kl);
printf("worst KL at a position %.8f\n", klmax);
printf("\nn_ubatch 2048 (prompt fits, no split)\n");
printf("n_ubatch 512 (prompt is split into 2 micro-batches)\n");
printf("max |logit difference| %.6f\n", worst);
printf("\n%s\n", kl > 0.01
? "FAIL: n_ubatch changed the distribution."
: "ok: n_ubatch left the distribution intact.");
status = kl > 0.01 ? 2 : 0;
} else {
fprintf(stderr, "decode produced no logits\n");
}
llama_model_free(model);
}
llama_backend_free();
}
return status;
}CMakeLists.txt
cmake_minimum_required(VERSION 3.14)
project(repro CXX)
set(LLAMA_BUILD_COMMON OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_TESTS OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_TOOLS OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_SERVER OFF CACHE BOOL "" FORCE)
add_subdirectory(llama.cpp)
add_executable(repro repro.cpp)
target_link_libraries(repro PRIVATE llama)
target_compile_features(repro PRIVATE cxx_std_17)curl -L -C - -o /tmp/Qwen3.5-0.8B-Q4_K_M.gguf \
https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/resolve/main/Qwen3.5-0.8B-Q4_K_M.gguf
curl -L -C - -o /tmp/gemma-4-E2B_q4_0-it.gguf \
https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf/resolve/main/gemma-4-E2B_q4_0-it.gguf
ls -l /tmp/Qwen3.5-0.8B-Q4_K_M.gguf /tmp/gemma-4-E2B_q4_0-it.gguf
./repro /tmp/Qwen3.5-0.8B-Q4_K_M.gguf 99 qwen35 architecture
./repro /tmp/gemma-4-E2B_q4_0-it.gguf 99 gemma4 architectureTwo models of different architectures, 768-token prompt, n_ubatch 2048
against 512.
/tmp/Qwen3.5-0.8B-Q4_K_M.gguf (qwen35), from
https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF
backend argmax differs mean KL worst KL max|dlogit|
HIP (upstream) 15/768 1.95% 0.07477930 7.8480584 14.109375
Vulkan 0/768 0.00% 0.00000187 0.0001609 0.304688
CPU 0/768 0.00% 0.00000000 0.0000000 0.000000
HIP + one-line fix 0/768 0.00% 0.00048698 0.0728900 1.828125
/tmp/gemma-4-E2B_q4_0-it.gguf (gemma4), from
https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf
backend argmax differs mean KL worst KL max|dlogit|
HIP (upstream) 170/768 22.14% 1.17912350 18.3745292 40.190187
Vulkan 0/768 0.00% 0.00000318 0.0004254 0.877997
HIP + one-line fix 1/768 0.13% 0.00008664 0.0066724 2.086188
Both architectures are affected and both are fixed by the same one-line
change. gemma-4 is the worse of the two: upstream HIP picks a different most
likely token at 22 percent of positions. CPU and Vulkan are bit-exact or near
it across the split.
This is not specific to how any one program calls the API. In
src/llama-context.cpp:
cparams.n_ubatch = std::min(cparams.n_batch, params.n_ubatch == 0 ? params.n_batch : params.n_ubatch);n_ubatch defaults to 512. llama_kv_cache::init_batch then splits the batch
by n_ubatch and llama_context::decode loops over process_ubatch. That is
the ordinary prompt-processing path, so any caller submitting more than
n_ubatch tokens in one llama_decode is affected, including the shipped
tools and the server.
Severity scales with the number of splits
Perplexity over one sequence, one llama_decode, a ~350-token prompt of
gemma-4-E2B BF16, changing only n_ubatch. Measured with a small in-house
scorer rather than an upstream tool, so treat it as illustrating the shape of
the failure rather than as the primary evidence:
-ub 64 -> 792.9211 (6 micro-batches)
-ub 128 -> 80446.7741 (3 micro-batches)
-ub 256 -> 1614.8257 (2 micro-batches)
-ub 384 -> 10.0350 (no split) CORRECT
-ub 512 -> 10.0350 (no split) CORRECT
The corruption is not a fixed offset. It compounds with each additional
micro-batch.
Not caught by test-backend-ops
test-backend-ops passes 14564 of 14565, because it does not exercise the
host-buffer placement path. The one failure is unrelated and much too small to
explain the above, but is a genuine failure on this target:
TOPK_MOE(ne=[31,22,1,1],n_expert_used=8,with_norm=1,bias_probs=0,
gating_func=0,scale_w=0.000000)
[VIEW] ERR = 0.000382444 > 0.000000100 FAIL
What it is not
Each tested and ruled out:
- Not a quantisation kernel. BF16 is affected and involves no dequantisation.
Q8_0, Q4_K_M and Q4_0 are affected equally. - Not flash attention. Broken with
-fa on,offandauto. - Not HIP graphs.
GGML_CUDA_DISABLE_GRAPHS=1changes nothing. - Not the matmul path.
GGML_CUDA_FORCE_MMQ=1andGGML_CUDA_FORCE_CUBLAS=1
change the wrong value without fixing it. - Not KV offload alone.
-nkvochanges the wrong value without fixing it. - Not model or architecture specific. Reproduced on gemma-4 E2B and on
Qwen3.5-0.8B, which has no sliding-window attention. - Not non-determinism. Every configuration reproduces bit-identically, so this
is incorrect data rather than a race. - Not new at HEAD. Built and reproduced at a94d563 (2026-08-13), the point
AMD's fork branched from.
Related issues, and why this is not a duplicate of them
GitHub suggests three. None describe this failure, which is silent numerical
corruption on a single GPU with no crash.
-
Misc. bug: ggml-cuda: restore prop.integrated for HIP builds; #16308 hardcode breaks iGPU classification and supports_buft for AMD APUs #23977 is the origin, not a duplicate. Its fix, PR ggml-cuda : restore prop.integrated on HIP builds #24233, is what
introduced this. See the First Bad Commit section below. -
Misc. bug: Incapable IGP crashes llama.cpp when capable GPU is present #23152, incapable IGP crashes llama.cpp when a capable GPU is present.
A crash, on Windows, with a discrete GPU also present. Here there is one GPU,
nothing crashes, and the run completes with wrong numbers. -
Misc. bug: gfx1152 broken unless I manually export HSA_OVERRIDE_GFX_VERSION=11.5.0 #19949, gfx1152 broken unless
HSA_OVERRIDE_GFX_VERSION=11.5.0is set.
A segfault on a different chip caused by runtime chip recognition. Setting
that override on gfx1151 does not help and aborts instead, since it
contradicts the compiled--offload-arch. Unrelated. -
Eval bug: Broken/no Gemma 3n output on CUDA (Nvidia Jetson Orin Nano) #15034 is the same class of corruption on the CUDA side and is why
integratedis disabled there. It is the closest relative of this report.
First Bad Commit
This might be a regression with a traceable history, and the naive fix may undo
someone else's bug fix.
- Eval bug: Broken/no Gemma 3n output on CUDA (Nvidia Jetson Orin Nano) #15034 (2025-08): Gemma 3n produced garbled output on a Jetson Orin Nano, an
integrated GPU. Fixed by PR cuda : Disable host buffers on integrated GPUs (#15034) #16308, which hardcodedintegrated = false. - Misc. bug: ggml-cuda: restore prop.integrated for HIP builds; #16308 hardcode breaks iGPU classification and supports_buft for AMD APUs #23977 (2026-06): that hardcode broke iGPU classification on AMD APUs, where
a machine with both an iGPU and a dGPU had work scheduled onto both as if
both were discrete. The reporter traced it to
ggml_backend_cuda_device_supports_buftreturning the wrong value and
proposed restoringprop.integratedfor HIP. - PR ggml-cuda : restore prop.integrated on HIP builds #24233 (2026-07-16) implemented that and is the commit that introduced the
#if defined(GGML_USE_HIP)branch now in the tree.
Possible Root cause
ggml/src/ggml-cuda/ggml-cuda.cu, device init:
#if defined(GGML_USE_HIP)
info.devices[id].integrated = prop.integrated;
#else
info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
#endifintegrated is disabled for CUDA because it produced corrupted output
(#15034), and re-enabled for HIP by PR #24233 to fix iGPU classification
(#23977). gfx1151 (Strix Halo) is an integrated UMA part, so HIP reports
integrated = true and takes the path CUDA disabled for the corruption
reason. See the history bullets above: both branches exist for good reasons
and the flag is being asked to mean two different things.
That flag then lets ggml_backend_cuda_device_supports_buft accept pinned host
buffers as GPU-usable:
static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;
const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated;
return (ggml_backend_buft_is_cuda(buft) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft));
}So deleting the HIP branch would restore correctness here and re-break
#23977. That is oneline-fix.patch, and it is included only to demonstrate
that this flag is the cause.
--- a/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/ggml/src/ggml-cuda/ggml-cuda.cu
@@ -305,11 +305,7 @@
-#if defined(GGML_USE_HIP)
- info.devices[id].integrated = prop.integrated;
-#else
info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
-#endifIt is not the recommended fix.
Proposed fix
Keep integrated for device classification, which is what #23977 needs, and
stop treating host buffers as GPU-usable, which is what causes the corruption.
Those are two separate concerns sharing one flag.
AMD's fork does exactly this, in commit 865374b on branch gfx11, and it is
amd-fix.patch here:
static bool ggml_cuda_is_gfx1151(const int device) {
return ggml_cuda_info().devices[device].cc == GGML_CUDA_CC_RDNA3_5 + 1;
}
static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;
const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated;
const bool direct_host_access = integrated && !ggml_cuda_is_gfx1151(dev_ctx->device);
return (ggml_backend_buft_is_cuda(buft) && buft->device == dev) ||
(direct_host_access && ggml_backend_buft_is_cuda_host(buft));
}Whether the exclusion should be gfx1151 only, or all HIP integrated targets, or
whether the underlying coherency problem can be fixed instead, is a question
for people who know the driver. We can only report that gfx1151 is affected and
that this shape of fix resolves it.
Relevant log output
Logs
# corpus: standard WikiText-2 test split
# curl -sLO https://huggingface.co/datasets/ggml-org/ci/resolve/main/wikitext-2-raw-v1.zip
# unzip wikitext-2-raw-v1.zip
#
# Only -ub changes between the two runs of each pair. Everything else is identical.
### HIP build, Qwen3.5-0.8B-Q4_K_M
$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf -f wiki.test.raw \
-c 2048 -b 2048 -ub 512 --chunks 8 -ngl 99 -fa off
Final estimate: PPL = 1565.9828 +/- 87.11104
$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf -f wiki.test.raw \
-c 2048 -b 2048 -ub 2048 --chunks 8 -ngl 99 -fa off
Final estimate: PPL = 13.7743 +/- 0.45458
### HIP build, gemma-4-E2B q4_0
$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf -f wiki.test.raw \
-c 2048 -b 2048 -ub 512 --chunks 8 -ngl 99 -fa off
perplexity: calculating perplexity over 8 chunks, n_ctx=2048, batch_size=2048, n_seq=1
[1]6030.1563,[2]7605.1239,[3]8028.6236,[4]8303.2742,[5]7819.0489,[6]6628.9725,[7]6531.4533,[8]6473.3124,
Final estimate: PPL = 6473.3124 +/- 382.43584
$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf -f wiki.test.raw \
-c 2048 -b 2048 -ub 2048 --chunks 8 -ngl 99 -fa off
Final estimate: PPL = 37.1891 +/- 1.62087
### Vulkan build, same machine, same GPU, same files
$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf ... -ub 512
Final estimate: PPL = 13.7938 +/- 0.45571
$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf ... -ub 2048
Final estimate: PPL = 13.7891 +/- 0.45529
$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf ... -ub 512
Final estimate: PPL = 37.0287 +/- 1.61418
$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf ... -ub 2048
Final estimate: PPL = 37.0170 +/- 1.61392
### Summary
-ub 512 -ub 2048 ratio
HIP Qwen 1565.9828 13.7743 113.7x
HIP gemma 6473.3124 37.1891 174.1x
Vulkan Qwen 13.7938 13.7891 1.00x
Vulkan gemma 37.0287 37.0170 1.00x
Vulkan is flat. HIP is not. -ub is the only variable.
Activity
Error00101 commentedon Sep 3, 2026
Have the same gfx1151 (Strix Halo - RDNA3.5)
llama.cpp is compiled from source as a fat binary with all backends included (currently on: 7798007)
Devices
./llama-perplexity_fixed --list-devicesAvailable devices:
ROCm0: AMD Radeon 8060S Graphics (98304 MiB, 33936 MiB free)
Vulkan0: AMD Radeon 8060S Graphics (RADV STRIX_HALO) (114688 MiB, 19584 MiB free)
BLAS: OpenBLAS (0 MiB, 0 MiB free)
I did some slight testing and perplexity explodes when using the unfixed ROCm version.
The fix (865374b) was implemented by me and it works.
All models are run with the default params the only thing that changes is the backend
Summary:
I normaly run DeepseekV4-0731 with ROCm. I observed substantialy improved answers from it (far less rambly and more precise with its answers) but i have no benchmark for that. If needed i will provide a benchmark for that as well.
If other tests are needed i am happy to help.
Edit: I found the RPC option (Sorry!)
sedgewickmm18 commentedon Sep 6, 2026
@Error00101
I applied your patch to commit 7620399 (from 09/06), ran a build based ROCm 10.0 on a Strix Halo and tested the build with an MCP tool scenario (context > 50k token) with unsloth/Qwen3-Coder-Next-GGUF:UD-Q6_K_XL
Without the fix I always ended up with a garbage response, with the fix I get stable and sane results - as with llama-rocm build b1291 from lemonade-sdk, the last known good build.
fix(llamacpp): patch gfx1151 host-access bug, ROCm is correct again (#…
Error00101 commentedon Sep 7, 2026
Sorry for my poor choice of words! I did not create the patch.
I simply pulled the patch and applied it to my local repository and did some testing.
noamsto commentedon Sep 7, 2026
Confirming the fix works, and reporting one gap:
865374bbmisses gfx1150.The check is
cc == GGML_CUDA_CC_RDNA3_5 + 1, i.e. gfx1151 only. gfx1150 sits atGGML_CUDA_CC_RDNA3_5exactly, so it falls straight through — and it is equally affected.gfx1150 (Strix Point) is broken the same way
Ryzen AI 9 HX PRO 370 / Radeon 890M, gfx1150, kernel 7.2.2, ROCm 7.2.3, llama.cpp b10566. Same corpus and flags throughout (
-c 512 --chunks 6 --seed 42 -t 8):Same signature as gfx1151: deterministic (a repeat gave byte-identical
250,459.9596),-ngl 0reproduces the CPU figure exactly, corruption already present at partial offload (-ngl 16), and Vulkan correct on the same box. Two unrelated model families, so it is not a MoE or Gemma quirk.For reference, the gfx1151 side on this hardware (Ryzen AI MAX+ 395), Qwen3.5-4B: CPU 6.8056, Vulkan 6.8067, ROCm 1334.00.
Widening the predicate fixes it, with no regression on gfx1151
Built at master
465e49b9(b10830) withCMAKE_HIP_ARCHITECTURES=gfx1150;gfx1151, Qwen3.5-4B on both chips:-nglIdentical at every offload depth and identical across both chips, against a CPU reference of 6.79.
GGML_CUDA_CC_IS_RDNA3_5already exists incommon.cuhand covers gfx1150/1151/1152/1153.Two smaller notes
test-backend-opsdoes not catch this. It passes 100% on gfx1151, including 1660MUL_MATcases — but q6_K is only exercised atn= 1–9 and 64, and maxmacross all types is 6273, while the tensor that triggers the worst corruption here istoken_embd.weightat2560 x 248320. Worth knowing given how often this bug class resurfaces on RDNA3.5 and gets fixed one model architecture at a time.Independent of packaging or ROCm version. Reproduced identically with AMD's own prebuilt (
lemonade-sdk/llamacpp-rocmb1325, bundled ROCm 10.1.0a20260822, llama.cpp3ad1b): 1334.57 on gfx1151. Also worth noting b1291 is cited earlier in this thread as the last known-good build, which matches — b1325 reproduces.Happy to open a PR for the one-line widening if that is useful.
fix(llamacpp): widen the host-access patch from gfx1151 to all RDNA3.5
fix(llamacpp): widen the host-access patch from gfx1151 to all RDNA3.5 (
Error00101 commentedon Sep 8, 2026
It seems like upstream is fixed:
d4389a4
They unconidtionaly disabled the Hostbuffer. Some comentary about it can be found here. :
#27311 (comment)
Will close my PR as the fix seems to be to disable hostbuffers entirely.
noamsto commentedon Sep 8, 2026
Thanks — the revert covers gfx1150 too, so the per-arch widening isn't needed. Switching to carrying
d4389a4duntil nixpkgs catches up.huppiflupp commentedon Sep 14, 2026
Another gfx1151 data point, which may help narrow this. Not a claim that the issue is fixed —
my configuration differs from yours in three ways that could matter, listed below.
What I see
Same machine, same model, same corpus,
-c 2048 --chunks 6(so chunk length > the 512 defaultn_ubatch, matching your trigger condition):9731ad3(2026-08-18)bfdc321(2026-09-14)bfdc321(2026-09-14)So the defect is clearly present on this hardware at
9731ad3, and I cannot reproduce it attoday's master.
HIP_LAUNCH_BLOCKING=1on master changes nothing (7.0098 either way), which isconsistent with whatever was wrong no longer being live on my path.
Worth knowing for anyone bisecting: throughput does not move
9731ad3bfdc321The broken build is exactly as fast as the correct one.
llama-benchcannot see this defect atall — it loads, runs, emits tokens at full speed and reports no error. I had already published a
benchmark comparison based on the August build before I thought to check perplexity, and had to
retract the numbers.
How my setup differs from yours — any of these could be the discriminator
mem_info_vram_total~1 GiB)The memory difference is the one I would look at first. On Strix Halo a large carved-out VRAM
block and a GTT-backed pool are genuinely different allocation paths, and #28113 (MoE garbage on
gfx1151) plus #28537 (batch-shared logit corruption) were both filed in the same window as this
one, which makes me suspect a family of issues rather than one bug.
If it would help, I can re-run on this box with a dense model closer to yours, or with a carved
VRAM block instead of GTT, to isolate which of those four rows matters. Hardware is a Ryzen AI
MAX+ 395 / Radeon 8060S, 128 GB.
huppiflupp commentedon Sep 18, 2026
Follow-up to my 09-14 numbers: master
ec92815HIP on gfx1151 gives PPL 5.7835 (MoE) and 5.7176 (dense), matching Vulkan (5.7879 / 5.7401). With #28604 this looks fixed here; fine to close from my side.jimw567 commentedon Sep 23, 2026
ok. The issue is caused by PR #24233, which was reverted by https://github.com/ggml-org/llama.cpp/pull/28604/changes . With that revert, the output correctness is retored.
Another open PR #27311 that may cause some issue in this area again. I will keep a close eye on that one
@leok7v Is it ok to close the issue as it stands?
leok7v commentedon Sep 23, 2026
It will take me couple of days to verify but if you are confident you can close it w/o waiting, we can always reopen it later. Would be good to have a test for it too. Should be simple to test and patch earlier.
jimw567 commentedon Sep 24, 2026
Take your time. I will only close once you confirm. Thanks
tczank commentedon Sep 25, 2026
Independent verification while you finish yours — with a twist: on my machine the corruption reproduces on an integrated target that is not gfx1151, and a one-line A/B pins it causally on the reverted branch.
Setup: AMD Ryzen 7 7800X3D iGPU (gfx1036,
hipGetDevicePropertiesreportsintegrated=1) + RX 7900 XTX (gfx1100,integrated=0), Arch Linux, kernel 7.2.6, ROCm 7.2.4 / HIP 7.2.53211, llama.cpp 9731ad3 (build 10497, pre-revert) and a02c7f5 (build 106, current master), both-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100;gfx1036 -DCMAKE_BUILD_TYPE=Release. Model Qwen3.5-0.8B-Q4_K_M,wiki.test.raw,-c 2048 --chunks 6(huppiflupp's protocol), plus the issue'srepro.cppcompiled unmodified against each tree.Perplexity:
-ub 512(split)-ub 2048d4389a4ddrevert applied on topSame binary, same config: the dGPU is identical to CPU at displayed precision (
integrated=0, host path never taken), while the iGPU corrupts only when the prompt is split. Applying only the single line ofd4389a4dd(integrated = prop.integrated→false) to9731ad3flips 1804.77 → 15.7781. (Master dGPU == master CPU as well: 15.6994 / 15.7766.)repro.cpp: pre-revert iGPU FAIL — 16/768 argmax differ, mean KL 0.077 and 0.134 across two runs, worst KL 7.6503 (both runs). Same pre-revert binary on the dGPU: PASS (mean KL 1.6e-4); master dGPU and CPU: PASS (3.5e-5).
Two points that may matter for the issue:
Not gfx1151-specific. gfx1036 (RDNA2 desktop iGPU) corrupts at least as hard as the reported boxes (ppl ~1805 here vs 727 at huppiflupp's
9731ad3and 84.7 at leok7v's). The blanket revert was the right shape; a per-arch carve-out that keeps direct host access outside gfx115x (the fork's approach as quoted in fix: gfx1151, wrong logits #28574) would leave gfx1036 broken. For Scheduler UMA ring buffer (+ sanitizer and fixes) #27311: whatever restores host buffers needs to cover gfx1036 too, not just RDNA3.5.Caution when re-running the repro on master with an integrated target: on gfx1036 it does not return to pass state at master — mean KL 0.176–0.183 across three invocations (threshold 0.01), varying run-to-run, while perplexity stays clean (15.76 / 15.73). Per-position diagnostics show the divergence sits entirely in microbatch-2+ positions (first at position 542, worst KL 10–21 at the final positions 760–767, max |logit difference| ~20; mid-prompt positions are argmax-identical). It also occurs with the revert applied and does not occur on the dGPU — i.e. a batch-boundary / run-to-run numerics effect specific to this iGPU, separate from the fixed host-buffer corruption. So on such boxes: a repro FAIL at master alone does not mean the bug is still live — check perplexity.
On the test ask — two candidate levels:
ggml_backend_cuda_device_supports_buftdoes not acceptcuda_hostunless host placement is explicitly enabled; would have failed on any integrated box during the regression window and needs no special hardware;Repro of the table's first row: