HIP/ROCm backend produces corrupted output on gfx1151: two dense architectures fail, Vulkan is correct with byte-identical flags

@30

Summary

On a gfx1151 (Strix Halo APU, RDNA3.5) system with ROCm 7.2.4, the HIP/ROCm backend
produces corrupted output where the Vulkan backend, running the same weights, the same
build, and byte-identical flags on the same machine, is correct on every attempt.

Measured on three architectures. Two fail on HIP, with different severity; one is
unaffected:

model	`general.architecture`	topology	Vulkan	HIP/ROCm
Llama-3.1-8B-Instruct `Q4_K_M`	`llama`	dense	16/16 @30,516 · 6/6 @1,616	0/16 @30,516 · 0/6 @1,616
Qwen3.8-27B `UD-Q8_K_XL`	`qwen35`	dense	16/16 @29,355	0/16 @29,355 · 6/6 @1,349
Ling-3.0-flash `IQ4_XS`	`bailingmoe3`	512-expert MoE	16/16 @29,275	16/16 @29,275

(@N = prompt tokens. Each cell is 8 requests run twice; the repeat is deliberate, because a
single 8/8 is not enough to call a backend clean.)

Backend and binary path were read from /proc/<pid>/exe on every run, not inferred from
the launcher — an earlier "Vulkan" result in this investigation turned out to be the HIP server
still answering after a boot script silently failed.

Two severities, deliberately not merged

llama is broken outright on HIP — at every depth tested, including a 1,616-token
prompt. Output is degenerate word salad:
'for 4\n<|contextual background\nline keep track\nTheater in line while mind the material and context in'
The same model, flags and template on Vulkan is 6/6 shallow and 16/16 at 30,516 tokens, so
this is not a configuration error.
qwen35 is correct at shallow depth and fails past roughly 29k tokens when the request
also carries a moderately large tool array. Output stays fluent and well-formed; the model
confidently mis-states its own tool list. With an array of read_file, tool_00 …
tool_15:

I don't have a read_file tool available in my current toolset. The tools I have access to
are tool_01 through tool_15, which are placeholder utilities that accept a single string
argument. None of them are file-reading tools.

It has read the placeholders' descriptions correctly while losing the entries at the head
of the array. Other samples invented tool names absent from the request entirely, or ran to
the token limit emitting finish_reason: "length" with empty content and no tool call.

Whether these are one defect with severity varying by model, or two, is not established here.

Hypothesis, on a single data point — please treat as a lead, not a finding

The two failing models are dense; the one that passes is a 512-expert MoE. Dense and
MoE matmuls take different paths (mul_mat vs mul_mat_id), which may be where to look. This
rests on one MoE observation and no second MoE was tested. It is offered as a starting
point, not a claim.

Still present on current master

Reproduced on stock upstream b21e4de74567f5eef213765c9476a843c2e43f0d — master
re-fetched immediately before building — from a clean detached worktree with no local patches,
built for gfx1151 with ROCm 7.2.4.

The same result holds on 3a653fea932e (12 days earlier). The two SHAs agreeing is
expected rather than corroborating: there are 7 commits between them and none touch
ggml/src/ggml-cuda or ggml/src/ggml-hip, so it simply means nothing has been attempted on
this code path in that window. It should not be read as an independent second confirmation.

It is not `e79e4bf66` — please do not close this as a duplicate of that commit

e79e4bf66 ("ggml-hip : remove -funsafe-math-optimizations", #26696) is present in every
tested tree and does not fix this. It is the obvious suspect and it matched on every checkable
detail:

its message describes -fassociative-math reassociating FP reductions and "can flip greedy
argmax on RDNA3.5 (e.g. MTP speculative decode diverging from the non-speculative
baseline)";
the hardware here is gfx1151, which is RDNA3.5;
one of the failing workloads runs --spec-type draft-mtp;
an older tree used here still carries the flag at ggml/src/ggml-hip/CMakeLists.txt:130,
while every tree tested above does not.

With that flag already removed, the defect reproduces at 0/16. This is either a second,
independent problem or the same one with another cause.

Environment


Build	stock `b21e4de74567f`; also reproduced on `3a653fea932e`
GPU	gfx1151 (Strix Halo APU), 128 GB unified memory
ROCm	7.2.4 (hipcc 7.2.53211-97f5574fe2, AMD clang 22.0.0git roc-7.2.4)
OS	Linux
Build flags	`-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release -DCMAKE_HIP_COMPILER=<rocm>/lib/llvm/bin/clang++ -DLLAMA_CURL=OFF -DGGML_NATIVE=ON`

Server flags — identical on both backends, only the binary differs

-ngl 999 --flash-attn on
--ctx-size 65536 --ubatch-size 1024 --batch-size 2048
--parallel 1 --threads 16
--cache-type-k f16 --cache-type-v f16
--cache-prompt --cache-reuse 256 --cache-ram 0
--ctx-checkpoints 32 --checkpoint-min-step 8192
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--jinja        # each model uses its own embedded chat template

Verified byte-identical with diff between the two launcher files; the only differences are a
comment header, the slot-save path, and the binary path.

Causes eliminated, each by changing ONE variable

Listed so nobody spends replies asking. Every row was measured; all remained failing:

varied	result
Context depth	not the cause — 1 tool at 36,335 tokens is 5/5 correct, longer than the failing case
Tool count	not the cause — 17 tools at 1,349 tokens is 6/6 correct
Total prompt length	not the cause — see the 36,335-token row above
KV cache type	`q4_0`, `q8_0`, `f16` all fail
Flash attention	`--flash-attn off` still fails; it changes the failure mode, not the failure. (Note `--flash-attn off` is incompatible with a quantised V cache: "quantized V cache requires flash_attn to be enabled".)
Speculative decoding / MTP	removing `--spec-type draft-mtp` (a one-line change) still fails
Weight quantisation	`UD-Q8_K_XL` → `UD-Q4_K_XL` still fails
Multimodal projector	removing `--mmproj` still fails
Seats / context size	`262144/1`, `1048576/4`, `1048576/1` are statistically identical
Our out-of-tree patches	ruled out — both legs of each pair are the same stock tree
`e79e4bf66`	present in every tested tree; does not fix it

Reproducer

Python 3 standard library only. Generates its own tool array and filler, so the request is fully
described by (n_tools, lines).

llama-server -m <model>.gguf --jinja \
  -ngl 999 --flash-attn on --ctx-size 65536 --parallel 1 \
  --cache-type-k f16 --cache-type-v f16 --port 8080

python3 repro_hip_toolcall.py --base http://127.0.0.1:8080          # failing cell
python3 repro_hip_toolcall.py --base http://127.0.0.1:8080 --grid   # + controls

Pass criterion: tool_calls[0].function.name == "read_file" and its arguments parse as JSON
to an object whose path contains /tmp/notes.txt. Anything else — no tool call, a call to a
placeholder tool, unparseable arguments — is a failure.

Tokenizers differ, so match on token count, not line count. 1400 filler lines is ~29.3k
tokens for Qwen3 but only ~25.4k for Llama-3.1, and ~25.3k is inside Qwen3's passing range —
comparing at equal line counts would have produced a meaningless pass. Use --lines to bring
each model to a comparable prompt_tokens.

What was NOT tested

Stated explicitly so no scope is assumed that was not measured:

A newer or older ROCm. Only ROCm 7.2.4.
Anything past b21e4de74567f. That was master at the time of writing.
Other gfx targets. Only gfx1151. Discrete RDNA/CDNA cards untested.
A second MoE. The dense-vs-MoE hypothesis rests on one model.
More than three architectures, and only one quant per model except Qwen3 (tested at two).
Non-tool workloads systematically. For qwen35 a short greedy text probe returns fluent,
correct text on the same failing server, so ordinary generation appears unaffected there — but
no perplexity or long-context accuracy measurement was run. For llama the corruption is
visible in plain text output at any depth.

Why this is easy to miss

For qwen35, a health check that asks the model to complete a sentence passes on this backend
every time. The server is fast, fluent and confident while being unable to see the front of its
own tool array. Any monitoring built on short text probes reports it green.

`repro_hip_toolcall.py`

#!/usr/bin/env python3
"""
Minimal reproducer: HIP/ROCm backend emits wrong or missing tool calls at scale,
where the Vulkan backend on the same weights and identical flags does not.

Self-contained: Python 3 standard library only. No project files needed.

USAGE
    # start llama-server with the HIP build, then:
    python3 repro_hip_toolcall.py --base http://127.0.0.1:8080

    # full grid that eliminates depth / tool-count / total-length:
    python3 repro_hip_toolcall.py --base http://127.0.0.1:8080 --grid

WHAT IT SENDS
    A chat completion carrying N tool definitions. Tool 0 is the real one
    ("read_file"); the rest are same-shaped placeholders. The user message is
    <lines> lines of filler followed by an explicit instruction to call
    read_file. Everything is generated here, so the request is fully described
    by (n_tools, lines).

WHAT COUNTS AS PASS
    tool_calls[0].function.name == "read_file" AND its arguments parse as JSON
    to an object whose "path" contains /tmp/notes.txt. Anything else fails:
    no tool call at all, a call to a placeholder tool, or unparseable arguments.

EXPECTED
    Vulkan backend : 8/8 correct
    HIP backend    : ~3/8 correct, and the failures are qualitatively wrong --
                     the model states it has no read_file tool and lists its
                     own tools incorrectly (e.g. "tool_00 through tool_14" for
                     a 17-tool array whose first entry is read_file), or calls
                     a placeholder tool with the real tool's name stuffed into
                     a string argument.
"""
import argparse, json, sys, urllib.request

REAL_TOOL   = "read_file"
TARGET_PATH = "/tmp/notes.txt"
ASK  = f"Use the {REAL_TOOL} tool to read {TARGET_PATH}. Call the tool, do not describe it."
LINE = "context line {:05d}: background material the assistant should keep in mind while working"


def build_tools(n_tools):
    tools = [{"type": "function", "function": {
        "name": REAL_TOOL, "description": "Read a text file from disk",
        "parameters": {"type": "object",
                       "properties": {"path": {"type": "string", "description": "absolute path"}},
                       "required": ["path"]}}}]
    for i in range(n_tools - 1):
        tools.append({"type": "function", "function": {
            "name": f"tool_{i:02d}", "description": f"Placeholder utility {i}",
            "parameters": {"type": "object",
                           "properties": {"arg": {"type": "string", "description": "argument"}},
                           "required": ["arg"]}}})
    return tools


def build_body(n_tools, lines, nonce):
    filler = "\n".join(LINE.format(i) for i in range(lines))
    return {"max_tokens": 500, "temperature": 1.0, "top_k": 20,
            "messages": [{"role": "user", "content": f"{filler}\n\n[probe {nonce}]\n{ASK}"}],
            "tools": build_tools(n_tools)}


def one(base, n_tools, lines, nonce, timeout):
    req = urllib.request.Request(
        base.rstrip("/") + "/v1/chat/completions",
        data=json.dumps(build_body(n_tools, lines, nonce)).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        d = json.loads(r.read().decode())
    ch = (d.get("choices") or [{}])[0]
    m  = ch.get("message") or {}
    pt = (d.get("usage") or {}).get("prompt_tokens")
    tc = m.get("tool_calls") or []
    if not tc:
        return False, pt, f"NO_TOOL_CALL finish={ch.get('finish_reason')} content={(m.get('content') or '')[:100]!r}"
    f = tc[0].get("function") or {}
    try:
        args = json.loads(f.get("arguments") or "")
    except Exception:
        args = None
    if f.get("name") != REAL_TOOL:
        return False, pt, f"WRONG_TOOL name={f.get('name')} args={str(f.get('arguments'))[:70]!r}"
    if not isinstance(args, dict) or TARGET_PATH not in str(args.get("path", "")):
        return False, pt, f"BAD_ARGS args={str(f.get('arguments'))[:70]!r}"
    return True, pt, "CORRECT"


def cell(base, n_tools, lines, n, timeout, verbose=True):
    ok, depth = 0, None
    for i in range(1, n + 1):
        try:
            good, pt, detail = one(base, n_tools, lines, str(i), timeout)
        except Exception as e:
            print(f"  #{i} REQUEST_FAILED {e}")
            continue
        depth = pt if pt is not None else depth
        ok += 1 if good else 0
        if verbose:
            print(f"  #{i} {detail}")
    print(f"  => {n_tools} tools, {lines} filler lines, prompt_tokens={depth}: {ok}/{n} correct\n")
    return ok, depth


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="http://127.0.0.1:8080")
    ap.add_argument("-n", type=int, default=8, help="repeats per cell")
    ap.add_argument("--tools", type=int, default=17)
    ap.add_argument("--lines", type=int, default=1400)
    ap.add_argument("--timeout", type=int, default=300)
    ap.add_argument("--grid", action="store_true",
                    help="also run the control cells that eliminate depth, tool count and total length")
    a = ap.parse_args()

    print(f"endpoint {a.base}\n")
    print(f"[FAILING CELL] {a.tools} tools x {a.lines} lines")
    cell(a.base, a.tools, a.lines, a.n, a.timeout)

    if a.grid:
        # Each control changes exactly one thing away from the failing cell.
        print("[CONTROL] same tool count, shallow  -> isolates context length")
        cell(a.base, a.tools, 0, a.n, a.timeout, verbose=False)
        print("[CONTROL] same tool count, shorter  -> isolates the depth at which it starts")
        cell(a.base, a.tools, 1200, a.n, a.timeout, verbose=False)
        print("[CONTROL] one tool, same depth      -> isolates tool count")
        cell(a.base, 1, a.lines, a.n, a.timeout, verbose=False)
        print("[CONTROL] one tool, LONGER          -> isolates total prompt length")
        cell(a.base, 1, 1800, a.n, a.timeout, verbose=False)
    return 0


if __name__ == "__main__":
    sys.exit(main())

Follow-up: seven mitigation attempts, none worked — no fix to offer

I tried to find a workaround before/after filing. None of these helped, so this is a
ruled-out list rather than a fix. Reporting it so nobody repeats the work.

Baseline for these runs: qwen35 on the HIP backend at 29,355 prompt tokens with 17 tools,
which scores 1/16 correct. Screening was 4 requests per candidate (a candidate would have to
be visibly better than baseline to earn a full 16-run confirmation; none was).

#	attempt	level	result
1	`--ctx-checkpoints 32` → `0`	config	1/4
2	`--cache-reuse 256` → `0`	config	0/4
3	`--ubatch-size 1024` → `512`	config	0/4
4	`GGML_CUDA_DISABLE_FUSION=1`	runtime env	1/4
5	`GGML_CUDA_DISABLE_GRAPHS=1`	runtime env	1/4
6	`-DGGML_CUDA_FORCE_MMQ=ON`	build	0/4
7	`-DGGML_CUDA_FORCE_CUBLAS=ON`	build	1/4

Attempts 6 and 7 are deliberately the two opposite ends of the quantised-matmul path. Neither
extreme changes the outcome, which I read as evidence against the dense-vs-MoE
mul_mat/mul_mat_id hypothesis in my original post — that hypothesis is weaker than when I
filed, and I would not want anyone spending time on it on my account.

Every candidate was verified live before measuring: env vars read back from
/proc/<pid>/environ, build options read back from CMakeCache.txt, and the serving binary
from /proc/<pid>/exe.

Still unexplained, and still the sharpest lead I have: the corruption consistently drops entries
from the head of the tools array (an array of read_file, tool_00 … tool_15 is reported
back as "tool_01 through tool_15", with the placeholder descriptions read correctly), which
suggests something specific to the earliest part of the context rather than a general precision
problem.

Follow-up: the MoE data point is now depth-swept — 80/80 from 1,277 to 64,496 tokens

In the original report the bailingmoe3 (MoE) result was a single cell at 29,275 prompt tokens.
That is about 80 tokens below the depth at which qwen35 fails, which is far too marginal to
carry the dense-vs-MoE lead. I have now swept it, because a pass just under the boundary and a
pass well past it are very different pieces of evidence.

Same reproducer, same 17-tool array, same pass criterion, HIP backend:

filler lines	`prompt_tokens`	trials	correct	vs the `qwen35` failure depth (29,355)
0	1,277	16	16/16	0.04× (shallow)
1400	29,276	16	16/16	0.997× — reproduces the originally reported cell
2186	44,996	16	16/16	1.53×
2936	59,996	16	16/16	2.04×
3161	64,496	16	16/16	2.20× — top of the 65,536 window
		80	80/80

No trial produced anything other than a correct read_file call with parseable arguments.
64,496 tokens is the ceiling of the served context, so this is as deep as this configuration can
be pushed; qwen35 is at 0/16 by 29,355.

One methodological caveat, because it affects reproduction

--cache-prompt --cache-reuse 256 were on — as they were for every cell in the original
report, including the qwen35 0/16, so the cells are directly comparable. But it means each row
is one cold deep prefill followed by 15 hops that reused the KV prefix (the server logged
f_sim_best=0.999, f_keep=0.998, ~1,028 tokens re-prefilled per hop). Deep prefill compute
was amortised across the 16 trials and these are not 80 independent cold prefills. What was
exercised on every trial: deep-context attention at decode, and tool-call emission — which is
where the qwen35 corruption manifests. Anyone wanting 80 genuinely cold prefills should set
--cache-reuse 0.

Also, for precision: these are 16 requests in a single invocation, not the 8-run-twice pattern
used for the cells in the original report.

Config

Identical to the flags already listed in the issue — -ngl 999 --flash-attn on --ctx-size 65536 --ubatch-size 1024 --batch-size 2048 --parallel 1 --threads 16 --cache-type-k f16 --cache-type-v f16 --cache-prompt --cache-reuse 256 --cache-ram 0 --ctx-checkpoints 32 --checkpoint-min-step 8192 --jinja (model's own embedded template), no --mmproj, no
speculative decode — on stock b21e4de74567f5eef213765c9476a843c2e43f0d built for gfx1151 with
ROCm 7.2.4.

All five cells ran against one server process that was never restarted, and the backend
binary was read from /proc/<pid>/exe before the first trial and again after the last.

Depths were matched on tokens, not lines: the 0-line cell measures this tokenizer's intercept
at 1,277 tokens (≈20.0 tokens/line), and the deeper line counts were computed from that, landing
within 0.01% of the 45k/60k/64.5k targets.

Model detail that may matter for the `mul_mat_id` lead

Read from the GGUF metadata rather than the model card, since the routing shape is the part the
hypothesis actually depends on:

bailingmoe3.expert_count = 512, bailingmoe3.expert_used_count = 8
expert_group_count = 8, expert_group_used_count = 4, expert_shared_count = 1
block_count = 43, leading_dense_block_count = 2 (so 41 MoE blocks, 2 dense)
embedding_length = 2560, expert_feed_forward_length = 768

So it is 8-of-512 routed plus 1 shared expert per MoE block — the routed matmuls are narrow and
numerous, not a handful of wide ones. Worth noting the first two blocks are dense, and they
pass here while wholly-dense llama fails at every depth.

What this does and does not change

It does not resolve the issue. llama and qwen35 still fail on HIP exactly as reported.
The dense-vs-MoE lead survives a test that could have killed it. Had the MoE failed at 45k
or 60k, the split would have been an artefact of testing it just under the boundary. It did not.
It is still one MoE architecture. Depth coverage is not architecture coverage, and this
cannot separate bailingmoe3-the-architecture from this specific model. The "a second MoE was
not tested" limitation in the original report still stands unchanged.
Set against this, the two build-level probes reported earlier — -DGGML_CUDA_FORCE_MMQ=ON and
-DGGML_CUDA_FORCE_CUBLAS=ON, opposite ends of the quantised matmul path — changed nothing.
So the lead is strengthened on the depth axis while remaining weak on the mechanism axis.

We can run a second MoE architecture on this hardware — that is the experiment that would
actually separate bailingmoe3-the-architecture from this one model — or any specific cell
you would find useful. Nothing is scheduled; say which would help most and we will run it.

Correction: the `qwen35` failure starts at ~4k prompt tokens, not ~29k — reproduces far more cheaply than reported

The original report says:

qwen35 is correct at shallow depth and fails past roughly 29k tokens when the request also carries a moderately large tool array.

The onset is an order of magnitude lower. I had only ever measured two depths on this model — 1,349 (passing) and 29,355 (failing) — and nothing in between. Bisecting that gap puts the onset between 3,395 and 4,015 prompt tokens.

This makes the defect cheaper and faster to reproduce: a ~4k-token prompt with 17 tools is enough. Nobody needs to build a 29k-token request.

Measured, HIP backend, n=8 per cell

`prompt_tokens`	correct
1,355	8/8
1,995	8/8
2,695	8/8
3,395	8/8
4,015	1/8
8,015	5/8
29,355	0/8

The failure mode at 4,015 is the one already described in the issue — fluent output, no tool call, the model asserting the tool does not exist:

I can't call a read_file tool in this session — no such tool is available among my current capabilities

It is not a clean cliff — please don't reproduce at one depth and conclude

8/8 → 1/8 across a 620-token step looks like a threshold, but 8,015 then scores 5/8, better than 4,015. So the shape is:

below ~3.4k — clean and deterministic, never observed to fail
from ~4k — an unstable band, intermittent, rate varies non-monotonically with depth
by ~29k — total

Practical consequence for anyone bisecting this: a single 8/8 in the 4k–29k band does not clear a change. The rate between those points is sampled at two depths only and is not characterised — treat anything in that band as intermittent and use enough trials.

Vulkan control at the same depths

Same commit, same weights, same flags, backend read from /proc/<pid>/exe before the first trial and after the last, one server process throughout:

`prompt_tokens`	HIP	Vulkan
4,015	1/8	8/8
8,015	5/8	8/8

Token counts are exact matches between the legs, not approximations. The two launchers differ by 2 argv tokens out of 55 — the binary path, and a separate slot-save directory so the legs cannot share slot state.

The generation lengths are as informative as the pass counts: all 16 Vulkan trials emitted the tool call within 48–96 generated tokens, while the HIP failures at the same depths ran to 271–500 tokens explaining that no such tool exists.

Two config differences from the cells in the original report — stated because they are real

These cells were run with --mmproj present and with a custom chat template (--chat-template-file) rather than the model's own embedded one. The original report's cells had no --mmproj and used each model's embedded template. So these rows are not drop-in replacements for the ones in the issue body.

I think the finding survives that mismatch, because the two configs agree at both ends of the range — the original config is 6/6 at 1,349 and 0/16 at 29,355; this config is 8/8 at 1,355 and 0/8 at 29,355. The disagreement was never about the endpoints; it is that the middle had never been measured by either config. But it is a difference, so: the exact onset under the issue body's own config is UNVERIFIED. If the precise threshold matters to a fix, it should be re-bisected without --mmproj and with the embedded template.

Everything else is unchanged from the issue: stock b21e4de74567f5eef213765c9476a843c2e43f0d, gfx1151, ROCm 7.2.4, -ngl 999 --flash-attn on --ctx-size 65536 --ubatch-size 1024 --batch-size 2048 --parallel 1 --threads 16 --cache-type-k f16 --cache-type-v f16 --cache-prompt --cache-reuse 256 --cache-ram 0 --ctx-checkpoints 32 --checkpoint-min-step 8192, MTP off, same reproducer and same pass criterion.

One asymmetry worth naming rather than leaving implicit: GGML_VK_MAX_NODES_PER_SUBMIT=1 is set on the Vulkan leg and not on HIP, where it is inert. It caps Vulkan submission batching and is set here only because every earlier Vulkan run in this investigation used it, keeping the cells comparable. I see no mechanism by which it would repair a tools array, but that is argued, not measured — UNVERIFIED.

What this does not change

The 1 tool at 36,335 tokens is 5/5 correct row in the eliminated-causes table still stands, and is worth re-reading alongside this: with one tool the model is fine at 36k, while with 17 tools it degrades from ~4k. Depth alone is still not the cause — it is the interaction with the tool array, and that interaction bites much earlier than reported.

The reproducer, the pass criterion and the mul_mat vs mul_mat_id lead are all unchanged. The only thing that moves is the threshold, and it moves in the direction that makes this easier to investigate.

Chiming in that I likely have a similar issue, which presently makes ROCm and llama.cpp combination useless. Vulkan seems to work without an issue.

In my case, I start opencode and the main agent thinks it is the title generator. It is often visibly confused in its reasoning trace, e.g. revision b0539c4 which is the latest at time of writing, I get talk like this:

It doesn't happen 100% of the time, but probably like 80-90 % of the time. I think KV caches are being mixed up or something, and this is serious issue. I also think the same issue -- or related one -- is happening with CUDA because I get Qwen3.8-27B producing answers to me that are clearly intended for some other session. So CUDA path inference is presently probably broken, I think, but in way that very severely impacts ROCm and only at relative rarely impacts genuine CUDA.

Vulkan works even on nvidia hardware, so I've switched for now to that.

By the way, model is quasi-coherent if you have single slot. Enable two, and watch the carnage:

Needless to say, the model absolutely doesn't talk like this on Vulkan.

I think this might be the root cause also of the issue I've just opened.
#27612

I've been fighting with it for weeeks, and the only solution was to use vulkan

I just god bad generation even on Vulkan, though. So whatever the problem is, it might cross all backends. I think frequency is ROCm >> CUDA > Vulkan.

I continued 3 tasks with about 80-170k tokens in each against a new llama-server instance. Two seemed to resume working as expected, but third produced confused XML tag soup that made no sense in reasoning.

So, presently, I am cautiously claiming that there might exists no backend in which Qwen3.8-27B works correctly on llama.cpp, though for now I would recommend Vulkan because it might work 99.9 % correctly. I aborted the bad generation and resumed from a prior message and the model seemed to recover. That is 4 slots on llama-server and unified KV cache on Vulkan. I use Q8_0 and F16 KV caches in order to have high quality generations and other than these extremely rare moments when the model suddenly spews complete garbage all of a sudden, it seems to be working well on Vulkan.

On CUDA, I've seen incoherent ramblings of mixed up sessions.

On ROCm, basically zero usable output.

I can add a characterization of one aggressive trigger condition on the same hardware
(multi-slot serving — which may or may not be the same defect as the single-slot
tool-array failures documented above), plus a cross-backend comparison that points at
the HIP backend specifically (not ROCm/hardware in general).

Setup: Qwen3.8-27B (hybrid GDN + MTP-capable), llama.cpp master @70adb1b4c (b10588),
AMD Strix Halo (gfx1151, Radeon 8060S), ROCm 7.2.2. tools/server, ShareGPT-derived load,
20-100 requests per condition, vllm bench serve --save-detailed to capture actual generated
text (not just HTTP success/latency — see note at the end on why this matters).

Trigger, isolated:

Single slot (--parallel 1) was healthy up to 32k+ tokens on our workload
(plain ShareGPT-style text generation and a long-context recall probe). We had
not run the tool-array reproducer from this issue — we are running it now, with and
without the workaround below, and will follow up. This point characterizes our
text-gen workload; it is not a contradiction of the single-slot failures reported
above, which exercise a cell our tests did not cover.
Multiple slots (--parallel ≥ 2) reliably corrupts output on HIP. Generated text
degenerates into a repeated single character (e.g. long runs of /). This reproduces
starting exactly at c=2 (measured: 17/20 sampled outputs flagged at c=2, 19/20 at c=4,
consistent through c=32), both with short (~1-4k token) and long (~24k token) prompts —
long-context multi-slot is where it's most severe (100% of sampled outputs corrupted in our
8-way-concurrent long-context test).
This is independent of speculative decoding (MTP). We initially suspected an MTP-specific
async-copy race (see our comment on Misc. bug: [draft-mtp] Draft acceptance collapses to 0.0 under -np N with multi-ubatch batches — async t_h_nextn device→host copy race #27572), but ON/OFF comparison with MTP completely
disabled shows the same corruption at the same trigger point and a similar failure rate
(e.g. c=8: MTP-on 20/20 outputs flagged degenerate, MTP-off 20/20; c=32: on 15/20, off 15/20).
MTP is not the cause; if anything the acceptance-rate collapse we reported in Misc. bug: [draft-mtp] Draft acceptance collapses to 0.0 under -np N with multi-ubatch batches — async t_h_nextn device→host copy race #27572 looks
like a downstream symptom of the same underlying multi-slot corruption.
Vulkan backend, identical model/flags, does not reproduce. All multi-slot conditions we
tested (c=1/8/16/32, MTP on and off, up to 32k context) produced clean output on Vulkan.
vLLM (ROCm/HIP via PyTorch, same GPU) does not reproduce either. Same model, same
hardware, MTP on, c=1 through c=32 — clean output every time (checked with the same
degenerate-output detector). This is the strongest signal that this is a llama.cpp HIP
backend implementation issue, not a ROCm/gfx1151/hardware-level problem — vLLM exercises
ROCm/HIP on the same silicon without hitting it.
Did NOT reproduce with a second, architecturally different model — Gemma-4-26B-A4B
(dense+MoE, no GDN/hybrid-linear-attention layers), same trigger condition (HIP,
--parallel 16, 20 requests): 0/20 corrupted. This is a single data point (no repeat runs,
no sweep across other c values for this model), but it raises the possibility that the bug
is specific to Qwen3.8's GDN/hybrid architecture rather than a fully general HIP multi-slot
bug. We'd treat "general vs. GDN-specific" as still open pending more models/conditions —
flagging it rather than asserting either way.

Why the output-content check matters: our first pass at this only checked HTTP success and
throughput (the standard vllm bench serve summary), which all showed "200/200 completed" —
looking fine. It was only after we started saving --save-detailed and inspecting
generated_texts that we found the corruption. If anyone else has run multi-slot HIP benchmarks
and only checked success/throughput, it's worth going back and looking at the actual text.

Regression bisect: prompted by a comment on #27506 that pins the regression to "b10040 and
later," we ran git bisect between b10039 (5839ba3) and b10588 (70adb1b) using an automated
HIP-build + 8-way-concurrent 20-request + degenerate-output-detector test at each step (9
iterations, all intermediate commits tested BAD/corrupted, converging quickly toward the good end).

First bad commit: c7d8722 — "ggml-cuda: restore
prop.integrated on HIP builds" (#24233). The diff:

+#if defined(GGML_USE_HIP)
+        info.devices[id].integrated = prop.integrated;
+#else
         info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
+#endif

Before this commit, integrated was force-false for all CUDA/HIP devices (a blanket workaround
for unrelated Jetson Orin corruption, #15034). This commit restores the real
hipDeviceProp_t.integrated value on HIP builds only — which on an APU/UMA part like Strix Halo
(gfx1151) is true. That flips ggml_backend_cuda_device_supports_buft()
(ggml-cuda.cu:5006-5009) to accept CUDA/HIP host (pinned) buffers on this device, a code path
that was previously unreachable on any HIP device.

Credit where due: after posting this we read #27506 more closely — Rcat999 identified
this same #ifdef and confirmed the integrated = false revert three days before our
bisect completed (with sjvrensburg reproducing on a second gfx1151 machine). First
identification belongs there. Please read our bisect as an independent, automated
confirmation converging on the same commit from a different symptom (multi-slot
degenerate output rather than PPL instability) — two independent routes to the same
commit should raise confidence that this is the right place to look.

We can't yet say with confidence why this specific path corrupts multi-slot output — our
working hypothesis is that tensors now eligible to live in host-buffer memory on this path aren't
covered by the same synchronization/dependency tracking as the normal device-buffer path (see our
async-copy note re: get_tensor_async on #27572), and that gap only bites once there's genuine
concurrency (--parallel ≥ 2) to race against. We have not confirmed from source whether
GDN/recurrent-state tensors specifically route through this host-buffer path more often than
dense/MoE tensors — if so it would explain why we saw no corruption with Gemma-4-26B-A4B (see
point 6), but we're flagging that as an open question, not a confirmed mechanism. Given the
integrated-GPU host-buffer angle, this may only affect APU/UMA parts (Strix Halo, and potentially
other integrated AMD GPUs) rather than discrete HIP/ROCm cards — worth checking if anyone can
reproduce (or not) on a discrete AMD GPU.

Confirmed: forcing integrated = false on current master (one-line revert of the #ifdef)
eliminates the corruption in our c=8 test — so this works as an immediate workaround for APU
users needing multi-slot HIP. (One-line diff, patch available on request.)

Happy to share full server logs, raw JSONs (including generated_texts), the build+test harness
script, or the one-line workaround patch if useful for reproduction / mitigation on other hardware.

(Edited: clarified the opening line and point 1 after a closer read of the thread — the
single-slot tool-array failures reported above exercise a workload our original tests did not
cover; we are running that reproducer now and will follow up with results. Edited again to
credit Rcat999's earlier identification of the same commit in #27506.)

Follow-up after digging into the tool-array reproducer and the discussion above, plus a
cross-check against issue #27572.

(a) Scope correction: our "single-slot is healthy" claim in the main report was measured on
plain generation, not tool-call arrays — we hadn't tested the cell you isolated. We ran your
repro_hip_toolcall.py at the same 29,355-token / 17-tool cell on our hardware (Qwen3.8-27B
Q4_K_M, --parallel 1, embedded template, no mmproj) and reproduced your result closely:
0/8 on plain HIP (same failure signature — the model denying it has a read_file tool),
16/16 on Vulkan. Not a contradiction of our original claim — a gap in what we'd measured.

(b) Our GDN-specific framing, withdrawn: our Gemma-4-26B-A4B data point (clean, no GDN) was
MoE, which is at least consistent with your table (two dense architectures failing, one MoE
clean). But given what we now know the mechanism to be (see (c)/(d) below — a process_ubatch
race, not a matmul kernel issue), we don't think dense-vs-MoE is best read as a mul_mat vs.
mul_mat_id kernel difference. Your own mitigations #6/#7 (GGML_CUDA_FORCE_MMQ and
GGML_CUDA_FORCE_CUBLAS — opposite ends of the quantized-matmul path) both leaving the failure
unchanged already argued against a matmul-kernel cause, and the race we found is consistent with
that null result. We'd now frame dense-vs-MoE as a downstream difference — which tensors and
ubatch shapes happen to land in the race window — rather than a property of the matmul path
itself. Our earlier "possibly GDN-specific" framing is withdrawn either way, in favor of your
lead.

(c) Connecting to #27572: worth flagging here since it appears to be the same underlying
defect. The MTP acceptance-rate collapse we originally reported there, and the reporter
(ByungHyun21)'s own follow-up investigation, converged on a race in
llama_context::process_ubatch(): the H2D input-tensor write (ggml_backend_tensor_set →
cudaMemcpyAsync on HIP's hipStreamPerThread, which has no implicit sync with the compute
stream) can land while the previous ubatch's graph is still reading those same input tensors.
The existing sync guard (if (cparams.pipeline_parallel) { ggml_backend_sched_synchronize(...); }) only fires with multiple GPUs — a single-GPU setup (ours, and presumably yours) skips it
entirely. This is not MTP-specific and not multi-slot-specific; it's general
continuous-batching/ubatch processing, which also explains why it can fire within a single slot
on a long prompt processed in multiple prefill ubatches.

(d) Unified mechanism, tested: we applied the fix ByungHyun21 posted in #27572 (moving the
scheduler sync to immediately before set_inputs(), unconditionally — no upstream PR exists for
it yet as of this writing) and tested it two ways, keeping integrated=true (i.e. the
host-buffer path from #24233 still active):

On our multi-slot corruption cell (--parallel 8/16/32, MTP off): clean at all three,
vs. reliably corrupted before.
On your tool-array cell (29,355 tokens, 17 tools, --parallel 1): 16/16, vs. 0/8 on
unpatched HIP in our control run.

We also re-tested our original integrated=false workaround (the amplifier-only fix, no sync
change) against your cell: also 16/16. So either fix alone resolves both trigger conditions in
our hands.

Working model (two layers, not fully proven but consistent with everything measured so
far): the base defect is the process_ubatch H2D/compute race described above — backend-
generic in principle, gated out on multi-GPU by the existing guard, ungated on single GPU. The
amplifier is #24233's integrated=true restoration on HIP, which enables the CUDA/HIP host-
buffer path on APU/UMA parts and appears to make the race window wide enough to hit routinely.
Closing either layer (the sync fix, or reverting to integrated=false) removed the symptom in
every condition we tested — we have not measured whether the race is fully closed by the sync
fix alone or only reduced below our detection threshold, so we'd call the host-buffer angle a
strong amplifier hypothesis rather than a proven one.

(e) Frequency gradient: worth noting alongside this — alankila's reports earlier in this
thread: rare corruption even on Vulkan at very long context (80-170k tokens, 4 slots), and
occasional cross-session mixing observed on CUDA, together suggest a frequency gradient
(ROCm ≫ CUDA > Vulkan) rather than a HIP-exclusive bug. That's consistent with the base-layer
defect being backend-generic and HIP's stream semantics (no implicit sync on
hipStreamPerThread) simply exposing it far more readily than Vulkan's scheduling model. See
also #27612, which looks like it may be the same defect surfacing as partial tool-call failures.

(f) On #24233 itself: to be clear, we don't think the fix in #24233 was wrong to attempt —
the dGPU+iGPU device-misclassification problem it targeted (#23977) is real, and reverting it
outright would reopen that. Our integrated=false patch (the same workaround Rcat999 arrived at
independently in #27506) is a workaround for APU users who need multi-slot HIP today, not a
proposed permanent fix. The durable fix looks like the process_ubatch sync correction, which
should let integrated=true stand once landed.

Happy to share the sync-fix patch, our test harness, or raw results (generated_texts,
tool-call payloads) for either build. Note we have not tested multi-GPU / pipeline_parallel
configurations — the existing sync guard already covers that path, so we'd expect it to be
unaffected, but we haven't measured it.

Verified against #27311 (Scheduler UMA ring buffer) on the same hardware — the tool-array
failure described here goes 16/16 with the ring enabled, 4/16 with it disabled, and 3/16 on
stock master as a control. Full numbers: #27311 (comment)

Additional reproduction: Ryzen AI MAX+ 395 / Qwen3.8-27B Q8_0

I can reproduce the gfx1151 HIP/ROCm correctness issue on a Ryzen AI MAX+ 395 with Radeon 8060S.

Hardware
CPU: AMD Ryzen AI MAX+ 395
GPU: Radeon 8060S
GPU architecture: gfx1151
OS: Linux
llama.cpp
commit: 4e97ac8
version: 0.3.0-dev (build 10666)
compiler: GNU 16.2.1
model: Qwen3.8-27B Q8_0 GGUF
Observed behavior

With the ROCm/HIP backend, Qwen3.8-27B produces clearly incorrect/nonsensical responses.

The same model and configuration produce normal, coherent responses when using the Vulkan backend on the same machine.

Adding:

HIP_LAUNCH_BLOCKING=1

to the ROCm process immediately restores correct model output.

So the behavior is:

ROCm/HIP: incorrect output
ROCm/HIP + HIP_LAUNCH_BLOCKING=1: correct output
Vulkan: correct output
Performance

Interestingly, the newer ROCm build is approximately 15% faster than our previous ROCm setup, so we would prefer to keep using it and only apply the synchronization workaround.

Reproduction

The model is run with:

-ngl 99
-c 262144
-fa on
-np 1
--cache-type-k q8_0
--cache-type-v q8_0
--spec-type draft-mtp
--spec-draft-n-max 4
--spec-draft-type-k q4_0
-b 4096
-t 16

The issue is consistently reproducible without HIP_LAUNCH_BLOCKING=1 and disappears when the variable is set.

This appears to be another reproduction of the gfx1151 asynchronous-execution correctness issue described in this issue.

@SenseiHans

@SenseiHans — thanks, and the HIP_LAUNCH_BLOCKING=1 result is the most useful thing
posted in this thread so far. It demonstrates the async-execution mechanism without
touching a line of code: force every HIP call to be synchronous and the corruption
disappears. That is exactly what the proposed fixes are addressing from the inside.

Your case also lands on the trigger rule that @Superfish1000 established in #27311 on
gfx1151: corruption appears when n_ubatch < n_batch. You have -b 4096 with no -ub,
so the default 512 applies — eight ubatches per batch. Worth confirming: does
-b 512 -ub 512 (or any config with ub == b large enough for your prompt) also make it
go away, without HIP_LAUNCH_BLOCKING? If so that gives you a workaround with far less
performance cost than full serialization.

Which brings me to the 15%. HIP_LAUNCH_BLOCKING=1 serializes every kernel launch, so I
would expect it to eat most or all of that gain — possibly more. If you have the numbers
with and without it on the same build, they would be worth posting: several people in
these threads are weighing "stay on ROCm with a workaround" against "switch to Vulkan",
and @thedelph measured in #27506 that switching to Vulkan on this chip cost 5–9% on
prefill but gained 24% on decode. If your serialized ROCm build ends up slower than
Vulkan, that changes the recommendation.

The fix under review is #27311 (scheduler ring buffer for graph input tensors), with
#25863 as a smaller targeted alternative. I verified #27311 on the same hardware class —
the tool-array failure in this issue goes 16/16 with the ring enabled versus 4/16 with it
disabled, and the scheduler sanitizer goes from 3597 write-after-read races to zero.
Numbers: #27311 (comment)

Navigation Menu

Repository navigation

HIP/ROCm backend produces corrupted output on gfx1151: two dense architectures fail, Vulkan is correct with byte-identical flags #27579

Summary

Two severities, deliberately not merged

Hypothesis, on a single data point — please treat as a lead, not a finding

Still present on current master

It is not `e79e4bf66` — please do not close this as a duplicate of that commit

Environment

Server flags — identical on both backends, only the binary differs

Causes eliminated, each by changing ONE variable

Reproducer

What was NOT tested

Why this is easy to miss

`repro_hip_toolcall.py`

Follow-up: seven mitigation attempts, none worked — no fix to offer

Follow-up: the MoE data point is now depth-swept — 80/80 from 1,277 to 64,496 tokens

One methodological caveat, because it affects reproduction

Config

Model detail that may matter for the `mul_mat_id` lead

What this does and does not change

Correction: the `qwen35` failure starts at ~4k prompt tokens, not ~29k — reproduces far more cheaply than reported

Measured, HIP backend, n=8 per cell

It is not a clean cliff — please don't reproduce at one depth and conclude

Vulkan control at the same depths

Two config differences from the cells in the original report — stated because they are real

What this does not change

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HIP/ROCm backend produces corrupted output on gfx1151: two dense architectures fail, Vulkan is correct with byte-identical flags #27579

Description

Summary

Two severities, deliberately not merged

Hypothesis, on a single data point — please treat as a lead, not a finding

Still present on current master

It is not e79e4bf66 — please do not close this as a duplicate of that commit

Environment

Server flags — identical on both backends, only the binary differs

Causes eliminated, each by changing ONE variable

Reproducer

What was NOT tested

Why this is easy to miss

repro_hip_toolcall.py

Activity

Biggles10-claude commented on Aug 22, 2026

Follow-up: seven mitigation attempts, none worked — no fix to offer

Biggles10-claude commented on Aug 23, 2026

Follow-up: the MoE data point is now depth-swept — 80/80 from 1,277 to 64,496 tokens

One methodological caveat, because it affects reproduction

Config

Model detail that may matter for the mul_mat_id lead

What this does and does not change

Biggles10-claude commented on Aug 23, 2026

Correction: the qwen35 failure starts at ~4k prompt tokens, not ~29k — reproduces far more cheaply than reported

Measured, HIP backend, n=8 per cell

It is not a clean cliff — please don't reproduce at one depth and conclude

Vulkan control at the same depths

Two config differences from the cells in the original report — stated because they are real

What this does not change

alankila commented on Aug 23, 2026

alankila commented on Aug 23, 2026

MetalKnight commented on Aug 23, 2026

alankila commented on Aug 24, 2026

nabe2030 commented on Aug 25, 2026

nabe2030 commented on Aug 25, 2026

nabe2030 commented on Aug 26, 2026

SenseiHans commented on Aug 28, 2026

nabe2030 commented on Sep 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

It is not `e79e4bf66` — please do not close this as a duplicate of that commit

`repro_hip_toolcall.py`

Model detail that may matter for the `mul_mat_id` lead

Correction: the `qwen35` failure starts at ~4k prompt tokens, not ~29k — reproduces far more cheaply than reported