Eval bug: server -np 4 --kv-unified returns other requests' responses verbatim on integrated HIP GPU (gfx1151) — bisected to c7d87229

Summary

Under parallel mixed load (-np 4 --kv-unified), llama-server on an integrated HIP GPU (Strix Halo / gfx1151, 128 GB UMA) returns complete responses that verbatim belong to a different, earlier request. A large-prompt request receives, as its entire response, the literal answer text of a short request from an earlier round. We also observed a chimeric response fusing tokens of two different requests' expected outputs.

Bisected to a single commit: c7d8722 "ggml-cuda : restore prop.integrated on HIP builds" (#24233).

build	result (validated responses, nonce-checked)
`505b1ed` (parent window start)	clean — 0 violations / 168
`5839ba3` (= parent of `c7d8722`)	clean — 0 / 36
`c7d8722`	cross-request replays: 3 hard / 36
`e8f19cc` (window end)	cross-request replays: 7 hard + 1 chimera / 96

The same harness against CPU builds of both window endpoints shows no build-dependent difference (model-compliance noise only) — consistent with the culprit being the HIP integrated-memory path. This presumably explains why it went unnoticed: discrete GPUs are unaffected; it needs an integrated AMD GPU (prop.integrated == 1), multiple slots and concurrent mixed traffic.

Environment

AMD Ryzen AI Max+ 395 (Strix Halo), gfx1151, 128 GB unified memory, ROCm 7.14, Ubuntu 24.04
Build: cmake -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DGPU_TARGETS=gfx1151 -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ (Release)
Server: llama-server -m <model> -ngl 999 -c 65536 -np 4 --kv-unified --jinja --chat-template-kwargs '{"enable_thinking":false}'
Model: Qwen3.6-35B-A3B MoE (Q4-class quant). First hit this in production behind LocalAI (same symptom after its vendored-llama.cpp bump crossed this commit); reproducing on vanilla llama-server rules the wrapper out.

Reproduction

Load pattern per round (~30 s, script below): (B) short "repeat exactly this code: " request → (C) large-prompt (~10k tokens) request whose system prompt requires echoing a per-request nonce in the first line → (D) burst of 4 concurrent short nonce requests (2 streamed / 2 plain). Every response is validated to contain its own nonce and no nonce of an earlier request.

On affected builds the round-N large-prompt request receives a literal foreign nonce as its complete response, typically from a round-(N−1) burst request. Verbatim from run logs at e8f19cc:

R4/C got='pd-20260722-115649-3-d1'      (expected: reply echoing pc-...-4-c + a summary)
R5/C got='pd-20260722-115649-4-d2'
R6/C got='pd-20260722-115649-1-d1'
R7/C got='pd-20260722-115649-6-d1'
R8/C got='pd-20260722-115649-7-d1'
R3/C got='<!-- run:pc-20260722-115649-2-d0 -->'   <-- chimera: pc-...-2 stem fused with a -d0 burst suffix

In production we additionally observed: a poisoned slot re-serving its stored answer to unrelated requests 20+ minutes later (the victim's usage then reports the foreign request's token counts), and occasional degeneration into repeated single characters (/////...).

repro-crossover.py (needs httpx)

#!/usr/bin/env python3
"""Repro: llama-server cross-request response mix-up under -np 4 --kv-unified.

Each round: (B) short "repeat exactly: <nonce>" request, (C) large-prompt
(~10k tokens) request whose system prompt requires echoing a per-request nonce,
(D) burst of 4 concurrent short nonce requests (2 streamed / 2 plain).
Every response must contain its OWN nonce and no nonce of an EARLIER request.

Usage: python3 repro-crossover.py --url http://127.0.0.1:8080 [--rounds 8]
Requires: pip install httpx
"""
import argparse, asyncio, json, random, string, sys, time
import httpx

FILLER = [
    "Quantized MoE models reach higher aggregate throughput under continuous batching.",
    "The KV cache grows linearly with context length and dominates memory for long prompts.",
    "Speculative decoding only pays off when the draft acceptance rate is high enough.",
    "A reranker only improves retrieval when the candidate set is broad enough.",
]

def big_prompt(seed, chars=30000):
    rnd = random.Random(seed)
    lines = [f"## Research block {seed}"]
    while sum(len(l) for l in lines) < chars:
        tag = "".join(rnd.choices(string.ascii_lowercase + string.digits, k=8))
        lines.append(f"- [{tag}] {rnd.choice(FILLER)}")
    lines.append("\nSummarize the key points in 5 sentences.")
    return "\n".join(lines)

seen, violations, checks = [], [], 0

def check(phase, rnd, nonce, text):
    global checks
    checks += 1
    stem = nonce.rsplit("-", 1)[0]
    problems = []
    if nonce not in text and not (phase == "C" and stem in text):
        problems.append("own_nonce_missing")
    problems += [f"REPLAY_of:{o}" for o in seen if o != nonce and o in text]
    seen.append(nonce)
    if problems:
        violations.append((rnd, phase, problems, text[:120]))
        print(f"  VIOLATION R{rnd}/{phase}: {problems} got={text[:80]!r}", flush=True)

async def echo_req(c, url, model, nonce, stream=False):
    p = {"model": model, "max_tokens": 40, "temperature": 0,
         "messages": [{"role": "user", "content": f"Repeat back exactly this code and nothing else: {nonce}"}]}
    if not stream:
        r = await c.post(f"{url}/v1/chat/completions", json=p)
        return r.json()["choices"][0]["message"]["content"]
    p["stream"] = True
    out = []
    async with c.stream("POST", f"{url}/v1/chat/completions", json=p) as r:
        async for line in r.aiter_lines():
            if line.startswith("data: ") and line != "data: [DONE]":
                try: out.append(json.loads(line[6:])["choices"][0]["delta"].get("content") or "")
                except Exception: pass
    return "".join(out)

async def synth_req(c, url, model, nonce, prompt):
    sysmsg = f"The FIRST line of your reply must be exactly `<!-- run:{nonce} -->`."
    r = await c.post(f"{url}/v1/chat/completions", json={
        "model": model, "max_tokens": 600,
        "messages": [{"role": "system", "content": sysmsg},
                     {"role": "user", "content": sysmsg + "\n\n" + prompt}]})
    return r.json()["choices"][0]["message"]["content"]

async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", default="http://127.0.0.1:8080")
    ap.add_argument("--model", default="default")
    ap.add_argument("--rounds", type=int, default=8)
    a = ap.parse_args()
    ts = time.strftime("%H%M%S")
    async with httpx.AsyncClient(timeout=600.0) as c:
        for rnd in range(1, a.rounds + 1):
            bp = big_prompt("cachehit" if rnd % 2 == 0 else f"uniq-{ts}-{rnd}")
            nb = f"pb-{ts}-{rnd}-b"
            check("B", rnd, nb, await echo_req(c, a.url, a.model, nb))
            nc = f"pc-{ts}-{rnd}-c"
            check("C", rnd, nc, await synth_req(c, a.url, a.model, nc, bp))
            nd = [f"pd-{ts}-{rnd}-d{i}" for i in range(4)]
            res = await asyncio.gather(*[echo_req(c, a.url, a.model, n, stream=(i % 2 == 0))
                                         for i, n in enumerate(nd)], return_exceptions=True)
            for n, r in zip(nd, res):
                if not isinstance(r, Exception):
                    check("D", rnd, n, r)
            print(f"round {rnd} done — violations so far: {len(violations)}/{checks}", flush=True)
    print(f"\nRESULT: {len(violations)} violations / {checks} validated responses")
    return 1 if violations else 0

if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Happy to run additional experiments on this hardware (it reproduces deterministically within 2–4 rounds).

Additional context for triage: the integrated-GPU path was already disabled once for exactly this symptom class — the surrounding code still carries // Temporarily disabled due to issues with corrupted output (e.g. #15034). c7d8722 re-enabled it for HIP only; our bisect shows the corruption is still present on HIP iGPUs (gfx1151/UMA). Re-disabling it (revert) fixes the mix-up on our hardware with no measurable throughput cost in our runs so far — we are running that revert as a downstream patch until a proper fix lands.

fix(llama-cpp): revert upstream c7d87229 — prop.integrated on HIP iGP…

I've been fighting similar issues, but happens on parallel=1 / no-kv-unified=true as well, the model returning stuff that e.g. coding agent use cannot handle at all (wrong tool use, random tokens, verbatim repeated thinking block as user message multiple times, etc... Also arrived on the same commit in the debugging effort.

Machine: Framework 13
GPU: AMD Radeon 890M iGPU — gfx1150 (RDNA3.5, Strix Point), integrated (prop.integrated == 1)
APU: AMD Ryzen AI 9 HX 370
Unified memory: 96 GB
ROCm / HIP: 7.2.4
OS / kernel: Arch Linux, 7.1.4-arch1-1
Model: unsloth Qwen3.6-35B-A3B-MTP GGUF, Q8_0

Swithching to --device Vulkan0 with everything else the same mitigates the error.

Also happy to try different stuff.

I can confirm this as well, additionally when using MiniMax M2.7 on affected builds, I get random tokens, a bunch of garbled output. On Qwen models I just got wrong tools in the response (e.g. when asked, the model responded with the Exa websearch tools, which I have not set as an MCP).

Machine:

GMKtec EVO-X2 AI Mini PC
CPU: AMD Ryzen™ AI Max+ 395
GPU: Radeon 8060S Graphics
Unified memory: 128 Gb
ROCm / HIP: 7.2.4 and 7.14.0 (both fail)
OS: Ubuntu 26.04 w/ Linux 7.0.0-28-generic
Models: MiniMax M2.7 UD-IQ3_XXS (garbled output), Qwen3.5 122B A10B Q5_K_M (wrong tools)

I also switched over to a Vulkan-based build and it fixed all issues.

Yes, this is a must fix for gfx1151! I have experienced broken tool calling from VSCode to Qwen3-Coder-Next-80b-A3b and Qwen3.6-27b-MTP and I had a Hermes agent go haywire thanks to this issue. I just applied the patch referenced above and everything is rosy now! OP might add to title Breaks Qwen Tool Calling from VSCode/Agents. (I also was using Vulkan in the meantime without issues.)

This commit also caused token_embd.weight of Qwen 27B Q8 to move to the slower Strix Halo when used in layer split mode with R9700, both HIP. PP dropped from 1300t/s to 600t/s. override-tensor helped, but it shouldn't be required.

Thanks for the detailed investigation and reproduction.

A fix is being worked on in #25863 and is currently awaiting review. Affected gfx1150/gfx1151 users are welcome to test the PR and share their results while it awaits merge.

Independent reproduction on our own gfx1151 hardware (AMD Ryzen AI Max+ 395 / Radeon 8060S, ROCm 7.2.2), with a much stronger signal than a single occurrence.

Setup: Qwen3-Coder-Next-Q4_K_M (46GB, MoE), -ngl 999 -c 65536 --jinja --no-mmap, three builds:

control = e3546c794 (predates this bug's bisected commit, c7d87229)
affected = current HEAD (4310aa4f8), contains c7d87229
patched = HEAD with a local revert of the 4-line HIP-guard hunk from c7d87229

Used the nonce-based cross-request repro script from this issue, -np 4 --kv-unified, 192 rounds (1152 checks) per build:

Build	Violations / Checks
control	0 / 1152
affected	1151 / 1152
patched	0 / 1152

So on this hardware, reverting just the 4-line HIP guard from c7d87229 eliminates the failure entirely, and it's absent on the commit predating that change.

One caveat worth being precise about: the failure mode we see is not a literal replay of another request's cached text — every violation is the same response degenerating into a fixed-length run of ? characters (invalid/garbage token output), never a readable fragment of another request's response. So this may not be pixel-for-pixel the same symptom described in the original report, but it's the same class of bug (commit-specific, -np4 --kv-unified-specific, concurrent-load-triggered correctness failure on an integrated HIP GPU), fixable with the same revert.

Separately, we also tested -np 1 with no --kv-unified on the affected commit as a possible workaround: 0 violations / 240 checks (48 + a follow-up 192-round strengthening pass). Not exhaustive, but no correctness failures observed in that posture on our hardware.

Happy to share full logs/scripts if useful.

serve: force -np 1, document known upstream quirks

I reproduced on AMD Ryzen AI MAX+ PRO 395 + ROCm: 7.14.0 + llama.cpp (master 01818e4) by running unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf + mmproj-F16, btw, PR #25863 tested on the same hardware class as this issue's bisect — differential results below

Here is the results:

build	greedy anchor after load
`4df29be` (pinned)	FAIL - output "////////////////"
`01818e4` (master)	FAIL - same signature
master + PR #25863	PASS x2 - clean "OK"
master, no --mmproj	FAIL - reproduces without vision

unpatched: 3/3 runs FAIL (one run each, three different unpatched configs) — patched: 2/2 PASS.

one early attempt ran under concurrent 16-job compile load — anchor passed, 2/8 streams stopped early; all tabled results above are idle-host runs; small samples, not claiming determinism

Here is the reproducing configs:

llama-server with -np 8 (split KV, n_ctx_slot=16384), -ngl 99
8 parallel streams x 256 completion tokens (sustained load)
greedy anchor request: "Reply with exactly: OK" (temperature 0)
-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release

Here is the Full receipts

Full receipts (exact flags, per-stream shapes, verbatim anchor output)

https://raw.githubusercontent.com/AIwork4me/Qwen3.8-27B-ROCm/main/docs/results/upstream-controls/e0-build714-4df29be4.json
https://raw.githubusercontent.com/AIwork4me/Qwen3.8-27B-ROCm/main/docs/results/upstream-controls/e1-master-01818e49.json
https://raw.githubusercontent.com/AIwork4me/Qwen3.8-27B-ROCm/main/docs/results/upstream-controls/e2-master-pr25863.json
https://raw.githubusercontent.com/AIwork4me/Qwen3.8-27B-ROCm/main/docs/results/upstream-controls/e3-master-nommproj.json

I'd like to test revised patches / run further diagnostics on this hardware

This is the same "////" signature after sustained multi-stream load on gfx1151 / ROCm 7.14 — differential test of PR #25863 on this hardware reported in #25992

Independent bisect converging on the same first bad commit, with two additional failure modes and a verified workaround. (Full disclosure: this investigation was run by Claude driving my machine; I'm relaying its findings, which I've seen reproduce on my own hardware.)

Environment: GMKtec EVO-X2 (Ryzen AI Max+ 395 / Radeon 8060S, gfx1151, 128 GB unified), Ubuntu 24.04.4, kernel 6.17.0-35, ROCm 7.2.1, llama.cpp master 60addddf, model llama3.3-70B Q4_K_M (-ngl 999 -c 8192).

Failure mode A: single-request long-context corruption (no `-np`, no `--kv-unified`)

Prompts ≳5K tokens produce degenerate output on the HIP backend — short prompts are fine. Deterministic reproducer: a synthetic 5.4K-token "chronicle" of 180 distinct records in the system prompt, then one retrieval question at --temp 0.

master HIP, -fa on: The village of Emberfall a village was a year 137, shortly after a year 137. Record a village…
master HIP, -fa off: same class of garbage → not flash-attention-specific
master HIP, GGML_CUDA_FORCE_CUBLAS=1: same garbage → not MMQ-specific
master Vulkan build, same GPU: correct answer
pre-c7d8722 HIP (tested dded58b4): correct answer

git bisect between dded58b4 (good) and 60addddf (bad), 5.4K-token probe at --temp 0 as the test, converged on:

c7d8722922a2599dc4d77f8808d8e6c2fde5e7a2 is the first bad commit
    ggml-cuda : restore prop.integrated on HIP builds (#24233)

Same commit as the cross-request replay already bisected in this issue.

Failure mode B: `GGML_CUDA_NO_PINNED=1` → gfxhub page faults + hard hang

Trying to dodge the host-buffer path via GGML_CUDA_NO_PINNED=1 on master HIP does not fall back cleanly — llama-cli faults the GPU VM during load and hangs permanently (survives SIGTERM):

amdgpu 0000:c5:00.0: amdgpu:  Process llama-cli pid 303395 thread llama-cli pid 303395
amdgpu 0000:c5:00.0: amdgpu:   in page starting at address 0x000078664c808000 from client 10
amdgpu 0000:c5:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32775)

Verified workaround: neutralize the commit

Forcing the pre-#24233 behavior on master (ggml/src/ggml-cuda/ggml-cuda.cu, the GGML_USE_HIP branch added by c7d8722):

info.devices[id].integrated = false;

restores correct output on the 5.4K-token probe with no performance cost (llama-bench, 70B Q4_K_M, -fa 1):

test	master + workaround	master (corrupted output)	pre-c7d8722 (`dded58b4`)
pp512	131.2	128.4	105.9
pp2048	127.5	126.2	85.9
tg128	5.02	5.00	4.99

Note

The line c7d8722 replaced was itself a workaround for corrupted output on Jetson Orin (#15034) — i.e., integrated = true appears to expose the same underlying host-buffer bug on both platforms, and #24233 re-exposed it on AMD APUs. The corruption is silent and only manifests at depth, so it survives casual testing (all our short-prompt smoke tests passed; a 5K-token retrieval probe caught it).

Can confirm c7d8722 seems to break rocm (7.14 and 7.2.4, Linux 7.2) build on Strix Halo with/without unified KV and even with np=1; garbled output that breaks opencode entirely (just returns a few words like "User wants to" and stops after the initial prompt).

I pulled the latest master and manually reverted just c7d8722 and things started working normally again, though with a significant hit to prefill performance (~1300t/s -> 1000t/s on Ornith 1.5 35B/A3B Q8_0).

docs: corrections from an internal fact-check (measurement labels, bu…

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

Eval bug: server -np 4 --kv-unified returns other requests' responses verbatim on integrated HIP GPU (gfx1151) — bisected to c7d87229 #25992

Summary

Environment

Reproduction

Failure mode A: single-request long-context corruption (no `-np`, no `--kv-unified`)

Failure mode B: `GGML_CUDA_NO_PINNED=1` → gfxhub page faults + hard hang

Verified workaround: neutralize the commit

Note

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Eval bug: server -np 4 --kv-unified returns other requests' responses verbatim on integrated HIP GPU (gfx1151) — bisected to c7d87229 #25992

Description

Summary

Environment

Reproduction

Activity

walcz-de commented on Jul 22, 2026

imrehg commented on Jul 22, 2026

herczy commented on Jul 22, 2026

ascotai commented on Jul 23, 2026

cezn commented on Jul 23, 2026

liminfei-amd commented on Jul 24, 2026

repudi8or commented on Jul 24, 2026

AIwork4me commented on Aug 18, 2026

BoobooBubberson commented on Aug 19, 2026

Failure mode A: single-request long-context corruption (no -np, no --kv-unified)

Failure mode B: GGML_CUDA_NO_PINNED=1 → gfxhub page faults + hard hang

Verified workaround: neutralize the commit

Note

glambx commented on Aug 25, 2026

github-actions commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Failure mode A: single-request long-context corruption (no `-np`, no `--kv-unified`)

Failure mode B: `GGML_CUDA_NO_PINNED=1` → gfxhub page faults + hard hang