Skip to content

Regression after c7d8722: llama-server generates infinite "/" tokens in agent mode #26209

Closed as not planned
Closed as not planned
Regression after c7d8722: llama-server generates infinite "/" tokens in agent mode#26209
Labels
bug-unconfirmedstale

Description

@puzhengwu
puzhengwu
opened on Jul 28, 2026

Name and Version

llama-server --version
version: 10156 (91f8c9c)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

HIP

Hardware

AMD MAX+ 395 gfx1151 + rocm7.14

/mnt/data/llama.cpp/rocm/bin/llama-server -m /mnt/data/models/Qwen3.5-4B-GGUF/Qwen3.5-4B-UD-Q4_K_XL.gguf -mm /mnt/data/models/Qwen3.5-4B-GGUF/mmproj-F16.gguf --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 -a Qwen3.5-4B --chat-template-kwargs '{"enable_thinking": false}' -c 262144 --host 0.0.0.0 --port 9003

/mnt/data/llama.cpp/vulkan/bin/llama-server -m /mnt/data/models/ornith-1.0-35b-Q4_K_M.gguf -mm /mnt/data/models/Qwen3.6-35B-A3B/mmproj-F16.gguf -a Ornith -c 262144 --host 0.0.0.0 --port 9003

/mnt/data/llama.cpp/rocm/bin/llama-server -m /mnt/data/models/ornith-1.0-35b-Q4_K_M.gguf -mm /mnt/data/models/Qwen3.6-35B-A3B/mmproj-F16.gguf -a Ornith -c 262144 --host 0.0.0.0 --port 9003

Models

No response

Problem description & steps to reproduce

Whether running on ROCm or Vulkan, executing an opencode-like agent results in infinite output of "/" characters regardless of the input prompt.

First Bad Commit

No response

Relevant log output

Logs 




Activity
  1. puzhengwu
    added 
    bug-unconfirmed
     on Jul 28, 2026
  2. puzhengwu commented on Jul 30, 2026 
    @puzhengwu
    puzhengwu
    on Jul 30, 2026
    Author
    rocm/bin/llama-server --version
    
version: 9844 (6c5de1c)
    
I rolled back llama.cpp to a version from one month ago, and the issue disappeared, indicating that it was introduced in recent versions.
  3. puzhengwu commented on Aug 3, 2026 
    @puzhengwu
    puzhengwu
    on Aug 3, 2026
    Author
    After multiple tests, it has been confirmed that this issue was introduced in commit c7d8722.
  4. puzhengwu
    changed the title [-]Eval bug: Whether running on ROCm or Vulkan, executing an opencode-like agent results in infinite output of "/" characters regardless of the input prompt.[/-] [+]Regression after c7d8722: llama-server generates infinite "/" tokens in agent mode[/+] on Aug 3, 2026
  5. coderashed commented on Aug 4, 2026 
    @coderashed
    coderashed
    on Aug 4, 2026
    Bisected this on a Strix Halo box (gfx1151, ROCm 7.2.2, debian 13, built with GGML_HIP=ON GPU_TARGETS=gfx1151):
    

    5839ba3  CUDA: dedup MoE gate/up activation quantization (#25441)    GOOD
    
c7d8722  ggml-cuda : restore prop.integrated on HIP builds (#24233)  BAD
    

    So the commit in the title is confirmed as the first bad one. fwiw I don't think agent mode matters, it's just prompt length. Anything over ~2k tokens that gets split across multiple llama_decode calls degenerates into one token repeated forever. The actual token varies with the model vocab: gemma4 26b gives me , a deepseek v4 model gives token 14 which renders as an empty string, qwen apparently gives /. Probably the same thing as #26239 and #25436.
    

    If I request n_probs the top candidates are ids 0-4 with null logprobs, i.e. the logits are NaN. That also explains why it never stops, EOG can't get sampled and the webui defaults to n_predict=-1.
    

    Things I ruled out on the same machine: happens with plain f16 KV cache, with --flash-attn off, and with GGML_CUDA_DISABLE_GRAPHS=1. -b 1024 doesn't move the threshold down (a 1.5k prompt split into two decode calls is fine, >2k is not), which lines up with the -b 6144 -ub 6144 observation in #26239 and it needs both the batch split and >2k total. CPU-only is clean at every size. Also once a long request has corrupted, following ~1.9k requests on the same server often come back garbage too, short ones recover.
    

    c7d8722 makes supports_buft() accept cuda_host buffers on integrated GPUs again. That flag was forced off in #16308 in the first place because of corrupted output on Jetson Orin (#15034), which is also an integrated device, so I suspect this is the same host-buffer coherency issue that workaround was papering over. Locally reverting to unconditional integrated = false fixes everything here clean output up to 9k-token prompts with f16 and quantized KV.
    

    I stumbled on this and my branch isn't clean cut off of master (it has KVarN implemented.) So I will test a clean patch using my Strix Halo later today to verify.
  6. simongonzalezdc commented on Aug 19, 2026 
    @simongonzalezdc
    simongonzalezdc
    on Aug 19, 2026
    Independent repro on a second Strix Halo machine (gfx1151, ROCm 7.2.4, GGML_HIP build), with a datapoint that widens the blast radius: the multimodal path hits this on every request on integrated GPUs — image embeddings push the prompt past the ~2k-token split threshold, so it's not just long agent transcripts.
    

    Setup: Qwen3.8-27B Q4_K_XL + mmproj-F16 (27-layer ViT), llama-server, MTP + capped ngram spec decode, q4_0 KV.
    

    Before the revert (build contains c7d8722):
    

      

    • 6/6 vision requests (image + one-line question) degenerate to / repeated forever, finish_reason=length. Solid colors, gradient, chart screenshot — identical failure regardless of image. Raw log: vision-v5-results.log
      • 

    • Short text-only prompts on the same server, same session: clean (HumanEval 30-problem subset 28/30 — those prompts sit well under 2k tokens).
      • 

    

    After reverting c7d8722 only (same quant, same mmproj, single-commit delta):
    

      

    • vision: 5/6 correct, 0.3–1.8 s per answer, all finish_reason=stop (the one miss is a grading artifact — "stacked bar chart" answering a describe-this-chart question). Raw log: vision-v5-after-fix.log
      • 

    • text regression: 28/30 again, same two failing problems (HumanEval/50, /145). Raw log: bench-post-fix-regression.log
      • 

    

    One red herring worth flagging for triage: because every vision request failed while text at the same quant was fine, we initially read this as "Q4 weights can't process F16 vision embeddings." The quant was irrelevant — the same Q4 GGUF serves vision cleanly once the buffer regression is reverted.
    

    Build disclosure: fork of master @ 9d57ce4 (Aug 15) carrying #27083 + #26653 cherry-picks + this revert; the before/after pair above differs only by the revert. Master still accepts host buffer types on integrated devices as of today (ggml_backend_cuda_device_supports_buft in ggml-cuda.cu).
    

    Possibly the same root: #25992 (gfx1151, bisected to c7d8722, -np 4 --kv-unified cross-request leakage) and #23577 (repeated //// with MTP on CUDA).
    

    Raw data + conditions: https://github.com/KyaniteLabs/qwen38-27b-strix-halo/tree/main/results/vision-2026-08-19
  7. simongonzalezdc commented on Aug 20, 2026 
    @simongonzalezdc
    simongonzalezdc
    on Aug 20, 2026
    Follow-up to our vision repro above, with a datapoint that may matter more for triage: this regression also silently corrupts long-context retrieval on integrated GPUs — text-only, no vision involved — in a way that looks like model behavior.
    

    Controlled A/B on the same box (Strix Halo, gfx1151, ROCm 7.2.4, GGML_HIP), Qwen3.8-27B Q4_K_XL, 262k ctx, q4_0 KV, temp 0:
    

      

    • Harness: synthetic ~198k-token haystacks (5800 paragraphs, fixed seed → byte-identical prompts across arms), one "access code" needle planted at a depth, question asks for the exact code, max_tokens 40.
      • 

    • Pre-revert build (contains c7d8722): 1/6 depths retrieved. The other five return degenerate attractors — the literal string ok, or an unrelated markdown risk-template repeated to finish_reason=length (same attractor text across different needles).
      • 

    • Post-revert (only delta = reverting c7d8722 / integrated = false): 6/6 depths retrieve the exact code, every cell finish_reason=stop.
      • 

    

    Anchor pair (token-identical): one needle at 50% depth, usage.prompt_tokens = 198228 in both eras, same seed and flags — pre-revert answers ok; post-revert answers COBALT-8835-DELTA.
    

    Why we nearly missed it: short-prompt text was unaffected — code benchmarks (<2k-token prompts) scored normal on the broken build while deep-context needle tests failed on the same server, and we initially attributed the failures to the model's architecture (it's a GDN/linear-attention hybrid, which made a "decay" story plausible). Anyone running long-context evaluation on iGPU builds since c7d8722 may be measuring this bug rather than their model.
    

    Depth-by-depth table + raw logs: https://github.com/KyaniteLabs/qwen38-27b-strix-halo/tree/main/results/needle-format-2026-08-19
    

    Build disclosure: fork of master @ 9d57ce4 carrying #27083 + #26653 cherry-picks + this revert; the A/B pair above differs only by the revert.
  8. simongonzalezdc commented on Aug 21, 2026 
    @simongonzalezdc
    simongonzalezdc
    on Aug 21, 2026
    Independent validation of #25863 on the affected hardware class (Strix Halo, gfx1151, ROCm/HIP), in case it helps merge confidence:
    

    Paired battery vs a locally c7d8722-reverted build - identical model (Qwen3.8-27B Q4_K_XL + mmproj F16), identical server flags (262144 ctx, KV q4_0, flash-attn, single slot), identical fixtures; the binary was the only delta.
    

      

    • Real-UI screenshot QA (6 Chrome-rendered screenshots: bar chart, data table, form, terminal, progress bars, line chart; keyword grading): PR build 6/6, reverted build 6/6, answers byte-identical (e.g. "Beta", "312", "503", "up").
      • 

    • Long-context needle retrieval (10k / 12k / 16k token haystacks, arbitrary alphanumeric code planted at 50%, exact-match grading): PR build 3/3 exact, reverted build 3/3 exact, zero degenerate outputs, all finish_reason=stop. Server-reported prompt token counts identical across arms (10013 / 12018 / 16015), so the fixtures were byte-identical between arms.
      • 

    

    On gfx1151 the PR matches the c7d8722-reverted behavior on both the vision class and the text-only long-context class described in this issue.
    

    Raw logs, harness, and per-launch server logs: https://github.com/KyaniteLabs/qwen38-27b-strix-halo/tree/main/results/pr25863-validation-2026-08-21
  9. Puddles-sh commented on Aug 30, 2026 
    @Puddles-sh
    Puddles-sh
    on Aug 30, 2026
    Independent reproduction on current master (b10687), pure llama.cpp
    

    Confirming this on a Strix Halo iGPU (gfx1151) with the repository tools only — no Ollama in the
    
loop. The corruption is specific to llama-server's first large-prompt eval; single-shot
    
llama-completion on the same build/GPU/model/prompt is unaffected, which lines up with the
    
ROCm_Host write-after-read described here.
    

    Environment
    

      

    • llama.cpp: version 0.3.0-dev (build 10687, commit c841aeeb8)
      • 

    • Backend: ROCm / HIP (libggml-hip); device gfx1151 (Radeon 8060S iGPU, Ryzen AI Max+ 395)
      • 

    • Kernel: 6.18.9; model: a qwen3-8b Q4_K GGUF, -c 16384 (not model-specific — a gemma-class
      
model corrupts the same way)
      • 

    

    Symptom
    

    With a large (~12k-token) system prompt, the first llama-server /completion request returns
    
empty / garbage / echoed input instead of a coherent answer. The GPU is genuinely in use
    
(prompt eval ~1.2k tok/s, not a CPU fallback).
    

    Reproduction (fully synthetic prompt — indexed dummy rules + a named needle key)
    

    # 1. build the prompt (regenerates at any size)
python3 gen_prompt.py --target-tokens 12000 > prompt.txt

# 2. CORRECT — single-shot completion (no server buffer reuse)
llama-completion -m MODEL.gguf -f prompt.txt -c 16384 -b 1024 -ub 1024 \
  --temp 0 --top-k 1 -s 0 -n 40 -fa auto -ngl 999 -no-cnv
#    -> recalls the needle (VELVET-ORBIT-2846)

# 3. CORRUPTED — llama-server, request fired IMMEDIATELY after /health
llama-server -m MODEL.gguf --host 127.0.0.1 --port 8099 -c 16384 -b 1024 -ub 1024 \
  -ngl 999 -fa auto --keep 4 --context-shift --no-webui &
# wait only until /health returns, then immediately:
curl -s http://127.0.0.1:8099/completion \
  -d '{"prompt":"<contents of prompt.txt>","n_predict":40,"temperature":0,"top_k":1,"seed":0,"cache_prompt":true}'
#    (ready-to-run body.json version in the zip's commands.md)
#    -> empty / garbage; the needle is NOT recalled

# 4. CORRECT — identical server, but sleep ~3s after /health before the first request
#    -> recalls the needle

    

    Timing is the trigger
    

    The fault only appears when the first request lands in the window right after /health reports
    
ready, i.e. before the async GPU/buffer state settles on the first eval:
    

    


    
path
result

    
    


    
llama-server, request immediately after /health
corrupted (3/3, ×2 trials)

    

    
llama-server, ~3s delay after /health
correct (3/3, ×2 trials)

    

    
single-shot llama-completion
correct (5/5)

    
    

    

    12/12 consistent. Because single-shot completion (one prompt eval, no server-side buffer reuse) is
    
always correct while the server path corrupts on the first eval, this points at the server's
    
prompt-token-buffer handling rather than the GPU math or the grammar.
    

    Suspected cause
    

    Consistent with #24233 re-enabling direct ROCm_Host compute on iGPUs → the write-after-read this
    
issue describes. Offered as consistent, not a claimed bisect. Happy to test PR #25863 once a
    
build is available.
    

    first observed via Ollama 0.33.2 (which bundles a different llama.cpp commit, d222767)
    
— it reproduces identically, but the above is pure llama.cpp so Ollama is not required.
    

    g6e-repro.zip
  10. puzhengwu commented on Aug 31, 2026 
    @puzhengwu
    puzhengwu
    on Aug 31, 2026
    Author
    export LD_LIBRARY_PATH=/opt/rocm/core-10.0/lib:/opt/rocm/core-10.0/lib64:$LD_LIBRARY_PATH
    
➜  ~ /mnt/data/llama.cpp/rocm/bin/llama-server --version
    
version: 0.3.0-dev (build 10713, commit 557614e)
    
built with GNU 13.3.0 for Linux x86_64
    

    diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
    
index a6fc655c4..55da02d24 100644
    
--- a/ggml/src/ggml-cuda/ggml-cuda.cu
    
+++ b/ggml/src/ggml-cuda/ggml-cuda.cu
    
@@ -303,7 +303,7 @@ static ggml_cuda_device_info ggml_cuda_init() {
    
info.default_tensor_split[id] = total_vram;
    
total_vram += device_vram;
    
#if defined(GGML_USE_HIP)
    

      

    • 

         info.devices[id].integrated = prop.integrated;

      

      • 

    

      

    • 

         info.devices[id].integrated = false;//prop.integrated;

      

      • 

    

    #else
    
info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
    
#endif
    
Image
  11. github-actions
    added 
    stale
     on Oct 1, 2026
  12. roverdrubber
    added a commit that references this issue on Oct 15, 2026
    cuda/hip: compute quantized matmuls in F32 on AMD, and port upstream …
    3372ce5
  13. github-actions commented on Oct 16, 2026 
    @github-actions
    github-actionsbot
    on Oct 16, 2026 – with GitHub Actions
    Contributor
    This issue was closed because it has been inactive for 14 days since being marked as stale.
  14. github-actions
    closed this as not plannedon Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Metadata
Assignees
No one assigned
    Labels
    bug-unconfirmedstale
    Type
    No type
    Projects
    No projects
      Milestone
      No milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions