Eval bug: qwen4exp / deepseek-v4 abort at first decode on Vulkan (RADV, gfx1151)

[shikanime]

Name and Version

llama.cpp 0.4.0, revs 18443257 and 96ffdc4. My build pin/config: pkgs/llama-cpp/default.nix (source rev + patches), and shikanime-labs/machines#1357 is the backend flip this issue motivated.

Operating systems

Linux (NixOS 26.11.20260829.d2f6794).

GGML backends

Vulkan (RADV) — fails as below. ROCm (HIP/rocblas), same rev, same GPU — qwen4exp passes (model loads and decodes: 8-token completion, ~19 tok/s, no asserts, no device loss). deepseek-v4 on ROCm: untested.

Hardware

AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 128 GB unified LPDDR5X (UMA) (Minisforum MS-S1 MAX)

Models

• qwen4exp: unsloth/Qwen3.8-Flash-Next-GGUF, revision UD-Q4_K_XL
• deepseek-v4: lmstudio-community/DeepSeek-V4-Flash-0731-GGUF, revision MXFP4

Problem description & steps to reproduce

On AMD Strix Halo (Radeon 8060S iGPU, gfx1151, RADV), any decode of qwen4exp (Qwen3.8-Flash-Next) or deepseek-v4 (DeepSeek-V4-Flash) fails. The same build and GGUF of qwen4exp runs cleanly on the same GPU with the ROCm backend (deepseek-v4 not yet retried on ROCm).

Failure modes on Vulkan/RADV:

1. Full offload (--n-gpu-layers 999, both archs): weights load to 100%, then the first llama_context::decode aborts:

   /build/source/ggml/src/ggml-backend.cpp:283: GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds") failed
   

2. Partial offload (deepseek-v4, --n-gpu-layers 48): the assert is bypassed, weights load, and the first graph compute loses the device:

   terminate called after throwing an instance of 'vk::DeviceLostError'
     what():  vk::Queue::submit: ErrorDeviceLost
   

   Worker process dies with SIGSEGV (exit 139). Backtrace: vk_queue_handle_unsynchronized::submit → ggml_backend_graph_compute → rpc_server::graph_compute.

Backtrace of the assert path (router child, verbatim frames): ggml_abort → ggml_backend_tensor_get_async (ggml-backend.cpp:283) → llama_context::decode → llama_decode → common_init_from_params → server_context_impl::load_model.

Reproduced identically: over RPC (Vulkan device held by a worker) and node-local; with and without --no-warmup; with and without --fit off; on two revs (above), the newer of which equals master at test time.

Older-arch models (qwen3.8-27b) run continuously on the same RADV device without incident, so the defect is specific to these new-arch graphs.

The Kubernetes manifests: shikanime-labs/manifests/apps/llama-cpp and shikanime-labs/manifests/apps/llama-cpp-rpc

Steps to reproduce:

llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL --n-gpu-layers 999 -c 2048
# send any single completion; the ggml-backend.cpp:283 abort fires at first decode

DeviceLost variant: lmstudio-community DeepSeek-V4-Flash MXFP4 GGUF with --n-gpu-layers 48.

First Bad Commit

Unknown. Both tested revs fail; the newer (96ffdc4) equals master HEAD at test time, so this reproduces on current master.

Relevant log output

Assert path (full offload, first decode):

/build/source/ggml/src/ggml-backend.cpp:283: GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds") failed

DeviceLost path (deepseek-v4, --n-gpu-layers 48, worker SIGSEGV exit 139):

terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Queue::submit: ErrorDeviceLost

Related but distinct (checked before filing)

• Eval bug: draft-mtp DeviceLost during *prompt* on AMD RADV — common_speculative_process runs llama_decode(ctx_dft) after every prefill ubatch #27306: RADV/gfx1151 DeviceLost, but mid-prefill under draft-MTP draft decodes, process survives as a zombie. Mine: first decode, no speculative decoding, process dies.
• qwen4exp (PR #27742): multi-segment prompts degrade to '//////' on gfx1151 — deterministic repro #27797: qwen4exp on gfx1151 outputs garbage (//////), no crash.
• Eval bug: qwen4exp QSA indexer top-k selects a different cell set each run on CUDA (CUB DeviceTopK over tied block scores) #28497: QSA indexer top-k nondeterminism on CUDA, wrong results, no crash.
• Suspect area consistent with unmerged new-arch op work: deepseek-v4 CPU ops (ggml: add DeepSeek V4 hyperconnection + KV ops (CPU) #23122) and qwen4exp QSA decode reworks. At full offload the failure surfaces as a host-side tensor read OOB; at mixed placement it surfaces as a malformed submit that loses the device.

[0cc4m]

What's the command you launch llama-server with?

[shikanime]

This is the configuration I tried to run, among the multiple combination:

hf = unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL
n-gpu-layers = 999
no-warmup = true
rpc = llama-cpp-rpc.shikanime.svc.cluster.local:50052

and llama-cpp-rpc doesn't have any particular flags, just the host, port and cache.

[shikanime]

Additional data: node-local HIP (ROCm) reproduction matrix — RPC ruled out, artifact integrity ruled out

Setup: same GPU as the issue (Radeon 8060S / gfx1151, 128 GB UMA), llama.cpp 0.4.0 (96ffdc4). Local patches touch only ggml-rpc.cpp accept-loop behavior and metrics auth; runs 3–4 remove RPC from the path entirely.

All runs: Qwen3.8-Flash-Next, --n-gpu-layers 999, --no-warmup, single 8-token completion. Identical abort every time: ggml-backend.cpp:283 "tensor read out of bounds" at the first llama_context::decode, via ggml_backend_tensor_get_async (the output-tensor read at the end of decode).

#	Backend	GGUF	RPC	mmproj	Result
1	HIP	unsloth UD-Q4_K_XL (pre-restructure shards)	yes	loaded	abort, first decode
2	HIP	lmstudio-community Q4_K_M	yes	loaded	abort, first decode
3	HIP	lmstudio-community Q4_K_M	no	loaded	abort, first decode
4	HIP	lmstudio-community Q4_K_M	no	--no-mmproj	abort, first decode

Runs 1–2 were initially suspected to be artifact poisoning (unsloth's restructured per-quant subdirectories serve a metadata-only first shard — n_tensors=0, verified via HTTP Range); the intact lmstudio GGUF aborts identically, so artifact integrity is not the discriminator.

Observations:

• Transport- and placement-independent: aborts over RPC, node-local, with and without the vision projector.
• CUDA and Metal users on adjacent revs decode this model fine (PR models: Qwen3.8-Flash-Next MTP #28243 thread), pointing at a gfx1151-specific defect spanning both the Vulkan and HIP backends.
• Working hypothesis: for the qwen4exp graph on gfx1151, the backend allocates an output/logits tensor smaller than the offset + size the host requests at the end of the first decode — an allocation-size mismatch rather than an RPC bug.

[0cc4m]

Not sure what exactly is going on there, but the Qwen model is very close to your memory limit, maybe over it depending on how much of it you made available to your GPU. The Deepseek model is definitely far over your memory size. It may run with mmap, but slowly. I can tell you that I can run bartowski's Qwen 3.8 Flash Next IQ4_XS, which fits fully into memory, on my 128GB Strix Halo.

[shikanime]

Root cause found (gfx1151, llama.cpp 0.4.0): the failing pods ran with LLAMA_ARG_EMBEDDINGS=true in the container environment. That flag restricts the server to embedding-only use; a generative model still loads, but the first decode reads the output tensor out of bounds and trips this exact assert (ggml/src/ggml-backend.cpp:283).

Reproduction matrix (Qwen3.8-Flash-Next Q4_K_M, 3 shards, HIP/gfx1151, node-local, no RPC):

• no env overrides, ctx 2048: PASS (full completion)
• LLAMA_ARG_EMBEDDINGS=true alone, otherwise identical: ASSERT on first decode, same signature
• same plus flash-attn on + q8_0/q8_0 KV cache: PASS
• in-container with the variable inherited: ASSERT; same command under env -u LLAMA_ARG_EMBEDDINGS: PASS

The earlier RPC and artifact-integrity eliminations stand, and flash-attn + quantized KV are exonerated. On our side the fix was dropping the env var — the router was accidentally configured for embedding mode while serving generative models. A clearer error than an offset assert when a generative model is loaded under embeddings-only restrictions might still be worthwhile, but no llama.cpp defect is implicated. Thanks for the pointers.