Eval bug: Decreased perfomance because of input layers on CPU for AMD Strix Halo

Name and Version

version 1 (1705d43)
build with Clang 23.0.0 for Windows

Operating systems

Windows

GGML backends

HIP

Hardware

AMD Strix Halo 128GB

Models

Qwen 3.6 35B MoE Q8

Problem description & steps to reproduce

I was fighting with 30% cpu usage and decreased GPU utilization on AMD Strix halo machine when running Qwen 3.6 35B MoE Q8 generation on ROCm. Vulkan does not have that problem and needs only ~5% cpu to run tg.
GLM found this issue in src/llama-model.cpp around line 1300:

// there is very little benefit to offloading the input layer, so always keep it on the CPU
pimpl->dev_input = { cpu_dev, &pimpl->cpu_buft_list };

This keeps tok_embd on CPU even with -ngl 99, which forces GET_ROWS(tok_embd, inp_tokens) onto the CPU backend at the start of every forward pass. On Stix Halo where CPU and GPU share same DDR5 it hurts a lot.

Possible fix is to mirror the output-layer policy for the input layer when full offload is requested.

Before fix:
41.33 t/s tg (30% CPU usage)

After full GPU offload:
47.24 t/s tg (2% CPU usage)

First Bad Commit

No response

Relevant log output

n/a

Oh this would explain why ROCm is behind compared to Vulkan on token generation speeds. Good find!

Why would this affect ROCm but not Vulkan? It also runs on CPU for Vulkan then.

Dunno, different buffer management? Might be a similar issue as to how --mmap is unusable on ROCm right now while it works fine on Vulkan.

Sorry im not that yet deep into llm engine programming, but this is what my llm says:
Vulkan does not suffer because its async H2D copy actually works (copy_from_host=true), so 2nd split only adds minimal overhead.
Rocm HIP backend hardcodes copy_from_host=false (with comment 'this change is not beneficial for hip') forcing a fallback path that does sync(source) + sync(dst) + copy per input tensor - 10 inputs * 2 syncs = 20 syncs per token, what kills TG perfomance.

Yeah, makes sense. Basically, it makes sense to only offload input tensors to CPU for non-integrated devices.

Dunno, different buffer management? Might be a similar issue as to how --mmap is unusable on ROCm right now while it works fine on Vulkan.

there are few issues there too, from my attempts to hack around windows memory limits i found that mapping actually does not map but copies memory to gpu area, doubling memory usage basically

Yeah, makes sense. Basically, it makes sense to only offload input tensors to CPU for non-integrated devices.

First step should be to check the ROCm backend's assumption about async copies, not conditional llama.cpp core changes.

Why would you not want to offload input tensors to the device in case of iGPU? The only rationale for not doing that is to save memory, that doesn't apply in case of iGPU systems.

Why would you not want to fix async behaviour of the backend? I'm not saying this should not be looked at, just that first priority for fixes is always within the backend.

I can't reproduce any performance increase from this on Linux with Strix Halo.

@Vasili-Sk

@Vasili-Sk can you post the patches that give you the improved performance measures?

I prepared agent workflow for a friend, but it needs sanity check since made by ai:
https://pastebin.com/dBDRuJku
I will apply these patches step by step against main branch later, since i used strix halo fork for main work.

im still investigating this issue from different sides.
apparently on windows adding --poll 0 for llama-bench alsmost fixes this problem and increases tg1024 40.8->47
applied fixes still contribute to minor boost for Q8 MoE model adding +60pp and +0.9tg
llama-server does not have increased CPU usage because it does additional work and have no cpu load show up, unlike llama-bench

So here's the issue - I've tried adding all the fixes here: https://github.com/pwilkin/llama.cpp/commits/fix-igpu-input-layer/

I even added the missing CUDA kernels so that the entire graph now lives on ROCm - including GET_ROWS.

However, I haven't been able to reproduce any of the performance gains - in fact, the performance has regressed a bit compared to mainline.

which OS?

Linux.

Added all current patches to https://github.com/Vasili-Sk/llama.cpp/tree/win-tg-perfomance against mainstream that verified any benefit
Main issue was --poll 50 in windows for llama-bench caused 30% CPU load that degraded performance down to 41 tg, not visible when running llama-server. Currently it is still behind on pp from strix halo fork, it is under investigation

Patch Decomposition Report — Strix Halo (gfx1151) ROCm on Upstream Master

Test rig

Source: llama.cpp upstream master @ e8f19cc0a (b9996-era), branch win-tg-perfomance
Build: Ninja, ROCm 7.1 (patched SDK), gfx1151
Model: Qwen3.6-35B-A3B (Q8_0 = 35.19 GiB; Q6_K = 27.19 GiB)
Flags: -p 16384 -n 1024 -ngl 99 -ctk q8_0 -ctv q8_0 -fa 1 -b 4096 -ub 2048 -r 3 --mmap 0 --no-host 1 --poll 0
Bench: llama-bench, results saved with git SHA baked into filename.

Patches deemed already-shipped upstream (SKIPPED — no source change)

Patch	Upstream commit	Reason skipped
enable `integrated = prop.integrated` for HIP	`c7d872292` "ggml-cuda : restore prop.integrated on HIP builds (#24233)" (liminfei-amd, 2026-07-16)	Same intent as our strix-fork R11 Anchor 1 — zero-copy path already live on APU.
async H2D `copy_from_host` for HIP	Path REMOVED upstream — `cpy_tensor_async` is now CUDA-to-CUDA only	Superseded by the integrated=true zero-copy path; nothing to apply.

Outcome per state (live)

State	Patch added	Q8 pp16384	Q8 tg1024	Q6 pp16384	Q6 tg1024	Δ tg Q8	Δ tg Q6	Δ pp Q8	Δ pp Q6	Verdict	Commit SHA
0	(baseline upstream, no patches)	1110.55 ±0.59	48.66 ±0.03	954.06 ±1.78	55.74 ±0.12	—	—	—	—	reference	`e8f19cc0a`
1	+R4 (hipDeviceScheduleBlockingSync for gfx1151)	1125.96 ±4.67	48.65 ±0.04	955.40 ±0.12	55.68 ±0.05	−0.02%	−0.11%	+1.39%	+0.14%	NEUTRAL (within ±0.5% noise) → reverted	(none)
2	+R6 (input layer / tok_embd GPU offload)	1138.29 ±2.44	48.89 ±0.05	954.22 ±3.37	55.72 ±0.06	+0.47%	−0.04%	+2.50%	+0.02%	PROFIT on Q8 (pp +2.5%, tg +0.47%); NEUTRAL on Q6_K → committed	`9ac96fe62`
3	+R11 batch gate (copy-to-VRAM for prefill, zero-copy for TG) on top of +R6	1159.24 ±3.05	48.86 ±0.08	991.52 ±5.19	55.66 ±0.06	−0.06%	−0.11%	+1.84%	+3.91%	PROFIT on PP for both quants, neutral TG → committed	`9c4afcf6b`

Cumulative scoreboard (vs baseline upstream, --poll 0)

Quant / Test	Baseline upstream	+R6	+R6 +R11gate	Total Δ vs baseline
Q8_0 pp16384	1110.55	1138.29	1159.24	+4.39%
Q8_0 tg1024	48.66	48.89	48.86	+0.41%
Q6_K pp16384	954.06	954.22	991.52	+3.93%
Q6_K tg1024	55.74	55.72	55.66	−0.14% (neutral)

@Vasili-Sk

@Vasili-Sk — excellent benchmarking work here, exactly the kind of data needed to move this forward.

We're running the same hardware (AMD Strix Halo gfx1151, Windows 11, 32 GB unified RAM) and hit the same CPU-offload performance wall. We've confirmed the baseline upstream behavior on LM Studio's bundled llama-server (v2.25.2 runtime) and are eager to test your win-tg-perfomance branch patches.

Would you be willing to share a compiled Windows binary (llama-server.exe + any required HIP/ROCm DLLs) built from your win-tg-perfomance branch? We'd like to drop it into LM Studio's engine folder and validate the gains on our rig while we wait for these fixes to land upstream and get bundled into a release.

If you'd prefer not to distribute a full build, even a pointer to your build flags/toolchain would help us compile it ourselves. We saw you shared an agent workflow on pastebin — happy to follow a similar guide if one exists for the Windows ROCm build.

Also flagging a related issue we filed today that may be of interest: #26089 (VRAM enumeration returning 0 on Windows Strix Halo when BIOS GPU allocation is set to Auto — affects both ROCm and Vulkan).

Hardware: AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo), 128 GB unified LPDDR5X — ROCm reports 96 GB as GPU VRAM, 32 GB reserved for system.

Model: Laguna APEX compact — 54.4 GB, 3.70 BPW, 118B total / ~8B active MoE.

Backend tested: Lemonade b10236, TheRock ROCm 7.13.0 for gfx1151. Lemonade automatically sets OCL_SET_SVM_SIZE=262144 for this arch.

Observed:

GPU VRAM showed ~26 GB used (Lemonade UI) despite 96 GB available and a 54.4 GB model
Remaining ~28 GB of model loaded into the 32 GB system RAM pool — visible system stutter as a result
TTFT for 22K token prefill: ~5 minutes before Hermes client timed out
Estimated prefill throughput: ~82 t/s

Contrast — Vulkan b10241 on the same hardware/model:

Full 54.4 GB loaded into GPU memory, no system RAM pressure, no stutter
TTFT for same 22K token prefill: 109 seconds
Prefill throughput: 202 t/s average (364 t/s peak at 2K tokens)
Generation: 33.25 t/s

The partial offload to system RAM on the ROCm path appears to be distinct from (or in addition to) the input-layer CPU sync overhead described in the original report. Even with OCL_SET_SVM_SIZE=262144 and TheRock's gfx1151-specific stack, the ROCm allocator appears to be capping GPU allocation well below the available 96 GB for this model size.

@Vasili-Sk

@Vasili-Sk @pwilkin I have confirmed the root cause. the line 1287 in src/llama-model.cpp hardcodes dev_input to CPU , on UMA architectures (Strix Halo, other iGPUs, Radeon 890M) this causes cross-backend synchronization overhead on every forward pass even though CPU and GPU share the same physical memory.

the output layer (dev_output) already uses get_layer_buft_list() which follows -ngl.
what i think a fix for dev_input would be:
when i_gpu_start == 0 (full offload requested), assign dev_input via get_layer_buft_list(0) instead of hardcoding CPU. I think this will handle the offloads on discrete GPUs.

@sanjayahari

@sanjayahari — this matches exactly what we're seeing on AMD Ryzen AI MAX+ 395 (Strix Halo gfx1151), Windows 11, 32 GB unified RAM, using LM Studio's bundled llama-server (v2.27.1 runtime).

Baseline TG on our hardware via the OpenAI-compatible API (Qwen3.5-9B Q6_K, 400 tokens):

ROCm 2.27.1: ~27 tok/s
Vulkan 2.27.1: ~27 tok/s (parity after the prop.integrated fix)

The \dev_input\ hardcode to CPU as the root cause is consistent with what we observed — the gap between ROCm and Vulkan closed in 2.27.x, but both backends are likely still paying the cross-backend sync cost you describe on every forward pass.

We can reproduce on Windows with this exact hardware class and are happy to test a compiled build or a patch against 2.27.1 if you or @Vasili-Sk can share one. We have a working build environment and can run llama-bench against the fix to provide before/after numbers.

(Related: we also filed lmstudio-ai/lms#589 tracking the VRAM enumeration failure on this hardware — same root platform, different symptom.)

Adding a data point with a Vulkan control arm on the same machine, which may help separate the input-layer policy from HIP-specific overhead.

Setup: Strix Halo (gfx1151), Linux, Docker. Same model on both arms (bartowski/Ornith-1.5-35B-A3B Q4_K_M, a 35B-A3B MoE), same build b10666-4e97ac86e, same per-slot context (131072), -ngl 999 on both. Official images server-rocm-b10666 and server-vulkan-b10666. Measured via /completions, cache_prompt=false, temp=0, one arm at a time.

CPU time per generated token, as a cgroup usage_usec delta across a 300-token generation:

backend	tg	CPU ms/token	busy cores
Vulkan	47.9 t/s	8.25	0.39
ROCm	41.3 t/s	45.49	1.86

5.5x more CPU time per token on ROCm. Since the llama-model.cpp policy this issue describes is backend-agnostic, both arms should pay the input-layer GET_ROWS cost equally — so a difference this large suggests most of the ROCm CPU load is HIP runtime synchronisation (the sustained ~1.9 busy cores while the GPU is the bottleneck look like spin-waiting), with the input layer being a smaller, shared component on top.

That doesn't argue against the fix proposed here — it would help both backends — but it may mean the CPU usage reported in the OP has two independent contributors.

Throughput and power during generation (400 tokens, GPU clock/power sampled at 0.2 s):

backend	tg	sclk mean	busy%	package power
Vulkan	47.8 t/s	2624 MHz	83%	101.1 W
ROCm	40.3 t/s	2663 MHz	85%	115.9 W

Worth noting the iGPU clocks slightly higher under ROCm at comparable occupancy, so on this shared-TDP part the CPU spin is not visibly starving the GPU of clock headroom — at equal clock and equal busy% ROCm just yields fewer tokens. The ~15 W package delta lines up with the ~1.5 extra busy cores. Net efficiency is 0.47 vs 0.35 tokens/joule.

One coincidence that may or may not mean anything: the 41.33 → 47.24 t/s reported in the OP brackets our two arms almost exactly (ROCm 41.3, Vulkan 47.8). If that holds up, the fix here would put ROCm generation roughly where Vulkan already is on this hardware.

Caveats: both containers were resident simultaneously (~20 GB mlock'd each), so absolute numbers are depressed by memory pressure; the relative comparison should hold since the idle arm was not computing. Single runs, not averaged over repeats. Happy to re-run anything specific, or with only one container loaded, if that would be useful.

docs(adr): record ROCm A/B measurements in ADR 0003

Linux data point with the fix gated to integrated GPUs only (ggml_backend_dev_type == GGML_BACKEND_DEVICE_TYPE_IGPU and full offload), which addresses the "detrimental to non-UMA" concern in #27426 by construction:

diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index e195f50d0..661c3da0f 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -1533,8 +1533,17 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
     };
 
     // assign the input layer
-    // there is very little benefit to offloading the input layer, so always keep it on the CPU
+    // there is very little benefit to offloading the input layer, so keep it on the CPU --
+    // except on integrated GPUs with full offload: CPU and GPU share one memory, so the
+    // CPU copy saves nothing, while the CPU GET_ROWS costs a cross-backend sync every
+    // forward pass (#25700). Discrete GPUs are unaffected by construction.
     pimpl->dev_input = { cpu_dev, &pimpl->cpu_buft_list };
+    if (act_gpu_layers > n_layer_all) {
+        const auto first = get_layer_buft_list(0);
+        if (first.dev != cpu_dev && ggml_backend_dev_type(first.dev) == GGML_BACKEND_DEVICE_TYPE_IGPU) {
+            pimpl->dev_input = first;
+        }
+    }
 
     // assign the repeating layers to the devices according to the splits
     pimpl->dev_layer.resize(n_layer_all);

Strix Halo (395, gfx1151), Linux 7.2, master ec92815, llama-server /completion, 3×300 tokens:

HIP	master	input layer on iGPU
Qwen3.6-35B-A3B IQ4_XS tg	49.05 t/s	49.39 t/s
CPU per token (server process)	20.8 ms	20.6 ms
Qwen3.8-27B Q4_K_M tg	11.85 t/s	11.86 t/s

The fix is effective (the ROCm0 buffer grows by 515 MiB, token_embd now on the GPU) and PPL is unchanged, but on current master there is no measurable gain. Possibly #28604 (host buffers disabled on iGPUs) removed the sync cost the original reports saw. The high CPU usage on HIP remains either way (~1 core at 49 t/s vs 6.6 ms/token = 0.4 core on Vulkan, same model), so that part isn't the input layer. Can someone who saw the 41→47 t/s gain re-check on a build after #28604?

Navigation Menu

Repository navigation

Eval bug: Decreased perfomance because of input layers on CPU for AMD Strix Halo #25700

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

11 remaining items

Patch Decomposition Report — Strix Halo (gfx1151) ROCm on Upstream Master

Patches deemed already-shipped upstream (SKIPPED — no source change)

Outcome per state (live)

Cumulative scoreboard (vs baseline upstream, --poll 0)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Eval bug: Decreased perfomance because of input layers on CPU for AMD Strix Halo #25700

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

Mushoz commented on Jul 15, 2026

0cc4m commented on Jul 15, 2026

pwilkin commented on Jul 15, 2026

Vasili-Sk commented on Jul 15, 2026

pwilkin commented on Jul 15, 2026

Vasili-Sk commented on Jul 15, 2026

0cc4m commented on Jul 15, 2026

pwilkin commented on Jul 15, 2026

0cc4m commented on Jul 15, 2026

11 remaining items

0cc4m commented on Jul 15, 2026

pwilkin commented on Jul 15, 2026

Vasili-Sk commented on Jul 15, 2026

Vasili-Sk commented on Jul 16, 2026

pwilkin commented on Jul 16, 2026

Vasili-Sk commented on Jul 16, 2026

pwilkin commented on Jul 16, 2026

Vasili-Sk commented on Jul 17, 2026

Patch Decomposition Report — Strix Halo (gfx1151) ROCm on Upstream Master

Patches deemed already-shipped upstream (SKIPPED — no source change)

Outcome per state (live)

Cumulative scoreboard (vs baseline upstream, --poll 0)

cflury4784 commented on Jul 24, 2026

WareWolf-MoonWall commented on Aug 7, 2026

sanjayahari commented on Aug 9, 2026

cflury4784 commented on Aug 10, 2026

hgnnmnn commented on Aug 28, 2026

huppiflupp commented on Sep 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions