[amangupta-tether]

Overview

Prefill speedups for Strix Halo (Ryzen AI MAX+ 395, gfx1151), mainly on Vulkan, measured on Qwen3.8 Flash-Next (qwen4exp) and Qwen3.8 27B (qwen35). This covers the work that pays off at short context; attention at depth (QSA selected-key attention, multi-row prefill flash attention) will follow in a separate PR.

Three kinds of change, one commit per optimization:

• Upstream llama.cpp commits that temp-11018 (last merged Sep 17) was missing. Flash-Next prefill doubles on both ROCm and Vulkan with the first five commits, most likely from llama_prefetch_rows (llama: llama_prefetch_rows ggml-org/llama.cpp#29599): upstream built at the merge base c9a5eeeb3 runs at fabric's speed, and the gain appears between that commit and Oct 1, the window that contains llama: llama_prefetch_rows ggml-org/llama.cpp#29599. The commit was not measured on its own. Also the int8 coopmat1 matmul for RDNA3/4 (vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 ggml-org/llama.cpp#27952), the 512-expert MUL_MAT_ID row-id hoist (vulkan: raise the hoisted row-id limit for mul_mat_id from 256 to 512 experts ggml-org/llama.cpp#28501), the IQ3_S/IQ4_XS Vulkan MMQ kernels (vulkan: add IQ3_S MMQ matmul kernels ggml-org/llama.cpp#28822, vulkan: add IQ4_XS MMQ/MMV matmul kernels ggml-org/llama.cpp#28415), the qwen4exp hc_post gate fusion (vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain ggml-org/llama.cpp#29520) and top-k MoE fusion (cuda: top-k MoE should always fire ggml-org/llama.cpp#28432).
• Vulkan changes from halo-box/strix-llama.cpp, the RDNA 3.5 fork, none of them in upstream: tiled transpose for the delta-net conv-state concat, the direct GDN conv, vec4 stores and the MoE router-weight epilogue in the int8 MMQ, the short-row RMS norm fusions, and the small-M operand swap. The cherry-picks keep their original authors. The two GDN conv commits (8ee3e0848, e092d0386) are by Nathan Wilson (strix f4d6f9e8d, e24924856, 2dfe8e521), ported by hand.
• One new change: on RDNA 3.5 with HIP >= 7.15 (ROCm 10.0), dense Q8_0 matmuls with more than 512 rows use dequantization + hipBLAS instead of MMQ (e10627480, narrowed in f2df7a3c9). With ROCm 7.2.2 hipBLAS is slower for these matmuls, so older HIP keeps MMQ.

ggml-org#29182 is picked and then reverted by ggml-org#29936, as upstream did. The revert is neutral here (Flash-Next pp2048 672.5 vs 669.6 t/s) and is kept to match upstream.

Additional information

Results

GMKtec EVO-X2: Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151), 64 GiB GPU carve-out, Mesa 26.0.8 RADV, ROCm 10.0. llama-bench -ngl 999 -fa 1 -b 2048 -ub 2048 -p 2048 -n 128; Flash-Next adds -lzm on.

Vulkan, pp2048 at empty context (t/s):

model	temp-11018	this PR	upstream a0e6dc738	strix-llama.cpp deb5e2f
Qwen3.8 Flash-Next UD-IQ4_XS	233	684	486	730
Qwen3.8 27B UD-Q8_K_L	234	328	267	335
Qwen3.8 27B Q4_K_M	248	305	253	309

ROCm at empty context (t/s), same builds, -p 2048 -n 512, 2 runs:

model	test	temp-11018	this PR	upstream a0e6dc738	strix-llama.cpp deb5e2f
Qwen3.8 Flash-Next UD-IQ4_XS	pp2048	256.1	525.2	518.8	1189.7
	tg512	21.50	22.95	22.98	27.23
Qwen3.8 27B Q4_K_M	pp2048	309.1	308.0	320.7	349.5
	tg512	12.31	12.31	12.30	12.42

On ROCm, this PR brings Flash-Next level with upstream, through the same first five commits. The 27B Q4_K_M is unchanged: its matmuls are Q4_K/Q6_K, which the new Q8_0 hipBLAS rule (e10627480) does not cover. With HIP 7.15 that rule takes Qwen3.8 27B UD-Q8_K_L from 310 to 354 t/s. On a ROCm 7.2.2 box it measured 317 -> 288 t/s, so it only applies from HIP 7.15 on. strix-llama.cpp's ROCm lead comes from its BF16 WMMA matmul (MMB) and fused qwen4exp HIP kernels, which are not part of this PR. Decode is unchanged or slightly faster in every case.

Per commit, Vulkan pp2048 unless noted. Each number comes from the same build with and without the commit, or the commit's env switch off/on, interleaved:

commit	change	Flash-Next	27B Q8
6a349d6ef	llama_prefetch_rows (ggml-org#29599), with the four commits before it	233 -> 429 (ROCm 257 -> 528)	—
e10627480	RDNA3.5 Q8_0 -> hipBLAS above 512 rows (ROCm, HIP >= 7.15 after f2df7a3c9)	unchanged	310 -> 354
8ee3e0848	tiled concat transpose	432 -> 484	240 -> 258
e092d0386	direct GDN conv	483 -> 491	260 -> 261
f964031f4	int8 coopmat1 matmul (ggml-org#27952)	492 -> 520	262 -> 320
105e1ad7f	int8 MMQ vec4 stores	519 -> 535	321 -> 326
1be77e089	router weights in the MMQ MUL_MAT_ID epilogue	540 -> 557	—
6f69f4bd1	hc_post gate fusion (ggml-org#29520)	prefill flat, tg 25.94 -> 26.22	—
854e8164d	short-row RMS norm fusions	551 -> 565	flat
672f167ed	512-expert row-id hoist (ggml-org#28501)	575 -> 677	—
2f3fcbe9d	small-M operand swap	670 -> 682 (inject matmul 1805 -> 557 us/call)	flat

Fabric-specific adaptations

• vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 ggml-org/llama.cpp#27952: a non-contiguous quantized src0, which fabric supports by dequantizing it to f16, no longer takes the int8 path. Without this, 3 MUL_MAT cases failed (src1 was quantized to q8_1 for the f16 pipeline). Also AMD_RDNA4 goes into the enum in ggml-vulkan.cpp, and fabric's small-tile warp width (30b7e5162) is kept.
• vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain ggml-org/llama.cpp#29520 and the strix commits target upstream's split Vulkan headers (ggml-vulkan-types.h and the others). Their declarations go into ggml-vulkan.cpp here.
• The small-M swap keeps fabric's wider src1 types (BF16, quantized) on the existing M = 1 route; only the new transposed route is restricted.
• The strix RMS_NORM_SCALE test is test_rms_norm_scale_short, because fabric already has a test_rms_norm_scale. Fabric's harness has no scheduler-allocation mode, so the strix tests that used it run on the plain allocator.

Testing

On gfx1151, Vulkan0: test-backend-ops MUL_MAT 2123/2123, MUL_MAT_ID 1199/1199, MUL_MAT_ID_FUSION 22/22, CONCAT 226/226, CONCAT_SSM_CONV_SILU 12/12, SSM_CONV 53/53, DSV4_HC_POST 9/9, RMS_NORM 60, RMS_NORM_MUL_MUL 10, RMS_NORM_SCALE 12, RMS_NORM_MUL_ADD 36, L2_NORM 30 and NORM 60, all passing. The env-gated changes were also tested with their gate off.

test-recurrent-state-rollback (qwen35, deepseek4, nemotron-h and kimi-k3 test models) passes on Vulkan0 with GGML_SCHED_DEBUG_REALLOC=1 after 587d94865. On ROCm0: MUL_MAT 2279/2279, TOPK_MOE 416/416 and ARGSORT 46/46.

Perplexity (wikitext, c = 2048):

• Identical per chunk with each change on and off for the concat transpose and direct conv (27B) and the router-weight epilogue (Flash-Next).
• vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 ggml-org/llama.cpp#27952 quantizes activations to q8_1, so 27B PPL changes slightly, and now matches upstream Vulkan chunk for chunk.
• The short-row norm changes the reduction order: 27B PPL 6.2256 -> 6.2192 over 4 chunks.

Not tested: CUDA, Metal, OpenCL and Vulkan on other vendors. The int8 coopmat1 path, the vec4 stores and the router-weight epilogue only engage on RDNA3/RDNA4. The concat transpose, the direct GDN conv, the small-M swap and the row-id hoist apply on every Vulkan device but were only run on RADV. The CUDA-side commits (ggml-org#28432, ggml-org#29599) and the HIP rule were only built and run on gfx1151.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude Code (Claude Opus 5.5) did the cherry-picks, conflict resolution and hand ports, built and ran the benchmarks and test-backend-ops on the EVO-X2, and drafted this description; the commits carry Co-Authored-By trailers.

🤖 Generated with Claude Code

[aegioscy]

Focused review of the two areas in code_review_and_refactoring_instructions.md: two refactoring suggestions and one adjacent CUDA allocation-boundary issue. Static source/index checks only; no GPU build or benchmark was run for this review.

[aegioscy]

Flatten the activation and common top-k suffix. The bias branches repeat the reshape/argsort/view/gather sequence. Select the activation once, insert the optional ADD between the shared operations, and calculate the retained VIEW index in one place. Keep out_nodes: these outputs are needed by the fused path and cannot be elided. This preserves the existing operation order for sigmoid, sqrt-softplus, and softmax.

Suggested change

	if (args.sigmoid) {
	ops.insert(ops.end(), { GGML_OP_UNARY });
	} else if (args.sqrt_softplus) {
	ops.insert(ops.end(), { GGML_OP_UNARY, GGML_OP_SQRT });
	} else {
	ops.insert(ops.end(), { GGML_OP_SOFT_MAX });
	}
	const int i_probs = i + (int) ops.size() - 1; // last node of the gating activation
	if (args.prob_bias) {
	ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, GGML_OP_VIEW,
	GGML_OP_GET_ROWS });
	out_nodes[0] = i_probs + 4;
	} else {
	ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS });
	out_nodes[0] = i_probs + 3;
	}
	if (args.sigmoid) {
	ops.push_back(GGML_OP_UNARY);
	} else if (args.sqrt_softplus) {
	ops.insert(ops.end(), { GGML_OP_UNARY, GGML_OP_SQRT });
	} else {
	ops.push_back(GGML_OP_SOFT_MAX);
	}
	const int i_probs = i + static_cast<int>(ops.size()) - 1;
	ops.push_back(GGML_OP_RESHAPE);
	if (args.prob_bias) {
	ops.push_back(GGML_OP_ADD);
	}
	ops.insert(ops.end(), { GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS });
	out_nodes[0] = i_probs + (args.prob_bias ? 4 : 3);

[amangupta-tether]

Done in 6d95fca, as suggested: the reshape, the optional ADD and the shared top-k suffix are pushed once, out_nodes[0] = i_probs + (args.prob_bias ? 4 : 3), and out_nodes is kept. The op order is unchanged; TOPK_MOE 416/416 on ROCm.

[aegioscy]

Hoist the invariant offsets and outer bounds checks. Cache the three buffer offsets once, compute row/column bases outside the loops, and move the fixed row/column guards around the unrolled loops. Keep the per-iteration c/r checks and the unconditional barrier for partial tiles. I kept p.ne11 for the src1 guard; the dispatch predicate currently guarantees ne01 == ne11, but this makes the source extent explicit. Any performance gain should be confirmed from generated code or a benchmark; this is a source-level refactor.

Suggested change

	void main() {
	const uint tx = gl_LocalInvocationID.x;
	const uint ty = gl_LocalInvocationID.y;
	const uint row = gl_WorkGroupID.x * 32 + tx;
	// src0 is already contiguous, copy it straight through.
	if (gl_WorkGroupID.y == 0 && row < p.ne01) {
	for (uint i0 = ty; i0 < p.ne00; i0 += 8) {
	data_d[get_doffset() + row*p.nb21 + i0*p.nb20] = D_TYPE(data_a[get_aoffset() + row*p.nb01 + i0*p.nb00]);
	}
	}
	[[unroll]] for (uint j = 0; j < 32; j += 8) {
	const uint c = gl_WorkGroupID.y * 32 + ty + j;
	if (c < p.ne10 && row < p.ne11) {
	tmp[ty + j][tx] = A_TYPE(data_b[get_boffset() + c*p.nb10 + row*p.nb11]);
	}
	}
	barrier();
	const uint col = gl_WorkGroupID.y * 32 + tx;
	[[unroll]] for (uint j = 0; j < 32; j += 8) {
	const uint r = gl_WorkGroupID.x * 32 + ty + j;
	if (col < p.ne10 && r < p.ne11) {
	data_d[get_doffset() + r*p.nb21 + (p.ne00 + col)*p.nb20] = D_TYPE(tmp[tx][ty + j]);
	}
	}
	}
	void main() {
	const uint tx = gl_LocalInvocationID.x;
	const uint ty = gl_LocalInvocationID.y;
	const uint row = gl_WorkGroupID.x * 32 + tx;
	const uint d_offset_base = get_doffset();
	const uint a_offset_base = get_aoffset();
	const uint b_offset_base = get_boffset();
	if (gl_WorkGroupID.y == 0 && row < p.ne01) {
	const uint d_row_ptr = d_offset_base + row * p.nb21;
	const uint a_row_ptr = a_offset_base + row * p.nb01;
	for (uint i0 = ty; i0 < p.ne00; i0 += 8) {
	data_d[d_row_ptr + i0 * p.nb20] = D_TYPE(data_a[a_row_ptr + i0 * p.nb00]);
	}
	}
	if (row < p.ne11) {
	const uint b_row_ptr = b_offset_base + row * p.nb11;
	[[unroll]] for (uint j = 0; j < 32; j += 8) {
	const uint c = gl_WorkGroupID.y * 32 + ty + j;
	if (c < p.ne10) {
	tmp[ty + j][tx] = A_TYPE(data_b[b_row_ptr + c * p.nb10]);
	}
	}
	}
	barrier();
	const uint col = gl_WorkGroupID.y * 32 + tx;
	if (col < p.ne10) {
	const uint d_col_ptr = d_offset_base + (p.ne00 + col) * p.nb20;
	[[unroll]] for (uint j = 0; j < 32; j += 8) {
	const uint r = gl_WorkGroupID.x * 32 + ty + j;
	if (r < p.ne11) {
	data_d[d_col_ptr + r * p.nb21] = D_TYPE(tmp[tx][ty + j]);
	}
	}
	}
	}

[amangupta-tether]

Done in c3fe78c, as suggested: offsets read once, row/column bases and the fixed row/column guards moved outside the unrolled loops, the per-iteration c/r checks and the unconditional barrier kept. CONCAT 226/226, CONCAT_SSM_CONV_SILU 12/12 and SSM_CONV 53/53 on Vulkan0. Perf is neutral: with the direct GDN conv off (GGML_VK_SSMCONV_DIRECT=0), so the transpose kernel actually runs, Qwen3.8 27B pp2048 measures 1.47-1.59 ms per CONCAT before and 1.49-1.52 ms after (48 calls, 2 runs each). In the default path the concat is fused into CONCAT_SSM_CONV_SILU and only its tail runs (8.5 us either way).

[aegioscy]

[P1] Anchor allocation dependencies to the final fused node. add_alloc_deps uses its second argument as an actual cgraph->nodes[last_node] index. A match of ops.size() nodes ends at i + ops.size() - 1 (already stored in out_nodes[1]), so the current call points one node past the match. If the fusion ends at the graph boundary it reads outside the valid node range, and dependencies can be keyed to a node the scheduler never visits, losing the intended lifetime protection. Use the retained weights node and cover a top-k fusion at the end of the graph.

Suggested change

	add_alloc_deps(i, i + ops.size());
	add_alloc_deps(i, out_nodes[1]);

[amangupta-tether]

Fixed in 28dc651: add_alloc_deps(i, out_nodes[1]), so the deps are keyed on the weights node that ends the match. ROCm (gfx1151): TOPK_MOE 416/416 and ARGSORT 46/46; Flash-Next pp2048 525.7 t/s, unchanged. Upstream master has the same i + ops.size() call, so this should go upstream as well.

[gianni-cor]

[P1] Keep the direct SSM conv path opt-in until recurrent graph allocation is stable. On this PR head, GGML_SCHED_DEBUG_REALLOC=1 ctest -R ^test-recurrent-state-rollback$ aborts with unexpected graph reallocation (graph size = 170, nodes = 170, leafs = 58) in the Vulkan jobs. Running the same test with GGML_VK_SSMCONV_DIRECT=0 passes, which isolates the regression to this default-enabled path. This currently fails both Vulkan NVIDIA jobs and the Apple Vulkan job. Please correct the allocation dependencies so recurrent replays retain an identical graph allocation, or leave the optimization disabled by default until that is fixed.

[amangupta-tether]

Fixed in 587d948 without turning the path off. Vulkan's graph_optimize registered the x/state alloc deps only for concats that passed the full direct-conv check, which rejects ubatches under 8 tokens. So the graph gained dependency nodes only for long ubatches, and its node count changed with the ubatch size. After the scheduler reserved for the largest ubatch, a short ubatch needed a fresh allocation, and the next short ubatch with more tokens but the same node count tripped the debug check. The minimum token count now applies only to the runtime match, so the deps are registered for every ubatch size and the graph no longer changes shape. Short ubatches still run the plain concat + conv, with x and the state kept alive through the SiLU. Reproduced on RADV (gfx1151) first: GGML_SCHED_DEBUG_REALLOC=1 test-recurrent-state-rollback with the qwen35 test model aborted before the fix and passes after, as do the deepseek4, nemotron-h and kimi-k3 variants. CONCAT 226/226, CONCAT_SSM_CONV_SILU 12/12, SSM_CONV 53/53. Vulkan pp2048/tg128 is unchanged (27B Q8 328/7.93 both ways; Flash-Next tg 26.19 vs 26.18).

[gianni-cor]

[P1] Do not add this shape unconditionally while WebGPU cannot dispatch it. This case generates 270001 workgroups in X, but WebGPU limits each dispatch dimension to 65535. Both WebGPU CI jobs abort at DispatchWorkgroups(270001, 1, 1) rather than reporting an unsupported case. Please gate this case out for WebGPU, or implement a split/strided dispatch before enabling it in the generic backend test suite.

[amangupta-tether]

Fixed in e1cbcda: test_rms_norm now skips WebGPU when the case has more than 65535 rows (WebGPU dispatches one workgroup per row in X). The {16, 270001} case still runs on every other backend, since it is there to cover the Vulkan grid past 65535 workgroups. RMS_NORM 60/60 on Vulkan0.

[gianni-cor]

[P1] Revisit this gfx1151 cutoff because it regresses the target machine with the exact PR model. On qvac-dev-strix-1, Qwen3.8-27B-UD-Q8_K_L with -ngl 999 -fa 1 -b 2048 -ub 2048 -p 2048 -n 512 -r 2 gives pp2048 288.33 +/- 0.80 t/s on the new default hipBLAS path versus 316.71 +/- 4.04 t/s when the prior MMQ path is forced with GGML_CUDA_RDNA3_5_Q8_0_MMQ_MAX=100000. That is about a 9% prompt-processing regression, while generation is effectively unchanged (7.96 vs 8.00 t/s). This is opposite to the improvement stated in the added comment, so the global 512-row cutoff should be revised or gated more narrowly.

[amangupta-tether]

Thanks for measuring this. The difference is the ROCm version, not the hardware: on our gfx1151 box (TheRock ROCm 10.0, HIP 7.15) the same model and settings still give pp2048 354 t/s with hipBLAS against 308 t/s with MMQ forced (2 interleaved runs each). Your 288 vs 317 is ROCm 7.2.2 (HIP 7.2). f2df7a3 enables the 512-row default only for HIP_VERSION >= 7.15, the version where it was measured, and keeps MMQ for all row counts before that, so your setup gets the 317 t/s path by default. GGML_CUDA_RDNA3_5_Q8_0_MMQ_MAX still overrides it either way. I don't have a HIP version between 7.2 and 7.15 to find the actual crossover, so the cutoff is conservative.

[gianni-cor]

Latest benchmark results from qvac-dev-strix-1 using the same model requested for the PR comparison:

• Model: Qwen3.8-27B-UD-Q8_K_L
• SHA-256: 2a13bba36d2efa213f9275abc430c40e4d914b145bfde976f30f1bb5b7e23ac2
• Size: 28,045,695,904 bytes
• GPU target: gfx1151
• ROCm: 7.2.2
• HIP command parameters: -ngl 999 -fa 1 -b 2048 -ub 2048 -p 2048 -n 512 -r 2

Path	pp2048	tg512
New default hipBLAS path	288.33 +/- 0.80 t/s	7.96 +/- 0.01 t/s
Prior MMQ path, forced with GGML_CUDA_RDNA3_5_Q8_0_MMQ_MAX=100000	316.71 +/- 4.04 t/s	8.00 +/- 0.00 t/s

The new default is about 9.0% slower for prompt processing on this machine; generation speed is effectively unchanged. This does not reproduce the pp2048 310 -> 354 t/s improvement stated in the source comment.

For the same Q8_K_L model through Vulkan, the latest result was pp2048 364.32 +/- 0.99 t/s, with tg128 8.06 t/s.

[amangupta-tether]

Thanks for the runs. All three findings are addressed in 587d948, e1cbcda and f2df7a3; details are on each thread.

On the Q8_0 numbers: with ROCm 7.2.2 the new default is now MMQ again (the 317 t/s path). The hipBLAS rule only applies from HIP 7.15 (ROCm 10.0) on, where it measures 308 -> 354 t/s on our gfx1151.

The one gpu-metal failure on this run (MUL_MAT_VEC_FUSION with ptq1_0, Metal only) is not touched by this PR. The same job passed twice on #396, which contains all of these commits, so it looks flaky. #396 is rebased onto the new head.