Repository navigation
HIP: verify windows past 4 rows and past 32 rows fall off the stream and K-parallel tiles #1605
Description
On AMD Ryzen AI MAX+ 395, Radeon 8060S gfx1151 (40 CUs, 128 GB unified memory, measured 225.6 GB/s read), Ubuntu 26.04, archive HIP 7.1 (clang 21), TensorFold 7c068895, TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP.
The decode stream tile takes 1, 2 or 4 rows (zig/src/hip/launches/affine.zig), so a 5-8-row verify window reads every weight twice; and with 4 verify rows per stream, the 9th concurrent stream crosses 32 rows where the routed tile switches from the K-parallel tile to the GEMM tile (1.03 -> 2.74 ms for a gate/up pair in hip-affine-tune). This caps deeper MTP and 8+ streams.
Proposal
An 8-row stream variant and a routed selection that keeps verify-sized plans off the 128-row GEMM tile, with equal bytes per row and served receipts at 1/4/8/12 streams.
Activity
gilby commentedon Oct 29, 2026
Measured on AMD Ryzen AI MAX+ 395 (Radeon 8060S
gfx1151), TensorFold7c068895,TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP, MTP 3 drafts.zig/src/native/hip.zigrefuses--mtp-draftsabove 3 ("the HIP engine drafts at most 3"), so a stream verifies at most 4 rows, which the existing R=4 stream tile covers. An 8-row stream variant would only matter if that cap is raised.tools/bench_concurrent.py --levels 1,4,8,9,12 --reps 3 --temperatures 0 --alonewith--parallel 12 --context 16384, aggregate tok/s code / chat: 1 stream 50.2 / 44.1, 4 streams 90.3 / 73.0, 8 streams 99.9 / 80.3, 9 streams 97.7 / 78.4 (-2.2% / -2.4% vs 8), 12 streams 105.0 / 84.0. The kp3-to-GEMM switch at 33 rows costs about 2% served, and the only remedy found is the routed-plan selection bound already proposed in HIP: bound the routed prompt's K-parallel tile by pairs per item, so gfx1201 can take it for Qwen3.6's short prompts #1557, so I'm not opening a competing change.ashhart commentedon Oct 30, 2026
Thank you, @gilby, for measuring both cases on gfx1151. Your numbers answer the first one: HIP drafts at most 3 tokens, so a stream verifies at most 4 rows, and the existing 4-row stream tile covers that. An 8-row variant only matters if that cap is raised. For the second one, the switch to the GEMM tile at 33 rows costs about 2% served, and the remedy is the routed plan's pairs-per-item bound from #1557. That bound is now on main through #1607, tuned for gfx1201 so far, and
hip-affine-tuneproposes the bound for a card. If you run it on your gfx1151 and send the resulting table row as a small PR, we'll gate it with the 1/4/8/9/12-stream cells you used here. We're closing this one as investigated. Please reopen if the 33-row step is still there once gfx1151 has its own bound.