[Strix Halo / gfx1151] Vulkan decode-at-depth regression: master @0a37251f4 vs old fork pin @5621d2b5c (-22% tg@48K, FA path dominant)

[baraxnaxgaming-commits]

Hardware: AMD Ryzen AI Max+ 395 (gfx1151, Radeon 8060S), 128 GB UMA, Mesa RADV STRIX_HALO.
Model: Qwen3.8 Flash Next qwen4exp A3B, Q4_0_ROCMFP4_FAST flat GGUF (~93 GB), DKQ=256, GQA 24q/2kv.
Backends compared: --device Vulkan0 only (ROCm excluded on the old pin - it has no HIP backend).
Env both arms: GGML_CUDA_DISABLE_GRAPHS=1 (CUDA knob; inert for Vulkan, kept constant), -ngl 99.

Repro (llama-bench, repetitions 2, stddevs <= 0.84 t/s)

llama-bench -m <flat.gguf> -p 512 -n 32 -d 0,16384,49152 \
            -b 2048 -ub 512 -ngl 99 -fa on|off -r 2 --device Vulkan0 -o csv

Results (tg32 t/s per warm-KV depth)

tree	commit	FA	@0	@16k	@48k
old fork pin	5621d2b5c	on	27.18	24.56	22.72
strix master	0a37251f4	on	27.81	23.84	17.67
old fork pin	5621d2b5c	off	26.77	20.05	8.58
strix master	0a37251f4	off	28.62	19.60	7.72

Delta (master vs old pin): FA on: +2.3% / -2.9% / -22.2% at 0/16K/48K.
FA off: +6.9% / -2.2% / -10.0%.

Reading

• Depth-only regression: parity (or better) at depth 0, ~-3% at 16K, -22% at 48K with FA on.
• With FA off both trees collapse (non-FA KV reads at 48K are slow everywhere - expected), but
  the master-vs-pin gap shrinks from -5.05 t/s to -0.86 t/s absolute at 48K.
• => ~83% of the loss is attention-path-specific at depth; a ~17% residual sits in the non-FA
  decode path (KV cache layout / rope / fattn_vec fallback changes upstream between the pins).
• Same model file both arms (flat GGUF), so this is not a ple16/split-PLE artifact; the old pin
  loads the flat file fine and lands within 3.5% of its own ple16 numbers (22.72 vs 23.54 tg@48K).

Context / asks

• Observed while gating a HIP cutover for this model: server-side decode ladders reproduce it
  (new-tree Vulkan0 arm 16.97 t/s at ~48K ctx vs old production binary 22.37; identical probe
  suite, all correctness needles clean on both).
• The CUDA/HIP side of master is separately not our suspect here: on ROCm, -fa off is WORSE than
  -fa on (5.66 vs 11.72 t/s w48), i.e. the FA kernels help there; this issue is Vulkan-specific.
• CI does not cover this class (no Vulkan depth ladder). Bisect candidates between ~2026-09-05
  and master HEAD: upstream fattn/fattn_vec churn, KV cache layout changes (kv-unified era),
  RDNA pipeline-selection changes in ggml-vulkan. Happy to re-run any candidate build on the
  Strix Halo with this exact ladder.

Evidence files (share): r29-bench-newtree-{vulkan}-, r30-bench-oldflat-vk-, r31-bench--faoff- CSVs;
server ladders r28/r29 verdicts. Happy to provide raw CSVs on request.

[baraxnaxgaming-commits]

r32 profiling update: the depth regression is dominated by COPY ops, not FA kernels

This corrects my earlier read that ~83% of the loss sits "inside the attention path" - at node level most of it is CONT (contiguous copies) that bill near attention.

Method: llama-bench -p 512 -n 32 -d 0 / -d 49152, -fa on, --device Vulkan0, GGML_VK_PERF_LOGGER=1 (per-op GPU timings), same model file (Qwen3.8-Flash-Next flat GGUF) in both trees: old fork 5621d2b5c vs current master lineage 0a37251f4. I aggregated decode-graph windows only (flush windows containing a FLASH_ATTN_EXT with q(...,1,...)), so prefill noise is excluded. Busy/token reconciles with wall clock: old 39.3 ms busy / 42.5 ms wall = 23.5 t/s; new 52.8 / 56.5 = 17.7 t/s - i.e. the budget below closes the gap.

Per-token decode cost at depth ~49K, new minus old (ms/token):

op	delta
CONT	+7.7
RMS_NORM_MUL	+1.6
ADD	+1.3
MULTI_ADD	+0.8
FLASH_ATTN_EXT (decode kernel)	+0.6
SCALE	+0.5
TOP_K family	+0.4
total	~+13.5

Key observations:

• CONT on the old tree is flat at ~2.8 us/node regardless of depth. On the new tree it scales with n_kv: 2.5 us/node at d0 -> 47.8 us/node at 49K, with ~200 CONT nodes per decode token. Something in the new graph materializes/copies n_kv-sized tensors on every decode token.
• The FA decode kernel itself is only +28% (218 us vs 280 us per call at k=49408, hd 256, GQA 24/2) = ~0.6 ms/token of the gap. Not the driver.
• QSA top-k is cheaper on the new tree (fused TOPK_QSA radix sub-op) than the old tree's TOP_K op; it is exonerated.

Suspect area: the QSA mask path at decode time - build_attn_qsa / llama-memory-hybrid-idx (the fill/set_rows/add chain over n_kv and its CONT views). A bisect should target graph-construction changes between the 5621d2b5c era and 0a37251f4, not fattn_vec or the topk shaders.

Happy to re-run any arm with extra flags/dumps; raw per-op logs are archived on my side.

Written by Hermes Agent (posted by baraxnaxgaming-commits).