From 1b97c5409474052b9dede2d5bb4b7ced76f66995 Mon Sep 17 00:00:00 2001 From: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com> Date: Thu, 19 Nov 2026 16:23:08 -0800 Subject: [PATCH] Model K (D3D12): 16-byte aligned loads in the wide stage and split-m kernels (-52.8 us summed dispatch time at 1440p on RADV, byte-identical) The wide stage kernels (k_stagew2_e3 / d3 / e4 / d4, k_stagew4_e4 / d4 / m, k_block_wide.hlsl) and the phase kernels of the split stage m (k_split_m_*, k_block_split.hlsl) load their weight, bias and activation fragments through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits every 16-byte load into four global_load_b32 with four address additions: k_stagew2_e3 compiled to 1236 dword loads and no wider one, k_stagew2_d3 to 1276, k_stagew4_e4 to 1044, k_stagew4_d4 to 1040. The same pattern as the narrow stages; the wide stages are already at 192 VGPRs and 8 waves, so only the load count is left to win. KW_LD_A16 (default 0 in k_wide.hlsli; set to 1 by k_block_wide.hlsl and k_block_split.hlsl before their includes) makes the new helper kw_ld128, which kw_ldg and the 13 direct 16-byte load sites of the two files use, clear the low 4 address bits. The premise: arena allocations and the packed section bases are 256-byte aligned (k_pack.cpp pads every section to 256 bytes; the runtime's allocator hands out 256-byte steps), the per-lane bases the kernels derive from them add 2 * (256 * tile + 8 * lane) bytes, and every remaining offset is a multiple of 16 bytes (the HIP kernels' own uint4 loads), so each masked address is the requested one. No arithmetic, summation order or accumulator chain changes. One side effect: k_split_m_w2 grows from 192 to 216 VGPRs (one wave per SIMD less) because its address arithmetic shrinks while more loaded data is live. It is a 4 us kernel and it got faster (4.04 -> 2.68 us); the frame is no slower with it (summed dispatch time 1485.6 us with it changed, 1487.1 us with it left at the old shader), so the define stays file-level instead of excluding one phase. k_wide.hlsli, k_block_wide.hlsl and k_block_split.hlsl are in the include closure of the softmax self-check digest, so kSourceDigest in clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own smx_selfcheck_digest.digest(); the HIP reference hashes in the header are unchanged. That line is also rewritten by the narrow-stage aligned-loads change (#290) and by the Swin / decoder one (#294, wg_platform.hlsli): the three conflict on it, and whichever lands last must recompute it from the combined sources. DXIL scope, from hashing a full build of this tree (Model M and K, 108 files) against main's: 19 files change: k_stagew2_d3, k_stagew2_e3, k_stagew2_d4, k_stagew2_e4, k_stagew4_d4, k_stagew4_e4, k_stagew4_m, their __rodd forms, and k_split_m_attn, k_split_m_attn__rodd, k_split_m_wo, k_split_m_w1, k_split_m_w2. The other 89 are identical, including every Model M kernel, the narrow stages, the input stage and the output kernels. The timed measurements below ran the 8 mix-form kernels that a default frame launches (k_stagew2_e3, k_stagew2_d3, k_stagew4_e4, k_stagew4_d4 and the four k_split_m phases), byte-identical to this tree's. The other 11 changed files were run in a second check from this tree's DXIL, below. Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile "Default" (overclocked): absolute times are on that profile; the interleaved arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only the named DXIL files swapped; a synthetic native D3D12 frame harness): - No-card gfx1201 instructions: k_stagew2_e3 6861 -> 5372 (-21.7 %), k_stagew2_d3 7092 -> 5599 (-21.1 %), k_stagew4_e4 5571 -> 4271 (-23.3 %), k_stagew4_d4 5649 -> 4309 (-23.7 %); memory instructions 1252 -> 325, 1288 -> 331, 1056 -> 273, 1048 -> 268; VGPRs unchanged, no spills. - Byte identity at 200x120 (5 frames), 1279x721 and 2560x1440 (3 frames): a same-session pair of base runs and the candidate, 17 files per frame (the buffers k.Np, k.Hc, k.L, k.Hh, k.x0, k.S0-S4, k.A, k.B, k.ctap, k.ctapT, k.scalars, exposure.pair and the filter output; k.N is only written with k_head=full), every file present, non-empty, the same length, compared over its full length: base pair 85/85, 51/51, 51/51, candidate equal in all of them, and the create-time softmax self-check passes on the new kernels (smx=mix split=m in the report). - Controls built from this tree with the mask changed to ~31u. (a) The wrong mask in the 8 mix-form kernels: the self-check rejects the mix form and the runtime falls back twice, to the round-to-odd ports and, because the poisoned k_split_m_wo / w1 / w2 phases are shared, to the one-launch stage m (smx=rodd in the report); the frame output is then produced by the surviving, unchanged shaders and equals the base, so this is a route-selection control, not a byte-difference control. (b) The wrong mask in the mix and the __rodd forms: create is refused at all three sizes ("the D3D12 softmax stages are not exact on this driver"). (c) The wrong mask in k_stagew2_e4__rodd (with the mix form of k_stagew2_e3 poisoned so that the round-to-odd route is taken): create is refused at 200x120 and 3840x2160 naming "round-to-odd form k_stagew2_e4__rodd@1536x1280". (d) The wrong mask in the mix k_stagew2_e4 only, at 3840x2160: the self-check rejects it and the runtime takes the round-to-odd ports; a route-selection control, no byte comparison. - Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL. Summed GPU dispatch time (profiler-on runs; all valid dispatch durations of a frame summed, the timing_barrier dispatch excluded (0.36 us, including it gives 1538.8 -> 1486.0 us); frames whose unchanged k_in_tiled_uniform kernel is more than 3 % above its run's minimum are dropped as low-clock, 271 and 282 of 300 kept; unfiltered 1537.8 -> 1485.6 us): 1538.4 -> 1485.6 us (-52.8 us, -3.4 %). k_stagew2_e3 46.3 -> 34.2, k_stagew2_d3 40.6 -> 28.7, k_stagew4_e4 39.4 -> 27.9, k_stagew4_d4 38.3 -> 26.6 us; the split phases attn 4.12 -> 3.80, wo 2.80 -> 1.82, w1 3.76 -> 2.84, w2 4.04 -> 2.68 us; every other kernel unchanged within 0.6 us. Two further measures per arm, both medians of 300 frames with paired per-round differences: (1) the GPU frame span, the sum of the runtime's four stage timestamp intervals (exposure, input, network, output): profiler on 1.6302 -> 1.5777 ms (-52.5 us; paired median -52.2 us, 5 of 5 rounds faster), profiler off 1.6450 -> 1.6153 ms (-29.7 us; paired median -28.5 us, 5 of 5). (2) The host wall-clock time of each frame call (QueryPerformanceCounter around it): profiler on 4.7037 -> 4.6440 ms (-59.7 us; paired -59.5 us, 5 of 5; inflated by the profiler's readback), profiler off 1.7722 -> 1.7418 ms (-30.4 us; paired -36.0 us, 5 of 5). Busy-sample shader clock about 2.8 GHz with the profiler on (throttle status zero in all samples) and 3.13-3.15 GHz with it off; the profiler-off runs carry throttle status on about 61 % of busy samples in every arm (236 of 388 base, 229 of 382 this change), comparable between the arms. The summed dispatch time can count overlapping work. - The 11 changed files the timing did not use (the __rodd forms, k_stagew2_d4, k_stagew2_e4, k_stagew4_m), from this tree's complete DXIL, each frame compared byte for byte (17 files per frame, full lengths) with main's build on the same route, a same-session pair of main runs as the baseline: * mix route, all 19 changed files, 200x120, 1279x721, 2560x1440 and a 3840x2160 output (1920x1080 render; its base map has more than 24 windows, so e4 / d4 take the NG = 2 kernels k_stagew2_e4 / k_stagew2_d4): equal in all four, smx=mix; the 3840x2160 frames launch k_stagew2_d4 / e4. * round-to-odd route, forced without a developer build by replacing the mix-form k_stagew2_e3 with the wrong-mask build (the self-check rejects it and takes the round-to-odd ports, which passed it: smx=rodd): equal at 200x120, 2560x1440 and 3840x2160 with the split stage m (k_split_m_attn__rodd, k_stagew2_e3 / d3 / d4 / e4 __rodd launched at 3840x2160), and with stage m as one launch (the mix-form k_split_m_wo also replaced; k_stagew4_m__rodd launched) at 200x120 and 3840x2160. * mix route with the one-launch stage m (only k_split_m_wo replaced, so the split form fails its check and stage m takes the one launch k_stagew4_m): equal at 200x120 and 1279x721, smx=mix. Dispatch lists of one frame are saved for the 3840x2160 mix, round-to-odd split and round-to-odd one-launch runs and for the 200x120 one-launch mix run. The NG = 4 round-to-odd shaders k_stagew4_e4__rodd / k_stagew4_d4__rodd have no dispatch list: their use in the 200x120 and 2560x1440 round-to-odd frames follows from the stage routing and the runs' reports, and the create-time self-check dispatches them (512x384 cases). The create-time self-check runs both rounding forms of each case on the selected stage-m route: with the split on (the default) the 17 changed shaders other than k_stagew4_m and k_stagew4_m__rodd run at every start and are compared with the HIP reference hashes, and k_stagew4_m / __rodd are not dispatched at create; they are covered by the one-launch frame runs above (and run at create when the split is off or its check fails). Not tested: the Windows driver; RDNA 3 (the K kernels do not compile on the no-card gfx1100 route here). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com> --- clean/k/dx12/hlsl/k_wide.hlsli | 9 ++++++++- clean/k/dx12/kernels/k_block_split.hlsl | 11 ++++++----- clean/k/dx12/kernels/k_block_wide.hlsl | 17 +++++++++-------- clean/k/runtime/k_smx_selfcheck.h | 2 +- 4 files changed, 24 insertions(+), 15 deletions(-) diff --git a/clean/k/dx12/hlsl/k_wide.hlsli b/clean/k/dx12/hlsl/k_wide.hlsli index 3b8b0b3a..aa6cac1d 100644 --- a/clean/k/dx12/hlsl/k_wide.hlsli +++ b/clean/k/dx12/hlsl/k_wide.hlsli @@ -154,7 +154,14 @@ uint4 kw_pin(uint4 v) } // global fragments: pointer (arena byte offset) + element (half) offset -uint4 kw_ldg(uint p, uint e) { return wg_ld128(p, e * 2u); } +// KW_LD_A16 (set by the wide-stage kernel files): 16-byte loads with the alignment made visible to the driver. Load4 is known +// 4-byte aligned only and compiles to four dword loads; every base is a 256-byte aligned runtime allocation (k_pack.cpp kAlign) and +// every offset a multiple of 16 bytes (HIP's own uint4 loads), so clearing the low 4 address bits changes no address. +#ifndef KW_LD_A16 +#define KW_LD_A16 0 +#endif +uint4 kw_ld128(uint base, uint off) { return wg_mem.Load4((base + off) & (KW_LD_A16 ? ~15u : ~0u)); } +uint4 kw_ldg(uint p, uint e) { return kw_ld128(p, e * 2u); } void kw_stg(uint p, uint e, uint4 v) { wg_st128(p, e * 2u, v); } // Mirror about the edge token, edge not repeated (spec 41; k_common.h wgk_mirror), branch free. diff --git a/clean/k/dx12/kernels/k_block_split.hlsl b/clean/k/dx12/kernels/k_block_split.hlsl index 2387060e..ccef0bd1 100644 --- a/clean/k/dx12/kernels/k_block_split.hlsl +++ b/clean/k/dx12/kernels/k_block_split.hlsl @@ -28,6 +28,7 @@ // @wg-kernel k_split_m_wo co=k_block_wide.co block=128 wave=32 args=s184:p19.b32,p,p,p,p entry=ks_wo defines=KS_PHASE=2,KW_C=160,KW_HEADS=8,KW_IN=0,KW_CIN=160,KW_CNEXT=0,KW_NG=4 // @wg-kernel k_split_m_w1 co=k_block_wide.co block=128 wave=32 args=s184:p19.b32,p,p,p,p entry=ks_w1 defines=KS_PHASE=3,KW_C=160,KW_HEADS=8,KW_IN=0,KW_CIN=160,KW_CNEXT=0,KW_NG=4 // @wg-kernel k_split_m_w2 co=k_block_wide.co block=128 wave=32 args=s184:p19.b32,p,p,p,p entry=ks_w2 defines=KS_PHASE=4,KW_C=160,KW_HEADS=8,KW_IN=0,KW_CIN=160,KW_CNEXT=0,KW_NG=4 +#define KW_LD_A16 1 // 16-byte aligned loads (hlsl/k_wide.hlsli) #include "wg_platform.hlsli" WG_ARGS_BEGIN @@ -111,7 +112,7 @@ uint ks_vidx(int row, int c) { return (uint)(256 + row * 8 + (c ^ (row & 7))) & static uint ksA_qkv; // product s = (ct KT + kt) 3 + {q, k, v} of the head's q / k / v chains (k_block_wide.hlsl's stream order of a round) -uint4 ks_wq(int s) { return wg_ld128(ksA_qkv, (uint)(((s % 3) * 2 * KT + s / 3) * 512)); } +uint4 ks_wq(int s) { return kw_ld128(ksA_qkv, (uint)(((s % 3) * 2 * KT + s / 3) * 512)); } #define KS_TAKE(s) rg[(s) % KS_RD] #define KS_NEXT(s) { if ((s) + KS_RD < SA) rg[(s) % KS_RD] = ks_wq((s) + KS_RD); } @@ -236,7 +237,7 @@ WG_KERNEL(ks_wo) const uint zr = ks_zrow(win, L.t, hh); uint4 wf[DT], zf[DT]; [unroll] for (int d0 = 0; d0 < DT; ++d0) { - wf[d0] = wg_ld128(wo, (uint)(d0 * 512)); + wf[d0] = kw_ld128(wo, (uint)(d0 * 512)); zf[d0] = kw_ldg(P_sz, zr + 16u * d0); } const uint4 xr = kw_ldg(P_x, L.mtok * KW_C + 8u * hh + 16u * ot); @@ -268,7 +269,7 @@ WG_KERNEL(ks_w1) uint4 a[KT]; uint4 wf[KS_HB][KT]; [unroll] for (int kt0 = 0; kt0 < KT; ++kt0) { - [unroll] for (int m0 = 0; m0 < KS_HB; ++m0) wf[m0][kt0] = wg_ld128(w1, (uint)((m0 * KT + kt0) * 512)); + [unroll] for (int m0 = 0; m0 < KS_HB; ++m0) wf[m0][kt0] = kw_ld128(w1, (uint)((m0 * KT + kt0) * 512)); a[kt0] = kw_ldg(P_sa, cr + 16u * kt0); } uint4 b1v[KS_HB]; @@ -312,8 +313,8 @@ WG_KERNEL(ks_w2) [unroll] for (int b = 0; b < NB2; ++b) { // the hidden tiles 0 .. HT - 1, in order uint4 rw[KS_B2], rh[KS_B2]; [unroll] for (int j0 = 0; j0 < KS_B2; ++j0) { - rw[j0] = wg_ld128(w2, (uint)((b * KS_B2 + j0) * KT * 512)); - rh[j0] = wg_ld128(hb, (uint)((b * KS_B2 + j0) * 32)); + rw[j0] = kw_ld128(w2, (uint)((b * KS_B2 + j0) * KT * 512)); + rh[j0] = kw_ld128(hb, (uint)((b * KS_B2 + j0) * 32)); } uint f = b == 0 ? b2v.x ^ x1.x : 0u; [unroll] for (int j1 = 0; j1 < KS_B2; ++j1) f ^= rw[j1].x ^ rh[j1].x; diff --git a/clean/k/dx12/kernels/k_block_wide.hlsl b/clean/k/dx12/kernels/k_block_wide.hlsl index 0303fd57..64793be0 100644 --- a/clean/k/dx12/kernels/k_block_wide.hlsl +++ b/clean/k/dx12/kernels/k_block_wide.hlsl @@ -20,6 +20,7 @@ // @wg-kernel k_stagew4_d4__rodd co=k_block_wide.co block=512 wave=32 args=s184:p19.b32 entry=kw_stage defines=K_SMX=1,KW_C=128,KW_HEADS=4,KW_IN=1,KW_CIN=160,KW_CNEXT=0,KW_NG=4,KW_RD=12 // @wg-kernel k_stagew2_d4__rodd co=k_block_wide.co block=256 wave=32 args=s184:p19.b32 entry=kw_stage defines=K_SMX=1,KW_C=128,KW_HEADS=4,KW_IN=1,KW_CIN=160,KW_CNEXT=0,KW_NG=2 // @wg-kernel k_stagew2_d3__rodd co=k_block_wide.co block=256 wave=32 args=s184:p19.b32 entry=kw_stage defines=K_SMX=1,KW_C=96,KW_HEADS=4,KW_IN=1,KW_CIN=128,KW_CNEXT=0,KW_NG=2,KW_RD=4 +#define KW_LD_A16 1 // 16-byte aligned loads (hlsl/k_wide.hlsli) #include "wg_platform.hlsli" WG_ARGS_BEGIN @@ -76,29 +77,29 @@ uint4 kw_wld(int s) { if (s < B_R) { // expand: product s = kt NK + k (tile wave + k NW) const int kt = s / max(NK, 1), k = s % max(NK, 1); - return wg_ld128(kwA_wa, (uint)((k * NW * KTI + kt) * 512)); + return kw_ld128(kwA_wa, (uint)((k * NW * KTI + kt) * 512)); } if (s < B_C) { const int r = (s - B_R) / RLEN, f = (s - B_R) % RLEN; if (f < SQ) // f = (ct KT + kt) 3 + {q, k, v}; head r NG + grp - return wg_ld128(kwA_qkv, (uint)((r * NG * 3 * 2 * KT + (f % 3) * 2 * KT + f / 3) * 512)); + return kw_ld128(kwA_qkv, (uint)((r * NG * 3 * 2 * KT + (f % 3) * 2 * KT + f / 3) * 512)); const int f2 = f - SQ; // f2 = j NOWN + i: z tile 2 r NG + j, owned tile i if ((KT % NG) == 0) // every owned tile present: grp + NG i, one address - return wg_ld128(kwA_wo[0], (uint)(((f2 % NOWN) * NG * DT + 2 * r * NG + f2 / NOWN) * 512)); - return wg_ld128(kwA_wo[f2 % NOWN], (uint)((2 * r * NG + f2 / NOWN) * 512)); + return kw_ld128(kwA_wo[0], (uint)(((f2 % NOWN) * NG * DT + 2 * r * NG + f2 / NOWN) * 512)); + return kw_ld128(kwA_wo[f2 % NOWN], (uint)((2 * r * NG + f2 / NOWN) * 512)); } if (s < B_M) { const int c = (s - B_C) / CLEN, f = (s - B_C) % CLEN; if (f < S1) { // hidden tile c CH + grp MM + m const int m = KW_ILV ? f % MM : f / KT, kt = KW_ILV ? f / MM : f % KT; - return wg_ld128(kwA_w1, (uint)(((c * CH + m) * KT + kt) * 512)); + return kw_ld128(kwA_w1, (uint)(((c * CH + m) * KT + kt) * 512)); } const int f2 = f - S1; // f2 = j NOWN + i: hidden tile c CH + j, owned tile i if ((KT % NG) == 0) - return wg_ld128(kwA_w2[0], (uint)(((c * CH + f2 / NOWN) * KT + (f2 % NOWN) * NG) * 512)); - return wg_ld128(kwA_w2[f2 % NOWN], (uint)((c * CH + f2 / NOWN) * KT * 512)); + return kw_ld128(kwA_w2[0], (uint)(((c * CH + f2 / NOWN) * KT + (f2 % NOWN) * NG) * 512)); + return kw_ld128(kwA_w2[f2 % NOWN], (uint)((c * CH + f2 / NOWN) * KT * 512)); } - return wg_ld128(kwA_wz, (uint)((s - B_M) * 512)); // merge: tile min(wave, NTM - 1), kt = s - B_M + return kw_ld128(kwA_wz, (uint)((s - B_M) * 512)); // merge: tile min(wave, NTM - 1), kt = s - B_M } // product s of the stream: its fragment, and the refill of its ring slot with product s + KW_RD diff --git a/clean/k/runtime/k_smx_selfcheck.h b/clean/k/runtime/k_smx_selfcheck.h index 788f8302..acfbe98a 100644 --- a/clean/k/runtime/k_smx_selfcheck.h +++ b/clean/k/runtime/k_smx_selfcheck.h @@ -8,7 +8,7 @@ namespace k_smx_selfcheck { -constexpr char kSourceDigest[] = "a47546460b78b46b3ae0775661c6c4160ad20af09b6106c16322f8d164136264"; +constexpr char kSourceDigest[] = "8df2d07b3e3e564587fd52903fcb4b93532454167d700aa45f6e1da2c7f5fa1e"; constexpr int kCount = 133; constexpr const char* kNames[kCount] = { "k_stage_e2@512x384/o4,4",