From 8184fb8fe59d13ac6cca38b662ecea1ee86e5152 Mon Sep 17 00:00:00 2001 From: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com> Date: Thu, 19 Nov 2026 15:19:19 -0800 Subject: [PATCH] Swin c64 block and fused decoder (D3D12): 16-byte aligned loads (swin_block_c64_nc, d1p12/swin_block_c64_add_dec0_nc; -101.6 us summed dispatch time at 1440p on RADV, byte-identical) wg_ld128 and wg_ld4f (so wg_load8f, st_fetch's weight staging, the gamma / bias tiles and the activation fragments) read through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits each into four global_load_b32 with four address additions: d1p12 compiled to 398 dword loads and no wider one, swin_block_c64_nc to 584 memory instructions. The new macro WG_LD_A16 (default 0, set through defines= on the two kernel declarations) clears the low 4 address bits in those two helpers. The premise holds for these kernels: every base is a per-stage arena allocation (256-byte aligned, device_d3d12.cpp kNativeAlign), the packed block sections are 16-byte multiples (L_G1 = 16, HEAD_BYTES / MLP_BYTES, the bias and gamma tiles), and the activation pixel pitches are multiples of 64 bytes, so the masked address is the requested one. No arithmetic, summation order or WMMA operand changes. It is scoped to these two kernels because a first draft that masked the helpers for every Swin kernel pushed swin_block_c96_pm_rev_nc and swin_block_c96_add_xe_xo_nc from 192 to 256 VGPRs under ACO. With the default 0 every other kernel is unchanged: a full Model M build of this tree against main's differs in exactly these two DXIL files (45 of 47 Model M files are identical); the builds with Model K (106 of 108) and Model L (99 of 101) differ in those same two files only, every K-specific and L-specific binary is identical. wg_platform.hlsli is in the include closure of Model K's softmax self-check digest, so kSourceDigest in clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own smx_selfcheck_digest.digest() (the HIP reference hashes in the header are unchanged: no K shader code changes). Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile "Default" (overclocked): absolute times are on that profile; the interleaved arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only these two DXIL files swapped): - No-card gfx1201: d1p12 4578 -> 4122 instructions, memory instructions 538 -> 247, 192 VGPRs / 8 waves unchanged; swin_block_c64_nc 5441 -> 4921, 584 -> 302, 240 VGPRs unchanged; no spills. - Byte identity (whole-frame runs, synthetic model), from the DXIL of this tree, at 64x40 -> 96x60, 853x481 -> 1279x721 and 1706x960 -> 2560x1440 on the default fusion route, and at the two smaller sizes again with xd removed from WG_D3D12_FUSION (pm,xe,rv,xo,lc,ll,le): below 1.2 M render pixels the default route runs the xd form of the decoder (d1p12/swin_block_c64_add_dec0_xd_nc, unchanged by this commit), so the xd-less runs are the ones that launch the changed d1p12/swin_block_c64_add_dec0_nc (profiler dispatch lists of both routes at both sizes are saved with the runs). Each group has a same-session pair of base runs, the candidate and two controls built from this tree with only the helper changed (mask ~31u; address + 16). 93 dumps per run; 60 live buffers are compared: all except b0-b4.out, dec2.out, enc1.merged, post.display and frame-0 state *_prev / motion_history (unwritten or reused memory), dec0.B (unused with pack12) and dec1.out (dec1 runs inside dec0 on these routes and leaves out1 unwritten); dec0.C only on its active prefix, the first rw*rh*4 bytes, a shorter file is a failure; every expected dump must exist. Result, all five groups: base pair 60/60, candidate 60/60, both controls 24/60. - Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL. Summed GPU dispatch time (from the profiler-on runs: all valid dispatch durations of a frame summed; frames whose unchanged input_encoder kernel is more than 3 % above its run's minimum are dropped as low-clock, 266-272 of 300 kept per arm; unfiltered also given): 2597.1 -> 2495.5 us (-101.6 us, -3.9 %; unfiltered 2607.3 -> 2498.5). d1p12 437.6 -> 368.4 us, c64 block 288.4 -> 261.9 us. Profiler-on runs: busy shader-clock medians 3.04-3.12 GHz per run, throttle status zero in all 951 (base) and 936 (this change) busy samples. Host frame time (frame_io_timed wall time, median of 300 frames): profiler on 2.6857 -> 2.5777 ms; profiler off 2.7062 -> 2.5479 ms (paired per-round differences -111 us and -143 us). The profiler-off runs carry throttle status 2 (hotspot up to 88-89 C, busy-sample clock medians 3.13-3.16 GHz) on 327 of 460 busy samples in the base arm and 317 of 439 in this change's arm (71 % and 72 %), so the profiler-off host numbers are comparable between the arms but not unthrottled. The summed dispatch time can count overlapping kernels; the host time is the wall-clock measure. Not tested: the Windows driver; RDNA 3 (these kernels do not compile on the no-card gfx1100 route here). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com> --- clean/dx12/hlsl/wg_platform.hlsli | 10 ++++++++-- clean/dx12/kernels/dec1_dec0.hlsl | 2 +- clean/dx12/kernels/swin_c64w2.hlsl | 2 +- clean/k/runtime/k_smx_selfcheck.h | 2 +- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/clean/dx12/hlsl/wg_platform.hlsli b/clean/dx12/hlsl/wg_platform.hlsli index 65b6eee5..4a6b1f36 100644 --- a/clean/dx12/hlsl/wg_platform.hlsli +++ b/clean/dx12/hlsl/wg_platform.hlsli @@ -68,10 +68,16 @@ uint wg_ld8(uint base, uint off) { uint a = base + off; return (wg_mem.Load(a & uint wg_ld16(uint base, uint off) { return (uint)wg_mem.Load(base + off); } uint wg_ld32(uint base, uint off) { return wg_mem.Load(base + off); } uint2 wg_ld64(uint base, uint off) { return wg_mem.Load2(base + off); } -uint4 wg_ld128(uint base, uint off) { return wg_mem.Load4(base + off); } +// WG_LD_A16 (kernel's defines=): 16-byte loads with the alignment made visible to the driver. Load4 is known 4-byte aligned +// only and compiles to four dword loads; a kernel that sets it guarantees that every base + offset of its wg_ld128 / +// wg_ld4f is a multiple of 16 bytes (arena allocations are 256-byte aligned, the packed sections are 16-byte multiples). +#ifndef WG_LD_A16 +#define WG_LD_A16 0 +#endif +uint4 wg_ld128(uint base, uint off) { return wg_mem.Load4((base + off) & (WG_LD_A16 ? ~15u : ~0u)); } uint4 wg_ld128_a16(uint base, uint off) { return wg_mem.Load4(base + off); } float wg_ldf(uint base, uint off) { return asfloat(wg_mem.Load(base + off)); } -float4 wg_ld4f(uint base, uint off) { return asfloat(wg_mem.Load4(base + off)); } +float4 wg_ld4f(uint base, uint off) { return asfloat(wg_mem.Load4((base + off) & (WG_LD_A16 ? ~15u : ~0u))); } void wg_st16(uint base, uint off, uint v) { wg_mem.Store(base + off, (uint16_t)v); } void wg_st32(uint base, uint off, uint v) { wg_mem.Store(base + off, v); } void wg_st64(uint base, uint off, uint2 v) { wg_mem.Store2(base + off, v); } diff --git a/clean/dx12/kernels/dec1_dec0.hlsl b/clean/dx12/kernels/dec1_dec0.hlsl index 909782b7..e08e1709 100644 --- a/clean/dx12/kernels/dec1_dec0.hlsl +++ b/clean/dx12/kernels/dec1_dec0.hlsl @@ -21,7 +21,7 @@ // both pixels of every lane and only the stores are conditional). dec0's FP16 latent update is done in FP32 with an FP16 // rounding after each step, not in half arithmetic (the driver contracts half mul + add into v_fma_mix_f32). // Bit-identical to the HIP kernel on the synthetic frames (clean/dx12/compare.py, both cases, every buffer). -// @wg-kernel d1p12/swin_block_c64_add_dec0_nc co=dec1_dec0__pack12.co block=256 wave=32 args=p,p,p,u,u,u,u,p,p,p,p,p,p,p,b56 defines=WG_GELU_T5=0 +// @wg-kernel d1p12/swin_block_c64_add_dec0_nc co=dec1_dec0__pack12.co block=256 wave=32 args=p,p,p,u,u,u,u,p,p,p,p,p,p,p,b56 defines=WG_GELU_T5=0,WG_LD_A16=1 // // DEC0_ONLY (port d1p12/dec0_q7; Model M on the FP16-form shared stages, WG_M_FORM=q7, where dec1 runs as the q7 block // l_q7_block_c64_dec and stores its map): the same kernel without the block. The epilogue's input o[0..3] (lane (t, hh): diff --git a/clean/dx12/kernels/swin_c64w2.hlsl b/clean/dx12/kernels/swin_c64w2.hlsl index 4ba1a5ae..1769335a 100644 --- a/clean/dx12/kernels/swin_c64w2.hlsl +++ b/clean/dx12/kernels/swin_c64w2.hlsl @@ -1,7 +1,7 @@ // SPDX-License-Identifier: MIT // swin_block_c64_nc (enc1's Swin block) with TWO windows per wave in 128-lane workgroups (tracks lev3-swin, r3-c64w2). // -// @wg-kernel swin_block_c64_nc co=swin.co block=256 group=128 gscale=1 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=64,SW_HEADS=2,SW_MLP=256,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_W2=1 +// @wg-kernel swin_block_c64_nc co=swin.co block=256 group=128 gscale=1 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=64,SW_HEADS=2,SW_MLP=256,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_W2=1,WG_LD_A16=1 // Model L's dec1 block swin_block_c64_add_ex_nc (swin_c64ex.hlsl) runs this kernel too (SW_W2_NAME), with the block // input x + xadd (SW_ADD) and the fused expand (SW_ECL: swin_block.hlsl's fused_expand with the 4 waves; each wave then // reads its two windows' records). diff --git a/clean/k/runtime/k_smx_selfcheck.h b/clean/k/runtime/k_smx_selfcheck.h index 788f8302..fde024d8 100644 --- a/clean/k/runtime/k_smx_selfcheck.h +++ b/clean/k/runtime/k_smx_selfcheck.h @@ -8,7 +8,7 @@ namespace k_smx_selfcheck { -constexpr char kSourceDigest[] = "a47546460b78b46b3ae0775661c6c4160ad20af09b6106c16322f8d164136264"; +constexpr char kSourceDigest[] = "97b3541f3d19948aad3b6e7adcfc9e757d39dcb6228892fdb2ebdbe11c046ded"; constexpr int kCount = 133; constexpr const char* kNames[kCount] = { "k_stage_e2@512x384/o4,4",