Repository navigation
Model M (D3D12): 16-byte aligned loads in the c64 Swin block and fused decoder (-102 us GPU time per frame at 1440p on RADV, byte-identical) - #294
Merged
phantoslayer21 merged 2 commits intoNov 20, 2026
Conversation
…_block_c64_nc, d1p12/swin_block_c64_add_dec0_nc; -101.6 us summed dispatch time at 1440p on RADV, byte-identical) wg_ld128 and wg_ld4f (so wg_load8f, st_fetch's weight staging, the gamma / bias tiles and the activation fragments) read through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits each into four global_load_b32 with four address additions: d1p12 compiled to 398 dword loads and no wider one, swin_block_c64_nc to 584 memory instructions. The new macro WG_LD_A16 (default 0, set through defines= on the two kernel declarations) clears the low 4 address bits in those two helpers. The premise holds for these kernels: every base is a per-stage arena allocation (256-byte aligned, device_d3d12.cpp kNativeAlign), the packed block sections are 16-byte multiples (L_G1 = 16, HEAD_BYTES / MLP_BYTES, the bias and gamma tiles), and the activation pixel pitches are multiples of 64 bytes, so the masked address is the requested one. No arithmetic, summation order or WMMA operand changes. It is scoped to these two kernels because a first draft that masked the helpers for every Swin kernel pushed swin_block_c96_pm_rev_nc and swin_block_c96_add_xe_xo_nc from 192 to 256 VGPRs under ACO. With the default 0 every other kernel is unchanged: a full Model M build of this tree against main's differs in exactly these two DXIL files (45 of 47 Model M files are identical); the builds with Model K (106 of 108) and Model L (99 of 101) differ in those same two files only, every K-specific and L-specific binary is identical. wg_platform.hlsli is in the include closure of Model K's softmax self-check digest, so kSourceDigest in clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own smx_selfcheck_digest.digest() (the HIP reference hashes in the header are unchanged: no K shader code changes). Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile "Default" (overclocked): absolute times are on that profile; the interleaved arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only these two DXIL files swapped): - No-card gfx1201: d1p12 4578 -> 4122 instructions, memory instructions 538 -> 247, 192 VGPRs / 8 waves unchanged; swin_block_c64_nc 5441 -> 4921, 584 -> 302, 240 VGPRs unchanged; no spills. - Byte identity (whole-frame runs, synthetic model), from the DXIL of this tree, at 64x40 -> 96x60, 853x481 -> 1279x721 and 1706x960 -> 2560x1440 on the default fusion route, and at the two smaller sizes again with xd removed from WG_D3D12_FUSION (pm,xe,rv,xo,lc,ll,le): below 1.2 M render pixels the default route runs the xd form of the decoder (d1p12/swin_block_c64_add_dec0_xd_nc, unchanged by this commit), so the xd-less runs are the ones that launch the changed d1p12/swin_block_c64_add_dec0_nc (profiler dispatch lists of both routes at both sizes are saved with the runs). Each group has a same-session pair of base runs, the candidate and two controls built from this tree with only the helper changed (mask ~31u; address + 16). 93 dumps per run; 60 live buffers are compared: all except b0-b4.out, dec2.out, enc1.merged, post.display and frame-0 state *_prev / motion_history (unwritten or reused memory), dec0.B (unused with pack12) and dec1.out (dec1 runs inside dec0 on these routes and leaves out1 unwritten); dec0.C only on its active prefix, the first rw*rh*4 bytes, a shorter file is a failure; every expected dump must exist. Result, all five groups: base pair 60/60, candidate 60/60, both controls 24/60. - Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL. Summed GPU dispatch time (from the profiler-on runs: all valid dispatch durations of a frame summed; frames whose unchanged input_encoder kernel is more than 3 % above its run's minimum are dropped as low-clock, 266-272 of 300 kept per arm; unfiltered also given): 2597.1 -> 2495.5 us (-101.6 us, -3.9 %; unfiltered 2607.3 -> 2498.5). d1p12 437.6 -> 368.4 us, c64 block 288.4 -> 261.9 us. Profiler-on runs: busy shader-clock medians 3.04-3.12 GHz per run, throttle status zero in all 951 (base) and 936 (this change) busy samples. Host frame time (frame_io_timed wall time, median of 300 frames): profiler on 2.6857 -> 2.5777 ms; profiler off 2.7062 -> 2.5479 ms (paired per-round differences -111 us and -143 us). The profiler-off runs carry throttle status 2 (hotspot up to 88-89 C, busy-sample clock medians 3.13-3.16 GHz) on 327 of 460 busy samples in the base arm and 317 of 439 in this change's arm (71 % and 72 %), so the profiler-off host numbers are comparable between the arms but not unthrottled. The summed dispatch time can count overlapping kernels; the host time is the wall-clock measure. Not tested: the Windows driver; RDNA 3 (these kernels do not compile on the no-card gfx1100 route here). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
3 tasks done
…hantoslayer21#290 + this PR Recomputed with smx_selfcheck_digest.py on the merged tree (both PRs changed the same line). Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Owner
|
Note 🤖 Claude Opus 5.5 responding on behalf of phantoslayer21 I pushed one merge commit to this branch (47af6f7). It merges main, which now has #290 and #292, and sets All three kernel PRs were checked on the native Windows driver (RX 9070 XT, driver 32.0.32015.7029):
|
j0nsh1n
added a commit
to j0nsh1n/wronggpu
that referenced
this pull request
Nov 20, 2026
…ned sources phantoslayer21#294 changed the same kSourceDigest line. The value is recomputed with clean/k/dx12/smx_selfcheck_digest.py on the merged tree (stale() is None; test_k_smx_selfcheck_digest passes). No other conflict. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
j0nsh1n
added a commit
to j0nsh1n/wronggpu
that referenced
this pull request
Nov 20, 2026
…inux launcher clean/installer/CMakeLists.txt conflicted with phantoslayer21#291: main links rstrtmgr for non-MSVC test builds, while this branch already names rstrtmgr for every WIN32 build of wg_core (record.cpp's #pragma comment(lib) is read by MSVC only). The branch's version is kept; it covers phantoslayer21#291's case. Verified: fresh Linux build (WRONGGPU_TESTING=ON, Release, Ninja), ctest 24/24 passed, launcher-native included. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
phantoslayer21
pushed a commit
that referenced
this pull request
Nov 20, 2026
…kernels (-52.8 us summed dispatch time at 1440p on RADV, byte-identical) (#295) The wide stage kernels (k_stagew2_e3 / d3 / e4 / d4, k_stagew4_e4 / d4 / m, k_block_wide.hlsl) and the phase kernels of the split stage m (k_split_m_*, k_block_split.hlsl) load their weight, bias and activation fragments through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits every 16-byte load into four global_load_b32 with four address additions: k_stagew2_e3 compiled to 1236 dword loads and no wider one, k_stagew2_d3 to 1276, k_stagew4_e4 to 1044, k_stagew4_d4 to 1040. The same pattern as the narrow stages; the wide stages are already at 192 VGPRs and 8 waves, so only the load count is left to win. KW_LD_A16 (default 0 in k_wide.hlsli; set to 1 by k_block_wide.hlsl and k_block_split.hlsl before their includes) makes the new helper kw_ld128, which kw_ldg and the 13 direct 16-byte load sites of the two files use, clear the low 4 address bits. The premise: arena allocations and the packed section bases are 256-byte aligned (k_pack.cpp pads every section to 256 bytes; the runtime's allocator hands out 256-byte steps), the per-lane bases the kernels derive from them add 2 * (256 * tile + 8 * lane) bytes, and every remaining offset is a multiple of 16 bytes (the HIP kernels' own uint4 loads), so each masked address is the requested one. No arithmetic, summation order or accumulator chain changes. One side effect: k_split_m_w2 grows from 192 to 216 VGPRs (one wave per SIMD less) because its address arithmetic shrinks while more loaded data is live. It is a 4 us kernel and it got faster (4.04 -> 2.68 us); the frame is no slower with it (summed dispatch time 1485.6 us with it changed, 1487.1 us with it left at the old shader), so the define stays file-level instead of excluding one phase. k_wide.hlsli, k_block_wide.hlsl and k_block_split.hlsl are in the include closure of the softmax self-check digest, so kSourceDigest in clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own smx_selfcheck_digest.digest(); the HIP reference hashes in the header are unchanged. That line is also rewritten by the narrow-stage aligned-loads change (#290) and by the Swin / decoder one (#294, wg_platform.hlsli): the three conflict on it, and whichever lands last must recompute it from the combined sources. DXIL scope, from hashing a full build of this tree (Model M and K, 108 files) against main's: 19 files change: k_stagew2_d3, k_stagew2_e3, k_stagew2_d4, k_stagew2_e4, k_stagew4_d4, k_stagew4_e4, k_stagew4_m, their __rodd forms, and k_split_m_attn, k_split_m_attn__rodd, k_split_m_wo, k_split_m_w1, k_split_m_w2. The other 89 are identical, including every Model M kernel, the narrow stages, the input stage and the output kernels. The timed measurements below ran the 8 mix-form kernels that a default frame launches (k_stagew2_e3, k_stagew2_d3, k_stagew4_e4, k_stagew4_d4 and the four k_split_m phases), byte-identical to this tree's. The other 11 changed files were run in a second check from this tree's DXIL, below. Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile "Default" (overclocked): absolute times are on that profile; the interleaved arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only the named DXIL files swapped; a synthetic native D3D12 frame harness): - No-card gfx1201 instructions: k_stagew2_e3 6861 -> 5372 (-21.7 %), k_stagew2_d3 7092 -> 5599 (-21.1 %), k_stagew4_e4 5571 -> 4271 (-23.3 %), k_stagew4_d4 5649 -> 4309 (-23.7 %); memory instructions 1252 -> 325, 1288 -> 331, 1056 -> 273, 1048 -> 268; VGPRs unchanged, no spills. - Byte identity at 200x120 (5 frames), 1279x721 and 2560x1440 (3 frames): a same-session pair of base runs and the candidate, 17 files per frame (the buffers k.Np, k.Hc, k.L, k.Hh, k.x0, k.S0-S4, k.A, k.B, k.ctap, k.ctapT, k.scalars, exposure.pair and the filter output; k.N is only written with k_head=full), every file present, non-empty, the same length, compared over its full length: base pair 85/85, 51/51, 51/51, candidate equal in all of them, and the create-time softmax self-check passes on the new kernels (smx=mix split=m in the report). - Controls built from this tree with the mask changed to ~31u. (a) The wrong mask in the 8 mix-form kernels: the self-check rejects the mix form and the runtime falls back twice, to the round-to-odd ports and, because the poisoned k_split_m_wo / w1 / w2 phases are shared, to the one-launch stage m (smx=rodd in the report); the frame output is then produced by the surviving, unchanged shaders and equals the base, so this is a route-selection control, not a byte-difference control. (b) The wrong mask in the mix and the __rodd forms: create is refused at all three sizes ("the D3D12 softmax stages are not exact on this driver"). (c) The wrong mask in k_stagew2_e4__rodd (with the mix form of k_stagew2_e3 poisoned so that the round-to-odd route is taken): create is refused at 200x120 and 3840x2160 naming "round-to-odd form k_stagew2_e4__rodd@1536x1280". (d) The wrong mask in the mix k_stagew2_e4 only, at 3840x2160: the self-check rejects it and the runtime takes the round-to-odd ports; a route-selection control, no byte comparison. - Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL. Summed GPU dispatch time (profiler-on runs; all valid dispatch durations of a frame summed, the timing_barrier dispatch excluded (0.36 us, including it gives 1538.8 -> 1486.0 us); frames whose unchanged k_in_tiled_uniform kernel is more than 3 % above its run's minimum are dropped as low-clock, 271 and 282 of 300 kept; unfiltered 1537.8 -> 1485.6 us): 1538.4 -> 1485.6 us (-52.8 us, -3.4 %). k_stagew2_e3 46.3 -> 34.2, k_stagew2_d3 40.6 -> 28.7, k_stagew4_e4 39.4 -> 27.9, k_stagew4_d4 38.3 -> 26.6 us; the split phases attn 4.12 -> 3.80, wo 2.80 -> 1.82, w1 3.76 -> 2.84, w2 4.04 -> 2.68 us; every other kernel unchanged within 0.6 us. Two further measures per arm, both medians of 300 frames with paired per-round differences: (1) the GPU frame span, the sum of the runtime's four stage timestamp intervals (exposure, input, network, output): profiler on 1.6302 -> 1.5777 ms (-52.5 us; paired median -52.2 us, 5 of 5 rounds faster), profiler off 1.6450 -> 1.6153 ms (-29.7 us; paired median -28.5 us, 5 of 5). (2) The host wall-clock time of each frame call (QueryPerformanceCounter around it): profiler on 4.7037 -> 4.6440 ms (-59.7 us; paired -59.5 us, 5 of 5; inflated by the profiler's readback), profiler off 1.7722 -> 1.7418 ms (-30.4 us; paired -36.0 us, 5 of 5). Busy-sample shader clock about 2.8 GHz with the profiler on (throttle status zero in all samples) and 3.13-3.15 GHz with it off; the profiler-off runs carry throttle status on about 61 % of busy samples in every arm (236 of 388 base, 229 of 382 this change), comparable between the arms. The summed dispatch time can count overlapping work. - The 11 changed files the timing did not use (the __rodd forms, k_stagew2_d4, k_stagew2_e4, k_stagew4_m), from this tree's complete DXIL, each frame compared byte for byte (17 files per frame, full lengths) with main's build on the same route, a same-session pair of main runs as the baseline: * mix route, all 19 changed files, 200x120, 1279x721, 2560x1440 and a 3840x2160 output (1920x1080 render; its base map has more than 24 windows, so e4 / d4 take the NG = 2 kernels k_stagew2_e4 / k_stagew2_d4): equal in all four, smx=mix; the 3840x2160 frames launch k_stagew2_d4 / e4. * round-to-odd route, forced without a developer build by replacing the mix-form k_stagew2_e3 with the wrong-mask build (the self-check rejects it and takes the round-to-odd ports, which passed it: smx=rodd): equal at 200x120, 2560x1440 and 3840x2160 with the split stage m (k_split_m_attn__rodd, k_stagew2_e3 / d3 / d4 / e4 __rodd launched at 3840x2160), and with stage m as one launch (the mix-form k_split_m_wo also replaced; k_stagew4_m__rodd launched) at 200x120 and 3840x2160. * mix route with the one-launch stage m (only k_split_m_wo replaced, so the split form fails its check and stage m takes the one launch k_stagew4_m): equal at 200x120 and 1279x721, smx=mix. Dispatch lists of one frame are saved for the 3840x2160 mix, round-to-odd split and round-to-odd one-launch runs and for the 200x120 one-launch mix run. The NG = 4 round-to-odd shaders k_stagew4_e4__rodd / k_stagew4_d4__rodd have no dispatch list: their use in the 200x120 and 2560x1440 round-to-odd frames follows from the stage routing and the runs' reports, and the create-time self-check dispatches them (512x384 cases). The create-time self-check runs both rounding forms of each case on the selected stage-m route: with the split on (the default) the 17 changed shaders other than k_stagew4_m and k_stagew4_m__rodd run at every start and are compared with the HIP reference hashes, and k_stagew4_m / __rodd are not dispatched at create; they are covered by the one-launch frame runs above (and run at create when the split is off or its check fails). Not tested: the Windows driver; RDNA 3 (the K kernels do not compile on the no-card gfx1100 route here). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An exact speed-up for two Model M kernels on the RX 9070 XT under Linux (RADV through vkd3d-proton):
swin_block_c64_ncand the fused decoderd1p12/swin_block_c64_add_dec0_nc. Output bytes are unchanged.wg_ld128andwg_ld4fread throughRWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits each 16-byte load into four dword loads; the fused decoder compiles to 398global_load_b32and no wider load. A new macroWG_LD_A16(default 0) clears the low 4 address bits in those two helpers. It is set only on these two kernels'@wg-kernellines, because masking every Swin kernel pushed two c96 kernels from 192 to 256 VGPRs under ACO. Every base here is a 256-byte aligned arena allocation, the packed sections are 16-byte multiples and pixel pitches are multiples of 64 bytes, so no address changes. Same idea as #290 for Model K and #280's 8-byte loads inpost_tf.The scope is two files:
wg_platform.hlsliis in Model K's softmax self-check digest, sokSourceDigestis recomputed withsmx_selfcheck_digest.py. The HIP reference hashes do not change. Model K (D3D12): 16-byte aligned loads in the narrow stage kernels (-24 us at 1440p on RADV, byte-identical) #290 changes the same line: whichever of the two merges second needs that one line recomputed on the combined tree.Measured on main 11d3bbc with only this commit, RX 9070 XT, Wine + vkd3d-proton + RADV, synthetic model, overclocked profile. Both arms use one runtime DLL; only the two DXIL files are swapped.
xdremoved fromWG_D3D12_FUSION, because below 1.2 M render pixels the default route uses the_xddecoder form, which this change doesn't touch. Two wrong-address controls are caught (24/60).Not tested: the Windows driver and RDNA 3. Reviewed before opening by two other models.
Checklist
Signed-off-by:line (DCO 1.1).REUSE.toml(no new files).