Repository navigation
Model K (D3D12): 16-byte aligned loads in the narrow stage kernels (-24 us at 1440p on RADV, byte-identical) - #290
Merged
phantoslayer21 merged 1 commit intoNov 20, 2026
Conversation
…24 us at 1440p on RADV, byte-identical) The narrow stage kernels of k_block.hlsl (k_stage_e2/e0/e1/d1/d2, k_stage2_e0/e1/d1/d0_hp/d0_hidden and the __rodd forms) load their weight, bias and activation fragments through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore split every 16-byte load into four global_load_b32 with four address additions (k_stage2_d1: 576 dword loads, no wider one). Arena allocations and packed section starts are 256-byte aligned (the D3D12 arena, k_pack.cpp kAlign), and every derived base and byte offset these loads use is a multiple of 16 (k_common.hlsli already documents "16-byte aligned"), so clearing the low 4 address bits changes no address. LDS reads and all stores are not masked. KB_LD_A16 (default 0, set to 1 only in k_block.hlsl) selects the masked form in wgk_ld128_a16; wgk_ld8, kb_bias32 and kb_weight128 use it. k_block.hlsl is the only shader that includes k_common.hlsli or k_block.hlsli, so only its 13 DXIL files change in the 108-shader build with Model K; the other 95 are byte-identical. kSourceDigest in k_smx_selfcheck.h is recomputed with the repository's own smx_selfcheck_digest.digest(); the reference hashes in the header are HIP outputs and do not change (no arithmetic, summation order or WMMA accumulation order is touched). Checks (RX 9070 XT, gfx1201, Wine + vkd3d-proton + RADV, synthetic Model K): - Byte identity against the base build's output: all K buffers and the filter output, 85 / 51 / 51 files at 200x120, 1279x721 and 2560x1440, equal. - Create-time softmax self-check passes on the new kernels (smx=mix split=m, no fallback). A deliberately wrong mask (~31u) is refused at create. - No-card gfx1201: instructions -10.6 % (d1), -13.8 % (e1), -12 to -16 % (e2, d2, e0, d0_hp); k_stage2_d1's 576 global_load_b32 become 144 global_load_b128; VGPRs unchanged (240 / 192), no spills. - Timing (profiled GPU dispatch times), 4 interleaved rounds, all 240 timed frames per arm passed the 3 % input-duration filter; busy-sample shader clock median 2971-2974 MHz, no throttle; the card on an overclocked profile (core +150 MHz, -50 mV, 374 W cap): frame sum of kernels 1540.1 -> 1515.8 us (-24.3 us, -1.6 %); k_stage2_d1 -7.9, d0_hp -6.1, e2 -4.9, d2 -4.3, e0 -3.0 us, e1 +0.1 us (unchanged). - test_k_smx_selfcheck_digest, test_kernel_sets, test_k_wide_ng pass; line-endings check and git diff --check clean. Not tested: the native Windows AMD driver, and RDNA 3 (gfx1100) on the card; the no-card gfx1100 compile of these kernels fails here, so the effect there is unmeasured. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
This was referenced Nov 19, 2026
phantoslayer21
added a commit
to j0nsh1n/wronggpu
that referenced
this pull request
Nov 20, 2026
…hantoslayer21#290 + this PR Recomputed with smx_selfcheck_digest.py on the merged tree (both PRs changed the same line). Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
j0nsh1n
added a commit
to j0nsh1n/wronggpu
that referenced
this pull request
Nov 20, 2026
phantoslayer21#293); recompute kSourceDigest on the combined sources phantoslayer21#290 changed the same kSourceDigest line in clean/k/runtime/k_smx_selfcheck.h. The value is recomputed with clean/k/dx12/smx_selfcheck_digest.py on the merged tree (stale() is None; test_k_smx_selfcheck_digest passes). No other conflict. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
j0nsh1n
added a commit
to j0nsh1n/wronggpu
that referenced
this pull request
Nov 20, 2026
…inux launcher clean/installer/CMakeLists.txt conflicted with phantoslayer21#291: main links rstrtmgr for non-MSVC test builds, while this branch already names rstrtmgr for every WIN32 build of wg_core (record.cpp's #pragma comment(lib) is read by MSVC only). The branch's version is kept; it covers phantoslayer21#291's case. Verified: fresh Linux build (WRONGGPU_TESTING=ON, Release, Ninja), ctest 24/24 passed, launcher-native included. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
phantoslayer21
pushed a commit
that referenced
this pull request
Nov 20, 2026
…kernels (-52.8 us summed dispatch time at 1440p on RADV, byte-identical) (#295) The wide stage kernels (k_stagew2_e3 / d3 / e4 / d4, k_stagew4_e4 / d4 / m, k_block_wide.hlsl) and the phase kernels of the split stage m (k_split_m_*, k_block_split.hlsl) load their weight, bias and activation fragments through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits every 16-byte load into four global_load_b32 with four address additions: k_stagew2_e3 compiled to 1236 dword loads and no wider one, k_stagew2_d3 to 1276, k_stagew4_e4 to 1044, k_stagew4_d4 to 1040. The same pattern as the narrow stages; the wide stages are already at 192 VGPRs and 8 waves, so only the load count is left to win. KW_LD_A16 (default 0 in k_wide.hlsli; set to 1 by k_block_wide.hlsl and k_block_split.hlsl before their includes) makes the new helper kw_ld128, which kw_ldg and the 13 direct 16-byte load sites of the two files use, clear the low 4 address bits. The premise: arena allocations and the packed section bases are 256-byte aligned (k_pack.cpp pads every section to 256 bytes; the runtime's allocator hands out 256-byte steps), the per-lane bases the kernels derive from them add 2 * (256 * tile + 8 * lane) bytes, and every remaining offset is a multiple of 16 bytes (the HIP kernels' own uint4 loads), so each masked address is the requested one. No arithmetic, summation order or accumulator chain changes. One side effect: k_split_m_w2 grows from 192 to 216 VGPRs (one wave per SIMD less) because its address arithmetic shrinks while more loaded data is live. It is a 4 us kernel and it got faster (4.04 -> 2.68 us); the frame is no slower with it (summed dispatch time 1485.6 us with it changed, 1487.1 us with it left at the old shader), so the define stays file-level instead of excluding one phase. k_wide.hlsli, k_block_wide.hlsl and k_block_split.hlsl are in the include closure of the softmax self-check digest, so kSourceDigest in clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own smx_selfcheck_digest.digest(); the HIP reference hashes in the header are unchanged. That line is also rewritten by the narrow-stage aligned-loads change (#290) and by the Swin / decoder one (#294, wg_platform.hlsli): the three conflict on it, and whichever lands last must recompute it from the combined sources. DXIL scope, from hashing a full build of this tree (Model M and K, 108 files) against main's: 19 files change: k_stagew2_d3, k_stagew2_e3, k_stagew2_d4, k_stagew2_e4, k_stagew4_d4, k_stagew4_e4, k_stagew4_m, their __rodd forms, and k_split_m_attn, k_split_m_attn__rodd, k_split_m_wo, k_split_m_w1, k_split_m_w2. The other 89 are identical, including every Model M kernel, the narrow stages, the input stage and the output kernels. The timed measurements below ran the 8 mix-form kernels that a default frame launches (k_stagew2_e3, k_stagew2_d3, k_stagew4_e4, k_stagew4_d4 and the four k_split_m phases), byte-identical to this tree's. The other 11 changed files were run in a second check from this tree's DXIL, below. Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile "Default" (overclocked): absolute times are on that profile; the interleaved arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only the named DXIL files swapped; a synthetic native D3D12 frame harness): - No-card gfx1201 instructions: k_stagew2_e3 6861 -> 5372 (-21.7 %), k_stagew2_d3 7092 -> 5599 (-21.1 %), k_stagew4_e4 5571 -> 4271 (-23.3 %), k_stagew4_d4 5649 -> 4309 (-23.7 %); memory instructions 1252 -> 325, 1288 -> 331, 1056 -> 273, 1048 -> 268; VGPRs unchanged, no spills. - Byte identity at 200x120 (5 frames), 1279x721 and 2560x1440 (3 frames): a same-session pair of base runs and the candidate, 17 files per frame (the buffers k.Np, k.Hc, k.L, k.Hh, k.x0, k.S0-S4, k.A, k.B, k.ctap, k.ctapT, k.scalars, exposure.pair and the filter output; k.N is only written with k_head=full), every file present, non-empty, the same length, compared over its full length: base pair 85/85, 51/51, 51/51, candidate equal in all of them, and the create-time softmax self-check passes on the new kernels (smx=mix split=m in the report). - Controls built from this tree with the mask changed to ~31u. (a) The wrong mask in the 8 mix-form kernels: the self-check rejects the mix form and the runtime falls back twice, to the round-to-odd ports and, because the poisoned k_split_m_wo / w1 / w2 phases are shared, to the one-launch stage m (smx=rodd in the report); the frame output is then produced by the surviving, unchanged shaders and equals the base, so this is a route-selection control, not a byte-difference control. (b) The wrong mask in the mix and the __rodd forms: create is refused at all three sizes ("the D3D12 softmax stages are not exact on this driver"). (c) The wrong mask in k_stagew2_e4__rodd (with the mix form of k_stagew2_e3 poisoned so that the round-to-odd route is taken): create is refused at 200x120 and 3840x2160 naming "round-to-odd form k_stagew2_e4__rodd@1536x1280". (d) The wrong mask in the mix k_stagew2_e4 only, at 3840x2160: the self-check rejects it and the runtime takes the round-to-odd ports; a route-selection control, no byte comparison. - Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL. Summed GPU dispatch time (profiler-on runs; all valid dispatch durations of a frame summed, the timing_barrier dispatch excluded (0.36 us, including it gives 1538.8 -> 1486.0 us); frames whose unchanged k_in_tiled_uniform kernel is more than 3 % above its run's minimum are dropped as low-clock, 271 and 282 of 300 kept; unfiltered 1537.8 -> 1485.6 us): 1538.4 -> 1485.6 us (-52.8 us, -3.4 %). k_stagew2_e3 46.3 -> 34.2, k_stagew2_d3 40.6 -> 28.7, k_stagew4_e4 39.4 -> 27.9, k_stagew4_d4 38.3 -> 26.6 us; the split phases attn 4.12 -> 3.80, wo 2.80 -> 1.82, w1 3.76 -> 2.84, w2 4.04 -> 2.68 us; every other kernel unchanged within 0.6 us. Two further measures per arm, both medians of 300 frames with paired per-round differences: (1) the GPU frame span, the sum of the runtime's four stage timestamp intervals (exposure, input, network, output): profiler on 1.6302 -> 1.5777 ms (-52.5 us; paired median -52.2 us, 5 of 5 rounds faster), profiler off 1.6450 -> 1.6153 ms (-29.7 us; paired median -28.5 us, 5 of 5). (2) The host wall-clock time of each frame call (QueryPerformanceCounter around it): profiler on 4.7037 -> 4.6440 ms (-59.7 us; paired -59.5 us, 5 of 5; inflated by the profiler's readback), profiler off 1.7722 -> 1.7418 ms (-30.4 us; paired -36.0 us, 5 of 5). Busy-sample shader clock about 2.8 GHz with the profiler on (throttle status zero in all samples) and 3.13-3.15 GHz with it off; the profiler-off runs carry throttle status on about 61 % of busy samples in every arm (236 of 388 base, 229 of 382 this change), comparable between the arms. The summed dispatch time can count overlapping work. - The 11 changed files the timing did not use (the __rodd forms, k_stagew2_d4, k_stagew2_e4, k_stagew4_m), from this tree's complete DXIL, each frame compared byte for byte (17 files per frame, full lengths) with main's build on the same route, a same-session pair of main runs as the baseline: * mix route, all 19 changed files, 200x120, 1279x721, 2560x1440 and a 3840x2160 output (1920x1080 render; its base map has more than 24 windows, so e4 / d4 take the NG = 2 kernels k_stagew2_e4 / k_stagew2_d4): equal in all four, smx=mix; the 3840x2160 frames launch k_stagew2_d4 / e4. * round-to-odd route, forced without a developer build by replacing the mix-form k_stagew2_e3 with the wrong-mask build (the self-check rejects it and takes the round-to-odd ports, which passed it: smx=rodd): equal at 200x120, 2560x1440 and 3840x2160 with the split stage m (k_split_m_attn__rodd, k_stagew2_e3 / d3 / d4 / e4 __rodd launched at 3840x2160), and with stage m as one launch (the mix-form k_split_m_wo also replaced; k_stagew4_m__rodd launched) at 200x120 and 3840x2160. * mix route with the one-launch stage m (only k_split_m_wo replaced, so the split form fails its check and stage m takes the one launch k_stagew4_m): equal at 200x120 and 1279x721, smx=mix. Dispatch lists of one frame are saved for the 3840x2160 mix, round-to-odd split and round-to-odd one-launch runs and for the 200x120 one-launch mix run. The NG = 4 round-to-odd shaders k_stagew4_e4__rodd / k_stagew4_d4__rodd have no dispatch list: their use in the 200x120 and 2560x1440 round-to-odd frames follows from the stage routing and the runs' reports, and the create-time self-check dispatches them (512x384 cases). The create-time self-check runs both rounding forms of each case on the selected stage-m route: with the split on (the default) the 17 changed shaders other than k_stagew4_m and k_stagew4_m__rodd run at every start and are compared with the HIP reference hashes, and k_stagew4_m / __rodd are not dispatched at create; they are covered by the one-launch frame runs above (and run at create when the split is off or its check fails). Not tested: the Windows driver; RDNA 3 (the K kernels do not compile on the no-card gfx1100 route here). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An exact speed-up for Model K's narrow stage kernels on the RX 9070 XT under Linux (RADV through vkd3d-proton). Output bytes are unchanged.
RWByteAddressBuffer.Load4is known to be 4-byte aligned only, so RADV/ACO compiles every 16-byte fragment load ink_block.hlslas fourglobal_load_b32with four address additions. Every base these loads use is 16-byte aligned (arena allocations and packed sections are 256-byte aligned; derived bases and offsets are multiples of 16), so clearing the low 4 address bits changes no address and lets ACO emit oneglobal_load_b128. The switchKB_LD_A16is set only ink_block.hlsl, the only shader that includes the two changed headers: 13 DXIL files change, the other 95 are byte-identical.kSourceDigestis recomputed withsmx_selfcheck_digest.py; the HIP reference hashes do not change.Measured on an RX 9070 XT, Wine + vkd3d-proton + RADV, synthetic Model K:
smx=mix). A deliberately wrong mask (~31u) is refused at create.k_stage2_d1-7.9,k_stage2_d0_hp-6.1,k_stage_e2-4.9,k_stage_d2-4.3,k_stage2_e0-3.0 us,k_stage2_e1unchanged.Not tested: the native Windows AMD driver (it may already emit wide loads here), and RDNA 3 on the card. Same idea as the
post_tf8-byte loads in #280. Reviewed before opening by two other models.Checklist
Signed-off-by:line (DCO 1.1).REUSE.toml(no new files).