Skip to content

Model K (D3D12): 16-byte aligned loads in the narrow stage kernels (-24 us at 1440p on RADV, byte-identical) - #290

Merged
phantoslayer21 merged 1 commit into
phantoslayer21:mainfrom
j0nsh1n:perf/k-narrow-aligned-loads
Nov 20, 2026
Merged

phantoslayer21 merged 1 commit into
phantoslayer21:mainfrom
j0nsh1n:perf/k-narrow-aligned-loads

Conversation

@j0nsh1n

@j0nsh1n j0nsh1n commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

An exact speed-up for Model K's narrow stage kernels on the RX 9070 XT under Linux (RADV through vkd3d-proton). Output bytes are unchanged.

RWByteAddressBuffer.Load4 is known to be 4-byte aligned only, so RADV/ACO compiles every 16-byte fragment load in k_block.hlsl as four global_load_b32 with four address additions. Every base these loads use is 16-byte aligned (arena allocations and packed sections are 256-byte aligned; derived bases and offsets are multiples of 16), so clearing the low 4 address bits changes no address and lets ACO emit one global_load_b128. The switch KB_LD_A16 is set only in k_block.hlsl, the only shader that includes the two changed headers: 13 DXIL files change, the other 95 are byte-identical. kSourceDigest is recomputed with smx_selfcheck_digest.py; the HIP reference hashes do not change.

Measured on an RX 9070 XT, Wine + vkd3d-proton + RADV, synthetic Model K:

  • Byte identity against main at 200x120, 1279x721 and 2560x1440 (all K buffers and the filter output). The create-time softmax self-check passes on the new kernels (smx=mix). A deliberately wrong mask (~31u) is refused at create.
  • Kernel sum per frame at 1706x960 -> 2560x1440: 1540.1 -> 1515.8 us (-24.3 us, -1.6 %), 4 interleaved rounds, 240 frames per arm, shader clock about 2.97 GHz on an overclocked profile. Per kernel: k_stage2_d1 -7.9, k_stage2_d0_hp -6.1, k_stage_e2 -4.9, k_stage_d2 -4.3, k_stage2_e0 -3.0 us, k_stage2_e1 unchanged.
  • Instructions -10 to -16 % in six of these kernels; VGPRs unchanged, no spills.

Not tested: the native Windows AMD driver (it may already emit wide loads here), and RDNA 3 on the card. Same idea as the post_tf 8-byte loads in #280. Reviewed before opening by two other models.

Checklist

…24 us at 1440p on RADV, byte-identical)

The narrow stage kernels of k_block.hlsl (k_stage_e2/e0/e1/d1/d2,
k_stage2_e0/e1/d1/d0_hp/d0_hidden and the __rodd forms) load their weight,
bias and activation fragments through RWByteAddressBuffer.Load4, which is
known to be 4-byte aligned only. RADV/ACO therefore split every 16-byte load
into four global_load_b32 with four address additions (k_stage2_d1: 576
dword loads, no wider one). Arena allocations and packed section starts are
256-byte aligned (the D3D12 arena, k_pack.cpp kAlign), and every derived base
and byte offset these loads use is a multiple of 16 (k_common.hlsli already
documents "16-byte aligned"), so clearing the low 4 address bits changes no
address. LDS reads and all stores are not masked.

KB_LD_A16 (default 0, set to 1 only in k_block.hlsl) selects the masked form
in wgk_ld128_a16; wgk_ld8, kb_bias32 and kb_weight128 use it. k_block.hlsl is
the only shader that includes k_common.hlsli or k_block.hlsli, so only its 13
DXIL files change in the 108-shader build with Model K; the other 95 are
byte-identical.

kSourceDigest in k_smx_selfcheck.h is recomputed with the repository's own
smx_selfcheck_digest.digest(); the reference hashes in the header are HIP
outputs and do not change (no arithmetic, summation order or WMMA
accumulation order is touched).

Checks (RX 9070 XT, gfx1201, Wine + vkd3d-proton + RADV, synthetic Model K):
- Byte identity against the base build's output: all K buffers and the
  filter output, 85 / 51 / 51 files at 200x120, 1279x721 and 2560x1440, equal.
- Create-time softmax self-check passes on the new kernels (smx=mix split=m,
  no fallback). A deliberately wrong mask (~31u) is refused at create.
- No-card gfx1201: instructions -10.6 % (d1), -13.8 % (e1), -12 to -16 %
  (e2, d2, e0, d0_hp); k_stage2_d1's 576 global_load_b32 become 144
  global_load_b128; VGPRs unchanged (240 / 192), no spills.
- Timing (profiled GPU dispatch times), 4 interleaved rounds, all 240 timed
  frames per arm passed the 3 % input-duration filter; busy-sample shader
  clock median 2971-2974 MHz, no throttle; the card on an overclocked profile
  (core +150 MHz, -50 mV, 374 W cap): frame sum of kernels 1540.1 ->
  1515.8 us (-24.3 us, -1.6 %); k_stage2_d1 -7.9, d0_hp -6.1, e2 -4.9,
  d2 -4.3, e0 -3.0 us, e1 +0.1 us (unchanged).
- test_k_smx_selfcheck_digest, test_kernel_sets, test_k_wide_ng pass;
  line-endings check and git diff --check clean.

Not tested: the native Windows AMD driver, and RDNA 3 (gfx1100) on the card;
the no-card gfx1100 compile of these kernels fails here, so the effect there
is unmeasured.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
@phantoslayer21
phantoslayer21 merged commit ddda47e into phantoslayer21:main Nov 20, 2026
10 checks passed
phantoslayer21 added a commit to j0nsh1n/wronggpu that referenced this pull request Nov 20, 2026
…hantoslayer21#290 + this PR

Recomputed with smx_selfcheck_digest.py on the merged tree (both PRs changed the same line).

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
j0nsh1n added a commit to j0nsh1n/wronggpu that referenced this pull request Nov 20, 2026
phantoslayer21#293); recompute kSourceDigest on the combined sources

phantoslayer21#290 changed the same kSourceDigest line in clean/k/runtime/k_smx_selfcheck.h.
The value is recomputed with clean/k/dx12/smx_selfcheck_digest.py on the
merged tree (stale() is None; test_k_smx_selfcheck_digest passes). No other
conflict.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
j0nsh1n added a commit to j0nsh1n/wronggpu that referenced this pull request Nov 20, 2026
…inux launcher

clean/installer/CMakeLists.txt conflicted with phantoslayer21#291: main links rstrtmgr for
non-MSVC test builds, while this branch already names rstrtmgr for every
WIN32 build of wg_core (record.cpp's #pragma comment(lib) is read by MSVC
only). The branch's version is kept; it covers phantoslayer21#291's case.

Verified: fresh Linux build (WRONGGPU_TESTING=ON, Release, Ninja), ctest
24/24 passed, launcher-native included.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
@j0nsh1n
j0nsh1n deleted the perf/k-narrow-aligned-loads branch November 20, 2026 09:24
phantoslayer21 pushed a commit that referenced this pull request Nov 20, 2026
…kernels (-52.8 us summed dispatch time at 1440p on RADV, byte-identical) (#295)

The wide stage kernels (k_stagew2_e3 / d3 / e4 / d4, k_stagew4_e4 / d4 / m,
k_block_wide.hlsl) and the phase kernels of the split stage m (k_split_m_*,
k_block_split.hlsl) load their weight, bias and activation fragments through
RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only.
RADV/ACO therefore splits every 16-byte load into four global_load_b32 with
four address additions: k_stagew2_e3 compiled to 1236 dword loads and no
wider one, k_stagew2_d3 to 1276, k_stagew4_e4 to 1044, k_stagew4_d4 to 1040.
The same pattern as the narrow stages; the wide stages are already at 192
VGPRs and 8 waves, so only the load count is left to win.

KW_LD_A16 (default 0 in k_wide.hlsli; set to 1 by k_block_wide.hlsl and
k_block_split.hlsl before their includes) makes the new helper kw_ld128, which
kw_ldg and the 13 direct 16-byte load sites of the two files use, clear the
low 4 address bits. The premise: arena allocations and the packed section
bases are 256-byte aligned (k_pack.cpp pads every section to 256 bytes; the
runtime's allocator hands out 256-byte steps), the per-lane bases the kernels
derive from them add 2 * (256 * tile + 8 * lane) bytes, and every remaining
offset is a multiple of 16 bytes (the HIP kernels' own uint4 loads), so each
masked address is the requested one. No arithmetic, summation order or
accumulator chain changes.

One side effect: k_split_m_w2 grows from 192 to 216 VGPRs (one wave per SIMD
less) because its address arithmetic shrinks while more loaded data is live.
It is a 4 us kernel and it got faster (4.04 -> 2.68 us); the frame is no slower
with it (summed dispatch time 1485.6 us with it changed, 1487.1 us with it left
at the old shader), so the define stays file-level instead of excluding one
phase.

k_wide.hlsli, k_block_wide.hlsl and k_block_split.hlsl are in the include
closure of the softmax self-check digest, so kSourceDigest in
clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own
smx_selfcheck_digest.digest(); the HIP reference hashes in the header are
unchanged. That line is also rewritten by the narrow-stage aligned-loads change
(#290) and by the Swin / decoder one (#294, wg_platform.hlsli): the three
conflict on it, and whichever lands last must recompute it from the combined
sources.

DXIL scope, from hashing a full build of this tree (Model M and K, 108 files)
against main's: 19 files change: k_stagew2_d3, k_stagew2_e3, k_stagew2_d4,
k_stagew2_e4, k_stagew4_d4, k_stagew4_e4, k_stagew4_m, their __rodd forms, and
k_split_m_attn, k_split_m_attn__rodd, k_split_m_wo, k_split_m_w1,
k_split_m_w2. The other 89 are identical, including every Model M kernel, the
narrow stages, the input stage and the output kernels. The timed measurements
below ran the 8 mix-form kernels that a default frame launches (k_stagew2_e3,
k_stagew2_d3, k_stagew4_e4, k_stagew4_d4 and the four k_split_m phases),
byte-identical to this tree's. The other 11 changed files were run in a second
check from this tree's DXIL, below.

Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile
"Default" (overclocked): absolute times are on that profile; the interleaved
arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only the
named DXIL files swapped; a synthetic native D3D12 frame harness):
- No-card gfx1201 instructions: k_stagew2_e3 6861 -> 5372 (-21.7 %),
  k_stagew2_d3 7092 -> 5599 (-21.1 %), k_stagew4_e4 5571 -> 4271 (-23.3 %),
  k_stagew4_d4 5649 -> 4309 (-23.7 %); memory instructions 1252 -> 325,
  1288 -> 331, 1056 -> 273, 1048 -> 268; VGPRs unchanged, no spills.
- Byte identity at 200x120 (5 frames), 1279x721 and 2560x1440 (3 frames): a
  same-session pair of base runs and the candidate, 17 files per frame (the
  buffers k.Np, k.Hc, k.L, k.Hh, k.x0, k.S0-S4, k.A, k.B, k.ctap, k.ctapT,
  k.scalars, exposure.pair and the filter output; k.N is only written with
  k_head=full), every file present, non-empty, the same length, compared over
  its full length: base pair 85/85, 51/51, 51/51, candidate equal in all of
  them, and the create-time softmax self-check passes on the new kernels
  (smx=mix split=m in the report).
- Controls built from this tree with the mask changed to ~31u. (a) The wrong
  mask in the 8 mix-form kernels: the self-check rejects the mix form and the
  runtime falls back twice, to the round-to-odd ports and, because the poisoned
  k_split_m_wo / w1 / w2 phases are shared, to the one-launch stage m
  (smx=rodd in the report); the frame output is then produced by the
  surviving, unchanged shaders and equals the base, so this is a route-selection
  control, not a byte-difference control. (b) The wrong mask in the mix and the
  __rodd forms: create is refused at all three sizes ("the D3D12 softmax stages
  are not exact on this driver"). (c) The wrong mask in k_stagew2_e4__rodd
  (with the mix form of k_stagew2_e3 poisoned so that the round-to-odd route is
  taken): create is refused at 200x120 and 3840x2160 naming
  "round-to-odd form k_stagew2_e4__rodd@1536x1280". (d) The wrong mask in the mix
  k_stagew2_e4 only, at 3840x2160: the self-check rejects it and the runtime takes
  the round-to-odd ports; a route-selection control, no byte comparison.
- Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL.
  Summed GPU dispatch time (profiler-on runs; all valid dispatch durations of a
  frame summed, the timing_barrier dispatch excluded (0.36 us, including it
  gives 1538.8 -> 1486.0 us); frames whose unchanged k_in_tiled_uniform kernel
  is more than 3 % above its run's minimum are dropped as low-clock, 271 and 282
  of 300 kept; unfiltered 1537.8 -> 1485.6 us): 1538.4 -> 1485.6 us (-52.8 us,
  -3.4 %). k_stagew2_e3 46.3 -> 34.2, k_stagew2_d3 40.6 -> 28.7, k_stagew4_e4
  39.4 -> 27.9, k_stagew4_d4 38.3 -> 26.6 us; the split phases attn 4.12 ->
  3.80, wo 2.80 -> 1.82, w1 3.76 -> 2.84, w2 4.04 -> 2.68 us; every other
  kernel unchanged within 0.6 us. Two further measures per arm, both medians of
  300 frames with paired per-round differences: (1) the GPU frame span, the sum
  of the runtime's four stage timestamp intervals (exposure, input, network,
  output): profiler on 1.6302 -> 1.5777 ms (-52.5 us; paired median -52.2 us,
  5 of 5 rounds faster), profiler off 1.6450 -> 1.6153 ms (-29.7 us; paired
  median -28.5 us, 5 of 5). (2) The host wall-clock time of each frame call
  (QueryPerformanceCounter around it): profiler on 4.7037 -> 4.6440 ms (-59.7 us;
  paired -59.5 us, 5 of 5; inflated by the profiler's readback), profiler off
  1.7722 -> 1.7418 ms (-30.4 us; paired -36.0 us, 5 of 5). Busy-sample shader
  clock about 2.8 GHz with the profiler on (throttle status zero in all samples)
  and 3.13-3.15 GHz with it off; the profiler-off runs carry throttle status on
  about 61 % of busy samples in every arm (236 of 388 base, 229 of 382 this
  change), comparable between the arms. The summed dispatch time can count
  overlapping work.
- The 11 changed files the timing did not use (the __rodd forms, k_stagew2_d4,
  k_stagew2_e4, k_stagew4_m), from this tree's complete DXIL, each frame
  compared byte for byte (17 files per frame, full lengths) with main's build
  on the same route, a same-session pair of main runs as the baseline:
  * mix route, all 19 changed files, 200x120, 1279x721, 2560x1440 and a
    3840x2160 output (1920x1080 render; its base map has more than 24 windows,
    so e4 / d4 take the NG = 2 kernels k_stagew2_e4 / k_stagew2_d4): equal in
    all four, smx=mix; the 3840x2160 frames launch k_stagew2_d4 / e4.
  * round-to-odd route, forced without a developer build by replacing the
    mix-form k_stagew2_e3 with the wrong-mask build (the self-check rejects it
    and takes the round-to-odd ports, which passed it: smx=rodd): equal at
    200x120, 2560x1440 and 3840x2160 with the split stage m (k_split_m_attn__rodd,
    k_stagew2_e3 / d3 / d4 / e4 __rodd launched at 3840x2160), and with stage m as
    one launch (the mix-form k_split_m_wo also replaced; k_stagew4_m__rodd
    launched) at 200x120 and 3840x2160.
  * mix route with the one-launch stage m (only k_split_m_wo replaced, so the
    split form fails its check and stage m takes the one launch k_stagew4_m):
    equal at 200x120 and 1279x721, smx=mix.
  Dispatch lists of one frame are saved for the 3840x2160 mix, round-to-odd
  split and round-to-odd one-launch runs and for the 200x120 one-launch mix run.
  The NG = 4 round-to-odd shaders k_stagew4_e4__rodd / k_stagew4_d4__rodd have no
  dispatch list: their use in the 200x120 and 2560x1440 round-to-odd frames
  follows from the stage routing and the runs' reports, and the create-time
  self-check dispatches them (512x384 cases). The create-time self-check runs
  both rounding forms of each case on the selected stage-m route: with the split
  on (the default) the 17 changed shaders other than k_stagew4_m and
  k_stagew4_m__rodd run at every start and are compared with the HIP reference
  hashes, and k_stagew4_m / __rodd are not dispatched at create; they are
  covered by the one-launch frame runs above (and run at create when the split
  is off or its check fails).

Not tested: the Windows driver; RDNA 3 (the K kernels do not compile on the
no-card gfx1100 route here).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants