Skip to content

Model M (D3D12): 16-byte aligned loads in the c64 Swin block and fused decoder (-102 us GPU time per frame at 1440p on RADV, byte-identical) - #294

Merged
phantoslayer21 merged 2 commits into
phantoslayer21:mainfrom
j0nsh1n:perf/m-swin-aligned-loads
Nov 20, 2026
Merged

phantoslayer21 merged 2 commits into
phantoslayer21:mainfrom
j0nsh1n:perf/m-swin-aligned-loads

Conversation

@j0nsh1n

@j0nsh1n j0nsh1n commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

An exact speed-up for two Model M kernels on the RX 9070 XT under Linux (RADV through vkd3d-proton): swin_block_c64_nc and the fused decoder d1p12/swin_block_c64_add_dec0_nc. Output bytes are unchanged.

wg_ld128 and wg_ld4f read through RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only. RADV/ACO therefore splits each 16-byte load into four dword loads; the fused decoder compiles to 398 global_load_b32 and no wider load. A new macro WG_LD_A16 (default 0) clears the low 4 address bits in those two helpers. It is set only on these two kernels' @wg-kernel lines, because masking every Swin kernel pushed two c96 kernels from 192 to 256 VGPRs under ACO. Every base here is a 256-byte aligned arena allocation, the packed sections are 16-byte multiples and pixel pitches are multiples of 64 bytes, so no address changes. Same idea as #290 for Model K and #280's 8-byte loads in post_tf.

The scope is two files:

Measured on main 11d3bbc with only this commit, RX 9070 XT, Wine + vkd3d-proton + RADV, synthetic model, overclocked profile. Both arms use one runtime DLL; only the two DXIL files are swapped.

  • Byte identity: 60 of 60 written dumps at 96x60, 1279x721 and 2560x1440, against two base runs from the same session. The two smaller sizes were run a second time with xd removed from WG_D3D12_FUSION, because below 1.2 M render pixels the default route uses the _xd decoder form, which this change doesn't touch. Two wrong-address controls are caught (24/60).
  • Timing at 1706x960 -> 2560x1440, profiler on:
    • summed GPU dispatch time 2597.1 -> 2495.5 us (-101.6 us, -3.9 %);
    • fused decoder 437.6 -> 368.4 us, c64 block 288.4 -> 261.9 us;
    • host frame time 2.686 -> 2.578 ms.
  • Profiler off: host frame time 2.706 -> 2.548 ms. Both arms ran with the card's throttle status set on about 70 % of samples, so those two numbers compare with each other but are not unthrottled times.

Not tested: the Windows driver and RDNA 3. Reviewed before opening by two other models.

Checklist

…_block_c64_nc, d1p12/swin_block_c64_add_dec0_nc; -101.6 us summed dispatch time at 1440p on RADV, byte-identical)

wg_ld128 and wg_ld4f (so wg_load8f, st_fetch's weight staging, the gamma /
bias tiles and the activation fragments) read through RWByteAddressBuffer.Load4,
which is known to be 4-byte aligned only. RADV/ACO therefore splits each
into four global_load_b32 with four address additions: d1p12 compiled to 398
dword loads and no wider one, swin_block_c64_nc to 584 memory instructions.

The new macro WG_LD_A16 (default 0, set through defines= on the two kernel
declarations) clears the low 4 address bits in those two helpers. The premise
holds for these kernels: every base is a per-stage arena allocation (256-byte
aligned, device_d3d12.cpp kNativeAlign), the packed block sections are 16-byte
multiples (L_G1 = 16, HEAD_BYTES / MLP_BYTES, the bias and gamma tiles), and
the activation pixel pitches are multiples of 64 bytes, so the masked address
is the requested one. No arithmetic, summation order or WMMA operand changes.

It is scoped to these two kernels because a first draft that masked the
helpers for every Swin kernel pushed swin_block_c96_pm_rev_nc and
swin_block_c96_add_xe_xo_nc from 192 to 256 VGPRs under ACO. With the default
0 every other kernel is unchanged: a full Model M build of this tree against
main's differs in exactly these two DXIL files (45 of 47 Model M files are
identical); the builds with Model K (106 of 108) and Model L (99 of 101) differ
in those same two files only, every K-specific and L-specific binary is
identical.

wg_platform.hlsli is in the include closure of Model K's softmax self-check
digest, so kSourceDigest in clean/k/runtime/k_smx_selfcheck.h is recomputed
with the repository's own smx_selfcheck_digest.digest() (the HIP reference
hashes in the header are unchanged: no K shader code changes).

Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT
profile "Default" (overclocked): absolute times are on that profile; the
interleaved arms are comparable; all on upstream main 11d3bbc, one runtime
DLL, only these two DXIL files swapped):
- No-card gfx1201: d1p12 4578 -> 4122 instructions, memory instructions
  538 -> 247, 192 VGPRs / 8 waves unchanged; swin_block_c64_nc 5441 -> 4921,
  584 -> 302, 240 VGPRs unchanged; no spills.
- Byte identity (whole-frame runs, synthetic model), from the DXIL of this tree, at
  64x40 -> 96x60, 853x481 -> 1279x721 and 1706x960 -> 2560x1440 on the default
  fusion route, and at the two smaller sizes again with xd removed from
  WG_D3D12_FUSION (pm,xe,rv,xo,lc,ll,le): below 1.2 M render pixels the
  default route runs the xd form of the decoder (d1p12/swin_block_c64_add_dec0_xd_nc,
  unchanged by this commit), so the xd-less runs are the ones that launch the
  changed d1p12/swin_block_c64_add_dec0_nc (profiler dispatch lists of both
  routes at both sizes are saved with the runs). Each group has a same-session
  pair of base runs, the candidate and two controls built from this tree with
  only the helper changed (mask ~31u; address + 16). 93 dumps per run; 60 live
  buffers are compared: all except b0-b4.out, dec2.out, enc1.merged,
  post.display and frame-0 state *_prev / motion_history (unwritten or reused
  memory), dec0.B (unused with pack12) and dec1.out (dec1 runs inside dec0 on
  these routes and leaves out1 unwritten); dec0.C only on its active prefix,
  the first rw*rh*4 bytes, a shorter file is a failure; every expected dump must
  exist. Result, all five groups: base pair 60/60, candidate 60/60, both
  controls 24/60.
- Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL.
  Summed GPU dispatch time (from the profiler-on runs: all valid dispatch
  durations of a frame summed; frames whose unchanged input_encoder kernel is
  more than 3 % above its run's minimum are dropped as low-clock, 266-272 of
  300 kept per arm; unfiltered also given): 2597.1 -> 2495.5 us (-101.6 us,
  -3.9 %; unfiltered 2607.3 -> 2498.5). d1p12 437.6 -> 368.4 us, c64 block
  288.4 -> 261.9 us. Profiler-on runs: busy shader-clock medians 3.04-3.12 GHz
  per run, throttle status zero in all 951 (base) and 936 (this change) busy
  samples. Host frame time (frame_io_timed wall time, median of 300 frames):
  profiler on 2.6857 -> 2.5777 ms; profiler off 2.7062 -> 2.5479 ms (paired
  per-round differences -111 us and -143 us). The profiler-off runs carry
  throttle status 2 (hotspot up to 88-89 C, busy-sample clock medians
  3.13-3.16 GHz) on 327 of 460 busy samples in the base arm and 317 of 439 in
  this change's arm (71 % and 72 %), so the profiler-off host numbers are
  comparable between the arms but not unthrottled. The summed dispatch time
  can count overlapping kernels; the host time is the wall-clock measure.

Not tested: the Windows driver; RDNA 3 (these kernels do not compile on the
no-card gfx1100 route here).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…hantoslayer21#290 + this PR

Recomputed with smx_selfcheck_digest.py on the merged tree (both PRs changed the same line).

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21

Copy link
Copy Markdown
Owner

Note

🤖 Claude Opus 5.5 responding on behalf of phantoslayer21

I pushed one merge commit to this branch (47af6f7). It merges main, which now has #290 and #292, and sets kSourceDigest to f6ae0495… for the combined tree. I recomputed that with smx_selfcheck_digest.py, and test_k_smx_selfcheck_digest.py passes. Your shader changes are untouched.

All three kernel PRs were checked on the native Windows driver (RX 9070 XT, driver 32.0.32015.7029):

#290 and #292 are merged. This one merges once CI is green.

@phantoslayer21
phantoslayer21 merged commit 6294fa3 into phantoslayer21:main Nov 20, 2026
10 checks passed
j0nsh1n added a commit to j0nsh1n/wronggpu that referenced this pull request Nov 20, 2026
…ned sources

phantoslayer21#294 changed the same kSourceDigest line. The value is recomputed with
clean/k/dx12/smx_selfcheck_digest.py on the merged tree (stale() is None;
test_k_smx_selfcheck_digest passes). No other conflict.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
j0nsh1n added a commit to j0nsh1n/wronggpu that referenced this pull request Nov 20, 2026
…inux launcher

clean/installer/CMakeLists.txt conflicted with phantoslayer21#291: main links rstrtmgr for
non-MSVC test builds, while this branch already names rstrtmgr for every
WIN32 build of wg_core (record.cpp's #pragma comment(lib) is read by MSVC
only). The branch's version is kept; it covers phantoslayer21#291's case.

Verified: fresh Linux build (WRONGGPU_TESTING=ON, Release, Ninja), ctest
24/24 passed, launcher-native included.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
@j0nsh1n
j0nsh1n deleted the perf/m-swin-aligned-loads branch November 20, 2026 09:24
phantoslayer21 pushed a commit that referenced this pull request Nov 20, 2026
…kernels (-52.8 us summed dispatch time at 1440p on RADV, byte-identical) (#295)

The wide stage kernels (k_stagew2_e3 / d3 / e4 / d4, k_stagew4_e4 / d4 / m,
k_block_wide.hlsl) and the phase kernels of the split stage m (k_split_m_*,
k_block_split.hlsl) load their weight, bias and activation fragments through
RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only.
RADV/ACO therefore splits every 16-byte load into four global_load_b32 with
four address additions: k_stagew2_e3 compiled to 1236 dword loads and no
wider one, k_stagew2_d3 to 1276, k_stagew4_e4 to 1044, k_stagew4_d4 to 1040.
The same pattern as the narrow stages; the wide stages are already at 192
VGPRs and 8 waves, so only the load count is left to win.

KW_LD_A16 (default 0 in k_wide.hlsli; set to 1 by k_block_wide.hlsl and
k_block_split.hlsl before their includes) makes the new helper kw_ld128, which
kw_ldg and the 13 direct 16-byte load sites of the two files use, clear the
low 4 address bits. The premise: arena allocations and the packed section
bases are 256-byte aligned (k_pack.cpp pads every section to 256 bytes; the
runtime's allocator hands out 256-byte steps), the per-lane bases the kernels
derive from them add 2 * (256 * tile + 8 * lane) bytes, and every remaining
offset is a multiple of 16 bytes (the HIP kernels' own uint4 loads), so each
masked address is the requested one. No arithmetic, summation order or
accumulator chain changes.

One side effect: k_split_m_w2 grows from 192 to 216 VGPRs (one wave per SIMD
less) because its address arithmetic shrinks while more loaded data is live.
It is a 4 us kernel and it got faster (4.04 -> 2.68 us); the frame is no slower
with it (summed dispatch time 1485.6 us with it changed, 1487.1 us with it left
at the old shader), so the define stays file-level instead of excluding one
phase.

k_wide.hlsli, k_block_wide.hlsl and k_block_split.hlsl are in the include
closure of the softmax self-check digest, so kSourceDigest in
clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own
smx_selfcheck_digest.digest(); the HIP reference hashes in the header are
unchanged. That line is also rewritten by the narrow-stage aligned-loads change
(#290) and by the Swin / decoder one (#294, wg_platform.hlsli): the three
conflict on it, and whichever lands last must recompute it from the combined
sources.

DXIL scope, from hashing a full build of this tree (Model M and K, 108 files)
against main's: 19 files change: k_stagew2_d3, k_stagew2_e3, k_stagew2_d4,
k_stagew2_e4, k_stagew4_d4, k_stagew4_e4, k_stagew4_m, their __rodd forms, and
k_split_m_attn, k_split_m_attn__rodd, k_split_m_wo, k_split_m_w1,
k_split_m_w2. The other 89 are identical, including every Model M kernel, the
narrow stages, the input stage and the output kernels. The timed measurements
below ran the 8 mix-form kernels that a default frame launches (k_stagew2_e3,
k_stagew2_d3, k_stagew4_e4, k_stagew4_d4 and the four k_split_m phases),
byte-identical to this tree's. The other 11 changed files were run in a second
check from this tree's DXIL, below.

Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile
"Default" (overclocked): absolute times are on that profile; the interleaved
arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only the
named DXIL files swapped; a synthetic native D3D12 frame harness):
- No-card gfx1201 instructions: k_stagew2_e3 6861 -> 5372 (-21.7 %),
  k_stagew2_d3 7092 -> 5599 (-21.1 %), k_stagew4_e4 5571 -> 4271 (-23.3 %),
  k_stagew4_d4 5649 -> 4309 (-23.7 %); memory instructions 1252 -> 325,
  1288 -> 331, 1056 -> 273, 1048 -> 268; VGPRs unchanged, no spills.
- Byte identity at 200x120 (5 frames), 1279x721 and 2560x1440 (3 frames): a
  same-session pair of base runs and the candidate, 17 files per frame (the
  buffers k.Np, k.Hc, k.L, k.Hh, k.x0, k.S0-S4, k.A, k.B, k.ctap, k.ctapT,
  k.scalars, exposure.pair and the filter output; k.N is only written with
  k_head=full), every file present, non-empty, the same length, compared over
  its full length: base pair 85/85, 51/51, 51/51, candidate equal in all of
  them, and the create-time softmax self-check passes on the new kernels
  (smx=mix split=m in the report).
- Controls built from this tree with the mask changed to ~31u. (a) The wrong
  mask in the 8 mix-form kernels: the self-check rejects the mix form and the
  runtime falls back twice, to the round-to-odd ports and, because the poisoned
  k_split_m_wo / w1 / w2 phases are shared, to the one-launch stage m
  (smx=rodd in the report); the frame output is then produced by the
  surviving, unchanged shaders and equals the base, so this is a route-selection
  control, not a byte-difference control. (b) The wrong mask in the mix and the
  __rodd forms: create is refused at all three sizes ("the D3D12 softmax stages
  are not exact on this driver"). (c) The wrong mask in k_stagew2_e4__rodd
  (with the mix form of k_stagew2_e3 poisoned so that the round-to-odd route is
  taken): create is refused at 200x120 and 3840x2160 naming
  "round-to-odd form k_stagew2_e4__rodd@1536x1280". (d) The wrong mask in the mix
  k_stagew2_e4 only, at 3840x2160: the self-check rejects it and the runtime takes
  the round-to-odd ports; a route-selection control, no byte comparison.
- Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL.
  Summed GPU dispatch time (profiler-on runs; all valid dispatch durations of a
  frame summed, the timing_barrier dispatch excluded (0.36 us, including it
  gives 1538.8 -> 1486.0 us); frames whose unchanged k_in_tiled_uniform kernel
  is more than 3 % above its run's minimum are dropped as low-clock, 271 and 282
  of 300 kept; unfiltered 1537.8 -> 1485.6 us): 1538.4 -> 1485.6 us (-52.8 us,
  -3.4 %). k_stagew2_e3 46.3 -> 34.2, k_stagew2_d3 40.6 -> 28.7, k_stagew4_e4
  39.4 -> 27.9, k_stagew4_d4 38.3 -> 26.6 us; the split phases attn 4.12 ->
  3.80, wo 2.80 -> 1.82, w1 3.76 -> 2.84, w2 4.04 -> 2.68 us; every other
  kernel unchanged within 0.6 us. Two further measures per arm, both medians of
  300 frames with paired per-round differences: (1) the GPU frame span, the sum
  of the runtime's four stage timestamp intervals (exposure, input, network,
  output): profiler on 1.6302 -> 1.5777 ms (-52.5 us; paired median -52.2 us,
  5 of 5 rounds faster), profiler off 1.6450 -> 1.6153 ms (-29.7 us; paired
  median -28.5 us, 5 of 5). (2) The host wall-clock time of each frame call
  (QueryPerformanceCounter around it): profiler on 4.7037 -> 4.6440 ms (-59.7 us;
  paired -59.5 us, 5 of 5; inflated by the profiler's readback), profiler off
  1.7722 -> 1.7418 ms (-30.4 us; paired -36.0 us, 5 of 5). Busy-sample shader
  clock about 2.8 GHz with the profiler on (throttle status zero in all samples)
  and 3.13-3.15 GHz with it off; the profiler-off runs carry throttle status on
  about 61 % of busy samples in every arm (236 of 388 base, 229 of 382 this
  change), comparable between the arms. The summed dispatch time can count
  overlapping work.
- The 11 changed files the timing did not use (the __rodd forms, k_stagew2_d4,
  k_stagew2_e4, k_stagew4_m), from this tree's complete DXIL, each frame
  compared byte for byte (17 files per frame, full lengths) with main's build
  on the same route, a same-session pair of main runs as the baseline:
  * mix route, all 19 changed files, 200x120, 1279x721, 2560x1440 and a
    3840x2160 output (1920x1080 render; its base map has more than 24 windows,
    so e4 / d4 take the NG = 2 kernels k_stagew2_e4 / k_stagew2_d4): equal in
    all four, smx=mix; the 3840x2160 frames launch k_stagew2_d4 / e4.
  * round-to-odd route, forced without a developer build by replacing the
    mix-form k_stagew2_e3 with the wrong-mask build (the self-check rejects it
    and takes the round-to-odd ports, which passed it: smx=rodd): equal at
    200x120, 2560x1440 and 3840x2160 with the split stage m (k_split_m_attn__rodd,
    k_stagew2_e3 / d3 / d4 / e4 __rodd launched at 3840x2160), and with stage m as
    one launch (the mix-form k_split_m_wo also replaced; k_stagew4_m__rodd
    launched) at 200x120 and 3840x2160.
  * mix route with the one-launch stage m (only k_split_m_wo replaced, so the
    split form fails its check and stage m takes the one launch k_stagew4_m):
    equal at 200x120 and 1279x721, smx=mix.
  Dispatch lists of one frame are saved for the 3840x2160 mix, round-to-odd
  split and round-to-odd one-launch runs and for the 200x120 one-launch mix run.
  The NG = 4 round-to-odd shaders k_stagew4_e4__rodd / k_stagew4_d4__rodd have no
  dispatch list: their use in the 200x120 and 2560x1440 round-to-odd frames
  follows from the stage routing and the runs' reports, and the create-time
  self-check dispatches them (512x384 cases). The create-time self-check runs
  both rounding forms of each case on the selected stage-m route: with the split
  on (the default) the 17 changed shaders other than k_stagew4_m and
  k_stagew4_m__rodd run at every start and are compared with the HIP reference
  hashes, and k_stagew4_m / __rodd are not dispatched at create; they are
  covered by the one-launch frame runs above (and run at create when the split
  is off or its check fails).

Not tested: the Windows driver; RDNA 3 (the K kernels do not compile on the
no-card gfx1100 route here).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants