Skip to content

Model K (D3D12): 16-byte aligned loads in the wide stages (-53 us GPU time per frame at 1440p on RADV, byte-identical) - #295

Merged
phantoslayer21 merged 3 commits into
phantoslayer21:mainfrom
j0nsh1n:perf/k-wide-aligned-loads
Nov 20, 2026
Merged

phantoslayer21 merged 3 commits into
phantoslayer21:mainfrom
j0nsh1n:perf/k-wide-aligned-loads

Conversation

@j0nsh1n

@j0nsh1n j0nsh1n commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

An exact speed-up for Model K's wide stages on the RX 9070 XT under Linux (RADV through vkd3d-proton). It is the wide-stage counterpart of #290, which covers the narrow stages. Output bytes are unchanged.

The wide stages' 16-byte loads go through Load4, which is known to be 4-byte aligned only, so ACO splits each one into four dword loads. The wide stages issued only global_load_b32. KW_LD_A16 (default 0, set in k_block_wide.hlsl and k_block_split.hlsl) clears the low 4 address bits at the 13 kw_ld128 sites. Allocations and packed section bases are 256-byte aligned, and the per-lane bases and offsets are 16-byte multiples, so no address changes. Instructions in the four timed wide stages drop 21-24 %, with VGPRs unchanged at 192 and no spills.

k_split_m_w2 goes from 192 to 216 VGPRs, and is still faster (4.04 -> 2.68 us).

Scope:

Measured on main 11d3bbc with only this commit, synthetic Model K, overclocked profile; both arms use one runtime DLL.

  • Byte identity: checked against main on the same route.
    • Mix route at 200x120, 1279x721, 2560x1440 and 3840x2160 (the 4K frame runs the NG=2 kernels).
    • Round-to-odd route and the one-launch stage m, both reached through the runtime's own self-check fallbacks.
    • The create-time softmax self-check passes (smx=mix). A wrong mask in the __rodd forms is refused at create.
  • Timing at 1706x960 -> 2560x1440, 5 interleaved rounds:
    • summed GPU dispatch time 1538.4 -> 1485.6 us (-52.8 us, -3.4 %);
    • each wide stage is about 12 us faster;
    • GPU frame span -52.5 us with the profiler on, -29.7 us with it off;
    • host wall time -30.4 us with the profiler off.
    • With the profiler off, both arms ran with the card's throttle status set on about 60 % of samples, at equal clocks.

Not tested: the Windows driver and RDNA 3. Reviewed before opening by two other models; their findings were about the commit message and our test tooling, and are fixed.

Checklist

…kernels (-52.8 us summed dispatch time at 1440p on RADV, byte-identical)

The wide stage kernels (k_stagew2_e3 / d3 / e4 / d4, k_stagew4_e4 / d4 / m,
k_block_wide.hlsl) and the phase kernels of the split stage m (k_split_m_*,
k_block_split.hlsl) load their weight, bias and activation fragments through
RWByteAddressBuffer.Load4, which is known to be 4-byte aligned only.
RADV/ACO therefore splits every 16-byte load into four global_load_b32 with
four address additions: k_stagew2_e3 compiled to 1236 dword loads and no
wider one, k_stagew2_d3 to 1276, k_stagew4_e4 to 1044, k_stagew4_d4 to 1040.
The same pattern as the narrow stages; the wide stages are already at 192
VGPRs and 8 waves, so only the load count is left to win.

KW_LD_A16 (default 0 in k_wide.hlsli; set to 1 by k_block_wide.hlsl and
k_block_split.hlsl before their includes) makes the new helper kw_ld128, which
kw_ldg and the 13 direct 16-byte load sites of the two files use, clear the
low 4 address bits. The premise: arena allocations and the packed section
bases are 256-byte aligned (k_pack.cpp pads every section to 256 bytes; the
runtime's allocator hands out 256-byte steps), the per-lane bases the kernels
derive from them add 2 * (256 * tile + 8 * lane) bytes, and every remaining
offset is a multiple of 16 bytes (the HIP kernels' own uint4 loads), so each
masked address is the requested one. No arithmetic, summation order or
accumulator chain changes.

One side effect: k_split_m_w2 grows from 192 to 216 VGPRs (one wave per SIMD
less) because its address arithmetic shrinks while more loaded data is live.
It is a 4 us kernel and it got faster (4.04 -> 2.68 us); the frame is no slower
with it (summed dispatch time 1485.6 us with it changed, 1487.1 us with it left
at the old shader), so the define stays file-level instead of excluding one
phase.

k_wide.hlsli, k_block_wide.hlsl and k_block_split.hlsl are in the include
closure of the softmax self-check digest, so kSourceDigest in
clean/k/runtime/k_smx_selfcheck.h is recomputed with the repository's own
smx_selfcheck_digest.digest(); the HIP reference hashes in the header are
unchanged. That line is also rewritten by the narrow-stage aligned-loads change
(phantoslayer21#290) and by the Swin / decoder one (phantoslayer21#294, wg_platform.hlsli): the three
conflict on it, and whichever lands last must recompute it from the combined
sources.

DXIL scope, from hashing a full build of this tree (Model M and K, 108 files)
against main's: 19 files change: k_stagew2_d3, k_stagew2_e3, k_stagew2_d4,
k_stagew2_e4, k_stagew4_d4, k_stagew4_e4, k_stagew4_m, their __rodd forms, and
k_split_m_attn, k_split_m_attn__rodd, k_split_m_wo, k_split_m_w1,
k_split_m_w2. The other 89 are identical, including every Model M kernel, the
narrow stages, the input stage and the output kernels. The timed measurements
below ran the 8 mix-form kernels that a default frame launches (k_stagew2_e3,
k_stagew2_d3, k_stagew4_e4, k_stagew4_d4 and the four k_split_m phases),
byte-identical to this tree's. The other 11 changed files were run in a second
check from this tree's DXIL, below.

Checks (RX 9070 XT, gfx1201, Linux/RADV, vkd3d-proton under Wine, LACT profile
"Default" (overclocked): absolute times are on that profile; the interleaved
arms are comparable; all on upstream main 11d3bbc, one runtime DLL, only the
named DXIL files swapped; a synthetic native D3D12 frame harness):
- No-card gfx1201 instructions: k_stagew2_e3 6861 -> 5372 (-21.7 %),
  k_stagew2_d3 7092 -> 5599 (-21.1 %), k_stagew4_e4 5571 -> 4271 (-23.3 %),
  k_stagew4_d4 5649 -> 4309 (-23.7 %); memory instructions 1252 -> 325,
  1288 -> 331, 1056 -> 273, 1048 -> 268; VGPRs unchanged, no spills.
- Byte identity at 200x120 (5 frames), 1279x721 and 2560x1440 (3 frames): a
  same-session pair of base runs and the candidate, 17 files per frame (the
  buffers k.Np, k.Hc, k.L, k.Hh, k.x0, k.S0-S4, k.A, k.B, k.ctap, k.ctapT,
  k.scalars, exposure.pair and the filter output; k.N is only written with
  k_head=full), every file present, non-empty, the same length, compared over
  its full length: base pair 85/85, 51/51, 51/51, candidate equal in all of
  them, and the create-time softmax self-check passes on the new kernels
  (smx=mix split=m in the report).
- Controls built from this tree with the mask changed to ~31u. (a) The wrong
  mask in the 8 mix-form kernels: the self-check rejects the mix form and the
  runtime falls back twice, to the round-to-odd ports and, because the poisoned
  k_split_m_wo / w1 / w2 phases are shared, to the one-launch stage m
  (smx=rodd in the report); the frame output is then produced by the
  surviving, unchanged shaders and equals the base, so this is a route-selection
  control, not a byte-difference control. (b) The wrong mask in the mix and the
  __rodd forms: create is refused at all three sizes ("the D3D12 softmax stages
  are not exact on this driver"). (c) The wrong mask in k_stagew2_e4__rodd
  (with the mix form of k_stagew2_e3 poisoned so that the round-to-odd route is
  taken): create is refused at 200x120 and 3840x2160 naming
  "round-to-odd form k_stagew2_e4__rodd@1536x1280". (d) The wrong mask in the mix
  k_stagew2_e4 only, at 3840x2160: the self-check rejects it and the runtime takes
  the round-to-odd ports; a route-selection control, no byte comparison.
- Timing, 5 interleaved rounds x 60 frames at 1706x960 -> 2560x1440, same DLL.
  Summed GPU dispatch time (profiler-on runs; all valid dispatch durations of a
  frame summed, the timing_barrier dispatch excluded (0.36 us, including it
  gives 1538.8 -> 1486.0 us); frames whose unchanged k_in_tiled_uniform kernel
  is more than 3 % above its run's minimum are dropped as low-clock, 271 and 282
  of 300 kept; unfiltered 1537.8 -> 1485.6 us): 1538.4 -> 1485.6 us (-52.8 us,
  -3.4 %). k_stagew2_e3 46.3 -> 34.2, k_stagew2_d3 40.6 -> 28.7, k_stagew4_e4
  39.4 -> 27.9, k_stagew4_d4 38.3 -> 26.6 us; the split phases attn 4.12 ->
  3.80, wo 2.80 -> 1.82, w1 3.76 -> 2.84, w2 4.04 -> 2.68 us; every other
  kernel unchanged within 0.6 us. Two further measures per arm, both medians of
  300 frames with paired per-round differences: (1) the GPU frame span, the sum
  of the runtime's four stage timestamp intervals (exposure, input, network,
  output): profiler on 1.6302 -> 1.5777 ms (-52.5 us; paired median -52.2 us,
  5 of 5 rounds faster), profiler off 1.6450 -> 1.6153 ms (-29.7 us; paired
  median -28.5 us, 5 of 5). (2) The host wall-clock time of each frame call
  (QueryPerformanceCounter around it): profiler on 4.7037 -> 4.6440 ms (-59.7 us;
  paired -59.5 us, 5 of 5; inflated by the profiler's readback), profiler off
  1.7722 -> 1.7418 ms (-30.4 us; paired -36.0 us, 5 of 5). Busy-sample shader
  clock about 2.8 GHz with the profiler on (throttle status zero in all samples)
  and 3.13-3.15 GHz with it off; the profiler-off runs carry throttle status on
  about 61 % of busy samples in every arm (236 of 388 base, 229 of 382 this
  change), comparable between the arms. The summed dispatch time can count
  overlapping work.
- The 11 changed files the timing did not use (the __rodd forms, k_stagew2_d4,
  k_stagew2_e4, k_stagew4_m), from this tree's complete DXIL, each frame
  compared byte for byte (17 files per frame, full lengths) with main's build
  on the same route, a same-session pair of main runs as the baseline:
  * mix route, all 19 changed files, 200x120, 1279x721, 2560x1440 and a
    3840x2160 output (1920x1080 render; its base map has more than 24 windows,
    so e4 / d4 take the NG = 2 kernels k_stagew2_e4 / k_stagew2_d4): equal in
    all four, smx=mix; the 3840x2160 frames launch k_stagew2_d4 / e4.
  * round-to-odd route, forced without a developer build by replacing the
    mix-form k_stagew2_e3 with the wrong-mask build (the self-check rejects it
    and takes the round-to-odd ports, which passed it: smx=rodd): equal at
    200x120, 2560x1440 and 3840x2160 with the split stage m (k_split_m_attn__rodd,
    k_stagew2_e3 / d3 / d4 / e4 __rodd launched at 3840x2160), and with stage m as
    one launch (the mix-form k_split_m_wo also replaced; k_stagew4_m__rodd
    launched) at 200x120 and 3840x2160.
  * mix route with the one-launch stage m (only k_split_m_wo replaced, so the
    split form fails its check and stage m takes the one launch k_stagew4_m):
    equal at 200x120 and 1279x721, smx=mix.
  Dispatch lists of one frame are saved for the 3840x2160 mix, round-to-odd
  split and round-to-odd one-launch runs and for the 200x120 one-launch mix run.
  The NG = 4 round-to-odd shaders k_stagew4_e4__rodd / k_stagew4_d4__rodd have no
  dispatch list: their use in the 200x120 and 2560x1440 round-to-odd frames
  follows from the stage routing and the runs' reports, and the create-time
  self-check dispatches them (512x384 cases). The create-time self-check runs
  both rounding forms of each case on the selected stage-m route: with the split
  on (the default) the 17 changed shaders other than k_stagew4_m and
  k_stagew4_m__rodd run at every start and are compared with the HIP reference
  hashes, and k_stagew4_m / __rodd are not dispatched at create; they are
  covered by the one-launch frame runs above (and run at create when the split
  is off or its check fails).

Not tested: the Windows driver; RDNA 3 (the K kernels do not compile on the
no-card gfx1100 route here).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
phantoslayer21#293); recompute kSourceDigest on the combined sources

phantoslayer21#290 changed the same kSourceDigest line in clean/k/runtime/k_smx_selfcheck.h.
The value is recomputed with clean/k/dx12/smx_selfcheck_digest.py on the
merged tree (stale() is None; test_k_smx_selfcheck_digest passes). No other
conflict.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ned sources

phantoslayer21#294 changed the same kSourceDigest line. The value is recomputed with
clean/k/dx12/smx_selfcheck_digest.py on the merged tree (stale() is None;
test_k_smx_selfcheck_digest passes). No other conflict.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
@phantoslayer21
phantoslayer21 merged commit 8775610 into phantoslayer21:main Nov 20, 2026
10 checks passed
@j0nsh1n
j0nsh1n deleted the perf/k-wide-aligned-loads branch November 21, 2026 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants