Skip to content

Model L: C = 32 Swin block kernel (HIP) from Model M's block template - #102

Merged
phantoslayer21 merged 2 commits into
mainfrom
model-l/c32-block-gpu
Nov 13, 2026
Merged

phantoslayer21 merged 2 commits into
mainfrom
model-l/c32-block-gpu

Conversation

@phantoslayer21

Copy link
Copy Markdown
Owner

Step P8 of Model L. Refs #89

  • clean/l/kernels/l_swin.hip includes clean/kernels/swin.hip for its templates only (WG_SWIN_NO_EXPORTS, as dec1_dec0.hip does) and exports l_swin_block_c32 / l_swin_block_c32_nc (C = 32, 2 heads, MLP 128). The arithmetic, packed weights (clean/runner/packing.py pack_block) and launch geometry are Model M's block kernels'. It also exports l_swin_block_c64, the same template at Model M's width, used only to check that this file builds the template exactly as swin.co does.
  • clean/l/runner/l_c32_check.py: the GPU check below.

No existing file changes except clean/l/README.md; Model M's code objects are untouched.

What the GPU check proves

Run of 2026-11-13 on an RX 9070 XT: python clean/l/runner/l_c32_check.py --build <dir> --rocm <ROCm> --seeds 8, synthetic data only (48 cases: map sizes 5x7 to 48x64, window offsets (0,0), (2,0), (0,2), (2,2), random C = 32 weights and FP8 inputs).

  • l_swin_block_c32 against the NumPy oracle (clean/l/reference/oracle.py swin_block with the kernels' sigmoid GELU): 6 of 1,407,744 output values differ, each by exactly one FP8 step. No value is more than one step off and none is non-finite. The worst case has 0.0077 % of its values differing, and its mean absolute error is 2.4e-6 of the reference RMS. Model M's per-stage rule (at most 2 % of values more than one step off, mean error at most 1 % of RMS) holds in every case.
  • l_swin_block_c32_nc (no inner clamps, used when the weights' flags prove the range) gives byte-identical output to l_swin_block_c32 in all 48 cases.
  • l_swin_block_c64 (this file's build of the template) gives byte-identical output to Model M's swin_block_c64 from swin.co in all 48 cases.

What stays unproven while the FP8 WMMA accumulation rule is open

  • The kernel-order CPU model cannot yet reproduce the GPU's FP8 WMMA sums byte for byte in the general case. On the same GPU, probes show that sums of products without an accumulator are correctly rounded (exponent spreads up to 20), and that an accumulator 2 to 24 bits smaller than the largest product is truncated toward minus infinity. Random full-range cases still match no candidate rule. So the C = 32 kernel is checked against the float64 oracle by Model M's tolerance rule, not byte for byte against a CPU model.
  • The rare one-step differences above are therefore not attributed to a specific operation (WMMA sums or the FP32 vs float64 path of the softmax / RMSNorm / GELU).
  • Until the rule is settled, the HIP kernel on the GPU is the byte reference for its D3D12 port, as for Model M.

Tested

  • GPU check above.
  • python -m unittest discover -s clean/l/tests -t .: passes (CPU tests, unchanged by this PR).
  • python tools/ci/check_line_endings.py: 0 problems.

Not tested

  • The host lane emulator (it needs Linux).
  • Runtime use: the runtime does not launch Model L's kernels yet.

…its GPU check

clean/l/kernels/l_swin.hip includes clean/kernels/swin.hip for its templates
only (WG_SWIN_NO_EXPORTS, as dec1_dec0.hip does) and exports
l_swin_block_c32[_nc] (2 heads, MLP 128) with Model M's block arithmetic,
packed weights and launch geometry, plus l_swin_block_c64 (the template at
Model M's width) for the check that this file builds it as swin.co does.

clean/l/runner/l_c32_check.py (GPU, synthetic): the C = 32 kernel against the
oracle, _nc against the clamping kernel, and l_swin_block_c64 against Model
M's swin_block_c64.

Refs #89

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…rence; README rows

Refs #89

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 marked this pull request as ready for review November 13, 2026 23:04
@phantoslayer21
phantoslayer21 merged commit 378e9fe into main Nov 13, 2026
2 checks passed
@phantoslayer21
phantoslayer21 deleted the model-l/c32-block-gpu branch November 13, 2026 23:04
j0nsh1n pushed a commit to j0nsh1n/wronggpu that referenced this pull request Nov 14, 2026
…ts HIP kernel

l_swin.hlsl instantiates swin_block.hlsl as l_swin_block_c32_nc (C = 32, 2 heads,
MLP 128, HIP's GELU chain and MLP unit cut), the D3D12 port of phantoslayer21#102's HIP kernel.
swin_block.hlsl (and the generated fused_swin.hlsl) gain the RMS-norm scale for
C = 32 (1/32); Model M's DXILs are unchanged. vgpr_limits.json bounds the kernel at
96 VGPRs and build_dx12.py checks l_swin_* kernels too. kernel_replay.cpp runs single
kernels on the native D3D12 device from case files; l_d3d12_check.py drives HIP and
D3D12 on the same synthetic inputs and compares whole output buffers byte for byte,
with Model M's swin_block_c96_nc as a control.

Refs phantoslayer21#89

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant