Skip to content

Model K: exact speedup, stage m split across workgroups (-23.5 to -31.25 us at 1080p Q) - #283

Merged
phantoslayer21 merged 14 commits into
mainfrom
kl-speed/k-deep-split-opus
Nov 18, 2026
Merged

phantoslayer21 merged 14 commits into
mainfrom
kl-speed/k-deep-split-opus

Conversation

@phantoslayer21

@phantoslayer21 phantoslayer21 commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

What changed

Model K, lever deep-split (probe, stage m only). Stage m runs as 4 launches instead of one k_stagew4_m launch:

  • k_split_m_attn (and k_split_m_attn__rodd twin): attention per window and head
  • k_split_m_wo: Wo per output tile
  • k_split_m_w1: W1 per 2 hidden tiles
  • k_split_m_w2: W2 per output tile

Files: clean/k/dx12/kernels/k_block_split.hlsl (new), clean/k/runtime/runtime_k.cpp and runtime_k.h (split path, behind the existing k_split option), clean/dx12/kernel_sets.json, clean/dx12/isa_checks.json, clean/k/dx12/README.md, clean/k/dx12/smx_selfcheck_digest.py, clean/k/tests/test_k_smx_selfcheck_digest.py, clean/k/runtime/k_smx_selfcheck.h, plus the native tests under clean/k/tests/.

Review fixes

  • Automatic fallback. A split stage m (k_split=m, the default) that fails the create-time self-check no longer refuses create. Stage m takes the one k_stagew4_m launch (same bytes, same reference hashes), the self-check runs again on that path, and one create note says so. Create refuses only when the one-launch path fails too; the error names both failures. Developer hook: k_smx_test=split_fail (WG_DEV_DIAGNOSTICS builds).
  • Self-check digest rebind. kSourceDigest in k_smx_selfcheck.h is rebound to the digest of the reviewed tree (ef7da006...6fa92b; stale() is None). The 133 names and reference hashes are unchanged. Proof, D3D12 report hook (k_smx_test=report, dev-diagnostics build, no HIP at runtime): 133/133 mix and 133/133 round-to-odd matched the HIP reference hashes, 0 mismatches.
  • Native tests (new). Native-against-native, no HIP reference run: split=m vs k_split=off gives the same raw bytes of the output and every written buffer, history and exposure state after every frame, at 200x120 and odd 201x121, RGBA16F and dense RGB with auto-exposure, each side starting from its own sentinel. Launch trace checks the phase launches under split=m and the round-to-odd forms. k_split_m_wo/w1/w2 ISA check covers the no-softmax phases. Fallback test (split_fail) matches k_split=off frames.
  • Raw compare covers buffers of any byte length (odd sizes no longer stop at the 4-byte check); CPU tests cover lengths 1, 2, 3 mod 4 and 20698.
  • Split scratch is allocated before the k_fill block and is covered by its sentinel.
  • Tile size guard. k_block_split.hlsl refuses tile sizes that do not divide HT at compile time.
  • Docs and header. README window counts state the measured sizes; k_smx_selfcheck.h carries only the generator's text.

Test results: the new native tests and CPU tests are run by CI on this head (see the checks list). No native-test log for this head is attached to this PR.

Byte checks (main vs this branch, same output)

Final output fnv1a64 is identical in every run: 1080p Q 1b975fbba17891e3, 1440p Q 5fb3d1ab49ad8f90, both equal to REFS.json.

  • Native e2e, 600 frames, main vs branch: odd 1601x901, 1080p Q, 1440p Q identical, first differing frame null, nonfinite 0.
  • Element-wise, split on vs off, 120 frames: 0 differing values in every saved frame (odd, 1080p Q, 1440p Q).
  • Changed DXIL vs main: 5 new kernels, 154 unchanged. Mutant (Wo chain reversed) is refused by the create-time check.

Timing (kab.py, paired ABBA, 1500 frames, 4 blocks, A = main, B = this branch)

Range across sessions, stage m probe only:

  • 1080p Quality: -23.5 to -31.25 us (faster in every session; floor 5 us). Sessions: v1 -23.5 us, r2 (this head's build) -31.25 us.
  • 1440p Quality: -14.5 to -20.1 us (faster in both sessions). Sessions: v1 -20.1 us (one noisy A/A block), quiet re-run -14.5 us.

Records: C:\WrongGPU-clean-out\kl-speed\K\time\ (deep-split-opus-v1, deep-split-opus-v2-1440pQ, deep-split-r2).

Review verdict

Exactness passes on every leg above. Speed is faster at both shapes across sessions. The stage m probe is the only thing measured; the full 7-level split (65 us plan figure) is not measured. A cross-family second reader has not been run on this record.

Ready for review. Owner merges only.

Stage m runs as four launches on the native D3D12 device instead of one
k_stagew4_m launch with one 512-lane workgroup per window (6 at 1080p):
k_split_m_attn per (window, head), k_split_m_wo per (window, output tile),
k_split_m_w1 per (window, two hidden tiles), k_split_m_w2 per (window,
output tile), joined by dispatch boundaries. Each accumulator is the same
WMMA chain (operands, order, initial value) as k_stagew4_m and every FP16
value handed between launches is the one k_stagew4_m keeps, so the output
bytes are unchanged. Operand loads of each chain are fenced (opaque-zero
XOR) so the driver issues them together instead of one latency per WMMA.

KRuntime::launch redirects k_stagew4_m[__rodd] when k_split=m (default on
backend 3; k_split=off restores the one launch), so the create-time
self-check runs the split form for its m case against the unchanged HIP
reference hashes. Self-check kernel sources and regions are untouched
(kSourceDigest unchanged). isa_checks: one v_fma_mix per weight in
k_split_m_attn, none in the __rodd twin.
The native D3D12 device runs stage m's phase kernels (k_split_m_*) for the
self-check's m case (KRuntime::launch redirect), but the digest that binds
k_smx_selfcheck.h to its sources covered only k_stagew4_m. Add the five phase
kernels to the digest's kernel set and mark the k_split option, the redirect
and split_m / split_alloc as SMX-SELFCHECK-SOURCE regions; digest test cases
for k_block_split.hlsl and the redirect.

The header's kSourceDigest is NOT rebound here (the 133 hashes are unchanged;
rebinding needs coordinator / oversight OK): until then normal builds refuse
this tree and the lever builds --test-only.
The D3D12 report hook (k_smx_test=report, developer build) on the native
D3D12 device reproduced all 133 HIP reference hashes of
k_smx_selfcheck.h in both softmax forms (mix and round-to-odd), 0
mismatches, probe 0 / 0, create smx=mix. kSourceDigest is rebound to the
digest of this tree (smx_selfcheck_digest.digest()); the header's hashes
and names are unchanged. Record:
R:\WrongGPU-clean\tmp\kl-speed-K\deep-split\diag\smx-report-diag.json.
check_fallback_frames expected k_stagew4_m__rodd on backend 3, but with
k_split=m (the default) stage m runs as k_split_m_attn__rodd and the
k_split_m_wo / w1 / w2 launches. The expectation now follows the option:
the phase launches with split=m, and a k_split=off leg expects
k_stagew4_m__rodd. The forbidden mix forms follow the same rule (the
phases wo / w1 / w2 have no softmax and run in both forms).
…ails

A split stage m (k_split=m) that fails the create-time self-check no
longer refuses create. Stage m takes the one k_stagew4_m launch (the same
bytes and the same reference hashes), the self-check runs again on that
path, and one create note says so, in the pattern of the round-to-odd
note. Create refuses only when the one-launch path fails the self-check
too; the error names both failures.

Developer hook: k_smx_test=split_fail (WG_DEV_DIAGNOSTICS builds) makes
the split form count as failed, so the fallback can be tested. With
k_split=off it has no effect.
…e fallback

Native-against-native tests (no HIP reference run, so they run on the
developer device without the HIP legs):
- split=m (the default) against k_split=off: the same raw bytes of the
  output and of every written buffer, history and exposure state after
  every frame, at 200 x 120 and at an odd output size (201 x 121 from
  131 x 79), in RGBA16F and in dense RGB with auto-exposure. Each side
  starts from its own sentinel (0x5A, 0xC3), so a word neither side wrote
  fails.
- The launch trace: the phase launches with k_split=m, k_stagew4_m with
  k_split=off, and the round-to-odd rounding of the softmax stages with
  k_smx_test=mix_fail on both.
- A k_split value other than m or off fails create in one line.
- The kernels line names split=m only when stage m is split.
- k_smx_test=split_fail: create falls back to the one launch, one note
  names both paths, and the frames match k_split=off.

The both_fail refusal names the split form's failure as well as the
one-launch path's, and with k_split=off it does not.
The provenance lines added by hand to k_smx_selfcheck.h named a local
path and the internal review process. The generator
(clean/k/dx12/gen_k_smx_selfcheck.py) does not write them: its output
for the same entries and digest is now byte-identical to the header.
The provenance of the rebound digest is in its commit message.
…e HT

The W2 batches (HT / KS_B2) and the W1 groups (HT / KS_HB) need HT = 4
KW_C / 16 divisible by both sizes; a #error now refuses any other
choice at compile time. The comment ties the runtime's W1 grid (HT / 2
groups per window) to KS_HB = 2, so a change of KS_HB changes that
launch as well.
split_alloc now runs before the k_fill block, and the four scratch
buffers (Z, X1, A, H) join the fill list, so k_fill's sentinel covers
them as it covers the rest of what a frame writes. Their byte sizes are
recorded at allocation (split_bytes_). The W1 launch carries a comment
that ties its HT / 2 groups per window to KS_HB = 2 in the kernel source.
The three phases have no softmax, so they must have no v_fma_mix. The
check records that, next to the attention phase's count.
…easured

Stage m's window count is 6 to 24 at 1080p and 1440p output, more at 4K
and in the self-check; the sentence said only the first range.
kSourceDigest moves from 1a51cb9d... to ef7da006529622c2fa87d1d3371038cdf85467b1fb0010b361f334784b6fa92b,
the value smx_selfcheck_digest.digest() gives for this tree (stale() is None). Only the digest line changes:
the 133 names and the reference hashes are unchanged.

Provenance: dev-diagnostics runtime built from this tree (DLL sha256 b8b97de8...), k_smx_test=report on
backend 3 (D3D12, native, no HIP), record kl-speed-K deep-split smx-report-r3, 2026-11-18:
- mix: 133/133 equal to the HIP reference hashes, 0 mismatches
- rodd: 133/133 equal to the HIP reference hashes, 0 mismatches
compare_raw required a multiple of 4 bytes, so the native odd-size legs of
test_native_split_m_matches_one_launch and test_native_split_fallback stopped
at k.ctapT (2 bytes per render pixel: 2 x 131 x 79 = 20698 bytes) before any
state or head buffer was compared. The whole 4-byte words are still compared
as uint32; the 1 to 3 trailing bytes are compared byte for byte under the
same sentinel rules. Equal data still passes; a changed trailing byte, an
untouched trailing byte (counted) and a size mismatch fail.

CPU tests in test_k_raw_compare.py cover odd lengths 1, 2, 3 mod 4 and
20698: equal passes, a differing last byte and first trailing byte fail, a
trailing byte neither side wrote is counted and limited, a write only on
D3D12 fails, a size mismatch fails.
@phantoslayer21 phantoslayer21 changed the title K-deep-split: exact speedup (-31.25 us at 1080p Q) Model K: exact speedup, stage m split across workgroups (-23.5 to -31.25 us at 1080p Q) Nov 18, 2026
@phantoslayer21
phantoslayer21 marked this pull request as ready for review November 18, 2026 19:56
@phantoslayer21
phantoslayer21 merged commit e82d856 into main Nov 18, 2026
10 checks passed
@phantoslayer21
phantoslayer21 deleted the kl-speed/k-deep-split-opus branch November 18, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant