Repository navigation
Model K: exact speedup, stage m split across workgroups (-23.5 to -31.25 us at 1080p Q) - #283
Merged
Merged
Conversation
Stage m runs as four launches on the native D3D12 device instead of one k_stagew4_m launch with one 512-lane workgroup per window (6 at 1080p): k_split_m_attn per (window, head), k_split_m_wo per (window, output tile), k_split_m_w1 per (window, two hidden tiles), k_split_m_w2 per (window, output tile), joined by dispatch boundaries. Each accumulator is the same WMMA chain (operands, order, initial value) as k_stagew4_m and every FP16 value handed between launches is the one k_stagew4_m keeps, so the output bytes are unchanged. Operand loads of each chain are fenced (opaque-zero XOR) so the driver issues them together instead of one latency per WMMA. KRuntime::launch redirects k_stagew4_m[__rodd] when k_split=m (default on backend 3; k_split=off restores the one launch), so the create-time self-check runs the split form for its m case against the unchanged HIP reference hashes. Self-check kernel sources and regions are untouched (kSourceDigest unchanged). isa_checks: one v_fma_mix per weight in k_split_m_attn, none in the __rodd twin.
The native D3D12 device runs stage m's phase kernels (k_split_m_*) for the self-check's m case (KRuntime::launch redirect), but the digest that binds k_smx_selfcheck.h to its sources covered only k_stagew4_m. Add the five phase kernels to the digest's kernel set and mark the k_split option, the redirect and split_m / split_alloc as SMX-SELFCHECK-SOURCE regions; digest test cases for k_block_split.hlsl and the redirect. The header's kSourceDigest is NOT rebound here (the 133 hashes are unchanged; rebinding needs coordinator / oversight OK): until then normal builds refuse this tree and the lever builds --test-only.
The D3D12 report hook (k_smx_test=report, developer build) on the native D3D12 device reproduced all 133 HIP reference hashes of k_smx_selfcheck.h in both softmax forms (mix and round-to-odd), 0 mismatches, probe 0 / 0, create smx=mix. kSourceDigest is rebound to the digest of this tree (smx_selfcheck_digest.digest()); the header's hashes and names are unchanged. Record: R:\WrongGPU-clean\tmp\kl-speed-K\deep-split\diag\smx-report-diag.json.
check_fallback_frames expected k_stagew4_m__rodd on backend 3, but with k_split=m (the default) stage m runs as k_split_m_attn__rodd and the k_split_m_wo / w1 / w2 launches. The expectation now follows the option: the phase launches with split=m, and a k_split=off leg expects k_stagew4_m__rodd. The forbidden mix forms follow the same rule (the phases wo / w1 / w2 have no softmax and run in both forms).
…ails A split stage m (k_split=m) that fails the create-time self-check no longer refuses create. Stage m takes the one k_stagew4_m launch (the same bytes and the same reference hashes), the self-check runs again on that path, and one create note says so, in the pattern of the round-to-odd note. Create refuses only when the one-launch path fails the self-check too; the error names both failures. Developer hook: k_smx_test=split_fail (WG_DEV_DIAGNOSTICS builds) makes the split form count as failed, so the fallback can be tested. With k_split=off it has no effect.
…e fallback Native-against-native tests (no HIP reference run, so they run on the developer device without the HIP legs): - split=m (the default) against k_split=off: the same raw bytes of the output and of every written buffer, history and exposure state after every frame, at 200 x 120 and at an odd output size (201 x 121 from 131 x 79), in RGBA16F and in dense RGB with auto-exposure. Each side starts from its own sentinel (0x5A, 0xC3), so a word neither side wrote fails. - The launch trace: the phase launches with k_split=m, k_stagew4_m with k_split=off, and the round-to-odd rounding of the softmax stages with k_smx_test=mix_fail on both. - A k_split value other than m or off fails create in one line. - The kernels line names split=m only when stage m is split. - k_smx_test=split_fail: create falls back to the one launch, one note names both paths, and the frames match k_split=off. The both_fail refusal names the split form's failure as well as the one-launch path's, and with k_split=off it does not.
The provenance lines added by hand to k_smx_selfcheck.h named a local path and the internal review process. The generator (clean/k/dx12/gen_k_smx_selfcheck.py) does not write them: its output for the same entries and digest is now byte-identical to the header. The provenance of the rebound digest is in its commit message.
…e HT The W2 batches (HT / KS_B2) and the W1 groups (HT / KS_HB) need HT = 4 KW_C / 16 divisible by both sizes; a #error now refuses any other choice at compile time. The comment ties the runtime's W1 grid (HT / 2 groups per window) to KS_HB = 2, so a change of KS_HB changes that launch as well.
split_alloc now runs before the k_fill block, and the four scratch buffers (Z, X1, A, H) join the fill list, so k_fill's sentinel covers them as it covers the rest of what a frame writes. Their byte sizes are recorded at allocation (split_bytes_). The W1 launch carries a comment that ties its HT / 2 groups per window to KS_HB = 2 in the kernel source.
The three phases have no softmax, so they must have no v_fma_mix. The check records that, next to the attention phase's count.
…easured Stage m's window count is 6 to 24 at 1080p and 1440p output, more at 4K and in the self-check; the sentence said only the first range.
kSourceDigest moves from 1a51cb9d... to ef7da006529622c2fa87d1d3371038cdf85467b1fb0010b361f334784b6fa92b, the value smx_selfcheck_digest.digest() gives for this tree (stale() is None). Only the digest line changes: the 133 names and the reference hashes are unchanged. Provenance: dev-diagnostics runtime built from this tree (DLL sha256 b8b97de8...), k_smx_test=report on backend 3 (D3D12, native, no HIP), record kl-speed-K deep-split smx-report-r3, 2026-11-18: - mix: 133/133 equal to the HIP reference hashes, 0 mismatches - rodd: 133/133 equal to the HIP reference hashes, 0 mismatches
compare_raw required a multiple of 4 bytes, so the native odd-size legs of test_native_split_m_matches_one_launch and test_native_split_fallback stopped at k.ctapT (2 bytes per render pixel: 2 x 131 x 79 = 20698 bytes) before any state or head buffer was compared. The whole 4-byte words are still compared as uint32; the 1 to 3 trailing bytes are compared byte for byte under the same sentinel rules. Equal data still passes; a changed trailing byte, an untouched trailing byte (counted) and a size mismatch fail. CPU tests in test_k_raw_compare.py cover odd lengths 1, 2, 3 mod 4 and 20698: equal passes, a differing last byte and first trailing byte fail, a trailing byte neither side wrote is counted and limited, a write only on D3D12 fails, a size mismatch fails.
phantoslayer21
marked this pull request as ready for review
November 18, 2026 19:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Model K, lever deep-split (probe, stage m only). Stage m runs as 4 launches instead of one
k_stagew4_mlaunch:k_split_m_attn(andk_split_m_attn__roddtwin): attention per window and headk_split_m_wo: Wo per output tilek_split_m_w1: W1 per 2 hidden tilesk_split_m_w2: W2 per output tileFiles:
clean/k/dx12/kernels/k_block_split.hlsl(new),clean/k/runtime/runtime_k.cppandruntime_k.h(split path, behind the existing k_split option),clean/dx12/kernel_sets.json,clean/dx12/isa_checks.json,clean/k/dx12/README.md,clean/k/dx12/smx_selfcheck_digest.py,clean/k/tests/test_k_smx_selfcheck_digest.py,clean/k/runtime/k_smx_selfcheck.h, plus the native tests underclean/k/tests/.Review fixes
k_split=m, the default) that fails the create-time self-check no longer refuses create. Stage m takes the onek_stagew4_mlaunch (same bytes, same reference hashes), the self-check runs again on that path, and one create note says so. Create refuses only when the one-launch path fails too; the error names both failures. Developer hook:k_smx_test=split_fail(WG_DEV_DIAGNOSTICS builds).kSourceDigestink_smx_selfcheck.his rebound to the digest of the reviewed tree (ef7da006...6fa92b;stale()is None). The 133 names and reference hashes are unchanged. Proof, D3D12 report hook (k_smx_test=report, dev-diagnostics build, no HIP at runtime): 133/133 mix and 133/133 round-to-odd matched the HIP reference hashes, 0 mismatches.k_split_m_wo/w1/w2ISA check covers the no-softmax phases. Fallback test (split_fail) matches k_split=off frames.k_block_split.hlslrefuses tile sizes that do not divide HT at compile time.k_smx_selfcheck.hcarries only the generator's text.Test results: the new native tests and CPU tests are run by CI on this head (see the checks list). No native-test log for this head is attached to this PR.
Byte checks (main vs this branch, same output)
Final output fnv1a64 is identical in every run: 1080p Q
1b975fbba17891e3, 1440p Q5fb3d1ab49ad8f90, both equal to REFS.json.Timing (kab.py, paired ABBA, 1500 frames, 4 blocks, A = main, B = this branch)
Range across sessions, stage m probe only:
Records:
C:\WrongGPU-clean-out\kl-speed\K\time\(deep-split-opus-v1,deep-split-opus-v2-1440pQ,deep-split-r2).Review verdict
Exactness passes on every leg above. Speed is faster at both shapes across sessions. The stage m probe is the only thing measured; the full 7-level split (65 us plan figure) is not measured. A cross-family second reader has not been run on this record.
Ready for review. Owner merges only.