Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion clean/dx12/isa_checks.json
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
{
"_doc": "build_dx12.py ISA check: for each kernel listed, the RGA (gfx1201) ISA must contain exactly the given number of instructions whose mnemonic matches each pattern (a regular expression, matched at the start of the mnemonic). Model K's softmax weights (clean/k/dx12/hlsl/k_smx.hlsli) are exact only when the driver folds the mix form p + e into one v_fma_mixlo/hi_f16 rounding per weight: 32 weights per lane per softmax round, i.e. 32 x heads x windows per workgroup for the narrow stages (e2 / d2: 2 heads, 1 window; d1: 2 heads, 2 windows) and 32 x heads / NG for the wide stages (e3 / d3 and the NG = 2 e4 / d4: 2 rounds; NG = 4 e4 / d4: 1; m: 8 heads / NG 4 = 2). A missing fold leaves v_cvt_f16_f32 after an FP32 add (a second rounding) and fails here. The __rodd ports (round to odd) must have none. The native D3D12 runtime checks the same fold on the player's driver at create (k_mix_probe). WG_DX12_ISA_CHECK=0 skips the check.",
"_doc": "build_dx12.py ISA check: for each kernel listed, the RGA (gfx1201) ISA must contain exactly the given number of instructions whose mnemonic matches each pattern (a regular expression, matched at the start of the mnemonic). Model K's softmax weights (clean/k/dx12/hlsl/k_smx.hlsli) are exact only when the driver folds the mix form p + e into one v_fma_mixlo/hi_f16 rounding per weight: 32 weights per lane per softmax round, i.e. 32 x heads x windows per workgroup for the narrow stages (e2 / d2: 2 heads, 1 window; d1: 2 heads, 2 windows) and 32 x heads / NG for the wide stages (e3 / d3 and the NG = 2 e4 / d4: 2 rounds; NG = 4 e4 / d4: 1; m: 8 heads / NG 4 = 2; the phase-split m attention k_split_m_attn: one head per workgroup = 1). A missing fold leaves v_cvt_f16_f32 after an FP32 add (a second rounding) and fails here. The __rodd ports (round to odd) must have none. The native D3D12 runtime checks the same fold on the player's driver at create (k_mix_probe). WG_DX12_ISA_CHECK=0 skips the check.",
"k_stage_e2": {"v_fma_mix(lo|hi)_f16": 64},
"k_stage_d2": {"v_fma_mix(lo|hi)_f16": 64},
"k_stage2_d1": {"v_fma_mix(lo|hi)_f16": 128},
"k_stagew2_e3": {"v_fma_mix(lo|hi)_f16": 64},
"k_stagew4_e4": {"v_fma_mix(lo|hi)_f16": 32},
"k_stagew2_e4": {"v_fma_mix(lo|hi)_f16": 64},
"k_stagew4_m": {"v_fma_mix(lo|hi)_f16": 64},
"k_split_m_attn": {"v_fma_mix(lo|hi)_f16": 32},
"k_split_m_wo": {"v_fma_mix(lo|hi)_f16": 0},
"k_split_m_w1": {"v_fma_mix(lo|hi)_f16": 0},
"k_split_m_w2": {"v_fma_mix(lo|hi)_f16": 0},
"k_stagew4_d4": {"v_fma_mix(lo|hi)_f16": 32},
"k_stagew2_d4": {"v_fma_mix(lo|hi)_f16": 64},
"k_stagew2_d3": {"v_fma_mix(lo|hi)_f16": 64},
Expand All @@ -18,6 +22,7 @@
"k_stagew4_e4__rodd": {"v_fma_mix(lo|hi)_f16": 0},
"k_stagew2_e4__rodd": {"v_fma_mix(lo|hi)_f16": 0},
"k_stagew4_m__rodd": {"v_fma_mix(lo|hi)_f16": 0},
"k_split_m_attn__rodd": {"v_fma_mix(lo|hi)_f16": 0},
"k_stagew4_d4__rodd": {"v_fma_mix(lo|hi)_f16": 0},
"k_stagew2_d4__rodd": {"v_fma_mix(lo|hi)_f16": 0},
"k_stagew2_d3__rodd": {"v_fma_mix(lo|hi)_f16": 0},
Expand Down
20 changes: 20 additions & 0 deletions clean/dx12/kernel_sets.json
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,26 @@
"k_out_pixel_direct",
"k_io.co"
],
[
"k_split_m_attn",
"k_block_wide.co"
],
[
"k_split_m_attn__rodd",
"k_block_wide.co"
],
[
"k_split_m_w1",
"k_block_wide.co"
],
[
"k_split_m_w2",
"k_block_wide.co"
],
[
"k_split_m_wo",
"k_block_wide.co"
],
[
"k_stage2_d0_hidden",
"k_block.co"
Expand Down
20 changes: 19 additions & 1 deletion clean/k/dx12/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ refuse a header whose digest differs; `WG_DX12_SMX_DIGEST=0` lets a negative-con
`build.py` never embeds a test-only, bypassed, define-built, unprovenanced or stale entry, and the payload packager
refuses a library whose build manifest does not record that check (`clean/tests/test_dev_diagnostics_off.py`).

The test hook `k_smx_test` (`mix_fail`, `both_fail`, `rodd_unchecked`, `report`) exists only in runtime libraries built
The test hook `k_smx_test` (`mix_fail`, `both_fail`, `rodd_unchecked`, `split_fail`, `report`) exists only in runtime libraries built
with `build.py --dev-diagnostics`; a release build refuses it (`test_k_runtime_d3d12.py`:
`test_native_softmax_switch_is_logged`, `test_native_softmax_refusals`, `test_native_fallback_frames_*`). Negative
controls, never shipped: `K_SMX_MUT=1` (the mix form's residual dropped), `K_SMX_MUT4=1` (dropped only at window offset
Expand All @@ -113,6 +113,24 @@ identical.
Known limit, shared with Model M's ports: D3D12 flushes FP32 denormals (inputs and results) to zero and HIP keeps them.
`k_expo_scalars` only meets one with a pre-exposure above about 8.5e37, so the check's inputs keep clear of them.

## Phase-split stage m (runtime option `k_split`, native D3D12 device)

`kernels/k_block_split.hlsl` runs stage m (`k_stagew4_m`: 6 to 24 windows at 1080p/1440p output (more at 4K and in the self-check), one 512-lane workgroup each) as four launches
spread over many more workgroups: `k_split_m_attn` (one per window and head: normalization, q / k / v, softmax, z;
`__rodd` twin), `k_split_m_wo` (one per window and output tile: Wo, x1, a), `k_split_m_w1` (one per window and two hidden
tiles: W1, activation) and `k_split_m_w2` (one per window and output tile: W2, x2, the block output), joined by the
dispatch boundaries. Every accumulator is the same WMMA chain (operands, order, initial value) as in `k_stagew4_m` and every
FP16 value handed between the launches is the one `k_stagew4_m` keeps at that point, so the output bytes are the same.
Default on the native D3D12 device (`k_split=m`; `k_split=off`: the one launch; other devices: the one launch). The runtime
redirects `k_stagew4_m[__rodd]` in `KRuntime::launch`, so the create-time self-check runs the split form for its m case
against the unchanged HIP reference hashes (kernels line `split=m`). The self-check digest (`smx_selfcheck_digest.py`)
covers the phase kernels and the runtime's `k_split` option, redirect and phase launches. `k_split` is parsed on every
device: a value other than `m` or `off` fails create there too.

If the create-time self-check fails on the split form of stage m, create does not refuse: stage m takes the one launch,
the self-check runs again on that path, and one note of create says so (the output is the same). Create refuses only
when the one-launch path fails the self-check too. The developer hook `k_smx_test=split_fail` forces the first failure.

## Close approximations (runtime option `k_approx`, opt-in)

Every port above is exact, and nothing set is the exact path. `k_approx` selects close approximations of two stages on
Expand Down
Loading
Loading