Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions clean/dx12/kernels/hyb_gemm.hlsl
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,13 @@
// groups one group at a time, the expand those of one sub-pixel k at a time when SW_EX_SPLIT (expand_c128: the four
// sub-pixels' 48 KB do not fit D3D12's 32 KB).
// HIP twins: the same names in swin_hyb.co (clean/kernels/swin_hyb.hip: swin.hip's patch_merge / expand with HYB).
// The expands run with their workgroups in reverse order (WG_REV_ORDER, 1-D dispatches; the same bytes), opposite to
// the bottleneck / dec2 block before them, so each starts on the rows its producer wrote last (Model L's hybrid chain).
//
// @wg-kernel patch_merge_c64_hyb co=- block=256 wave=32 args=p,p,p,u,u defines=SW_PM_C=64,SW_PM_CO=96,SW_PM_SPLIT=1
// @wg-kernel patch_merge_c96_hyb co=- block=256 wave=32 args=p,p,p,u,u defines=SW_PM_C=96,SW_PM_CO=128,SW_PM_SPLIT=4
// @wg-kernel expand_c96_hyb co=- block=256 wave=32 args=p,p,p,u,u defines=SW_EX_CL=96,SW_EX_C=64,SW_EX_SPLIT=0
// @wg-kernel expand_c128_hyb co=- block=256 wave=32 args=p,p,p,u,u defines=SW_EX_CL=128,SW_EX_C=96,SW_EX_SPLIT=1
// @wg-kernel expand_c96_hyb co=- block=256 wave=32 args=p,p,p,u,u defines=SW_EX_CL=96,SW_EX_C=64,SW_EX_SPLIT=0,WG_REV_ORDER=1
// @wg-kernel expand_c128_hyb co=- block=256 wave=32 args=p,p,p,u,u defines=SW_EX_CL=128,SW_EX_C=96,SW_EX_SPLIT=1,WG_REV_ORDER=1
// *_scanmut the ISA rules' mutants (probe=1, outside the manifest; build_dx12.py requires one per model hybrid
// kernel): no RN16 before the FP8 pack (HYB_SCAN_MUTANT=1), rule 2 must fail (no norm tree here: rule 1
// has nothing to find)
Expand Down
4 changes: 2 additions & 2 deletions clean/dx12/kernels/hyb_swin.hlsl
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@
// (the table's shared_dec_border) and the skip map with the fine mirror; the member input is RN16(E + S).
// HIP twins: the same names in swin_hyb.co (clean/kernels/swin_hyb.hip).
//
// @wg-kernel swin_block_c64_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=64,SW_HEADS=2,SW_MLP=256,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c64_hyb
// @wg-kernel swin_block_c64_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=64,SW_HEADS=2,SW_MLP=256,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c64_hyb,WG_REV_ORDER=1
// @wg-kernel swin_block_c64_add_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=64,SW_HEADS=2,SW_MLP=256,SW_ADD=1,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c64_add_hyb
// @wg-kernel swin_block_c96_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=96,SW_HEADS=4,SW_MLP=384,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c96_hyb
// @wg-kernel swin_block_c96_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=96,SW_HEADS=4,SW_MLP=384,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c96_hyb,WG_REV_ORDER=1
// @wg-kernel swin_block_c96_add_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=96,SW_HEADS=4,SW_MLP=384,SW_ADD=1,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c96_add_hyb
// @wg-kernel swin_block_c128_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u defines=SW_C=128,SW_HEADS=4,SW_MLP=512,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_HYB=1,SW_ENTRY_OVERRIDE=swin_block_c128_hyb
// swin_block_c64_hyb_scanmut / _c128_hyb_scanmut the fused-square scan's mutants (probe=1, outside the manifest; see
Expand Down
6 changes: 4 additions & 2 deletions clean/dx12/kernels/l_swin.hlsl
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@
// l_swin_block_c32_enc_nc the encoder block: FP16 halo map [H + 4][W + 4][32] in (SW_SRC 1), FP8 out
// with its workgroups in reverse order (WG_REV_ORDER, wg_platform.hlsli; the same bytes): it
// reads first the rows of x2 the linears wrote last, and its last-written rows of x3 are the
// ones l_project_gather reads first (more cache hits at both hand-offs)
// ones l_project_gather reads first (more cache hits at both hand-offs). WG_REV_ORDER
// reverses the group index of a 1-D dispatch only (a multi-dimensional launch must flatten
// it first); every kernel declared with it here and in hyb_swin.hlsl / hyb_gemm.hlsl is 1-D
// l_swin_block_c32_dec_nc the decoder block: FP8 map in with the token-grid mirror (SW_SRC 2), FP16 out (SW_OUT_F16)
// l_swin_block_c32_enc_lin_nc D3D12 only: the encoder block with the two input linears as its prologue (SW_SRC 3;
// byte for byte the pair l_linear_32_32_fused -> _enc_nc, which stays the fallback route):
Expand Down Expand Up @@ -43,7 +45,7 @@
// form, enc0.Win1, enc0.Win2 as the last arguments) and _dec_hyb with l_coef_pack12 as its
// epilogue (SW_OUT_COEF12, COEF_CVT_HW); the same bytes as the launches they replace
// @wg-kernel l_swin_block_c32_enc_lin_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u,p,p defines=SW_C=32,SW_HEADS=2,SW_MLP=128,SW_ADD=0,SW_SAFE=1,SW_ECL=0,SW_HPU=8,SW_GELU=0,SW_SRC=3,SW_HYB=1,SW_ENTRY_OVERRIDE=l_swin_block_c32_enc_lin_hyb,WG_REV_ORDER=1
// @wg-kernel l_swin_block_c32_dec_coef12_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u,p,p,p,p,u,u defines=SW_C=32,SW_HEADS=2,SW_MLP=128,SW_ADD=1,SW_SAFE=1,SW_ECL=0,SW_HPU=8,SW_GELU=0,SW_OUT_F16=1,SW_OUT_COEF12=1,COEF_CVT_HW=1,SW_HYB=1,SW_ENTRY_OVERRIDE=l_swin_block_c32_dec_coef12_hyb
// @wg-kernel l_swin_block_c32_dec_coef12_hyb co=- block=256 wave=32 args=p,p,p,p,u,u,u,u,p,p,p,p,u,u defines=SW_C=32,SW_HEADS=2,SW_MLP=128,SW_ADD=1,SW_SAFE=1,SW_ECL=0,SW_HPU=8,SW_GELU=0,SW_OUT_F16=1,SW_OUT_COEF12=1,COEF_CVT_HW=1,SW_HYB=1,SW_ENTRY_OVERRIDE=l_swin_block_c32_dec_coef12_hyb,WG_REV_ORDER=1
// ..._enc_lin_hyb_dump / _dec_coef12_hyb_dump (+ _mutdump: SW_HYB_MUT_ORDER=4) their stage dumps (probe=1, check only:
// l_hyb_stage_check.py --set fused, the oracle per stage); _scanmut: the fused-square scan's
// mutants (no contraction barrier, built beside every hybrid build, outside the manifest)
Expand Down
Loading