[WIP]: feat(cutedsl-sm100): BWD LoopK with warp-specialized pipeline + IndexSparse/BlockSparse support#334
Open
cennn wants to merge 7 commits into
Open
[WIP]: feat(cutedsl-sm100): BWD LoopK with warp-specialized pipeline + IndexSparse/BlockSparse support#334cennn wants to merge 7 commits into
cennn wants to merge 7 commits into
Conversation
Add structural foundation for SM100 BWD InnerLoopK: - TMEM layout: dQ persistent, S/P→dK overlapped, dP/dS, dV - SMEM: multi-stage sK/sV/sKt for K streaming - Scheduler: outer Q blocks, inner K range via get_n_block_min_max - load_loop_k: Q/dO fixed, K/V pipeline streaming - mma_loop_k: dQ accumulated, dK/dV fresh per iter - kernel dispatch: conditional LoopK vs LoopQ for load/mma warps 1-CTA only, hdim<=128. compute_loop and reduce warps pending.
- compute_loop: LoopK conditionals for block extraction, LSE/dPsum pre-wait/post-release, per-iter K-block mask, skip dKV epilogue - dKV_reduce_loop_k: per-K-iter dK/dV TMA atomic reduce + end-of-tile dQ reduce (mirrors dQacc_reduce pattern with 3 TMEM T2R copies) - kernel dispatch: route reduce warps to dKV_reduce_loop_k for LoopK - force dKV_postprocess=True for LoopK (fp32 dKacc/dVacc accumulation) - frontend: add swap_bwd_qk_loop param to _flex_flash_attn_bwd + compile key
…se support - Fix 5 pipeline deadlocks in BWD LoopK warp-specialized kernel: 1. Separate pipeline_K_load from pipeline_Q to avoid barrier conflict 2. Reorder K release before acquire in single-stage buffer 3. Remove unnecessary pipeline_dQ empty wait in LoopK 4. Fix pipeline_dKV consumer group to use reduce_warps for LoopK 5. Add producer_state_Q.advance() after commit in load_loop_k - Add IndexSparse BWD LoopK: indices-driven K/V load with sparse dKV store - Add BlockSparse BWD LoopK: block-level sparse mask support - Add index_attn_indices_to_block_sparse conversion in sparse_utils.py - Fix int64->int32 dtype for mask_block_idx in sparse_utils.py - Disable 2CTA mode when swap_bwd_qk_loop is enabled - Remove unused variables (tidx, producer_state_dO) in load_loop_k Verified: Dense LoopK bit-identical to LoopQ, BlockSparse MHA pass, IndexSparse all pass on B300 SM103.
…env vars - Enable PackGQA for BWD SM100 (was globally disabled). Host-side reorder lse/dpsum/dq_accum to match CuTe column-major packed layout. Fix causal mask and block boundary computations for packed coordinates. - Remove LoopK-only restriction for IndexSparse BWD, allowing LoopQ path. - Add MAGI_ATTENTION_FFA_CUTEDSL_INNER_DIR_MAX_TO_MIN env var to control BWD inner loop direction (both LoopQ and LoopK). - Add MAGI_ATTENTION_FFA_CUTEDSL_MASK_MODE=dispatch env var for causal zone-based mask dispatch, skipping R2P on fully-valid tiles.
…guard - Add transpose_block_sparse_tensors() to convert forward-direction (M→N) block sparse tensors to backward-direction (N→M) with correct coarse Q-block granularity (subtile_factor * tile_m). - When index_attn_indices is used with LoopQ (swap_bwd_qk_loop=False), auto-transpose the block sparse tensors for the backward path. - Add index_attn_indices to disable_2cta conditions to prevent 2-CTA mode with block sparsity.
When block_sparse_tensors have head_dim=NHK but the kernel indexes by head_idx in NHQ space (pack_gqa=False), expand NHK→NHQ via repeat_interleave. When pack_gqa=True (head_idx maps to KV space), pass num_head_kv to normalization instead. Verified: BlockSparse + IndexSparse × LoopQ/LoopK × PackGQA/NoPG all 8 combinations PASS (B=1, Sq=512, NHQ=8, NHK=2, HD=128).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
在 CuTe-DSL SM100 BWD 内核中新增 InnerLoopK 路径(
swap_bwd_qk_loop=True),支持外层遍历 Q blocks、内层流式迭代 K/V blocks。在此基础上支持 IndexSparse 和 BlockSparse 的 BWD LoopK,使得 IndexAttn 等稀疏注意力场景可以选择 LoopK 方向以获得更好的 dQ 存储效率(dQ TMEM 累积 + 单次 store,而非 LoopQ 的 atomicAdd dQ)。核心改动
BWD LoopK warp-specialized pipeline
load_loop_k(producer)、mma_loop_k(consumer)、dKV_reduce_loop_k(reduce warps)三个 LoopK 专用方法pipeline_K_load(与pipeline_Q分离,避免 mbarrier 冲突)cpasync_reduce_bulk_add_f32原子归约到 GMEMIndexSparse + BlockSparse BWD LoopK
index_attn_indices_to_block_sparse():token-level indices(total_q, NHK, topk)→ block-levelBlockSparseTensors(GPU 矢量化,scatter_构建 presence matrix)prepare_block_sparse_bwd_loopk():forward-direction normalize(per M-block → N-block list)get_curr_blocksparse_tensors+get_m_block_from_iter_bwd其他
dKV_postprocess=True(fp32 累积 → dtype 转换 + softmax_scale 缩放)disable_2cta条件扩展)正确性验证
B300 SM103 环境,D=128, bf16:
Dense LoopK(vs LoopQ, bit-identical, max_diff=0.000000):
BlockSparse LoopK(vs LoopQ reference):
IndexSparse LoopK(vs LoopQ reference):
限制
dKV_postprocess路径(fp32 原子累加),尚未做 direct TMA store 优化