Repository navigation
cuda: SM70 Flash Attention selects an unsupported tile for eight-query GQA12 #15
Description
Problem and impact
An existing SM70 Flash Attention route selected a kernel variant with no compatible device code for a valid D256/GQA12/Q8 shape with eight query rows. The unchanged baseline reproduced flash_attn_ext_f16 has no device code compatible with CUDA arch 700, then an unspecified launch failure. This was a correctness prerequisite discovered during #9, separate from any sparse-attention or TG speed claim.
Expected behavior is a compiled supported kernel or a valid fallback, with CPU-reference numerical agreement.
Reproduction and confirmed cause
Baseline revision: 8d75def06db97c8736989c795be403faf214a9d4; the frozen baseline test-only patch was 53272e885123a4ef609ccd8da2de785526832e14d33e7ba022cc30b3eaf9c8b3. Hardware: V100-SXM2-16GB, CUDA 12.9, SM70-real, one GPU exposed, graphs disabled and normal fusion enabled. Tests used immutable explicit libraries after normal model unload.
The original case has F32 Q, Q8_0 K/V, hsk=hsv=256, nh=2, nr23=[12,1], KV=65664, eight query rows, one batch, F16 mask, F32 accumulation, permutation [0,2,1,3], no sinks/ALiBi/softcap, n_kv_max=2051 and eight disjoint finite mask ranges. It failed on the unchanged V100 baseline and passed on the RPC RTX 5060 Ti; adjacent 17-query controls also passed.
KV65664 is not divisible by FATTN_KQ_STRIDE=256, so dispatch disables GQA packing and selects ncols2=1. Eight queries previously selected ncols1=16, but the existing Volta device guard only supports total columns >=32. This is a host tile-selection defect, not an OOM or instruction failure. Whether ordinary two-slot MTP produces this exact shape was not established or needed for the isolated reproduction.
Scope, authority and implementation
The owner approved native attention correctness work within #9, including reviewed PRs, merge, matched native/RPC deployment and proxy acceptance. Scope preserves public APIs, GGUF formats, Q8 KV, P2P policy, and other architecture routes; it does not authorize proxy source changes.
PR #18, reviewed at d90c7653be3cea8d0ed7c41ab6330657fb3f1524, skips the 16-column branch when volta_mma_available(cc) identifies Volta compiled code, selecting the existing 32-column variant. It adds neighboring and fallback cases without changing attention math or tolerance. A distinct baseline long-unmasked tolerance failure remains separately recorded in #16.
Acceptance evidence
- Original SM70 reproduction passes with CPU numerical comparison.
- Adjacent widths, masks, strides, fallback controls and RPC GPU behavior pass.
- Independent review confirms the chosen Volta tile has compiled device code and other paths retain their fallback.
- PR merged, exact matched native/RPC source deployed through the model lifecycle, affected Flash behavior verified through proxy :8082.
The final merged pair is c7de202dfd9551c4a7ad3eef666789dcdb0bbf0d with an empty source diff. All 15 focused CPU-reference cases passed on both GPU architectures, including the original disjoint-mask QW8 case, the 11 PR18 cases, and four related PR21 controls. Every row was supported and passed. The final selector was hsk=256,hsv=256,nh=2,nr23=\[12,1\],kv=(8192|65664|65792),nb=(5|8|16|17),.*.
Normal restore completed; both loaded binary manifests match the exact final revision, and the owned idle hold was released/read back inactive. Three valid cold8192+512 requests also completed through proxy :8082 on that pair at C266240/ub768, original target/draft, full GPU layer offload, pipeline enabled and two configured slots. Full acceptance details are in the PR18 acceptance report.
Status and ownership
Codex /root completed the bounded defect's acceptance. Initial duplicate search inspected all then-existing six fork issues and eight PRs, including #9 and #11/#12; no matching tile defect was found. This issue is resolved. Parent #9 remains open for TG optimization and its own final acceptance.
Activity
novkien commentedon Oct 3, 2026
Work report - dispatch cause and implementation claim
Codex
/rootremains coordinator; delegateflash_attentionowns the bounded source implementation for this issue in a new isolated fork worktree. Next action: exclude the 16-column host branch whenvolta_mma_available(cc)is true, so it selects the already compiled 32-column variant, and add the original failing shape plus nearby controls. This matches the backend's highest-compiled-architecture predicate rather than assuming the physical device ID alone.Confirmed source chain: K length65664 is not divisible by
FATTN_KQ_STRIDE=256, so the dispatcher disables GQA packing. Volta then chooses ncols2=1; QW8 selects ncols1=16; the device implementation explicitly rejects total columns below32. The unchanged baseline reproduction confirms the defect independently of the new sparse path.The first sparse-positive fixtures used the same unaligned KV length and therefore did not actually select the proposed sparse16x4 kernel. Their numerical passes were dense-path controls. Those positive tests will be corrected to aligned KV65792 in #9; this issue preserves the unaligned failure as its regression test. No sparse optimization is accepted or deployed.
novkien commentedon Oct 3, 2026
PR #18 is open and linked to this issue (
Addresses novkien/llama.cpp-fork#15). It targets mastera60027610cffb451909b2ae3e2130f1a19b44d4eat reviewed headd90c7653be3cea8d0ed7c41ab6330657fb3f1524.The two-file change skips the 16-column Flash Attention MMA branch when the selected compiled architecture is Volta, allowing the existing 32-column specialization to handle the query. The focused 11-case matrix passed CUDA-versus-CPU comparison on the V100 and RTX 5060 Ti. The original masked QW8/KV65664 failure now passes; the no-mask dispatcher control uses KV8192 after the unchanged baseline failed the separate long-unmasked NMSE intake in #16. The tolerance and attention math are unchanged.
No required checks are reported for this branch; the remaining CI checks are queued or pending, with no failures visible at this handoff. This issue stays open for exact merged native/RPC build verification and live Flash behavior acceptance through the proxy on 8082.
novkien commentedon Oct 3, 2026
Acceptance completed on the exact merged native/RPC pair
c7de202dfd9551c4a7ad3eef666789dcdb0bbf0d(empty source diff, manifest suffixe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). PR #18's Volta minimum-tile repair is included in that revision.After normal model unload, the final immutable
test-backend-opsand libraries passed all 15 selected Flash Attention CPU-reference cases on the V100 and RTX 5060 Ti. The selector washsk=256,hsv=256,nh=2,nr23=\[12,1\],kv=(8192|65664|65792),nb=(5|8|16|17),.*. This includes the original Q8/KV65664/QW8 disjoint-mask reproduction, PR18's 11 neighboring/tile/stride/mask cases, and four related PR21 controls. Every selected row reported supported=1 and passed=1; no numerical tolerance was changed. Native and RPC runs took 14.35 s and 16.95 s.The normal lifecycle then restored the exact c7de202 pair. Post-load binary manifests on both hosts match the expected source ID; the owned idle hold was released and read back inactive. The final pair has also completed three valid cold8192+512 proxy :8082 requests at C266240/ub768 with the original target/draft, all target/draft layers on GPU, pipeline enabled and two configured slots. These requests establish affected Flash behavior on the deployed pair; the isolated operator matrix establishes the precise formerly failing shape.
Independent dispatch review and pre-merge reference evidence are in the PR18 handoff. All acceptance criteria for the bounded tile-selection defect are now met, so #15 can close. The parent TG optimization issue #9 remains open; this correctness repair is not a TG speed claim. Local evidence:
issue15-final-c7-reference,run-issue15-final-reference.log, andtg-graph-pure-screen/baselineunder the retained Flash9 artifact directory.