Skip to content

cuda: keep Volta Flash Attention tiles at least 32 columns - #18

Merged
novkien merged 1 commit into
masterfrom
codex/issue-15-volta-fa-min-tiles
Oct 3, 2026
Merged

novkien merged 1 commit into
masterfrom
codex/issue-15-volta-fa-min-tiles

Conversation

@novkien

@novkien novkien commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Addresses #15

SM70 Flash Attention has no device code for a 16-column MMA tile. With an unaligned KV length or no mask, the host can disable GQA packing and select that tile for a small query batch, aborting at runtime. Exclude the 16-column branch when the selected compiled architecture is Volta so the existing 32-column specialization handles the query. Other architecture choices remain unchanged.

Add 11 focused D256/GQA12 controls across query widths 5/8/16/17, aligned and unaligned KV lengths, Q8/F16, and cache views. The original unaligned QW8/Q8 shape aborts on baseline and passes with this guard. All 11 final cases pass CUDA-versus-CPU comparison on V100 and RTX 5060 Ti. CPU fixture/support checks and independent source review also pass; git diff --check is clean.

An additional long unmasked F16 numerical stress case also fails on the unchanged RPC baseline and is tracked separately in #16. The no-mask dispatcher control here uses KV8192; the numerical tolerance and kernel math are unchanged. Exact merged deployment and proxy acceptance remain in #15.

Codex assisted with implementation and review under the owner's approved Flash Next work.

@novkien
novkien merged commit 109ab65 into master Oct 3, 2026
15 of 24 checks passed
@novkien

novkien commented Oct 3, 2026

Copy link
Copy Markdown
Owner Author

Acceptance completed on the exact merged native/RPC pair c7de202dfd9551c4a7ad3eef666789dcdb0bbf0d (empty source diff, manifest suffix e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). PR #18's Volta minimum-tile repair is included in that revision.

After normal model unload, the final immutable test-backend-ops and libraries passed all 15 selected Flash Attention CPU-reference cases on the V100 and RTX 5060 Ti. The selector was hsk=256,hsv=256,nh=2,nr23=\[12,1\],kv=(8192|65664|65792),nb=(5|8|16|17),.*. This includes the original Q8/KV65664/QW8 disjoint-mask reproduction, PR18's 11 neighboring/tile/stride/mask cases, and four related PR21 controls. Every selected row reported supported=1 and passed=1; no numerical tolerance was changed. Native and RPC runs took 14.35 s and 16.95 s.

The normal lifecycle then restored the exact c7de202 pair. Post-load binary manifests on both hosts match the expected source ID; the owned idle hold was released and read back inactive. The final pair has also completed three valid cold8192+512 proxy :8082 requests at C266240/ub768 with the original target/draft, all target/draft layers on GPU, pipeline enabled and two configured slots. These requests establish affected Flash behavior on the deployed pair; the isolated operator matrix establishes the precise formerly failing shape.

Independent dispatch review and pre-merge reference evidence are in the PR18 handoff. All acceptance criteria for the bounded tile-selection defect are now met, so #15 can close. The parent TG optimization issue #9 remains open; this correctness repair is not a TG speed claim. Local evidence: issue15-final-c7-reference, run-issue15-final-reference.log, and tg-graph-pure-screen/baseline under the retained Flash9 artifact directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants