Repository navigation
cuda: keep Volta Flash Attention tiles at least 32 columns - #18
Conversation
Assisted-by: Codex
|
Acceptance completed on the exact merged native/RPC pair After normal model unload, the final immutable The normal lifecycle then restored the exact c7de202 pair. Post-load binary manifests on both hosts match the expected source ID; the owned idle hold was released and read back inactive. The final pair has also completed three valid cold8192+512 proxy :8082 requests at C266240/ub768 with the original target/draft, all target/draft layers on GPU, pipeline enabled and two configured slots. These requests establish affected Flash behavior on the deployed pair; the isolated operator matrix establishes the precise formerly failing shape. Independent dispatch review and pre-merge reference evidence are in the PR18 handoff. All acceptance criteria for the bounded tile-selection defect are now met, so #15 can close. The parent TG optimization issue #9 remains open; this correctness repair is not a TG speed claim. Local evidence: |
Addresses #15
SM70 Flash Attention has no device code for a 16-column MMA tile. With an unaligned KV length or no mask, the host can disable GQA packing and select that tile for a small query batch, aborting at runtime. Exclude the 16-column branch when the selected compiled architecture is Volta so the existing 32-column specialization handles the query. Other architecture choices remain unchanged.
Add 11 focused D256/GQA12 controls across query widths 5/8/16/17, aligned and unaligned KV lengths, Q8/F16, and cache views. The original unaligned QW8/Q8 shape aborts on baseline and passes with this guard. All 11 final cases pass CUDA-versus-CPU comparison on V100 and RTX 5060 Ti. CPU fixture/support checks and independent source review also pass; git diff --check is clean.
An additional long unmasked F16 numerical stress case also fails on the unchanged RPC baseline and is tracked separately in #16. The no-mask dispatcher control here uses KV8192; the numerical tolerance and kernel math are unchanged. Exact merged deployment and proxy acceptance remain in #15.
Codex assisted with implementation and review under the owner's approved Flash Next work.