Skip to content

Attention optimizations (part 2): channel-aware backward cache#232

Draft
azrael417 wants to merge 14 commits into
mainfrom
tkurth/attention-optimizations-part-2
Draft

Attention optimizations (part 2): channel-aware backward cache#232
azrael417 wants to merge 14 commits into
mainfrom
tkurth/attention-optimizations-part-2

Conversation

@azrael417

@azrael417 azrael417 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Second batch of spherical-attention work on top of the optimized kernels. User-visible piece:

  • Channel-aware backward cache for the serial neighborhood-attention CUDA backward. The two-pass flash-style backward previously re-gathered K/V and recomputed qdotk/gdotv in pass 2. For wide
    channel counts we now cache those per-neighbor dot products in shared memory in pass 1 and read them back in pass 2, eliminating the recompute. Gated to nchan >= 192 and compile-time templated
    (template + if constexpr) so the sub-192 path is byte-for-byte identical to the previous kernel — no effect on the common C=64 ERA5 workload.

Plus benchmark and test coverage for the wide-channel regime the cache targets.

Motivation

The backward pass was the remaining hotspot after the forward work in part 1. Profiling showed pass-2 recompute (re-gathering K/V and redoing the dot products) is pure redundant work when
there's shared-memory budget to hold the pass-1 results — which only pays off once the channel dimension is wide enough to dominate the per-neighbor cost, hence the nchan >= 192 gate.
WGMMA/tensor-core rewrites were prototyped and rejected (small skinny latency-bou Jump to bottom (ctrl+End) ↓ r), so this MR keeps the scalar/vectorized kernel and only removes redundant

This MR is marked as draft because #231 needs to be merged first.

@azrael417 azrael417 self-assigned this Jul 15, 2026
@azrael417 azrael417 changed the title Tkurth/attention optimizations part 2 Attention optimizations (part 2): channel-aware backward cache Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant