Skip to content

perf(core): prefetch tiles in the tiled matmuls, and flash attention with 8-key blocks and paired GQA heads - #105

Merged
xxpopielxx merged 3 commits into
mainfrom
perf/flash-attention-prefetch
Nov 13, 2026
Merged

xxpopielxx merged 3 commits into
mainfrom
perf/flash-attention-prefetch

Conversation

@xxpopielxx

Copy link
Copy Markdown
Member

Description

Two kernel speedups, stacked on #97 (it touches the same matmul kernels).

  • Tiled matmuls (matmulTiledVec4, matmulQuantTiled): each thread loads the next tile into registers while the current one computes. Same values, same math, so outputs are bit-identical.
  • Flash attention (attnFlash), with 8 lanes and heads up to 128 values:
    • blocks take 8 keys instead of 32, so the scores stay in registers;
    • two query heads on one kv head (GQA with an even group) share a workgroup, so each key and value block is loaded once and every read feeds both heads.
    • 4 lanes and wider heads (EmbeddingGemma 2 text, 256/512) keep their blocks and one head per workgroup.
    • Outputs differ by at most 1 ulp (summation order); the error against an f64 reference is unchanged.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  • pnpm typecheck, pnpm lint, pnpm --filter runntime test
  • pnpm --filter runntime bench:ab kernels -b feat/quant-conv -k matmul,conv,attention

New tests and bench cases:

  • SDPA_OPTIONS: three query heads per kv head (6/2, sinks, causal), run on every prefill route.
  • A/B: GQA 16/4 hd 64 in f32, EmbeddingGemma 2 sliding (4/2, hd 256, f16, window 512) and full (4/1, hd 512) layers.

A/B against #97 (M2 Pro): 115 cases, 28 faster, none slower. Matmul and conv outputs are bit-identical.

kernel case A (µs) B (µs) Δ
attention flash, privacy-filter 14/2 hd 64, window 128, sinks 428 374 -12.6%
attention flash f16, streaming encoder hd 64, causal 84.7 65.1 -23.2%
attention flash, GQA 16/4 hd 64, causal 163 124 -23.8%
attention flash packed f16 + segments, MiniLM bucket 512 237 191 -19.5%
attention flash f16, EmbeddingGemma 2 sliding 4/2 hd 256 1097 1086 same
attention flash, 4 lanes, hd 48 30.7 31.0 same
conv conv1d + gelu, Moonshine-tiny conv2 288→576 k7 s3 217 182 -16.1%
conv conv1d causal, Moonshine-streaming medium conv2 3651 3200 -12.3%
matmul tiledVec4 f16, YOLO26n PSA [64,400]×[400,400] 34.1 27.8 -18.6%
matmul tiledVec4 + addend, MiniLM fc2 bucket 512 369 330 -10.4%
matmul tiledVec4, B past 2^24 elements [9,2560]×[2560,9728] 1055 823 -21.9%
matmul quantW int4 whole [256,1024]×[1024,2304] 687 640 -6.8%

End to end (zoo models, M2 Pro, median wall time):

  • EmbeddingGemma 2: text -6 to -20%, image -7 to -13%, video -6 to -10%, audio -6 to -9%
  • all-MiniLM-L6-v2: -8 to -21%
  • privacy-filter: -2%
  • Moonshine, YOLO26 (detect, seg, pose), MobileNetV4, DepthART: unchanged, same outputs

Screenshots

N/A

Related issues

Stacked on #97.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

  • The two commits split cleanly: the matmul prefetch first, then flash attention.
  • The pairing and the 8-key blocks are limited to 8 lanes and heads up to 128 values: with wider heads (EmbeddingGemma 2 text) two heads no longer fit in registers and the text model got slower, so those keep the old path. The EmbeddingGemma 2 A/B cases guard this.

@xxpopielxx
xxpopielxx requested a review from chmjkb November 9, 2026 06:23
@xxpopielxx
xxpopielxx force-pushed the perf/flash-attention-prefetch branch from 3411331 to aee26a2 Compare November 10, 2026 17:49
@xxpopielxx
xxpopielxx requested a review from chmjkb November 10, 2026 17:52
Base automatically changed from feat/quant-conv to main November 12, 2026 13:28
… kernels

matmulTiledVec4 and matmulQuantTiled load tile t + 1 while tile t computes. The same values go
through the same math, so outputs are bit-identical.
… workgroup

With 8 lanes and heads up to 128 values, blocks take 8 keys so the scores stay in registers, and
two query heads on one kv head share a workgroup, so each key and value block feeds both. 4 lanes
and wider heads keep their blocks and one head per workgroup.
@xxpopielxx
xxpopielxx force-pushed the perf/flash-attention-prefetch branch from aee26a2 to f879680 Compare November 12, 2026 13:30
@xxpopielxx
xxpopielxx merged commit 3c1cf47 into main Nov 13, 2026
2 checks passed
@xxpopielxx
xxpopielxx deleted the perf/flash-attention-prefetch branch November 13, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants