Fold the sigmoid gate into the causal-attention reduce epilogue: operator -8.3% where it folds, +0.31% decode - #225

Closed

MichaelDementii wants to merge 3 commits into

MichaelDementii:perf/attention-fused-sigmoid-gate

Enhance your code review process with GitHub Actions

GitHub Actions make it easy to automate all your software workflows, now with world-class CI/CD.
Build, test, and deploy your code right from GitHub. Learn more about GitHub Actions.

Navigation Menu

Uh oh!

Fold the sigmoid gate into the causal-attention reduce epilogue: operator -8.3% where it folds, +0.31% decode - #225

Fold the sigmoid gate into the causal-attention reduce epilogue: operator -8.3% where it folds, +0.31% decode#225
MichaelDementii wants to merge 3 commits into
Neroued:masterfrom
MichaelDementii:perf/attention-fused-sigmoid-gate

Enhance your code review process with GitHub Actions

Linux, macOS, Windows, and containers

Matrix builds

Any language

Live logs

Built-in secret store

Multi-container testing