Skip to content

fix(graph): apply SIGMOID/TANH activations in MoE layer - #736

Merged
jakmro merged 1 commit into
cactus-compute:mainfrom
yus100:fix/moe-sigmoid-tanh-activations
Jun 22, 2026
Merged

jakmro merged 1 commit into
cactus-compute:mainfrom
yus100:fix/moe-sigmoid-tanh-activations

Conversation

@yus100

@yus100 yus100 commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor

The MoE layer ignored the SIGMOID and TANH activations. Both are valid options on the moe_layer builder, but the activation switch had no cases for them, so they silently fell back to SILU and produced wrong results.

This adds the two missing cases (the kernels cactus_sigmoid_f16 and cactus_tanh_f16 already existed). I also added a test for the MoE layer, which previously had none — it runs each activation through a tiny identity-weight MoE and checks the output matches the standalone activation kernel.

Confirmed the test fails before the fix and passes after. Run it with cactus test graph --suite nn.

The MoE layer's activation switch only handled GELU, GELU_ERF, RELU and
SILU, with SIGMOID and TANH silently falling through to the SILU default.
Both activations are valid `Activation` enum values accepted by the
`moe_layer` builder and both kernels (cactus_sigmoid_f16 / cactus_tanh_f16)
already exist, so a graph built with either produced wrong results.

Add the two missing cases and a regression test that runs an identity-weight
single-expert MoE layer for every activation and checks the output matches
the corresponding standalone activation kernel.

Signed-off-by: yus100 <90424373+yus100@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@HenryNdubuaku
HenryNdubuaku requested a review from kar-m June 17, 2026 16:01
@HenryNdubuaku

Copy link
Copy Markdown
Collaborator

Thanks for this, i have assigned a reviewer.

@cattermelon1234

Copy link
Copy Markdown
Contributor

nice work stanley

@yus100

yus100 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@kar-m

@jakmro
jakmro merged commit 4200620 into cactus-compute:main Jun 22, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants