Skip to content

feat: Add activation functions and Xavier/He weight init#5

Merged
ThomasHartDev merged 1 commit into
mainfrom
thomas/feat/activations-init
Jul 24, 2026
Merged

feat: Add activation functions and Xavier/He weight init#5
ThomasHartDev merged 1 commit into
mainfrom
thomas/feat/activations-init

Conversation

@ThomasHartDev

Copy link
Copy Markdown
Owner

Pulled the nonlinearities and the init schemes into their own module so the variance story is measurable on its own. Each activation carries a local backward for act'(z), and Xavier vs He are the usual closed forms (2 over fan_in+fan_out for symmetric acts, 2 over fan_in for ReLU). The interesting bit is forward_variance_profile: stack a few layers from unit noise and you can watch naive N(0,1) weights explode, He hold a ReLU stack near O(1), and Xavier quietly starve the same ReLU stack. That last case is the silent failure mode I wanted to pin down with a test rather than leave as folklore.

Derivatives are checked against finite differences, sample variances of the draws match the formulas, and the MLP path is left alone for now so this stays a focused slice.

@ThomasHartDev
ThomasHartDev merged commit 1c22340 into main Jul 24, 2026
2 checks passed
@ThomasHartDev
ThomasHartDev deleted the thomas/feat/activations-init branch July 24, 2026 21:28
Comment thread src/activations.py
Comment thread src/activations.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant