Skip to content

Feat: add mgu variants and elstm - #57

Merged
MartinuzziFrancesco merged 2 commits into
mainfrom
feat/mgu-variants-elstm
Sep 27, 2026
Merged

MartinuzziFrancesco merged 2 commits into
mainfrom
feat/mgu-variants-elstm

Conversation

@MartinuzziFrancesco

Copy link
Copy Markdown
Owner

No description provided.

MartinuzziFrancesco and others added 2 commits September 27, 2026 18:39
MGU1-3 (Heck & Salem, MWSCAS 2017; RecurrentLayers.jl#157) reduce the
MGU forget gate: MGU1 drops the input term, MGU2 also drops the bias,
MGU3 keeps only a learnable bias_f. The candidate and update are MGU's.

eLSTM (Irie et al., arXiv:2305.19044; RecurrentLayers.jl#94) is an LSTM
whose forget gate and candidate see c(t-1) only through element-wise
weights (weight_c), with an output gate on c(t) via weight_hh. Only c
recurs, so it is built on the decoupled single-state bases like
IntersectionRNN: h feeds the next layer, c_n is returned as state.
weight_c defaults to uniform in [-0.1, 0.1], close to the reference
code's N(0, 0.1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Disabled biases are registered as zero buffers, so the
`if hasattr(self, "bias_*")` guards in 15 custom reset_parameters were
always true: a non-zero bias_init / recurrent_bias_init (e.g. "ones")
overwrote biases the user had turned off. Guard on nn.Parameter instead,
which covers bias_ih, bias_hh, bias_ch and bias_mh regardless of which
flag controls them. The base reset_parameters was already correct since
it iterates named_parameters.

Adds test_cell_disabled_biases_stay_zero over every cell; it failed for
19 cells before this change. Found by Codex review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.50%. Comparing base (1a597f5) to head (d34b376).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
+ Coverage   96.24%   96.50%   +0.26%     
==========================================
  Files          46       47       +1     
  Lines        3489     3750     +261     
==========================================
+ Hits         3358     3619     +261     
  Misses        131      131              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MartinuzziFrancesco
MartinuzziFrancesco merged commit b16308e into main Sep 27, 2026
20 checks passed
@MartinuzziFrancesco
MartinuzziFrancesco deleted the feat/mgu-variants-elstm branch September 27, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants