Repository navigation
feat: add missing cells - #56
Merged
Merged
Conversation
Ports T-RNN and T-GRU from Balduzzi & Ghifary, "Strongly-Typed
Recurrent Neural Networks" (ICML 2016, arXiv:1602.02218), found
missing by diffing against the sister Julia libraries
RecurrentLayers.jl and LuxRecurrentLayers.jl. Verified against the
actual paper (fetched and read directly, not just the Julia ports,
which cite a couple of unrelated arXiv IDs in code comments for
TGRU/TLSTM — the underlying equations are correct in both libraries,
matching the real paper).
Both cells have no weight matrix acting on the hidden state at all,
by design ("strongly typed" architectures forbid a learned linear map
on h(t-1)): T-RNN mixes h(t-1) in only via an elementwise gate, and
T-GRU's recurrent weights act on the previous *input* x(t-1) rather
than h(t-1).
This means TGRU's second recurrent state is input_size-shaped for the
first layer but hidden_size-shaped for every subsequent stacked layer
(since layer k>=1's input is the previous layer's hidden_size
output), which breaks this library's shared multi-layer stacking loop
and shared test harness (both assume every double-state component is
uniformly hidden_size-shaped). Rather than change the shared base
class, TGRU gets its own forward() override that tracks per-layer
state as Python lists and returns the final previous-input state as a
tuple of per-layer tensors instead of one stacked tensor. TGRU is
excluded from the generic CELL_CASES/LAYER_CASES harnesses for this
reason and gets bespoke tests instead, including state-continuity,
per-layer value correctness, and batch_first checks (added after a
Codex audit flagged the initial bespoke tests as shape-only).
TLSTM, MCLSTM, and IntersectionRNN are not included here: they share
a deeper issue where the paper-defined "output" is a different tensor
from what needs to recur in time, which neither of this library's two
base classes can cleanly express. Follow-up needed before porting
those.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends base.py with two new interface pairs (DecoupledSingleStateCellBase /DecoupledSingleStateRecurrentLayerBase and their double-state counterparts) for cells whose per-timestep output is not the same tensor as what recurs in time or feeds the next layer in a stack. This is purely additive: existing SingleState/DoubleState classes and every existing cell are untouched. This was needed because TLSTM, MCLSTM, and IntersectionRNN all define an "output" that genuinely differs from their recurring state, which neither of this library's original two base classes could express (both conflate "returned output" with "next-timestep state" with "next-layer input" into a single tensor). Implements IntersectionRNN/IntersectionRNNCell (Collins, Sohl-Dickstein, Sussillo, "Capacity and Trainability in Recurrent Neural Networks", ICLR 2017, arXiv:1611.09913) on top of the new single-state pair — the first real user of the extension. Verified against the actual paper PDF (page 4, equations 15-20). Requires input_size == hidden_size, per the paper: y(t) is a gated mix of x(t) and a hidden_size-dimensional candidate. The paper's own experiments exclude single-layer use as "not appropriate" (documented, not enforced, since the per-step math stays well-defined). Bespoke tests throughout (not added to the shared CELL_CASES/ LAYER_CASES/LAYER_CLASSES harnesses, since even the generic repr test uses differing input/hidden sizes) added after a Codex audit found the initial set didn't independently verify that multi-layer stacking forwards y (output), not h (state), to the next layer, and didn't numerically check dropout is applied only between layers. Both fixed with an explicit cell-by-cell unroll oracle and a dropout test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports MC-LSTM from Ben-Ari & Shwartz-Ziv, "Sequence modeling using a memory controller extension for LSTM" (NIPS 2017 time series workshop), found missing by diffing against the sister Julia libraries. Verified against the actual paper PDF: every gate (f, i, o, m, n) is driven by the control vector v(t-1), never by h(t-1) -- the paper's central idea is decoupling the LSTM's "prediction" output h(t) from the memory-management role a standard LSTM's hidden state also has to play. Built on the DecoupledDoubleStateCellBase/RecurrentLayerBase pair added in the previous commit -- MCLSTM is the first real user of the double-state half of that extension (IntersectionRNN used the single-state half). Unlike TGRU/TLSTM, MCLSTM's recurring state (v, c) is uniformly hidden_size-shaped across every layer (no ragged-shape issue), so the layer class relies entirely on the inherited generic stacking loop with no custom forward() override, and fits cleanly into the shared LAYER_CASES/LAYER_CLASSES test harness (unlike TGRU and IntersectionRNN, which needed fully bespoke layer tests). The cell itself still needs bespoke tests, since its forward() returns Tuple[Tensor, Tuple[Tensor, Tensor]], structurally incompatible with the shared CELL_CASES harness's Tuple[Tensor, Tensor] assumption. While implementing this, found and fixed a real gap in both MCLSTMCell and the already-committed TGRUCell: neither handled the per-component-None state convention (e.g. cell(x, (None, some_c))) that every other double-state cell in this library supports (see janet_cell.py's forward for the reference pattern) -- both only checked "state is None" at the top level, not per-component. Fixed in both. Codex audit found no correctness bugs (23 focused tests + 28 independent audit cases + a dedicated dropout-placement probe all passed) but flagged real test-thoroughness gaps: the None-state tests only covered unbatched inputs, the state-continuity test discarded final states instead of comparing them, and the stacking-oracle test only checked the last timestep's output and one state component per layer instead of the full sequence and every layer. All four strengthened before commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports T-LSTM from Balduzzi & Ghifary, "Strongly-Typed Recurrent Neural Networks" (ICML 2016, arXiv:1602.02218) -- the same paper TRNN and TGRU (already committed) came from. Verified against the actual paper HTML (equations 25-29). This is the trickiest of the five cells this branch set out to add, since it combines two issues each independently solved for other cells on this branch: - Like TGRU, the recurrent weights act on the previous *input* x(t-1), not a hidden state, so the second recurring state component is ragged across stacked layers (input_size-shaped for layer 0, hidden_size-shaped for later layers). - Like MCLSTM and IntersectionRNN, h(t) is a pure output that plays no role in any subsequent computation -- only (c(t), x(t)) recur. TLSTM therefore needed a custom forward() override (like TGRU, for the ragged state) built on DecoupledDoubleStateCellBase (like MCLSTM, for the decoupled output) -- the first cell needing both fixes at once. The critical correctness risk was the layer-stacking loop routing the wrong tensor to the next layer (e.g. threading c or x_prev instead of h, which would silently produce a different, incorrect architecture while still passing shape-only checks). This is independently verified by test_tlstm_stacking_forwards_output_not_state, which calls the cells directly rather than reusing any of the layer's own routing logic. Codex audit found no implementation bugs -- it specifically confirmed the stacking oracle would catch a routing bug by mutation-testing it (deliberately routing the wrong tensor and confirming the test fails) -- but found two docstring inaccuracies in the TLSTM layer docstring: it claimed to be "unlike every other cell" in decoupling output from state (false; MCLSTM and IntersectionRNN already do this), and it described the recurring state using x(t-1) notation when it's actually x(t) that carries forward (becoming x(t-1) on the next call). Both fixed. This completes all 5 models found missing from the sister Julia libraries (RecurrentLayers.jl, LuxRecurrentLayers.jl) on this branch: TRNN, TGRU, TLSTM, MCLSTM, IntersectionRNN. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #56 +/- ##
==========================================
+ Coverage 95.18% 96.24% +1.05%
==========================================
Files 43 46 +3
Lines 2619 3489 +870
==========================================
+ Hits 2493 3358 +865
- Misses 126 131 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.