Skip to content

feat: add missing cells - #56

Merged
MartinuzziFrancesco merged 6 commits into
mainfrom
feat/missing-cells
Sep 20, 2026
Merged

MartinuzziFrancesco merged 6 commits into
mainfrom
feat/missing-cells

Conversation

@MartinuzziFrancesco

Copy link
Copy Markdown
Owner

No description provided.

MartinuzziFrancesco and others added 6 commits September 20, 2026 11:00
Ports T-RNN and T-GRU from Balduzzi & Ghifary, "Strongly-Typed
Recurrent Neural Networks" (ICML 2016, arXiv:1602.02218), found
missing by diffing against the sister Julia libraries
RecurrentLayers.jl and LuxRecurrentLayers.jl. Verified against the
actual paper (fetched and read directly, not just the Julia ports,
which cite a couple of unrelated arXiv IDs in code comments for
TGRU/TLSTM — the underlying equations are correct in both libraries,
matching the real paper).

Both cells have no weight matrix acting on the hidden state at all,
by design ("strongly typed" architectures forbid a learned linear map
on h(t-1)): T-RNN mixes h(t-1) in only via an elementwise gate, and
T-GRU's recurrent weights act on the previous *input* x(t-1) rather
than h(t-1).

This means TGRU's second recurrent state is input_size-shaped for the
first layer but hidden_size-shaped for every subsequent stacked layer
(since layer k>=1's input is the previous layer's hidden_size
output), which breaks this library's shared multi-layer stacking loop
and shared test harness (both assume every double-state component is
uniformly hidden_size-shaped). Rather than change the shared base
class, TGRU gets its own forward() override that tracks per-layer
state as Python lists and returns the final previous-input state as a
tuple of per-layer tensors instead of one stacked tensor. TGRU is
excluded from the generic CELL_CASES/LAYER_CASES harnesses for this
reason and gets bespoke tests instead, including state-continuity,
per-layer value correctness, and batch_first checks (added after a
Codex audit flagged the initial bespoke tests as shape-only).

TLSTM, MCLSTM, and IntersectionRNN are not included here: they share
a deeper issue where the paper-defined "output" is a different tensor
from what needs to recur in time, which neither of this library's two
base classes can cleanly express. Follow-up needed before porting
those.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends base.py with two new interface pairs (DecoupledSingleStateCellBase
/DecoupledSingleStateRecurrentLayerBase and their double-state
counterparts) for cells whose per-timestep output is not the same
tensor as what recurs in time or feeds the next layer in a stack. This
is purely additive: existing SingleState/DoubleState classes and every
existing cell are untouched.

This was needed because TLSTM, MCLSTM, and IntersectionRNN all define
an "output" that genuinely differs from their recurring state, which
neither of this library's original two base classes could express
(both conflate "returned output" with "next-timestep state" with
"next-layer input" into a single tensor).

Implements IntersectionRNN/IntersectionRNNCell (Collins,
Sohl-Dickstein, Sussillo, "Capacity and Trainability in Recurrent
Neural Networks", ICLR 2017, arXiv:1611.09913) on top of the new
single-state pair — the first real user of the extension. Verified
against the actual paper PDF (page 4, equations 15-20). Requires
input_size == hidden_size, per the paper: y(t) is a gated mix of x(t)
and a hidden_size-dimensional candidate. The paper's own experiments
exclude single-layer use as "not appropriate" (documented, not
enforced, since the per-step math stays well-defined).

Bespoke tests throughout (not added to the shared CELL_CASES/
LAYER_CASES/LAYER_CLASSES harnesses, since even the generic repr test
uses differing input/hidden sizes) added after a Codex audit found the
initial set didn't independently verify that multi-layer stacking
forwards y (output), not h (state), to the next layer, and didn't
numerically check dropout is applied only between layers. Both fixed
with an explicit cell-by-cell unroll oracle and a dropout test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports MC-LSTM from Ben-Ari & Shwartz-Ziv, "Sequence modeling using a
memory controller extension for LSTM" (NIPS 2017 time series
workshop), found missing by diffing against the sister Julia
libraries. Verified against the actual paper PDF: every gate (f, i,
o, m, n) is driven by the control vector v(t-1), never by h(t-1) --
the paper's central idea is decoupling the LSTM's "prediction" output
h(t) from the memory-management role a standard LSTM's hidden state
also has to play.

Built on the DecoupledDoubleStateCellBase/RecurrentLayerBase pair
added in the previous commit -- MCLSTM is the first real user of the
double-state half of that extension (IntersectionRNN used the
single-state half). Unlike TGRU/TLSTM, MCLSTM's recurring state (v, c)
is uniformly hidden_size-shaped across every layer (no ragged-shape
issue), so the layer class relies entirely on the inherited generic
stacking loop with no custom forward() override, and fits cleanly
into the shared LAYER_CASES/LAYER_CLASSES test harness (unlike TGRU
and IntersectionRNN, which needed fully bespoke layer tests). The cell
itself still needs bespoke tests, since its forward() returns
Tuple[Tensor, Tuple[Tensor, Tensor]], structurally incompatible with
the shared CELL_CASES harness's Tuple[Tensor, Tensor] assumption.

While implementing this, found and fixed a real gap in both
MCLSTMCell and the already-committed TGRUCell: neither handled the
per-component-None state convention (e.g. cell(x, (None, some_c)))
that every other double-state cell in this library supports (see
janet_cell.py's forward for the reference pattern) -- both only
checked "state is None" at the top level, not per-component. Fixed in
both.

Codex audit found no correctness bugs (23 focused tests + 28
independent audit cases + a dedicated dropout-placement probe all
passed) but flagged real test-thoroughness gaps: the None-state tests
only covered unbatched inputs, the state-continuity test discarded
final states instead of comparing them, and the stacking-oracle test
only checked the last timestep's output and one state component per
layer instead of the full sequence and every layer. All four
strengthened before commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports T-LSTM from Balduzzi & Ghifary, "Strongly-Typed Recurrent Neural
Networks" (ICML 2016, arXiv:1602.02218) -- the same paper TRNN and
TGRU (already committed) came from. Verified against the actual paper
HTML (equations 25-29). This is the trickiest of the five cells this
branch set out to add, since it combines two issues each independently
solved for other cells on this branch:

- Like TGRU, the recurrent weights act on the previous *input*
  x(t-1), not a hidden state, so the second recurring state component
  is ragged across stacked layers (input_size-shaped for layer 0,
  hidden_size-shaped for later layers).
- Like MCLSTM and IntersectionRNN, h(t) is a pure output that plays no
  role in any subsequent computation -- only (c(t), x(t)) recur.

TLSTM therefore needed a custom forward() override (like TGRU, for
the ragged state) built on DecoupledDoubleStateCellBase (like MCLSTM,
for the decoupled output) -- the first cell needing both fixes at
once. The critical correctness risk was the layer-stacking loop
routing the wrong tensor to the next layer (e.g. threading c or
x_prev instead of h, which would silently produce a different,
incorrect architecture while still passing shape-only checks). This
is independently verified by test_tlstm_stacking_forwards_output_not_state,
which calls the cells directly rather than reusing any of the layer's
own routing logic.

Codex audit found no implementation bugs -- it specifically confirmed
the stacking oracle would catch a routing bug by mutation-testing it
(deliberately routing the wrong tensor and confirming the test fails)
-- but found two docstring inaccuracies in the TLSTM layer docstring:
it claimed to be "unlike every other cell" in decoupling output from
state (false; MCLSTM and IntersectionRNN already do this), and it
described the recurring state using x(t-1) notation when it's
actually x(t) that carries forward (becoming x(t-1) on the next
call). Both fixed.

This completes all 5 models found missing from the sister Julia
libraries (RecurrentLayers.jl, LuxRecurrentLayers.jl) on this branch:
TRNN, TGRU, TLSTM, MCLSTM, IntersectionRNN.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Sep 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.42529% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.24%. Comparing base (d0c5ffc) to head (aee7c24).

Files with missing lines Patch % Lines
torchrecurrent/base.py 93.54% 4 Missing ⚠️
torchrecurrent/cells/trnn_cell.py 99.44% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #56      +/-   ##
==========================================
+ Coverage   95.18%   96.24%   +1.05%     
==========================================
  Files          43       46       +3     
  Lines        2619     3489     +870     
==========================================
+ Hits         2493     3358     +865     
- Misses        126      131       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MartinuzziFrancesco
MartinuzziFrancesco merged commit 1a597f5 into main Sep 20, 2026
21 checks passed
@MartinuzziFrancesco
MartinuzziFrancesco deleted the feat/missing-cells branch September 20, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants