Reproduce “A recurrent neural network without chaos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thomas Laurent, James von Brecht
Mathematical analysis reveals that popular recurrent neural networks like LSTMs and GRUs can exhibit chaotic internal dynamics even without external input. Researchers propose the Chaos-Free Network (CFN), a simpler recurrent architecture mathematically proven to be non-chaotic, which achieves perplexity scores comparable to LSTMs on language modeling datasets like Penn Treebank (e.g., 72.2 vs. 71.8 for 50M parameters).
Key insights
- LSTMs and GRUs can inherently exhibit strange attractors and chaotic dynamics when external inputs are absent, making their behavior sensitive to initial conditions.
- A simpler gated RNN (Chaos-Free Network - CFN) can be designed that is mathematically proven to be non-chaotic, with all trajectories converging to the zero state without input.
- Chaotic internal dynamics are not a prerequisite for strong performance in sequence modeling, as the provably non-chaotic CFN achieves competitive results against LSTMs and GRUs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.