Reproduce “Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Edo Cohen-Karlik, Itamar Menuhin-Gruman, Raja Giryes, Nadav Cohen, Amir Globerson
The paper from Tel Aviv University and Google Research establishes a theoretical framework explaining how overparameterized linear recurrent neural networks, trained with Gradient Descent, implicitly extrapolate to longer sequences. It reveals that this occurs when the training sequence length surpasses twice the teacher's intrinsic state space dimension, promoting the learning of low-dimensional state representations.
Key insights
- Gradient Descent implicitly biases overparameterized RNNs towards solutions that learn low-dimensional state spaces, enabling full extrapolation when the training sequence length k exceeds twice the teacher's intrinsic dimension d_hat (k > 2d_hat).
- The mechanism for this extrapolation stems from a deep connection to the statistical moment problem: matching a sufficient number of initial moments (impulse response elements) uniquely determines the full underlying system.
- The 'balancedness' of RNN parameters, where B=C^T, is an implicitly preserved property under Gradient Flow and is favored by common near-zero random initialization schemes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.