Reproduce “Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ivan Viakhirev, Kirill Borodin, Amirah Almutairi, Serguei Barannikov, Maxim Abramov, Grach Mkrtchian
An investigation into recurrent neural network dynamics reveals that controlling the finite-time behavior of recurrent operators is crucial for achieving reliable test-time depth and extrapolation, demonstrating that settling dynamical regimes lead to consistent accuracy improvements on harder, unseen instances, while non-settling regimes result in performance degradation.
Key insights
- The finite-time dynamical regime of a recurrent operator (settling, marginal, or drifting) reliably predicts its test-time depth behavior, including extrapolation and depth-safety.
- A fixed-point objective term can causally shift the dynamical regime from drifting to settling, thereby enabling depth-safe extrapolation and generalization gains.
- Algorithm-learning operators exhibit a structured, anisotropic Lyapunov spectrum and strong non-normality, indicating efficient computation within a globally contracting map, which serves as a strong predictor of extrapolation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.