Reproduce “State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shida Wang, Beichen Xue
Researchers from the National University of Singapore established that multi-layer state-space models with layer-wise non-linearity are universal approximators for continuous sequence-to-sequence functions, while also demonstrating that these models exhibit an asymptotic exponential decay in their memory, similar to traditional recurrent neural networks.
Key insights
- Layer-wise non-linearity is sufficient for multi-layer SSMs to achieve universal approximation of continuous sequence-to-sequence relationships, even with linear recurrence within each layer.
- SSMs, including advanced variants like S4, fundamentally exhibit an asymptotically exponential decay in their memory, indicating that the 'curse of memory' is not fully overcome.
- Two distinct constructive approaches for universal approximation (Kolmogorov-Arnold and Volterra Series) offer different implications for how network width might scale with sequence length.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.