Reproduce “Improved state mixing in higher-order and block diagonal linear recurrent networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Igor Dubinin, Antonio Orvieto, Felix Effenberger
New architectures for linear recurrent networks, Higher-order Linear Recurrent Units (H-LRU) and Block-Diagonal LRUs (BD-LRU), increase expressive power via structured state and channel mixing. These models, coupled with a novel L1-normalization scheme, achieve performance competitive with LSTMs on synthetic and language tasks while maintaining computational efficiency.
Key insights
- Structured state mixing, whether temporal (higher-order) or channel-wise (block-diagonal), is a primary determinant of LRNN expressivity, outperforming simply increasing network width.
- The proposed L1-normalization strategy is critical for the stable scaling and improved performance of higher-order and block-diagonal recurrent networks, especially as window/block sizes increase.
- The optimal block or window size for structured state mixing is task-dependent, implying that specific architectural inductive biases should be matched to the nature of the problem.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.