Reproduce “Mutual Information Scaling and Expressive Power of Sequence Models” on OpenResearch

About this paper

Huitao Shen

This work analyzes the mutual information scaling properties of Recurrent Neural Networks (RNNs) and self-attention networks (Transformers), demonstrating that linear and nonlinear RNNs inherently exhibit exponential decay of mutual information, while Transformers can capture the algebraic (power-law) decay observed in natural sequences. The research highlights a fundamental difference in how these architectures model long-range dependencies, linking it to concepts from statistical mechanics.

Key insights

• Linear and nonlinear RNNs, including LSTMs, fundamentally exhibit exponential decay of mutual information, limiting their capacity for true long-range dependencies due to hidden state compression.
• Transformers are significantly more efficient at capturing long-range temporal correlations, including power-law decay, attributed to their explicit attention mechanism across the entire sequence history.
• Many widely used natural language datasets exhibit a "non-uniformity problem" where validation set statistics, particularly long-range mutual information, differ from training sets, potentially hindering models' ability to learn true power-law correlations.

Summary by alphaXiv.