Reproduce “RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Suhan Guo, Jiahong Deng, Yi Wei, Hui Dou, Furao Shen, Jian Zhao
Researchers from Nanjing University developed RAM (Replace Attention with MLP), a strategy that substitutes attention mechanisms in state-of-the-art multivariate time series forecasting models with simpler Multi-Layer Perceptron (MLP) structures. This approach achieved an average FLOPs reduction of 62.579% for spatio-temporal models and 42.233% for long-term forecasting models, while maintaining or sometimes improving prediction accuracy.
Key insights
- The computational overhead of attention mechanisms in multivariate time series forecasting can be significantly reduced without substantial performance loss.
- The effectiveness of attention-based models largely stems from simpler MLP components like feedforward layers and residual connections, rather than the explicit pairwise relationship modeling of attention.
- Theoretical analysis suggests that attention mechanisms can degenerate into trivial mappings if softmax inputs are uniformly distributed, potentially explaining their dispensability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.