Reproduce “The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecasting” on OpenResearch

About this paper

Lefei Shen, Mouxiang Chen, Han Fu, Xiaoxue Ren, Xiaoyun Joy Wang, Jianling Sun, Zhuo Li, Chenghao Liu

Researchers from Zhejiang University, State Street Technology, and Salesforce Research systematically evaluated Transformer architectures for long-term time series forecasting, identifying key design choices for optimal performance. The study found that bi-directional attention with joint-attention, complete forecasting aggregation, and a direct-mapping paradigm consistently yield superior results. A combined model built on these principles outperformed several existing state-of-the-art models.

Key insights

• Bi-directional attention with joint-attention mechanisms proved more effective for LTSF than uni-directional or hybrid-masking approaches, especially when training data is limited.
• Complete forecasting aggregation, which processes latent embeddings from both look-back and forecasting windows together, consistently enhanced prediction accuracy.
• The direct-mapping forecasting paradigm significantly outperformed autoregressive methods by avoiding error accumulation and exposure bias.
• The optimal choice between LayerNorm and BatchNorm is dataset-dependent, with BatchNorm showing better performance on time series data containing more anomalies or non-stationary patterns.

Summary by alphaXiv.