Reproduce “Towards Lightweight Time Series Forecasting: a Patch-wise Transformer with Weak Data Enriching” on OpenResearch

About this paper

Meng Wang, Jintao Yang, Bin Yang, Hui Li, Tongxin Gong, Bo Yang, Jiangtao Cui

LiPFormer presents a lightweight Transformer-based model for time series forecasting, achieving high accuracy and efficiency through optimized architectural components. It incorporates a novel weak data enriching module that leverages future contextual information, enabling deployment on resource-constrained edge devices.

Key insights

• Traditional Transformer components like Layer Normalization and Feed Forward Networks, useful for NLP, can be detrimental to time series forecasting accuracy and resource efficiency, and their removal can improve performance.
• A combination of Inter-Patch and Cross-Patch attention effectively captures both local and global temporal dependencies in time series data, making explicit positional encodings unnecessary.
• Integrating future contextual information, even as 'weak labels,' through a contrastive learning dual encoder, significantly enhances forecasting accuracy, especially for predicting future changes.

Summary by alphaXiv.