Reproduce “SEMPO: Lightweight Foundation Models for Time Series Forecasting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hui He, Kun Yi, Yuanchi Ma, Qi Zhang, Zhendong Niu, Guansong Pang
Researchers from Beijing Institute of Technology and Singapore Management University developed SEMPO, a lightweight foundation model for time series forecasting, which achieves strong generalization with significantly reduced model size and pre-training data. The model demonstrated average reductions of 23.1% in MSE for zero-shot forecasting and 32.3% in MSE for few-shot scenarios on benchmark datasets, outperforming larger, more data-intensive foundation models.
Key insights
- Explicitly managing frequency-domain energy components through EASD effectively addresses the 'energy bias' in Transformers, leading to a more comprehensive utilization of pre-training data.
- A Mixture-of-Prompts mechanism in MoPFormer enables a lightweight Transformer to efficiently adapt to heterogeneous time series patterns across diverse domains with significantly fewer parameters than traditional large models.
- It is possible to achieve superior zero-shot and few-shot time series forecasting performance with substantially smaller models and less pre-training data, challenging the necessity of extreme scale for general-purpose time series FMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.