Reproduce “Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, Qiang Xu
Researchers developed aLLM4TS, a framework that adapts Large Language Models for general time series representation learning through a two-stage training strategy and a novel patch-wise decoder, achieving state-of-the-art performance in forecasting, anomaly detection, and few-shot learning across various datasets. This approach leverages the causal modeling strengths of LLMs to generate highly transferable temporal representations.
Key insights
- Reconceptualizing time series forecasting as a causal, next-patch prediction task naturally aligns with LLM pre-training, enabling a more effective transfer of sequence modeling capabilities.
- A small, shared patch-wise decoder significantly reduces parameters compared to sequence-level decoders, mitigating overfitting and forcing the LLM backbone to learn robust, disentangled temporal representations within each patch.
- Freezing most LLM backbone parameters during Stage II fine-tuning, while updating only a minimal set (e.0.01% of parameters), optimizes adaptation to specific contexts and prevents overfitting to downstream tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.