Reproduce “Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models” on OpenResearch

About this paper

Divij Gupta, Anubhav Bhatti, Surajsinh Parmar

Researchers at SpassMed Inc. evaluated various Parameter-Efficient Fine-Tuning (PEFT) techniques for adapting Time Series Foundational Models (TSFMs) to healthcare vital sign forecasting, finding that methods like FourierFT matched or surpassed LoRA and even outperformed state-of-the-art domain-specific models with significantly fewer trainable parameters. FourierFT on a small TSFM achieved superior performance by fine-tuning just 2,400 parameters, challenging traditional fine-tuning approaches.

Key insights

• PEFT methods consistently outperform both zero-shot and full fine-tuning for out-of-domain time series adaptation, demonstrating their effectiveness in leveraging pre-trained TSFMs for specialized tasks with limited data.
• Several explored PEFT techniques (BitFit, LayerNorm Tuning, VeRA, FourierFT) demonstrate performance comparable to or superior to LoRA, often requiring significantly fewer trainable parameters.
• The research highlights a critical trade-off between the number of fine-tuned parameters and model performance, where even smaller TSFMs, when efficiently adapted, can yield superior results over larger models or SOTA models trained from scratch.

Summary by alphaXiv.