Reproduce “Adapting Time Series Foundation Models through Data Mixtures” on OpenResearch

About this paper

Thomas L. Lee, Edoardo M. Ponti, Amos Storkey

MixFT adapts forecasting models to related time-series data by grouping context windows across datasets according to inferred patterns, then training a LoRA adapter for each group and routing new contexts to the matching adapter. Across ten evaluation datasets, MixFT has the best average rank among tested methods on small Chronos Bolt and Moirai-1.1-R: 2.0 and 2.1. This assumes evaluation series share sub-domains with fine-tuning data.

Key insights

• Time series datasets are often heterogeneous, containing multiple distinct underlying data generating processes or 'sub-domains' that do not align with arbitrary dataset boundaries.
• Fine-tuning specialized LoRA modules on data-driven sub-domains, rather than rigid dataset divisions, enables TSFMs to achieve finer-grained adaptation and better generalize to unseen time series contexts.
• A Bayesian GMM, fitted via variational inference with diagonal covariance priors, effectively and robustly identifies these meaningful sub-domains within the TSFM's embedding space.

Summary by alphaXiv.