Reproduce “Language Models Still Struggle to Zero-shot Reason about Time Series” on OpenResearch

About this paper

Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, Tim Althoff

Researchers from the University of Washington and University of Virginia introduced a novel synthetic dataset, TSandLanguage, to evaluate the zero-shot time series reasoning abilities of large language models. Their findings indicate that current LMs consistently struggle with tasks such as etiological reasoning, question answering, and context-aided forecasting on time series data, performing significantly below human benchmarks.

Key insights

• LMs' ability to generate accurate time series patterns or perform forecasting does not translate into genuine reasoning or deep understanding of temporal dynamics.
• Current LMs demonstrate a fundamental inability to effectively integrate numerical time series information with highly relevant textual context to improve reasoning or predictions.
• Performance gains observed in some LM evaluations were often attributable to models exploiting artifacts of the dataset generation process or relying on parametric knowledge rather than true time series interpretation.

Summary by alphaXiv.