Reproduce “Investigating Compositional Reasoning in Time Series Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Willa Potosnak, Cristian Challu, Mononito Goswami, Kin G. Olivares, Michał Wiliński, Nina Żukowska, Artur Dubrawski
Researchers from Carnegie Mellon University formalized compositional reasoning in Time Series Foundation Models (TSFMs) and developed a framework to evaluate their ability to synthesize fundamental patterns for complex forecasting. The study found that patch-based Transformers and MLP-based models exhibit strong compositional reasoning, often outperforming traditional statistical methods in zero-shot out-of-distribution scenarios.
Key insights
- Input patching is a critical design choice for Transformer-based TSFMs, significantly enabling their compositional reasoning capabilities.
- Architectural choices that optimize for in-distribution performance do not always translate to improved out-of-distribution compositional reasoning; smaller models or simpler layers can sometimes be more effective.
- Compositional understanding in TSFMs is reflected in their latent space embeddings, showing high similarity between representations learned from component parts and full compositions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.