Reproduce “HEARTS: Benchmarking LLM Reasoning on Health Time Series” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sirui Li, Shuhan Xiao, Mihir Joshi, Ahmed Metwally, Daniel McDuff, Wei Wang, Yuzhe Yang
HeaRTS introduces a benchmark to evaluate Large Language Models' reasoning abilities on diverse health time series, featuring real-world data across 12 domains and 110 hierarchical tasks. Evaluations revealed that current LLMs underperform specialized models and rely on simple heuristics rather than deep temporal reasoning, struggling particularly with high-resolution and complex physiological data.
Key insights
- LLMs exhibit a strong reliance on explicit instructions and low-complexity heuristics, rather than deep multi-step temporal reasoning, when processing health time series.
- Performance on health time series reasoning tasks shows a weak correlation with general LLM intelligence indices, indicating a need for domain- and data-specific skills.
- A shared, model-agnostic difficulty hierarchy exists across health time series domains, with high-resolution temporal data posing uniform bottlenecks for LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.