Reproduce “TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abdelrahman Abdallah, Mohammed Ali, Muhammad Abdul-Mageed, Adam Jatowt
TEMPO introduces a multi-domain benchmark for retrieval tasks requiring deep temporal reasoning, revealing that current state-of-the-art retrieval systems struggle significantly, with the best model achieving only 32.0 NDCG@10. The benchmark demonstrates that existing retrieval components often fail to provide temporally complete or relevant documents, leading to decreased performance in Retrieval-Augmented Generation (RAG) tasks compared to no-retrieval baselines.
Key insights
- Current state-of-the-art retrieval models, including reasoning-enhanced ones, exhibit fundamental limitations in handling complex temporal reasoning and providing comprehensive temporal coverage for queries.
- For complex temporal queries, existing retrieval systems can actively degrade RAG performance, providing incomplete or misleading context that leads to worse answers than generated without retrieval.
- Tasks requiring 'Trends & Cross-Period Comparison' pose the most significant challenge to current models, indicating a struggle to synthesize information across multiple time periods effectively.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.