Reproduce “A Dataset for Answering Time-Sensitive Questions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhu Chen, Xinyi Wang, William Yang Wang
The paper introduces TimeQA, a new dataset designed to evaluate and advance the temporal reasoning capabilities of NLP models when answering time-sensitive questions from unstructured text. Experiments reveal a significant performance gap, with the best model achieving 54.6% F1 on challenging temporal questions compared to 91.1% for humans.
Key insights
- A substantial gap exists between human (91.1% F1) and machine performance (54.6% F1 for best model) in temporal reasoning, particularly when questions require implicit temporal inference.
- Current QA models demonstrate a lack of robust and consistent internal temporal representations, as evidenced by a significant performance drop from 'easy' to 'hard' questions (13-15%) and inconsistencies in predictions when time specifiers are perturbed.
- The artificially synthesized questions in TimeQA are an effective proxy for natural language queries, allowing models trained on them to generalize well to human-paraphrased questions with only a small performance drop.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.