Reproduce “SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shruti Singh, Nandan Sarkar, Arman Cohan
Researchers from IIT Gandhinagar, Yale University, and AI2 developed SCIDQA, a dataset of 2,937 question-answer pairs derived from peer reviews, designed to challenge large language models in deep scientific reading comprehension. Evaluation revealed that state-of-the-art models struggle with the dataset's complexity, with many showing measurable improvements when provided with full paper context.
Key insights
- SCIDQA effectively challenges even state-of-the-art LLMs, demonstrating its utility as a benchmark for deep scientific reading comprehension.
- Providing full-text context significantly improves the performance of many LLMs, underscoring the importance of comprehensive document access for complex scientific queries.
- While proprietary models like GPT-4o achieve the highest scores, open-source models also show substantial gains with context, yet a considerable gap remains between AI and human expert-level understanding for highly complex questions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.