Reproduce “RACE: Large-scale ReAding Comprehension Dataset From Examinations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard Hovy
The RACE dataset provides a large-scale, human-curated benchmark for machine reading comprehension, sourcing nearly 100,000 questions from English examinations. This dataset reveals a significant performance gap between current AI models (~44% accuracy) and human ceiling performance (~95% accuracy), highlighting a need for models capable of deeper reasoning.
Key insights
- The RACE dataset emphasizes multi-sentence reasoning, with 59.2% of questions requiring inference beyond single-sentence analysis, significantly higher than other large-scale MRC datasets.
- RACE uniquely introduces large-scale questions testing passage summarization and attitude analysis, crucial for holistic comprehension but largely absent in prior benchmarks.
- Leveraging expert-designed English examination questions ensures high-quality content that specifically targets genuine reading comprehension and complex reasoning abilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.