Reproduce “Constructing Datasets for Multi-hop Reading Comprehension Across Documents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Johannes Welbl, Pontus Stenetorp, Sebastian Riedel
This work introduces a multi-hop reading comprehension task that requires models to combine evidence across multiple documents and releases two new datasets, WIKIHOP and MEDHOP, built to support this challenge. It demonstrates that while current neural RC models can perform multi-step inference, they struggle significantly with selecting relevant information from a larger, noisy set of documents.
Key insights
- Even with rigorous bias mitigation, state-of-the-art neural RC models show a notable gap compared to human performance on multi-hop cross-document tasks, indicating significant room for improvement.
- A major bottleneck for current multi-hop RC models is the intelligent selection of relevant documents from a larger, noisy set; model performance dramatically improves when provided only with the 'gold chain' of truly relevant documents.
- Masking candidate entities within documents can either improve performance by encouraging reliance on textual context (WIKIHOP) or hinder it by removing useful lexical cues (MEDHOP), depending on the dataset's inherent textual properties.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.