Reproduce “The NarrativeQA Reading Comprehension Challenge” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, Edward Grefenstette

NarrativeQA introduces a benchmark for answering questions about books and movie scripts, using questions written from summaries and testing systems on summaries or full stories. For full-story reading, the authors retrieve 200-token chunks before applying a neural reader. The Attention Sum Reader with 20 chunks scored 0.179 test MRR, below the no-context Seq2Seq baseline’s 0.202, showing limited traction by these tested systems.

Key insights

  • Existing reading comprehension models, including neural approaches, struggle significantly on NarrativeQA, especially when processing full stories.
  • A substantial gap persists between current model performance and human comprehension abilities on narrative understanding tasks.
  • The two-step retrieve-then-read approach for long documents is heavily bottlenecked by the effectiveness of the initial retrieval component.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.