Reproduce “Adversarial Examples for Evaluating Reading Comprehension Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Robin Jia, Percy Liang
Researchers from Stanford University introduced adversarial examples for reading comprehension systems, demonstrating that state-of-the-art models' F1 scores dropped dramatically when confronted with carefully constructed distracting sentences, while human performance remained largely unaffected. This work revealed that current systems often rely on superficial pattern matching rather than deep language understanding.
Key insights
- State-of-the-art reading comprehension models exhibit "overstability," meaning they fail to differentiate between semantically plausible but incorrect information and the true answer.
- Human understanding is robust to these adversarial distractors, indicating a significant disparity between human and machine comprehension of context and relevance.
- Model failures on adversarial examples are largely due to incorrectly predicting answer spans located within the distracting sentences, suggesting a reliance on shallow lexical overlap rather than deep semantic reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.