Reproduce “Fool Me Twice: Entailment from Wikipedia Gamification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, Jordan Boyd-Graber
Google Research and the University of Maryland developed FOOLMETWICE, a competitive multiplayer game designed to create a challenging dataset for natural language inference and fact verification. This gamification approach generated a dataset that integrates evidence retrieval and entailment, featuring reduced annotation artifacts and increased complexity compared to existing datasets like FEVER, which resulted in significantly lower performance for state-of-the-art models.
Key insights
- Gamification can effectively collect high-quality, adversarial NLP data by incentivizing the creation of complex, human-like reasoning challenges.
- Integrating evidence retrieval into dataset creation produces more realistic benchmarks for fact verification, addressing limitations of solely entailment-focused datasets.
- Annotation artifacts can be reduced by competitive human-in-the-loop processes, leading to datasets with fewer exploitable linguistic patterns and greater lexical diversity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.