Reproduce “NeoQA: Evidence-based Question Answering with Generated News Events” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Max Glockner, Xiang Jiang, Leonardo F. R. Ribeiro, Iryna Gurevych, Markus Dreyer
NEOQA introduces a benchmark using entirely fictional news events and entities to evaluate evidence-based question answering in Large Language Models (LLMs) for Retrieval-Augmented Generation (RAG). This approach addresses parametric knowledge interference, ensuring models rely solely on provided evidence and accurately deflect from unanswerable questions.
Key insights
- LLMs consistently struggle with unanswerable questions and detecting subtle inconsistencies or insufficient evidence, even when excelling at answerable questions.
- Shortcut reasoning is prevalent, especially when an "answer" might be inferred from partial evidence (e.g., missing bridge entities), leading models to answer incorrectly rather than deflect.
- A negative correlation exists between accuracy on answerable questions and accuracy on unanswerable questions, suggesting that models proficient at answering are more prone to guess when evidence is insufficient.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.