Reproduce “ASQA: Factoid Questions Meet Long-Form Answers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang
This paper introduces ASQA, a new dataset for long-form question answering focused on ambiguous factoid questions, and proposes the DR score, an automated evaluation metric. The dataset enables models to generate comprehensive explanations that resolve ambiguity, revealing a substantial performance gap between current state-of-the-art models and human capabilities.
Key insights
- High-quality retrieval of supporting documents is crucial for strong performance in generative long-form QA, but it is not sufficient; effective summarization and information synthesis are also essential.
- There is a substantial performance gap between current state-of-the-art generative models (e.g., T5 Open Book) and human performance on the ASQA task, indicating significant room for improvement in factual consistency and explanation generation.
- The proposed automated DR score reliably correlates with human judgments of overall answer quality, providing a scalable and objective metric for tracking progress in long-form QA.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.