Reproduce “ExpertQA: Expert-Curated Questions and Attributed Answers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, Dan Roth
University of Pennsylvania and University of Washington researchers introduce EXPERTQA, a large-scale, expert-curated dataset for evaluating the attribution and factual accuracy of large language models (LLMs) in domain-specific contexts. The study reveals that LLMs frequently struggle with providing reliable, attributable answers in specialized fields like medicine and law, with a significant rate of hallucinated references and incomplete attributions, while demonstrating the utility of EXPERTQA for improving model performance.
Key insights
- Current LLMs, even advanced ones like GPT-4, frequently struggle with providing reliably factual and attributable answers in specialized, high-stakes fields such as medicine and law.
- Automated evaluation metrics for attribution (AutoAIS) and factuality (FActScore) often show low recall compared to human expert judgments, indicating they may overestimate LLM performance and miss nuanced errors.
- The quality of attributions, including completeness, faithfulness to source, and source reliability, is paramount for LLM trustworthiness, with purely generative models prone to hallucinating plausible but false citations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.