Reproduce “RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min and 1 more
Researchers developed RAG-QA Arena, a new framework featuring the Long-form RobustQA (LFRQA) dataset, which offers multi-domain, human-written long-form answers for evaluating Retrieval-Augmented Generation systems. The framework uses LLMs as evaluators, achieving high correlation with human judgments and showing that even top LLMs attain only a 41.3% win rate against the human-quality LFRQA answers.
Key insights
- LFRQA's human-written long-form answers are empirically superior as a gold standard for evaluating modern generative LLMs, significantly preferred over concatenated extractive answers (83.9% human preference).
- LLMs can reliably serve as evaluators for RAG-QA systems, with GPT-4-0125-PREVIEW demonstrating strong correlation (Pearson > 0.52, Cohen's Kappa > 0.43) with human judgments for answer quality.
- Current state-of-the-art LLMs face considerable challenges in matching human-quality long-form answers, particularly in synthesizing information from multiple sources and maintaining coherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.