Reproduce “A Fixed-Budget, Cluster-Aware Standard for LLM-as-a-Judge Evaluation: A Multi-Hop RAG Stress Test” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Camilo Chacón Sartori, José H. García
A fixed-budget, cluster-aware evaluation standard is introduced for LLM-as-a-judge comparisons in multi-hop Retrieval-Augmented Generation (RAG) systems. This standard, when stress-tested, reveals that common evaluation protocols can overstate statistical significance and highlights the complementary roles of lexical and semantic retrieval signals.
Key insights
- Cluster correlation in RAG evaluation data can lead to a significant overstatement of statistical confidence in reported performance differences.
- Lexical and semantic retrieval signals are complementary, with hybrid evidence selection strategies often outperforming purely semantic methods.
- Observed performance advantages in RAG systems can be highly entangled with answer length and specific domain characteristics, necessitating granular analysis.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.