Reproduce “Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guneet Kohli
A study quantitatively measured the effective independence of LLM evaluation panels, revealing that a panel of nine frontier LLMs provides only about two independent votes. This finding, termed the "Condorcet gap," demonstrates that pervasive correlated errors negate the reliability benefits of majority voting and challenges current LLM evaluation practices.
Key insights
- A panel of nine frontier LLMs yields a remarkably low effective independence (n_eff = 2.18), meaning it provides only about two independent votes despite its size.
- A significant "Condorcet gap" exists, with actual majority-vote accuracy falling far below the theoretical expectation if judges voted independently (e.g., 22.0 percentage points on ChaosNLI-MNLI).
- Neither increasing panel size nor employing sophisticated aggregation methods effectively mitigates the independence deficit, pointing to inherent structural correlation in LLM judgments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.