Reproduce “Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Delip Rao, Chris Callison-Burch
The paper treats agreement scores for LLM judges as outputs of a measurement protocol, identifying choices about verdict extraction, rating thresholds, abstentions, and aggregation. In one rubric benchmark, 24 configurations yield accuracy from 0.551 to 0.899 and Cohen’s kappa from −0.067 to 0.068; extraction alone changes 703 of 2,007 decisions, so the spread does not establish strong judge performance.
Key insights
- Common binary agreement metrics (Pearson's r, Spearman's \rho, Kendall's \tau_b, \phi, MCC) are mathematically identical for non-degenerate binary verdicts, making multiple reporting redundant.
- Cohen's Kappa is sensitive to marginal rate mismatches between human and LLM judge, potentially appearing low even with high raw agreement if one label is highly prevalent.
- Methodological decisions for handling abstentions, ties, and aggregation fundamentally redefine the quantity being measured, significantly impacting reported agreement and rendering non-standardized comparisons invalid.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.