Reproduce “When LLM Judge Scores Look Good but Best-of-N Decisions Fail” on OpenResearch

About this paper

Eddie Landesberg

This research reveals that common global evaluation metrics for large language model (LLM) judges often overstate their ability for real-world best-of-N selection tasks. The paper demonstrates that a moderate global correlation (0.471) can mask a significantly lower within-prompt ranking ability (recovery rate of 21.0% for best-of-4 selection) due to weak within-prompt signal and high tie rates from coarse scoring.

Key insights

• Global correlation metrics for LLM judges primarily reflect agreement on broad prompt-level quality differences (baseline effects), largely independent of their ability to rank candidates within a single prompt.
• Coarse score quantization is a major bottleneck, leading to high tie rates (e.g., 66.5% of within-prompt pairwise comparisons showing identical scores) and severely hindering the selection of a unique best candidate.
• Explicit pairwise judging can substantially reduce tie rates and improve decision utility in some contexts, but its overall efficacy and efficiency depend on the evaluation setup and budget.

Summary by alphaXiv.