Reproduce “Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sadman Kabir Soumik
An independent researcher systematically evaluated nine bias mitigation strategies in LLM-as-a-Judge pipelines, revealing style bias as a dominant, underappreciated factor and identifying specific strategies, like the Combined Budget (S8), that significantly enhance human agreement rates and cost-efficiency for various judge models.
Key insights
- Style bias, particularly a strong preference for markdown formatting, is a dominant and often overlooked factor in LLM judge evaluations, with four out of five tested models exhibiting significant bias.
- Position bias was found to be negligible in contemporary LLM judges, suggesting improvements in modern instruction tuning compared to earlier findings.
- Verbosity bias is heterogeneous and quality-sensitive; some models (e.g., Gemini Pro, Llama) favor length regardless of content quality, while others (e.g., Claude, GPT-4o) penalize filler but reward genuine completeness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.