Reproduce “When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability” on OpenResearch

About this paper

Zongyou Yang, Yinghan Hou, Xiaokun Yang

Researchers from Imperial College London and Nanchang Institute of Technology audited the multi-dimensional reliability of LLM-as-judge systems, revealing that model upgrades do not uniformly improve reliability, biases persist even in high-accuracy judges, jury aggregation benefits are limited by correlated errors, and multi-agent debate protocols require comprehensive audit trails for interpretability.

Key insights

• LLM-as-judge upgrades are not uniformly interchangeable; reliability gains are often workload-dependent and do not consistently improve along all scaling or release paths.
• Stronger LLM judges exhibit reduced biases, but these biases persist even in high-accuracy models, necessitating continuous bias detection and mitigation strategies.
• The effectiveness of majority-vote juries is severely limited by high error correlation among individual judges, making error dependence a more critical factor than jury size.
• Multi-agent debate protocols can induce substantial decision shifts, but their interpretability and trustworthiness are critically dependent on comprehensive audit trails of all intermediate steps and parser statuses.

Summary by alphaXiv.