Reproduce “MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, Ronald Clark
Researchers from the University of Oxford and Yonsei University introduced MCJudgeBench, a benchmark for evaluating large language models as judges at the constraint level in multi-constraint instruction following tasks. This benchmark utilizes invariance perturbations to assess judge correctness and stability, revealing that aggregate accuracy can mask poor performance on minority label types and that judge reliability is sensitive to prompt variations.
Key insights
- LLM judge reliability is multi-dimensional, encompassing correctness, intrinsic consistency, and procedural stability, which cannot be fully captured by aggregate performance metrics.
- High overall correctness (CJAR) often masks weaker performance on minority 'partial' and 'no' labels, indicating an imbalance in judge's ability to detect nuanced failures.
- LLM judges exhibit higher sensitivity to changes in the evaluation prompt's formulation (procedural inconsistency) than to superficial, semantically equivalent changes in candidate responses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.