Reproduce “How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Laurène Vaugrante, Thilo Hagendorff
The authors systematically test how model, prompt, reasoning effort, and scoring scale affect LLM-as-a-judge evaluations of answer correctness, sentiment, and toxicity. Across 60 classification designs, judges average 96.5% accuracy, yet Base-to-Long prompts reduce leniency from 81.3% to 52.4%. Model choice also drives variation, especially on ambiguous answers; results are limited to these benchmarks and designs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.