Reproduce “Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim
A new diagnostic framework employs Item Response Theory (IRT) to evaluate the reliability of large language models used as evaluators, differentiating between intrinsic consistency and alignment with human judgments. The method identifies specific reliability issues such as prompt sensitivity, calibration mismatches, and potential construct validity gaps across NLP and vision-language tasks.
Key insights
- The IRT-GRM approach provides deeper diagnostic insights by explicitly separating underlying item quality from the LLM judge's measurement characteristics and prompt influences.
- Interpretable metrics ($C_V$, $ ho$, $ heta_{ratio}$, $D_W$) enable specific identification of issues like prompt sensitivity, calibration mismatches, or fundamental validity gaps.
- Prompt engineering (detailed instructions, CoT) primarily improves prompt stability, while rating scale choices can impact marginal reliability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.