Reproduce “A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong
This research introduces a framework for evaluating LLM-as-a-judge based on construct validity, formalizing both invariance and construct sensitivity, and empirically demonstrating that current LLM judges are significantly more invariant (S=0.945) than sensitive (R=0.319) to actual changes in a target construct. The study also reveals a consistent asymmetry in how judges detect changes in claim scope versus strength, and finds that public evaluation benchmarks are susceptible to construct-blind surface predictors.
Key insights
- Invariance and construct sensitivity are independent properties, meaning a judge can be highly reliable without accurately detecting changes in the underlying construct.
- LLM judges exhibit a structured failure mode, showing significantly higher sensitivity to changes in claim scope (0.383 R_scope) than in claim strength (0.262 R_strength).
- Many public LLM evaluation benchmarks "leak" surface-form information, with construct-blind predictors reproducing 55% to 67% of labels in paired evaluation modes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.