Reproduce “Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge” on OpenResearch

About this paper

Zijie Wang, Eduardo Blanco

The paper presents a rubric-generation method that creates instance-specific evaluation criteria from prompts without reference responses and fine-tunes a separate rubric generator using an LLM meta-judge’s comparisons of candidate rubrics. On MT-Bench, a Claude Sonnet 4 judge agreed with human majority choices on 83.69% of pairs, versus 81.72% for the strongest training-free comparison, with p less than 0.05; evaluation used the same benchmarks for training and testing and did not test out-of-domain transfer.

Key insights

• Instance-specific rubrics consistently outperform dataset-specific rubrics, demonstrating that tailoring evaluation criteria to individual inputs provides more effective guidance for LLM judges.
• The quality of LLM evaluation can be partially decoupled from the scale of the judge LLM; a smaller, specialized and fine-tuned rubric generator can produce higher-quality rubrics than a larger judge generating its own.
• Iterative preference learning, driven by a meta-judge without human annotation, successfully refines rubric generators, leading to consistent performance improvements and a qualitative shift towards more discriminative, dimension-labeled criteria.

Summary by alphaXiv.