Reproduce “Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks” on OpenResearch

About this paper

Delip Rao, Chris Callison-Burch

Autorubric, an open-source framework from the University of Pennsylvania, systematizes rubric-based LLM evaluation for non-verifiable tasks by unifying methodologies and introducing explicit rubric design choices. It demonstrated improved accuracy with calibration on grading tasks and provided actionable optimization signals for LLM agent development and reinforcement learning.

Key insights

• Few-shot calibration significantly improves LLM judge accuracy on complex grading tasks, while per-criterion analysis highlights specific areas of agreement or systematic disagreement.
• LLM judge family is the most significant source of variation in evaluation, and optimal mitigation strategies (e.g., ensemble size, shuffling) are model- and criterion-dependent, suggesting no universal "best" configuration.
• Per-criterion feedback generated by Autorubric provides effective, actionable optimization signals, enabling targeted skill revision for LLM agents and serving as continuous reward signals for reinforcement learning policies.

Summary by alphaXiv.