Reproduce “GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation” on OpenResearch

About this paper

Yifan Chen, Haitao Li, Qingyao Ai, Fengbin Zhu, Tat-Seng Chua, Min Zhang, Yiqun Liu

GenRubric learns to generate query-specific evaluation rubrics through self-evolution, using queries alone during reinforcement learning after supervised fine-tuning. It samples multiple rubrics, generates responses conditioned on them, and rewards rubrics whose responses satisfy criteria from other sampled rubrics. Across 700 queries, GenRubric-8B achieves Spearman rank correlation 0.4525 versus GPT-5.2’s 0.4343 against expert-rubric rankings; this measures ranking agreement, not answer quality.

Key insights

• Rubric quality can be implicitly estimated through 'rubric-induced self-consistency,' where the generalization of a rubric-induced response across other independently sampled rubrics for the same query serves as an annotation-free learning signal.
• Reinforcement learning, particularly with the cross-rubric comprehensiveness reward, effectively optimizes LLM parameters to generate high-quality evaluation rubrics without direct human supervision during the self-evolution process.
• A crucial supervised fine-tuning (SFT) cold-start phase provides foundational instruction-following and structured rubric generation capabilities, which are then significantly refined and enhanced by the self-evolutionary RL stage.

Summary by alphaXiv.