Reproduce “CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling” on OpenResearch

About this paper

Dengcan Liu, Fengkai Yang, Xiaohan Wang, Shurui Yan, Jiajun Chai, Jiahao Li, Yikun Ban, Zhendong Mao and 2 more

The CDRRM framework introduces a "Contrast-then-Synthesis" paradigm to generate fine-grained, interpretable rubrics for reward modeling, yielding improved preference prediction accuracy and enhanced robustness against evaluator biases like verbosity and position bias. The approach achieves these gains using significantly less training data than previous methods.

Key insights

• Contrastive analysis of preference pairs, grounded in explicit evidence, is highly effective for synthesizing succinct, discriminative, and context-aware evaluation rubrics.
• Explicitly conditioning preference judgments on structured rubrics provides transparent reasoning and helps mitigate systemic LLM evaluator biases, such as verbosity and position bias.
• The proposed framework enables high data efficiency, allowing the training of effective rubric generators and judge models with significantly fewer samples than traditional methods.

Summary by alphaXiv.