Reproduce “Libra: Assessing and Improving Reward Model by Learning to Think” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Meng Zhou, Bei Li, Jiahao Liu, Xiaowen Shi, Yang Bai, Rongxiang Weng, Jingang Wang, Xunliang Cai

Meituan researchers developed LIBRA, a framework that includes Libra Bench, a benchmark for evaluating reward models on complex mathematical reasoning, and Libra-RM, a series of generative reward models trained to "learn to think." Libra-RM achieves 81.7% accuracy on Libra Bench and demonstrates strong correlation with improved downstream policy model performance in DPO experiments, indicating its potential for scalable reinforcement learning with unlabeled data.

Key insights

  • Generative reward models explicitly trained with "learning-to-think" capabilities significantly outperform traditional discriminative RMs and general LLM-as-a-Judge methods on complex reasoning tasks.
  • The ability to accurately verify incorrect samples is a critical differentiator for robust reasoning RMs, with Libra-RM demonstrating superior performance in this aspect.
  • Incorporating non-judging (general reasoning and answering) data during SFT improves an RM's performance across both reasoning and general evaluation benchmarks, suggesting an intrinsic link between generative and evaluative capabilities in LLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.