Reproduce “Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chao Xue, Yao Wang, Mengqiao Liu, Di Liang, Xingsheng Han, Peiyang Liu, Xianjie Wu, Chenyao Lu and 6 more

An Efficient Generative Reward Model (E-GRM) reduces computational overhead in Large Language Models by dynamically triggering Chain-of-Thought reasoning only when internal model uncertainty indicates complexity, achieving a 62% reduction in latency and 49% in FLOPs on reasoning tasks. It also improves reward signal precision through a discriminative scoring module, leading to state-of-the-art performance on multiple reward modeling benchmarks.

Key insights

  • Model-internal uncertainty, derived from the convergence behavior of parallel model generations, serves as a reliable and task-agnostic signal for dynamically allocating computational resources (i.e., activating CoT only when needed).
  • Combining robust regression with discriminative ranking in a hybrid loss objective enables a scoring module to effectively estimate absolute quality and discern relative quality differences between reasoning paths.
  • Explicitly structuring reward functions to leverage paired preference data during alignment optimization enhances the precision and stability of the learning signal for improving reward model fidelity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.