Reproduce “Inference-Time Scaling for Generalist Reward Modeling” on OpenResearch

About this paper

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu

DeepSeek-AI and Tsinghua University developed Self-Principled Critique Tuning (SPCT) and DeepSeek-GRM models, a pointwise generative reward modeling approach that achieves state-of-the-art performance in generalist reward benchmarks. This method demonstrates effective inference-time scalability, allowing smaller models to match or exceed the performance of much larger models by leveraging increased compute during inference, thereby improving LLM alignment.

Key insights

• Proper learning methods, such as SPCT, can instill "scalable behaviors" in RMs, enabling them to consistently generate higher-quality reward signals with increased inference compute.
• Pointwise generative reward modeling inherently allows for input flexibility and diverse outputs, crucial for effective sampling-based inference-time scaling across general domains.
• Inference-time scaling with a well-trained GRM can achieve performance comparable to or better than traditional training-time scaling (larger model sizes), offering a more compute-efficient path to high-quality RMs.

Summary by alphaXiv.