Reproduce “Efficient Reasoning via Reward Model” on OpenResearch

About this paper

Yuhao Wang, Xiaopeng Li, Cheng Gong, Ziru Liu, Suiyun Zhang, Rui Liu, Xiangyu Zhao

Researchers from City University of Hong Kong and Huawei Research developed a Conciseness Reward Model (CRM) and an innovative Conciseness Reward Function (CRF) to enable Large Reasoning Models (LRMs) to generate more efficient and accurate solutions. This approach delivered an 8.1% improvement in Pass@1 accuracy and a 19.9% reduction in token length on the Qwen2.5-7B model, successfully mitigating common issues like length and training collapse.

Key insights

• An explicit dependency between outcome correctness and conciseness in the reward function is crucial for preventing "length collapse" and "training collapse," common pitfalls of simpler length penalties.
• Leveraging powerful LLMs for data augmentation and conciseness scoring provides a scalable and robust alternative to human annotation for training effective reward models.
• The proposed Conciseness Reward Function theoretically reduces the variance of the stochastic gradient estimator and improves convergence properties, leading to more stable and efficient reinforcement learning.

Summary by alphaXiv.