Reproduce “Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning” on OpenResearch

About this paper

Zikang Shan, Han Zhong, Liwei Wang, Li Zhao

GenAC revisits actor–critic learning for LLM reinforcement learning by replacing one-shot scalar critics with autoregressive critics that reason about partial solutions and the evolving actor’s capabilities. On mathematical reasoning tasks, it improves average accuracy to 51.90%, compared with 50.22% for GRPO, 49.93% for RLOO, and 50.39% for a conventional-critic PPO baseline, at roughly 2.1× the per-iteration cost of standard PPO.

Key insights

• Generative critics achieve lower value-prediction error, improve more consistently with model scale, and show less sensitivity to random seeds than discriminative scalar critics.
• Autoregressive value estimation is especially helpful for ranking candidate continuations: GenAC reaches top-1 ranking accuracies of 0.711, 0.492, and 0.332 with 2, 4, and 8 candidates, respectively.
• Conditioning on the current actor reduces critic error, while supervised pretraining mainly stabilizes reasoning format and length; subsequent RL pretraining provides most of the value-estimation improvement.

Summary by alphaXiv.