Reproduce “Variational Reasoning for Language Models” on OpenResearch

About this paper

Xiangxin Zhou, Zichen Liu, Haonan Wang, Chao Du, Min Lin, Chongxuan Li, Liang Wang, Tianyu Pang

A variational reasoning framework is introduced for large language models, treating thinking traces as latent variables within a probabilistic objective. This approach consistently improves accuracy over supervised finetuning and reinforcement learning methods across various reasoning benchmarks, while also demonstrating enhanced training stability and generalization.

Key insights

• The variational framework unifies existing methods like Rejection Sampling Finetuning and binary-reward RL, showing they can be re-expressed as local forward KL optimizations implicitly weighted by model accuracy.
• Common RL-style methods exhibit an unnoticed systematic bias towards easier questions, as their optimization inherently prioritizes problems where the model already performs well.
• An accuracy-based estimator for the probability of a correct answer given a thought process is theoretically and empirically superior, offering lower variance and more robust training compared to likelihood-based methods.

Summary by alphaXiv.