Reproduce “DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, Yuting Liu
This paper presents the Decoupled Group Reward Optimization (DGRO) algorithm, which improves Large Language Model reasoning by decoupling the exploration-exploitation trade-off in reinforcement learning. DGRO achieves superior accuracy on logical and mathematical reasoning tasks, and its theoretical analysis clarifies the influence of reward variance and decoupled coefficients on optimization dynamics.
Key insights
- Decoupling the regularization coefficient into separate policy gradient (β₁) and policy distance (β₂) parameters allows for more precise control over the exploration-exploitation trade-off in LLM fine-tuning.
- Reward variance influences learning dynamics; wider reward ranges can lead to better performance and different training patterns.
- The theoretical analysis provides an error bound for the soft value function and explains how β₁ and β₂ affect convergence and stability, offering guidance for parameter tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.