Reproduce “PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenquan Lu, Hai Huang, Enqi Liu, Randall Balestriero
PrAg-PO introduces a policy optimization method that incorporates diverse prompt templates and format rewards during reinforcement learning for large language models. This approach significantly improves mathematical reasoning accuracy, elicits diverse reasoning trajectories, and enhances training stability across various model architectures.
Key insights
- Prompt augmentation acts as a gradient regularizer, promoting more robust and generalizable learning by smoothing gradient signals and reducing overfitting to single prompt-reward interfaces.
- Explicit template-specific format rewards are crucial for eliciting and maintaining diverse reasoning structures, as their absence leads to models ignoring prompt instructions and reverting to free-form generation.
- The performance gains observed are a collective benefit of training with a diverse set of prompts, fostering robustness to prompt variations rather than being attributable to a single 'gold' prompt.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.