Reproduce “NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Lifan Yuan, Yin Cui, Haotian Ye, Tsung-Yi Lin and 3 more
Negative-aware Fine-Tuning (NFT) introduces a supervised learning method that enables large language models to learn from self-generated incorrect answers in math reasoning tasks. This approach achieves competitive performance with reinforcement learning algorithms while offering a more memory-efficient alternative for autonomous model improvement.
Key insights
- NFT reveals a theoretical equivalence between supervised learning and Group Reward Policy Optimization (GRPO) in strict-on-policy training, suggesting a deeper connection between SL and RL paradigms.
- Learning from self-generated negative examples is crucial for LLM self-improvement, providing significant performance gains over positive-only learning and fostering exploratory behavior.
- The ability to leverage negative feedback becomes increasingly important for larger models, where the performance gap between RFT and NFT widens more rapidly.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.