Reproduce “Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models” on OpenResearch

About this paper

Haitao Jiang, Wenbo Zhang, Jiarui Yao, Hengrui Cai, Sheng Wang, Rui Song

Researchers from North Carolina State University, UC Irvine, University of Illinois Urbana-Champaign, University of Washington, and Amazon present a systematic comparison of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Large Language Model post-training, establishing a unified framework that positions SFT as a special case of RL. Their analysis highlights a rapid shift towards hybrid SFT-RL paradigms in recent application studies.

Key insights

• SFT can be formally regarded as a special case of RL, where an indicator function for exact trajectory reproduction serves as a surrogate reward, facilitating the transfer of techniques between paradigms.
• Hybrid training paradigms, which combine SFT and RL objectives, have become the most common approach for LLM post-training, indicating a convergence leveraging their complementary strengths.
• SFT is effective for initializing models or generating positive samples but may reduce generalization, while RL offers better balance between performance and exploration but is prone to stability issues.

Summary by alphaXiv.