Reproduce “Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning” on OpenResearch

About this paper

Bowen Ding, Yuhan Chen, Jiayang Lyv, Jiyao Yuan, Qi Zhu, Shuangshuang Tian, Dantong Zhu, Futing Wang and 4 more

Researchers developed the Plasticity-Ceiling Framework to analyze LLM post-training for mathematical reasoning, establishing the sequential Supervised Fine-Tuning (SFT)-then-Reinforcement Learning (RL) pipeline as the superior approach. This method achieved a peak of 78.1 points on a composite mathematical benchmark, demonstrating that optimizing SFT to saturation maximizes the ultimate performance ceiling and preserves out-of-domain generalization.

Key insights

• The sequential SFT-then-RL pipeline consistently outperforms other post-training paradigms, indicating that a robust SFT foundation is crucial for maximizing performance in mathematical reasoning.
• The optimal SFT-to-RL transition point is identified as the SFT Stable or Mild Overfitting Regime, ensuring both high ultimate performance and preserved out-of-domain generalization.
• Data scale is the primary determinant of the post-training performance ceiling, with trajectory difficulty acting as a performance multiplier when data scale is limited; minimum SFT validation loss serves as a reliable predictive indicator.

Summary by alphaXiv.