Reproduce “When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff” on OpenResearch

About this paper

Runze Liu, Jiashun Liu, Xu Wan, Yuqian Fu, Ling Pan

This research investigates why Large Language Models (LLMs) trained with excessive Supervised Fine-Tuning (SFT) struggle to benefit from subsequent Reinforcement Learning (RL) and introduces "Rejuvenation," a post-hoc method. It restores model plasticity by combining global base-anchored linear fusion with attribution-guided neuron resetting, enabling robust SFT-to-RL transfer and improving performance on mathematical and agentic tasks, particularly enhancing out-of-distribution generalization.

Key insights

• Excessive SFT leads to substantial, sharp parameter shifts away from the base model and causes over-confident, low-entropy token distributions (e.g., entropy drops from 0.184 for ModSFT to 0.024 for OverSFT), impairing subsequent RL optimization.
• Over-trained models exhibit larger gradient norms during RL but result in smaller effective weight update magnitudes and reduced reward improvement, indicating a loss of "effective gradients" and inherent rigidity.
• The loss of model plasticity due to over-training is a primary factor in the SFT-to-RL handoff dilemma, preventing LLMs from fully leveraging reward signals for performance gains, especially on out-of-distribution tasks.

Summary by alphaXiv.