Reproduce “S-GRPO: Unified Post-Training for Large Vision-Language Models” on OpenResearch

About this paper

Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu

S-GRPO introduces a unified post-training framework that seamlessly integrates supervised learning into preference-based reinforcement learning for Large Vision-Language Models. This method effectively adapts models to specialized domains while robustly preserving general multimodal capabilities, overcoming issues like optimization collapse and catastrophic forgetting seen in prior approaches.

Key insights

• SFT can be viewed as a constrained RL process where ground-truth trajectories serve as high-advantage anchors within a policy gradient framework, enabling a unified approach to LVLM adaptation.
• Dynamically injecting ground-truth trajectories guarantees a non-zero standard deviation in group rewards, mathematically preventing optimization collapse and ensuring consistent advantage estimates even in sparse-reward environments.
• A conditional injection mechanism allows the model to seamlessly transition from expert-guided bootstrapping during initial struggles to autonomous policy exploration as its capabilities improve, without fixed curriculum schedules.

Summary by alphaXiv.