Reproduce “Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention” on OpenResearch

About this paper

Wenye Lin, Kai Han

The Surgical Post-Training (SPOT) framework enables Large Language Models to acquire new reasoning capabilities while preserving existing knowledge, employing a data rectification pipeline and a reward-based binary optimization objective. This approach achieved an average accuracy improvement of 6.2% for Qwen3-8B across reasoning tasks, enhanced general instruction following, and provided a more stable foundation for further reinforcement learning.

Key insights

• The KL-constrained reward formulation in preference optimization methods serves as an 'Elastic Tether' regularization, dynamically limiting policy divergence from the reference model and critically preventing catastrophic forgetting.
• Generating proximal on-policy data through 'surgical rectification' focuses learning on minimal, decision-critical changes in reasoning paths, which is more effective than direct off-policy data or pure on-policy sampling.
• Decoupling positive and negative supervision into a binary classification objective (SPoT-BCE/BCO) directly addresses the 'pull-up' effect and is more effective for learning objectively verifiable reasoning than relative preference ranking.

Summary by alphaXiv.