Reproduce “Self-Distilled Policy Gradient” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifeng Liu, Shiyuan Zhang, Yifan Zhang, Quanquan Gu
SDPG trains language models for mathematical reasoning by combining verifier-based policy optimization with full-vocabulary reverse-KL guidance from the same model conditioned on privileged context. It gates this guidance to verifier-favored rollouts and anchors training to a reference policy. On Qwen3-4B, both variants outperform GRPO and RLSD on three benchmarks’ final scores, but results are limited to these math experiments.
Key insights
- Principled integration of sparse global outcome rewards with dense token-level self-distillation, along with stability mechanisms, is crucial for robust and effective LLM training on complex reasoning tasks.
- The full-vocabulary reverse KL divergence used for on-policy distillation can be interpreted as a detached-sampling policy-gradient update, providing a theoretical understanding of its signal.
- KL regularization against a reference policy and positive-advantage gating are essential for preventing mode collapse and maintaining policy coherence, particularly in pure self-distillation settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.