Reproduce “RISE: Recursive Improvement via Self-Extrapolating Policy Distillation” on OpenResearch

About this paper

Yang Li, Semih Yavuz, Shafiq Joty

RISE turns a model’s reward-grounded training update into a self-extrapolating teacher and distills its token-level guidance back into the student. It reuses RLVR rollouts but adds a distillation phase. On Qwen3-8B trained on DAPOMath, RISE (weight) reaches an unweighted six-benchmark Math Avg of 62.7, versus 60.0 for GRPO. This supports gains in the tested setup, not universal self-improvement.

Key insights

• Training trajectories in LLM parameter or logit spaces are sufficiently low-dimensional and approximately linear, allowing for effective and safe extrapolation of the policy's future direction.
• Outcome rewards from RLVR are crucial for grounding the extrapolation in verifiable improvement, as purely self-distilled extrapolation or direct adoption of extrapolated parameters without distillation can lead to training instability or negligible gains.
• Token-level distillation effectively projects the extrapolated teacher's distribution onto the student, converting sparse outcome signals into a dense, fine-grained supervision signal that is critical for policy refinement.

Summary by alphaXiv.