Reproduce “Self-Distilled RLVR” on OpenResearch

About this paper

Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang and 2 more

RLSD uses teacher token-level assessments to reweight credit within visual reasoning responses, while verifier-derived rewards determine whether each response is reinforced or penalized. This separates the direction of updates from token-level credit magnitude. On Qwen3-VL-8B-Instruct, RLSD averages 56.18% across five benchmarks, compared with GRPO’s 53.86%; SDPO scores higher on ZeroBench. Results are limited mainly to multimodal reasoning.

Key insights

• OPSD's failure stems from a theoretical limitation: an irreducible conditional mutual information gap in its objective function, rendering the distribution-matching ill-posed.
• Privileged information leakage and performance degradation in OPSD are caused by a two-phase training dynamic where `r`-specific deviation components in gradients corrupt parameter updates.
• Decoupling credit assignment into direction (from verifiable outcomes) and magnitude (from self-distillation) allows for stable, leakage-free, and efficient fine-grained learning in LLMs.

Summary by alphaXiv.