Reproduce “On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models” on OpenResearch

About this paper

Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski, Federico Tombari

OPST trains models on self-generated questions from broad topic descriptions, without curated training questions or answers. A question-only student learns from token-level teacher guidance, while the teacher sees the generated candidate answer and rationale. On Gemma 4 E2B, MMLU rises from 37.5% to 47.7%, versus 38.3% for SFT and 37.3% for GRPO on the same data. Experiments use small models; verification checks consistency, not correctness.

Key insights

• Generated answers and rationales can serve as teacher context without being treated as ground-truth targets, making the training signal less directly dependent on exact synthetic labels.
• Filtering for self-consistency and student uncertainty helps shape a changing curriculum, though consistency does not establish factual correctness.
• The approach can improve targeted benchmarks, but transfer is uneven; the reported evidence is limited to selected model sizes and evaluation settings.

Summary by alphaXiv.