Reproduce “RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhaoning Yu, Will Su, Leitian Tao, Haozhu Wang, Aashu Singh, Hanchao Yu, Jianyu Wang, Hongyang Gao and 4 more

A self-driven reinforcement learning framework called RESTRAIN enables Large Language Models to improve reasoning capabilities without human-annotated labels by converting the absence of supervision and model uncertainty into robust learning signals. This approach surpasses existing unsupervised methods and nearly matches the performance of models trained with full gold-label supervision.

Key insights

  • The absence of gold labels and the presence of model uncertainty can be systematically leveraged as valuable and robust learning signals for LLMs.
  • A soft, frequency-aware weighting over the full distribution of model-generated answers yields more stable and effective learning than hard majority voting.
  • Explicitly penalizing highly inconsistent or low-confidence rollouts is crucial for preventing model collapse and encouraging exploration in unsupervised RL.
  • Pre-calculating and fixing prompt-level weights offline is vital for maintaining training stability and avoiding spurious feedback loops during dynamic policy optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.