Reproduce “Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunhao Tang, Sid Wang, Lovish Madaan, Rémi Munos

Meta researchers introduce JEPO (Jensen's Evidence lower bound Policy Optimization), a reinforcement learning algorithm enabling large language models to train effectively on datasets where output correctness is not easily verifiable. The method leverages an intrinsic likelihood signal derived from a latent chain-of-thought, demonstrating comparable performance to online RL on verifiable data, and outperforming supervised fine-tuning on complex, unverifiable mathematical proofs.

Key insights

  • Reinforcement Learning can be extended to unverifiable data by optimizing the likelihood of a given ground truth output conditioned on a sampled chain-of-thought (CoT).
  • Jensen's inequality offers a computationally tractable lower bound on the marginal log-likelihood of the ground truth, eliminating the need for expensive auxiliary inference models common in full ELBO methods.
  • JEPO's gradient estimate intrinsically incorporates both an RL-like term (using ground truth likelihood as reward) and a supervised learning term, exhibiting inherent variance reduction properties.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.