Reproduce “An Expectation-Maximization Perspective on Reinforcement Learning for LLM Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianbing Xu
EMPG trains reasoning models by sampling scored rationale-and-answer trajectories, then updating the model to favor higher-reward attempts. Its reward-weighted update omits probability-ratio corrections and clipping. Across tested Qwen2.5 models and tasks, results are generally comparable to or better than GRPO, with exceptions; on GSM8K, Qwen2.5-7B scores 0.888 versus 0.867. Qwen2.5-Math-7B traces are shorter, but elapsed inference time was not measured.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.