Reproduce “A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Audrey Huang, Nan Jiang, Sham M. Kakade, Sadhika Malladi, Akshay Krishnamurthy, Dylan J. Foster
The authors introduce Approximate Policy Mirror Descent (APMD), which repeatedly resets the KL-regularization reference to the latest policy and makes strongly regularized local updates. They define response coverage as the average reciprocal probability of generating an optimal response. In a constructed bandit, REINFORCE needs exponentially many gradient steps, while stylized PPO-NoClip has a polynomial bound under softmax-linear policies and linear rewards.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.