Reproduce “Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoqun Cao, Yurun Yuan, Tengyang Xie

The authors analyze off-policy updates for language-model post-training, showing when KL regularization and a prompt-grouped mean baseline preserve the optimum as a stationary point. This route relies on deterministic transitions; shifting the baseline gives a comparator-coverage regret guarantee under additional assumptions. In a synthetic binary-tree experiment, AsymRE regret is 59.1 × 10⁻³ versus GRPG’s 1329 × 10⁻³ at ε = 0.01, not an LLM result.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.