Reproduce “Batch size-invariance for policy optimization” on OpenResearch

About this paper

Jacob Hilton, Karl Cobbe, John Schulman

The authors introduce decoupled PPO: the behavior policy supplies importance sampling, while a separate recent policy controls updates. PPO-EWMA uses an exponentially weighted average of policy-network weights as update reference. Tests found approximate performance invariance across adjusted batch sizes and minimal degradation through about eight iterations of artificial data staleness; they do not establish robustness or speedup in asynchronous systems.

Key insights

• The "old" policy in PPO serves two distinct and separable purposes: importance sampling (requiring the behavior policy) and controlling the rate of policy change (which can use any recent policy).
• Policy optimization algorithms like PPO are better understood as natural policy gradient methods, where controlling the rate of policy change relative to a recent, averaged policy is paramount for stability and efficiency.
• Achieving batch size-invariance in policy optimization necessitates careful, specific adjustments to multiple hyperparameters, with the Adam step size being the most critical.

Summary by alphaXiv.