Reproduce “High-Dimensional Continuous Control Using Generalized Advantage Estimation” on OpenResearch

About this paper

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel

The authors analyze generalized advantage estimation (GAE) for policy-gradient reinforcement learning, combining temporal-difference residuals across time with exponentially decreasing weights. The parameter lambda trades lower variance against bias from an inaccurate value function, while gamma has a separate discounting effect. With trust-region updates, neural policies learn simulated biped and quadruped gaits; this demonstrates simulated control, not real-robot learning.

Key insights

• Generalized Advantage Estimation (GAE) provides a principled bias-variance tradeoff for policy gradient estimates through its lambda parameter, where intermediate values often yield superior performance over extreme values.
• GAE can be interpreted as reward shaping, where the approximate value function implicitly immediatezes long-term rewards, transforming the credit assignment problem.
• Using a well-trained value function is crucial for significant variance reduction in policy gradient methods, enabling more stable and efficient learning.

Summary by alphaXiv.