Reproduce “ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, Tom Zahavy
ReLOAD stabilizes constrained reinforcement learning by applying optimistic gradient updates to both the policy and its Lagrange multiplier, using current and previous gradients or value estimates. On oscillation-prone control tasks, ReLOAD-IMPALA achieved 549.0 ± 2.0 weighted reward on Walker height versus 487.5 ± 2.8 for OGD-IMPALA. Constraint violations remained nonzero, and guarantees cover only convex occupancy measures.
Key insights
- Standard primal-dual Mirror Descent methods, including Gradient Descent-Ascent (GDA), are theoretically shown to not guarantee last-iterate convergence for general convex-concave saddle-point problems, often leading to oscillations or divergence.
- Achieving last-iterate convergence in min-max games requires optimistic updates on both the primal (policy) and dual (Lagrange multipliers) sides of the optimization problem, as single-sided optimism is insufficient.
- The CMDP Lagrangian can be mapped to a monotone inclusion problem, allowing the application of generalized OMD, which can accommodate different Bregman divergences for policy and Lagrange multiplier updates, to prove LIC.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.