Reproduce “Reinforcement Learning by Value Gradients” on OpenResearch

About this paper

Michael Fairbank

The paper studies value-gradient learning for deterministic control with a differentiable model. The method learns how predicted reward changes with state, propagating gradients along greedy trajectories to expose better neighboring paths without local exploration. In a two-step toy problem, VGL(1) achieved 100% success in 1,596.8 iterations versus 244,122 for VL(1), but the result is local and limited to λ=1 conditions.

Key insights

• Value-gradients implicitly provide information about superior neighboring trajectories, eliminating the need for explicit exploration to find locally optimal solutions in deterministic environments.
• The VGLΩ(1) algorithm is mathematically equivalent to Policy Gradient Learning (specifically Back-Propagation Through Time for a greedy policy), providing the first robust convergence guarantees for a class of value function learning algorithms in deterministic control with function approximators.
• Residual gradient methods can converge to spurious local minima that do not correspond to optimal policies, whereas VGLΩ(1) performs true gradient ascent on the total reward.

Summary by alphaXiv.