Reproduce “Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spaces” on OpenResearch

About this paper

Guy Lorberbom, Chris J. Maddison, Nicolas Heess, Tamir Hazan, Daniel Tarlow

The authors introduce DirPG, a method that searches for policy-plausible action sequences with higher perturbed objectives, then updates the policy by contrasting a searched trajectory with an ordinary policy sample. In MiniGrid-MultiRoom-N6-v0, average return improved as search priority rose from 0 to 0.3, then declined at higher settings. Learning requires a simulator and complete episodes; deployment does not.

Key insights

• DirPG optimizes a specific objective function, a log-expectation of exponential returns, which precisely approximates the true policy gradient as the $\epsilon$ parameter approaches zero.
• The $\epsilon$ parameter directly controls the agent's preference for 'controllable risk' (policy-induced variance), leading to risk-seeking (for $\epsilon > 0$) or risk-averse (for $\epsilon < 0$) behaviors.
• The component of the gradient related to the $a_{opt}$ trajectory acts as an inherent control variate, providing a built-in mechanism for variance reduction.

Summary by alphaXiv.