Reproduce “Equivalence of stochastic and deterministic policy gradients” on OpenResearch

About this paper

Emo Todorov

The authors show that stochastic and deterministic policy gradients are equal under Gaussian control noise and quadratic control costs, with general dynamics and state costs. Moving noise from action selection into the dynamics preserves state transitions; the resulting state values differ only by a constant, while state-control value functions differ. The result does not establish that practical algorithms perform alike.

Key insights

• For Quadratic-Gaussian MDPs, the theoretically exact stochastic and deterministic policy gradients, along with natural gradients (after metric adjustment), are identical.
• The state-control value functions (Q-functions) are the only quantities that differ fundamentally between the stochastic and deterministic formulations, explaining why practical algorithm performance diverges.
• Any stochastic policy MDP can be systematically decomposed and transformed into an equivalent deterministic policy MDP, demonstrating a universal theoretical bridge between the two paradigms.

Summary by alphaXiv.