Reproduce “Refined Analysis of Entropy-Regularized Actor-Critic” on OpenResearch

About this paper

Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines

The authors analyze actor-critic learning in finite discounted tabular Markov decision processes with entropy regularization. They show that an exact critic makes actor-gradient variance vanish as the deterministic gradient approaches zero, while a learned critic leaves bias and variance tied to its estimation error. Under stated step-size and update conditions, the learned-critic method achieves a tilde O(1/epsilon) sample complexity; the exact-critic rate is tilde O(log(1/epsilon)).

Key insights

• Actor-Critic methods, when using an exact critic, function as strong variance-reduction algorithms, implying that the variance of the stochastic policy gradient estimator vanishes entirely as the algorithm approaches an optimal policy.
• The convergence rate of Ent-AC with an inexact critic is primarily driven by the critic's bias and variance, rather than the actor's sampling variance, which is absorbed into the descent term.
• The frequency of critic updates (`H`) relative to actor updates is theoretically and empirically crucial, as a larger `H` significantly reduces critic bias and variance, leading to faster and more stable convergence.

Summary by alphaXiv.