Reproduce “Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiqiang Tan
The authors analyze a single-loop entropy-regularized natural policy-gradient method for unregularized reinforcement learning. At each iteration, it takes one stochastic critic step using uncentered features, then updates the policy. With tabular features, exploratory restarts, unbiased Monte Carlo estimates, and a positive action gap, its average-iterate performance gap is \(\tilde O(T_{\text{total}}^{-2/3})\), versus \(O(T_{\text{total}}^{-1/2})\) without the margin; the faster rate is conditional, not worst-case.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.