Reproduce “Mirror Descent Actor Critic via Bounded Advantage Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ryo Iwaki
IBM Research - Tokyo developed Mirror Descent Actor Critic (MDAC), an actor-critic algorithm for continuous control that applies bounding functions to log-density terms within its critic loss. This approach stabilizes KL-entropy regularized methods, enabling them to outperform strong entropy-only regularized baselines like SAC in complex continuous control tasks.
Key insights
- Bounding the log-density terms (`α log π`) in the critic's loss is crucial for stabilizing KL-entropy regularized methods in continuous control, preventing divergence and enabling effective learning.
- Slower-saturating bounding functions (e.g., `clip(x/10, -1, 1)`) generally perform better than aggressive ones, as they retain more relative information about the advantage function.
- The bounding mechanism can be understood as a form of Bounded Advantage Learning (BAL) that reduces inherent approximation errors in the Bellman operator and ensures asymptotic convergence under specific conditions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.