Reproduce “When Maximum Entropy Misleads Policy Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruipeng Zhang, Ya-Chien Chang, Sicun Gao

Maximum Entropy Reinforcement Learning (MaxEnt RL) algorithms can actively mislead policy optimization in precise, real-world control tasks, despite their benefits in exploration and robustness. A theoretical framework, the Entropy Bifurcation Extension, demonstrates how entropy regularization biases agents toward suboptimal, high-entropy actions, a limitation mitigated by an adaptive approach that improves performance in challenging environments like quadrotor and vehicle control where standard MaxEnt RL fails.

Key insights

  • Entropy regularization, while beneficial for exploration and robustness, can bias MaxEnt RL policies towards suboptimal, high-entropy actions in critical states where precision is required.
  • The Entropy Bifurcation Extension, a novel theoretical construction, formally demonstrates how MaxEnt-optimal policies can be systematically misled to match arbitrary or even the worst possible behaviors.
  • The 'soft Q-value' landscapes in MaxEnt methods can be flattened by entropy regularization, obscuring the precise, low-entropy actions necessary for success in complex control tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.