Reproduce “Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuanyang Zhu, Zhi Wang, Yuanheng Zhu, Chunlin Chen, Dongbin Zhao

A novel unimodal policy based on Poisson distributions is introduced for discretizing continuous action spaces in on-policy reinforcement learning, achieving faster convergence and higher final rewards on MuJoCo tasks, such as 7072 ± 409 rewards on Humanoid-v3 compared to 4886 ± 226 for Gaussian policies, while significantly reducing network output units. This approach leverages action continuity, enabling on-policy methods to compete with state-of-the-art off-policy algorithms.

Key insights

  • Explicitly enforcing a unimodal probability distribution, such as with Poisson distributions, over discretized action spaces effectively leverages the inherent continuity of actions, leading to more stable and higher-performing policies.
  • The Poisson distribution, with its single rate parameter, provides a parameter-efficient way to model the probability distribution over discrete action bins derived from continuous dimensions.
  • The proposed architecture significantly reduces the number of network output units (M vs. M*K), which contributes to a theoretically lower variance for the policy gradient estimator and improved scalability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.