Reproduce “Model-based Reinforcement Learning for Parameterized Action Spaces” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Renhao Zhang, Haotian Fu, Yilin Miao, George Konidaris

DLPA introduces the first model-based reinforcement learning algorithm for Parameterized Action Markov Decision Processes (PAMDPs), achieving an average of 30 times higher sample efficiency and superior or comparable asymptotic performance over state-of-the-art model-free baselines across eight benchmarks and a complex robotics environment. It accomplishes this by learning an explicit dynamics model tailored for hybrid actions and employing a specialized Model Predictive Path Integral for planning.

Key insights

  • Model-based approaches can achieve substantially higher sample efficiency and robust performance in PAMDPs compared to data-hungry model-free methods.
  • Specialized inference model architectures, such as the sequential approach, are crucial for effectively representing the entangled discrete-continuous nature of parameterized actions within a dynamics model.
  • Using a multi-step prediction loss for model training and employing dedicated reward predictors for terminal states significantly improves the learned dynamics model's accuracy and utility for planning in PAMDPs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.