Reproduce “Action Space Shaping in Deep Reinforcement Learning” on OpenResearch

About this paper

Anssi Kanervisto, Christian Scheller, Ville Hautamäki

Researchers empirically evaluate the impact of various action space transformations on deep reinforcement learning agent performance across multiple video game environments. The study finds that removing unnecessary actions and discretizing continuous actions significantly improve learning efficiency and often enable agents to learn in complex settings, while converting multi-discrete to single discrete actions offers no consistent benefit and can introduce scaling challenges.

Key insights

• Empirical evidence suggests that modifying an agent's action space can significantly influence deep reinforcement learning efficiency and success, analogous to reward shaping.
• Removing unnecessary actions and discretizing continuous actions generally improve an agent's ability to learn, often being critical for learnability in complex environments.
• Converting multi-discrete action spaces into a single discrete action space does not consistently improve learning and can introduce combinatorial scaling challenges.

Summary by alphaXiv.