Reproduce “Revisiting Action Factorization for Complex Action Spaces” on OpenResearch

About this paper

Timothy Flavin, Sandip Sen

This research systematically evaluates various action factorization strategies across major deep reinforcement learning algorithms (DQN, PPO, SAC) and action space types (discrete, continuous, hybrid). The study provides empirical guidance for model selection, demonstrating that auto-regressive actions can achieve high performance at a greater computational cost, while branching dueling/VDN architectures offer a balanced approach with robust performance and efficiency.

Key insights

• Action factorization through branching dueling architectures (e.g., VDN-style) offers an effective balance of performance, computational cost, and implementation overhead across different D-RL algorithm families.
• Auto-regressive policies provide the most general high-performance solution for modeling complex action dependencies, though with increased computational cost and inference latency.
• Importance weighting in policy gradient methods, as seen in VDN-PPO, can reduce advantage variance in inactive action heads, leading to improved learning dynamics and modest but robust performance gains.

Summary by alphaXiv.