Reproduce “Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs” on OpenResearch

About this paper

Tianwei Ni, Benjamin Eysenbach, Ruslan Salakhutdinov

The paper demonstrates that recurrent model-free reinforcement learning, when carefully implemented and tuned, can serve as a highly competitive and general baseline for a diverse range of Partially Observable Markov Decision Processes (POMDPs). Their optimized implementation achieved performance on par with or superior to specialized state-of-the-art algorithms across 18 out of 21 environments, often with greater sample and computational efficiency.

Key insights

• The perceived poor performance of recurrent model-free RL is not an inherent limitation but a consequence of suboptimal implementation details and hyperparameter choices.
• Key design decisions significantly impact performance, including using separate recurrent encoders for the actor and critic, conditioning RNNs on rich inputs (such as rewards), and leveraging off-policy algorithms.
• The optimal RNN context length is highly task-specific and a crucial hyperparameter for performance, often requiring careful tuning.

Summary by alphaXiv.