Reproduce “EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song and 4 more

EnvPool is a high-performance C++ execution engine that accelerates parallel reinforcement learning environment simulations through an asynchronous, threadpool-based design. It enables up to 1 million frames per second for Atari and 3 million physics steps per second for MuJoCo environments on high-end hardware, significantly reducing RL agent training wall-time while maintaining sample efficiency.

Key insights

  • A C++ implementation combined with an asynchronous threadpool architecture dramatically boosts environment execution throughput compared to Python-level parallelization.
  • Asynchronous execution effectively addresses the 'long-tail problem' in parallel environments, maximizing throughput even with varying individual environment step times.
  • Optimized data transfer through lock-free buffers and direct memory sharing between C++ and Python significantly reduces overhead, enhancing overall system efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.