Reproduce “Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyu Zhang, Aishik Deb, Klaus Mueller
Researchers at Stony Brook University developed ExploRLer, a pluggable pipeline that enhances on-policy reinforcement learning by systematically exploring sparse regions of the parameter space at the iteration level. This approach, which leverages the Empty-Space Search Algorithm, consistently yields higher average returns and improved stability across various continuous control tasks compared to standard PPO and TRPO.
Key insights
- Visualizations show that current policy checkpoints often miss nearby, unexplored regions in the parameter space that contain policies yielding significantly higher returns.
- Iteration-level exploration, by leveraging structural information across multiple gradient steps, offers a computationally efficient and effective way to correct for surrogate-gradient bias.
- The on-policy design of ExploRLer, decoupling policy and value networks, allows for modular integration and maintains stability, avoiding issues common in off-policy contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.