Reproduce “Learning Off-Policy with Online Planning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harshit Sikchi, Wenxuan Zhou, David Held
The Learning Off-Policy with Online Planning (LOOP) framework, developed by Carnegie Mellon University, combines model-based online planning with model-free off-policy value learning to enhance sample efficiency and performance across online, offline, and safe reinforcement learning tasks. It addresses actor divergence through Actor Regularized Control (ARC) and achieves substantial improvements in robotic control benchmarks.
Key insights
- H-step lookahead policies offer a theoretical trade-off in optimality gap, reducing dependence on value function error while introducing model error dependence, proving advantageous when value function bias is significant.
- Separating computationally intensive online planning from efficient model-free value function learning allows for a practical and scalable hybrid reinforcement learning approach.
- Regularizing the behavior policy during online planning prevents actor divergence and stabilizes value function learning in hybrid actor-critic frameworks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.