Reproduce “Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, Joni Pajarinen
The REDQ+AdaptiveBC method dynamically adjusts behavior cloning regularization during online fine-tuning for reinforcement learning, building on pre-trained offline policies. This approach effectively prevents performance collapse and enables sample-efficient learning across various D4RL locomotion and manipulation tasks by adaptively balancing offline knowledge exploitation with online exploration.
Key insights
- Dynamically modulating the behavior cloning weight prevents performance collapse during the critical offline-to-online transition and facilitates faster adaptation.
- Fixed behavior cloning weights are highly task-dependent and sub-optimal, whereas an adaptive approach reduces hyperparameter tuning effort and improves generalization.
- Combining adaptive behavior cloning with randomized ensembles offers a robust framework for stable and sample-efficient learning in hybrid offline-to-online RL settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.