Reproduce “Residual Reinforcement Learning from Demonstrations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce, Cordelia Schmid
The paper introduces Residual Reinforcement Learning from Demonstrations (RRLfD), a method that first trains a base policy through behavioral cloning from demonstrations and then refines it with a reinforcement learning-trained residual policy. This approach enhances sample efficiency and robustness for vision-based robot manipulation tasks with sparse rewards, effectively eliminating the need for hand-engineered base controllers or dense reward functions.
Key insights
- Learning a data-driven base policy from demonstrations effectively eliminates the need for hand-engineered base controllers, broadening the applicability of residual RL.
- The additive residual formulation preserves the fundamental skills acquired from demonstrations, mitigating catastrophic forgetting during subsequent reinforcement learning.
- Reusing visual features learned during behavioral cloning allows the residual policy to effectively operate on high-dimensional visual inputs and fuse them with proprioceptive state.
- Residual policies not only improve task reliability but also optimize execution efficiency, demonstrating a 21% reduction in task completion time in experiments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.