Reproduce “ReIL: A Framework for Reinforced Intervention-based Imitation Learning” on OpenResearch

About this paper

Rom Parnichkun, Matthew N. Dailey, Atsushi Yamashita

The ReIL framework from researchers at the Asian Institute of Technology and The University of Tokyo presents a method for training robot agents by integrating imitation learning and reinforcement learning, efficiently utilizing human interventions. It addresses the value overestimation bias in intervention-based methods and uses a multi-task model, demonstrating stable performance in real-world mobile robot navigation without performance degradation over time.

Key insights

• Value overestimation bias in intervention-based reinforcement learning can be effectively mitigated by designing a reward function that assigns a constant negative reward upon intervention, specifically lower than any possible task reward, thus intrinsically penalizing such events.
• Combining off-policy reinforcement learning (TD3) with behavior cloning, while carefully weighting expert demonstrations and agent-generated data, significantly enhances sample efficiency and learning stability in intervention-based scenarios.
• A general, memory-aware model architecture, like an adapted SNAIL, can process diverse temporal inputs (demonstrations, past experiences, current observations) to reduce task-specific engineering and enable multi-task robot policies.

Summary by alphaXiv.