Reproduce “EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning” on OpenResearch

About this paper

Kunal Menda, Katherine Driggs-Campbell, Mykel J. Kochenderfer

Researchers at Stanford University and UIUC developed EnsembleDAgger, an interactive imitation learning algorithm that utilizes Bayesian deep learning to quantify the novice policy's uncertainty ('doubt'). This algorithm employs a state-dependent decision rule for expert intervention, demonstrating improved learning efficiency and enhanced safety, including zero failures in simple tasks and a better Pareto frontier in complex robotics environments.

Key insights

• A novice policy's epistemic uncertainty, measured by the variance of a deep ensemble, serves as an effective model-free proxy for familiarity with the state space and inversely, the proximity to potential failure states.
• Guiding expert intervention based on the novice's doubt enables more efficient learning of robust policies with stronger safety guarantees, as the novice is allowed to act primarily in regions where it is confident and familiar.
• Deep ensembles provide a scalable and practical method for estimating predictive uncertainty in high-dimensional imitation learning settings, making Bayesian uncertainty quantification viable for complex robotic tasks.

Summary by alphaXiv.