Reproduce “LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videos” on OpenResearch

About this paper

Noriaki Hirose, Catherine Glossop, Ajay Sridhar, Dhruv Shah, Oier Mees, Sergey Levine

LeLaN is a method that trains language-conditioned robot navigation policies by automatically generating synthetic language instructions and robot actions from readily available "in-the-wild" egocentric video data, leveraging foundation models for annotation. It enables training with diverse, unlabeled video content to improve navigation performance and generalization across robot platforms.

Key insights

• Abundant "in-the-wild" egocentric video data, despite lacking explicit labels, can be transformed into valuable training material for robot navigation policies.
• Foundation models, including Vision-Language Models, Large Language Models, and robot foundation models, can automate the complex and costly process of generating synthetic language instructions and corresponding robot actions.
• The diversity and scale of data generated through this automated pipeline can significantly improve the generalization capabilities of language-conditioned navigation policies across different environments and robot platforms.

Summary by alphaXiv.