Reproduce “Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos” on OpenResearch

About this paper

Bingyi Xia, Han Bao, Zhewei Chen, Hanjing Ye, Jingwen Yu, Yuhan Pang, Wenjun Xu, Jiankun Wang

Researchers from Southern University of Science and Technology and collaborators developed a scalable framework for learning urban navigation policies from 500 hours of in-the-wild egocentric videos, automatically annotating them with metric trajectories and structured Chain-of-Thought rationales. The framework systematically exposes long-tail navigation challenges based on empirical rarity and model-dependent difficulty, achieving superior planning performance on a video test set and improved real-world transfer after fine-tuning.

Key insights

• Web-scale in-the-wild egocentric videos provide a scalable and diverse data source for training robust urban navigation policies, capturing a broad empirical distribution of real-world scenarios.
• Integrating structured Chain-of-Thought (CoT) reasoning into VLA models significantly improves navigation planning performance and enhances the interpretability of an agent's decisions.
• The long tail of urban navigation can be empirically discovered and characterized by jointly analyzing distributional rarity of perception-motion patterns and model-dependent planning errors, revealing a partial but critical overlap between rare and difficult situations.

Summary by alphaXiv.