Reproduce “View Invariant Learning for Vision-Language Navigation in Continuous Environments” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Josh Qixuan Sun, Huaiyuan Weng, Xiaoying Xing, Chul Min Yeum, Mark Crowley

This paper introduces View Invariant Learning (VIL), a post-training framework that efficiently adapts existing Vision-Language Navigation in Continuous Environments (VLNCE) policies to diverse camera viewpoints, encompassing variations in height and angle. VIL improves navigation performance under varied viewpoints by 8-15% on R2R-CE and demonstrates zero-shot transfer capabilities on a physical robot, enhancing success rates from 20-28% to 44-48%.

Key insights

  • A systematic evaluation setting, V²-VLNCE, is essential for rigorously benchmarking and improving the robustness of navigation policies to diverse camera viewpoints.
  • Combining contrastive learning for robust visual features and knowledge distillation for waypoint prediction enables effective post-training adaptation of existing policies to viewpoint variations.
  • Efficient adaptation can be achieved by only training a small, lightweight adapter module within the student's waypoint predictor, preserving computational efficiency and leveraging pre-trained knowledge.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.