Reproduce “Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Zubair Irshad, Chih-Yao Ma, Zsolt Kira
This research introduces Robo-VLN, a continuous control benchmark for Vision-and-Language Navigation in 3D reconstructed environments, and proposes a Hierarchical Cross-Modal (HCM) agent. The HCM agent significantly outperforms flat baselines, achieving a 46% Success Rate and 40% SPL in unseen environments by decoupling high-level reasoning from low-level continuous control.
Key insights
- Hierarchical decomposition, explicitly decoupling high-level reasoning from low-level continuous control, is crucial for improving performance in complex, long-horizon VLN tasks.
- Formulating VLN as a continuous control problem in photo-realistic 3D environments better approximates real-world robotics, including handling obstacles and continuous movements.
- Employing separate cross-modal attention for RGB and Depth visual features, rather than early fusion, enhances the agent's ability to ground instructions with visual information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.