Reproduce “Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arun Balajee Vasudevan, Dengxin Dai, Luc Van Gool
ETH Zurich researchers introduced Talk2Nav, a large-scale dataset for long-range outdoor vision-and-language navigation in New York City, featuring fine-grained landmark and directional instructions. They also developed a cognitively inspired navigation agent using a dual attention mechanism and explicit spatial memory, which achieved an 11.92% SPL on their dataset and outperformed baselines on the TouchDown dataset.
Key insights
- Decomposing long navigation instructions into fine-grained landmark descriptions and local directional cues significantly improves annotation feasibility and learning effectiveness for long-range navigation.
- Modeling human cognitive processes, such as explicit spatial memory and differentiating between types of navigational language, leads to more robust and accurate autonomous navigation agents.
- An explicit 2D visual representation of the agent's traversed path (spatial memory) is more effective for spatial reasoning in long-range navigation than sequence-based memory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.