Reproduce “Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinshuai Song, Weixing Chen, Yang Liu, Weikai Chen, Guanbin Li, Liang Lin
Researchers from Sun Yat-sen University introduce Long-Horizon Vision-Language Navigation (LH-VLN) as a task to address multi-stage, real-world instructions, providing an automated data generation platform (NavGen) and a benchmark (LHPR-VLN). Their Multi-Granularity Dynamic Memory (MGDM) model demonstrates improved adaptive memory and sustained reasoning capabilities for these complex tasks, outperforming baselines and highlighting the crucial role of memory.
Key insights
- Existing VLN models perform very poorly on long-horizon tasks, often achieving zero success rates even on initial subtasks, underscoring the substantial difficulty of LH-VLN.
- Decomposition of complex instructions into subtasks improves individual subtask success, but a model's holistic understanding and coherent memory are essential for overall multi-stage task completion.
- Memory accumulated from previous stages in longer tasks paradoxically aids performance in subsequent subtasks, emphasizing the critical role of robust memory mechanisms for sustained reasoning.
- The Chain-of-Thought feedback and adaptive memory modules are crucial for solving LH-VLN tasks; their ablation results in a complete failure to achieve any success metrics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.