Reproduce “NavBench: Probing Multimodal Large Language Models for Embodied Navigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanyuan Qiao, Haodong Hong, Wenqi Lyu, Dong An, Siqi Zhang, Yutong Xie, Xinyu Wang, Qi Wu
NavBench, a new benchmark, evaluates Multimodal Large Language Models (MLLMs) for embodied navigation by assessing both navigation comprehension and step-by-step execution in zero-shot scenarios. The findings indicate MLLMs exhibit limitations in temporal reasoning and progress tracking, despite demonstrating real-world applicability via a robotic deployment pipeline.
Key insights
- A consistent correlation exists between MLLM navigation comprehension scores and their subsequent execution performance, indicating a link between understanding and acting.
- Current MLLMs, including advanced proprietary models, exhibit notable limitations in temporal reasoning and progress tracking, struggling with tasks involving sequential instruction order and identifying completed sub-instructions.
- Integrating structured topological map information into the MLLM's decision-making process significantly enhances navigation success rates, particularly for scenarios of medium difficulty.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.