Reproduce “VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xunyi Zhao, Gengze Zhou, Qi Wu

The Australian Institute for Machine Learning (AIML) at the University of Adelaide introduced VLN-MME, a modular and simulator-free framework designed to diagnose Multimodal Large Language Models (MLLMs) as language-guided visual navigation agents. This framework offers fine-grained error analysis, revealing MLLMs' 'poor context awareness' and 'perception-action gap' rather than simple success metrics in embodied tasks.

Key insights

  • MLLMs exhibit 'poor context awareness' and a 'local reasoning bias' in embodied navigation, frequently neglecting historical context and struggling with downstream action consequences.
  • Counterintuitively, advanced prompting strategies like Chain-of-Thought and self-reflection often led to performance degradation or minimal improvements for MLLMs in dynamic embodied tasks.
  • A 'perception-action gap' exists in current MLLMs, where visual recognition of objects frequently fails to translate into correct and timely navigation actions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.