Reproduce “Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, Saining Xie
The authors introduce VSI-Bench, a benchmark using indoor walkthrough videos to test whether models remember spatial relationships. It contains more than 5,000 question-answer pairs from 288 videos. They also prompt models to map object locations before answering. On Gemini-1.5 Pro’s relative-distance task, accuracy rises from 46.0 to 56.0 with its map and 66.0 with a ground-truth map; effects vary across models.
Key insights
- Spatial reasoning, specifically relational reasoning and egocentric-allocentric transformation, is identified as the primary bottleneck for multimodal large language models (MLLMs) in visual-spatial tasks.
- Linguistic prompting techniques like Chain-of-Thought surprisingly led to performance degradation in MLLMs on spatial reasoning tasks, indicating these methods do not inherently solve visual-spatial challenges.
- MLLMs tend to form localized, rather than globally consistent, cognitive maps from video observations, showing strong local spatial awareness but poor global understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.