Reproduce “SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bo Gu, Zhikang Zhang, Zizhuang Wei, Zhenyuan Chen, Lingyun Li, Zhuoyi Song
SpaceMind++ enhances Multimodal Large Language Models (MLLMs) by constructing a voxelized allocentric cognitive map from RGB videos, enabling precise and consistent spatial reasoning within 3D environments. This architecture achieves a state-of-the-art average score of 73.2 on the VSI-Bench benchmark and shows strong generalization across various spatial reasoning tasks.
Key insights
- Mimicking the brain's dual visual stream and allocentric cognitive maps enables MLLMs to build a persistent, world-centered spatial understanding, moving beyond auxiliary 3D features.
- Explicitly constructing a dynamic voxelized cognitive map allows for reorganizing fragmented egocentric observations into a stable 3D metric space, preserving object permanence and spatial topology.
- Coordinate-guided attention mechanisms, utilizing 3D coordinate embeddings and 3D RoPE, are crucial for grounding semantic interactions within the 3D metric structure, ensuring precise spatial reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.