Reproduce “Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chanyoung Gwak, Yoonwoo Jeong, Byungwoo Jeon, Hyunseok Lee, Jinwoo Shin, Minsu Cho
Cog3DMap builds an explicit 3D memory from successive views and supplies it to a language model for spatial reasoning. Point3R predicts positions and geometric features, while semantic features are combined into tokens that retain, refresh, or add observations as views arrive. On VSTI-Bench, Cog3DMap-8B scores 67.5 versus 58.8 for VLM-3R-7B; this comparison does not isolate the map’s effect.
Key insights
- An explicit, compact, and non-redundant 3D cognitive map, inspired by human cognition, enables MLLMs to perform more direct and interpretable spatial reasoning.
- Geometric features from 3D reconstruction models can sufficiently encode spatial information, making additional explicit 3D positional embeddings redundant when integrated effectively.
- Spatially aggregating and updating visual tokens in a 3D map significantly reduces visual token redundancy, improving computational efficiency, particularly for long sequences, without performance degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.