Reproduce “VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu and 10 more
VLM-3R addresses spatial reasoning from monocular video by estimating scene geometry and camera motion, converting them into tokens, and fusing them with vision and language. It needs no external depth sensor or prebuilt map at inference. On VSI-Bench, its 7B model scores 60.9 versus 57.7 for fine-tuned LLaVA-NeXT-Video-7B, an eight-task average, not validated temporal performance.
Key insights
- Directly deriving metric-scale 3D geometry and camera pose from monocular video, without external sensors or pre-built maps, can significantly enhance a VLM's spatial reasoning capabilities.
- Fusing implicit latent representations (spatial and view tokens) from a 3D reconstruction model through a cross-attention mechanism proves more effective and stable than directly fusing explicit point clouds for VLM tasks.
- Instruction tuning with a large volume of automatically generated 3D-aware spatial and spatio-temporal QA pairs is crucial for adapting VLMs to complex geometric and dynamic reasoning tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.