Reproduce “ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ting Huang, Zhenyu Zhang, Wenyuan Huang, Jian Yang, Hao Tang
ConsiSpace is a framework enhancing video spatial reasoning in multimodal large language models (MLLMs) by integrating geometric consistency into memory management and through a self-supervised learning objective. It achieves up to a 19.2-point average score increase on the VSI-Bench benchmark while significantly reducing GPU memory usage and inference time.
Key insights
- Integrating geometric consistency as both an evidence organization principle and an explicit learning signal is effective for improving efficiency and robustness in MLLM-based video spatial reasoning.
- Selectively writing and fusing video frames based on geometric changes (e.g., camera movement) can significantly reduce memory overhead and improve scalability without sacrificing crucial information.
- Self-supervised consistency learning across varying viewpoints provides a data-efficient method to enhance the stability and reliability of spatial predictions after initial supervised fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.