Reproduce “Cambrian-P: Pose-Grounded Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jihan Yang, Zifan Zhao, Xichen Pan, Shusheng Yang, Junyi Zhang, Bingyi Kang, Hu Xu, Shang-Wen Li and 1 more
Cambrian-P integrates camera pose as a core supervisory signal within multimodal large language models to enhance video understanding, achieving state-of-the-art performance on spatial reasoning benchmarks like VSI-Bench (73.7) and VSTI-Bench, and demonstrating competitive streaming camera pose estimation with an ATE of 0.078 on ScanNet. The approach enables MLLMs to develop a more geometrically grounded comprehension of video content, also improving performance on general video QA tasks.
Key insights
- Camera pose is a compact, fundamental 3D signal that enforces global consistency and disentangles camera and object motion, significantly improving MLLMs' spatial understanding.
- An interleaved training strategy, combining VQA and pose estimation with dynamic frame sampling and random jitter, effectively harmonizes the training dynamics of both tasks.
- Pseudo-annotated camera poses from in-the-wild videos provide a scalable and effective supervisory signal, enhancing general video QA performance beyond datasets with ground-truth pose.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.