Reproduce “Video Spatial Reasoning with Object-Centric 3D Rollout” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoran Tang, Meng Cao, Ruyang Liu, Xiaoxi Liang, Linglong Li, Ge Li, Xiaodan Liang
Researchers developed Object-Centric 3D Rollout (OCR), a novel reinforcement learning strategy to enhance Multi-modal Large Language Models' (MLLMs) video spatial reasoning by addressing "query-locked reasoning". A 3B-parameter model trained with OCR achieved 47.5% average accuracy on the VSI-Bench benchmark, outperforming larger MLLMs and proprietary models.
Key insights
- Explicitly degrading object-specific visual cues through 3D spatial perturbations during reinforcement learning effectively overcomes query-locked reasoning in MLLMs by compelling them to rely on surrounding context.
- A curriculum learning approach for object perturbation, starting with higher noise and gradually reducing it, fosters more robust and generalizable spatial reasoning paths.
- Leveraging high-quality, CoT-annotated supervised fine-tuning data, specifically curated for challenging spatial problems, is critical for cold-starting and improving MLLMs before reinforcement learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.