Reproduce “Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wentao Wang, Heqing Zou, Tianze Luo, Rui Huang, Yutian Zhao, Zhuochen Wang, Hansheng Zhang, Chengwei Qin and 3 more
Video-STR improves Multimodal Large Language Models' video spatio-temporal reasoning by integrating a graph-based reasoning mechanism within a reinforcement learning framework, outperforming prior methods and commercial models like GPT-4o on spatio-temporal benchmarks. It also contributes STV-205k, a large-scale dataset for training and evaluation.
Key insights
- Inter-object relation graphs provide a robust, rotation-invariant representation of multi-object scenes, effectively overcoming the limitations of viewpoint-dependent 2D map approaches by focusing on physical relationships.
- Reinforcement Learning with Verifiable Reward (RLVR) fosters superior generalization and genuine understanding in MLLMs for complex reasoning tasks, unlike Supervised Fine-Tuning (SFT) which often leads to memorization.
- Explicitly guiding MLLMs to infer underlying spatio-temporal topologies through a graph-based reasoning mechanism significantly enhances their ability to comprehend physical information and dynamic interactions in videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.