Reproduce “VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao and 2 more

VideoChat-R1, developed by researchers from Shanghai AI Laboratory, Nanjing University, and collaborators, implements Reinforcement Fine-Tuning (RFT) with Group Relative Policy Optimization (GRPO) to enhance the spatio-temporal perception of multimodal large language models (MLLMs) for video understanding. The framework achieves state-of-the-art performance, including a +31.8 mIoU improvement on Charades-STA for temporal grounding, while maintaining the model's general chat capabilities and introducing a temporal clue-driven reasoning schema for long videos.

Key insights

  • RFT with GRPO is data-efficient, significantly improving spatio-temporal perception without degrading the MLLM's original general capabilities or causing overfitting.
  • Joint training on multiple spatio-temporal perception tasks using GRPO yields synergistic improvements across all related benchmarks.
  • Rule-based reward functions, including those where an LLM acts as a judge for recall, effectively guide MLLM learning for complex video understanding tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.