Reproduce “MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li and 4 more

The MUSEG framework, developed by Tsinghua University and Alibaba Group, enhances Large Multimodal Models (MLLMs) for video temporal understanding by integrating multi-segment grounding and a timestamp-aware, phased reinforcement learning training approach. It achieved superior performance on temporal grounding tasks, such as 59.7 mIoU on Charades-STA for MUSEG-7B, and significantly improved generalization on time-sensitive video question answering, nearly doubling performance on complex understanding tasks compared to base models.

Key insights

  • Incorporating multi-segment grounding in training is crucial for preventing models from relying on simple shortcuts and instead fostering more comprehensive temporal understanding.
  • Explicitly encouraging timestamp-aware reasoning through a dedicated reward function in early training phases effectively guides MLLMs to focus on fine-grained temporal details.
  • A phased reward strategy, balancing initial structured guidance with later flexibility, optimizes the learning process for robust and generalizable temporal reasoning skills in MLLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.