Reproduce “ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqiang Xu, Qi Dai, Tian Xie, Yifan Yang, Kai Qiu, DongDong Chen, Zuxuan Wu, Chong Luo
ViaRL, a framework for video understanding, learns to identify and process only the most relevant temporal segments of a video based on a query. It achieves this through a two-stage iterative training process that combines reinforcement learning for frame selection with instruction tuning for a downstream MLLM, yielding superior performance on temporal grounding tasks and significant gains on benchmarks like MLVU and VideoMME.
Key insights
- Mimicking human-like attention by selectively focusing on relevant video segments significantly improves efficiency and performance for video understanding.
- Reinforcement learning, driven by downstream task performance (e.g., MLLM answer accuracy), can effectively train a temporal grounding selector without requiring explicit frame-level annotations.
- An iterative, two-stage training cycle between a frame selector and a language model can mutually amplify their capabilities, leading to stable and progressive improvements in video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.