Reproduce “STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning” on OpenResearch

About this paper

Xiaowen Zhang, Zhi Gao, Licheng Jiao, Lingling Li, Qing Li

The STVG-R1 framework by Zhang et al. introduces an object-centric visual prompting paradigm to reformulate spatial-temporal video grounding as an instance ID identification task, followed by reinforcement learning to fine-tune Vision-Language Models. This approach achieves new state-of-the-art results on several STVG benchmarks and demonstrates strong zero-shot generalization across various video understanding tasks, improving metrics like m vIoU by up to 6.8% over prior methods.

Key insights

• Reformulating dense coordinate prediction into instance-level ID identification, supported by explicit visual prompts, significantly mitigates visual-textual misalignment and VLM hallucination in video grounding tasks.
• Embedding simple, temporally consistent object identifiers as visual prompts provides effective and interpretable instance-level cues to VLMs, leading to performance improvements even in zero-shot settings without architectural modifications.
• Applying reinforcement learning with a task-driven reward function, encompassing temporal accuracy, spatial consistency, and response format, enables VLMs to optimize directly for coherent spatial-temporal reasoning in videos.

Summary by alphaXiv.