Reproduce “SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards” on OpenResearch

About this paper

Sheng Xia, Zhengqin Lai, Tianxiang Jiang, Kanghui Tian, Shoujun Zhou, Bin Li, Yi Wang

The Semantic Evidence Reward (SER) framework enables video Multimodal Large Language Models (MLLMs) to ground their reasoning in explicit spatio-temporal evidence through semantic claim verification, eliminating the need for dense box annotations. It achieved state-of-the-art performance on the V-STAR benchmark with 49.6% mLGM and 35.7% mAM, while also improving answer accuracy and evidence grounding across general video understanding tasks.

Key insights

• Replacing geometry-only evidence rewards with a semantic claim verification by a fixed referee VLM provides a more stable and semantically aligned credit assignment signal for reinforcement learning.
• Effective evidence-grounded video reasoning can be achieved without dense spatio-temporal box annotations, making the training process more scalable and robust to annotation mismatches.
• Explicitly grounding high-level linguistic reasoning in low-level visual evidence through structured claims creates a mutually reinforcing cycle that enhances both answer accuracy and the faithfulness of spatio-temporal reasoning in MLLMs.

Summary by alphaXiv.