Reproduce “SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning” on OpenResearch

About this paper

Mingyuan Wu, Jingcheng Yang, Shengyi Qian, Xudong Wang, Jize Jiang, Qifan Wang, Aashu Singh, Khoi Pham and 6 more

SVR-R1 is a multi-turn reinforcement learning framework that enhances Vision Language Models' (VLMs) multi-modal reasoning by utilizing the VLM's intrinsic self-verification mechanism as a learning signal during training. This approach yielded accuracy improvements of up to 19.1% on ChartQA and 16.5% on TableVQA compared to baseline Qwen2.5-VL models, fostering the internalization of self-correction to reduce explicit verification needs during inference.

Key insights

• VLMs can leverage their intrinsic self-verification capabilities for self-improvement during RL training, reducing reliance on external supervision and bootstrapping multi-modal reasoning.
• The iterative self-verification process can be internalized by the model, leading to increased confidence in initial answers and a reduction in the number of explicit self-correction turns needed during inference.
• SVR-R1's efficacy is more pronounced in refining answers for medium-difficulty questions rather than solving inherently intractable problems, suggesting utility in improving performance within a feasible difficulty range.

Summary by alphaXiv.