Reproduce “Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiahao Wang, Weiye Xu, Aijun Yang, Wengang Zhou, Lewei Lu, Houqiang Li, Xiaohua Wang, Jinguo Zhu
This research introduces Self-Consistency Sampling (SCS) to improve the faithfulness of multimodal large language models (MLLMs) trained with outcome-reward reinforcement learning. SCS addresses unfaithful reasoning, where models arrive at correct answers via flawed logic, by integrating a differentiable consistency reward, leading to an average absolute performance gain of 7.7 percentage points for Qwen2.5-VL-7B-Instruct.
Key insights
- Outcome-reward alone is insufficient for distinguishing genuinely correct reasoning from unfaithful lucky guesses in multiple-choice MLLM tasks.
- Perturbing reasoning trajectories through truncation and visual noise can effectively assess and, subsequently, improve the robustness and faithfulness of MLLM reasoning during RL training.
- Integrating a consistency reward into existing outcome-reward RL frameworks provides a computationally efficient method to enhance reasoning fidelity without requiring complex process reward models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.