Reproduce “MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoning” on OpenResearch

About this paper

Xuhui Zheng, Kang An, Ziliang Wang, Yuhang Wang, Faqiang Qian, Yichao Wu

A multimodal reinforcement pre-training framework, MMRPT, enhances visual reasoning in MLLMs by integrating masked vision-dependent reasoning and reinforcement learning during the pre-training phase. This method demonstrates consistent zero-shot performance gains across various benchmarks and significantly improves model robustness by preventing degradation on out-of-distribution tasks after specialized fine-tuning.

Key insights

• Cross-attention patterns in specific MLLM decoder layers (upper-middle) are highly indicative of vision-dependent textual segments, enabling effective automated data construction.
• Employing a high-capacity text-only LLM to refine mask targets alongside MLLM cross-attention ensures that masked tokens truly necessitate visual evidence, preventing reliance on linguistic priors.
• Directly incorporating reinforcement learning during pre-training, guided by structured outputs and a reward function combining format and answer correctness, fosters active visual inference and deeper semantic grounding.

Summary by alphaXiv.