Reproduce “MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG” on OpenResearch

About this paper

Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Yang, Bing Li, Chunfeng Yuan, Kang Rong, Fengyun Rao and 2 more

MMAgent-R^2 introduces an agentic multimodal retrieval-augmented generation (mRAG) framework that enables Vision-Language Models to perform dynamic visual verification and active search for Knowledge-based Visual Question Answering (KB-VQA). The system achieves new state-of-the-art results on E-VQA, InfoSeek, and MMhops benchmarks, significantly improving visual entity identification accuracy by up to 14.1% on E-VQA.

Key insights

• Integrating active visual reranking and rejection mechanisms directly within an agentic VLM significantly enhances the precision of visual entity identification and robustness against initial retrieval failures, moving beyond passive post-processing.
• Formulating mRAG as a reinforcement learning problem allows for joint optimization of retrieval, verification, and answer generation, enabling more autonomous decision-making and handling of complex, multi-hop reasoning tasks.
• Dense step-level verification rewards (for both successful reranking and correct rejections) provide a crucial learning signal that helps the agent learn when to confidently select a candidate and when to dynamically expand its search.

Summary by alphaXiv.