Reproduce “Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval” on OpenResearch

About this paper

Chunxu Liu, Jiyuan Yang, Ruopeng Gao, Yuhan Zhu, Feng Zhu, Rui Zhao, Limin Wang

Researchers from Nanjing University and SenseTime Research developed the Reasoning Guided Embeddings (RGE) framework, which leverages Multimodal Large Language Model generative reasoning to produce enhanced multimodal embeddings for retrieval tasks. The RGE 3B-parameter model achieved an average score of 70.1% on the MMEB benchmark, an absolute 4.9% improvement over non-reasoning baselines by integrating a rationale generation step with a specialized contrastive learning objective.

Key insights

• Explicitly leveraging MLLM's generative reasoning process significantly enhances the quality and discriminative power of multimodal embeddings for downstream tasks.
• It is critical to use self-generated rationales for contrastive learning to avoid information leakage, which occurs when oracle rationales are directly used and leads to performance degradation.
• Training with reasoning-guided objectives generally imbues the MLLM with superior embedding capabilities, even if explicit reasoning is not performed during inference.

Summary by alphaXiv.