Reproduce “Vision-Centric Activation and Coordination for Multimodal Large Language Models” on OpenResearch

About this paper

Yunnan Wang, Fan Lu, Kecheng Zheng, Ziyuan Huang, Ziqiang Li, Wenjun Zeng, Xin Jin

The VaCo framework enhances Multimodal Large Language Models (MLLMs) by intrinsically activating and coordinating vision-centric information. It utilizes query-based discriminative alignment to harness visual priors from multiple Vision Foundation Models (VFMs) and employs a Token Gateway Mask to resolve representational conflicts, leading to superior visual comprehension and efficiency. VaCo outperforms state-of-the-art MLLMs on benchmarks such as MMBench-English (78.6, +2.3 points over LLaVA-1.5) and MMMU (46.7, +4.9 points over LLaVA-1.5) with a 7B model, while using only a single visual encoder during inference.

Key insights

• Query-based discriminative alignment is a more effective strategy for integrating multiple VFMs compared to generative objectives or direct VFM token input methods.
• Explicit coordination mechanisms, such as the Token Gateway Mask, are crucial for mitigating representational conflicts that arise from combining diverse task-aware visual priors from multiple VFMs.
• Leveraging diverse VFMs, particularly those capturing 3D geometry, significantly enhances MLLM visual reasoning capabilities, addressing a key bottleneck in 2D image understanding.

Summary by alphaXiv.