Reproduce “Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin and 8 more
The EAGLE framework systematically explores design principles for multi-modal Large Language Models (MLLMs) that leverage a mixture of vision encoders. It introduces optimal strategies for integrating diverse visual experts and a novel pre-alignment training stage, leading to MLLMs that achieve state-of-the-art performance across various multimodal benchmarks, particularly excelling in high-resolution and OCR tasks.
Key insights
- Unfreezing and updating vision encoders during training consistently yields significant performance improvements for MLLMs.
- Simple channel concatenation emerged as the most effective and efficient fusion strategy for integrating features from multiple vision experts.
- A novel "Pre-Alignment" training stage proved crucial for bridging representational gaps between vision-focused encoders and the language model, boosting MLLM coherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.