Reproduce “Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qi Li, Yanzhe Zhao, Yongxin Zhou, Yameng Wang, Yandong Yang, Yuanjia Zhou, Jinxiang Liu
Magic-MM-Embedding introduces a framework for universal multimodal embedding that achieves state-of-the-art performance by combining aggressive visual token compression with a three-stage progressive training pipeline. This approach significantly reduces inference latency and training time across various multimodal retrieval benchmarks.
Key insights
- Aggressive visual token compression, up to a 75% reduction, can enhance both performance and inference efficiency when effectively integrated with a multi-stage training strategy, challenging the traditional efficiency-performance trade-off.
- A progressive, coarse-to-fine training pipeline is critical for restoring foundational MLLM capabilities and then systematically building discriminative and task-aware multimodal representations in compressed models.
- Using an MLLM as a 'judge' for curating high-quality hard negatives significantly improves training effectiveness by providing more challenging and relevant negative samples than conventional methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.