Reproduce “EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Linglin Jing, Yuting Gao, Zhigang Wang, Wang Lan, Yiwen Tang, Wenhai Wang, Kaipeng Zhang, Qingpei Guo

EvoMoE introduces an MoE-tuning framework for Multimodal Large Language Models (MLLMs) that addresses challenges of expert uniformity and router rigidity. The model achieves average performance gains of 0.9% to 1.4% across various MLLM sizes compared to MoE-LLaVA baselines, while efficiently activating only the top-1 expert.

Key insights

  • Expert diversity can be effectively achieved through a gradient-based evolution process rather than simple replication, leading to specialized experts.
  • Routing mechanisms in multimodal Mixture of Experts (MoE) models benefit significantly from dynamic, modality-aware adaptation, allowing more precise token-to-expert allocation.
  • Combining expert evolution with a dynamic router creates a synergistic effect, yielding more efficient and capable Multimodal Large Language Models (MLLMs).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.