Reproduce “Scaling Vision with Sparse Mixture of Experts” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, Neil Houlsby

Google Brain researchers introduce Vision-MoE (V-MoE), an architecture that integrates Mixture of Experts (MoE) layers into Vision Transformers, enabling efficient scaling to models with up to 15 billion parameters. The V-MoE models achieve higher accuracy compared to dense ViTs at similar computational costs, and the Batch Prioritized Routing (BPR) technique further reduces compute by focusing on the most informative image regions.

Key insights

  • MoE allows for scaling vision models up to 15 billion parameters, maintaining computational efficiency by activating only a subset of parameters per input.
  • Early MoE layers in V-MoE specialize in processing different image regions, while deeper layers specialize in discriminating between specific object classes.
  • Batch Prioritized Routing (BPR) enables dynamic and intelligent allocation of computational resources, focusing on the most informative image patches and discarding less useful ones.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.