Reproduce “On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingze Wang, Weinan E

This research provides a theoretical framework demonstrating that Mixture-of-Experts (MoE) networks can efficiently approximate complex structured tasks. Shallow MoEs are shown to overcome the curse of dimensionality when approximating functions on low-dimensional manifolds, while deep MoEs can model piecewise functions with an exponential number of distinct pieces by leveraging compositional sparsity.

Key insights

  • Shallow MoE networks can overcome the curse of dimensionality by decomposing functions on low-dimensional manifolds into localized, low-dimensional subproblems, with approximation rates dependent on the intrinsic dimension.
  • Deep MoE networks can model an exponentially large number (E^L) of distinct task pieces by leveraging compositional sparsity, where depth enables hierarchical composition and experts provide subtask specialization.
  • The gating mechanism in MoE networks plays a crucial role in performing accurate input-to-expert assignments, effectively acting as a learnable routing and partitioning mechanism.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.