Reproduce “Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, Jun Suzuki
A new initialization method called Drop-Upcycling enhances Mixture of Experts (MoE) model training by blending pre-trained knowledge with strategic expert diversification. This approach achieves superior performance and faster convergence while requiring approximately one-quarter of the training FLOPs compared to dense models of equivalent performance.
Key insights
- Successfully balancing the trade-off between transferring knowledge from a pre-trained dense model and encouraging expert specialization is crucial for efficient MoE training.
- Strategic partial re-initialization of expert weights, rather than full replication or purely random noise, fosters necessary diversity without discarding valuable prior knowledge.
- The re-initialization ratio ('r') significantly impacts performance, with 'r=0.5' providing an optimal blend for rapid initial progress and sustained long-term convergence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.