Reproduce “DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu
Researchers from Tsinghua University developed DECO, a sparse Mixture-of-Experts (MoE) architecture that achieves performance comparable to dense Transformer models under identical total parameter and training token budgets. DECO demonstrates significant inference speedups of up to 2.93x on edge devices compared to dense inference, making advanced LLMs more feasible for resource-constrained environments.
Key insights
- Non-gated MLP experts, when combined with ReLU-based routing, exhibit more stable activation trends than gated variants, simplifying architecture and reducing the need for aggressive regularization.
- The proposed NormSiLU, through inter-expert mean normalization and intra-expert RMS normalization, effectively stabilizes internal expert activations and output magnitudes, crucial for efficient parameter utilization.
- Learnable expert-wise scaling factors for router scores adaptively manage heterogeneity in expert output norms, contributing to improved overall model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.