Reproduce “One Student Knows All Experts Know: From Sparse to Dense” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fuzhao Xue, Xiaoxin He, Xiaozhe Ren, Yuxuan Lou, Yang You
Researchers at NUS and Huawei Noah’s Ark Lab developed a framework for integrating knowledge from sparse Mixture-of-Experts (MoE) models into a single, dense 'OneS' student model. This approach achieves significant performance retention from the MoE teacher while substantially improving inference speed and deployability by creating a hardware-friendly architecture.
Key insights
- It is possible to effectively transfer the high-capacity learning from a complex sparse MoE model to a compact, hardware-friendly dense student model, retaining significant performance.
- The SVD-KG method is particularly effective at extracting and merging essential knowledge components from multiple sparse experts into a single dense representation.
- Combining intelligent initialization through knowledge gathering with subsequent soft knowledge distillation is crucial for mitigating noise and refining the student's performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.