Reproduce “SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhao You, Shulin Feng, Dan Su, Dong Yu
SpeechMoE from Tencent AI Lab adapts the Mixture of Experts (MoE) architecture for acoustic models in Automatic Speech Recognition (ASR), achieving up to 23.0% relative Character Error Rate (CER) reduction on Mandarin benchmarks while maintaining comparable inference computational costs. The approach incorporates a novel router mechanism and specialized loss functions designed for speech features.
Key insights
- For low-level speech features, providing high-level distinctive information via a separate shared embedding network is crucial for effective router decision-making, unlike language models where previous layer outputs may suffice.
- Explicit L1 regularization on router probability distributions is more effective than previous balancing losses for promoting true sparsity and specialized expert selection.
- A differentiable mean importance loss can robustly balance expert utilization without hindering the sparse routing objectives, leading to improved expert diversity and overall model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.