Reproduce “Fast Inference of Mixture-of-Experts Language Models with Offloading” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Artyom Eliseev, Denis Mazur
A method combining MoE-specific parameter offloading with mixed-precision quantization enables interactive inference of large Mixture-of-Experts language models like Mixtral-8x7B on consumer-grade GPUs. This approach, developed by researchers at Moscow Institute of Physics and Technology and Yandex, allows for token generation speeds of 2-4 tokens per second on hardware with 12-16GB VRAM, making these models accessible for local deployment.
Key insights
- MoE models exhibit significant expert reuse and locality during autoregressive inference, suggesting caching frequently accessed experts can reduce data transfer.
- The hidden states from early transformer layers can effectively predict which experts will be activated in subsequent MoE layers, enabling proactive speculative loading.
- Quantizing the vast majority of MoE parameters (experts) to lower bitwidths (e.g., 2-bit or 3-bit) while maintaining higher precision for other parts of the model offers an optimal quality-size trade-off.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.