Reproduce “MoPEQ: Mixture of Mixed Precision Quantized Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Krishna Teja Chitty-Venkata, Jie Ye, Murali Emani
MoPEQ introduces a novel post-training mixed-precision quantization algorithm for Vision-Language Mixture-of-Experts (VLM-MoE) models, leveraging a data-free Hessian trace approximation to assess expert sensitivity for precision assignment. This approach reduces model size by approximately 1.5 times while preserving accuracy within 5% of full-precision baselines and consistently outperforms activation frequency-based methods.
Key insights
- Expert sensitivity, approximated by the Hessian trace, serves as a more robust indicator of quantization importance compared to activation frequency, particularly for MoE models designed for balanced expert usage.
- Model-wise mixed-precision assignment generally outperforms layer-wise allocation in validation performance across various tasks and models.
- Experts in deeper layers often exhibit lower sensitivity to quantization, implying that initial model layers require higher precision to preserve accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.