Reproduce “VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangshuo Qin, Zhiteng Li, Zheng Chen, Weihang Zhang, Linghe Kong, Yulun Zhang
The VEQ framework, developed by researchers from Shanghai Jiao Tong University and Tsinghua University, introduces a post-training quantization approach for Mixture-of-Experts Vision-Language Models that adaptively addresses both modality and expert heterogeneities. This framework significantly improves zero-shot accuracy in low-bit settings, achieving gains of up to 3.09% on Qwen3-VL-30B-A3B-Instruct for 3-bit weight quantization compared to previous state-of-the-art methods.
Key insights
- Text tokens exhibit significantly higher gradient magnitudes (22.4x on average) compared to vision tokens, indicating their disproportionately larger impact on model output and requiring differentiated quantization.
- MoE experts display substantial heterogeneity, including load imbalance, functional specialization (generalist vs. specialist experts), and routing biases, where only a few 'decisive' experts are critical for performance.
- Quantization methods must be 'dual-aware,' simultaneously accounting for modality-specific differences and expert-level dynamics, to achieve robust performance in low-bit MoE VLM compression.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.