Reproduce “Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jongseok Park, Sunga Kim, Zhenyu Gu, Ion Stoica, Alvin Cheung
The system applies inference-time masking of low-magnitude neuron activations within routed experts in pretrained MoE models, without changing weights or architecture. It integrates sparse execution into vLLM and leaves shared experts intact. Across eight models, cutoffs retain at least 95% of baseline average benchmark scores. On Qwen3.5-35B-A3B, it achieves 1.55× MoE-layer speedup at 84% sparsity, not end-to-end.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.