Reproduce “Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Cheng, Zhiyu Yang, Bo Ma, Hanzhang Jia
The authors compare four ways to deactivate experts in sparse mixture-of-experts models. Across five instruction-tuned checkpoints, the worst implementation follows the training-time weighting rule: renormalization for raw-probability models and unnormalized re-routing for normalized-weight models. For DeepSeek-V2-Lite, freezing and renormalization on the same disabled set yield perplexities of 17.21 and 258,621.64, respectively; weight handling is not established as the sole cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.