Reproduce “Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengze Xu, Maolin Deng, Tieyong Zeng
The authors analyze how a near-zero mixture-of-experts routing probability can recover. They decompose its motion into covariance among complementary routes and the suppressed expert’s relative gradient advantage, deriving conditional recovery-time orders under pathwise bounds. In a six-layer Transformer-MoE trained on AG News, projecting a selected class-conditional gradient contrast delayed confirmed recovery in both dispatch settings, but does not establish generality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.