Reproduce “Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zongfei Li
This research reveals that sparse Mixture-of-Experts (MoE) models exhibit a "selection-commitment mismatch," where the router's scores are sub-optimal for weighting already-selected experts. The study introduces Fixed-Dispatch Adaptive Aggregation (FDAA), a lightweight, trainable mechanism that learns token-adaptive aggregation weights, leading to significant cross-entropy reductions of up to -0.19367 on WikiText-103 for OLMoE-1B-7B and -0.01747 for DeepSeek-V2-Lite.
Key insights
- A "selection-commitment mismatch" is consistently observed across MoE architectures, where router scores poorly correlate with optimal expert commitment within the dispatched set (Spearman correlation of 0.030 for OLMoE).
- Downstream first-order gradients strongly predict optimal expert commitment (Spearman correlation of 0.595), indicating the necessary signal for aggregation improvement is present in the backward pass.
- Forward-only predictors fail to reliably distill this gradient-derived credit into effective aggregation policies, highlighting a representational bottleneck that necessitates direct optimization with the language modeling objective.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.