Reproduce “Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikolaos Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi, Leon Voukoutis, Vassilis Katsouros, Georgios Paraskevopoulos
Researchers at the Institute for Language and Speech Processing, Athena Research Center, developed Contrastive Routing for Modular Mixture-of-Experts (CoRM), which routes tokens by contrasting their distinctiveness from a dynamic, learned reference state. This mechanism improves zero-shot accuracy by up to +1.77 percentage points on reasoning benchmarks and yields lower language modeling perplexity, while maintaining minimal computational overhead.
Key insights
- Subtracting a dynamic, learned baseline of generic token structure from representations via EMA effectively concentrates the routing signal on token-specific, discriminative information.
- L2 normalization combined with a low-dimensional projection bottleneck provides inherent routing stability and encourages distinct expert specializations by partitioning the data manifold.
- Reconceptualizing expert selection as a contrastive competition leads to significantly enhanced functional modularity, reflected in structurally decorrelated expert projections and stronger alignment of routing boundaries with linguistic structures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.