Reproduce “DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuxuan Lou, Kai Yang, Geng Zhang, Yong Liu, Yang You

DivMoE converts a dense model into domain-specialized experts by splitting domain-adapted models and routing each token to experts from distinct groups. On Qwen3-1.7B-Base, after 500B Stage-2 tokens, it averages 55.6% across 15 benchmarks, versus 51.6% for Branch-Train-Mix; without the routing constraint, it scores 52.5%. This setup adds 40B specialist-training tokens and tests models only up to 4B parameters.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.