Reproduce “OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingze Shi, Zhangyang Peng, Yizhang Zhu, Yifan Wu, Guang Liu, Yuyu Luo
OmniMoE combines token-selected fine-grained experts with an always-active dense MLP, using grouped scheduling to make routed work more GPU-friendly. Expert-centric scheduling groups assignments for matrix operations. In separate experiments, a 6.4B-total model scores 50.9 average accuracy across seven benchmarks, while a fixed-200M benchmark reports 6.7 ms versus PEER’s 73 ms at 4,096 input tokens. These findings apply to their tested configurations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.