Reproduce “Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng and 6 more
The authors introduce Meta Experts, a sparse mixture-of-experts design in which layers share an expert pool rather than keeping separate layer-specific banks. The models keep top-8 routing, activating eight experts per token. In scaling tests, Meta Experts reach comparable language-modeling loss with 70–80% as many experts as Vanilla MoE; this shows storage-capacity savings, not faster training or inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.