Reproduce “CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenpeng Su, Xing Wu, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu and 1 more
CartesianMoE introduces a novel Mixture-of-Experts architecture that employs a Cartesian Product Layer for enhanced, "multiplication"-style knowledge sharing among fine-grained sub-experts. This approach achieves lower perplexity and improved performance across various downstream tasks in large language models, while also demonstrating significantly greater robustness to expert routing failures.
Key insights
- A "multiplication"-style knowledge sharing mechanism, inspired by collective matrix factorization, is more effective for expert interaction in MoE models than predominantly "addition"-based methods.
- Sequentially processing tokens through two distinct, independently routed sub-layers enables deeper representations and more flexible routing, contributing to superior model performance.
- A hierarchical knowledge sharing system that integrates global shared knowledge, group-wise shared knowledge via Cartesian products, and expert-specific knowledge provides enhanced model capabilities and robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.