Reproduce “Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuqing Luo, Ye Han, Pingzhi Li, Jiayin Qin, Jie Peng, Yang, Zhao, Tianlong Chen
Mozart introduces an algorithm-hardware co-design framework for efficiently training Mixture-of-Experts (MoE) Large Language Models on 3.5D wafer-scale chiplet architectures, achieving latency reductions up to 2.37 times over baseline methods.
Key insights
- Aligning the logical modularity of MoE models with physical hardware modularity on chiplets is crucial for enhancing training efficiency.
- Profiling expert co-activation patterns allows for strategic placement of experts, significantly reducing communication overhead during training.
- Fine-grained scheduling with streaming tokens and experts effectively hides DRAM access latency by overlapping communication and computation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.