Reproduce “Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Chaitanya Lakhchaura, Rishitej Reddy Vyalla, Erik Cambria and 2 more
Sonar-MoE reduces peak GPU memory in MoE fine-tuning by updating batched expert weights as gradients become ready, then releasing them. In bf16 Alpaca fine-tuning on OLMoE with SonicMoE, allocated memory fell from 55.55 GB to 43.57 GB, and median step time fell 18.7%; held-out loss was 1.2577 versus 1.2596. It clips dense, not expert, gradients; evidence covers two MoE families.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.