Reproduce “ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peng Jin, Zihan Qiu, Zekun Wang, Bo Zheng, Yang Xu, Tian Xie, Xiao Li, Huaqing Zhang and 3 more

ID Balancing adjusts routing biases using token-count errors and whether imbalance is worsening, without adding a balancing gradient to the language-modeling objective. In an 18.9B model with Top-3 routing over 768 experts, it lowers worst backbone MaxVio to 15.33 and training-average MinVio to 0.548; this is not a win on every module or setting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.