Reproduce “Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ya Wang, Zhijian Zhuo, Yutao Zeng, Xun Zhou, Jian Yang, Xiaoqing Li
Scale-Distribution Decoupling (SDD), developed by researchers from ByteDance and Peking University, introduces a method to disentangle the scale and distribution of weight matrices in Large Language Models. This technique enhances training stability and accelerates convergence, achieving up to 3.46% higher accuracy on tasks like ARC-Challenge compared to existing normalization baselines in dense and Mixture-of-Experts architectures.
Key insights
- Explicitly separating the scale and distribution of weight matrices fundamentally stabilizes gradient dynamics and prevents common issues like gradient explosion or vanishing.
- SDD mitigates 'feature collapse' by encouraging more diverse and distinct representations across network layers, thereby enabling more effective scaling for deeper models.
- The approach offers increased robustness to hyperparameter changes, allowing models to converge stably under conditions where other normalization methods fail.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.