Reproduce “Language Modeling at Scale” on OpenResearch

About this paper

Mostofa Patwary, Milind Chabbi, Heewoo Jun, Jiaji Huang, Gregory Diamos, Kenneth Church

Researchers at Baidu Research developed "Uniqueness," "Seeding," and "Compression" techniques to scale language models, achieving an 8.6x reduction in GPU memory and enabling training on up to 192 GPUs. This allowed for training on a 93GB Chinese corpus, yielding a 35% improvement in prediction accuracy with only a 1.25x increase in training time compared to a 3GB corpus.

Key insights

• Zipf's Law, which describes the statistical distribution of word frequencies, offers a critical opportunity to optimize memory and communication by only processing unique words per batch in distributed embedding layers.
• Controlled randomness in distributed training, specifically using a limited yet distinct set of seeds for sampled softmax, can maintain model accuracy while enabling substantial scalability gains.
• Combining algorithmic optimizations that leverage domain-specific insights (like word frequency distributions) with hardware-aware techniques (like mixed-precision communication) is essential for overcoming the unique bottlenecks in scaling large-vocabulary NLP models.

Summary by alphaXiv.