Reproduce “Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients” on OpenResearch

About this paper

Yizhou Liu, Jeff Gore

Researchers at MIT propose "neural scaling universality," asserting that pre-training loss in large language models adheres to fixed power-law exponents —1/3 for time/dataset size and 1 for inverse width/depth —driven by generic mechanisms. Empirical analysis of Pythia, Chinchilla, and Farseer models supports these fixed exponents and derived optimal design principles, including a L ~ C-1/6 scaling for compute-optimal training.

Key insights

• LLM pre-training loss exponents are fixed and universal within a specific class, determined by generic mechanisms, shifting research focus from exponent variation to coefficient characterization.
• Dataset size scaling is a derived consequence of dynamic time scaling and optimization dynamics, indicating pre-training is "step-limited" rather than fundamentally "data-limited."
• Coefficients of the scaling laws are critical for practical LLM design and optimization, directly dictating optimal model shapes and compute-optimal training strategies.
• The derived optimal model shape and compute allocation strategies, based on these fixed exponents, align well with empirical observations across multiple LLM families.

Summary by alphaXiv.