Reproduce “Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Andrei Mircea, Supriyo Chakraborty, Nima Chitsazan, Milind Naphade, Sambit Sahu, Irina Rish, Ekaterina Lobacheva

Investigating training dynamics, the paper uncovers "loss deceleration" as a universal phenomenon in language model training, mechanistically attributing it to "zero-sum learning" (ZSL) where per-example gradients destructively interfere. Scaling up models improves performance by mitigating ZSL, leading to lower loss at the onset of deceleration and faster improvement rates afterward.

Key insights

  • Loss deceleration, an abrupt slowdown in loss improvement, is a universal and consistent qualitative transition in language model training.
  • This deceleration is mechanistically caused by "zero-sum learning" (ZSL), characterized by destructive interference in per-example loss improvements and their underlying gradients.
  • Larger models mitigate ZSL after deceleration by maintaining more parameters with lower destructive interference, providing increased flexibility for sustained progress.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.