Reproduce “Scaling Laws for Autoregressive Generative Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown and 11 more

Researchers at OpenAI investigated the generality of scaling laws for autoregressive generative models beyond language, finding that Transformer performance consistently improves with power-law scaling across images, video, multimodal data, and mathematical problems. This work provides an information-theoretic interpretation of loss components and offers guidelines for optimal resource allocation in large-scale model training.

Key insights

  • Power-law scaling of cross-entropy loss is a universal property of autoregressive Transformers, extending consistently across diverse data modalities (images, video, multimodal, math).
  • The cross-entropy loss can be interpreted as the sum of the true data entropy (irreducible loss) and the KL divergence between the true and model distributions (reducible loss).
  • Optimal resource allocation dictates a strong preference for increasing model size over training duration, with optimal model size scaling approximately as $N_{opt} \propto C^{0.7}$ with compute budget.
  • Improvements in generative loss, even when approaching the theoretical minimum, consistently translate to enhanced performance and representation quality on downstream tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.