Reproduce “A Solvable Model of Neural Scaling Laws” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Maloney, Daniel A. Roberts, James Sully
Researchers developed a solvable statistical model that accurately reproduces empirical neural scaling laws. This model reveals how power-law data spectra and feature maps extending these spectra drive predictable performance gains, while a finite latent data dimension (M) ultimately dictates when these scaling laws break down.
Key insights
- The emergence of neural scaling laws is fundamentally tied to a power-law structure in the eigenvalue spectrum of the underlying data, with feature maps playing a critical role in effectively extending this spectrum.
- A newly identified scale, the intrinsic latent data dimension (M), governs the limits of scaling laws, predicting their breakdown when model parameters (N) or training data size (T) approach or exceed M.
- Optimal L2 regularization fundamentally alters model behavior, eliminating the characteristic double descent phenomenon and resulting in a monotonically decreasing test loss with increasing model complexity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.