Reproduce “A Neural Scaling Law from the Dimension of the Data Manifold” on OpenResearch

About this paper

Utkarsh Sharma, Jared Kaplan

Sharma and Kaplan (Johns Hopkins University) present a theoretical framework explaining neural scaling laws, establishing that the power-law dependence of neural network loss on model size (L ∝ N ^-α) is fundamentally determined by the intrinsic dimension (d) of the data manifold. Their work empirically validates the relationship α ≈ 4/d for ReLU networks across synthetic and real-world datasets, moving beyond empirical observation to a principled understanding of deep learning scalability.

Key insights

• The power-law scaling of neural network loss with model size is fundamentally determined by the intrinsic dimension of the data manifold relevant to the task.
• For ReLU networks using MSE or cross-entropy loss, the scaling exponent α is approximately 4/d, directly linking data complexity to the benefits of increased model scale.
• The intrinsic dimension of network activations remains largely constant within the scaling regime, suggesting it is a stable property of the data and task, independent of the model's capacity.

Summary by alphaXiv.