Reproduce “Scaling Laws for Deep Learning” on OpenResearch

About this paper

Jonathan S. Rosenfeld

Jonathan S. Rosenfeld's doctoral thesis at MIT systematically develops explicit scaling laws that quantitatively link deep learning generalization error to fundamental constituents like model size, data size, and pruning density. The work demonstrates that uncertainty and learning deficiency are the dominant error sources in current deep learning systems, paving the way for more principled design and future algorithmic advancements.

Key insights

• Generalization error in deep learning, for both dense models and pruned networks, follows predictable scaling laws that can be accurately approximated by simple rational functions.
• An 'error-preserving invariant' exists within pruned networks, demonstrating that combinations of depth, width, and pruning density can be interchanged to maintain constant generalization error.
• Uncertainty error (from data sampling stochasticity) and learning deficiency error (from optimization failures) are the primary limitations in current deep learning, not primarily a mismatch in the model's hypothesis class.

Summary by alphaXiv.