Reproduce “Neural Scaling Laws for Deep Regression” on OpenResearch

About this paper

Tilen Cadez, Kyoung-Min Kim

An empirical study investigated neural scaling laws in deep regression models, demonstrating power-law relationships between generalization error and both training dataset size and model capacity across diverse architectures. The work, using a parameter estimation task in condensed matter physics, found scaling exponents for error reduction that can be notably higher, up to 2.3, compared to typical classification models.

Key insights

• Neural scaling laws, characterized by power-law relationships, consistently apply to deep regression models, showing a reduction in generalization error with increased data and model size.
• The observed scaling exponents for deep regression, particularly with dataset size (up to 2.3), are notably higher than those typically reported for classification and language models, suggesting substantial performance gains are possible.
• Robust statistical measures (geometric mean of test loss) and fine-tuned training schemes (adjustable learning rate) are crucial for accurately capturing and observing consistent scaling behaviors, especially for smaller models or very large datasets.

Summary by alphaXiv.