Reproduce “A Constructive Prediction of the Generalization Error Across Scales” on OpenResearch

About this paper

Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit

A functional form is proposed to constructively predict neural network generalization error as a joint function of model and dataset size, demonstrating high accuracy in fitting empirical observations and extrapolating performance across a wide range of scales. This method provides a direct specification of the model configuration needed to achieve predicted error levels.

Key insights

• Generalization error for neural networks consistently follows power-law relationships with both model size and dataset size, with saturation observed at larger scales.
• A single, compact functional form can accurately describe the generalization error landscape across diverse tasks (vision, language), architectures (ResNet, VGG, LSTM), and optimizers (SGD, Adam).
• Small-scale measurements can be reliably extrapolated to accurately predict large-scale model performance, enabling principled design and efficient resource allocation without full-scale training.

Summary by alphaXiv.