Reproduce “Scaling Laws for Uncertainty in Deep Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mattia Rosso, Simone Rossi, Giulio Franzese, Markus Heinonen, Maurizio Filippone
Rosso et al. empirically demonstrate that predictive uncertainties in deep learning models exhibit universal power-law scaling with dataset size and, in some cases, model size, across various architectures and uncertainty quantification methods, providing a theoretical link to Singular Learning Theory.
Key insights
- Predictive uncertainties (total, aleatoric, epistemic) in deep learning models consistently follow power-law scaling laws (U(x) N^γ) with respect to dataset size and sometimes model size.
- Epistemic uncertainty generally decreases with increasing data, challenging the notion that it becomes negligible in data-rich deep learning settings.
- Formal connections between generalization error from Singular Learning Theory and total uncertainty in Bayesian linear models provide a theoretical basis for observed scaling behaviors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.