Reproduce “Fantastic Generalization Measures and Where to Find Them” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, Samy Bengio
This empirical study from Google evaluates over 40 complexity measures on more than 10,000 deep neural networks to identify which measures reliably predict generalization performance. The research finds that sharpness-based measures of the loss landscape and certain optimization-related metrics are the most consistent predictors of generalization, while many other measures exhibit spurious correlations driven by hyperparameter settings.
Key insights
- Many complexity measures that appear correlated with generalization are in fact spuriously correlated, primarily reflecting the influence of hyperparameters.
- Sharpness-based measures of the loss landscape consistently exhibit the strongest correlation with generalization performance.
- Optimization dynamics and noise injection during training (e.g., through dropout or batch size) play a crucial role in improving generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.