Reproduce “Learning Curves for Analysis of Deep Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Derek Hoiem, Tanmay Gupta, Zhizhong Li, Michal M. Shlapentokh-Rothman
A systematic methodology for utilizing learning curves to evaluate deep network classifiers is presented, enabling characterization of models by asymptotic error and data-reliance. This approach offers enhanced insights into the impact of architectural choices, pretraining strategies, optimization methods, and data augmentation on model performance across varying data scales.
Key insights
- The proposed `e_N` and `β_N` parameters provide a comprehensive and interpretable framework for comparing deep network design choices, revealing nuanced impacts beyond single-point performance metrics.
- Learning curves can be reliably estimated from a relatively small number of trained models (e.g., as few as four data points), making systematic analysis computationally accessible for diverse research groups.
- The systematic analysis demonstrates that various design decisions, such as pretraining and architecture, significantly influence both a model's ultimate performance and its data efficiency (data-reliance).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.