Reproduce “Benign Overfitting in Two-layer Convolutional Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuan Cao, Zixiang Chen, Mikhail Belkin, Quanquan Gu
This paper theoretically establishes the conditions for benign and harmful overfitting in two-layer convolutional neural networks trained by gradient descent, revealing a sharp phase transition based on the data's signal-to-noise ratio. The analysis demonstrates that deep CNNs can achieve strong generalization performance despite perfectly fitting noisy training data, a finding that extends beyond the traditional Neural Tangent Kernel regime.
Key insights
- A sharp phase transition exists between benign and harmful overfitting in two-layer CNNs, governed by the signal-to-noise ratio (SNR) of the data and sample size.
- The analysis moves beyond the Neural Tangent Kernel (NTK) regime, allowing for significant weight changes from initialization, reflecting more realistic deep learning dynamics.
- A novel 'signal-noise decomposition' of convolutional filters allows for precise tracking of signal learning versus noise memorization during gradient descent training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.