Reproduce “Early-Learning Regularization Prevents Memorization of Noisy Labels” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, Carlos Fernandez-Granda
Researchers at NYU developed Early-Learning Regularization (ELR), a framework that leverages theoretical insights into early learning to prevent deep neural networks from memorizing noisy labels. The ELR and its enhanced version, ELR+, achieved state-of-the-art or competitive performance across various simulated and real-world noisy datasets, including Clothing1M and WebVision, while maintaining computational efficiency.
Key insights
- Early learning and memorization are fundamental phenomena in high-dimensional classification, not just empirical observations, and can be formally proven even in simple linear models.
- The proposed `log(1 - <p,t>)` regularization term effectively sustains the gradient contribution from clean labels and dampens the detrimental influence of noisy labels, preventing memorization.
- Robust target estimation using temporal ensembling and the synergistic combination of semi-supervised learning techniques (like weight averaging and two networks) can significantly enhance robustness to label noise.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.