Reproduce “A Closer Look at Memorization in Deep Networks” on OpenResearch

About this paper

Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer and 3 more

The paper demonstrates that deep neural networks exhibit qualitatively distinct optimization behaviors when learning from real, structured data compared to noisy data. It shows deep networks prioritize learning simple, generalizable patterns before memorizing noise and that explicit regularization, like dropout, can selectively hinder the memorization of noise without impacting generalization.

Key insights

• Deep neural networks exhibit qualitatively different optimization behaviors when trained on real, structured data compared to purely random or noisy data.
• Gradient-based optimization in deep networks is "content-aware," prioritizing the learning of simple, generalizable patterns first before resorting to memorizing complex noise.
• Explicit regularization techniques, particularly dropout, can selectively hinder the memorization of noise without significantly compromising the network's ability to generalize on real, structured data.

Summary by alphaXiv.