Reproduce “When and how epochwise double descent happens” on OpenResearch

About this paper

Cory Stephenson, Tyler Lee

Researchers from Intel Labs developed a solvable linear model and validated it with deep neural networks to explain the mechanisms behind epochwise double descent (EDD), characterizing its dependence on model size and data noise. They identified conditions under which early stopping is suboptimal and proposed a method to eliminate EDD by analytically solving the final layer's weights, which improved generalization performance.

Key insights

• Epochwise double descent primarily arises from the training dynamics learning noise-affected, quickly learned features first, followed by clean, informative features associated with slower learning rates.
• The occurrence of EDD and whether early stopping is suboptimal depends critically on a specific noise threshold in the training data and the model's degree of over/under-parameterization.
• The necessity of a critical noise level for EDD to appear creates tension with the Effective Model Complexity hypothesis, suggesting EDD is not solely a function of increasing model complexity.

Summary by alphaXiv.