Reproduce “Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization” on OpenResearch

About this paper

Satrajit Chatterjee

The "Coherent Gradients" hypothesis posits that generalization in deep learning using Gradient Descent arises because per-example gradients for similar examples align, leading to reinforced updates that favor broadly applicable features over specific ones. Experiments with label noise and a "winsorized" SGD variant provide empirical support, showing reduced overfitting and faster learning of generalizable patterns.

Key insights

• Generalization in deep networks is a direct consequence of gradient dynamics, where coherent gradients from similar examples create stronger update signals, prioritizing generalizable patterns.
• The optimization process exhibits differential learning, rapidly fitting correctly labeled (pristine) examples via strong, coherent gradients before slowly accommodating noisy or random labels through weaker, less coherent gradient components.
• Actively suppressing weak per-example gradient components can reduce overfitting, suggesting a direct mechanistic link between gradient coherence and generalization.

Summary by alphaXiv.