Reproduce “L2 Regularization versus Batch and Weight Normalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Twan van Laarhoven
Radboud University researchers clarify that L2 regularization in deep neural networks with normalization layers primarily controls the effective learning rate by modulating weight scale, rather than directly simplifying the function. Their analysis establishes quantitative relationships where the optimal learning rate is proportional to 1/λ for SGD/Momentum and 1/√λ for adaptive optimizers.
Key insights
- L2 regularization, when used with normalization techniques, does not directly regularize the learned function but rather acts as a mechanism to control the scale of the network's weights.
- The scale of weights critically impacts the *effective learning rate* experienced by first-order optimization methods, as gradients in normalized layers scale inversely with weight magnitude.
- Second-order optimization methods like Newton's method are inherently scale-invariant, meaning their effective learning rate is not modulated by weight scale or L2 regularization in this context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.