Reproduce “Decorrelated Batch Normalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lei Huang, Dawei Yang, Bo Lang, Jia Deng
Decorrelated Batch Normalization (DBN) is introduced to perform complete whitening of activations within mini-batches, addressing a theoretical limitation of standard Batch Normalization. DBN consistently outperforms standard BN across diverse architectures and datasets, leading to faster convergence and improved generalization.
Key insights
- The "stochastic axis swapping" problem, where principal components permute between mini-batches due to minor eigenvalue variations, is a critical issue that destabilizes deep learning training when using PCA whitening.
- ZCA (Zero-phase Component Analysis) whitening is essential for robust decorrelation in mini-batch settings because it rotates PCA-whitened data back, preserving consistent axis alignment across batches.
- Complete whitening of activations leads to a lower condition number for the Fisher Information Matrix, indicating improved optimization properties for deep networks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.