Reproduce “Micro-Batch Training with Batch-Channel Normalization and Weight Standardization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille

Researchers from Johns Hopkins University developed Weight Standardization (WS) and Batch-Channel Normalization (BCN) to enable effective deep network training with micro-batches. These methods allow models to match or surpass the performance of large-batch Batch Normalization across various computer vision tasks, including achieving a 23.09% Top-1 error for ResNet-50 on ImageNet and up to a 2.1 APb improvement for Mask R-CNN on MS COCO compared to baselines.

Key insights

  • Batch Normalization's effectiveness stems from two factors: smoothing the loss landscape and avoiding harmful elimination singularities.
  • Channel-based normalization methods (like Group Normalization) can be augmented to mitigate micro-batch issues, but they still lack the global batch perspective needed to fully prevent elimination singularities.
  • Weight standardization can smooth the loss landscape by normalizing gradients and reducing the Lipschitz constant of the loss function with respect to weights, making optimization more stable.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.