Reproduce “Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule” on OpenResearch

About this paper

Naoki Sato, Hideaki Iiduka

Researchers at Meiji University established the theoretical existence of a critical batch size that minimizes computational cost in Two Time-Scale Update Rule (TTUR)-based Generative Adversarial Network (GAN) training. They also provided methods to estimate lower bounds for this batch size, which were numerically validated to optimize training efficiency across various GAN architectures.

Key insights

• For TTURs with constant learning rates, the number of training steps required for GANs to reach stationary points is a monotonically decreasing and convex function of the batch size.
• The total computational cost (SFO complexity, i.e., steps multiplied by batch size) for GAN training is a convex function of batch size, guaranteeing the existence of a specific 'critical batch size' that minimizes this cost.
• Lower bounds for the critical batch size can be estimated using known hyperparameters and architectural information, enabling practitioners to select efficient batch sizes prior to extensive empirical tuning.

Summary by alphaXiv.