Reproduce “Sample Efficiency of Data Augmentation Consistency Regularization” on OpenResearch

About this paper

Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S. Dhillon, Sujay Sanghavi, Qi Lei

Researchers from The University of Texas at Austin and Princeton University demonstrate that Data Augmentation Consistency (DAC) regularization is intrinsically more sample-efficient than Empirical Risk Minimization on Augmented Data (DA-ERM). DAC achieves lower generalization error by reducing effective problem complexity and exhibiting greater robustness to misspecified augmentations, reaching 70.56% accuracy on CIFAR-100 compared to DA-ERM's 69.40%.

Key insights

• Data Augmentation Consistency (DAC) regularization reduces the effective problem dimensions or complexity of the function class by enforcing invariance, leading to lower generalization error than DA-ERM.
• DAC demonstrates superior robustness to misspecified augmentations; its tunable regularization parameter allows it to converge to zero excess risk asymptotically, unlike DA-ERM which retains a persistent bias.
• The intrinsic efficiency benefits of DAC are more significant when using a limited number of augmented samples, highlighting its ability to extract more value from constrained augmentation budgets.

Summary by alphaXiv.