Reproduce “Unraveling Batch Normalization for Realistic Test-Time Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zixian Su, Jingwei Guo, Kai Yao, Xi Yang, Qiufeng Wang, Kaizhu Huang
Researchers at Xi'an Jiaotong-Liverpool University, University of Liverpool, and Duke Kunshan University identified that reduced class diversity, not just sample quantity, causes inaccurate Batch Normalization statistics in small test-time mini-batches. They developed an adaptive Test-time Exponential Moving Average (TEMA) and a layer-wise rectification strategy, which achieved state-of-the-art performance and exceptional robustness to mini-batch sizes down to one on corruption benchmarks.
Key insights
- Inaccurate Batch Normalization statistics in small test-time mini-batches primarily stem from substantially reduced class diversity, rather than just the number of samples.
- Test-time Exponential Moving Average (TEMA) can effectively expand the 'effective sample pool' to increase class diversity for target statistics estimation.
- Robust test-time adaptation requires dynamically balancing the timeliness of statistics with their class diversity through an adaptive momentum for TEMA and a layer-wise rectification strategy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.