Reproduce “BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping” on OpenResearch

About this paper

Taolin Zhang, Jinpeng Wang, Hang Guo, Tao Dai, Bin Chen, Shu-Tao Xia

BoostAdapter introduces a test-time adaptation strategy for Vision-Language Models (VLMs) that unifies insights from training-required and training-free methods through regional bootstrapping. This approach enhances VLM generalization and robustness across out-of-distribution and cross-domain benchmarks, improving average accuracy by 1.68% on ImageNet variants (ViT-B/16 backbone) over prior methods while maintaining high computational efficiency.

Key insights

• Gradient descent optimization of classifier weights using cross-entropy on high-quality augmented samples is functionally equivalent to feature retrieval from a cache composed of these samples, thereby bridging training-required and training-free TTA paradigms.
• The synergistic combination of instance-agnostic historical samples and instance-aware boosting samples (from regional bootstrapping) is crucial for comprehensive adaptation, exploiting both global domain shifts and fine-grained intra-sample information.
• Theoretical error bounds demonstrate that incorporating boosting samples, which provide higher local data density around a test instance, further reduces empirical risk and improves adaptation performance.

Summary by alphaXiv.