Reproduce “AcTTA: Rethinking Test-Time Adaptation via Dynamic Activation” on OpenResearch

About this paper

Hyeongyu Kim, Geonhui Han, Dosik Hwang

AcTTA introduces a Test-Time Adaptation (TTA) method that dynamically modulates activation functions during inference to mitigate performance degradation from data distribution shifts. This approach consistently enhances the robustness of existing TTA methods across diverse architectures and corruption types, achieving stable optimization with higher learning rates, demonstrating efficacy with small batch sizes, and significantly reducing continual forgetting.

Key insights

• Activation functions, traditionally fixed, can be reinterpreted as adaptable components to mitigate domain shift, broadening the scope of Test-Time Adaptation beyond normalization layers.
• Dynamically adjusting activation slopes and centers enables robust gradient propagation and re-centering of nonlinear responses, overcoming the "zero-centered rigidity" of standard activations.
• Localizing adaptation to the activation space provides a compact and effective mechanism that enhances existing TTA methods, improves optimization stability, and reduces continual forgetting.

Summary by alphaXiv.