Reproduce “MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation” on OpenResearch

About this paper

Ronyu Zhang, Aosong Cheng, Gaole Dai, Yulin Luo, Jiaming Liu, Li Du, Huanrui Yang, Dan Wang and 3 more

A new method called MoASE++ enables deep neural networks to continually adapt to changing, unlabeled target data streams by explicitly disentangling domain-agnostic structural features from domain-specific textural features. This approach achieved a 39.7% mean error rate on ImageNet-C, improving over previous state-of-the-art by 0.8%, and reached 62.5% mean mIoU on Cityscapes→ACDC, surpassing prior methods by 0.5%.

Key insights

• Explicitly disentangling domain-agnostic structural features from domain-specific textural features, inspired by the human visual system, can effectively counter texture bias and enhance model robustness in continual adaptation.
• Dynamic, input-adaptive routing and activation sparsity based on varying activation degrees offers a controllable and interpretable mechanism to balance model plasticity (adaptation to new domains) and stability (retention of past knowledge).
• Stabilizing unsupervised continual adaptation by aligning a student model to an EMA-anchored teacher through on-policy augmented views and reverse KL divergence significantly mitigates confirmation bias and improves the quality of supervisory signals.

Summary by alphaXiv.