Reproduce “Test-Time Distillation for Continual Model Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiao Chen, Jiazhen Huang, Zhiming Liu, Qinting Jiang, Fanding Huang, Jingyan Jiang, Zhi Wang
Tsinghua University researchers introduced Test-Time Distillation (TTD), a method for Continual Test-Time Adaptation that leverages a frozen Vision-Language Model as an external teacher to improve model adaptation. Their CoDiRe framework obtains state-of-the-art accuracy, reaching 60.69% on ImageNet-C and 80.38% on OfficeHome, by addressing challenges in external knowledge integration and constructing a robust blended supervisory signal.
Key insights
- Self-referential adaptation in CTTA can lead to unstable learning loops and error amplification under significant domain shifts, necessitating a stable external anchor signal.
- Directly using VLMs or naively blending predictions from heterogeneous models is suboptimal due to VLMs' generalist nature ("Generalist Trap") and inconsistent confidence distributions ("Entropy Bias").
- A dynamically weighted blended teacher, leveraging MSP for confidence and refined by optimal transport, effectively combines the complementary strengths of generalist VLMs and task-specific models for robust distillation targets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.