Reproduce “Prototype-Based Test-Time Adaptation of Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhaohong Huang, Yuxin Zhang, Wenjing Liu, Fei Chao, Rongrong Ji
A Prototype-Based Test-Time Adaptation (PTA) method is introduced for Vision-Language Models, allowing them to effectively adapt to distribution shifts by continuously accumulating class-specific knowledge via dynamic prototypes. This approach achieves state-of-the-art accuracy on various benchmarks, significantly outperforming many existing methods, while maintaining an inference speed that is 92% of the original CLIP model.
Key insights
- Continuous, cache-free accumulation of knowledge into dynamic prototypes from all test samples ensures stable performance and avoids cold-start issues or data loss.
- Adaptive weighting based on initial zero-shot confidence prevents early, potentially high-confidence, samples from excessively influencing prototype updates.
- Anchoring visually-derived prototypes to original text embeddings is critical to prevent semantic drift and maintain alignment within the VLM's joint embedding space.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.