Reproduce “Efficient Open Set Single Image Test Time Adaptation of Vision Language Models” on OpenResearch

About this paper

Manogna Sreenivas, Soma Biswas

Researchers from the Indian Institute of Science (IISc) developed ROSITA, a framework for Open-set Single-image Test-Time Adaptation (OSTTA) of Vision-Language Models (VLMs). It combines a dynamic open-set identifier with a novel contrastive loss, significantly improving the identification of unknown classes and classification of known classes while adapting only lightweight LayerNorm parameters of the vision encoder, achieving state-of-the-art performance across diverse benchmarks.

Key insights

• Updating only LayerNorm parameters of the VLM's vision encoder is the most effective and lightweight strategy for continuous, single-image adaptation, outperforming prompt-tuning and full fine-tuning.
• The proposed ReDUCe loss, particularly its contrastive component for undesired samples ($L_U$), is crucial for enhancing separability between known and unknown classes by actively pushing their feature representations apart.
• A dynamic optimal threshold, determined by 1D LDA on a moving window of similarity scores, provides a robust and adaptive mechanism for open-set identification in sequential data streams.

Summary by alphaXiv.