Reproduce “LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Mateusz Kozinski, Horst Possegger, Rogerio Feris, Horst Bischof
LaFTer proposes a label-free and parameter-efficient method to adapt vision-language models like CLIP to new visual classification tasks by leveraging large language models to generate synthetic text descriptions and using unlabeled target domain images for unsupervised fine-tuning. The approach achieved an average absolute gain of 6.7% over zero-shot CLIP and often surpassed few-shot supervised methods across 12 diverse datasets.
Key insights
- The strong cross-modal alignment in pre-trained vision-language models allows a classifier trained solely on text embeddings to effectively categorize image embeddings.
- Large Language Models can generate high-quality, diverse textual descriptions of visual classes, effectively serving as synthetic supervision to train a visual classifier without any labeled images.
- Combining this text-only trained classifier with online pseudo-labeling and parameter-efficient fine-tuning enables robust and high-performing unsupervised adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.