Reproduce “Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marco Mistretta, Alberto Baldrati, Marco Bertini, Andrew D. Bagdanov
This work introduces Knowledge Distillation Prompt Learning (KDPL), an unsupervised method that enables lightweight Vision-Language Models to learn generalizable prompts by distilling knowledge from a more powerful, frozen teacher VLM. The approach consistently improves zero-shot generalization across diverse unseen domains, datasets, and novel classes, even in class-agnostic settings where training class names are unknown.
Key insights
- Unsupervised knowledge distillation from a powerful teacher VLM enables a lightweight student VLM to learn generalizable prompts without requiring any ground-truth labels.
- A symmetric KL-divergence loss effectively transfers knowledge in unsupervised settings, improving student prompt performance and generalization.
- Effective prompt learning is achievable even in a 'class agnostic' scenario where neither instance labels nor training class names are known, via a dynamic class filtering strategy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.