Reproduce “A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyi Huang, Andy Zhou, Zijian Lin, Mu Cai, Haohan Wang, Yong Jae Lee
A new domain generalization method, RISE, distills knowledge from CLIP's language component into smaller models, outperforming existing state-of-the-art approaches on multiple benchmarks. Researchers at the University of Wisconsin-Madison, University of Illinois Urbana-Champaign, and Imperial College London developed this technique, showing that language provides a more concise and domain-invariant semantic representation for robust generalization.
Key insights
- CLIP's text embeddings provide more concise and domain-invariant semantic representations than image embeddings, proving critical for guiding models to generalize.
- Using an ensemble of 80 diverse prompt templates for generating generic text embeddings is more effective than single prompts for robust semantic guidance.
- Each component of the proposed training objective (standard classification, image-to-image distillation, and the novel language-guided losses) contributes positively to overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.